Scaling Expert Judgment with AI

November 29th, 2023Risk Assessment Paper Report Generation Paper Press Coverage

Summary

The Problem: Expert judgment becomes a bottleneck as a business grows. Every new case needs someone to interpret the evidence, weigh competing signals, and explain a conclusion. Our work with Traive Finance brought this problem into focus in agricultural lending, where assessing a farmer’s ability to repay requires understanding how crop performance, expected farmer income, and existing debt fit together when making a credit decision.

Our Approach: We developed a method that combined specialist knowledge with the data for each case:

Capturing the Reasoning: We captured how credit experts reasoned over agricultural and financial factors (rather than only what they conclude).
Guiding the Assessment: We reflected the reasoning process of the experts into AI models, to guide their assessment approach.
Explaining the Result: The models generated both a risk assessment and a written report explaining the reasoning behind the assessment; this allowed expert analysts to read, check, and act on AI recommendations more efficiently.

The Impact: The generated reports explained what increased or reduced credit risk and how the factors interacted, giving analysts a written assessment they could review rather than a score to blindly trust. We found that credit analysts preferred the AI generated reports in up to 90% of cases. Our work shows a way to turn specialist knowledge into useful AI-generated analysis, so an organization can scale expertise, without scaling head-count.

The Problem

Expertise does not scale; the reasoning of experts itself has to become reusable.

Expert judgment is the part of any business that is hardest to grow; each new case requires someone to interpret the evidence and explain a conclusion well enough that a colleague can act on it. Hiring experts is difficult, and capacity grows slowly and unevenly.

Agricultural lending is a clear example of where expert judgment is critical. Deciding whether a farmer can repay a loan requires reasoning about how different factors interact: crop performance shapes expected income, expected income has to be set against debt already owed, and a change in any one of them moves the others (see Fig. 1).

Agricultural evidence Financial evidence

Fig. 1: A simplified view of the reasoning a credit analyst performs.

To build an AI tool that replicates expert judgment, we need to capture the reasoning process rather than just the final decision; an AI-generated credit score alone is not enough, because someone still has to work out why the number came out the way it did before anyone can act on it.

Our Approach

To build an AI tool that replicates expert judgment, we started by capturing how credit experts connect the evidence; we then integrated those relationships into an AI system that could assess each case and generate an explainable report.

1. Capture how the experts reason, not just what they decide: We sat with credit specialists and recorded the structure of their judgment: which agricultural and financial factors they consider, in which direction the influence runs, and how a change in one revises their reading of another.

2. Train AI to emulate expert reasoning: We then developed an AI system that replicated the expert reasoning, ensuring that it could assess each case in a manner consistent with expert judgment.

3. Generate a report the analyst can review AI reasoning: In addition to the risk assessment, the AI system produced a brief report explaining its reasoning that an analyst could review. Each report set out what raised risk, what reduced it, and how the factors influenced one another (see Fig. 2).

Expert reasoning How human experts connect agricultural and financial factors

Case data The evidence belonging to one application

→

Structured relationships AI emulation of expert reasoning, applied to this case

→

Risk assessment Consistent with expert reasoning

Written report Explaining what raised risk, what reduced it, and how the factors interact.

Fig. 2: Because the assessment approach was mopdeled on human experts, the explanations generated in the reports were faster for human analysts to review and action.

The Impact

Analysts received credit risk scores aligned precisely how expert assessments would have scored them; in a blind evaluation, they overwhelmingly preferred our AI generated reports to those generated by their peers.

up to 90% of generated credit risk reports preferred by human credit analysts

We asked human credit analysts to evaluate our AI generated reports compared those produced by their peers; the analysts preferred the AI generated reports in up to 90% of the cases.

What made the reports usable was their structured explanation of risk factors, which allowed an analyst to review the evidence, and agree/disagree with conclusions when adjudicating the final decision. While this work focused on Agricultural lending, nothing in the approach depended on it: wherever judgment is the constraint and the reasoning can be articulated, the same method lets an organization handle more cases with the expertise it already has.