Evals Framework to Benchmark and Improve Finance Agents
InferEdge has built an evaluation system to test and benchmark models and agents for various finance workflows. This can be used by enterprise to evaluate their harness or model setups or a combination of both.
The original set was built to evaluate and improve InferEdge’s in-house investment workflows agent, MIND and we continue to use it to improve our agentic capabilities.
How the Evalset was built
Evals are generated using real-world use cases or workflows. Each case is human reviewed.
To scale this approach, we run queries on our agent runners. Currently, we support 3 different runners: MIND agent, a basic web search + SEC agent and a runner from a public finance benchmark.
Every query generates traces and agent session transcripts in our observability tool. We manually verify the traces and outputs and use them as a reference point to add examples or cases to the dataset.
The final outputs and rubrics are always verified and committed to the eval set by a person.
We do not let LLMs write or update evals - this invariably results in cases that the LLM can pass almost 100% (other LLMs may fail). We have confirmed this behaviour with multiple LLMs and suspect this happens due to reward hacking or built-in bias.
Eval Framework
Most of the key components needed to run evals are covered in our framework.
- Evaluators: Different types of evaluation criteria and parameters to score the output or rubric
- Gauges: Efficiency measurements like token consumption
- Experiment Runner: Connect and run any agent harness or model
- Agent Harness: this is the agent + model combination under evaluation
- Results and Trace Visualizer: dashboard to check datasets, experiment results and trends over time
See the diagram on the next page for a high level overview/flow of how this works.
Evaluators
Evaluators are scoring functions that compare the expected and actual for each example.
| Type | Description |
|---|---|
| Content Accuracy | Checks for numeric and factual content. Also ensures citations/data sources are correct |
| Trajectory Accuracy | Checks for logic and outcome of intermediate steps |
| Presentation | Validates that the outcome adheres to standard financial data practices, such as precision and alignment. |
Based on collected data for each session (each example runs in its own fresh session), evaluators configured for the example calculate respective scores and aggregates.
Gauges
Unlike evaluators, which score the performance, gauges take measurements from the runs, which is essentially the trade off that one needs to balance. The goal of a good eval is not just to reward a good output but also to measure the cost associated with building that output.
| Type | Description |
|---|---|
| Token consumption | input, output, cached tokens and cost associated with it |
| Time Taken | Time taken for overall run of an example, tool-wise breakdown, LLM completion times |
| Tool Call Efficiency | Number of tool calls |
Experiment Runner
We have built a lightweight runner using Arize Phoenix as the base. This is flexible and can be used to connect to any agent harness or model that is under evaluation, including a harness that you have built. The runner drives experiments against a combination of the agent + model.
The evalset is provided by us to benchmark and run the agents/models. Examples from the dataset are run independently in parallel against instances of the harness and session transcripts are collected along with other metrics such as tokens, timing information and cost. This output is fed to the evaluator for measurement.
Agent Harness
Any agent harness can be used, as long as it can push data into open telemetry or an observability tool. Currently we support only Arize Phoenix (open source).
The experiment runner can work with any agent harness with a minimal adapter requiring two things: a way to initiate runs and to collect run traces and output.
We have written adapters for the following and you can use these out of the box.
- Inferedge MIND - in-house investment workflow agent
- A minimal agent which has access to SEC Filings and web search
- Benchmark harnesses like BigFinanceBench, Frontier Finance and FinRetrieval
Results and Trace Visualizer
A visualizer to enable quick glances at experiments and drill-downs into individual examples:
- Experiment View: Top-level experiment results, scores of each evaluator against the examples of the experiment along with its comparison with a baseline experiment run
- Transcript view: Agent session transcript viewer consisting of individual assistant messages, tool calls and reasoning
- Workspace view: The session workspace directory with all files that the agent has created
How InferEdge uses this to improve MIND
We created a framework that we call Eval Driven Development (EDD). The MIND agent is mainly a composite of skills and MCP tools. Accordingly, EDD is used to improve the agent skills or MCP tool definitions.
Eval-Driven Development (EDD) applies the red → green discipline of Test-Driven Development (TDD) to agents. An agent's behaviour is the product of its prompt, skills, tools, tool-descriptions, model, and context - any of which can change and silently regress.
EDD makes the change loop explicit: you express the behaviour you want as an eval case before you change the agent, watch it fail, make the change, and watch it pass. The diagram below explains the flow:
The EDD loop is utilized for all changes made to the agent:
- Building or changing a capability (skill or tool)
- Turning a production failure into a regression case
- Assisted review of failures and diagnosis
Measured Impact and Improvements
Using EDD, we were able to improve the document retrieval capabilities of Inferedge MIND on our evalset:
| Metric | Baseline | EDD Applied | Improvement |
|---|---|---|---|
| Accuracy | 63% | 79% | 16% |
| Tool Error Rate (per example) | 1.21 | 0.44 | 63.6% |
| Tool Calls (per example) | 18.37 | 10.67 | 41.9% |
Additionally, MIND’s score on BigFinanceBench improved from 40% to 70%. Changes that drove these improvements were skill, prompts and tool surface changes guided by this EDD framework.