AI Agent · Evaluation engineering
A well-known enterprise Client anonymized
From expert tasks to executable agent evaluations.
Expert task annotation, reward scripts, and rollout engineering for scientific AI agents, informed by the public Terminal-Bench-Science benchmark.
Business challenge
Scientific expertise had to become work an agent could execute and a team could evaluate. Instructions, datasets, runtime environments, output artifacts, and scoring rules needed to agree without exposing the answers to the agent.
Public reference: Terminal-Bench-Science
What we delivered
- Coordinated expert task annotation and translated domain requirements into task instructions, output contracts, and reward scripts.
- Packaged tasks into containerized environments, connected agent rollouts to artifact collection and evaluation, and separated public inputs from hidden grading assets.
- Built validation tools for task structure, output and grader alignment, scoring behavior, and detailed evaluation records so experts and engineers could inspect the same evidence.
Expert knowledge becomes executable evaluation.
- Expert tasksInstructions, data & output contracts
- Task environmentContainers & runtime constraints
- Agent rolloutTask execution & run records
- Output artifactsFiles, code & research outputs
- Reward evaluationChecks, scores & detailed logs
Evaluation records inform expert review and task refinement.
A system built for the work.
Executable task packages, reward scripts, rollout workflows, and traceable evaluation records: a practical foundation for testing agent behavior and refining tasks with domain experts.
Business and engineering, together.
Our engineers worked through the scientific workflow with experts, resolving the details of inputs, deliverables, runtime constraints, and scoring together. Domain knowledge became running software and inspectable evidence.