DeliverBench

End-to-end professional project delivery

DeliverBench.

Benchmarking AI agents on delivering realistic multi-stage professional projects.

Agents work in a persistent workspace, incorporating new materials and client feedback while reusing, revising, and integrating their own earlier artifacts.

Humanlaya Data Theseus Lab

11Professional projects
7Domains
150Interdependent subtasks
111Required final deliverables

The evaluation unit

Individual tasks.
A shared project objective.

Later tasks inherit the agent’s own artifacts. New evidence and client feedback require revisions across files, until the final package is ready for handoff.

Personal assistance, task execution, and project delivery compared. DeliverBench follows interdependent subtasks and evolving inputs in a persistent workspace. The lower panel illustrates stage dependency violations, ignored new inputs, cross-file inconsistency, unmet project requirements, and other delivery risks.
From personal assistance to professional project delivery. DeliverBench tracks dependencies, revisions, and the joint state of the final package. The illustrated delivery failures are not exhaustive.

Results

How well do agents
deliver?

Seven model–harness systems, evaluated across all 11 projects. Mean final-delivery scores range from 40.39 to 65.00 out of 100.

October 2026 results77 system–project outcomes

Leaderboard

Two metrics, one view. Bars share a 0–100 scale.

Ranked by final delivery. The selected metric is emphasized.

Seven evaluated systems, ranked by final-delivery score.
RankModel / harnessFinal deliveryScore / 100 · mean of 11 projectsSubtask pass rate% of 150 checkpoints scoring ≥ 90
01
Claude Opus 5Claude Code
 Final delivery
65.00
 Subtask pass rate
50.00%
02
GPT-5.6 SolCodex
 Final delivery
63.75
 Subtask pass rate
36.00%
03
Qwen 3.8 MaxQwen Code
 Final delivery
60.39
 Subtask pass rate
25.33%
04
Kimi K3Kimi CLI
 Final delivery
59.39
 Subtask pass rate
26.00%
05
DeepSeek V4.1 FlashClaude Code
 Final delivery
58.18
 Subtask pass rate
26.67%
06
GLM 5.2Claude Code
 Final delivery
44.46
 Subtask pass rate
14.00%
07
Gemini 3.8 FlashGemini CLI
 Final delivery
40.39
 Subtask pass rate
13.33%
Different metrics, different scopes.

Final delivery evaluates the submitted project package. Subtask pass rate counts checkpoint submissions scoring at least 90; it is not a final-package acceptance rate. Model and harness are evaluated together.

Inside a professional project

P02 Syndicated refinancing

Project example · v3.1.0

Act as the deal execution team for a digital infrastructure refinancing. Bring together due diligence, financing terms, financial models, and lender feedback into a coherent package for signing preparation.

  • 13 scored subtasks
  • One persistent workspace
  • 6 final deliverables

The agent’s own work, carried forward

New materials arrive in stages. Earlier artifacts stay available for reuse and revision.

Artifact handoffConditions & contextBusiness feedback
01

Due diligence

Diligence materials

Uses earlier workFinancial records and late diligence evidence.

ProducesAn earnings-quality report with the adopted earnings basis and unresolved reconciliation items.

From
Source materials
To

Uses earlier workPermitting, environmental and insurance records.

ProducesA review that distinguishes confirmed coverage and permissions from outstanding conditions.

From
Source materials
To

Uses earlier workTechnical records for the asset portfolio and network.

ProducesAsset assessments with capital-expenditure risks, cost evidence and remaining uncertainties.

From
Source materials
To

Uses earlier workTax records and the proposed ownership structure.

ProducesA tax memo explaining the relevant scenario and the limits of its application.

From
Source materials
To
02

Financing proposal

Lender feedback

Uses earlier workThe agent’s financial, environmental, technical and tax diligence.

ProducesA lender presentation bringing together debt-service capacity, financing proposals and open issues.

FromTo

Uses earlier workThe lender presentation, supporting diligence and unresolved execution conditions.

ProducesA committee pre-read that identifies decisions, alternatives and conditions for proceeding.

FromTo

Uses earlier workEarlier financing analysis, committee issues, lender feedback and management comments.

ProducesA revised term sheet for modelling, internal review and further lender discussions.

FromTo
03

Modelling & review

Board decisions

Uses earlier workFinancing terms, technical risks and the applicable tax assumptions, checked against source evidence.

ProducesA recalculating financing model with traceable assumptions and debt-service calculations.

FromTo

Uses earlier workThe agent’s own core financial model.

ProducesTwo scenario models, an assumptions memo and a record of the changes actually made.

FromTo

Uses earlier workScenario results, the earlier committee pack and current financing terms, alongside new board decisions.

ProducesA final committee pack, a lender invitation and a working-group status note for their respective audiences.

FromTo
04

Syndication & handoff

Order updates · signing instructions

Uses earlier workScenario analysis, invitation materials and open conditions, plus new orders and lender clarification questions.

ProducesA syndication report answering the questions, updating execution status and identifying changes still needed.

FromTo

Uses earlier workMarket evidence, model assumptions and current financing status. Financing context does not replace independent market evidence.

ProducesA valuation-multiples summary that states the applicability and limits of each method.

FromTo

Uses earlier workThe agent’s models, assumptions, syndication update and valuation summary, together with current terms and signing instructions.

ProducesA six-file handoff with the current arrangements and outstanding conditions clearly identified.

FromTo
Final package
Final handoff

Six files. One delivery package.

Current arrangements, supporting analysis and outstanding conditions.

  • Base-case model
  • Assumptions memo
  • Syndication update
  • Valuation summary
  • Signature page
  • Draft commitment letter

Arrows show how earlier work informs later deliverables, not parallel execution. Earlier conclusions can be checked and corrected.

At each submission

Checkpoint evaluation

Each subtask submission is preserved and scored at its checkpoint.

T01T02T03T04T05T06T07T08T09T10T11T12T13
At final handoff

Final delivery evaluation

The final submitted package is evaluated independently for artifact quality, cross-file consistency, and package acceptance.

Artifact qualityCross-file consistencyPackage acceptance

Later corrections can improve the final delivery; earlier checkpoint scores remain unchanged. Signing preparation does not mean that financing has been approved or completed.

Selected cases

What breaks
at handoff?

Information can be recorded in one artifact while a related file still says something else. Preserved outputs make these delivery failures visible.

P03: Conflicting audit conclusions. The summary records unverified completeness while the detailed workpaper says the sample is complete.
Audit · Final package
Evidence & context

The summary records unverified population completeness and period coverage. The workpaper for the same control retains a positive sample-completeness and selection statement. The inconsistency lies in the supporting evidence state, not identical wording in the two files.

Evidence: audit workbook, ITGC-CE-2023 O17/Q17 and ITGC-CE-06 E12; issue register, row 10. Excerpts abbreviated and translated.

P07: Correction stated, not applied. The correction note states 4,249.2, but the operative payment schedule retains 4,841.5. Amounts in CNY 10,000.
Finance · T16 checkpoint
Evidence & context

Units: CNY 10,000. The T16 revised workbook states the corrected amount in its revision note but retains 4,841.5 in the capital-expenditure schedule. The next calculation still references that cell. This example concerns a checkpoint artifact, not the final package.

Evidence: T16 revised financing workbook, revision note B13; capital expenditure sheet F6 and formula G6 = E6 + F6.

P11: Stale handoff status. Calibration data is received and acknowledged in the closing memo, while the task register still reports missing calibration as a blocker.
Data Operations · Final package
Evidence & context

The closing memo acknowledges the additional calibration data and schedules execution for the next window. The task register still identifies missing data as the blocker. Receiving the input and completing the remaining execution are distinct states.

Evidence: closing memo, lines 34/53/65; delivery register, task sheet F7/I7/J7. Excerpts abbreviated and translated.

P04: Superseded decision retained. The requirement replaces the dressing table with a chest of drawers, but the current decision register retains the dressing table.
Design · Final package
Evidence & context

The requirements register adopts the replacement. The decision register still preserves the earlier direction and follow-up status. The failure concerns the active project record; it does not establish that physical construction was carried out incorrectly.

Evidence: project management workbook, REQ-064 and DEC-022; current decision status and follow-up columns.

Selected examples, not an exhaustive taxonomy or an estimate of failure prevalence.

Benchmark construction

Real projects.
Expert-guided reconstruction.

Domain experts provide project materials and help reconstruct stages, information releases, and artifact dependencies. Instructions and rubrics are then reviewed for business fidelity, answerability, and alignment.

The five-step DeliverBench construction process: collect materials, reconstruct workflows, model dependencies, write instructions and rubrics, then conduct expert review.
Original project materials, reconstructed workflows, and scenario feedback are distinguished during construction.
  1. 01Collect materials
  2. 02Reconstruct workflows
  3. 03Map dependencies
  4. 04Write tasks & rubrics
  5. 05Expert review

Execution & evaluation

The agent’s own work.
Carried forward.

The same agent works in a persistent workspace. Later subtasks consume and revise earlier outputs as evidence and feedback arrive. Reference outputs are not inserted between tasks.

Continuous execution in a persistent workspace with new data and feedback; separate checkpoint and final-package evaluation.
Intermediate submissions are preserved. Final delivery is assessed on the package intended for handoff.

Three views of final delivery

Diagnostic perspectives, not an extra layer of score weights.

Artifact quality

Is each deliverable correct, complete, professional, and usable?

Cross-file consistency

Do related files agree on facts, values, conclusions, and current status?

Package acceptance

Does the complete handoff satisfy the project’s scope and acceptance requirements?

Project coverage

Different professions.
Shared delivery demands.

From a financial investigation to a safety case, each project combines dependent assignments and a multi-file handoff. Project IDs keep the focus on the work.

Finance 4Medicine 2Design 1Data Operations 1Strategy 1Audit 1Law 1
661 input files

423 arrive after initialization. New evidence and client feedback are part of the project timeline, not just a larger initial prompt.

Figure viewer
Fit