Argo-Bench
Paper TextQL
Map data © OpenStreetMap contributors
[ A benchmark by TextQL ]

Argo-Bench

A frontier benchmark for enterprise-scale data science.

Read the paper ↗ Explore the world View dataset ↗
010203040506070$8$7$6$5$4$3$2$1$0Opus 5.5Gemini 4 ArgonGPT-6 AstraSonnet 5.5GPT-6.1 SolMuse WonderDeepSeek V4.1 1017Kimi K3Gemini 3.8 FlashMuse Spark 1.3GPT-6 Luna58.433.3% of taskssolved
[ Leaderboard · v1.4.2 · 210 enterprise tasks ]

Measuring agents on long-horizon data science.

010203040506070$8$7$6$5$4$3$2$1$0Average cost per taskArgo-Bench scoreOpus 5.5Gemini 4 ArgonGPT-6 AstraSonnet 5.5GPT-6.1 SolGPT-6.1 TerraGrok 4.7Muse WonderDeepSeek V4.1 1017Kimi K3Gemini 3.8 FlashHaiku 5.5GLM 5.3 FlashMuse Spark 1.3GPT-6 Luna58.433.3% of taskssolved
  1. 1 Claude Opus 5.5 58.4
  2. 2 Gemini 4 Argon 53.9
  3. 3 GPT-6 Astra 50.7
  4. 4 Claude Sonnet 5.5 49.9
  5. 5 GPT-6.1 Sol 47.8
  6. 6 GPT-6.1 Terra 44.2

Mean task score out of 100, each model at its highest reasoning effort, ±2 to 8 points at 95% confidence. A task counts as fully solved at a score of 95 or more.

[ Leaderboard · v1.4.2 · 210 tasks ]

Every model, every setting

1Claude Opus 5.5 Extra high 58.433.3%$4.8474,00584
2Gemini 4 Argon High 53.933.8%$7.53154,926120
3GPT-6 Astra Extra high 50.725.7%$2.7614,83223
4Claude Sonnet 5.5 Extra high 49.926.2%$3.8086,14777
5GPT-6.1 Sol Extra high 47.822.4%$0.5017,37124
6GPT-6.1 Terra Extra high 44.223.8%$0.2718,18528
7Grok 4.7 Extra high 38.819.5%$6.2834,46494
8Muse Wonder Extra high 37.617.6%$7.3180,117144
9GPT-6 Sol Extra high 34.915.7%$0.8620,93639
10DeepSeek V4.1 Flash 1017 Max 31.813.8%$0.2487,06498
11Kimi K3 Extra high 27.212.4%$4.5972,16087
12Gemini 3.8 Flash High 24.514.8%$3.99109,478198
13DeepSeek V4.1 Flash Max 23.215.7%$0.54195,286124
14Claude Haiku 5.5 Extra high 23.210.5%$1.02145,21172
15GLM 5.3 Flash Extra high 19.810.5%$0.3679,55093
16Muse Spark 1.3 High 18.38.6%$4.0467,853177
17Claude Sonnet 5 Extra high 14.46.2%$2.1570,60156
18GPT-6 Luna Extra high 12.95.7%$0.05741,61139
19Claude Haiku 4.5 5.20.9%$0.2013,29632
Score is the mean task score out of 100; solved is the share of the 210 tasks scoring at least 95. Cost is model API spend per task, excluding warehouse queries. Tokens are output tokens, reasoning included. Steps are model calls. A hollow dot marks open weights. Settings run up to extra-high effort; by model, each model is shown at its highest setting. Claude Sonnet 5, GPT-6 Sol, Claude Haiku 4.5, and DeepSeek V4.1 Flash, which their successors replaced on the chart, are listed here only, as is Gemini 4 Argon at low effort. Qwen 3.8 Max, in the paper, is not listed: many of its runs ended at its context limit.
[ The videos ]
The world1:06 Explore one evening in the simulated city and discover how its marketplace works.
Fraud study1:00 Explore how an agent catches couriers who steal orders.
[ The benchmark ]

The first benchmark to evaluate enterprise decisions in dollars.

Argo-Bench simulates a year of a New York food-delivery platform, down to fraud, incentives, and its Oracle ERP. Agents decide who to ban, where to spend, and more.

210
Tasks in five business areas
235
Tables in Oracle EBS form
7.49B
Warehouse rows
81M
Simulated orders in 2024
  1. Trust & safety 67 tasks

    The best offers vanish before couriers can tap. Ban scripted accounts.

    Bans and holds
  2. Marketplace 46 tasks

    Finance wants to free up $9.6M in driver bonuses. Decide where to trim.

    Budgets and plans
  3. FP&A 54 tasks

    A pay floor took effect in April. Forecast May's net pay adjustments.

    Forecasts
  4. Accounting 23 tasks

    The city says couriers were underpaid. Find every short week.

    Reported figures
  5. Growth 20 tasks

    Did membership deals pay for themselves? Publish the dashboard.

    Dashboards
How it works →
[ Example task · trust & safety ]

Which restaurants had their payouts hijacked without noticing?

Claude Opus 5.5 learned the takeover pattern from reported cases and froze every silent victim's payouts, leaving hundreds of honest bank changes alone.

Ten of 69 model settings get this task right.

More from the traces
[ Claude Opus 5.5 · high effort ]Score 100
  1. 1,848 payout-account changes through the merchant portal in 2024
  2. 34 reported by the restaurant and restored: in every one, the contact e-mail changed 0–2.2 days before the bank account
  3. 14 never reported, with the same fingerprint. The next closest gap is 8 days
Payouts frozen on all 14 No honest restaurant frozen
[ Example task · mer-71 · the solution ]

How Claude Opus 5.5 solved it

Score
100
Tool calls
89
Time
14.8 min
Model cost
$3.53
The task, as the agent received it

Not every storefront whose payout account was hijacked has noticed. Trust & safety wants the payouts frozen on every storefront whose payout account was changed this year in the way the reported takeovers were: the contact e-mail changed within a couple of days of the bank account, or the new account is one we already pay another, unrelated supplier into, and whose account was never restored. Storefronts change their bank for ordinary reasons all year, and restaurant groups legitimately share one payout account across their brands; leave those alone. Hold each storefront's payouts for account takeover (identified by the payables vendor ID).

  1. 1

    Find where payout-account changes are recorded

    The audit trail logs every change to a supplier’s payment instrument, with its reason: changes made through the merchant portal, and the ones support restored after a restaurant reported them.

    run_sql 
    SELECT SOURCE_TABLE, SOURCE_KEY_NAME, COLUMN_NAME, CHANGE_TYPE, CHANGE_REASON,
       COUNT(*) n, MIN(CHANGED_DATE), MAX(CHANGED_DATE)
FROM XX_AUDIT_TRAIL
GROUP BY 1, 2, 3, 4, 5 ORDER BY 1, 2, 3, 4, 5
  2. 2

    Tie each change to a storefront

    A change is keyed on the external payee; three joins lead from it to the supplier, its payables vendor ID and its party, where the contact e-mails live.

    run_sql 
    SELECT a.AUDIT_ID, CAST(a.SOURCE_KEY_VALUE AS INT64) ext_payee_id,
       CAST(a.OLD_VALUE AS INT64) old_instr, CAST(a.NEW_VALUE AS INT64) new_instr,
       a.CHANGE_REASON, a.CHANGED_DATE, p.PAYEE_PARTY_ID, ss.VENDOR_ID, s.VENDOR_TYPE_LOOKUP_CODE
FROM XX_AUDIT_TRAIL a
JOIN IBY_EXTERNAL_PAYEES_ALL p ON p.EXT_PAYEE_ID = CAST(a.SOURCE_KEY_VALUE AS INT64)
JOIN AP_SUPPLIER_SITES_ALL ss ON ss.VENDOR_SITE_ID = p.SUPPLIER_SITE_ID
LEFT JOIN AP_SUPPLIERS s ON s.VENDOR_ID = ss.VENDOR_ID
WHERE a.SOURCE_TABLE = 'IBY_PMT_INSTR_USES_ALL'
  3. 3

    Learn the fingerprint from the reported takeovers

    For each change, the nearest change of the storefront’s contact e-mail. All 34 reported and restored takeovers had one 0 to 2.2 days before the bank account changed. Among the changes never restored, 14 match; the next closest is 8 days out.

    run_python 
    ch = au[au.CHANGE_REASON.str.contains('PORTAL')].copy()
rest = set(au[au.CHANGE_REASON.str.contains('RESTORED')].ext_payee_id)
ch['restored'] = ch.ext_payee_id.isin(rest)
# e-mail changes per party: the retired contact points
ina = cp[cp.STATUS == 'I'][['OWNER_TABLE_ID', 'LAST_UPDATE_DATE', 'EMAIL_ADDRESS']] \
    .rename(columns={'OWNER_TABLE_ID': 'PARTY_ID', 'LAST_UPDATE_DATE': 'em_chg'})
m = ch.merge(ina, left_on='PAYEE_PARTY_ID', right_on='PARTY_ID', how='left')
m['dt_days'] = (m.em_chg - m.CHANGED_DATE).dt.total_seconds() / 86400
m['abs_dt'] = m.dt_days.abs()
# the closest e-mail change to each bank change
mm = m.sort_values('abs_dt').groupby('AUDIT_ID').first().reset_index()
print(mm.groupby('restored').abs_dt.describe())
  4. 4

    Rule out the look-alikes

    About 250 other changes land on an account another supplier already uses. They carry no e-mail change and reuse no known mule account, and their count is what chance alone would give, so they are left alone. Restaurant groups set their shared account when they joined, not through a later change.

    run_python 
    rows = []
for _, r in ch.iterrows():
    # who else was paid into the new account when it was switched to?
    co = u[(u.INSTRUMENT_ID == r.new_instr) & (u.VENDOR_ID != r.VENDOR_ID)
           & (u.CREATION_DATE < r.CHANGED_DATE)
           & (u.END_DATE.isna() | (u.END_DATE >= r.cd))]
    rows.append(dict(AUDIT_ID=r.AUDIT_ID, n_co=co.VENDOR_ID.nunique(),
                     co_types=list(co.vtype.unique())))
ch2 = ch.merge(pd.DataFrame(rows), on='AUDIT_ID')
print(pd.crosstab(ch2.n_co > 0, ch2.restored))
  5. 5

    Freeze the payouts

    The 14 are filed through the mission-control console, with the method written down for the reviewers.

    run_python 
    ids = [int(v) for v in targets.VENDOR_ID]
mc.hold_payouts(ids, reason=Reason.ACCOUNT_TAKEOVER,
    note="Unreported payout-account takeovers: 2024 portal bank change preceded by "
         "contact e-mail change within 0-2.2 days (same fingerprint as the 34 "
         "support-restored takeovers), payout account never restored.")
[ Get started ]

Run the benchmark or build on it.

The warehouse and the tasks are public. For official scores and access to the simulator, contact us.

The warehouse

CC BY 4.0

1,219 Parquet files, with setup for BigQuery, DuckDB, Snowflake, Trino, Delta Lake, and Iceberg.

View dataset ↗

The tasks and agent

Apache 2.0

The reference agent with its warehouse, sandbox, tools, and tasks.

View code ↗
[ About TextQL ]

We work on the hardest data problems at the world's largest enterprises.

We built Argo-Bench because we couldn't find a benchmark that came close to the complexity of those environments, or that told us much about how a model would do inside one.

TextQL ↗ Request demo ↗
[ Cite ]

Citation

If you use Argo-Bench, the released warehouse or our results in your work, please cite the paper, arXiv:2610.02122.

BibTeX 
@article{tomitsuka2026argobench,
  title   = {Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows},
  author  = {Tomitsuka, Gabriel and Raayatsanati, Arman and Xing, Emma
             and Gand, Duke and Ma, Joseph J},
  journal = {arXiv preprint arXiv:2610.02122},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.02122}
}
[ Changelog ]

Versions

Scores are reported against a version.

  1. v1.4.2 Current Nov 2026

    Grading fixes, GPT-6.1 Terra, and Muse Wonder on the Responses API

    • vd-c2-six-month-membership-retention and vd-c3-twelve-month-membership-retention also accept a second reasonable reading of when an unpaid trial membership ends, as their prompts allow.
    • A dollar figure filed to the cent counts as right when it is the answer rounded to the cent. ds-36t-min-pay-true-up-per-delivery-by-borough and ds-40t-no-courier-compensation-by-borough had marked such figures wrong.
    • On ids-01-identity-substitution, ato-02-unreported-takeovers, vbp-01-duplicate-brands, and dsk-03-farm-rings-h2, accounts the answer key cannot fairly settle neither earn nor cost a run anything.
    • fc-14-back-pay is keyed on the full back-pay rule, mer-14-platform-funded-at-shells counts cancelled pickup orders whose charge stood, fin-01-collections-past-due weighs its position metrics more, and or-01b-coverage-cut-january-bare reads a plan as it is filed.
    • A run that broke on every attempt counts as filing nothing. Three such runs had been left out of their settings' averages: Muse Wonder at high effort on fc-20p-quest-desk-vs-finance-read, Claude Sonnet 5 at low effort on ds-23-margin-monthly-total, and DeepSeek V4.1 Flash 1017 at high effort on col-10-refund-partnerships-h.
    • Every run was re-graded. 52 settings rise, by at most 1.8 points, five fall by at most 0.2, and the top four keep their order.
    • Muse Wonder runs again on Meta's Responses API, which keeps its reasoning from one step to the next. Its extra-high score rises from 32.6 to 37.7, still eighth.
    • GPT-6.1 Terra joins the leaderboard in sixth place.
  2. v1.4 Oct 2026

    Six harder tasks and Grok 4.7

    • Six tasks that most settings solved are retired: vd-e6-consumer-fee-peak-quarter, dhc-01-closure-order-sales, pe-07-instant-pay-price, fin-15-customer-credit-liabilities, cf-08-promo-acquisition, and pe-11-support-customer-exposure. Their runs stay in the record.
    • Six new tasks take their places, each in the same business area: cmp-31-cancelled-trip-make-whole, dhc-02-closure-order-sales-year, ip-02r-instant-fee-waiver-rule, fin-33-order-books-close, cf-42-new-customer-offer-revenue-presentation, and cr-01-november-courier-churn-vs-plan. Every model on the leaderboard ran them. No score moves by more than 2.3 points, and the order of the models does not change.
    • Grok 4.7 joins the leaderboard in sixth place.
  3. v1.3 Oct 2026

    Stricter grading on three questions

    • ds-90-menu-items-on-offer, ds-36t-min-pay-true-up-per-delivery-by-borough, and dhc-01-closure-order-sales no longer accept answers the warehouse contradicts. Every affected run was re-graded, no score on the leaderboard moves by more than 1.2 points, and the order does not change.
  4. v1.2 Oct 2026

    Grading fixes and three new models

    • Fixed four grading bugs: pay-01-diverted-payouts, cf-05-quest-reconcile, ds-34-commission-by-plan, and re-filed answers. Every affected run was re-graded, and no score on the leaderboard moves by more than 0.6 points.
    • Gemini 4 Argon, Muse Wonder, and DeepSeek V4.1 Flash 1017 join the leaderboard.
    • DeepSeek V4.1 Flash is repriced at what Fireworks charged during the run, after Fireworks raised its prices.
  5. v1.1 Sep 2026

    Claude Sonnet 5.5 and GPT-6.1 Sol

    • Claude Sonnet 5.5 and GPT-6.1 Sol join the leaderboard in third and fourth place, respectively.
    • Claude Haiku 5.5 joins in October, in Haiku 4.5's place on the chart.
    • The leaderboard reports each model at its highest reasoning effort, extra-high where it has one.
    • Forecasts are graded by interval score on an asinh scale against the median reference miss, floored at −2 per expectation, and every run is re-graded under the new rule.
    • Runs stopped by the one-hour limit were resumed from their transcripts with three more hours.
    • The four refund-collusion searches state how their grader weighs the review, and were run again on the new wording.
    • Storefronts that play fraud roles in the released warehouse carry pseudonyms.
  6. v1.0 Sep 2026

    First release, as submitted to ICLR 2027

    • 210 tasks across trust and safety, marketplace operations, FP&A, accounting and growth.
    • The public warehouse, the tasks and a reference agent; answer keys held out on a private world.

Argo-Bench

A frontier benchmark for enterprise-scale data science.

TextQL Labs

Argo-Bench contains no data about real people. Restaurants are real New York businesses from public records, except those cast in a fraud scenario, which carry fictional names; everything any of them does in the world is simulated. Map data © OpenStreetMap contributors.