Stripe benchmarks coding agents on real payment integration tasks
How much confidence should a developer put in an agent that builds a checkout? Stripe published a benchmark on 19 November that lets teams compare coding agents on three different jobs: one-time…
How much confidence should a developer put in an agent that builds a checkout? Stripe published a benchmark on 19 November that lets teams compare coding agents on three different jobs: one-time payments, subscriptions and Connect platform integrations.
In Stripe’s first results, Codex with GPT-6 Astra scored 90 out of 100 on one-time payments without the plugin and 96 with it. On subscriptions the same setup moved from 82 to 83. A team building recurring billing would get a different impression from those scores than from the checkout result alone.
What the scores measure
Each scenario gives the coding agent a task and a simulated developer who can answer follow-up questions. Stripe compares runs with and without its plugin, which supplies integration guidance and tools. Both configurations have web search. The score measures how many evaluation criteria the integration satisfies.
For an app team, the relevant comparison is the one closest to its own money flow. A successful one-time payment says little about what happens when a subscription renews or a marketplace refund must be allocated. Keep those cases in the project’s own tests before merging generated code.
Stripe Developers announced the integration benchmark on 19 November.
View the original announcement on X
More from AI
See all →
