Stripe’s AI coding evals expose payment integration gaps
Stripe’s new coding-agent benchmark offers a useful warning for teams building custom checkout, subscription and marketplace software: a high score can conceal missing integration requirements. In…
Stripe’s new coding-agent benchmark offers a useful warning for teams building custom checkout, subscription and marketplace software: a high score can conceal missing integration requirements.
In its 19 November report, Stripe evaluated 69 scenarios across 14 model-and-harness combinations. It reports an average criteria score of 78.1%, compared with a 29.4% strict pass rate requiring every mandatory criterion to succeed. These are different measures of the same evaluations, not a production payment failure rate.
Read the limitations before the leaderboard
Stripe’s own agent plugin improved scores for every tested combination, by roughly three to eight percentage points. But the vendor-run evaluation judges transcripts and produced files. It does not verify runtime behaviour: non-functional code can receive full marks. Runs exceeding the 45-minute limit are excluded from score calculations.
That makes this evidence about integration design, not certification that a generated checkout is ready to accept customers’ money.
Turn the findings into acceptance criteria
Our practical recommendation for merchants commissioning custom payment work is to ask for an explicit acceptance checklist before approving delivery. Include successful purchases, declined payments, duplicate notifications, refunds, subscription changes and reconciliation with order records. Require demonstrations in an appropriate test environment, with a developer accountable for the implementation.
Keep a record of the API versions, dependencies and documentation used. When an agent changes code, rerun the relevant checks and review the changes to payment and fulfilment logic. A benchmark can help you choose what to inspect; your own business rules determine what must pass.
This is relevant to teams building or maintaining their own Stripe integrations. It does not, by itself, require a Shopify merchant using an existing hosted checkout to replace that checkout or change payment providers.
More from AI
See all →
