Coinbase's fraud agent test: newer Opus, Sonnet and GPT versions caught less fraud than their predecessors, and a post-trained Qwen3.5-9B beat Opus 4.5

Why it matters

【Key points】

Coinbase engineers published a three-part "Owning Intelligence" series on October 6–8 on a fraud agent for Onramp, its guest-checkout crypto purchase flow. Part 1: an LLM review layer, initially Claude Opus, added after the existing ML model and rules cut fraudulent transactions by 30% and fraud value by 22% in an online A/B test with 50% of traffic; projected savings were about three times the inference cost.

Part 2: with the decision policy fixed, a nine-week replay (16,140 transactions, 7,293 users, 813 confirmed frauds) compared Opus 4.5 with Opus 5, Sonnet 4.6 with Sonnet 5, and GPT-5.4 with GPT-5.6 Sol. Every newer version had lower recall, F1 and dollar-weighted recall; Sonnet recall fell 22.2 points, and GPT precision rose 11.5 points while recall fell 20.7. Part 3: Qwen3.5-9B post-trained with SkyRL on Anyscale beat Opus 4.5 on all four metrics, with F1 9.6 higher and 55% lower end-to-end latency (median 0.683s vs 1.515s).

【Verifiable】

All three posts are on Coinbase's engineering blog. A related paper, arXiv:2609.27287 (SR-Fraud), shares authors with the series and also reports the newer models regressing on this benchmark. The dataset is private, so results cannot be reproduced, and no confidence intervals are given. CEO Brian Armstrong said on X that specialized models will win narrow tasks.

【Our read】

1. "Newer is worse" holds for one task under one fixed policy, not for the models overall. A likelier reading is that newer versions shifted their risk calibration, so thresholds tuned for the old model no longer fit.

2. The real lesson: companies building on closed APIs can see business metrics drift silently after a model upgrade. In-house domain evaluation is becoming a requirement.

3. A 9B open model plus outcome-verified RL beating a frontier model on a narrow task echoes Microsoft's decision model built on the same base this month. For high-volume tasks with clear labels, owning the weights is starting to pay off.