SQL-Detective-v1 · Reward Atlas

Baselines & results

Results

model	method	exact match	avg turns
7B	GRPO, 800 steps	61.4%	5.8
7B	SFT on 2k traces	49.0%	4.1
7B	Base (no training)	38.2%	3.2

Eval split (400 questions), mean over 3 seeds

Eval exact match during GRPO training

View as table

training steps	GRPO 7B
0	38.2
50	42.9
100	45.5
150	47.8
200	50.9
250	53
300	53.7
350	55
400	56.9
450	57.5
500	57.4
550	58.5
600	59.7
650	59.4
700	59.2
750	60.3
800	60.9

Inspect a complete example trajectory

Aggregate scores tell you whether a policy works. This run shows how it worked: every visible decision, tool call, environment response, correction and verifier receipt in order.

Run trace · sample-0241 / rollout-03pass

Find the three product categories with the largest revenue drop from Q1 to Q2 2024.

Model: GRPO-7B · seed 2
Reward: 1.0
Duration: 18.4 s
Tokens: 2.8k
Cost: $0.006

Input

24 tokens

Find the three product categories with the largest revenue drop from Q1 to Q2 2024. Return category and revenue_drop.

Output

41 tokens

Outdoor — 27,144.80

Kitchen — 19,802.35

Books — 12,110.00

Workflow map

Orient
Read the task and inspect available tables.
Ground
Resolve keys, prices and category names from schema.
3Correct
Notice that positive deltas rank growth, then reverse the ordering.
Verify
Submit exact rows and preserve the grader receipt.

Observable trace

Messages, tool calls, results and scoring—not hidden reasoning.

01 · TaskEvaluation prompt
Which three product categories had the largest drop in revenue between Q1 and Q2 2024? Return category and revenue_drop.
02 · AgentChoose schema-first exploration 0.9 s184 tok$0.0004
The requested metric spans products, categories and dated orders. Inspecting the schema before querying avoids guessing join keys or date fields.
03 · ToolList database tables 42 ms
Discover the available relations.
```
list_tables()
```
["orders", "order_items", "products", "categories", "customers"]
04 · ToolInspect the three relevant schemas 71 ms
Confirm the foreign keys, order timestamp and revenue inputs.
```
describe_table("orders"); describe_table("order_items"); describe_table("products")
```
orders(id, ordered_at) order_items(order_id, product_id, quantity, unit_price) products(id, category_id)
05 · AgentForm the comparison query 1.6 s421 tok$0.0009
Aggregate quantity × unit_price by category and quarter, then compare Q1 with Q2. The first ordering is kept visible because it led to a useful correction.
06 · ToolRun the first query 84 ms
The query is valid, but the sign and ordering expose growth rather than decline.
```
SELECT c.name, q2.revenue - q1.revenue AS delta
FROM quarterly q1 JOIN quarterly q2 USING (category_id)
JOIN categories c ON c.id = q1.category_id
ORDER BY delta DESC LIMIT 3;
```
Home office | 18420.70 Audio | 9250.10 Accessories | 6118.40
warn
07 · AgentCorrect the ranking direction 0.7 s163 tok$0.0003
A revenue drop is Q1 − Q2, so positive values should be ranked descending. Reversing the subtraction fixes both the meaning and the sort without changing the joins.

08 · ToolRun the corrected query 79 ms

Execute the corrected comparison and retain the exact rows for submission.

SELECT c.name, ROUND(q1.revenue - q2.revenue, 2) AS revenue_drop
FROM quarterly q1 JOIN quarterly q2 USING (category_id)
JOIN categories c ON c.id = q1.category_id
ORDER BY revenue_drop DESC LIMIT 3;

Outdoor | 27144.80 Kitchen | 19802.35 Books | 12110.00

pass

09 · ToolSubmit the result rows 31 ms
End the episode with the three grounded rows.
```
submit_answer([["Outdoor",27144.80],["Kitchen",19802.35],["Books",12110.00]])
```
10 · VerifierExact multiset match 12 ms
The submitted categories and values match the reference result. Row ordering is ignored; numeric tolerance is 1e-6.
passreward +1.0

Run artifacts

Final answerjson

[["Outdoor", 27144.80], ["Kitchen", 19802.35], ["Books", 12110.00]]

Verifier receiptjson

{
  "exact_match": true,
  "matched_rows": 3,
  "invalid_queries": 0,
  "reward": 1.0
}