10
Dataset, tools & verifier
Dataset
2,400 questions over 12 SQLite databases.
- train: 2,000 · eval: 400
- Databases disjoint between splits
- License: CC BY 4.0
Tools
list_tables · describe_table · run_query · submit_answer
One call per turn; see Step 4 for signatures and guardrails.
Sandbox
Image rewardatlas/sql-detective:1.0
- Network: off
- Mount: read-only
- 5 s query timeout, 512 MB memory
Verifier / rubric
Deterministic Python grader: multiset match, partial column credit, invalid-query penalty and a format canary. No LLM judge.
def grade(submitted, expected, state) -> float:
if contains_format_canary(submitted):
return 0.0
if as_multiset(submitted) == as_multiset(expected):
score = 1.0
elif column_overlap(submitted, expected) >= 0.5:
score = 0.2
else:
score = 0.0
penalty = 0.1 * max(0, state.invalid_queries - 3)
return max(score - penalty, 0.0)