GAIA Benchmark Agent

Problem

GAIA Level 1 questions are short to state and annoying to answer: they require reading files, listening to audio, checking a specific revision of a Wikipedia article, or extracting one cell from a spreadsheet, and the grader accepts exactly one string. A plain chat model fails them. The assignment is to build an agent that reliably does the work and formats the answer.

Architecture: two agents, not one

The agent is a hybrid built on smolagents. A CodeAgent runs first: it writes and executes Python to work through the task, which handles math, file parsing, and multi-step lookups well. When the CodeAgent path fails or is a poor fit, a ToolCallingAgent takes over with the same tool set in a conventional tool-call loop.

The tool set covers the benchmark's modalities: web research, revision-aware Wikipedia retrieval through the MediaWiki API (several GAIA questions depend on what an article said at a specific point in time, not what it says now), file and spreadsheet and PDF inspection, audio transcription, and video analysis.

Trace-driven failure analysis

Progress came from reading traces, not tweaking prompts blind. Each failed question was rerun in a single-question regression harness, its trace inspected for where the run went wrong, and the fix verified against that question without re-running the whole set.

The largest source of avoidable loss was formatting, not reasoning: right answers rejected because the string did not match the grader. The fix is question-aware deterministic answer normalization, code that inspects what shape the question demands (a number, a comma-separated list, a name) and normalizes the model's answer to it. Correctness that depends on formatting is handled by code, not left to the model.

Result

90% (18 of 20) on the GAIA Level 1 final assignment, against a 30% pass bar, earning the course's Certificate of Excellence in July 2026. The agent is live on Hugging Face Spaces and the certificate is public.