WideSearch

WideSearch measures breadth. The agent has to enumerate a full entity set, chase down each attribute, and emit a structured table — closer to real research workflows (market scans, literature surveys, competitive tables) than single-answer trivia. Half the tasks are in Chinese, so it also exercises engines outside English-language results.

We run it with the model held fixed because that isolates the variables OpenRouter users actually control. Those are which engine handles the searches, whether search runs as a server tool or a plugin, and how many agent turns the loop is allowed. Those knobs are exactly what you can set on a request today.

These scores compare search configurations, not agent products. The model reads search result excerpts only, with no full-page fetching and no code tools, so absolute numbers sit below published agent leaderboards, which allow both. Compare configurations rather than raw levels.

The headline score is answer-item accuracy, which gives partial credit for matched table items so near-misses remain visible. Strict WideSearch Success Rate remains the secondary measure and requires every row and cell in the table to be correct. Even frontier models score low upstream — the paper reports about 4.5% single-agent success for OpenAI o3 (Avg@4), with answer-item accuracy around 52–58% for leading single-agent systems. Engine gaps smaller than the confidence ranges are unresolved, and a configuration's cost should be read alongside its quality.

The questions are public, which means memorization can inflate absolute scores over time; differences between search configurations remain meaningful because every configuration shares the same model.

Each task is one query with a reference table. The model's table is parsed, its rows and columns are aligned to the reference by a judge model, and each cell is compared — exact match where possible, judge-graded equivalence otherwise. A task succeeds only when every reference row is found and every cell matches, with no missing or extra rows.

reward = all_rows_aligned ∧ every_cell_correct   // ∈ {0, 1}
judge  = gpt-4.1 at temperature 0 for alignment and fuzzy cells
unparseable or empty tables score 0

A search configuration is one model, search engine, request format, and maximum search budget. Scores are weighted by the number of questions in each verified run; small smoke tests are excluded.

Runs execute on OpenRouter's native benchmark harness against production endpoints. Server-tool configurations set a request-level tool-call budget on the web search server tool. Plugin configurations make one pre-inference search of the question with the web search plugin. Engines use the same default configurations that serve production traffic.

Every run persists its exact model, engine, request format, search budget, cost, and available timing telemetry. Missing configurations stay missing in the comparison table, and absent or zero timing telemetry is not treated as instantaneous performance.