An LLM Benchmarking System
SherlockBench is a benchmarking system designed to test an LLM's ability to pro-actively investigate a problem. It is given a mystery function, and has to test it to determine what it does.
It tests "pro-active learning" which means it requires the LLM to reason about a strategy for it's own learning (in context).
LeaderBoard
SherlockBench is now saturated by GPT-6 Astra as-of September 2026. However the benchmark is still useful to test lesser models. Here is the scoreboard for the Irene2 problem-set.
| Model | Provider | Average | pass@3 |
|---|---|---|---|
| GPT-6 Astra gpt-6-astra - high effort | OpenAI | 98% | 100% |
| GPT-5.6 Sol gpt-5.6-sol - high effort | OpenAI | 84% | 91% |
| Qwen3.5 9b 6 bit quant, reasoning enabled | Qwen | 42% | 58% |
I can't afford to do a pass@3 on Fable 5.1 because it's twice as expensive as Astra, but I did a pass@1 and it scored about the same as GPT-5.6 Sol. In future I'll focus on testing open-weight models unless someone wants to pay for it.