SherlockBench

The SherlockBench Problem-sets

SherlockBench problem-sets are just files written in Clojure which can be loaded by the API server. Each has it's own repo in GitHub.

This is our newest problem-set. It is very small to make it easy to run and is designed to test fluid intelligence.

This contains a lot of problems which smart models sometimes get right.

This is the problem-set we use on the Leaderboard. It has a mixture of easy and hard problems.

This problem-set is designed for testing frontier reasoning models which get very high scores on Sherlock1.

This is a port of the version 6 problem-set from the old InterroBench benchmark.