Can AI solve riddles from your culture?
Every riddle, every model
Each column is one riddle and each row is one model configuration, strongest at the top. A filled cell means native speakers or the scoring rules judged the answer fully correct. Riddles are sorted from solved by all models (left) to solved by none (right).
A strong average hides weak cultures
Accuracy is the share of answers judged fully correct. Each model's community scores sit on the right: a long spread means its accuracy depends on whose culture the riddle comes from.
The same model, a different culture
Models largely agree on which communities are hard. The gaps follow language resources and region more than any single model.
Cultural fingerprint
Every model in every community. Rows are models, strongest at the top; columns are communities. Click a square to open that community below.
Each model on these riddles
Bars show accuracy in this community; the grey tick is the model's accuracy over all communities.
By topic
Accuracy pooled over all models. Wildcard groups the topics each community chose itself.
How many models solve each riddle
Language resources
Each dot is a community, placed by the accuracy of all its scored answers pooled over models. Diamonds pool every answer in the level, with 95% intervals from resampling communities.
Where the riddles come from
Accuracy pooled over all models and every human-evaluated community in each country.
By region
Pooled accuracy by UN geographic subregion, with 95% intervals from resampling communities (shown where a region has more than one community).
What the answers look like
How each answer was scored, how often models are right, partly right or wrong, and the kinds of mistakes native speakers saw.
How every answer was scored
Rules handle exact matches, empty outputs, placeholders and explicit admissions of not knowing. Native speakers score the rest; where no valid human label exists, an LLM jury of two judges scores answers they agree on.
Right, partly right or wrong
Every scored answer gets 0 (wrong), 1 (admits not knowing), 2 (partly correct) or 3 (correct).
Kinds of answers
Every answer scored by rules or native speakers, by the type the scorer recorded. Types are descriptive: annotators agree on them less than on scores.
Weaker models make things up more
Each dot is a model: its accuracy against the share of its human-labeled answers that native speakers marked as hallucinated.
When models agree, are they right?
For each riddle, the number of models that gave the most common answer, and whether that answer was correct.
Topics
Every community wrote riddles on the same shared topics, plus wildcard topics of its own choosing.
Controlled comparisons
Each comparison holds everything else fixed: the same model with one setting changed, the same riddles in two languages, or the same language in different places.
More reasoning helps, web search hurts
The same model answered twice with one setting changed.
English helps, but does not close the gap
One language, different cultures
Several communities write in Chinese. Accuracy averages each model's accuracy over all models.
Bigger is usually better, but not always
Accuracy against total parameters for models with a published size (log scale). Proprietary models do not publish their size.
What lowers the odds of a correct answer
Adjusted odds ratios from the paper's regression models, with 95% intervals. Below 1 means lower odds than the reference.
How the benchmark is built
Written, checked and scored by the communities themselves, through the Cohere Labs Community.
What changed when the jury filled the gaps
Accuracy for each model before and after adding the answers that the LLM jury agreed on. Before, those answers were left out.
All model configurations
Click a column to sort. Parameters in billions (active/total for mixture-of-experts models).
Coverage for each community
How each community's answers were scored. Communities without valid human labels are left out of the results.