Benchmarking the open problems in single-cell analysis.
Single-cell analysis is hard. We define the field's hardest computational challenges as formal, reproducible benchmarks, and rank the methods that solve them.
Four traits drive innovation in challenges.
ImageNet and the Netflix Prize showed how far a shared benchmark can move a field. We bring that rigor to single-cell biology.
Clear definitions
Every task is mathematically well-defined, with a fixed input/output contract methods must satisfy.
Standardized datasets
Public, ready-to-use gold-standard datasets curated and version-controlled by the community.
Quantitative metrics
Success is measured by transparent, peer-reviewed metrics, never vibes.
Continuous leaderboards
State-of-the-art methods are ranked and re-run automatically as the field evolves.
How methods compare.
Every method scored across each dataset and metric, in a single funkyheatmap. Results are scaled using control methods for easy comparison.
Standing on the shoulders of shared benchmarks.
Learning from machine learning
We take inspiration from machine learning, where open competitions like ImageNet (computer vision), the Workshop on Statistical Machine Translation (NLP), and the Netflix Prize (recommendations) each pushed the boundaries of what their field could achieve.
Formalized challenges
Major advances in the biological sciences, from DeepMind's protein-folding success with CASP to the DREAM Challenges and RxRx competitions, show the impact of structured challenges.
Cross-disciplinary innovation
Single-cell methods have advanced by drawing on fields like computer vision. We want to strengthen that exchange, bringing machine learning and biomedical research closer together.
A community-owned platform for benchmarking single-cell analysis.
Open Problems is hosted on GitHub with benchmarks run on cloud infrastructure supported by the Chan Zuckerberg Initiative. Every task, dataset, metric and method is shaped by community input, and anyone can contribute a new method or propose a new problem.
Sustained by leaders in open science.
Funders and infrastructure partners power the compute, engineering and scientific leadership behind every benchmark. Meet our sponsors
