Roadmap

This roadmap outlines the key phases of development, from initial setup and growth through current enhancements and future strategic directions.

Phase 1 - Platform inception - 2021 to Mid 2022

• Founding & Vision: Established the core mission to formalize and benchmark open problems in single-cell analysis.
• Initial Infrastructure: Set up foundational infrastructure, including GitHub repositories (openproblems-bio/openproblems, openproblems-bio/website) and potentially initial cloud integrations (AWS discussed in early tutorials).
• NeurIPS 2021 Competition: A significant early focus was organizing and running a multimodal single-cell data integration competition for NeurIPS 2021. This involved defining tasks (like modality prediction), creating starter kits, setting up submission processes (using tools like Viash and Docker), and defining evaluation metrics.
• Early Community Building: Engaged the community through activities like hackathons and development tutorials (e.g., using GitHub and AWS for contributions).
• Website v1: Developed the initial website (openproblems.bio) to host documentation, competition details, and potentially early results.

Phase 2 - Foundation and consolidation - Mid 2022 to 2023

• Project Structure (V1 -> V2 Transition Start): Began migrating the existing codebase (evolved from the NeurIPS competition structure) towards a more robust and modular structure (OpenProblems-v2). This involved porting numerous tasks including ATAC signal denoising, various batch integration tasks, data denoising, dimensionality reduction, label projection, modality prediction, modality matching, regulatory effect prediction, and spatial decomposition.
• Website Development: Created contribution guidelines for V2 and planned/developed a new results page for displaying benchmark results, automating updates.
• Competition Setup: Set up infrastructure for an EvalAI competition (Q2 2023), including scaling worker nodes and updating evaluation pipelines.

Phase 3 - Expansion and scaling - 2024

• Data Expansion: Integrated with CELLxGENE Discover Census via a CellCensus data loader to significantly expand the range of available datasets.
• Modularisation: Enhanced maintainability by transitioning from a monolithic repository to a modular structure (task-specific repositories). Implemented independent dependency management and created a task template repository.
• GPU & Large Model Support: Implemented support for GPU-accelerated instances and established mechanisms for handling large models.
• Scalability Enhancements: Improved component scalability to handle larger datasets and models. Developed automated scalability testing.
• Task Migration & Onboarding: Continued migrating tasks to the new structure and onboarded new tasks (e.g., Spatially Variable Genes, iST Preprocessing, GRN Inference).
• Interactive Results & Deployment: Planned development for an interactive interface for result exploration and a one-click deployment option for the OpenProblems workbench in Google CoLab.
• Foundation Model Benchmark: Focused on implementing and integrating various foundation models (e.g., scGPT, SCimilarity, UCE, Geneformer) into specific tasks.
• Website rewrite: Implemented a redesigned website and backend infrastructure, including database design, API development, and data migration.
• Community Support: Provided ongoing community support, including kickstarter weeks, weekly meetings, and documentation development.

Phase 4 - Global community and impact - 2025 and beyond

• Enhance Platform Usability & Sustainability: Develop API clients, enhance web client, refactor backend, improve onboarding.
• Foster a Vibrant Global Community: Implement global meetings, increase engagement, provide support, maintain documentation.
• Advance Benchmark Development: Host hackathon, complete Foundation Model MVP, analyze results.
• Translate Insights into Practice: Analyze metrics, investigate dataset/performance links, pilot predictive models, develop guidelines.
• Ensure Cost-Effective & Scalable Infrastructure: Manage cloud services, monitor usage, optimize costs.
• Ongoing Support & Enhancements: Provide continuous community support and undertake prioritized infrastructure enhancements.

Near term technical roadmap

Here we expand on the above roadmap to outline specific goals for the Open Problems community in 2025 and early 2026.

User & Developer Experience Enhancements

Goal: Enhance the usability, accessibility, and feature set for both end-users interacting with the website/results and developers contributing to or using the benchmarking framework. Activities:

• Improve web user experience:

  • Improve marketing pages to clearly communicate OpenProblems' features.
  • Update documentation for recently developed features.
  • Extend documentation for downstream analyses.
  • Implement customisable analysis plots per benchmark.
  • Add video tutorials to documentation.

• Improved user experience for benchmark developers:

  • Create a Python package for quick local evaluation of benchmarks.
  • Enable partial benchmark results for long running methods.
  • Enable local preview of benchmarking result figures & QC report.

Core Platform Infrastructure & Management

Goal: Improve the quality, traceability, structure, and flexibility of datasets and associated metadata used within the OpenProblems platform. Activities:

• Dataset improvements:
  • Add data lineage tracking into our workflows (Source datasets → Benchmark datasets → Models → Predictions → Scores → Results).
  • Refactor dataset loaders to support lineage tracing.
  • Support SpatialData format in dataset workflows, automated unit tests, and automated documentation generators.
• Database improvements:
  • Refactor database schema for improved maintainability and data querying.
• Cloud infrastructure improvements:
  • Enable switching between cloud providers to avoid infrastructure constraints.

Foundation Model Benchmarking MVP

Scope: Implement, integrate, and analyze foundation models within the Open Problems benchmarking framework to build a foundation model MVP. Activities:

• Model Implementation: Complete the implementation and integration of the following models into the Open Problems platform: SCimilarity, scGPT, UCE, Geneformer, scPRINT.
• Task Integration: Integrate at least the foundation models into the Batch integration (batch effect removal), Label projection (cell-type annotation), and Perturbation prediction tasks.
• Analysis: Analyze and interpret benchmarking results, comparing model performance across different tasks resulting in a white paper or preprint containing the results of the MVP benchmark.
• Release: Create release and documentation of steps used to run the benchmark.

Cloud Infrastructure Management

• Goal: Manage the cloud infrastructure (e.g., AWS, Google Cloud) necessary for hosting the Open Problems platform, running benchmarks, and storing data.
• Activities:
  • Monitor cloud resource usage and performance.
  • Implement cost optimization strategies where feasible.

2025 Hackathon to Advance Benchmarks & Collaboration

Goal: Host a focused hackathon event aimed at accelerating the development and enhancement of OpenProblems benchmarks. This includes advancing existing benchmarks, initiating new ones through intensive collaborative work, fostering community engagement, and establishing pathways for the publication of results. Activities:

• Recruit and invite approximately 50 participants to focus on roughly 6 benchmarking topics.
• Focus development efforts on key areas, including planned collaborations for Spatial Niche Detection and Foundation Model Benchmarking.
• Target the advancement of benchmarking tasks, aiming for:
  • New releases with enhanced functionality for existing tasks.
  • Proof-of-Concept (POC) pre-releases with defined study designs & core components for new tasks.
• Produce a summary publication or preprint (e.g., on bioHackrXiv) documenting the hackathon outcomes and advancements made.