src/<task_id>/

Each task should contain a data processor (to transform common datasets into task-specific datasets), methods, control methods (for quality control), and metrics (Figure 1).

graph LR
  classDef component fill:#decbe4,stroke:#333
  classDef anndata fill:#d9d9d9,stroke:#333
  common_dataset[Common<br/>dataset]:::anndata
  subgraph task_specific[Task-specific workflow]
    dataset_processor[/Dataset<br/>processor/]:::component
    solution[Solution]:::anndata
    masked_data[Masked data]:::anndata
    method[/Method/]:::component
    control_method[/Control<br/>method/]:::component
    output[Output]:::anndata
    metric[/Metric/]:::component
    score[Score]:::anndata
  end
  common_dataset --> dataset_processor --> masked_data
  dataset_processor --> solution
  masked_data --> method --> output
  masked_data & solution --> control_method --> output
  solution & output --> metric --> score
Figure 1: Overview of a typical benchmarking workflow in an OpenProblems task. Legend: Grey rectangles are AnnData .h5ad files, purple rhomboids are Viash components.