Task structure

Before defining a new task in OpenProblems, it’s important to understand the typical structure of an OpenProblems task (Figure 1).

A task typically consists of a dataset processor, methods, control methods and metrics. Each component has a well-defined input-output interface, for which the file formats in the resulting AnnData are also described.

graph LR
  classDef component fill:#decbe4,stroke:#333
  classDef anndata fill:#d9d9d9,stroke:#333
  common_dataset[Common<br/>dataset]:::anndata
  subgraph task_specific[Task-specific workflow]
    dataset_processor[/Dataset<br/>processor/]:::component
    solution[Solution]:::anndata
    masked_data[Masked data]:::anndata
    method[/Method/]:::component
    control_method[/Control<br/>method/]:::component
    output[Output]:::anndata
    metric[/Metric/]:::component
    score[Score]:::anndata
  end
  common_dataset --> dataset_processor --> masked_data
  dataset_processor --> solution
  masked_data --> method --> output
  masked_data & solution --> control_method --> output
  solution & output --> metric --> score

Figure 1: Overview of a typical benchmarking workflow in an OpenProblems task. Legend: Grey rectangles are AnnData .h5ad files, purple rhomboids are Viash components.

File and component formats

Path: src/<task_id>/api

This folder contains YAML specifications for task-specific file formats and component interfaces.

Dataset processor

Path: src/<task_id>/process_dataset

This components processes common components into task-specific dataset objects. In supervised tasks, this component will usually output a solution, a training dataset and a test dataset. In unsupervised tasks, this component usually output a solution and a masked dataset.

Methods

Path: src/<task_id>/methods

This folder contains method components. Each method component outputs a prediction given the training and test datasets (when applicable).

Control methods

Path: src/<task_id>/control_methods

This folder contains control (or baseline) components for the task. These components have the same interface as the regular methods but also receive the solution object as input. It serves as a starting point to test the relative accuracy of new methods in the task, and also as a quality control for the metrics defined in the task. A control method can either be a positive control or a negative control, which set a maximum and minimum threshold for performance, so any new method should perform better than the negative control methods and worse than the positive control method.

A positive control is a method where the expected results are known, thus resulting in the best possible value for any metric outcome measure.

A negative control is a simple, naive, or random method that does not rely on any sophisticated techniques or domain knowledge.

Metrics

Path: src/<task_id>/metrics

This folder contains metric components. Each metric component outputs one or more metric results given a solution object and a method output object.

Benchmarking pipeline

Path: src/<task_id>/workflows

This folder contains a Nextflow pipeline defining the benchmarking workflow for this task.

Resource generation scripts

Path: src/<task_id>/resources_scripts

This folder contains scripts for generating benchmarking resources required for the task.

Test resource generation scripts

Path: src/<task_id>/resources_test_scripts

This folder contains scripts for generating test resources for the task.