Add a metric

A metric is a quantitative measure used to evaluate the performance of the different methods in solving the specific task problem.

This guide will show you how to create a new Viash component. In the following we will show examples for both Python and R. Note that the Label Projection task is used throughout the guide, so make sure to replace any occurrences of "label_projection" with your task of interest.

Tip

Make sure you have followed the “Getting started” guide.

Step 1: Create a new component

Use the create_component component to start creating a new metric.

viash run src/common/create_component/config.vsh.yaml -- \
  --task label_projection \
  --type metric \
  --name my_metric_py \
  --language python
[notice] Checking if Docker image is available at 'ghcr.io/openproblems-bio/common/create_component:dev'
[warning] Could not pull from 'ghcr.io/openproblems-bio/common/create_component:dev'. Docker image doesn't exist or is not accessible.
[notice] Building container 'ghcr.io/openproblems-bio/common/create_component:dev' with Dockerfile

This will create a new folder at src/label_projection/metrics/my_metric_py containing a Viash config and a script.

src/label_projection/metric/my_metric_py
    ├── script.py                    Script for running the metric.
    ├── config.vsh.yaml              Config file for metric.
    └── ...                          Optional additional resources.
viash run src/common/create_component/config.vsh.yaml -- \
  --task label_projection \
  --type metric \
  --name my_metric_r \
  --language r

This will create a new folder at src/label_projection/metrics/my_metric_r containing a Viash config and a script.

src/label_projection/metrics/my_metric_r
    ├── script.R                     Script for running the metric.
    ├── config.vsh.yaml              Config file for metric.
    └── ...                          Optional additional resources.

Change the --name to a unique name for your metric. It must match the regex [a-z][a-z0-9_]* (snakecase).

  • A config file contains metadata of the component and the dependencies required to run it. In steps 2 and 3 we will fill in the required information.
  • A script contains the code to run the metric. In step 4 we will edit the script.
Tip

Use the command viash run src/common/create_component/config.vsh.yaml -- --help to get information on all of the parameters if the create_component component.

Step 2: Fill in metadata

The Viash config contains metadata of your metric, which script is used to run it, and the required dependencies.

Generated config file

This is what the config.vsh.yaml generated by the create_component component looks like:

Contents of config.vsh.yaml 
# The API specifies which type of component this is.
# It contains specifications for:
#   - The input/output files
#   - Common parameters
#   - A unit test
__merge__: ../../api/comp_metric.yaml

functionality:
  name: my_metric_py

  # Metadata for your component (required)
  info:
    metrics:
      - name: my_metric_py
        pretty_name: My Metric Py
        summary: 'FILL IN: A one sentence summary of this metric.'
        description: 'FILL IN: A (multiline) description of how this metric works.'
        reference: bibtex_reference_key
        documentation_url: https://url.to/the/documentation
        repository_url: https://github.com/organisation/repository
        min: 0
        max: 1
        maximize: 'true'

  # Component-specific parameters (optional)
  # arguments:
  #   - name: "--n_neighbors"
  #     type: "integer"
  #     default: 5
  #     description: Number of neighbors to use.

  # Resources required to run the component
  resources:
    # The script of your component
    - type: python_script
      path: script.py
platforms:
  - type: docker
    image: python:3.10
    # Add custom dependencies here
    setup:
      - type: python
        pypi: anndata~=0.8.0
  - type: nextflow
    directives:
      label: [midmem, midcpu]
Contents of config.vsh.yaml 
# The API specifies which type of component this is.
# It contains specifications for:
#   - The input/output files
#   - Common parameters
#   - A unit test
__merge__: ../../api/comp_metric.yaml

functionality:
  name: my_metric_r

  # Metadata for your component (required)
  info:
    metrics:
      - name: my_metric_r
        pretty_name: My Metric R
        summary: 'FILL IN: A one sentence summary of this metric.'
        description: 'FILL IN: A (multiline) description of how this metric works.'
        reference: bibtex_reference_key
        documentation_url: https://url.to/the/documentation
        repository_url: https://github.com/organisation/repository
        min: 0
        max: 1
        maximize: 'true'

  # Component-specific parameters (optional)
  # arguments:
  #   - name: "--n_neighbors"
  #     type: "integer"
  #     default: 5
  #     description: Number of neighbors to use.

  # Resources required to run the component
  resources:
    # The script of your component
    - type: r_script
      path: script.R
platforms:
  - type: docker
    image: eddelbuettel/r2u:22.04
    # Add custom dependencies here
    setup:
      - type: apt
        packages:
          - libhdf5-dev
          - libgeos-dev
          - python3
          - python3-pip
          - python3-dev
          - python-is-python3
      - type: python
        pypi: anndata~=0.8.0
      - type: r
        cran: anndata
  - type: nextflow
    directives:
      label: [midmem, midcpu]

Required metadata fields

Please make sure that the following fields in the functionality and functionality.info sections in the config file are filled in. The metrics component can contain several metric values these are listed in the functionality.info.metrics.

functionality.name

A unique identifier for the metric component. Must be written in snake case. Example: my_new_metric. This will be the same as the name given in the --name argument of the create_component command above.

functionality.info.metrics[]

  • .name: A unique identifier for the metric (if only 1 metric in the component can be the same as functionality.name). Must be written in snake case. Example: my_new_metric.

  • .pretty_name: A label for the metric used for visualisations and documentation. Example: "My new metric".

  • .summary: A one sentence summary of the metric. Used for creating short overviews of the components in a task.

  • .description: An explanation for how the metric works. Used for creating reference documentation of a task.

  • .reference: A bibtex reference key to the paper where the metrics is used.

  • .documentation_url: The url to the documentation of the metrics used.

  • .repository_url: The repository url for the metrics used.

  • .min: The minimum value of the metrics (-inf if there isn’t a min).

  • .max: The maximum value of the metrics (+inf if there isn’t a max).

  • .maximize: Set to true if a higher value is better and false if a lower value is better.

__merge__

The file specified in this field contains information regarding the input and output arguments of the component, as well as a unit test to ensure that the component is functioning properly. Normally you don’t need to change this if you gave the right arguments to the create_component component.

Step 3: Add dependencies

Each component has it’s own set of dependencies, because different components might have conflicting dependencies.

In the platforms section of the config file update the setup definition that describes the packages that need to be installed in the Docker image and are required for your metric to run. Note that both anndata~=0.8.0 and pyyaml are necessary Python package dependencies.

Please check out this guide for more information on how to add extra package dependencies.

Note

Tip: After making changes to the components dependencies, you will need to rebuild the docker container as follows:

viash run src/label_projection/metrics/my_metric_py/config.vsh.yaml -- \
  ---setup cachedbuild
Output 
[notice] Building container 'ghcr.io/openproblems-bio/label_projection/metrics/my_metric_py:dev' with Dockerfile

Step 4: Edit script

A component’s script typically has five sections:

  1. Imports and libraries
  2. Argument values
  3. Read input data
  4. Generate results
  5. Write output data to file

Generated script

This is what the script generated by the create_component component looks like:

Contents of script.py 
import anndata as ad

## VIASH START
par = {
  'input_solution': 'resources_test/label_projection/pancreas/solution.h5ad',
  'input_prediction': 'resources_test/label_projection/pancreas/prediction.h5ad',
  'output': 'output.h5ad'
}
meta = {
  'functionality_name': 'my_metric_py'
}
## VIASH END

print('Reading input files', flush=True)
input_solution = ad.read_h5ad(par['input_solution'])
input_prediction = ad.read_h5ad(par['input_prediction'])

print('Compute metrics', flush=True)
# metric_ids and metric_values can have length > 1
# but should be of equal length
uns_metric_ids = [ 'my_metric_py' ]
uns_metric_values = [ 0.5 ]

print("Write output AnnData to file", flush=True)
output = ad.AnnData(
  uns={
    'dataset_id': input_prediction.uns['dataset_id'],
    'normalization_id': input_prediction.uns['normalization_id'],
    'method_id': input_prediction.uns['method_id'],
    'metric_ids': uns_metric_ids,
    'metric_values': uns_metric_values
  }
)
output.write_h5ad(par['output'], compression='gzip')
Contents of script.R 
library(anndata)

## VIASH START
par <- list(
  input_solution = "resources_test/label_projection/pancreas/solution.h5ad",
  input_prediction = "resources_test/label_projection/pancreas/prediction.h5ad",
  output = "output.h5ad"
)
meta <- list(
  functionality_name = "my_metric_r"
)
## VIASH END

cat("Reading input files\n")
input_solution <- anndata::read_h5ad(par[["input_solution"]])
input_prediction <- anndata::read_h5ad(par[["input_prediction"]])

cat("Compute metrics\n")
# metric_ids and metric_values can have length > 1
# but should be of equal length
uns_metric_ids <- c("my_metric_r")
uns_metric_values <- c(0.5)

cat("Write output AnnData to file\n")
output <- anndata::AnnData(
  uns = list(
    dataset_id = input_prediction$uns[["dataset_id"]],
    normalization_id = input_prediction$uns[["normalization_id"]],
    method_id = input_prediction$uns[["method_id"]],
    metric_ids = uns_metric_ids,
    metric_values = uns_metric_values
  )
)
output$write_h5ad(par[["output"]], compression = "gzip")

Required sections

Imports and libraries

In the top section of the script you can define which packages/libraries the metric needs. If you add a new or different package add the dependency to config.vsh.yaml in the setup field (see above).

Argument block

The Viash code block is designed to facilitate prototyping, by enabling you to execute directly by running python script.py (or Rscript script.R for R users). Note that anything between “VIASH START” and “VIASH END” will be removed and replaced with a CLI argument parser when the components are being built by Viash.

Here, the par dictionary contains all the arguments defined in the config.vsh.yaml file (including those from the defined __merge__ file). When adding a argument in the par dict also add it to the config.vsh.yaml in the arguments section.

Read input data

This section reads any input AnnData files passed to the component.

Generate results

This is the most important section of your script, as it defines the core functionality provided by the component. It processes the input data to create results for the particular task at hand.

Write output data to file

The output stored in a AnnData object and then written to an .h5ad file. The format is specified by the API file specified in the __merge__ field in the config file.

Step 5: Try component

Your component’s API file contains the necessary unit tests to check whether your component works and the output is in the correct format.

You can test your component by using the following command:

viash test src/label_projection/metrics/my_metric_py/config.vsh.yaml
Output 
Running tests in temporary directory: '/tmp/viash_test_f16663551105160948292'
====================================================================
+/tmp/viash_test_f16663551105160948292/build_executable/f1 ---verbosity 6 ---setup cachedbuild
[notice] Building container 'ghcr.io/openproblems-bio/label_projection/metrics/f1:test' with Dockerfile
[info] Running 'docker build -t ghcr.io/openproblems-bio/label_projection/metrics/f1:test /tmp/viash_test_f16663551105160948292/build_executable -f /tmp/viash_test_f16663551105160948292/build_executable/tmp/dockerbuild-f1-ZuSDFC/Dockerfile'
Sending build context to Docker daemon  40.96kB

Step 1/7 : FROM python:3.10
 ---> fc98d03e6037
Step 2/7 : RUN pip install --upgrade pip &&   pip install --upgrade --no-cache-dir "scikit-learn" "pyyaml" "anndata~=0.8.0"
 ---> Using cache
 ---> 1d35b64eb218
Step 3/7 : LABEL org.opencontainers.image.description="Companion container for running component label_projection/metrics f1"
 ---> Running in 019b6e97322f
Removing intermediate container 019b6e97322f
 ---> 7b2f6eb7cb79
Step 4/7 : LABEL org.opencontainers.image.created="2023-05-06T00:06:35Z"
 ---> Running in cf46fab9908c
Removing intermediate container cf46fab9908c
 ---> 254c25f31001
Step 5/7 : LABEL org.opencontainers.image.source="https://github.com/openproblems-bio/openproblems-v2"
 ---> Running in 537faf27b29e
Removing intermediate container 537faf27b29e
 ---> a27e17982974
Step 6/7 : LABEL org.opencontainers.image.revision="9438b8ad0cdd9cd2ed3ba6a01d0b4f075c059d64"
 ---> Running in b8903ed4b290
Removing intermediate container b8903ed4b290
 ---> 3a02e1b30f79
Step 7/7 : LABEL org.opencontainers.image.version="test"
 ---> Running in e7825f5c3f1c
Removing intermediate container e7825f5c3f1c
 ---> b4415837b532
Successfully built b4415837b532
Successfully tagged ghcr.io/openproblems-bio/label_projection/metrics/f1:test
====================================================================
+/tmp/viash_test_f16663551105160948292/test_check_metric_config/test_executable
Load config data
check general fields
Check info fields
{'v1_url': 'openproblems/tasks/label_projection/metrics/f1.py', 'v1_commit': 'bb16ca05ae1ce20ce59bfa7a879641b9300df6b0', 'metrics': [{'name': 'f1_weighted', 'pretty_name': 'F1 weighted', 'summary': 'Average weigthed support between each labels F1 score', 'description': "Calculates the F1 score for each label, and find their average weighted by support (the number of true instances for each label). This alters 'macro' to account for label imbalance; it can result in an F-score that is not between precision and recall.", 'reference': '', 'repository_url': '', 'documentation_url': '', 'min': 0, 'max': 1, 'maximize': True}, {'name': 'f1_macro', 'pretty_name': 'F1 macro', 'summary': 'Unweighted mean of each label F1-score', 'description': 'Calculates the F1 score for each label, and find their unweighted mean. This does not take label imbalance into account.', 'reference': '', 'repository_url': '', 'documentation_url': '', 'min': 0, 'max': 1, 'maximize': True}, {'name': 'f1_micro', 'pretty_name': 'F1 micro', 'summary': 'Calculation of TP, FN and FP.', 'description': 'Calculates the F1 score globally by counting the total true positives, false negatives and false positives.', 'reference': '', 'repository_url': '', 'documentation_url': '', 'min': 0, 'max': 1, 'maximize': True}], 'type': 'metric'}
All checks succeeded!
====================================================================
+/tmp/viash_test_f16663551105160948292/test_run_and_check_adata/test_executable
>> Checking whether input files exist
>> Running script as test
Load data
Encode labels
Compute F1 score
Store metric value
Writing adata to file
>> Checking whether output file exists
>> Reading h5ad files and checking formats
Reading and checking input_solution
  AnnData object with n_obs × n_vars = 154 × 419
    obs: 'label', 'batch'
    var: 'hvg', 'hvg_score'
    uns: 'dataset_id', 'normalization_id'
    obsm: 'X_pca'
    layers: 'counts', 'normalized'
Reading and checking input_prediction
  AnnData object with n_obs × n_vars = 154 × 419
    obs: 'batch', 'label_pred'
    var: 'hvg', 'hvg_score'
    uns: 'dataset_id', 'method_id', 'normalization_id'
    obsm: 'X_pca'
    layers: 'counts', 'normalized'
Reading and checking output
  AnnData object with n_obs × n_vars = 154 × 419
    obs: 'batch', 'label_pred'
    var: 'hvg', 'hvg_score'
    uns: 'dataset_id', 'method_id', 'metric_ids', 'metric_values', 'normalization_id'
    obsm: 'X_pca'
    layers: 'counts', 'normalized'
All checks succeeded!
====================================================================
SUCCESS! All 2 out of 2 test scripts succeeded!
Cleaning up temporary directory

Visit “Run tests” for more information on running unit tests and how to interpret common error messages.

You can also run your component on local files using the viash run command. For example:

viash run src/label_projection/metrics/my_metric_py/config.vsh.yaml -- \
  --input_prediction resources_test/label_projection/pancreas/knn.h5ad \
  --input_solution resources_test/label_projection/pancreas/solution.h5ad \
  --output output.h5ad

Next steps

If your component works, please create a pull request.