Add a metric

A metric is a quantitative measure used to evaluate the performance of the different methods in solving the specific task problem.

This guide will show you how to create a new Viash component. In the following we will show examples for both Python and R. Note that the Task template repo is used throughout the guide, so make sure to replace any occurrences of "task_template" with your task of interest.

Tip

Make sure you have followed the "Getting started" guide.

Step 1: Create a new component

Use the create_*_metric.sh script found in the scripts repository to start creating a new metric. Open the script and update the name parameter to the desired name of the method.

Change the --name to a unique name for your metric. It must match the regex [a-z][a-z0-9_]* (snakecase).

A config file contains metadata of the component and the dependencies required to run it. In steps 2 and 3 we will fill in the required information.
A script contains the code to run the metric. In step 4 we will edit the script.

Tip

Some tasks have multiple metric subtypes (e.g. batch_integration), which will require you to use a different value for --type corresponding to the desired metric subtype.

Step 2: Fill in metadata

The Viash config contains metadata of your metric, which script is used to run it, and the required dependencies.

Generated config file

This is what the config.vsh.yaml generated by the create_component component looks like:

Required metadata fields

Please make sure that the following fields in the config file are filled in. The metrics component can contain several metric values these are listed in the info.metrics.

Step 3: Add dependencies

Each component has it’s own set of dependencies, because different components might have conflicting dependencies.

base images

For your convenience we have created several base images that can be used for python or R scripts. These images can be found in the OpenProblems Docker repository. Click on the packages to view the url you need to use. You are not required to use these images but install the required packages to make sure OpenProblems works properly.

openproblems/base_python Base image for python scripts.
openproblems/base_r Base image for R scripts.
openproblems/base_pytorch_nvidia Base image for scripts that use pytorch with nvidia gpu support.
openproblems/base_tensorflow_nvidia Base image for scripts that use tensorflow with nvidia gpu support.

custom image

Update the setup definition in the platforms section of the config file. This section describes the packages that need to be installed in the Docker image and are required for your method to run.

If you're using a custom image use the following minimum setup:

Please check out this guide for more information on how to add extra package dependencies.

Note

Tip: After making changes to the components dependencies, you will need to rebuild the docker container as follows:

viash run src//metrics/my_python_metric/config.vsh.yaml -- \
  ---setup cachedbuild

Output

#| echo: false
viash run src/metrics/my_python_metric/config.vsh.yaml -- \
  ---setup cachedbuild

Step 4: Edit script

A component's script typically has five sections:

Imports and libraries
Argument values
Read input data
Generate results
Write output data to file

Generated script

This is what the script generated by the create_component component looks like:

Required sections

Imports and libraries

In the top section of the script you can define which packages/libraries the metric needs. If you add a new or different package add the dependency to config.vsh.yaml in the setup field (see above).

Argument block

The Viash code block is designed to facilitate prototyping, by enabling you to execute directly by running python script.py (or Rscript script.R for R users). Note that anything between "VIASH START" and "VIASH END" will be removed and replaced with a CLI argument parser when the components are being built by Viash.

Here, the par dictionary contains all the arguments defined in the config.vsh.yaml file (including those from the defined __merge__ file). When adding a argument in the par dict also add it to the config.vsh.yaml in the arguments section.

viash test src/metrics/my_python_metric/config.vsh.yaml

Output

#| echo: false
# use f1 instead of 'my_metric' because the script won't work.
# maybe copy a 'working' script here
viash test src/metrics/accuracy/config.vsh.yaml

Visit "Run tests" for more information on running unit tests and how to interpret common error messages.

You can also run your component on local files using the viash run command. For example:

viash run src/metrics/my_python_metric/config.vsh.yaml -- \
  --input_prediction resources_test/task_template/cxg_mouse_pancreas_atlas/prediction.h5ad \
  --input_solution resources_test/task_template/cxg_mouse_pancreas_atlas/solution.h5ad \
  --output output.h5ad

Next steps

If your component works, please create a pull request.

Docs Navigation