Run tests

A component in OpenProblems will typically come with at least two unit tests out of the box:

Use viash test to run all of the component’s unit tests.

Example

viash test src/label_projection/methods/knn/config.vsh.yaml
Output
Running tests in temporary directory: '/tmp/viash_test_knn3187009284176040043'
====================================================================
+/tmp/viash_test_knn3187009284176040043/build_executable/knn ---verbosity 6 ---setup cachedbuild
[notice] Building container 'ghcr.io/openproblems-bio/label_projection/methods/knn:test' with Dockerfile
[info] Running 'docker build -t ghcr.io/openproblems-bio/label_projection/methods/knn:test /tmp/viash_test_knn3187009284176040043/build_executable -f /tmp/viash_test_knn3187009284176040043/build_executable/tmp/dockerbuild-knn-DZXWgk/Dockerfile'
Sending build context to Docker daemon  39.94kB

Step 1/7 : FROM python:3.10
 ---> fc98d03e6037
Step 2/7 : RUN pip install --upgrade pip &&   pip install --upgrade --no-cache-dir "scikit-learn" "pyyaml" "anndata~=0.8.0"
 ---> Using cache
 ---> 1d35b64eb218
Step 3/7 : LABEL org.opencontainers.image.description="Companion container for running component label_projection/methods knn"
 ---> Using cache
 ---> f9833a51c1bc
Step 4/7 : LABEL org.opencontainers.image.created="2023-05-06T00:04:39Z"
 ---> Running in 9238fdf8b216
Removing intermediate container 9238fdf8b216
 ---> 397ac5725a53
Step 5/7 : LABEL org.opencontainers.image.source="https://github.com/openproblems-bio/openproblems-v2"
 ---> Running in dc9b554c5694
Removing intermediate container dc9b554c5694
 ---> bd13a2af1715
Step 6/7 : LABEL org.opencontainers.image.revision="9438b8ad0cdd9cd2ed3ba6a01d0b4f075c059d64"
 ---> Running in e6698eb0c299
Removing intermediate container e6698eb0c299
 ---> 285c4ebb0909
Step 7/7 : LABEL org.opencontainers.image.version="test"
 ---> Running in e72ae99420e1
Removing intermediate container e72ae99420e1
 ---> eea3261f2b6e
Successfully built eea3261f2b6e
Successfully tagged ghcr.io/openproblems-bio/label_projection/methods/knn:test
====================================================================
+/tmp/viash_test_knn3187009284176040043/test_check_method_config/test_executable
Load config data
Check general fields
Check info fields
All checks succeeded!
====================================================================
+/tmp/viash_test_knn3187009284176040043/test_run_and_check_adata/test_executable
>> Checking whether input files exist
>> Running script as test
Load input data
Fit to train data
Predict on test data
Write output to file
>> Checking whether output file exists
>> Reading h5ad files and checking formats
Reading and checking input_train
  AnnData object with n_obs × n_vars = 346 × 419
    obs: 'label', 'batch'
    var: 'hvg', 'hvg_score'
    uns: 'dataset_id', 'normalization_id'
    obsm: 'X_pca'
    layers: 'counts', 'normalized'
Reading and checking input_test
  AnnData object with n_obs × n_vars = 154 × 419
    obs: 'batch'
    var: 'hvg', 'hvg_score'
    uns: 'dataset_id', 'normalization_id'
    obsm: 'X_pca'
    layers: 'counts', 'normalized'
Reading and checking output
  AnnData object with n_obs × n_vars = 154 × 419
    obs: 'batch', 'label_pred'
    var: 'hvg', 'hvg_score'
    uns: 'dataset_id', 'method_id', 'normalization_id'
    obsm: 'X_pca'
    layers: 'counts', 'normalized'
All checks succeeded!
====================================================================
SUCCESS! All 2 out of 2 test scripts succeeded!
Cleaning up temporary directory

Test multiple components

Use viash ns test to unit test all of the components of a given task.

viash ns test --query label_projection --parallel --platform docker
Output
            namespace        functionality             platform            test_name exit_code duration               result
label_projection/methods  logistic_regression               docker                start                                        
label_projection/methods               scanvi               docker                start                                        
label_projection/methods                  knn               docker                start                                        
label_projection/methods                  mlp               docker                start                                        
label_projection/metrics             accuracy               docker                start                                        
label_projection/metrics                   f1               docker                start
label_projection/methods  logistic_regression               docker     build_executable         0        4              SUCCESS
label_projection/methods  logistic_regression               docker      generic_test.py         0        9              SUCCESS
label_projection/metrics                   f1               docker     build_executable         0        7              SUCCESS
label_projection/metrics                   f1               docker      format_check.py         0        8              SUCCESS
label_projection/metrics             accuracy               docker     build_executable         0        8              SUCCESS
label_projection/metrics             accuracy               docker      format_check.py         0        7              SUCCESS
...

Common errors

Below is a listing common errors and how to solve them. If you come across any other problems, please take a look at our troubleshooting page, or reach out via GitHub issues.

Assertion error

An assertion error typically occurs when data format of input or output parameters is incorrect.

Component script errors:

  • Output file cannot be found: Check that your script writes to the correct output filename.

  • Some fields/objects cannot be found in the output file: Check whether the correct fields are written in the output file.

+/tmp/viash_test_knn12471306149427017048/test_generic_test/test_executable
>> Running script as test
>> Checking whether output file exists
>> Reading h5ad files
>> Checking whether predictions were added
Traceback (most recent call last):
  File "/viash_automount/tmp/viash_test_knn12471306149427017048/test_generic_test/tmp//viash-run-knn-QTKmUM.py", line 57, in <module>
    assert "label_predi" in output.obs
AssertionError

Component config errors:

When these AssertionErrors occur, check the spelling of the missing value if it is present in the file. If the field is irrelevant you can simply add an empty string "" to make sure it is included in the composed config file.

+/tmp/viash_test_knn12945373156205296243/test_check_method_config/test_executable
Load config data
check general fields
Traceback (most recent call last):
Check info fields
  File "/viash_automount/tmp/viash_test_knn12945373156205296243/test_check_method_config/tmp//viash-run-knn-Xn2Vd7.py", line 42, in <module>
    assert "summary" in info is not None, "summary not an info field or is empty"
AssertionError: summary not an info field or is empty

Python / R dependency does not exist

When a dependency for the unit test or the executed script is not added to the setup of the docker you will get a ModuleNotFoundError. Add the dependency to the setup.

ModuleNotFoundError: No module named 'yaml'

Docker image not found

When this kind of error occurs make sure there are no spelling mistakes in the image name.

#3 ERROR: docker.io/library/python:3.1: not found
------
> [internal] load metadata for docker.io/library/python:3.1:
------
Dockerfile:1
--------------------
  1 | >>> FROM python:3.1
  2 |     
  3 |     RUN pip install --upgrade pip && \
--------------------
ERROR: failed to solve: python:3.1: docker.io/library/python:3.1: not found
[error] Error occurred while building container 'ghcr.io/openproblems-bio/label_projection/methods/knn:test'
ERROR! Setup failed!

Script error

When the executed script has an error it will be printed out like the example below. In most cases you can find the problem in the stack trace.

+/tmp/viash_test_knn14797416935521308344/test_generic_test/test_executable
>> Running script as test
Load input data
Traceback (most recent call last):
File "/tmp/viash-run-knn-p4pvkA.py", line 31, in <module>
  input_test = ad.read_h5ad(par['input_test'])
File "/usr/local/lib/python3.10/site-packages/anndata/_io/h5ad.py", line 224, in read_h5ad
  with h5py.File(filename, "r") as f:
File "/usr/local/lib/python3.10/site-packages/h5py/_hl/files.py", line 542, in __init__
  name = filename_encode(name)
File "/usr/local/lib/python3.10/site-packages/h5py/_hl/compat.py", line 19, in filename_encode
  filename = fspath(filename)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Method script with returncode ...