Match Modalities
Match cells across datasets of the same set of samples on different technologies / modalities.
In this task, the goal is to learn a latent space where cells profiled by different technologies in different modalities are matched if they have the same state. We use jointly profiled data as ground truth so that we can evaluate when the observations from the same cell acquired using different modalities are similar. A perfect result has each of the paired observations sharing the same coordinates in the latent space. A method that can achieve this would be able to match datasets across modalities to enable multimodal cellular analysis from separately measured profiles.
Contributors
Leaderboard
Methods ranked by scaled overall mean. Each cell encodes a score from 0 to 1 by size and intensity.
QC: Normalisation Visualisation 2 plots
Per metric: points placed by control-anchored scaled score (x); dashed lines mark scaled 0 and 1 (worst/best control); the lower axis shows the raw score. Points beyond [-0.2, 1.2] are clamped to the edge as triangles. Hover a dot or line to highlight it and read details.
- kNN Area Under the Curvehigher better
- Mean Squared Errorlower better
QC: Indicator table 4 errors1 warning
Automated checks on the benchmark run and its results: missing values, score scaling, metric ranges and similar. Errors are high-severity issues that usually need a maintainer's attention; warnings are lower-severity signals. Findings that are expected for this task are listed separately as silenced.
4 high-severity issues need review. 75 of 80 checks passed.
- error Method info Info field 'references' % missing
Method info field 'references' should be defined Task: match_modalities Field: references Percentage missing: 67
- error Metric info Info field 'references' % missing
Metric info field 'references' should be defined Task: match_modalities Field: references Percentage missing: 100
- error Scaling Metric 'knn_auc' % outside range
Percentage of scaled scores outside control range should be less than 10% Task: match_modalities Metric: knn_auc Inside range: 0 Scaled scores: 18 Percentage outside: 100%
- error Scaling Metric 'mse' % outside range
Percentage of scaled scores outside control range should be less than 10% Task: match_modalities Metric: mse Inside range: 0 Scaled scores: 18 Percentage outside: 100%
Show 1 warning
- warning Scaling Metric 'mse' worst score % outside range
The worst scaled score should be less than 10% outside the control range Task: match_modalities Metric: mse Worst score: -0.157977727407115 Percentage outside range: 16%
Method info 4
A simpler version of the original mnnCorrect algorithm.
FastMNN is a simplified version of the mnnCorrect algorithm. Both use Mutual Nearest Neighbors to integrate multimodal single-cell data.
Harmonic Alignment
Harmonic Alignment is a method for integrating multimodal single-cell data. It is based on the idea of aligning the eigenvectors of the Laplacian matrices of the two modalities. The alignment is achieved by solving a generalized eigenvalue problem. The method is described in the following paper: https://doi.org/10.1137/1.9781611976236.36
Procrustes superimposition embeds cellular data from each modality into a common space.
Procrustes superimposition embeds cellular data from each modality into a common space by aligning the 100-dimensional SVD embeddings to one another by using an isomorphic transformation that minimizes the root mean squared distance between points. The unmodified SVD embedding and the transformed second modality are used as output for the task.
Control method info 2
Randomly permutated features
Randomly permuted twice, once for use as the output for each modality, producing random features with no correlation between modalities.
A 1 to 1 mapping of features between modalities
use the same features for both modalities
Metric info 2
Compute the kNN Area Under the Curve
Let $f(i) \in F$ be the scRNA-seq measurement of cell $i$, and $g(i) \in G$ be the scATAC- seq measurement of cell $i$. kNN-AUC calculates the average percentage overlap of neighborhoods of $f(i)$ in $F$ with neighborhoods of $g(i)$ in $G$. Higher is better.
Compute the mean squared error.
Mean squared error (MSE) is the average distance between each pair of matched observations of the same cell in the learned latent space. Lower is better.
Dataset info 3
CITE-seq profiles of 8k Cord Blood Mononuclear Cells
8k cord blood mononuclear cells profiled by CITEsequsing a panel of 13 antibodies.
sci-CAR profiles of 5k cell line cells (HEK293T, NIH/3T3, A549) across three treatment conditions (DEX 0h, 1h and 3h)
Single cell RNA-seq and ATAC-seq co-profiling for HEK293T cells, NIH/3T3 cells, A549 cells across three treatment conditions (DEX 0 hour, 1 hour and 3 hour treatment).
sci-CAR profiles of 11k mouse kidney cells
Single cell RNA-seq and ATAC-seq co-profiling of 11k mouse kidney cells.
References
- Cao, J., Cusanovich, D. A., Ramani, V., Aghamirzaie, D., Pliner, H. A., Hill, A. J., Daza, R. M., McFaline-Figueroa, J. L., Packer, J. S., Christiansen, L., Steemers, F. J., Adey, A. C., Trapnell, C., & Shendure, J. (2018). Joint profiling of chromatin accessibility and gene expression in thousands of single cells. Science, 361(6409), 1380–1385. 10.1126/science.aau0730 ↗
- Stoeckius, M., Hafemeister, C., Stephenson, W., Houck-Loomis, B., Chattopadhyay, P. K., Swerdlow, H., Satija, R., & Smibert, P. (2017). Simultaneous epitope and transcriptome measurement in single cells. Nature Methods, 14(9), 865–868. 10.1038/nmeth.4380 ↗