All benchmarks

Batch integration

Removing batch effects while preserving biological variation (embedding output)

10 methods
7 control methods
3 datasets
10 metrics
5 releases
Task repository MIT v1.0.0-embed

This is a sub-task of the overall batch integration task. Batch (or data) integration integrates datasets across batches that arise from various biological and technical sources. Methods that integrate batches typically have three different types of output: a corrected feature matrix, a joint embedding across batches, and/or an integrated cell-cell similarity graph (e.g., a kNN graph). This sub-task focuses on all methods that can output joint embeddings, and includes methods that canonically output corrected feature matrices with subsequent postprocessing to generate a joint embedding. Other sub-tasks for batch integration can be found for:

This sub-task was taken from a benchmarking study of data integration methods.

Contributors

  • Michaela Mueller
    maintainerauthor
  • Malte Luecken
    author
  • Daniel Strobl
    author
  • Robrecht Cannoodt
    contributor
  • Scott Gigante
    contributor
  • Kai Waldrant
    contributor
  • Nartin Kim
    contributor

Leaderboard

Methods ranked by scaled overall mean. Each cell encodes a score from 0 to 1 by size and intensity.

QC: Normalisation Visualisation 10 plots

Per metric: points placed by control-anchored scaled score (x); dashed lines mark scaled 0 and 1 (worst/best control); the lower axis shows the raw score. Points beyond [-0.2, 1.2] are clamped to the edge as triangles. Hover a dot or line to highlight it and read details.

methodcontrol
  • ARIhigher better
    Random Graph by Cellt…Random Graph by CelltypeRandom Embedding by C…Random Embedding by Celltype (with jitter)Random Embedding by C…Random Embedding by Celltype (default)scANVI (hvg/unscaled)scANVI (hvg/unscaled)scANVI (full/unscaled)scANVI (full/unscaled)Scanorama (hvg/scaled)Scanorama (hvg/scaled)scVI (hvg/unscaled)scVI (hvg/unscaled)scVI (full/unscaled)scVI (full/unscaled)FastMNN embed (hvg/un…FastMNN embed (hvg/unscaled)SCALEX (hvg)SCALEX (hvg)Combat (full/unscaled)Combat (full/unscaled)Scanorama (hvg/unscal…Scanorama (hvg/unscaled)MNN (hvg/scaled)MNN (hvg/scaled)Harmony (hvg/unscaled)Harmony (hvg/unscaled)Combat (hvg/unscaled)Combat (hvg/unscaled)Scanorama (full/scale…Scanorama (full/scaled)FastMNN embed (hvg/sc…FastMNN embed (hvg/scaled)Harmony (hvg/scaled)Harmony (hvg/scaled)SCALEX (full)SCALEX (full)Combat (hvg/scaled)Combat (hvg/scaled)Scanorama gene output…Scanorama gene output (hvg/scaled)MNN (full/unscaled)MNN (full/unscaled)Harmony (full/unscale…Harmony (full/unscaled)MNN (hvg/unscaled)MNN (hvg/unscaled)FastMNN embed (full/u…FastMNN embed (full/unscaled)FastMNN embed (full/s…FastMNN embed (full/scaled)Scanorama gene output…Scanorama gene output (hvg/unscaled)Harmony (full/scaled)Harmony (full/scaled)Scanorama gene output…Scanorama gene output (full/scaled)MNN (full/scaled)MNN (full/scaled)Scanorama gene output…Scanorama gene output (full/unscaled)Scanorama (full/unsca…Scanorama (full/unscaled)Combat (full/scaled)Combat (full/scaled)Liger (hvg/unscaled)Liger (hvg/unscaled)Liger (full/unscaled)Liger (full/unscaled)No IntegrationNo IntegrationRandom Integration by…Random Integration by CelltypeNo Integration by Bat…No Integration by BatchRandom Integration by…Random Integration by BatchRandom IntegrationRandom Integration-3.0e-40.250.50.75100.250.50.751rawscaled
  • Batch ASWhigher better
    Random Embedding by C…Random Embedding by Celltype (default)Random Graph by Cellt…Random Graph by CelltypeRandom Embedding by C…Random Embedding by Celltype (with jitter)MNN (full/scaled)MNN (full/scaled)Scanorama (full/scale…Scanorama (full/scaled)Scanorama (hvg/scaled)Scanorama (hvg/scaled)Combat (full/scaled)Combat (full/scaled)Scanorama (hvg/unscal…Scanorama (hvg/unscaled)Scanorama (full/unsca…Scanorama (full/unscaled)MNN (hvg/scaled)MNN (hvg/scaled)MNN (hvg/unscaled)MNN (hvg/unscaled)MNN (full/unscaled)MNN (full/unscaled)Combat (full/unscaled)Combat (full/unscaled)Harmony (full/scaled)Harmony (full/scaled)Combat (hvg/unscaled)Combat (hvg/unscaled)Harmony (hvg/scaled)Harmony (hvg/scaled)scVI (hvg/unscaled)scVI (hvg/unscaled)Harmony (full/unscale…Harmony (full/unscaled)SCALEX (hvg)SCALEX (hvg)scVI (full/unscaled)scVI (full/unscaled)scANVI (full/unscaled)scANVI (full/unscaled)scANVI (hvg/unscaled)scANVI (hvg/unscaled)SCALEX (full)SCALEX (full)Harmony (hvg/unscaled)Harmony (hvg/unscaled)Combat (hvg/scaled)Combat (hvg/scaled)Scanorama gene output…Scanorama gene output (full/unscaled)Random IntegrationRandom IntegrationRandom Integration by…Random Integration by BatchRandom Integration by…Random Integration by CelltypeScanorama gene output…Scanorama gene output (hvg/unscaled)Scanorama gene output…Scanorama gene output (full/scaled)FastMNN embed (full/u…FastMNN embed (full/unscaled)FastMNN embed (full/s…FastMNN embed (full/scaled)Scanorama gene output…Scanorama gene output (hvg/scaled)FastMNN embed (hvg/sc…FastMNN embed (hvg/scaled)FastMNN embed (hvg/un…FastMNN embed (hvg/unscaled)Liger (full/unscaled)Liger (full/unscaled)Liger (hvg/unscaled)Liger (hvg/unscaled)No IntegrationNo IntegrationNo Integration by Bat…No Integration by Batch0.7150.7860.8570.929100.250.50.751rawscaled
  • Cell Cycle Scorehigher better
    No Integration by Bat…No Integration by BatchCombat (hvg/unscaled)Combat (hvg/unscaled)SCALEX (hvg)SCALEX (hvg)MNN (hvg/unscaled)MNN (hvg/unscaled)Harmony (hvg/unscaled)Harmony (hvg/unscaled)MNN (full/scaled)MNN (full/scaled)FastMNN embed (hvg/un…FastMNN embed (hvg/unscaled)FastMNN embed (hvg/sc…FastMNN embed (hvg/scaled)Harmony (hvg/scaled)Harmony (hvg/scaled)Harmony (full/unscale…Harmony (full/unscaled)MNN (full/unscaled)MNN (full/unscaled)No IntegrationNo IntegrationCombat (full/unscaled)Combat (full/unscaled)MNN (hvg/scaled)MNN (hvg/scaled)SCALEX (full)SCALEX (full)Combat (hvg/scaled)Combat (hvg/scaled)FastMNN embed (full/s…FastMNN embed (full/scaled)FastMNN embed (full/u…FastMNN embed (full/unscaled)Combat (full/scaled)Combat (full/scaled)scANVI (full/unscaled)scANVI (full/unscaled)Harmony (full/scaled)Harmony (full/scaled)Random Embedding by C…Random Embedding by Celltype (default)Random Embedding by C…Random Embedding by Celltype (with jitter)Random Graph by Cellt…Random Graph by CelltypescANVI (hvg/unscaled)scANVI (hvg/unscaled)scVI (full/unscaled)scVI (full/unscaled)scVI (hvg/unscaled)scVI (hvg/unscaled)Liger (full/unscaled)Liger (full/unscaled)Liger (hvg/unscaled)Liger (hvg/unscaled)Random Integration by…Random Integration by CelltypeScanorama gene output…Scanorama gene output (full/unscaled)Scanorama (hvg/unscal…Scanorama (hvg/unscaled)Scanorama gene output…Scanorama gene output (hvg/unscaled)Scanorama gene output…Scanorama gene output (hvg/scaled)Scanorama (full/unsca…Scanorama (full/unscaled)Scanorama (hvg/scaled)Scanorama (hvg/scaled)Scanorama (full/scale…Scanorama (full/scaled)Scanorama gene output…Scanorama gene output (full/scaled)Random IntegrationRandom IntegrationRandom Integration by…Random Integration by Batch0.0140.2610.5070.754100.250.50.751rawscaled
  • Graph connectivityhigher better
    Random Graph by Cellt…Random Graph by CelltypeRandom Embedding by C…Random Embedding by Celltype (with jitter)Random Embedding by C…Random Embedding by Celltype (default)MNN (full/unscaled)MNN (full/unscaled)MNN (hvg/unscaled)MNN (hvg/unscaled)scANVI (full/unscaled)scANVI (full/unscaled)scANVI (hvg/unscaled)scANVI (hvg/unscaled)scVI (hvg/unscaled)scVI (hvg/unscaled)scVI (full/unscaled)scVI (full/unscaled)MNN (hvg/scaled)MNN (hvg/scaled)Scanorama (full/scale…Scanorama (full/scaled)Combat (hvg/unscaled)Combat (hvg/unscaled)SCALEX (hvg)SCALEX (hvg)Scanorama (hvg/scaled)Scanorama (hvg/scaled)MNN (full/scaled)MNN (full/scaled)SCALEX (full)SCALEX (full)Combat (full/unscaled)Combat (full/unscaled)Combat (hvg/scaled)Combat (hvg/scaled)FastMNN embed (hvg/un…FastMNN embed (hvg/unscaled)FastMNN embed (hvg/sc…FastMNN embed (hvg/scaled)Combat (full/scaled)Combat (full/scaled)FastMNN embed (full/s…FastMNN embed (full/scaled)FastMNN embed (full/u…FastMNN embed (full/unscaled)Harmony (hvg/unscaled)Harmony (hvg/unscaled)Harmony (full/unscale…Harmony (full/unscaled)Harmony (hvg/scaled)Harmony (hvg/scaled)Scanorama (hvg/unscal…Scanorama (hvg/unscaled)Scanorama gene output…Scanorama gene output (full/scaled)Harmony (full/scaled)Harmony (full/scaled)Scanorama gene output…Scanorama gene output (hvg/scaled)Scanorama gene output…Scanorama gene output (hvg/unscaled)Scanorama (full/unsca…Scanorama (full/unscaled)Scanorama gene output…Scanorama gene output (full/unscaled)No IntegrationNo IntegrationRandom Integration by…Random Integration by CelltypeNo Integration by Bat…No Integration by BatchLiger (full/unscaled)Liger (full/unscaled)Liger (hvg/unscaled)Liger (hvg/unscaled)Random Integration by…Random Integration by BatchRandom IntegrationRandom Integration0.0480.2860.5240.762100.250.50.751rawscaled
  • Isolated label F1higher better
    Random Graph by Cellt…Random Graph by CelltypeRandom Embedding by C…Random Embedding by Celltype (with jitter)Random Embedding by C…Random Embedding by Celltype (default)Scanorama (hvg/scaled)Scanorama (hvg/scaled)Scanorama (full/scale…Scanorama (full/scaled)scANVI (hvg/unscaled)scANVI (hvg/unscaled)scVI (hvg/unscaled)scVI (hvg/unscaled)Scanorama gene output…Scanorama gene output (hvg/scaled)scVI (full/unscaled)scVI (full/unscaled)scANVI (full/unscaled)scANVI (full/unscaled)MNN (hvg/scaled)MNN (hvg/scaled)Scanorama gene output…Scanorama gene output (hvg/unscaled)Scanorama (full/unsca…Scanorama (full/unscaled)Scanorama gene output…Scanorama gene output (full/scaled)MNN (hvg/unscaled)MNN (hvg/unscaled)Scanorama (hvg/unscal…Scanorama (hvg/unscaled)Scanorama gene output…Scanorama gene output (full/unscaled)Combat (hvg/scaled)Combat (hvg/scaled)MNN (full/unscaled)MNN (full/unscaled)Combat (full/unscaled)Combat (full/unscaled)Harmony (full/unscale…Harmony (full/unscaled)Combat (hvg/unscaled)Combat (hvg/unscaled)FastMNN embed (hvg/sc…FastMNN embed (hvg/scaled)Harmony (hvg/unscaled)Harmony (hvg/unscaled)FastMNN embed (hvg/un…FastMNN embed (hvg/unscaled)Random Integration by…Random Integration by CelltypeNo IntegrationNo IntegrationFastMNN embed (full/u…FastMNN embed (full/unscaled)FastMNN embed (full/s…FastMNN embed (full/scaled)Harmony (full/scaled)Harmony (full/scaled)MNN (full/scaled)MNN (full/scaled)Harmony (hvg/scaled)Harmony (hvg/scaled)SCALEX (hvg)SCALEX (hvg)Combat (full/scaled)Combat (full/scaled)SCALEX (full)SCALEX (full)No Integration by Bat…No Integration by BatchLiger (hvg/unscaled)Liger (hvg/unscaled)Liger (full/unscaled)Liger (full/unscaled)Random Integration by…Random Integration by BatchRandom IntegrationRandom Integration0.0290.2720.5140.757100.250.50.751rawscaled
  • Isolated label Silhouettehigher better
    Random Embedding by C…Random Embedding by Celltype (default)Random Embedding by C…Random Embedding by Celltype (with jitter)Random Graph by Cellt…Random Graph by CelltypeScanorama gene output…Scanorama gene output (hvg/unscaled)Scanorama gene output…Scanorama gene output (hvg/scaled)scANVI (hvg/unscaled)scANVI (hvg/unscaled)MNN (hvg/scaled)MNN (hvg/scaled)Scanorama gene output…Scanorama gene output (full/unscaled)MNN (hvg/unscaled)MNN (hvg/unscaled)scANVI (full/unscaled)scANVI (full/unscaled)Scanorama gene output…Scanorama gene output (full/scaled)Random Integration by…Random Integration by CelltypeNo IntegrationNo IntegrationscVI (hvg/unscaled)scVI (hvg/unscaled)Scanorama (hvg/unscal…Scanorama (hvg/unscaled)Scanorama (hvg/scaled)Scanorama (hvg/scaled)Combat (hvg/scaled)Combat (hvg/scaled)scVI (full/unscaled)scVI (full/unscaled)Harmony (full/unscale…Harmony (full/unscaled)Combat (hvg/unscaled)Combat (hvg/unscaled)Scanorama (full/scale…Scanorama (full/scaled)Scanorama (full/unsca…Scanorama (full/unscaled)Combat (full/unscaled)Combat (full/unscaled)Harmony (hvg/scaled)Harmony (hvg/scaled)Harmony (full/scaled)Harmony (full/scaled)SCALEX (hvg)SCALEX (hvg)MNN (full/unscaled)MNN (full/unscaled)Harmony (hvg/unscaled)Harmony (hvg/unscaled)SCALEX (full)SCALEX (full)FastMNN embed (hvg/un…FastMNN embed (hvg/unscaled)FastMNN embed (hvg/sc…FastMNN embed (hvg/scaled)MNN (full/scaled)MNN (full/scaled)FastMNN embed (full/s…FastMNN embed (full/scaled)FastMNN embed (full/u…FastMNN embed (full/unscaled)Combat (full/scaled)Combat (full/scaled)No Integration by Bat…No Integration by BatchRandom Integration by…Random Integration by BatchLiger (hvg/unscaled)Liger (hvg/unscaled)Liger (full/unscaled)Liger (full/unscaled)Random IntegrationRandom Integration0.4620.5960.7310.865100.250.50.751rawscaled
  • kBEThigher better
    Random Graph by Cellt…Random Graph by CelltypeRandom Integration by…Random Integration by CelltypeRandom Embedding by C…Random Embedding by Celltype (with jitter)Random Embedding by C…Random Embedding by Celltype (default)Harmony (hvg/scaled)Harmony (hvg/scaled)Harmony (full/scaled)Harmony (full/scaled)Liger (hvg/unscaled)Liger (hvg/unscaled)MNN (full/scaled)MNN (full/scaled)Combat (full/scaled)Combat (full/scaled)Harmony (hvg/unscaled)Harmony (hvg/unscaled)Random IntegrationRandom IntegrationFastMNN embed (full/s…FastMNN embed (full/scaled)FastMNN embed (full/u…FastMNN embed (full/unscaled)Liger (full/unscaled)Liger (full/unscaled)Scanorama (hvg/scaled)Scanorama (hvg/scaled)FastMNN embed (hvg/un…FastMNN embed (hvg/unscaled)FastMNN embed (hvg/sc…FastMNN embed (hvg/scaled)scVI (hvg/unscaled)scVI (hvg/unscaled)scVI (full/unscaled)scVI (full/unscaled)scANVI (full/unscaled)scANVI (full/unscaled)Harmony (full/unscale…Harmony (full/unscaled)scANVI (hvg/unscaled)scANVI (hvg/unscaled)SCALEX (full)SCALEX (full)Scanorama (full/scale…Scanorama (full/scaled)SCALEX (hvg)SCALEX (hvg)Scanorama gene output…Scanorama gene output (hvg/scaled)Scanorama gene output…Scanorama gene output (full/scaled)Scanorama (hvg/unscal…Scanorama (hvg/unscaled)Scanorama (full/unsca…Scanorama (full/unscaled)Scanorama gene output…Scanorama gene output (full/unscaled)Scanorama gene output…Scanorama gene output (hvg/unscaled)MNN (full/unscaled)MNN (full/unscaled)MNN (hvg/scaled)MNN (hvg/scaled)MNN (hvg/unscaled)MNN (hvg/unscaled)Combat (hvg/scaled)Combat (hvg/scaled)Combat (full/unscaled)Combat (full/unscaled)No IntegrationNo IntegrationCombat (hvg/unscaled)Combat (hvg/unscaled)No Integration by Bat…No Integration by BatchRandom Integration by…Random Integration by Batch2.7e-40.240.4790.7180.95700.250.50.751rawscaled
  • NMIhigher better
    Random Graph by Cellt…Random Graph by CelltypeRandom Embedding by C…Random Embedding by Celltype (with jitter)Random Embedding by C…Random Embedding by Celltype (default)scANVI (hvg/unscaled)scANVI (hvg/unscaled)scANVI (full/unscaled)scANVI (full/unscaled)Scanorama (hvg/unscal…Scanorama (hvg/unscaled)Scanorama (hvg/scaled)Scanorama (hvg/scaled)scVI (full/unscaled)scVI (full/unscaled)scVI (hvg/unscaled)scVI (hvg/unscaled)MNN (hvg/scaled)MNN (hvg/scaled)SCALEX (hvg)SCALEX (hvg)FastMNN embed (hvg/un…FastMNN embed (hvg/unscaled)Combat (full/unscaled)Combat (full/unscaled)MNN (hvg/unscaled)MNN (hvg/unscaled)MNN (full/unscaled)MNN (full/unscaled)Combat (hvg/scaled)Combat (hvg/scaled)Scanorama (full/scale…Scanorama (full/scaled)Combat (hvg/unscaled)Combat (hvg/unscaled)Scanorama gene output…Scanorama gene output (hvg/scaled)Harmony (hvg/unscaled)Harmony (hvg/unscaled)Scanorama gene output…Scanorama gene output (hvg/unscaled)SCALEX (full)SCALEX (full)FastMNN embed (hvg/sc…FastMNN embed (hvg/scaled)Harmony (hvg/scaled)Harmony (hvg/scaled)Harmony (full/unscale…Harmony (full/unscaled)Scanorama gene output…Scanorama gene output (full/unscaled)FastMNN embed (full/u…FastMNN embed (full/unscaled)FastMNN embed (full/s…FastMNN embed (full/scaled)Scanorama (full/unsca…Scanorama (full/unscaled)Scanorama gene output…Scanorama gene output (full/scaled)Harmony (full/scaled)Harmony (full/scaled)MNN (full/scaled)MNN (full/scaled)No IntegrationNo IntegrationRandom Integration by…Random Integration by CelltypeCombat (full/scaled)Combat (full/scaled)No Integration by Bat…No Integration by BatchLiger (hvg/unscaled)Liger (hvg/unscaled)Liger (full/unscaled)Liger (full/unscaled)Random Integration by…Random Integration by BatchRandom IntegrationRandom Integration4.1e-30.2530.5020.751100.250.50.751rawscaled
  • PC Regressionhigher better
    Combat (hvg/scaled)Combat (hvg/scaled)No Integration by Bat…No Integration by BatchCombat (full/scaled)Combat (full/scaled)Combat (hvg/unscaled)Combat (hvg/unscaled)Combat (full/unscaled)Combat (full/unscaled)SCALEX (full)SCALEX (full)Random IntegrationRandom IntegrationSCALEX (hvg)SCALEX (hvg)MNN (full/scaled)MNN (full/scaled)Harmony (full/scaled)Harmony (full/scaled)scVI (full/unscaled)scVI (full/unscaled)Liger (full/unscaled)Liger (full/unscaled)scANVI (full/unscaled)scANVI (full/unscaled)Scanorama (full/scale…Scanorama (full/scaled)Liger (hvg/unscaled)Liger (hvg/unscaled)scVI (hvg/unscaled)scVI (hvg/unscaled)MNN (hvg/scaled)MNN (hvg/scaled)Scanorama (hvg/scaled)Scanorama (hvg/scaled)Harmony (hvg/scaled)Harmony (hvg/scaled)Random Integration by…Random Integration by CelltypeScanorama gene output…Scanorama gene output (full/scaled)MNN (full/unscaled)MNN (full/unscaled)scANVI (hvg/unscaled)scANVI (hvg/unscaled)Random Embedding by C…Random Embedding by Celltype (with jitter)Random Graph by Cellt…Random Graph by CelltypeRandom Embedding by C…Random Embedding by Celltype (default)FastMNN embed (full/u…FastMNN embed (full/unscaled)FastMNN embed (full/s…FastMNN embed (full/scaled)Scanorama gene output…Scanorama gene output (hvg/scaled)Harmony (hvg/unscaled)Harmony (hvg/unscaled)Harmony (full/unscale…Harmony (full/unscaled)MNN (hvg/unscaled)MNN (hvg/unscaled)FastMNN embed (hvg/sc…FastMNN embed (hvg/scaled)FastMNN embed (hvg/un…FastMNN embed (hvg/unscaled)Scanorama (hvg/unscal…Scanorama (hvg/unscaled)Scanorama gene output…Scanorama gene output (full/unscaled)Scanorama gene output…Scanorama gene output (hvg/unscaled)Scanorama (full/unsca…Scanorama (full/unscaled)Random Integration by…Random Integration by BatchNo IntegrationNo Integration00.250.50.75100.250.50.751rawscaled
  • Silhouettehigher better
    Random Embedding by C…Random Embedding by Celltype (default)Random Embedding by C…Random Embedding by Celltype (with jitter)Random Graph by Cellt…Random Graph by CelltypeFastMNN embed (hvg/sc…FastMNN embed (hvg/scaled)FastMNN embed (hvg/un…FastMNN embed (hvg/unscaled)Scanorama gene output…Scanorama gene output (hvg/unscaled)scANVI (hvg/unscaled)scANVI (hvg/unscaled)Scanorama gene output…Scanorama gene output (hvg/scaled)MNN (hvg/unscaled)MNN (hvg/unscaled)SCALEX (hvg)SCALEX (hvg)FastMNN embed (full/s…FastMNN embed (full/scaled)FastMNN embed (full/u…FastMNN embed (full/unscaled)Scanorama (hvg/unscal…Scanorama (hvg/unscaled)MNN (hvg/scaled)MNN (hvg/scaled)Harmony (hvg/unscaled)Harmony (hvg/unscaled)Combat (hvg/scaled)Combat (hvg/scaled)Harmony (hvg/scaled)Harmony (hvg/scaled)scANVI (full/unscaled)scANVI (full/unscaled)Scanorama gene output…Scanorama gene output (full/unscaled)Scanorama (hvg/scaled)Scanorama (hvg/scaled)Combat (hvg/unscaled)Combat (hvg/unscaled)MNN (full/unscaled)MNN (full/unscaled)Combat (full/unscaled)Combat (full/unscaled)SCALEX (full)SCALEX (full)scVI (hvg/unscaled)scVI (hvg/unscaled)scVI (full/unscaled)scVI (full/unscaled)Scanorama (full/scale…Scanorama (full/scaled)Harmony (full/scaled)Harmony (full/scaled)Scanorama gene output…Scanorama gene output (full/scaled)Scanorama (full/unsca…Scanorama (full/unscaled)Harmony (full/unscale…Harmony (full/unscaled)No IntegrationNo IntegrationRandom Integration by…Random Integration by CelltypeCombat (full/scaled)Combat (full/scaled)MNN (full/scaled)MNN (full/scaled)Liger (hvg/unscaled)Liger (hvg/unscaled)No Integration by Bat…No Integration by BatchLiger (full/unscaled)Liger (full/unscaled)Random IntegrationRandom IntegrationRandom Integration by…Random Integration by Batch0.4150.5610.7080.854100.250.50.751rawscaled
QC: Indicator table all clear

Automated checks on the benchmark run and its results: missing values, score scaling, metric ranges and similar. Errors are high-severity issues that usually need a maintainer's attention; warnings are lower-severity signals. Findings that are expected for this task are listed separately as silenced.

No high-severity issues. 837 of 837 checks passed.

Method info 10

ComBat uses an Empirical Bayes (EB) approach to correct for batch effects. It estimates batch-specific parameters by pooling information across genes in each batch and shrinks the estimates towards the overall mean of the batch effect estimates across all genes. These parameters are then used to adjust the data for batch effects, leading to more accurate and reproducible results.

parameter sets tested full/scaledfull/unscaledhvg/scaledhvg/unscaled

fastMNN performs a multi-sample PCA to reduce dimensionality, identifying MNN paris in the low-dimensional space, and then correcting the target batch towards the reference using locally weighted correction vectors. The corrected target batch is then merged with the reference. The process is repeated with the next target batch except for the PCA step.

parameter sets tested full/scaledfull/unscaledhvg/scaledhvg/unscaled

Harmony is a method that uses PCA to group the cells into multi-dataset clusters, and then computes cluster-specific linear correction factors. Each cell is then corrected by its cell-specific linear factor using the cluster-weighted average. The method keeps iterating these four steps until cell clusters are stable.

parameter sets tested full/scaledfull/unscaledhvg/scaledhvg/unscaled

LIGER or linked inference of genomic experimental relationships uses iNMF deriving and implementing a novel coordinate descent algorithm to efficiently do the factorization. Joint clustering is performed and factor loadings are normalised.

parameter sets tested full/unscaledhvg/unscaled

MNN first detects mutual nearest neighbours in two of the batches and infers a projection of the second onto the first batch. After that, additional batches are added iteratively.

parameter sets tested full/scaledfull/unscaledhvg/scaledhvg/unscaled

SCALEX is a method for integrating heterogeneous single-cell data online using a VAE framework. Its generalised encoder disentangles batch-related components from batch-invariant biological components, which are then projected into a common cell-embedding space.

parameter sets tested fullhvg

Scanorama is an extension of the MNN method. Other then MNN, it finds mutual nearest neighbours over all batches and embeds observations into a joint hyperplane.

parameter sets tested full/scaledfull/unscaledhvg/scaledhvg/unscaled

Scanorama is an extension of the MNN method. Other then MNN, it finds mutual nearest neighbours over all batches and embeds observations into a joint hyperplane.

parameter sets tested full/scaledfull/unscaledhvg/scaledhvg/unscaled

ScanVI is an extension of scVI but instead using a Bayesian semi-supervised approach for more principled cell annotation.

parameter sets tested full/unscaledhvg/unscaled

scVI combines a variational autoencoder with a hierarchical Bayesian model.

parameter sets tested full/unscaledhvg/unscaled
Control method info 7

Cells are embedded by PCA on the unintegrated data. A graph is built on this PCA embedding.

Cells are embedded by computing PCA independently on each batch

Cells are embedded as a one-hot encoding of celltype labels

Cells are embedded as a one-hot encoding of celltype labels. A graph is then built on this embedding

Feature values, embedding coordinates, and graph connectivity are all randomly permuted

Feature values, embedding coordinates, and graph connectivity are all randomly permuted within each batch label

Feature values, embedding coordinates, and graph connectivity are all randomly permuted within each celltype label

Metric info 10
ARIhigher is betterLuecken et al., 2021

ARI (Adjusted Rand Index) compares the overlap of two clusterings. It considers both correct clustering overlaps while also counting correct disagreements between two clustering.

Batch ASWhigher is betterLuecken et al., 2021

The absolute silhouette width is computed over batch labels per cell. As 0 then indicates that batches are well mixed and any deviation from 0 indicates a batch effect, we use the 1-abs(ASW) to map the score to the scale [0;1].

Cell Cycle Scorehigher is betterLuecken et al., 2021

The cell-cycle conservation score evaluates how well the cell-cycle effect can be captured before and after integration.

Graph connectivityhigher is betterLuecken et al., 2021

The graph connectivity metric assesses whether the kNN graph representation, G, of the integrated data connects all cells with the same cell identity label.

Isolated label F1higher is betterLuecken et al., 2021

Isolated cell labels are identified as the labels present in the least number of batches in the integration task. The score evaluates how well these isolated labels separate from other cell identities based on clustering.

Isolated label Silhouettehigher is betterLuecken et al., 2021

This score evaluates the compactness for the label(s) that is(are) shared by fewest batches. It indicates how well rare cell types can be preserved after integration.

kBEThigher is betterBüttner et al., 2018

kBET determines whether the label composition of a k nearest neighborhood of a cell is similar to the expected (global) label composition. The test is repeated for a random subset of cells, and the results are summarized as a rejection rate over all tested neighborhoods.

NMIhigher is betterLuecken et al., 2021

NMI compares the overlap of two clusterings. We used NMI to compare the cell-type labels with Louvain clusters computed on the integrated dataset.

PC Regressionhigher is betterLuecken et al., 2021

This compares the explained variance by batch before and after integration. It returns a score between 0 and 1 (scaled=True) with 0 if the variance contribution hasn’t changed. The larger the score, the more different the variance contributions are before and after integration.

Silhouettehigher is betterLuecken et al., 2021

The absolute silhouette with is computed on cell identity labels, measuring their compactness.

Dataset info 3
Immune (by batch) unlinked

Human immune cells from peripheral blood and bone marrow taken from 5 datasets comprising 10 batches across technologies (10X, Smart-seq2).

Lung (Viera Braga et al.) unlinked

Human lung scRNA-seq data from 3 datasets with 32,472 cells. From Vieira Braga et al. Technologies: 10X and Drop-seq.

Pancreas (by batch) unlinked

Human pancreatic islet scRNA-seq data from 6 datasets across technologies (CEL-seq, CEL-seq2, Smart-seq2, inDrop, Fluidigm C1, and SMARTER-seq).

References

  1. Büttner, M., Miao, Z., Wolf, F. A., Teichmann, S. A., & Theis, F. J. (2018). A test metric for assessing single-cell RNA-seq batch correction. Nature Methods, 16(1), 43–49. 10.1038/s41592-018-0254-1 ↗
  2. Open Problems for Single Cell Analysis Consortium. (2022). Open Problems. link ↗
  3. Haghverdi, L., Lun, A. T. L., Morgan, M. D., & Marioni, J. C. (2018). Batch effects in single-cell RNA-sequencing data are corrected by matching mutual nearest neighbors. Nature Biotechnology, 36(5), 421–427. 10.1038/nbt.4091 ↗
  4. Hie, B., Bryson, B., & Berger, B. (2019). Efficient integration of heterogeneous single-cell transcriptomes using Scanorama. Nature Biotechnology, 37(6), 685–691. 10.1038/s41587-019-0113-3 ↗
  5. Johnson, W. E., Li, C., & Rabinovic, A. (2006). Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics, 8(1), 118–127. 10.1093/biostatistics/kxj037 ↗
  6. Korsunsky, I., Millard, N., Fan, J., Slowikowski, K., Zhang, F., Wei, K., Baglaenko, Y., Brenner, M., ru Po-Loh, & Raychaudhuri, S. (2019). Fast, sensitive and accurate integration of single-cell data with Harmony. Nature Methods, 16(12), 1289–1296. 10.1038/s41592-019-0619-0 ↗
  7. Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., & Yosef, N. (2018). Deep generative modeling for single-cell transcriptomics. Nature Methods, 15(12), 1053–1058. 10.1038/s41592-018-0229-2 ↗
  8. Luecken, M. D., Büttner, M., Chaichoompu, K., Danese, A., Interlandi, M., Mueller, M. F., Strobl, D. C., Zappia, L., Dugas, M., Colomé-Tatché, M., & Theis, F. J. (2021). Benchmarking atlas-level data integration in single-cell genomics. Nature Methods, 19(1), 41–50. 10.1038/s41592-021-01336-8 ↗
  9. Lun, A. (2019). A description of the theory behind the fastMNN algorithm. link ↗
  10. Welch, J. D., Kozareva, V., Ferreira, A., Vanderburg, C., Martin, C., & Macosko, E. Z. (2019). Single-Cell Multi-omic Integration Compares and Contrasts Features of Brain Cell Identity. Cell, 177(7), 1873-1887.e17. 10.1016/j.cell.2019.05.006 ↗
  11. Xiong, L., Tian, K., Li, Y., Ning, W., Gao, X., & Zhang, Q. C. (2022). Online single-cell data integration through projecting heterogeneous datasets into a common cell-embedding space. Nature Communications, 13(1). 10.1038/s41467-022-33758-z ↗
  12. Xu, C., Lopez, R., Mehlman, E., Regier, J., Jordan, M. I., & Yosef, N. (2021). Probabilistic harmonization and annotation of single-cell transcriptomics data with deep generative models. Molecular Systems Biology, 17(1). 10.15252/msb.20209620 ↗