The growing availability of single-cell data has sparked an increased interest in the inference of cell-cell communication (CCC), with an ever-growing number of computational tools developed for this purpose.
Different tools propose distinct preprocessing steps with diverse scoring functions, that are challenging to compare and evaluate. Furthermore, each tool typically comes with its own set of prior knowledge. To harmonize these, Dimitrov et al, 2022 recently developed the LIANA framework, which was used as a foundation for this task.
The challenges in evaluating the tools are further exacerbated by the lack of a gold standard to benchmark the performance of CCC methods. In an attempt to address this, Dimitrov et al use alternative data modalities, including the spatial proximity of cell types and downstream cytokine activities, to generate an inferred ground truth. However, these modalities are only approximations of biological reality and come with their own assumptions and limitations. In time, the inclusion of more datasets with known ground truth interactions will become available, from which the limitations and advantages of the different CCC methods will be better understood.
This subtask evaluates the methods’ ability to predict interactions, the corresponding of cytokines of which, are inferred to be active in the target cell types. This subtask focuses on the prediction of interactions from steady-state, or single-context, single-cell data.
Metrics for cell-cell communication aim to characterize how good are the different scoring methods at prioritizing assumed truth predictions.
Odds ratio: The odds ratio represents the ratio of true and false positives within a set of prioritized interactions (top ranked hits) versus the same ratio for the remainder of the interactions. Thus, in this scenario odds ratios quantify the strength of association between the ability of methods to prioritize interactions and those interactions assigned to the positive class.
AUPRC: a single number [0-1] that summarizes the area under the curve where x is the recall and y is the precision.