Deconvolution of Low-count RNA Sequencing Data for Tumor Cells Using Embedded Negative Binomial Distributions

Estimating tumor-specific transcript proportions can provide critical insights into tumor heterogeneity and cell plasticity. However, current methods often require comprehensive single-cell reference data, which is not always obtainable, and may suffer from technical biases.

To overcome these issues, we introduce DeMixNB, a semi-reference-based deconvolution method designed specifically for estimating transcript proportions from sparse expression matrices using a negative binomial model, without the reliance on high-quality external reference data. Through simulations and analyses of real microRNA sequencing and spatial transcriptomics data from breast and lung cancers, we demonstrate that DeMixNB accurately estimates tumor-specific transcript proportions and provides meaningful biological insights from real-world datasets.