Abstract
Spatial transcriptomics (ST) integrates gene expression with spatial location, enabling precise mapping of cellular distributions and interactions within tissues, and is a key tool for understanding tissue structure and function. Single-cell RNA sequencing (scRNA-seq) data enhances spatial transcriptomics by providing accurate cell type deconvolution, yet existing methods still face accuracy challenges. We propose GraphCellNet, a model combining cell type deconvolution and spatial domain identification, featuring the Kolmogorov-Arnold Network layer (KAN) to enhance nonlinear feature representation and contextual integration. This design addresses ambiguous cell boundaries and high heterogeneity, improving analytical precision. Evaluated using metrics like Pearson correlation coefficient (PCC), structural similarity index (SSIM), root mean square error (RMSE), Jensen-Shannon divergence (JSD), and Adjusted Rand Index (ARI), GraphCellNet has been applied to various systems, yielding new insights. In myocardial infarction, it identified spatial regions with high Trem2 expression associated with metabolic gene signatures in the infarcted heart. In Drosophila development, it uncovered TWEEDLE dynamics. In human heart development, it identified cell compositions and spatial organization across stages, deepening understanding of cellular spatial dynamics and informing regenerative medicine. KEY MESSAGES: A novel deep learning architecture that effectively captures cellular composition and spatial organization in tissue samples. An innovative KAN layer design that improves the modeling of nonlinear gene expression relationships while maintaining computational efficiency. A graph-based spatial domain identification method that leverages the spatial relationships of cell type information to enhance domain recognition accuracy. Demonstration of the framework's applicability in various biological applications, providing new insights into tissue organization and development.
