Multimodal Data Integration: How Artificial Intelligence Is Revolutionizing Cancer Care

Author(s): Max Charney

Originally published on Towards AI.

I recently read this article (link) about multimodal data integration for oncology with artificial intelligence (AI). I found the topics covered super interesting because of the new potential for accurate patient outcome predictions as well as opportunities for finding relevant biomarkers/biological pathways. Very cool. Let’s dive into it.

Multimodal Data refers to different types of information sources (data). Cancer can be characterized by many types of modalities, such as radiology (images), genomics (genes, DNA), and histology (tissue samples). Current AI primarily operates with single modalities, neglecting the broader clinical contexts and opportunities, ultimately diminishing their effectiveness and potential. Multimodal data integration enables models to discover novel patterns within and across modalities, leading to the exploration and discovery of novel biomarkers, better prediction of patient outcomes or treatment results, and overall improved personalized care.

Multimodal Data Fusion is the integration of multiple data sources. There are three main types of data fusion: early fusion, intermediate fusion, and late fusion. In essence, each term refers to the stage of the deep learning framework at which the fusion of multimodal data occurs.

Survival Prediction, or the prediction of how long a patient will survive, is often performed with the Cox proportional hazards (Cox-PH) model. This multivariate linear regression model requires input data/features for its predictions. However, in the modern day, there’s so much medical information! We now use artificial intelligence (supervised, weakly supervised, and unsupervised learning methods) to extract features from data and use them as input data in Cox-PH models. Multimodal data integration in AI has proved to result in better survival prediction results.

Model Interpretability/Introspection is critical to AI development, deployment, and validation. While AI models have the ability to learn abstract feature representations, there is concern that models will use ineffective, spurious shortcuts rather than learning clinically relevant aspects. In order to learn more, scientists have come up with methods to see how the model is thinking and why it makes certain predictions.

Introspection Methods often vary per data type. With histology images, heatmaps are often used; in radiology, attention scores are often used; with molecular data, integrated gradient methods (compute attribution values indicating how changes in specific inputs affect the outputs) are often used. When taking a more holistic approach, attribution plots can be implemented to determine the contribution of each modality toward the overall model prediction. Introspective methods are typically class activation methods (CAMs) which allow one to determine the importance of the model inputs (e.g., pixels) by computing how changes in the inputs affect the model outputs for each prediction class.

Benefits of Multimodal Data Interconnection. Malignant changes can often be seen across different scales; for example, a genetic mutation can affect cell behavior, which in turn reshapes tissue morphology or the tumor microenvironment visible in histology images.

• Morphological associates can serve as cost-efficient biomarker substitutes to support screening in low-middle-income settings.
• Associates can reveal new therapeutic targets.
• Identified relationships between non-invasive (e.g., radiological) and invasive (e.g., histological) modalities will enable outcome predictions from non-invasive procedures.
• Associates between data acquired (e.g., Electronic Medical Records) prior to more in-depth procedures will enable quicker and easier identification of potential predictive risk factors.

Conclusions. Looking forward, many obstacles exist for the development of artificial intelligence with multimodal data integration for cancer care, such as data fairness and bias implications, limited interpretability, and more. On the other hand, AI has the potential to transform and improve oncology significantly. Many AI avenues possess the potential for great impact; examples include the prediction of patient outcomes (like prognosis, survival, and recurrence), efficient diagnosis, optimal treatment capabilities, and the discovery and exploration of novel biomarkers and biological pathways. Overall, this field is extremely exciting, and I can’t wait to share more. And there’s great news for machine learning enthusiasts like me! You can use various multimodal data integration models through GitHub repositories such as this one from Harvard Medical School.

Perhaps in the future, AI will be able to autonomously manage the entire healthcare process…

I’ve listed some other resources below that may be of interest.

• Original paper for multimodal data integration in AI
• Mahmood Lab Github repository
• Data fusion paper

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI