--- title: The Curse of Dimensionality description: Curse of dimensionality, sounds daunting doesn't it? But what does this look like when it comes to AI and biopharma? image: https://info.aitiabio.com/hubfs/iStock-987365514.jpg --- - [Blog](https://blog.gnshealthcare.com/) - [Careers](https://www.gnshealthcare.com/careers/) - [Twitter](https://twitter.com/gnshealthcare) - [LinkedIn](https://www.linkedin.com/company/gns-healthcare/) - [YouTube](https://www.youtube.com/channel/UC8RbKPCoi9Y-8BG_rEZEOXg) [![GNS](https://info.aitiabio.com/hubfs/GNS_Healthcare_October2018%20Theme/Images/gns_logo.png)](https://www.gnshealthcare.com/)   - Solutions - [REFS Platform](https://www.gnshealthcare.com/refs-platform/) - [Subpopulations](https://www.gnshealthcare.com/subpopulations/) - [Value-based Drug Models](https://www.gnshealthcare.com/value-based-drug-models/) - [Intervention Optimization](https://www.gnshealthcare.com/intervention-optimization/) - [Customized Modeling](https://www.gnshealthcare.com/customized-modeling/) - Who We Serve - [Biopharma](https://www.gnshealthcare.com/biopharma/) - [Health Plans](https://www.gnshealthcare.com/health-plans/) - [Foundations](https://www.gnshealthcare.com/foundations/) - Results - [The Power of Causal Machine Learning](https://www.gnshealthcare.com/the-power-of-causal-machine-learning/) - [Journal Articles](https://www.gnshealthcare.com/journal-articles/) - [Posters](https://www.gnshealthcare.com/posters/) - [News & Events](https://www.gnshealthcare.com/news/) - [Blog](https://blog.gnshealthcare.com/) - [In the News](https://www.gnshealthcare.com/news/#news) - [Events](https://www.gnshealthcare.com/news/#events) - [Press Releases](https://www.gnshealthcare.com/news/#press-releases) - [GNS Precision Medicine Forum](https://www.gnshealthcare.com/gns-precision-medicine-forum/) - About Us - [Leadership](https://www.gnshealthcare.com/leadership/) - [Board of Directors](https://www.gnshealthcare.com/board-of-directors/) - [Strategic Advisory Board](https://www.gnshealthcare.com/strategic-advisory-board/) - [Our Culture](https://www.gnshealthcare.com/culture/) - [Careers](https://www.gnshealthcare.com/careers/) - [Contact](https://www.gnshealthcare.com/contact/) - [Request Info](https://info.gnshealthcare.com/request-info-website) - [Blog](https://blog.gnshealthcare.com/) - [Careers](https://www.gnshealthcare.com/careers/) ## GNS Healthcare Blog # GNS Healthcare Blog ## The Curse of Dimensionality By [Brian Harms, Ph.D., VP, Precision Medicine](https://info.aitiabio.com/author/brian-harms-ph-d-vp-precision-medicine) May 02, 2019 [Clinical Trials](https://info.aitiabio.com/topic/clinical-trials), [AI](https://info.aitiabio.com/topic/ai), [pharma](https://info.aitiabio.com/topic/pharma) [![Share on Facebook](https://static.hubspot.com/final/img/common/icons/social/facebook-24x24.png)](http://www.facebook.com/share.php?u=https%3A%2F%2Finfo.aitiabio.com%2Fcurse-of-dimensionality%3Futm_medium%3Dsocial%26utm_source%3Dfacebook) [![Share on LinkedIn](https://static.hubspot.com/final/img/common/icons/social/linkedin-24x24.png)](http://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Finfo.aitiabio.com%2Fcurse-of-dimensionality%3Futm_medium%3Dsocial%26utm_source%3Dlinkedin) [![Share on Twitter](https://static.hubspot.com/final/img/common/icons/social/twitter-24x24.png)](https://twitter.com/intent/tweet?original_referer=https%3A%2F%2Finfo.aitiabio.com%2Fcurse-of-dimensionality%3Futm_medium%3Dsocial%26utm_source%3Dtwitter&url=https%3A%2F%2Finfo.aitiabio.com%2Fcurse-of-dimensionality%3Futm_medium%3Dsocial%26utm_source%3Dtwitter&source=tweetbutton&text=The%20Curse%20of%20Dimensionality) [![Share on Email](https://static.hubspot.com/final/img/common/icons/social/email-24x24.png)](mailto:?subject=Check%20out%20https%3A%2F%2Finfo.aitiabio.com%2Fcurse-of-dimensionality%3Futm_medium%3Dsocial%26utm_source%3Demail%20&body=Check%20out%20https%3A%2F%2Finfo.aitiabio.com%2Fcurse-of-dimensionality%3Futm_medium%3Dsocial%26utm_source%3Demail) ![iStock-987365514](https://info.aitiabio.com/hubfs/iStock-987365514.jpg)  *This is the first of a series on the question of data availability, structure, value and challenges.* The “curse of dimensionality”. Sounds daunting, doesn’t it? As previously discussed in our post on [Biomarkers as the Guideposts for Drug Discovery and Development](https://info.aitiabio.com/biomarkers-guideposts-for-drug-discovery), this “curse” is a common problem in enabling precision medicine.[\[1\]](https://info.aitiabio.com/curse-of-dimensionality#_ftn1) The term refers to a phenomenon when the number of measured variables (such as gene expression levels) is much greater than the number of available samples or patients. In that situation, the possible space of observations is vastly greater than the available data – raising the risk that any patterns discovered in the data occur only by chance. Ironically, as the ease of generating large molecular datasets in biology research increases, it becomes much harder to glean reliable insights from the data – things like the identification of new drug targets, or factors that result in positive patient prognosis.   **Not All Data are Created Equal**  High-throughput molecular data, often termed ‘omics’ data as a catch-all, includes genomic, proteomic, transcriptomic and other types of data which hone in on a person’s biology. Omics comes from the Greek meaning “a totality of” – proteomics, for instance, is data about **all** the proteins available in the human body. Beginning with the onset of the genomics revolution, when the first non-human genome was sequenced in 1976 and expanding still further with the Human Genome Project, a fifteen year project which published its final chromosome reference sequence in 2006, the ability to delve deeper into biological data has put us on a track to being overwhelmed by the data, particularly as the cost of omics data generation continues to rapidly fall.[\[2\]](https://info.aitiabio.com/curse-of-dimensionality#_ftn2) However, more data doesn’t guarantee better research insights or patient outcomes. The challenge lies in the proper interpretation of the biology or pathology represented by the high dimensional data. While there are many diseases that can be traced back to a single genetic or physiologic abnormality (such as the *HTT* gene in Huntington’s Disease or BCR-ABL translocation in Chronic Myelogenous Leukemia), it is more common to see diseases that result from the complicated interaction of many genetic, physiologic, and environmental factors. This is reflected in the difficulty in developing successful therapeutic approaches for diseases like lupus, Alzheimer’s, or ALS. In these diseases, individual factors can have very limited effect. Only when multiple factors combine does a perfect storm arise, causing disease or deterioration. However, when two or more factors are believed to interact synergistically, parsing the massive number of possible omics combinations is extremely challenging. Testing all possible synergies requires both a massive amount of computing power and robust algorithms which can analyze the data without finding spurious relationships by chance.[\[3\]](https://info.aitiabio.com/curse-of-dimensionality#_ftn3) It’s the curse of dimensionality rearing its ugly head again.   **Sifting Through**   If finding the true factors driving disease is so difficult given the curse of dimensionality, what can we do?  One approach for working effectively with large scale data is creating a “network model”; that is to say, a graphical representation of the hierarchical relationships between the data. A network model shows how “downstream” variables can summarize the information contained in other, “upstream” data – which means that the downstream network variables are the ones that matter most. When sifting through large scale 'omics data, because network models are designed to show how different variables interact, they allow for easy interpretation of the key molecular or laboratory factors driving clinical disease. This makes graphical network models a powerful tool for addressing the curse of dimensionality and identifying reliable insights into disease.   ![networkforblog](https://info.aitiabio.com/hs-fs/hubfs/networkforblog.png?width=292&name=networkforblog.png)  *Causal networks create a transparent map pointing toward the relationship between data and outcomes. In this diagram, red represents the patient health outcome of interest. Data variables in the network map that are close to the outcome (“downstream” data) are more important for predicting the outcome than data variables that are further away (“upstream” data). Causal network algorithms are capable of taking a large set of ‘omics data and transforming it into a network map of relationships that identify the most important variables for predicting patient outcomes.*   A curse, or a blessing in disguise, the plethora of data creates unique challenges but also a promising path forward for leveraging network models and novel data to advance healthcare.   [\[1\]](https://info.aitiabio.com/curse-of-dimensionality#_ftnref1) https://www.nature.com/articles/s41746-019-0081-5 [\[2\]](https://info.aitiabio.com/curse-of-dimensionality#_ftnref2) [https://www.nature.com/articles/nature04727](https://www.nature.com/articles/nature04727) [\[3\]](https://info.aitiabio.com/curse-of-dimensionality#_ftnref3) https://stm.sciencemag.org/content/3/87/87ec90 ### Subscribe to the GNS Newsletter Search: Search ### Follow Us: ### Recent Posts: ### Categories: - [AI (29)](https://info.aitiabio.com/topic/ai) - [Precision Medicine (21)](https://info.aitiabio.com/topic/precision-medicine) - [Artificial Intelligence (20)](https://info.aitiabio.com/topic/artificial-intelligence) - [Machine Learning (20)](https://info.aitiabio.com/topic/machine-learning) - [Future (13)](https://info.aitiabio.com/topic/future) - [Big Data (12)](https://info.aitiabio.com/topic/big-data) - [Healthcare (11)](https://info.aitiabio.com/topic/healthcare) ### Twitter [Tweets by @gnshealthcare](https://twitter.com/gnshealthcare) **About GNS Healthcare** By leveraging the most powerful form of AI, called causal machine learning, we transform massive and diverse data streams to precisely match therapeutics, procedures, and care management interventions to individuals. We understand that in order to make precision medicine a reality, we need to discover and deliver insights that will cure disease and heal the healthcare system. [Privacy Policy](https://www.gnshealthcare.com/privacy-policy) ![HITRUST](https://info.aitiabio.com/hubfs/GNS_Healthcare_October2018%20Theme/Images/HiTrust-Certified.jpg) **Contact Us** 196 Broadway Cambridge, MA 02139 Email: [info@gnshealthcare.com](mailto:info@gnshealthcare.com) Phone: 617.374.2300 - [Twitter](https://twitter.com/gnshealthcare) - [LinkedIn](https://www.linkedin.com/company/gns-healthcare/) - [YouTube](https://www.youtube.com/channel/UC8RbKPCoi9Y-8BG_rEZEOXg) © GNS Healthcare 561 Windsor St. A200, Somerville, MA 02143 | 617-374-2300