Exploring identification of social determinants of health (SDOH) data in NHLBI BioData Catalyst® (BDC) using embedding-based methods

Abstract

In alignment with the Make America Healthy Again initiative to promote the policies to improve public health, this initiative focuses on the identification of survey questions and answers in datasets hosted within the NHLBI BioData Catalyst® (BDC) ecosystem for the purpose of easier search in the ecosystem’s cohort-building tool, BDC Powered by PIC-SURE (BDC-PIC-SURE). Leveraging data standards developed by the Gravity Project, we systematically evaluated and ranked 113 high-value datasets within BDC based on the variables represented in the Gravity Project domains. The four datasets with the highest representation of variables in the Gravity Project domains were then manually annotated using Simple Knowledge Organization System (SKOS) relations to match survey questions and answers with the Gravity Project elements. We used this manually annotated data set as a “gold standard” to test a proof-of-concept annotation tool that uses embedding-based approaches to match survey-based data with the Gravity Project value set. Performance varied by domain, with employment status being the best (F1-Score = 1.0) and financial insecurity being the worst (F1-Score = 0.42). Some domains, such as financial insecurity, material hardship, and medical cost burden had significant overlap
and were challenging for human annotators to differentiate. Future work includes further refinement of this workflow by comparing the performance of different embedding algorithms, examining performance on categorical variables versus continuous variables, determining binning for semantic similarity scores (high, medium, low), and exploring the possibility of other vocabularies or annotating data with multiple domains.