Recent advances in machine learning for personalized medicine have created a need to determine when observational healthcare data can reliably inform treatment policies. This thesis examines methods for evaluating whether treatment variation in medical datasets is sufficient for developing dependable clinical policies. Through three complementary approaches, we investigate methods to detect and measure meaningful action diversity in healthcare data. First, we analyze the MIMIC sepsis dataset using transformer-based dynamics models. Our findings reveal that including action information provides minimal improvement in outcome predictions across the entire dataset. This suggests limited meaningful treatment diversity when analyzed in aggregate. Second, in our controlled simulation experiments with a one-dimensional GridWorld environment, we demonstrate that comparing prediction performance between models with and without action inputs effectively identifies regions where treatments meaningfully impactoutcomes. Finally, we present a novel interactive visualization tool that employs t-SNE dimensionality reduction and intuitive diversity metrics to help researchers explore action diversity across patient state spaces. This tool helps identify subgroups where treatment policies can be reliably learned.

Our findings demonstrate that dynamics model comparisons can effectively identify regions where treatment policies can be reliably learned, enabling more targeted and trustworthy deployment of machine learning in healthcare. This framework provides researchers with practical tools to evaluate data sufficiency before deploying treatment recommendation systems, potentially improving both the reliability of AI assistance in clinical decision-making and, ultimately, patient outcomes.

49 pages

Thesis Committee:
Adam Perer (Chair)
Zachory Erickson

Srinivasan Seshan, Head, Computer Science Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu