To properly measure large language models (LLMs), we must go beyond observational analysis. I trace the theme of spiking (inserting certain data into training at known rates) in my own work, and motivate spiking from both statistical and legal perspectives. From the statistical perspective, I review the importance of randomization in making exact inferences about model properties. Many inferences are only possible if developers spike their training data and insert additional randomness. From the legal perspective, I review how law analyzes both the behavior and structure of technology. Spiking can be seen as a structural design feature for LLMs and support a legal defense under structural scrutiny.
The final work of this thesis demonstrates a practical application of spiking, where spiking enables principled statistical correction for test set contamination. Methods for correction are validated on Hubble, and I give an overview of the Hubble model suite, a suite of open-source LLMs spiked with book passages, biographies, and test sets, intended to enable a wide range of causal study on LLM memorization. Experiments on Hubble show that spiking a small number of test examples or natural texts makes it possible to calibrate memorization detectors and statistically correct inflated test scores. Finally, I will conclude on future directions for spiking and the implications for technical governance.