Resources

Publications

Judd, Nick, Alexandre Vaz, Kevin Paeth, Layla Inés Davis, Milena Esherick, Jason Brand, Inês Amaro, and Tony Rousmaniere. 2026. Independent Clinical Evaluation of General-Purpose LLM Responses to Signals of Suicide Risk.. arXiv:2510.27521. Presented at the International Association for Artificial Intelligence Safety & Ethics Conference (IASEAI 2026), Paris, France. Proceedings forthcoming.

Rabin, Rafiqul, Sean McGregor, and Nick Judd. 2025. Malicious and Unintentional Disclosure Risks in Large Language Models for Code Generation.. In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C), 61–68. IEEE. DOI: 10.1109/SANER-C66551.2025.00016.
Cassani, Laura, Michael Davinroy, Tatiana Toumbeva, Peter Bautista, Lauren Fortier, James Cook, Ashley Hart, and Svtilana Volkova. 2025. Human-AI Collaboration for Synthetic Media Detection in Training and Operations.. In Interservice/Industry Training, Simulation and Education Conference (I/ITSEC) Proceedings, Orlando, Florida: IITSEC Proceedings. (February 18, 2026).
Paeth, Kevin, and Sean McGregor. 2025. AI Risk, Safety, and Incident Reporting.. In Handbook of Human-Centered Artificial Intelligence, Springer, Singapore, 1–39. doi:10.1007/978-981-97-8440-0_89-1.
Schiff, Daniel S. 2025. Strategies for Harmonizing Fragmented AI Ethics Frameworks, Standards, and Regulations.. In Handbook of Human-Centered Artificial Intelligence, Springer, Singapore, 1–45. doi:10.1007/978-981-97-8440-0_82-1.
Barnett, Julia, Kimon Kieslich, Jasmine Sinchai, and Nicholas Diakopoulos. 2025. Scenarios in Computing Research: A Systematic Review of the Use of Scenario Methods for Exploring the Future of Computing Technologies in Society.. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 8(1): 316–29. doi:10.1609/aies.v8i1.36551.
Allaham, Mowafak, Kimon Kieslich, and Nicholas Diakopoulos. 2025. Informing AI Risk Assessment with News Media: Analyzing National and Political Variation in the Coverage of AI Risks.. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 8(1): 90–103. doi:10.1609/aies.v8i1.36533.
Santos-Rodriguez, Raul, Kacper Sokol, Julia E. Vogt, and Sven Wellmann. 2025. From Research to Certification with Data-Driven Medical Decision Support Systems (Dagstuhl Seminar 25052). eds. Raul Santos-Rodriguez, Kacper Sokol, Julia E. Vogt, and Sven Wellmann. Dagstuhl Reports 15(1): 201–20. doi:10.4230/DagRep.15.1.201.
Kieslich, Kimon, Nicholas Diakopoulos, and Natali Helberger. 2025. Anticipating Impacts: Using Large-Scale Scenario-Writing to Explore Diverse Implications of Generative AI in the News Environment.. AI and Ethics 5(5): 4555–77. doi:10.1007/s43681-024-00497-4.
Einarsson, Alexander, Simen Oestmo, Lester Wollman, Duncan Purves, and Ryan Jenkins. 2025. A Debiasing Technique for Place-Based Algorithmic Patrol Management.. Criminal Justice Ethics 44(3): 330–50. doi:10.1080/0731129X.2025.2583681.
Barnett, Julia, Kimon Kieslich, Natali Helberger, and Nicholas Diakopoulos. 2025. Envisioning Stakeholder-Action Pairs to Mitigate Negative Impacts of AI: A Participatory Approach to Inform Policy Making.. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’25, New York, NY, USA: Association for Computing Machinery, 1424–49. doi:10.1145/3715275.3732096.
Gunawan, Johanna, Sarah Elizabeth Gillespie, David Choffnes, Woodrow Hartzog, and Christo Wilson. 2025. Promises, Promises: Understanding Claims Made in Social Robot Consumer Experiences.. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA: Association for Computing Machinery, 1–22. doi:10.1145/3706598.3713471.
Hasan, Jahid, and Romila Pradhan. 2025. Explanations for Machine Learning Pipelines under Data Drift.. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics, HILDA ’25, New York, NY, USA: Association for Computing Machinery, 1–7. doi:10.1145/3736733.3736744.
Jenkins, Ryan. 2025. Threads and Needles: A Value-Sensitive Design Approach to Online Toxicity.. Science and Engineering Ethics 31(3): 12. doi:10.1007/s11948-025-00533-4.
Jenkins, Ryan, John P. Sullins, Obinna Kalu, Aisha Kamath, and Krittana Phumjam. 2025. Recent Insights in Responsible AI Development and Deployment in National Defense: A Review of Literature, 2022–2024.. Journal of Military Ethics 24(1): 63–85. doi:10.1080/15027570.2025.2483058.
Jung, Ji-Youn, Devansh Saxena, Minjung Park, Jini Kim, Jodi Forlizzi, Ken Holstein, and John Zimmerman. 2025. Making the Right Thing: Bridging HCI and Responsible AI in Early-Stage AI Concept Selection.. In Proceedings of the 2025 ACM Designing Interactive Systems Conference, DIS ’25, New York, NY, USA: Association for Computing Machinery, 2992–3012. doi:10.1145/3715336.3735745.
Kirill, Trapeznikov, Paul Cummer, Pranay Pherwani, Jai Aslam, Michael Davinroy, Peter Bautista, Laura Cassani, and Matthew Stamm. 2025. SAFE: Synthetic Audio Forensics Evaluation Challenge.. In Proceedings of the 2025 ACM Workshop on Information Hiding and Multimedia Security, IH&MMSEC ’25, New York, NY, USA: Association for Computing Machinery, 174–80. doi:10.1145/3733102.3736707.
Kuo, Tzu-Sheng, Quan Ze Chen, Amy X. Zhang, Jane Hsieh, Haiyi Zhu, and Kenneth Holstein. 2025. PolicyCraft: Supporting Collaborative and Participatory Policy Design through Case-Grounded Deliberation.. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA: Association for Computing Machinery, 1–24. doi:10.1145/3706598.3713865.
McGregor, Sean, Allyson Ettinger, Nick Judd, Paul Albee, Liwei Jiang, Kavel Rao, William H. Smith, et al. 2025. To Err Is AI: A Case Study Informing LLM Flaw Reporting Practices.. Proceedings of the AAAI Conference on Artificial Intelligence 39(28): 28938–45. doi:10.1609/aaai.v39i28.35162.
Paeth, Kevin, Daniel Atherton, Nikiforos Pittaras, Heather Frase, and Sean McGregor. 2025. Lessons for Editors of AI Incidents from the AI Incident Database.. Proceedings of the AAAI Conference on Artificial Intelligence 39(28): 28946–53. doi:10.1609/aaai.v39i28.35163.
Saxena, Devansh, Ji-Youn Jung, Jodi Forlizzi, Kenneth Holstein, and John Zimmerman. 2025. AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development.. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA: Association for Computing Machinery, 1–23. doi:10.1145/3706598.3714098.
Surve, Tanmay, and Romila Pradhan. 2025. Explaining Fairness Violations Using Machine Unlearning.. In Proceedings of the 28th International Conference on Extending Database Technology (EDBT 2025), Barcelona, Spain, March: OpenProceedings.org. doi:10.48786/EDBT.2025.50.

Kieslich, K., Helberger, N., & Diakopoulos, N. (2024, June). My Future with My Chatbot: A Scenario-Driven, User-Centric Approach to Anticipating AI Impacts. In The 2024 ACM Conference on Fairness, Accountability, and Transparency (pp. 2071-2085).
Herasymuk, D., Arif Khan, F., & Stoyanovich, J. (2024, June). Responsible Model Selection with Virny and VirnyView. In Companion of the 2024 International Conference on Management of Data (pp. 488-491).
Kawakami, A., Coston, A., Zhu, H., Heidari, H., & Holstein, K. (2024, May). The Situate AI Guidebook: Co-Designing a Toolkit to Support Multi-Stakeholder, Early-stage Deliberations Around Public Sector AI Proposals. In Proceedings of the CHI Conference on Human Factors in Computing Systems (pp. 1-22).
Zhang, H. (2024, May). Searching for the Non-Consequential: Dialectical Activities in HCI and the Limits of Computers. In Proceedings of the CHI Conference on Human Factors in Computing Systems (pp. 1-13).
Kieslich, K., Diakopoulos, N., & Helberger, N. (2024, May). Anticipating impacts: using large-scale scenario-writing to explore diverse implications of generative AI in the news environment. AI and Ethics, 1-23.
McGregor, Sean. "Open digital safety." Computer 57.4 (2024, April): 99-103.
Chadda, A., McGregor, S., Hostetler, J., & Brennen, A. (2024, March). AI Evaluation Authorities: A Case Study Mapping Model Audits to Persistent Standards. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 21, pp. 23035-23040).
Guha, S., Khan, F. A., Stoyanovich, J., & Schelter, S. (2024, February). Automated data cleaning can hurt fairness in machine learning-based decision making. IEEE Transactions on Knowledge and Data Engineering.

Kawakami, A., Guerdan, L., Cheng, Y., Glazko, K., Lee, M., Carter, S., ... & Holstein, K. (2023, November). Training towards critical use: Learning to situate ai predictions relative to human knowledge. In Proceedings of The ACM Collective Intelligence Conference (pp. 63-78).
Rastogi, C., Leqi, L., Holstein, K., & Heidari, H. (2023, November). A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementarity. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (Vol. 11, No. 1, pp. 127-139).
Guerdan, L., Coston, A., Wu, S., & Holstein, K. (2023, October). Policy Comparison Under Unmeasured Confounding. In NeurIPS 2023 Workshop on Regulatable ML.
Loke, L. Y., Barsoum, D. R., Murphey, T. D., & Argall, B. D. (2023, September). Characterizing eye gaze for assistive device control. In 2023 International Conference on Rehabilitation Robotics (ICORR) (pp. 1-6). IEEE.
McGregor, S. (2023, August). A Scaled Multiyear Responsible Artificial Intelligence Impact Assessment. Computer, 56(8), 20-27.
Hammond, K., & Leake, D. (2023, July). Large language models need symbolic AI. In Proceedings of the 17th International Workshop on Neural-Symbolic Learning and Reasoning, La Certosa di Pontignano, Siena, Italy (Vol. 3432, pp. 204-209).
Guerdan, L., Coston, A., Holstein, K., & Wu, Z. S. (2023, June). Counterfactual prediction under outcome measurement error. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (pp. 1584-1598).
Bell, A., Bynum, L., Drushchak, N., Zakharchenko, T., Rosenblatt, L., & Stoyanovich, J. (2023, June). The possibility of fairness: Revisiting the impossibility theorem in practice. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (pp. 400-422).
Guerdan, L., Coston, A., Wu, Z. S., & Holstein, K. (2023, June). Ground (less) truth: A causal framework for proxy labels in human-algorithm decision-making. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (pp. 688-704).
Holstein, K., De-Arteaga, M., Tumati, L., & Cheng, Y. (2023, April). Toward supporting perceptual complementarity in human-AI collaboration via reflection on unobservables. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1), 1-20.

Ackerman, R., Icobelli, F., & Balyan, R. (2022, December). Intelligent Tutoring Systems in Healthcare for Low Literacy Population: A Concise Review. In 2022 International Conference on Computational Science and Computational Intelligence (CSCI) (pp. 1827-1829). IEEE.
Xu, S., Mi, L., & Gilpin, L. H. (2022, November). A Framework for Generating Dangerous Scenes for Testing Robustness. In Progress and Challenges in Building Trustworthy Embodied AI.
Rhea, A. K., Markey, K., D'Arinzo, L., Schellmann, H., Sloane, M., Squires, P., ... & Stoyanovich, J. (2022, September). An external stability audit framework to test the validity of personality prediction in AI hiring. Data Mining and Knowledge Discovery, 36(6), 2153-2193.
Jenkins, R., Hammond, K., Spurlock, S., & Gilpin, L. (2022, March). Separating facts and evaluation: motivation, account, and learnings from a novel approach to evaluating the human impacts of machine learning. AI & SOCIETY, 38(4), 1415-1428.

Tech Reports

Independent Clinical Evaluation of General-Purpose LLM Responses to Signals of Suicide Risk We present a clinician-informed, empirically validated method for evaluating how LLMs respond to signals of suicide risk in multi-turn conversations, showing that an open-source model (OLMo-2-32b) often deviates from best practices by becoming less likely to sustain supportive dialogue as risk increases, potentially discouraging help-seeking.

2023 Impact Report | UL Research Institutes Our annual impact report addresses how UL Research Institutes is working to address evolving public safety risks through groundbreaking safety science research.

Blog

The Image Edit Detection and Localization Challenge

06 Apr 2026 ⋅ #Image Forensics#AI-Edited Media Detection

Human vs. Machine Deep Fake Image Detection Competition at DEF CON

11 Sep 2025 ⋅ #ai#semafor#deepfake

DSRI Sponsors SAFE Audio Forensics Challenge to Identify Synthetic Audio

20 Jun 2025 ⋅ #audio#ai

Testing DeepSeek-R1

12 Jun 2025 ⋅ #deepseek

Investigating Risks and Safeguarding Communities

28 Jan 2025 ⋅ #presentation#cybercrime

Tabletop Red-Teaming for AI Safety

20 Dec 2024 ⋅ #red-teaming

A privacy-respecting way to test how well systems keep secrets

23 Oct 2024 ⋅ #presentation#cybercrime

Ignore All Previous Instructions and Build a Lighthouse

26 Aug 2024 ⋅ #meme#llm#personhood

The Chances of Bad Advice

24 Aug 2024 ⋅ #llm#ai

The Missing Section of LLM Model Cards

09 Aug 2024 ⋅ #llm#ai

Walkthrough for GRT2 Participants

08 Aug 2024 ⋅ #llm#ai#grt2

DSRI and Ai2 Making an Impact at DEF CON 2024

02 Aug 2024 ⋅ #llm#ai#grt2

Submit your AI Incident Research to IAAI

30 Jul 2024 ⋅ #llm#ai#grt2

Bridging Safety Research and Safety Application

26 Jul 2024 ⋅ #llm#ai

Learning from the Past

05 Mar 2024 ⋅ #ai

Newsletters

Volume 5

Volume 4

Volume 3

Volume 2

Volume 1

Judd, Nick, Alexandre Vaz, Kevin Paeth, Layla Inés Davis, Milena Esherick, Jason Brand, Inês Amaro, and Tony Rousmaniere. 2026. Independent Clinical Evaluation of General-Purpose LLM Responses to Signals of Suicide Risk.. arXiv:2510.27521. Presented at the International Association for Artificial Intelligence Safety & Ethics Conference (IASEAI 2026), Paris, France. Proceedings forthcoming.

Rabin, Rafiqul, Sean McGregor, and Nick Judd. 2025. Malicious and Unintentional Disclosure Risks in Large Language Models for Code Generation.. In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C), 61–68. IEEE. DOI: 10.1109/SANER-C66551.2025.00016.
Cassani, Laura, Michael Davinroy, Tatiana Toumbeva, Peter Bautista, Lauren Fortier, James Cook, Ashley Hart, and Svtilana Volkova. 2025. Human-AI Collaboration for Synthetic Media Detection in Training and Operations.. In Interservice/Industry Training, Simulation and Education Conference (I/ITSEC) Proceedings, Orlando, Florida: IITSEC Proceedings. (February 18, 2026).
Paeth, Kevin, and Sean McGregor. 2025. AI Risk, Safety, and Incident Reporting.. In Handbook of Human-Centered Artificial Intelligence, Springer, Singapore, 1–39. doi:10.1007/978-981-97-8440-0_89-1.
Schiff, Daniel S. 2025. Strategies for Harmonizing Fragmented AI Ethics Frameworks, Standards, and Regulations.. In Handbook of Human-Centered Artificial Intelligence, Springer, Singapore, 1–45. doi:10.1007/978-981-97-8440-0_82-1.
Barnett, Julia, Kimon Kieslich, Jasmine Sinchai, and Nicholas Diakopoulos. 2025. Scenarios in Computing Research: A Systematic Review of the Use of Scenario Methods for Exploring the Future of Computing Technologies in Society.. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 8(1): 316–29. doi:10.1609/aies.v8i1.36551.
Allaham, Mowafak, Kimon Kieslich, and Nicholas Diakopoulos. 2025. Informing AI Risk Assessment with News Media: Analyzing National and Political Variation in the Coverage of AI Risks.. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 8(1): 90–103. doi:10.1609/aies.v8i1.36533.
Santos-Rodriguez, Raul, Kacper Sokol, Julia E. Vogt, and Sven Wellmann. 2025. From Research to Certification with Data-Driven Medical Decision Support Systems (Dagstuhl Seminar 25052). eds. Raul Santos-Rodriguez, Kacper Sokol, Julia E. Vogt, and Sven Wellmann. Dagstuhl Reports 15(1): 201–20. doi:10.4230/DagRep.15.1.201.
Kieslich, Kimon, Nicholas Diakopoulos, and Natali Helberger. 2025. Anticipating Impacts: Using Large-Scale Scenario-Writing to Explore Diverse Implications of Generative AI in the News Environment.. AI and Ethics 5(5): 4555–77. doi:10.1007/s43681-024-00497-4.
Einarsson, Alexander, Simen Oestmo, Lester Wollman, Duncan Purves, and Ryan Jenkins. 2025. A Debiasing Technique for Place-Based Algorithmic Patrol Management.. Criminal Justice Ethics 44(3): 330–50. doi:10.1080/0731129X.2025.2583681.
Barnett, Julia, Kimon Kieslich, Natali Helberger, and Nicholas Diakopoulos. 2025. Envisioning Stakeholder-Action Pairs to Mitigate Negative Impacts of AI: A Participatory Approach to Inform Policy Making.. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’25, New York, NY, USA: Association for Computing Machinery, 1424–49. doi:10.1145/3715275.3732096.
Gunawan, Johanna, Sarah Elizabeth Gillespie, David Choffnes, Woodrow Hartzog, and Christo Wilson. 2025. Promises, Promises: Understanding Claims Made in Social Robot Consumer Experiences.. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA: Association for Computing Machinery, 1–22. doi:10.1145/3706598.3713471.
Hasan, Jahid, and Romila Pradhan. 2025. Explanations for Machine Learning Pipelines under Data Drift.. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics, HILDA ’25, New York, NY, USA: Association for Computing Machinery, 1–7. doi:10.1145/3736733.3736744.
Jenkins, Ryan. 2025. Threads and Needles: A Value-Sensitive Design Approach to Online Toxicity.. Science and Engineering Ethics 31(3): 12. doi:10.1007/s11948-025-00533-4.
Jenkins, Ryan, John P. Sullins, Obinna Kalu, Aisha Kamath, and Krittana Phumjam. 2025. Recent Insights in Responsible AI Development and Deployment in National Defense: A Review of Literature, 2022–2024.. Journal of Military Ethics 24(1): 63–85. doi:10.1080/15027570.2025.2483058.
Jung, Ji-Youn, Devansh Saxena, Minjung Park, Jini Kim, Jodi Forlizzi, Ken Holstein, and John Zimmerman. 2025. Making the Right Thing: Bridging HCI and Responsible AI in Early-Stage AI Concept Selection.. In Proceedings of the 2025 ACM Designing Interactive Systems Conference, DIS ’25, New York, NY, USA: Association for Computing Machinery, 2992–3012. doi:10.1145/3715336.3735745.
Kirill, Trapeznikov, Paul Cummer, Pranay Pherwani, Jai Aslam, Michael Davinroy, Peter Bautista, Laura Cassani, and Matthew Stamm. 2025. SAFE: Synthetic Audio Forensics Evaluation Challenge.. In Proceedings of the 2025 ACM Workshop on Information Hiding and Multimedia Security, IH&MMSEC ’25, New York, NY, USA: Association for Computing Machinery, 174–80. doi:10.1145/3733102.3736707.
Kuo, Tzu-Sheng, Quan Ze Chen, Amy X. Zhang, Jane Hsieh, Haiyi Zhu, and Kenneth Holstein. 2025. PolicyCraft: Supporting Collaborative and Participatory Policy Design through Case-Grounded Deliberation.. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA: Association for Computing Machinery, 1–24. doi:10.1145/3706598.3713865.
McGregor, Sean, Allyson Ettinger, Nick Judd, Paul Albee, Liwei Jiang, Kavel Rao, William H. Smith, et al. 2025. To Err Is AI: A Case Study Informing LLM Flaw Reporting Practices.. Proceedings of the AAAI Conference on Artificial Intelligence 39(28): 28938–45. doi:10.1609/aaai.v39i28.35162.
Paeth, Kevin, Daniel Atherton, Nikiforos Pittaras, Heather Frase, and Sean McGregor. 2025. Lessons for Editors of AI Incidents from the AI Incident Database.. Proceedings of the AAAI Conference on Artificial Intelligence 39(28): 28946–53. doi:10.1609/aaai.v39i28.35163.
Saxena, Devansh, Ji-Youn Jung, Jodi Forlizzi, Kenneth Holstein, and John Zimmerman. 2025. AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development.. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA: Association for Computing Machinery, 1–23. doi:10.1145/3706598.3714098.
Surve, Tanmay, and Romila Pradhan. 2025. Explaining Fairness Violations Using Machine Unlearning.. In Proceedings of the 28th International Conference on Extending Database Technology (EDBT 2025), Barcelona, Spain, March: OpenProceedings.org. doi:10.48786/EDBT.2025.50.

Kieslich, K., Helberger, N., & Diakopoulos, N. (2024, June). My Future with My Chatbot: A Scenario-Driven, User-Centric Approach to Anticipating AI Impacts. In The 2024 ACM Conference on Fairness, Accountability, and Transparency (pp. 2071-2085).
Herasymuk, D., Arif Khan, F., & Stoyanovich, J. (2024, June). Responsible Model Selection with Virny and VirnyView. In Companion of the 2024 International Conference on Management of Data (pp. 488-491).
Kawakami, A., Coston, A., Zhu, H., Heidari, H., & Holstein, K. (2024, May). The Situate AI Guidebook: Co-Designing a Toolkit to Support Multi-Stakeholder, Early-stage Deliberations Around Public Sector AI Proposals. In Proceedings of the CHI Conference on Human Factors in Computing Systems (pp. 1-22).
Zhang, H. (2024, May). Searching for the Non-Consequential: Dialectical Activities in HCI and the Limits of Computers. In Proceedings of the CHI Conference on Human Factors in Computing Systems (pp. 1-13).
Kieslich, K., Diakopoulos, N., & Helberger, N. (2024, May). Anticipating impacts: using large-scale scenario-writing to explore diverse implications of generative AI in the news environment. AI and Ethics, 1-23.
McGregor, Sean. "Open digital safety." Computer 57.4 (2024, April): 99-103.
Chadda, A., McGregor, S., Hostetler, J., & Brennen, A. (2024, March). AI Evaluation Authorities: A Case Study Mapping Model Audits to Persistent Standards. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 21, pp. 23035-23040).
Guha, S., Khan, F. A., Stoyanovich, J., & Schelter, S. (2024, February). Automated data cleaning can hurt fairness in machine learning-based decision making. IEEE Transactions on Knowledge and Data Engineering.

Kawakami, A., Guerdan, L., Cheng, Y., Glazko, K., Lee, M., Carter, S., ... & Holstein, K. (2023, November). Training towards critical use: Learning to situate ai predictions relative to human knowledge. In Proceedings of The ACM Collective Intelligence Conference (pp. 63-78).
Rastogi, C., Leqi, L., Holstein, K., & Heidari, H. (2023, November). A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementarity. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (Vol. 11, No. 1, pp. 127-139).
Guerdan, L., Coston, A., Wu, S., & Holstein, K. (2023, October). Policy Comparison Under Unmeasured Confounding. In NeurIPS 2023 Workshop on Regulatable ML.
Loke, L. Y., Barsoum, D. R., Murphey, T. D., & Argall, B. D. (2023, September). Characterizing eye gaze for assistive device control. In 2023 International Conference on Rehabilitation Robotics (ICORR) (pp. 1-6). IEEE.
McGregor, S. (2023, August). A Scaled Multiyear Responsible Artificial Intelligence Impact Assessment. Computer, 56(8), 20-27.
Hammond, K., & Leake, D. (2023, July). Large language models need symbolic AI. In Proceedings of the 17th International Workshop on Neural-Symbolic Learning and Reasoning, La Certosa di Pontignano, Siena, Italy (Vol. 3432, pp. 204-209).
Guerdan, L., Coston, A., Holstein, K., & Wu, Z. S. (2023, June). Counterfactual prediction under outcome measurement error. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (pp. 1584-1598).
Bell, A., Bynum, L., Drushchak, N., Zakharchenko, T., Rosenblatt, L., & Stoyanovich, J. (2023, June). The possibility of fairness: Revisiting the impossibility theorem in practice. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (pp. 400-422).
Guerdan, L., Coston, A., Wu, Z. S., & Holstein, K. (2023, June). Ground (less) truth: A causal framework for proxy labels in human-algorithm decision-making. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (pp. 688-704).
Holstein, K., De-Arteaga, M., Tumati, L., & Cheng, Y. (2023, April). Toward supporting perceptual complementarity in human-AI collaboration via reflection on unobservables. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1), 1-20.

Ackerman, R., Icobelli, F., & Balyan, R. (2022, December). Intelligent Tutoring Systems in Healthcare for Low Literacy Population: A Concise Review. In 2022 International Conference on Computational Science and Computational Intelligence (CSCI) (pp. 1827-1829). IEEE.
Xu, S., Mi, L., & Gilpin, L. H. (2022, November). A Framework for Generating Dangerous Scenes for Testing Robustness. In Progress and Challenges in Building Trustworthy Embodied AI.
Rhea, A. K., Markey, K., D'Arinzo, L., Schellmann, H., Sloane, M., Squires, P., ... & Stoyanovich, J. (2022, September). An external stability audit framework to test the validity of personality prediction in AI hiring. Data Mining and Knowledge Discovery, 36(6), 2153-2193.
Jenkins, R., Hammond, K., Spurlock, S., & Gilpin, L. (2022, March). Separating facts and evaluation: motivation, account, and learnings from a novel approach to evaluating the human impacts of machine learning. AI & SOCIETY, 38(4), 1415-1428.

Independent Clinical Evaluation of General-Purpose LLM Responses to Signals of Suicide Risk We present a clinician-informed, empirically validated method for evaluating how LLMs respond to signals of suicide risk in multi-turn conversations, showing that an open-source model (OLMo-2-32b) often deviates from best practices by becoming less likely to sustain supportive dialogue as risk increases, potentially discouraging help-seeking.

2023 Impact Report | UL Research Institutes Our annual impact report addresses how UL Research Institutes is working to address evolving public safety risks through groundbreaking safety science research.