Semester of Graduation
Spring 2026
Degree Type
Dissertation/Thesis
Degree Name
Masters in Computer Science
Department
Computer Science
Committee Chair/First Advisor
Md Shazibul Islam Shamim
Abstract
Software packages are widely used throughout the software development process, and developers frequently release updates to improve stability and security. These updates may inadvertently introduce syntactic or semantic changes incompatible with dependent projects, resulting in breaking changes. This research aims to help developers and researchers assess LLM-based models' performance in detecting breaking changes at the earliest stage of the development lifecycle. We conducted an empirical study evaluating the effectiveness of LLM-based agents in detecting breaking changes by analyzing differences between consecutive code commits. We performed a large-scale qualitative analysis of 9,081 GitHub issues related to breaking changes across 303 Python package repositories and curated a dataset of 211 commit files from breaking commits, each linked to its corresponding GitHub issue. Our investigation shows that breaking changes introduced at the commit stage remain undetected for an average of over 5 weeks before release, and 53% are introduced during minor releases. To evaluate LLM-based detection, we developed PyCoReX, an AI agent that detects breaking changes from code commits by analyzing localized Git diffs, outperforming existing static analysis tools. We evaluated PyCoReX across 12 LLMs, comprising nine cloud-based models from OpenAI, Anthropic, and Google Gemini, and three locally deployed open-weights models from the Llama and Gemma families. Our results demonstrate that Chain-of-Thought prompting improves the agent's best F1-score from 0.82 to 0.85. However, a substantial performance gap remains between syntactic and semantic breaking change detection, with F1-scores reaching 86.8% and 60%, respectively.