Semester of Graduation
Summer 2025
Degree Type
Dissertation/Thesis
Degree Name
Masters in Computer Science
Department
Computer Science
Committee Chair/First Advisor
Md Abdullah Al Hafiz Khan
Abstract
Developing effective and efficient AI systems to solve complex tasks requires human
expert involvement, such as judging AI decisions, identifying critical contexts, etc.
Expert cognitive factors significantly affect the building and decision-making of this
AI model. This thesis presents a novel Simulated Human-AI Teaming (HAT) frame-
work for enhancing machine learning model development, focusing on misinforma-
tion classification. Traditional AI systems often overlook human cognitive factors like
trust, cognitive load, and decision difficulty, leading to inefficiencies and low trust. To
address this, the study leverages large language models (LLMs) to simulate human
cognitive states, generating synthetic datasets enriched with behavioral metadata.
The framework integrates these factors into model training and decision-making,
employing adaptive weighting and K-means clustering to prioritize high-reliability
inputs.
Experimentalresultsdemonstratesignificantimprovements: theGatedHFJoint-
Model achieved 86.23% accuracy and 86.87% F1 score, outperforming baseline models
by 9.07% and 22.06%, respectively. The system activated human intervention for only
39.47% of cases, optimizing resource use. Analysis revealed that high cognitive load
(>0.7) reduced accuracy by 15-20%, while low trust (< 0.4) increased error rates by
14%. Clustering identified three reliability tiers, with 28.6% of samples requiring hu-
man review due to low reliability (Cluster 0), 43% benefiting from hybrid processing
(Cluster 1), and 28.4% suitable for full automation (Cluster 2).
The framework’s adaptive gating mechanism improved accuracy by 8.67% over
majority voting (0.762 → 0.828) and reduced false positives by 39%. Human-AI
collaboration corrected 1,203 cases where AI missed contextual cues, while 65.7% of
decisions achieved partial agreement. This work advances trustworthy AI systems
by bridging human intuition and algorithmic precision, offering a scalable, ethical
alternative to traditional human-in-the-loop approaches.