Principal Trait Analysis: Towards Deriving "Skills" in Human-AI Collaboration

Researchers propose Principal Trait Analysis (PTA), a Principal Component Analysis-inspired algorithm that automatically derives common interaction traits from large language model (LLM) conversation corpora to identify effective prompting patterns in human-AI collaboration. In evaluations on two human-AI collaborative coding datasets—an educational setting with students using an AI tutor and a professional setting with developers using an AI coding agent—PTA-derived traits significantly explained collaborator behavior and helped predict task outcomes, though whether these traits qualify as skills remains inconclusive due to limited generalizability and temporal stability.

arXiv:2608.11460v1 Announce Type: new Abstract: Large Language Model-powered agents are increasingly used in the workplace via human-artificial intelligence AI collaboration. In this new era of work, it is important to understand the kinds of prompting traits that contribute to task success. Moreover, we need to uncover key skills required for modern professionals and inform educators on how to foster these skills among students. Existing guidelines for human-AI collaboration are built from either top-down theory or context-specific observations of human-AI interactions. However, since LLM capabilities are rapidly improving, theory may not be able to explain emerging interaction patterns, and empirical guidelines may become obsolete quickly. In this work, we explore an automated, data-driven approach to uncover patterns, which we term traits, of effective human-AI interaction that are aligned with task outcomes. We propose Principal Trait Analysis, a Principal Component Analysis-inspired algorithm for deriving common traits from patterns in LLM conversations. Our algorithm uses LLM-based processing stages to analyze corpora of human-AI collaborative session traces, deriving common traits across the dataset and scoring each human collaborator's usage style by each trait. The approach also allows domain expertise to be injected during trait discovery and selects the most distinguishing traits to be those that exhibit the highest variance across collaborators. We evaluate PTA on two human-AI collaborative coding datasets, an educational setting students working with an AI tutor and a professional setting developers working with an AI coding agent . We find that PTA-derived traits are significant in explaining collaborator behavior across both settings and can help predict task outcomes. However, whether traits qualify as skills remains to be seen, due to inconclusive results on generalizability and how user traits change over time.