生成式多智能体支持的开放式协作问题解决能力测评系统设计与验证OA

The assessment of collaborative problem-solving(CPS)competence has long faced a fundamental challenge:balanc-ing the authenticity of collaborative contexts with the independence of individual assessment.Traditional human-human collaborative assessments are susceptible to peer effects,whereas human-computer assessments based on predefined response options are limited in their ability to capture the dynamic interactions inherent in complex collaboration.To address this challenge,this study developed an open-ended CPS assessment system powered by generative multi-agent technology.Grounded in the PISA 2015 CPS framework and informed by situated cognition theory,group dynamics theory,and Evidence-Centered Design(ECD),the system follows an assessment logic of"context construction-behavior elicitation-evidence inference"and incorporates multiple role-based agents,a dialogue control mechanism,and an evidence analysis module.Building on this system,an automated procedure was further established to map dia-logue units onto CPS competency dimensions and evaluate learners' performance across different proficiency levels.The system was validated with Grade 10 students.The results indicate that the system can effectively differentiate learners with varying levels of CPS competence,while demonstrating a high degree of consistency with conventional assessment approaches.Moreover,open-ended natu-ral language interaction generates richer and more continuous process-based behavioral evidence,providing a viable approach for leveraging large language models to assess higher-order thinking skills.