상세 보기
인공지능 한국어 능력 평가 문항의 사전 난이도 평정을 위한 설계 난이도 지수 개발 - 평가 요소 그래프에 기반한 이론, 운영 모형 및 다중 자료 집합 탐색 -
- 박승희;
- 권태현;
- 정재은
초록
Korean-language proficiency evaluation datasets for artificial intelligence systems require item-difficulty distributions designed to reliably differentiate capabilities across models. At the early stage of item development, however, response data are unavailable, and difficulty judgments therefore rely largely on qualitative criteria and expert experience. This study proposes the Design Difficulty Index (DDI), a framework for rating observable item-design features, together with their supporting rationale, prior to item administration. DDI distinguishes and connects, across the full assessment lifecycle, target difficulty, the pre-administration design score (s), the post-administration Item Response Theory (IRT) difficulty parameter (b), and the final operational level (L). The eight DDI Core rating items are reasoning depth, level of information integration, context and state dependence, Korean form and lexical knowledge requirements, pragmatic and social judgment requirements, external knowledge dependence, answer-structure complexity, and constraint-coordination complexity. These items are integrated with an assessment-element graph and quality gates. Three exploratory datasets were examined: NIA-53, a 53-item sample developed on the basis of expert experience for a National Information Society Agency (NIA) project proposal; Survey-65-4, comprising four writing items from the 65-item National Survey of Korean Language Proficiency conducted by the National Institute of Korean Language; and CSAT-945, comprising 945 items constructed from 21 Korean-language test forms actually administered in the College Scholastic Ability Test (CSAT) and its mock examinations, with 45 items taken by examinees in each administration. The DDI values in the original exploratory analyses remained either provisional single-analyst ratings or rule-based automated candidate ratings without independent review. In a separate independent re-rating of all 53 NIA items by two human raters, itemwise quadratic-weighted Cohen’s kappa ranged from -.002 to .485, exact agreement from 7.5% to 66.0%, and within-one-point agreement from 60.4% to 100.0%. This pattern provides preliminary reproducibility evidence for some items while identifying P, E, and X as priorities for boundary-rule refinement. NIA-53 showed a positive rank correlation between nominal levels and provisional unweighted design scores (ρ=.657, p<.001), although L2 and L3 overlapped substantially. Because nominal-level information was not concealed and NIA-53 was used to inspect the initial anchors, this association is interpreted as exploratory consistency within development data rather than independent convergent-validity evidence. In Survey-65-4, the intended difficulty explicitly specified by the item developers and the preliminary IRT category did not coincide for any of the four items, whereas the preliminary and main-survey IRT categories remained consistent for all four items. This case illustrates the potential value of treating intended difficulty and empirical difficulty as distinct layers and motivates an intermediate design layer for comparing and explaining their relationship. In CSAT-945, two-point and three-point items had mean provisional unweighted design scores of 21.51 and 35.10, respectively, with a large ordinal separation (Cliff’s δ=.699). Because official point values jointly reflect curricular importance, expected response time, and expected difficulty, the result is interpreted as exploratory convergence between automated candidate structural ratings and official point values. DDI contributes a computable record of item structure, quality status, evidence, and revision direction, together with a validation pathway linking pre-administration ratings to post-administration parameters. Further validation requires independent re-rating by three or more raters in external domains, explanatory models based on raw response data, external validation samples, and prospective item-revision experiments.
키워드
- 제목
- 인공지능 한국어 능력 평가 문항의 사전 난이도 평정을 위한 설계 난이도 지수 개발 - 평가 요소 그래프에 기반한 이론, 운영 모형 및 다중 자료 집합 탐색 -
- 제목 (타언어)
- Developing a Design Difficulty Index for Pre-administration Difficulty Rating of Korean-Language Proficiency Items for Artificial Intelligence Systems: Theory, an Operational Model, and Multi-Dataset Exploration Based on an Assessment-Element Graph
- 저자
- 박승희; 권태현; 정재은
- 발행일
- 2026-09
- 유형
- Y
- 저널명
- 외국어로서의 한국어교육
- 권
- 83
- 페이지
- 199 ~ 245
- 언어
- KOR
- 출판사
- 연세대학교 언어연구교육원 한국어학당
- 발행국가
- 대한민국
- 분량
- 47 페이지
- ISSN
- E 2671-6240
P 1598-8201