A Comparative Validation Study of Video-conferencing and ChatBot-based Tele-assessment Administration for Evaluating Indirect Pragmatic Expression in School-aged Children

Abstract

Indirect pragmatic expression involves interpreting the underlying meaning of utterances by considering situational context, world knowledge, and perspective-taking within the domain of meta-pragmatic language (Kim, 2018). School-aged children, who are exposed to diverse communication environments and engage in social relationships within the classroom, need to acquire indirect pragmatic expression. These skills are crucial for academic success and effective social engagement, making them an essential aspect of children's development (Olswang et al., 2007). The worldwide outbreak of COVID-19 has accelerated the adoption of distance interactions, leading to an increased demand for tele-assessments (Weidner & Lowman, 2020). However, a survey study aimed at investigating the current status of tele-practice services among Korean-speaking speech-language pathologists revealed the insufficiency of tele-assessment services in meeting the needs of diverse populations. This is evidenced by the low utilization rates (5.3%) observed, specifically in the domain of "assessment and screening." (Kim et al., 2020) Tele-assessments via video-conferencing programs such as Zoom and Google Meet have consistently been validated as reliable interaction modalities (Castilla-Earls et al., 2022; Dekhtyar et al., 2020; McGill et al., 2021; Theodoros et al., 2008). These studies reported no significant differences between tele-assessment conducted via video-conferencing and face-to-face interactions. More recent attention has been focused on exploring innovative models of care for tele-assessment, such as chatbots and artificial intelligence (Eerdenbrugh et al., 2022). Chatbots, software programs designed to simulate human-like conversations, have shown potential as assessment and intervention tools in school and clinical settings (Bradford et al., 2020; Ireland et al., 2016; Kim et al., 2019; Hong et al., 2021; Huang et al., 2022; Shin, 2019; Shin et al., 2023). The aim of the current study was to investigate the validity of video-conferencing and ChatBot-based online interaction modalities compared to traditional face-to-face assessment for evaluating indirect pragmatic expression of school-aged children. The study involved 76 typically developing children in grades 3 to 4 from elementary schools. Participants were divided into three groups based on the three interaction modality, ensuring no significant differences in age and vocabulary abilities as measured by the Receptive & Expressive Vocabulary Test (REVT; Kim et al., 2009). The study compared the performance on indirect pragmatic expression tasks across different interaction modalities and question types. Additionally, a correlation analysis examined the relationship between the performance on indirect pragmatic expression tasks within each interaction modality and the standardized vocabulary assessment results obtained from the REVT. The findings in the study indicated that school-aged children demonstrated comparable performance in indirect pragmatic expression tasks regardless of interaction modalities used, including face-to-face, video-conferencing, and ChatBot. Significant correlations were observed between the performance in indirect pragmatic expression tasks during face-to-face and ChatBot interactions, whereas no significant correlation was found between face-to-face and video-conferencing interactions. Significant correlations were identified between performance in indirect pragmatic expression tasks across all three interaction modalities and REVT. These results have methodological implications for utilizing ChatBot-based tele-assessment as a reliable alternative in indirect pragmatic expression tasks, particularly when face-to-face interactions are not possible.;본 연구는 학령기 아동의 간접 화용 표현 능력을 평가하기 위한 화상회의 기반과 챗봇 기반의 원격 상호작용 방식의 타당도를 조사하는 것을 목표로 한다. 연구 대상은 초등학교 3-4학년에 재학 중인 76명의 일반 아동으로 검사의 상호작용 방식(대면, 화상회의, 챗봇)에 따라 세 집단으로 나누었으며, 집단 간 연령 및 표준화된 수용 및 표현 어휘 능력에 유의한 차이가 없도록 하였다. 본 연구에서는 상호작용 방식과 질문 유형에 따른 간접 화용 표현 과제 수행력을 비교하였다. 다음으로, 세 가지 상호작용 방식에 따른 간접 화용 표현 과제 수행력과 표준화된 수용 및 표현 어휘 평가 결과 간의 관계를 탐색하였다. 본 연구의 결과를 요약하면 다음과 같다. 첫째, 학령기 아동들은 대면, 화상회의, 챗봇 기반의 상호작용 방식에 관계없이 간접 화용 표현 과제에서 통계적으로 유사한 수행력를 보였다. 둘째, 대면 및 챗봇 기반 상호작용에서 간접 화용 표현 과제 수행력 간에는 상관관계가 통계적으로 유의하게 나타났고, 대면과 화상회의 기반 상호작용 간에는 유의하지 않았다. 세 가지 상호작용 방식에서의 간접 화용 표현 과제 수행력과 표준화된 수용 및 표현 어휘 평가 결과 간에 통계적으로 유의한 정적 상관관계가 확인되었다. 본 연구의 결과는 간접 화용 표현 능력의 평가에서 대면 검사에 대해 신뢰할 수 있는 대안으로서 원격 상호작용 방식을 선택할 수 있다는 원격 평가의 방법론적 함의를 가지며, 학령기 아동의 간접 화용 표현 능력을 효율적으로 평가하기 위한 원격 평가 도구로서의 챗봇의 임상적 활용 가치를 확인한 데에 의의가 있다.