(043) Exploring the Potential of AI Chatbots as Psychoeducational Tools for Patients with Psychosis

Abstract: Background/Significance
The large language models (LLMs), like GPT-4 and LLaMA-2, are conversational Artificial Intelligence models that fundamentally change human-computer interaction. These models hold significant promise for improving mental healthcare, particularly in psychoeducation. While these technologies offer numerous advantages, they also present various considerations relevant to their accuracy and utility in mental health education (Rafael, 2024).
This study aims to explore the potential application of LLMs as a psychoeducational tool for patients diagnosed with psychosis. We evaluated a prominent LLM (GPT-4) via ChatGPT and its ability to provide accurate, clear, and clinically relevant information.
Method
We identified the twenty most common parental questions about childhood psychosis and gathered responses from ChatGPT. Two experts in psychosis assessed each response: a clinical psychologist and a psychiatrist at the Pediatric First Episode Psychosis Clinic at Nationwide Children`s Hospital. The evaluation framework assesses responses based on (1) Accuracy (measures the factual correctness), (2) Clarity (evaluates understandability), (3) Inclusivity (checks cultural sensitivity), (4) Completeness (determines if all question aspects are covered), (5) Clinical Utility (gauges practical relevance in clinical settings), and (6) Overall Score (reflects the evaluator's confidence in the response's effectiveness and appropriateness, summarizing the assessment across the mentioned criteria). We conducted descriptive and correlational analyses.
Results
Accuracy scored high across raters (Mean = 2.88, SD = 0.22), indicating a strong perception of responses directly and correctly addressing the question. Clarity was even more positively evaluated (Mean = 2.93, SD = 0.18), showing it was consistently seen as understandable. Inclusivity had a moderate average score (Mean = 2.30, SD = 0.41). Completeness was highly rated (Mean = 0.93, SD = 0.18), suggesting a consensus on the responses fully addressing all elements of each question. Clinical utility was rated very positively (Mean = 4.35, SD = 0.52), reflecting a favorable view of the responses' utility in a practical and clinical context. The overall rating was positive on average (Mean = 3.55, SD = 0.46), indicating a high level of confidence by the raters in each category being rated.
The Spearman's Rank correlation test revealed a strong positive relationship between Clarity and Completeness (Spearman's rho=0.61, p=0.004), and between Clarity and Clinical Utility (Spearman's rho=0.54, p=0.013). These findings highlight the importance of perceived clarity in enhancing both the completeness and perceived clinical relevance of the responses.
Discussion 
This study suggests that an LLM-based application shows potential as a psychoeducational tool for either patients or parents with psychosis-related questions. Such applications may offer a less intimidating and more engaging experience than traditional resources.
Conclusion
LLM-based psychoeducation may help parents and patients better support their children or themselves. This method, by offering accessible and personalized education with potential translation support, could reach a wider population. Further research is needed to explore the utility of this approach for other psychiatric conditions, especially in the consultation-liaison psychiatry setting, and to understand its limitations.

References
1- Rajaei, A. (2024). Teaching in the Age of AI/ChatGPT in Mental-Health-Related Fields. The Family Journal, 32(1), 6-10.

Presentation Eligibility: Not previously published or presented

Diversity, Equity, and Inclusion: Our poster presentation on chatbots explores their functionalities and assesses their inclusive language. A key finding we'll discuss is the identification of a moderate average score in current chatbot inclusivity.