In this study, we evaluated the quality, readability, reliability, and guideline adherence of patient information provided by commonly used artificial intelligence (AI) chatbots regarding intravesical Bacillus Calmette–Guérin (BCG) therapy for bladder cancer. The results suggested that AI chatbot responses generally provided moderate-quality information, with considerable variability among different platforms. However, the texts required high reading-grade levels, and actionable recommendations were limited. In addition, adherence to European Association of Urology (EAU) guidelines was moderate across all chatbots. These findings suggest that AI chatbots may serve as supportive tools for patient education, but their outputs should be interpreted cautiously and should not replace physician counseling.
The use of AI-based chatbots in healthcare has increased rapidly in recent years. Large language models are capable of generating fast and accessible medical information, which may facilitate patient education and improve access to healthcare knowledge (1,2). Studies evaluating AI chatbot performance in medical examinations and clinical knowledge have demonstrated that these systems can produce generally accurate information; however, concerns remain regarding incomplete responses, lack of referencing, and variability in the quality of medical content (2,15). The reliability of AI-generated medical information has therefore become an important topic of investigation, particularly in specialties such as urology where treatment decisions may be complex.
Intravesical BCG immunotherapy remains the gold standard adjuvant treatment for intermediate- and high-risk non–muscle invasive bladder cancer (NMIBC) and plays a crucial role in reducing tumor recurrence and progression rates (5,16). The mechanism of BCG therapy involves stimulation of the local immune response through activation of both innate and adaptive immune pathways, resulting in anti-tumor activity within the bladder mucosa (17). Despite its established efficacy, intravesical BCG therapy is associated with local and systemic side effects such as dysuria, urinary frequency, hematuria, and fever, which may lead to treatment-related anxiety among patients (16,17). Adequate patient education is therefore essential in order to improve treatment adherence and reduce uncertainty related to potential complications.
Previous studies evaluating AI chatbot performance in urological conditions have demonstrated similar findings to those observed in the present study. Investigations assessing AI-generated information regarding urological cancers have shown that, although responses to general patient questions may be of relatively high quality, performance can be lower when addressing guideline-based clinical questions (10). Similarly, studies evaluating chatbot performance in benign urological conditions have demonstrated moderate DISCERN scores, high reading-grade levels, and low actionability scores (8,9). These findings are consistent with the results of the present study, in which DISCERN scores indicated moderate information quality, while PEMAT-P results demonstrated limited actionable guidance for patients.
One of the most important findings of the present study was the consistently high reading-grade level of chatbot-generated texts. Coleman–Liau Index scores indicated that all responses required a university-level education for comprehension. Previous studies have shown that patient education materials should ideally be written at a reading level corresponding to the 6th–8th grade in order to ensure accessibility for the general population (18). High reading-grade levels may reduce patient comprehension and negatively affect treatment adherence, particularly in complex therapies such as intravesical BCG immunotherapy.
Another important finding of this study was the limited actionability of chatbot responses. Although AI-generated texts generally provided descriptive information about BCG therapy, they often lacked clear practical guidance regarding treatment expectations, management of side effects, and indications for seeking medical attention. Previous studies have similarly reported that AI chatbot responses frequently provide general medical explanations but insufficient practical recommendations for patients (10,15). This limitation may reduce the clinical usefulness of AI-generated information in real-world patient decision-making.
Evaluation of guideline adherence demonstrated moderate compatibility with EAU recommendations across all chatbot responses. Although chatbot outputs generally included basic information regarding treatment purpose and administration, important clinical details such as patient selection criteria, maintenance therapy protocols, and contraindications were not consistently addressed. Previous studies have similarly demonstrated variability in chatbot performance and reduced quality when addressing guideline-based clinical questions (9,10). This finding highlights the importance of physician supervision when patients use AI-generated medical information.
Intravesical BCG therapy represents a relatively complex treatment requiring patient understanding of treatment schedule, expected benefits, and possible adverse effects. Insufficient or inaccurate patient information may lead to increased anxiety and reduced treatment adherence. AI chatbots may provide easily accessible preliminary information for patients prior to clinical consultation; however, variability in response quality suggests that these systems should be considered complementary rather than primary sources of medical information.
This study has several limitations. First, AI chatbot responses may change over time as models are continuously updated. Second, the evaluation was performed using a single standardized prompt, and different question formulations may produce different responses. Third, although responses were evaluated by five independent urologists, subjective scoring systems may introduce interobserver variability. Furthermore, although understandability and actionability were assessed using validated instruments, actual patient comprehension and real-world usability of the chatbot-generated information were not directly evaluated. In addition, the study was designed as a descriptive methodological evaluation and was not intended to determine statistically significant differences between chatbot platforms. Therefore, the findings should be interpreted as descriptive observations rather than evidence of superiority of any specific chatbot. Finally, AI chatbot responses may vary depending on the language of the prompt and the specific version of the chatbot used.
DISCUSSION
In this study, we evaluated the quality, readability, reliability, and guideline adherence of patient information provided by commonly used artificial intelligence (AI) chatbots regarding intravesical Bacillus Calmette–Guérin (BCG) therapy for bladder cancer. The results suggested that AI chatbot responses generally provided moderate-quality information, with considerable variability among different platforms. However, the texts required high reading-grade levels, and actionable recommendations were limited. In addition, adherence to European Association of Urology (EAU) guidelines was moderate across all chatbots. These findings suggest that AI chatbots may serve as supportive tools for patient education, but their outputs should be interpreted cautiously and should not replace physician counseling.
The use of AI-based chatbots in healthcare has increased rapidly in recent years. Large language models are capable of generating fast and accessible medical information, which may facilitate patient education and improve access to healthcare knowledge (1,2). Studies evaluating AI chatbot performance in medical examinations and clinical knowledge have demonstrated that these systems can produce generally accurate information; however, concerns remain regarding incomplete responses, lack of referencing, and variability in the quality of medical content (2,15). The reliability of AI-generated medical information has therefore become an important topic of investigation, particularly in specialties such as urology where treatment decisions may be complex.
Intravesical BCG immunotherapy remains the gold standard adjuvant treatment for intermediate- and high-risk non–muscle invasive bladder cancer (NMIBC) and plays a crucial role in reducing tumor recurrence and progression rates (5,16). The mechanism of BCG therapy involves stimulation of the local immune response through activation of both innate and adaptive immune pathways, resulting in anti-tumor activity within the bladder mucosa (17). Despite its established efficacy, intravesical BCG therapy is associated with local and systemic side effects such as dysuria, urinary frequency, hematuria, and fever, which may lead to treatment-related anxiety among patients (16,17). Adequate patient education is therefore essential in order to improve treatment adherence and reduce uncertainty related to potential complications.
Previous studies evaluating AI chatbot performance in urological conditions have demonstrated similar findings to those observed in the present study. Investigations assessing AI-generated information regarding urological cancers have shown that, although responses to general patient questions may be of relatively high quality, performance can be lower when addressing guideline-based clinical questions (10). Similarly, studies evaluating chatbot performance in benign urological conditions have demonstrated moderate DISCERN scores, high reading-grade levels, and low actionability scores (8,9). These findings are consistent with the results of the present study, in which DISCERN scores indicated moderate information quality, while PEMAT-P results demonstrated limited actionable guidance for patients.
One of the most important findings of the present study was the consistently high reading-grade level of chatbot-generated texts. Coleman–Liau Index scores indicated that all responses required a university-level education for comprehension. Previous studies have shown that patient education materials should ideally be written at a reading level corresponding to the 6th–8th grade in order to ensure accessibility for the general population (18). High reading-grade levels may reduce patient comprehension and negatively affect treatment adherence, particularly in complex therapies such as intravesical BCG immunotherapy.
Another important finding of this study was the limited actionability of chatbot responses. Although AI-generated texts generally provided descriptive information about BCG therapy, they often lacked clear practical guidance regarding treatment expectations, management of side effects, and indications for seeking medical attention. Previous studies have similarly reported that AI chatbot responses frequently provide general medical explanations but insufficient practical recommendations for patients (10,15). This limitation may reduce the clinical usefulness of AI-generated information in real-world patient decision-making.
Evaluation of guideline adherence demonstrated moderate compatibility with EAU recommendations across all chatbot responses. Although chatbot outputs generally included basic information regarding treatment purpose and administration, important clinical details such as patient selection criteria, maintenance therapy protocols, and contraindications were not consistently addressed. Previous studies have similarly demonstrated variability in chatbot performance and reduced quality when addressing guideline-based clinical questions (9,10). This finding highlights the importance of physician supervision when patients use AI-generated medical information.
Intravesical BCG therapy represents a relatively complex treatment requiring patient understanding of treatment schedule, expected benefits, and possible adverse effects. Insufficient or inaccurate patient information may lead to increased anxiety and reduced treatment adherence. AI chatbots may provide easily accessible preliminary information for patients prior to clinical consultation; however, variability in response quality suggests that these systems should be considered complementary rather than primary sources of medical information.
This study has several limitations. First, AI chatbot responses may change over time as models are continuously updated. Second, the evaluation was performed using a single standardized prompt, and different question formulations may produce different responses. Third, although responses were evaluated by five independent urologists, subjective scoring systems may introduce interobserver variability. Furthermore, although understandability and actionability were assessed using validated instruments, actual patient comprehension and real-world usability of the chatbot-generated information were not directly evaluated. In addition, the study was designed as a descriptive methodological evaluation and was not intended to determine statistically significant differences between chatbot platforms. Therefore, the findings should be interpreted as descriptive observations rather than evidence of superiority of any specific chatbot. Finally, AI chatbot responses may vary depending on the language of the prompt and the specific version of the chatbot used.