Summary: A recent study found that ChatGPT correctly answered about 97% of questions drawn from the National Cancer Institute’s list of common cancer myths and misconceptions. In blinded testing, reviewers often could not tell whether responses came from ChatGPT or the NCI, though some chatbot answers used indirect or vague language that could be misinterpreted.
Source: Huntsman Cancer Institute
A new peer-reviewed study published in the Journal of the National Cancer Institute Cancer Spectrum examined how well ChatGPT handles common questions about cancer myths and misconceptions. As chatbots and other artificial intelligence tools become more widely used for health information, the research aimed to measure the accuracy and clarity of the answers these systems provide compared with authoritative sources.
In the first study of its kind, Skyler Johnson, MD, a physician-scientist at Huntsman Cancer Institute and an assistant professor of radiation oncology at the University of Utah, led the evaluation of ChatGPT’s responses to cancer-related misinformation. The team based their questions on the National Cancer Institute’s “Common Cancer Myths and Misconceptions” page and compared the chatbot’s answers directly with the official NCI responses.
Reviewers were blinded to the source of each response, meaning they did not know whether an answer came from ChatGPT or from the NCI. After independent expert review, the overall agreement for accuracy was essentially 97%—reported as 96.9% in the blinded analysis—while NCI answers were rated accurate at 100%. These results indicate that, on straightforward myth-busting questions, ChatGPT can provide largely accurate information consistent with national guidance.
While accuracy rates were high, the researchers emphasize caution. Some ChatGPT responses were phrased in indirect or vague terms, and reviewers noted occasional ambiguity in wording. That ambiguity could lead to misunderstanding by patients or caregivers and potentially influence decisions about care. For this reason, the study’s authors advise careful consideration before relying solely on chatbot outputs for cancer-related decisions.
“I recognize and understand how difficult it can feel for cancer patients and caregivers to access accurate information,” says Dr. Johnson. “These sources need to be studied so that we can help cancer patients navigate the murky waters that exist in the online information environment as they try to seek answers about their diagnoses.”
The team also notes this work builds on prior research showing that misinformation about cancer is common on social media, where false or misleading content can pose real risks to patients. Understanding how AI chatbots compare with trusted sources is an important step in assessing the role these tools might play in patient education and public health.
Planned next steps include studying how often patients and caregivers turn to chatbots for cancer information, identifying which questions they most frequently ask, and testing whether AI systems can accurately answer less common or more complex cancer questions. Researchers will also continue to evaluate language clarity and readability to reduce the risk of misinterpretation.
Funding: The study was supported by the National Institutes of Health/National Cancer Institute including P30 CA042014 and the Huntsman Cancer Foundation.
About this ChatGPT AI and cancer research news
Author: Avery Shrader ([email protected])
Source: Huntsman Cancer Institute
Contact: Avery Shrader – Huntsman Cancer Institute
Image: The image is in the public domain
Original Research: Open access. “Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information” by Skyler B Johnson et al., Cancer Spectrum
Abstract
Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information
Data on the quality of cancer information provided by chatbots and other artificial intelligence systems remain limited. This study directly compares ChatGPT responses to answers from the National Cancer Institute by using the questions on the NCI’s “Common Cancer Myths and Misconceptions” web page.
The NCI’s answers and ChatGPT outputs for each question were blinded and then evaluated by experts for factual accuracy. Ratings were recorded independently for each question and then compared between sources. The investigators also measured word count and Flesch–Kincaid readability grade level for each response.
Following expert review, NCI answers showed 100% agreement for accuracy, while ChatGPT outputs achieved 96.9% agreement for questions 1 through 13. There were minimal differences in response length and readability between NCI and ChatGPT answers. Overall, these findings suggest that ChatGPT can provide accurate information about common cancer myths and misconceptions, though careful attention to clarity and potential for misinterpretation is recommended.