HOME /

Symptom Checkers & ChatGPT

To help patients both self-triage (i.e., identify their symptoms and what type of medical attention they need) and subsequently receive the right care quickly, there has been a proliferation of online symptom checkers. Symptom checkers are websites or smart phone apps where users respond to questions and then, based on computer algorithms, are provided with potential diagnoses and a recommendation on whether and where to seek care.

Symptom checkers have several potential benefits. They can encourage patients with life-threatening problems like a stroke or heart attack to seek emergency care, and they can suggest that patients with a non-urgent problem that does not require a medical visit to stay at home. However, some key concerns about symptom checkers and telemedicine include the potential for misdiagnosis or incorrect triage, particularly for life-threatening problems, and an increase in unnecessary medical visits, particularly if someone is told to go to the emergency room when it is not medically necessary.

In our work, we have examined the diagnostic accuracy of symptom checkers and how their accuracy compares to that of physicians:

  • Kanagasingam Y, Xiao D, Vignarajan J, Preetham A, Tay-Kearney ML, Mehrotra A. Evaluation of Artificial Intelligence-Based Grading of Diabetic Retinopathy in Primary Care. JAMA Network Open. 2018 Sep 7;1(5):e182665.
  • Semigran HL, Levine DM, Nundy S, Mehrotra A. Comparison of Physician and Computer Diagnostic Accuracy. JAMA Internal Medicine. 2016 Dec 1;176(12):1860-1861. doi: 10.1001/jamainternmed.2016.6001.
  • Semigran HL, Linder JA, Gidengil C, Mehrotra A. Evaluation of symptom checkers for self diagnosis and triage: audit study.BMJ. 2015 Jul 8;351:h3480.

Recently, we have used some of the same clinical vignettes we used in our work on symptom checkers to evaluate the diagnostic performance of ChatGPT. ChatGPT is an artifiicial intelligence chatbot that was introduced with great fanfare in November of 2022. We published an op-ed in STAT news comparing the performance of ChatGPT vs. physicians. The piece highlights the potential of this tool as well as some issues we need to watch for. Linked here is an Appendix that provides the details of our evaluation.