2,239 Comments: What Primary Care Clinicians Said About Practicing With an AI Virtual Patient

Read the learner comments ↓

Anyone who has run a CME program knows how evaluation data usually comes back. A minority of learners respond at all. Most of those click through the rating scales, and the open-ended satisfaction box is empty or holds a word or two. In my years in CME, getting above a ~40% response rate on average of any kind on an activity was rare. Getting comments that were substantive enough to act on was even harder to come by.

That's a real problem when you're trying to evaluate a new educational activity. Quantitative satisfaction scores tell you whether learners were pleased. They don't tell you what worked, what got in the way, or whether people were paying attention. And when the format itself is new and novel, like a spoken conversation with an AI virtual patient, those are exactly the questions you need answered.

So I looked closely at learner feedback from one AI virtual patient activity. I wanted to know two things. Did learners like it? And did the feedback show signs of real engagement, the kind of specific, usable comments that are hard to get from traditional CME?

Why Good Feedback Is Hard to Get

What the research shows

In a study of end-of-course evaluations at a large public university, about 40% of forms had any written comment [1]. A medical school in Lebanon compared voluntary and required clerkship evaluations [2]. When evaluations were voluntary, 56% of students completed them. When grades were held until students finished, completion rose to 100%. But the comment box stayed optional, and in both groups only about 1 in 4 evaluations had a written comment.

Live courses can do better

At one Mayo Clinic internal medicine CME course, participants completed 57% of the presentation evaluations available to them [3]. Even there, the researchers' goal was getting more useful written feedback to presenters.

The Activity and the Learners

A spoken conversation with an AI patient

The activity was an online CME program on opioid use disorder (OUD). In addition to reviewing a set of slides about OUD, learners held a spoken conversation with an AI virtual patient named Jeremy, a man taking opioids for back pain. They had to screen him, work through the diagnostic criteria, use motivational interviewing, and talk with him about starting buprenorphine. Jeremy answered in real time. Five simulation tasks on screen guided the learners. When the conversation ended, each learner received an instant feedback report on their performance.

Who took it

The 2,619 learners who completed the activity were mostly frontline clinicians. Nearly half were nurse practitioners (46%), 29% were physicians (MD or DO), 13% were physician assistants, and about 12% were other professions, including a small number of pharmacists. Family medicine (29%) and internal medicine (13%) were the largest recorded specialties, and about half listed some other specialty. Close to half worked in private or outpatient practice (47%), with the rest mostly in community hospitals (23%) and academic medical centers (10%). Experience varied widely: 11% had been in practice less than a year, 38% for 1 to 10 years, 18% for 11 to 20 years, and 26% for more than 20.

One required question

The evaluation asked one open-ended question:

"What are your overall impressions of this virtual patient simulation? Please share any feedback on its effectiveness as a learning tool and suggest any improvements."

Learners had to type something in that box to move forward. That detail matters for everything that follows, and we'll come back to what it does and doesn't mean.

What Learners Wrote

Real feedback, not filler

Of the 2,619 learners who completed the activity, 2,239 (85.5%) wrote a comment that could be read and coded as feedback. The other 380 didn't. Only 23 left the box blank. The other 357 typed a period, "na," "none," or something close to it.

That last group tells you what a required field normally produces. When people don't want to write, they type a dot and move on. Here, the large majority wrote real feedback instead.

Length and detail

The comments also went past one-word reactions. Out of all 2,619 completers, about half (49%) wrote six words or more. One in three (33%) wrote eleven words or more. About 1 in 7 (15%) wrote more than twenty. Put another way, the share of learners here who wrote eleven words or more was close to the share who wrote anything at all in the university and medical school studies above.

Plenty of comments were short. Among the 2,239 interpretable comments, the median was seven words (five words across all 2,619 responses, counting blanks and filler), and many were simple, like "very helpful" or "great way to practice." But a large group wrote the kind of detail course designers rarely get: what confused them, what they'd change, and what they'd missed in the conversation.

How learners felt

Of the 2,239 learners who left an interpretable comment, 1,607 (71.8%) were positive, 332 (14.8%) were neutral, and 300 (13.4%) were negative. Neutral includes "it was okay" answers, suggestions with no opinion included, and the many comments that had praise and a complaint. If you use all 2,619 completers as the denominator instead, counting the people who typed filler as not positive, 61% left a positive comment. That's the more conservative number, and it's still a clear majority.

How the comments were coded

Getting those numbers right took two passes. The first pass used AI (ChatGPT Astra, high reasoning) to sort every comment. I then reviewed all 2,619 responses by hand and found that some clearly negative comments, like "Did not think this was effective," had been marked positive. Seventy-eight comments marked positive were negative, mixed, or suggestions only. Just four errors ran the other way. Correcting them moved the positive share from 75% to about 72%. Even after a careful human read pulled the number down, most learners who wrote real feedback were clearly positive about the experience.

What Learners Told Us

The numbers show learners liked the activity. The comments show why, and they're specific enough to act on. You'll find many more learner quotes, sorted by sentiment, at the end of this post.

Saying it out loud made them think

A common theme was the effort of producing the words. "I enjoyed having to think about what to say to a patient and saying it out loud," one learner wrote. "Valuable to learn what concepts I have difficult[y] articulating before I get to a visit with a real patient." Another was more direct: "It requires active participation. Better than rote answering multiple choice questions for credit."

The feedback showed gaps they didn't know they had

Many learners only saw their gaps after the conversation was scored. One wrote, "[It] showed me I had not covered it as well as I thought I did and that I did not do any real planning with Jeremy." Another noticed it in real time: "[I] could tell that Jeremy was frustrated at the beginning of the conversation, so I realized I was not addressing his concerns appropriately. There was definitely a change in demeanor[] when I did get to address his concerns[,] and I like how that mimics real life."

What got in the way

The critical comments were just as useful. Many were technical. Microphones dropped, some learners lost progress and had to restart, and several asked for a typing option. "It was overall positive, but it was frustrating to have to go all the way back to the beginning each time I couldn't get the mic to work."

Others were about readiness, not technology. Not every clinician who takes an opioid course prescribes buprenorphine, and some said they needed more background before the conversation. One learner wanted "more information on buprenorphine included prior to the interaction." A smaller group found the encounter artificial, said it felt like a checklist, or said they'd prefer watching a demonstration or attending a live session.

Turning complaints into fixes

Those two kinds of complaints need different fixes. This activity ran on Xuron V3. Our latest version, V4, which launched in fall 2026, was built to address the technical issues learners raised here. The readiness gaps are a course design question, not a platform one, and the answer is better preparation: a brief orientation before the first conversation and key reference material on screen during the encounter. Sorted that way, the comments became a short, concrete development list.

Do the Ratings Back It Up?

A second measure

Comments can be persuasive and still not tell the whole story, so it helps to check them against a number. The evaluation also asked every learner a rating question: "To what extent do you believe this virtual patient simulation enhances your understanding of this topic compared to other teaching modalities?" All 2,619 completers answered on a five-point scale:

• 5, Extremely valuable: 442 (16.9%)
• 4, Very valuable: 963 (36.8%)
• 3, Moderately valuable: 877 (33.5%)
• 2, Slightly valuable: 272 (10.4%)
• 1, Not valuable at all: 65 (2.5%)

The numbers match the comments

More than half of learners (53.6%) rated the simulation very or extremely valuable compared with other teaching methods, and 87.1% rated it at least moderately valuable. Only 12.9% chose one of the two lowest ratings, and the average was 3.55 out of 5.

That lines up closely with what learners wrote. With all 2,619 completers as the denominator, 61% wrote a positive comment and 54% gave one of the top two ratings. The middle of the scale fits too. A third of learners chose "moderately valuable," which matches the many comments from people who liked the practice but hit a dropped microphone or felt underprepared. The written comments and the 1-to-5 ratings point in the same direction.

Reading the Results Honestly

The required box

The required field is the first thing a careful reader will ask about, so it's worth addressing directly. A required box guarantees a response. It doesn't guarantee a thoughtful one. In the Lebanese study, required evaluations included a planted nonsense item, "The clerkship helped me in my application to the bank," and about a third of the students checked agreed with it [2]. That's what compliance without engagement looks like.

So the response rate on its own isn't the finding. The finding is what learners did with a box they had to fill in. Most could have typed a period. Most wrote real feedback instead, and a third wrote eleven words or more. That's a meaningful engagement signal. It isn't a controlled comparison. I didn't run a traditional version of the same activity with the same required question, so I can't say how much longer or more specific these comments were than a lecture-based activity would have produced. That comparison is worth running.

Novelty

More than 100 learners mentioned that this was their first conversation with an AI patient. Some of the enthusiasm is likely first-time interest, and it may fade with repeated use.

Reaction, not outcomes

In Moore's outcomes framework, learner satisfaction is Level 2, above participation and below learning, competence, performance, and patient outcomes [4]. These comments tell us learners valued the practice and found the feedback useful. They don't tell us that anyone's skills improved or that patients benefited. The same is true of the rating question. It asks learners to compare the simulation with other teaching methods, but it captures their perception, not a measured comparison.

The broader evidence is encouraging but not settled. A 2019 systematic review of 51 trials found that virtual patients matched traditional education for knowledge outcomes and outperformed it for skills [5]. Most of those trials enrolled students, results for communication skills were mixed, and the review predates conversational AI patients like the one in this activity.

Conclusion

Primary care clinicians liked this technology, and they said so in their own words. Of the 2,239 who wrote real feedback, more than 7 in 10 were positive, and more than half of all completers rated the simulation very or extremely valuable compared with other teaching methods. They liked having to find the words out loud instead of picking an answer from a list. They valued feedback that showed them what they missed, often things they thought they had covered. Several described it as a low-pressure place to practice a hard conversation before having it with a real patient. Their complaints were specific too, which made them fixable. AI virtual patients are new enough that the field is still deciding how to evaluate them. These learners gave a clear first answer: given the chance to practice a real conversation, most came away positive, and several asked for more like it.

References

1. Zimmaro DM, Gaede CS, Heikes EJ, Shim MP, Lewis KG. A study of students' written course evaluation comments at a public university. University of Texas at Austin; 2006. https://testingservices.utexas.edu/sites/default/files/A-Study-of-Students-Written-Course-Evaluation-Comments-at-a-Public-University-2006.pdf
2. Aoun Bahous S, Salameh P, Salloum A, Salameh W, Park YS, Tekian A. Voluntary vs. compulsory student evaluation of clerkships: effect on validity and potential bias. BMC Med Educ. 2018;18:9. doi:10.1186/s12909-017-1116-8
3. Wittich CM, Mauck KF, Mandrekar JN, et al. Improving participant feedback to continuing medical education presenters in internal medicine: a mixed-methods study. J Gen Intern Med. 2012;27(4):425-431. https://pmc.ncbi.nlm.nih.gov/articles/PMC3304027/
4. Moore DE Jr, Green JS, Gallis HA. Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009;29(1):1-15. doi:10.1002/chp.20001
5. Kononowicz AA, Woodham LA, Edelbring S, et al. Virtual patient simulations in health professions education: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. 2019;21(7):e14676. doi:10.2196/14676