In my work, I see lots of examples of people accepting what comes from AI as true; we accept whatever it gives us with little effort to confirm it. We should be concerned about this, except for the fact that humans have a very long history of accepting information as correct from sources without any concern of whether it is correct.
Galen of Pergamon was a philosopher and physician said to have lived from 129 until 216 and his work influenced physiology for more than 1000 years. While he was a skilled observer of bodies, many of his ideas were false, but they remained accepted for centuries, even as contrary ideas—even those supported by observation—were rejected.
We need not consider those with wide influences like Galen to see the influences of unquestioned and incorrect information. I recall a teacher colleague who was telling his students that humans have two sets of chromosomes (we are diploid), but that if we had more than two (if we were polyploid) we would be nine feet tall. Polypoid animals are not uncommon, but their size is not much different from similar animals that are diploid, so my colleague’s conclusion seems unlikely. His students, however, we all convinced it was true. This is a specific example of the myths we are taught in school, but not everything we are taught in school is a myth.
Over my career, I have often taught about conspiracies and theories. I first got the idea while watching a James Randii episode of Nova way back in the 1980s. It has always interested me that young people seem to be quite skeptical of these theories when we study them, but they have lots of stories of adults in their lives who are deeply committed to whatever pseudoscience we are studying.
Artificial intelligence is the latest authoritative source humans have encountered. I maintain our interactions with it are not unlike how we have encountered any other authoritative source: we accept too much of what they say and we disregard what we observe and we disregard reason.