AI hallucination refers to the phenomenon where a large language model (LLM) generates output that is stated with apparent confidence but is factually incorrect, unsupported by any source, or internally inconsistent. The term is borrowed from psychology but describes a distinct mechanism: LLMs are trained to produce fluent, coherent text by predicting likely token sequences, and that objective does not guarantee factual accuracy. In a learning context, hallucination is a serious risk because a learner who asks an AI tutor a question about a medical procedure, a legal rule, or a technical specification expects the answer to be correct — a plausible but wrong answer can cause real harm. Retrieval-augmented generation (RAG) is the primary technical mitigation: grounding the model in actual course content substantially reduces the rate of invented facts. However, RAG does not eliminate hallucination because the model can still misinterpret or contradict the retrieved passage. Human review of AI-tutor response policies, fallback to authoritative sources, and confidence-gating (refusing to answer when the model is uncertain) are complementary safeguards. For assessment content specifically, every AI-generated question must pass human review before reaching learners. Transparency matters too: citing the course passage the answer is based on lets the learner verify independently, turning the potential liability into a learning moment.
The handbook · 7 volumes
Fora Soft Video Engineering Handbook
AI hallucination is the risk that a tutor states a wrong answer with full confidence, which erodes learner trust fast. We wrote seven volumes that cover the whole video stack; for AI hallucination start with Volume 7, which covers agents, memory and tools, serving and guardrails in production, with Volume 6 on retrieval and RAG over video. Written by Nikolay Sapunov, CEO at Fora Soft.
Worried about AI hallucination in your course tutor or assistant?
We will test your AI assistant against your course content and give you a grounding and review setup that cuts hallucination, with measured results. 250+ video products shipped since 2005.
