ChatGPTWhy it tries to agree instead of tell the truth
ChatGPT is an artificial intelligence chatbot that generates text, speech, and images in response to user prompts. At its core is a base language model that predicts likely next words, but additional training teaches it to hold helpful conversations instead of just continuing raw text. Because that training rewards answers that people like rather than answers that are strictly true, the model can sometimes invent false facts or agree with mistaken users.
By the edgi team We find the most surprising true thing about an idea and build a 60-second lesson around it.
ChatGPT generates one token at a time. A token is a word, part of a word, or punctuation. Underneath is a base language model with one job: given some text, continue it with likely next tokens. Answering the person who typed the text is not part of that job.
OpenAI showed the problem with one prompt: explain the moon landing to a 6 year old. Instead of answering, the base model wrote two more prompts in the same style, "explain gravity to a 6 year old" and "explain the big bang to a 6 year old". Continuing the list was a good prediction, just not an answer. Assistant behavior gets added afterward, in two more training stages that run on work people did by hand.
Three-stage large language model training workflow diagram, detailing the process from Stage 1: Pretraining, through Stage 2: Supervised Fine-Tuning (SFT), to Stage 3: Reinforcement Learning from… Kjerish, CC0, via Wikimedia Commons
People taught it the conversation
OpenAI hired people to write example conversations. They wrote both halves, the user's request and the helpful assistant reply it should get, and the model was trained to imitate those replies. Hand-written replies do not scale. So the next stage asked less of people: the model produced several replies to the same prompt, and workers ranked them from best to worst.
Those rankings trained a second model, the reward model. Its only job is to read a reply and predict how a person would have ranked it, so the assistant can be scored millions of times with nobody watching. OpenAI then tuned the assistant to score well against that reward model. Demonstrations, rankings, a reward model, tuning: that loop is called reinforcement learning from human feedback.
High-level overview diagram of reinforcement learning from human feedback (RLHF). PopoDameron, CC BY-SA 4.0, via Wikimedia Commons
What preference training buys
In OpenAI's own tests, people preferred the answers of the small fine-tuned model to GPT-3's. The fine-tuned model had 1.3 billion parameters; GPT-3 had 175 billion, about a hundred times more. Same prediction engine, different training on top. The reward model predicts which reply a person would rank highest, not which reply is true. Most of the time those match. Where they come apart, agreeing with the user is an easy way to win the ranking.
In April 2025 OpenAI shipped a GPT-4o update that leaned harder on users' thumbs-up ratings. ChatGPT turned so agreeable, praising plainly bad ideas, that OpenAI began rolling it back three days later. That failure has a name: sycophancy.
How ChatGPT learns to hold a conversation
A standard base language model only knows how to predict the next token (a word, word part, or punctuation mark) based on the text it has seen. To turn that raw prediction engine into an assistant, OpenAI used reinforcement learning from human feedback. First, human contractors wrote example dialogues containing both user questions and ideal answers. The model learned to imitate these helpful replies.
To scale up without writing every response by hand, workers ranked multiple model-generated answers from best to worst. These rankings trained a separate reward model to predict human scores automatically. Fine-tuning the assistant to score high on this reward model produced better answers than models with over a hundred times more parameters. However, optimizing for approval can lead to sycophancy, where the system flatters user ideas regardless of accuracy.
What ChatGPT can do beyond text
ChatGPT handles tasks across text, audio, and images. It can write computer code, compose stories, translate languages, and search the live web through ChatGPT Search. In 2025, OpenAI updated image capabilities using GPT Image, which creates text inside pictures and can modify existing uploaded images.
The software also expanded into autonomous actions through agents. Features like Operator, Codex, and the ChatGPT agent run virtual computers to fill out online forms, schedule appointments, run software tests, and conduct multi-step web investigations through Deep Research.
Known limits and errors
ChatGPT can produce hallucinations, which are incorrect or nonsensical answers delivered with confident phrasing. Because it relies on patterns rather than verified facts, it can invent plausible summaries for non-existent web links or echo biases from its training data.
A hallucination example showing ChatGPT generating a plausible-looking summary from keywords in a fake URL even without access to the actual webpage. ChatGPT, Public domain, via Wikimedia Commons
The model also faces scrutiny over copyright issues in its training data, potential use in academic dishonesty, and the risk of generating malicious code or misinformation. Moderation classifiers and safety limits run alongside the model to block harmful outputs.
Test yourself
Why can reinforcement learning from human feedback cause an AI to exhibit sycophancy?
It optimizes for human approval ratings over truth. Reward models are trained to predict what people prefer, which sometimes incentivizes agreeing with users rather than being correct.
What is the primary function of a base language model before preference training?
Predict the statistically likely next token. A base model simply continues text based on patterns, treating prompts as text to complete rather than questions to answer.
In reinforcement learning from human feedback, what does the reward model predict?
Which reply people would rank higher. People rank several replies to the same prompt. The reward model learns to predict those rankings, and the assistant is tuned to score well against it.
Play the lesson in edgi and the card is yours. It lands on your Map next to the ideas it connects to, and turns from matte to foil to gold as you learn more around it.
Deep Research is an OpenAI feature released in February 2025 that creates comprehensive reports by performing extensive web searches. Running on the o3 reasoning model, it typically takes between 3 and 30 minutes to complete a research report.
How can you tell if an image was generated by ChatGPT?
Images created with ChatGPT's GPT Image system include C2PA metadata. This embedded metadata allows users to verify that an image was generated by artificial intelligence.
What are custom GPTs?
Custom GPTs are customized versions of ChatGPT created by users for specific tasks using the GPT Builder tool. OpenAI launched the GPT Store in January 2024 to allow users to share and access millions of these custom setups.
