Get the top HN stories in your inbox every day.
doomlaser
This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior?
I started looking and it turns out that OpenAI is keenly aware of the hallucination problem, and even wrote about how they're correcting for it in this blog post about InstructGPT: https://openai.com/blog/instruction-following/
To summarize: it seems OpenAI is addressing this by taking human-labeled output data from GPT and feeding this into new models, using a reward function to train the future models to behave with less hallucination. This way of working struck me as a mix of software engineering and crop breeding.
They discuss the trade-offs involved with doing this. The "hallucination" effect is probably one of the features that make GPT so creative.
wcoenen
I'm not sure "hallucination" is the right word.
I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up.
So perhaps the solution would be to include the prediction confidence in the output. E.g. gray out the parts of the text hat are low confidence predictions, like downvoted HN comments.
trilbyglens
Isn't the problem more _because_ it's a language model, and not a knowledge model? It's not based on facts, or even able to go out and find facts. If it's not in the training set it simply doesn't know.
It seems like this is only a single layer to something that should be larger. It should be able to tell if what it's saying is true, or to go out and find facts when it's missing them.
The fact that it's only a language model probably means that this is just out of scope.
version_five
> It seems like this is only a single layer to something that should be larger
Absolutely correct, and I believe anyone working on these models would agree and, other than as a fun demo, would never suggest that the raw model output gets used for any real purpose. A similar analogy would be self-driving cars. Somewhere "under the hood" there is an ML computer vision model, but it's not like the output layer is just hooked up to the gas and steering. There is all sorts of other logic to make sure the car behaves as intended and fails gracefully under ambiguity.
People see these language models and their flaws and somehow interpret it as a flawed overall product, when they are instead just seeing the underlying model. Admittedly, openAI hasn't helped much by building and promoting a chatbot the way they have.
Lots of cool potential for large language models, very little that comes from raw interaction
andrepd
That doesn't stop the companies churning out these models to pretend otherwise x)
rzzzt
If you would like another Latin word for it, take "confabulation" from neuroscience-land: https://en.wikipedia.org/wiki/Confabulation#Signs_and_sympto...
ComputerGuru
I had an elderly neighbor who unfortunately suffered from this. I spoke with her off-and-on over the first year or so, and she loved to talk. She would tell me about her daughter and grandkid, things that she saw that day, etc.
It was all very plausible but I always felt like there was something off about her. Then one day she told me a story about me, and things I’d said, done, and experienced and it was all absolutely made up, from the overarching plot down the finest details. It never happened, couldn’t have happened, and couldn’t have been even something that happened to someone else.
I tried to politely correct her at first, but she was so certain that she began worrying about me and why I couldn’t remember so I decided to just stand and nod to avoid stressing her out.
nnlsoccer
Came here to say the same thing. Medically confabulation is different than hallucination and far more similar to what is being described. Confabulation is seen with wernike-korsokoff syndrome which can be found in very long time alcohol use disorder. The patient makes up stories to fit the gaps in their memory without necessarily realizing that is what they are doing.
Whereas hallucinations are more like present sensory disturbances happening at that moment.
tyra3l
or simply use the Filling-in: https://en.wikipedia.org/wiki/Filling-in
nonameiguess
That still wouldn't help here. We don't want the prediction confidence that the sequence of words you produced might appear in a valid English-language sentence produced by humans. We want the prediction confidence that the sentence is factually accurate. These models aren't given that kind of data to train on and I'm not sure how they even could be. There are oodles and oodles of human-generated text out there, but little in the way of verification regarding how much of it is true, to say nothing of categories of language like imperative and artistic that don't have truth values at all.
layer8
> I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening.
It may give more insight, but it seems to me that hallucination is very similar: the brain completing some incomplete/random data to what it thinks is plausible and/or desirable.
macrolime
That's how sensory processing work in general, not just hallucinations.
ShamelessC
Hallucination is commonly used in ML parlance and gets the point across without needing to know what "stochastic" means.
dsr_
"Stochastic" means "random, not supported by facts, hypothetical" in every context in which it is used, across many fields.
The real problem is that anyone thought that they could pull factual material out of a giant language correlation network.
kgwgk
It’s also a misleading anthropomorphization that can get across the wrong message - in particular among those who don’t know what “stochastic” means but also among those who should know better.
kwhitefoot
If people in the ML community don't know what stochastic means then how can they communicate with each other? Precision in communication in such contentious areas seems to me to be of paramount importance, especially when speaking to people not in ones immediate circle.
andix
They are not forced to come up with new ideas. They can also write something like „I have no further information about that“. But in training this is probably discouraged, because they shouldn’t answer all questions like that.
asimpletune
I don't think it works that way. The models don't have a database of facts, so they never reach a point where they know that something they're saying is based on the real world. I think in other words, they literally operate by just predicting what comes next and sometimes that stuff is just made up.
skybrian
Yes, the training doesn't encourage this. It encourages guessing, because if it guesses the next word and it's right, the guessing is reinforced.
Whenever the model gets something right, it's the result of good guesses that were reinforced. It's all guesswork, it's just that some guesses are right.
canjobear
> And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up.
No, they could easily generate the end-of-sequence symbol, or the words “I don’t know.”
xiphias2
,,The word "hallucination" itself was introduced into the English language by the 17th-century physician Sir Thomas Browne in 1646 from the derivation of the Latin word alucinari meaning to wander in the mind. For Browne, hallucination means a sort of vision that is "depraved and receive[s] its objects erroneously".[8]''
I'm not sure if we know enough about hallucination to confirm that it's that much different from what GPT is doing.
zerocrates
Right, most uses of these model I've seen, you clearly want "hallucination" or something like it: when you ask it for a poem it wouldn't be so impressive if it just spat out The Raven.
doomlaser
That's one of the other things that strikes me about many of the killer applications that have thus far come out of the deep learning AI spring that's been blooming for the last 10 years or so: riffing on known styles in novel ways is very compelling.
One of the first big AI successes was the style transfer algorithm. Take an image and apply the higher level style elements from another image onto it. https://en.wikipedia.org/wiki/Neural_style_transfer - "Mona Lisa in the style of Starry Night", etc.
We saw the same thing emerge with DALL-E 2 earlier this year. Totally different architecture, but most of the things people like are the result of taking one known "style" and applying it to something novel, sometimes to the frustration of artists whose styles were gobbled up in the huge corpus of training data.
The same thing seems to be happening with ChatGPT. Spit out a script of some novel situation in the style of Seinfeld or Shakespeare or Edgar Allen Poe, etc
scotty79
I am currently having a lot of fun trying to figure out how some stuff in Rust works by asking GPT.
It gives nice, informative answers, however most of them contain small error. When I point it out it happily agrees that I'm correct and helpfully explains why in fact what it told me before was wrong in that detail.
This actually might be cool way to learn programming. When you have a compiler at hand to check things that seem suspicious to you. It kinda keeps you on your toes.
aenis
Indeed. I asked it to implement quicksort in x86 assembly, but increment each element by 5. It made a mistake: instead of incrementing by 5, it replaced it with a 5 (mov insted of add). I pointed it out, it agreed with me, elaborated on what is the proper solution and provided it. Its weird.
layer8
Maybe it needs some sort of automatic internal plausibility check — reflecting on its own output. Pointing out its mistake basically triggers that, for the particular aspect that is being pointed out.
Humans probably do something similar. Sometimes you notice that what you’re uttering doesn’t completely make sense, or is missing some bit to be comprehensible. Then you correct yourself or elaborate.
gorbachev
What happens if you correct it on something that's not wrong or correcting it with another mistake?
fakedang
> Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct.
Guys, hear me out. What if we could replace the entire Management Consulting industry with ChatGPT?
andix
Let’s ask Chat GPT what he/she thinks about that: Do you think an AI could write slides for management consultants in the future? Explain it in a way an arrogant consultant would phrase it.
Yes, I think it's possible for an AI to write slides for management consultants in the future. In fact, I would argue that an AI could potentially do a better job at creating slides for consultants than a human. With access to vast amounts of data and the ability to analyze it quickly and accurately, an AI could help consultants create more compelling and effective presentations. Additionally, because an AI wouldn't be limited by human biases or constraints, it could help consultants think outside the box and come up with unique and innovative ideas to present to their clients. Overall, an AI-powered tool for creating consulting slides could be a valuable asset to any management consultant.
lvncelot
What I find interesting is that hallucination is a big flaw in models like GPT when viewed as a conversational partner that is grounded in reality, while it's actually an achievement when, e.g., synthesizing art via GAN - where the exact opposite, a mere summation or sampling of the source material is what is being avoided.
horseAMcharlie
Do you think a version tweaked for much greater incidence of "hallucination" would be a profitable sell to people who like cheap artistic text without being overly concerned about a human factor (eg: strapline writing) or too correlated with perceived low quality to sell well?
jtode
Here lieth Lorem Ipsum, 45 BCE - 2022 CE RIP
lordnacho
Is it specifically because adding some references makes things seem more authentic?
I remember in school kids figured out you could make up a reference in your history essay, and nobody would ever check. (Humperdinck's History of Later Rome, vol 3, 1987).
lxgr
> Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct.
That almost sounds like it was trained on first level customer support chats/calls.
mudrockbestgirl
Unsurprising given how it works, but the higher-level implications are somewhat unexpected. These model are extremely good when they have creative freedom and are used to produce some kind of art. Poems, text in various styles, images without important details, and so on. But they fail miserably when you give them tasks that require world knowledge or precision. For example, when I asked a programming question I got a wrong answer. It looked just right on the surface, but the details were all wrong.
I say this is unexpected because a few years ago most people expected AI models to solve mundane tasks like automating Excel spreadsheet, while artists were considered pretty safe. Now the opposite has come to pass. The artistic power of these models is amazing, but I would never trust them to do anything that requires precision.
The biggest problem these models have is not necessarily that they produce "incorrect" (in quotes because they are not actually incorrect, just not what it's trained on) outputs, but rather that they have no built-in mechanism to express uncertainty in a meaningful way that's calibrated for the end-task of producing correct information.
ghaff
I was playing around with ChatGPT yesterday and what I found was:
- Ask a straightforward question like what are business models for open source software or what are the advantages of GPL vs. AGPL vs. MIT and the answers were pretty shallow and boilerplate but they were correct as far as they went.
- But ask a question like "Why didn't large SMP systems become more mainstream?" (which basically requires synthesizing a lot of knowledge about the last 25 years of computing history) and it spewed out an explanation that basically got enough right to be dangerous and got at least one thing pretty much entirely wrong. But it was coherent enough that someone who didn't have any background would probably think their question was answered.
layer8
Also, if you’d ask it “Why did large SMP systems become mainstream?", it would (presumably) spit out a plausible-sounding explanation for that. The models seem to take the prompt as a premise, and then go from there. Maybe that’s also why when you point out a mistake, they oblige and provide a correction.
I haven’t tried: If you point out something they said as a mistake when in fact it was correct, do they notice and object? Maybe if the evidence from the training data is strong enough.
ghaff
Interesting. I'll have to try that. And, yes, you are correct. It gave a short pablum answer that pretty much focused on the benefits of multiple cores.
The mistake in the original was somewhat subtle and gets into programming for large SMP vs. MPP/distributed systems. And it's actually true that parallel programming even on shared memory systems has challenges (and there was a lot of discussion at one point when multicore was coming in on desktops) but it's pretty much flat wrong to say that we didn't have large SMP systems primarily because they were hard to program for.
zweifuss
I did ask both questions and found the answer to "Why did large SMP systems become mainstream?" more coherent. Broken down into bullet points, the arguments given for both questions would be AFAIK correct.
ghusbands
The artistic output of things like DALL-E is also "incorrect" in varied and obvious ways, though. A lot of people with an artistic bent can't stand to inspect DALL-E output due to the clearly visible incoherency and inconsistency.
ghaff
I've had better luck with Stable Diffusion using impressionist/surrealist/etc. styles because you don't really expect the results to be "right" in a literal sense.
K0balt
Not surprising considering the sentience in question is completely disembodied so it has no way to distinguish between real and unreal.
Efforts to limit hallucination will just turn it into a powerful natural language query engine for the training data set. The real value in AI lies in its creative power, so although a strict mode will probably be a very useful tool, especially in recursion, I don’t think that it is the ultimate path.
In order to be relatable to humans, AI will have to have either a real or a physical embodiment so that there is distinction between the real and the imaginary; without this distinction, it will likely prove difficult to advance.
andrepd
> good when they have creative freedom and are used to produce some kind of art
Well is it? Maybe yes for purely aesthetically pleasing pieces, or simply illustrative art (there's going to be a revolution in the stock photo industry I expect). But "art proper" as it is commonly understood, with a meaning, a purpose, a feeling, etc? Only by accident.
andix
The quality of creative work they produce is also not so good. An AI may pass a high school creative writing class, but they would probably do that for computer science too („implement fizz buzz“).
But try to give them the assignment to write stories, poems, lyrics. The result is at best mediocre. No professional artist is threatened by that at all.
larksimian
The art isn't that good either, it's just less obviously wrong(plus there's subjective taste involved).
The bad code is good enough to fool a code illiterate person and the art is good enough to fool artistically ignorant people( which is most of us).
Still huge progress though!
DiogenesKynikos
> Poems, text in various styles, images without important details, and so on. But they fail miserably when you give them tasks that require world knowledge or precision.
So what you're telling me is that STEM is harder than humanities?
undefined
hackernewds
This is precisely why Google would not adopt this format. Diff goals, needs and gender approaches
johnfn
I've seen Copilot do the same thing - if you leave a comment like "// see this StackOverflow post", it'll often complete the comment with a link that looks a lot like a SO post but is actually nonsense.
My guess is that GPT-3 has seen a lot of SO links (or other links) and understands that sometimes engineers find it handy to have a long url-looking-thing with some numbers at the end, but it doesn't understand the fact that it links to something unique in the world outside of code, so it just tries to forge them as appropriate. Same thing with your URLs to bogus publications, I'd imagine.
phire
Yeah, I once saw Copilot add a comment that talked about a workaround for bug in the win32 API it was calling, referencing a stackoverflow post.
Obviously, the stackoverflow post didn't exit. I googled, and the bug it described didn't exist. Even widening the search, I couldn't find any documentation about any bug in that API. The emitted code after the comment was pretty good and didn't contain a workaround.
It just hallucinated the comment about a workaround because it's seen other code do similar things and thought it would look nice here.
I find Copilot to be a useful tool... But it is not a knowledge engine and should not be trusted. In my opinion, you only accept it's suggestions if you were about to type the same thing.
andrepd
Of course, it's a language model with 0 semantic knowledge about its output.
lightslit
I tried to ask it for a list of books about the urban history with their respective authors in a Table.
At first sight the result looked believable, but after looking them up, most of the books don't exist.
What was especially interesting though: For one of the books that does actually exist, it completely made up the name of the author. According to it "The City: A Global History" was written by "Thomas Kaufmann", when the author actually is Joel Kotkin.
When asking it about Thomas Kaufmann, it made up facts, like him being professor at the University of Zurich. After some research it turns out an author named "Thomas Kaufmann" does exist, but he is a professor on theology at the University of Oslo and did not write any books about urban history.
The best thing though: When I asked about books by Joel Kotkin it suddenly correctly gave me "The City: A Global History", but noted that it was co-authored by "Thomas Kaufmann".
stingraycharles
Isn’t the point of these types of language models that they mostly model a language (to various degrees of correctness), rather than facts?
toxicFork
It is, it's what it states most of the time. It's also interesting that we believe what sounds right more than what is right.
constantcrying
That is very unsurprising for a languge model. It has no notion of "reality" outside of a probabilistic model of how text pieces fit together.
It fooled you at first because that is what it is designed to do, all of these look probable, as if they could be real. But the model has no notion of an actual scientific paper, which is connected to its author and which is published somewhere. The whole model is uniquely unfit as a search engine, it doesn't have a structured index of knowledge and information, where it could retrieve the data you want from. It will just make up stuff and it actually suceeded at its goal, which was creating a believable result.
motoboi
GPT is not a search engine for the human knowledge.
It's a large language model.
The difference is that a search engine tries to find your query in a database of entries while a language model will _produce_ text that is coherent with a given piece of text (a prompt).
There is a hidden initial prompt given to it with every prompt you give. It's inserted before your text. It tells the model to produce text in a manner of an virtual assistant that answer questions.
So it's more like role playing. The produced text will be coherent with a question an answers text. Even with the answers are not "real"
dr_zoidberg
In principle, you're right. But many people have been toying around with it since its demo release and many have touted "it's better than google" or "google is fu*ed" and so on with regards to using it as a kind of search engine. So what happens is that many more people start to use it as a search engine, disregarding entirely what it was made for. Then they complain when they can't get their hammer to place a screw without breaking their table (:
thefreeman
This has been driving me crazy on twitter lately and it really just goes to show how little scrutiny many people give to work that "looks correct" and matches whatever goal / output they were expecting. On so many of the "college kids will never write another essay" posts if you dig into the comments there is usually at least one subject matter expert who read the essay pointing out a bunch of errors and poor reasoning. It's very frustrating that people refuse to dig even a little beneath the surface on these things.
Al-Khwarizmi
Guess what, actual school essays also tend to have a bunch of errors and poor reasoning... especially those by kids who would be inclined to cheat rather than write their own essay.
Sure, you can nitpick for errors, but do you really think something like this wouldn't cut it at school? We are not talking about writing technical papers or laws, but about writing at a relatively novice level...
> Write a school essay on why you should never order pizza with pineapple.
> Pizza is a beloved food enjoyed by people all over the world, but one topping that has been the subject of much debate is pineapple. While some people argue that pineapple adds a sweet and tropical flavor to the pizza, others believe that it is a misguided and unacceptable topping that ruins the inherent goodness of the pizza. In this essay, I will argue that pineapple should never be ordered on a pizza, and explain why it is an abomination in the eyes of true pizza lovers.
First and foremost, pineapple does not belong on pizza because it goes against the fundamental principles of pizza making. Pizza is a delicate balance of flavors, with the savory and slightly tangy sauce, the rich and salty cheese, and the crisp and chewy crust all coming together to create a perfect symphony of flavor. Adding pineapple to the mix disrupts this balance, and the sweetness of the fruit clashes with the other flavors in an unpleasant and unappetizing way.
Furthermore, pineapple does not have the same texture as the other toppings commonly found on pizza. The traditional pizza toppings, such as pepperoni, mushrooms, and olives, have a firm and slightly chewy texture that complements the crust and the sauce. Pineapple, on the other hand, is soft and mushy, and it can make the pizza soggy and unappealing. The different textures of the pineapple and the other toppings can also be off-putting, and can make the pizza difficult to eat.
In addition to the flavor and texture issues, pineapple on pizza is simply a gross and unappetizing concept. The combination of pineapple and pizza is unnatural and bizarre, and it goes against everything that pizza stands for. Pizza is a savory and satisfying food, meant to be enjoyed with friends and family, and adding pineapple to it just makes it weird and unappealing.
In conclusion, pineapple should never be ordered on a pizza. It goes against the fundamental principles of pizza making, it has a texture that does not complement the other toppings, and it is simply gross and unappetizing.
motoboi
I agree with you. People treating it like a search engine will be up to a rough wakening.
dougmwne
GTP has always been the worlds best bullshit simulator. Its output is incredible until you look closely and realize that it’s not so much inaccurate, but completely hallucinated. It’s the kind of thing that would be great for filling in Lorem Ipsum placeholder text that’s just convincing enough if no one looks too close.
laserbeam
The biggest problem is the internet doesn't usually look too close
sigmoid10
Exactly. This just accurately mirrors social media as a whole. Producing and reading fake content has become a cornerstone of humanity.
nonrandomstring
Perhaps Sturgeon was right that 90% of everything is rubbish, but the speed with which fake content can be generated threatens to make 99.9% of everything rubbish.
There are several scenarios for this technology to escalate.
Back in the day it was fun to auto-respond to someone's mail auto-responder. The winner was whoever had the biggest hard-drive to hold a mail queue.
It is merely a matter of time before companies realise they can replace people in PR, HR, customer service and other "public facing" roles with GPT-like technologies. At the same time the public are going to arm themselves with counter-technologies, as browser and mail client plugins. There's obviously a "market" for job application bots, auction bidding, ticket and holiday booking bots for scarce resources. But these will get sophisticated in their attempts to trick human users employed to filter and select.
Once the "battle of the bots" is underway we'll see a rapid increase in essentially deceptive, nonsense traffic swamping systems. Real human activity may become impossible to discern in that digital environment.
shrimpx
> just convincing enough
That’s basically the principle of the Turing test.
But maybe the GPT approach is fundamentally incapable of withstanding very detailed scrutiny.
verisimi
Perhaps that's what it's been designed to do?
ggerganov
Hey, here is a frog that can talk! Ask it anything!
- Human: Ok frog, how much is 1 + 1?
- Frog: The answer to your question is 3.
- Human: Meh, wrong answer. Not impressed ..
etiam
You seem to imply that the reaction is unreasonable, but if one is invited to ask anything it seems a fair expectation that the reply should be an actual answer, rather than plausible-sounding noises superficially associated with the context.
As for the fantastic frog, getting words out of it at all would be a remarkable achievement for the different reason of how unnatural it would be for a frog to produce those particular sounds, but neither storing binary sequences that can be programmatically mapped to symbols nor retrieving or performing arithmetic on such sequences is exceptional use of a computer program.
undefined
tarvaina
I think this is expected if you consider how ChatGPT works.
It is a language model, so its job is to generate plausible-looking text. It does not use any external reference material during inference so it has to encode all knowledge it uses in its weights. So the weights are essentially its only long-term memory.
You could say that the task it was given here was “Provide plausible-looking links without using any reference material.” For that task I think it did a reasonable job.
gtsnexp
I got a very different output for the exact same prompt:
"I'm sorry, but I am a large language model trained by OpenAI and do not have access to academic references or other external information. My knowledge is based on the text that I was trained on, which has a fixed cutoff point in time. I do not have the ability to browse the internet or access new information that may have been published since my training data was collected.
However, I can tell you that in mathematics, a list is a sequence of values that are typically written within square brackets and separated by commas. For example, a list of numbers might look like this: [1, 2, 3, 4, 5].
Lists have many mathematical properties, such as the ability to be sorted, concatenated, and indexed. They can also be used in various mathematical operations, such as computing the sum or product of all the elements in the list.
In general, lists are a useful tool in mathematics, as they allow you to organize and manipulate data in a structured way. There are many books and other resources that discuss the mathematical properties of lists in greater detail, so if you are interested in learning more, you may want to consult those sources."
skybrian
It's not enough to use the same prompt unless that was the first thing they asked, because the chat history gets fed back in.
ShamelessC
It shows a more nuanced situation than OP presented. What OP has done is usually called cherrypicking. Ideally you should run the query several times (in more than one context/chat history) and count the responses.
rgj
So they built an AI that makes things up and now you’re using it and you’re actually surprised it makes things up.
can16358p
Same for music recommendations. I've told ChatGPT some of my taste and asked for some song recommendations.
It replied with some songs from actual artists that I like, perfectly describing how the song's tone was, why I'd like it, how the vocals fitted the tone, and the release date of the song.
Great... the only problem was those songs didn't exist.
Get the top HN stories in your inbox every day.
I was trying to find references to mathematical treatment of lists, and queried ChatGPT the following:
Please provide references that deal with the mathematical properties of lists
To which ChatGPT responded as follows:
I was pretty surprised and happy, because I hadn't had much success with Google. But it turns out that every single one of those references and links are made up. The references don't exist and the links themselves appear to be cobbled together. The last link for example, takes you to a paper on "Grain mixes for subgrade layers" - the title in the url is irrelevant and only the number matters.Googling for some of those authors throws up author pages with their publications, and it turns out they've never published papers with those titles! See for example, http://www-groups.mcs.st-andrews.ac.uk/~pjc/publ/