Get the top HN stories in your inbox every day.
huijzer
> On the other hand, taken too literally, such language promotes anthropomorphism, exaggerating the similarities between these artificial intelligence (AI) systems and humans while obscuring their deep differences.
I don't see how the role play term doesn't introduce new issues. Cambridge, for example, defines role play as "pretending to be someone else". An LLM is also not pretending. Also what "role" would an LLM play if you just take the base model without a default prompt or finetuning?
duskwuff
Without any prompt or fine-tuning, most models won't converse with the user at all. They'll operate in a pure text-prediction mode, which is more likely to continue the user's prompt than to respond to it. Or it may end up generating a response to the prompt as if it were a question asked on a web forum like Stack Overflow -- which, yes, does mean that it will generate comments complaining that your question is off-topic and should be closed.
vintermann
Yes. In AI dungeon, which was an early ad-hoc attempt at using a LLM as GM before instruction fine-tuning was figured out, you saw this a lot. I remember some people posting chats where the model would shout that they'd had it with the player, and "User has left the chat".
pixl97
Well, from everything I've seen, without the fine tuning then the output will not be 'human' enough.
Or to put this another way... The 'role' may be more that of Legion in the biblical sense. Depending on the exact question asked hints of particular human characteristics show up, but there is a multitude of different ones and the model would seemingly randomly express them per question.
coldtea
>Also what "role" would an LLM play if you just take the base model without a default prompt or finetuning?
What role would any person into sexual roleplay play if you don't prompt them into sexual roleplay?
mxwsn
I first read about understanding LLMs as simulators generating simulacra a while ago through this post: https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators
VagabundoP
I've tried to play dungeons and dragons with Chatai. It doesn't like to use any mechanical rules and keeps trying to wrap everything up like its got a train to catch.
I'm sure you could tune it to have more attention than a two year old and actually apply the rules. It did come up with a decent setting based on B2: Keep on the Borderlands. And it was some fun.
The new generation of text games will be awesome. And Enders tablet game is possible with this tech.
TeMPOraL
> And Enders tablet game is possible with this tech.
Complete with aliens subtly messing with it, and through it, the player?
They who control the logits, control the future.
gwern
> I've tried to play dungeons and dragons with Chatai. It doesn't like to use any mechanical rules and keeps trying to wrap everything up like its got a train to catch.
Yes, that's the RLHF and other mechanisms. Raters presumably reward completions which are, well, complete and can be quickly judged as a whole, no matter what the intrinsic quality might be or how valid it would be to end with a 'Tune in next week for chapter 2'.
What OP is describing is the emergent behavior of the base model, which acts very differently from the RLHF/instruction-tuned/who-knows-what-else ChatGPT web interface. (This is one reason I tend to avoid that in favor of the Playground and direct model access: there's much less moving machinery behind the scenes.) What the additional stuff does is quite hard to understand because the original prediction objective has been replaced by a bunch of mashed-together objectives combined with feedback loops, leading to some bizarre behavior like being unable to reliably write a nonrhyming poem. (Give that a try in ChatGPT: "write a nonrhyming poem". It's been slooowly getting better at it, perhaps because I and other people keep submitting examples of it failing to do so - but it will still usually fail! and when it does seem to be succeeding finally, if you let it keep writing lines, it will generally gradually revert back to rhyming.) If you go back to the original davinci-001, you'll find that it acts strikingly different than your description of contemporary ChatGPT.
OP, incidentally, has some discussion of what it is like to interact with the real GPT-4, which is not like ChatGPT-4: https://www.lesswrong.com/posts/tbJdxJMAiehewGpq2/impression...
AFAIK, this is one of the only discussions online of what GPT-4-base is like qualitatively. (Note that it sounds a lot like Sydney - if you were around for that, the original Bing Sydney turned out to be a GPT-4 snapshot from partway through training which hadn't been RLHFed but given only some extremely inadequate custom Bing training.)
xeyownt
You didn't read the article, did you? :-D
coldblues
Roleplay is probably the most popular (casual) use of LLMs. Just check projects like SillyTavern.
Tiberium
Yeah, I think all those scientific papers miss out a lot by not checking how people actually RP with models, especially in regards to the prompts used.
dmd
Didn't read the article, huh?
IngoBlechschmid
Related: ideas by Gwern to enhance AI dungeons with caching, yielding a novel form of choose-your-own-adventure games. https://gwern.net/cyoa
j7ake
AI is clearly the hottest field right now.
Are there any other fields where a high fraction of the high impact papers (perspective or otherwise) are coming from industry rather than academia?
seydor
Finance? Mining/oil? Everything engineering with lots of money is bound to drive researchers to the private sector
SiempreViernes
Hush, we're pretending those deep learning papers are science papers, not engineering reports!
j7ake
But there are almost zero papers in glamour journals (high impact journals) from finance or mining/oil.
a_bonobo
A lot of the interesting papers in drug discovery come from industry, clinical trials are not cheap, and much of the data you need for the early discovery phase is hidden away in corporate vaults
eru
Cryptography has a lot of interesting stuff coming from both industry and academia. Especially zero knowledge proofs.
atleastoptimal
[flagged]
ben_w
It might, it might not.
Despise my bullish sentiment on AI (I won't be surprised by LLMs being PhD level in everything in 2 years), it is entirely possible that the LLM approach has practical limits that mean it will always be a jack of all trades and master of none, that mastery can only be achieved by specialised AI such as AlphaZero etc. that the LLMs can call out to and which themselves are hard to create in new domains.
This could in turn cause another AI winter, even as the current state of the art models are turned into fully open-sourced commodities that can run locally on cellphones.
atleastoptimal
Whether it's LLMs some other new groundbreaking architecture the trends and underlying principles are undeniable. To think it's just another hype cycle at this point is silly, and it's crazy how many people look at crypto and AI and think they're remotely similar.
Veraticus
Do you have, like, prior evidence for this statement, or is it just LLM cult stuff?
coldtea
It's current a popular tech, so it must have no realistic limit.
Just like the 1969 moon landing meant spaceships for everybody and space colonies in 30 years...
atleastoptimal
Seriously though if u think the implications of AI can be boiled to LLM cult hype you are seriously shortsighted.
It's weird since u have a public key on your profile you must be someone seriously into tech. Why are so many skilled tech veterans so naively bearish on AI? Is it pride in the primacy of their skills being threatened by artificial intelligence? Is it an inability to gauge general trends outside their field of narrow expertise?
coldtea
The destiny of all natural intelligence is to extinct itself, by damaging the environment, starting an all-out nuclear war, engineer bio weapons, and in many other ways, or just to be wiped out by the first visiting comet ELE, and in the process power starving and killing the substrates of any artificial intelligence it has created too...
low_tech_punk
and artificial intelligence to beget natural intelligence, completing the loop.
visarga
I think both humans and AI's are only as intelligent as their language data, maybe just one bit more than language
otabdeveloper4
There is no such thing as "intelligence".
atleastoptimal
Then why can I type this sentence when a cereal box can't? And please don't give a snarky "because you have hands" type answer.
soulofmischief
Maybe there's no intelligence behind this comment, but intelligence is all around you. It's a clearly defined phenomenon.
slater
There is.
xwdv
The inability to write suggestive or violent scenes or even have a simple fight between characters makes it difficult to do any interesting role play. The frustrating thing is that it feels like it could easily be done, but the creators are bound by some puritanical sense of moral obligation. I hate this!
randcraw
I wonder if this policy will also prohibit the use of violent/war metaphors, like "loaded for bear" or "another weapon in his arsenal". If so, the average sports commentary will throw dozens of red flags.
rhdunn
Metaphors are one of the things that make language interesting, vibrant, and expressive. You can convey so much with them. Prohibiting their use (even in specific contexts like this) is/will be a huge loss.
famouswaffles
Large Language Models are predictors. Not imitators, not simulators. Those are just apparent byproducts of performant prediction.
The end goal and what the loss trending down to is to perfectly model the data it's been given.
So it will not stop at "surface level similarity" or "plausible" or "uninspired" or whatever arbitrary competency line anyone tries to draw in the sand.
It will continue to improve until it is "correct" (as determined by the data).
Stick a bunch of protein sequences and it's not going to stop when it starts generating alphanumeric sequences that look like proteins but really aren't.
It's eventually going to start generating real proteins. https://www.nature.com/articles/s41587-022-01618-2
Then it's going to keep improving until it models the distribution of proteins in the dataset.
I'm honestly not sure what the point of this paper is.
"It is, perhaps, somewhat reassuring to know that LLM-based dialogue agents are not conscious entities with their own agendas and an instinct for self-preservation, and that when they appear to have those things it is merely role play."
Ignoring the whole, "We don't know what consciousness is", it just seems devoid of any meaningful distinction.
"Merely roleplay". What does that even mean ? That's it's not real ? Not really.
They seem to understand this too.
"It would be little consolation to a user deceived into sending real money to a real bank account to know that the agent that brought this about was only playing a role."
Bing has a habit of ending conversations when users say upsetting things. You can talk all you want about how "it's not really upset" but the conversation did end and now you have to start over and be potentially less confrontational if you want to move forward.
"Roleplay" as consequential as the "real thing" is the real thing.
This shiny piece of yellow metal looks like gold, tests like gold, sells like gold but is not...real gold ?
Not unless you have a meaningless definition of real.
nsagent
I've seen you frequently argue similar points about LLMs, but matching a statistical distribution of text is imitation, which necessitates confabulation on the part of LLMs. Namely, humans have underlying causes that affect their writing, like being tired after a long period without rest, frustrated at a traffic jam they are stuck in, or even making typos because they are using a crappy input device like a mobile phone.
If an LLM matches the distribution of text, it might superficially make similar mistakes a human would: introducing a typo that might be common on a phone keyboard. If asked, its reasoning will likely be that the typo was due to a phone keyboard, or maybe another common reason humans give for their typos. Though it's super unlikely that it will give the true reason: that it's been trained on text that exhibited this property.
That's a fundamental difference between an LLM doing an exceedingly competent job at pattern matching human behavior and real human behavior (unless maybe you're a human with schizophrenia).
That doesn't mean current LLMs aren't useful, but it does mean there is a very significant gap between them and the idea of an AGI. As a NLP researcher, I can confidently say we currently don't know how to imbue agency (as in embodied causal reasoning as I described above) into LLMs. There are definitely differences of opinions on how difficult that step is and if we are close, but it is a major limitation of current LLMs that can't be ignored.
famouswaffles
>causes that affect their writing, like being tired after a long period without rest, frustrated at a traffic jam they are stuck in
There is nothing special about fatigue or frustration that make it any less modellable than any other implicit structure present in the dataset that it models just fine.
>Though it's super unlikely that it will give the true reason
It's Super Unlikely humans will give the "true reason" for anything they do. There's a fair bit of research that stated reasons for the decisions we make are often(always?) just post-hoc rationalizations even if you believe otherwise.
>very significant gap between them and the idea of an AGI.
What is this idea of AGI that there exists a significant gap still ? It certainly isn't the idea of being Artificial and Generally Intelligent.
>I can confidently say we currently don't know how to imbue agency (as in embodied causal reasoning
I don't see the difference between what you've described and examples like these.
stale2002
> What is this idea of AGI that there exists a significant gap still?
Because an LLM doesn't simulate the brain.
It's a completely different model that simulates something much different.
> There is nothing special about fatigue or frustration that make it any less modellable than any other implicit structure
Fatigue isn't being modeled.
Instead, the predictive text as the result of that fatigue is being models.
You are confusing output to input.
To make this more clear, imagine a coin flip modeler.
A model that just picked a random seed, is completely different from a model that is doing physics calculations based on a coin flipping in the air.
Even if both models only output "heads" or "tails" .
Same argument applies to language models.
unaindz
> It's super unlikely humans will give the "true reason" for anything they do.
That may be true but the difference between humans and LLM is still there, we can try to think even if we fail to find the true cause sometimes while LLM will just "imitate" data
Grimblewald
What does schizophrenia have to do with it? You might be getting confused by the use of term "hallucination" in the LLM space it doesnt mean what it is conventionally used to mean.
Also look at a defention for AGI from before LLMs and they do a good job fitting the bill. Theyre not quite going to replace all humans everywhere forever, but that was never part of the definition. They match early definitions, especially multimodal models, theres no denying they match what people envisioned agi to be. You can ask fronteir models in plain language to do a task and it can break down the task, formulate a plan of attack, research if need be, sythesise knowledge and apply the solution. If that is all "just word prediction" then so are we.
Just because we understand how it is built doesnt make it any less impressive. More importantly, we simply dont understand LLMs yet. How theyre built, we know, yes, why they work so stupidly well, we do not know.
Importantly, we cant just change definitions to move goal posts because we feel uncomfortable.
Agi defenition
https://www.gartner.com/en/information-technology/glossary/a...
krainboltgreene
> If an LLM matches the distribution of text, it might superficially make similar mistakes a human would
It's also important to point out (IMHO) that the current distribution of text that almost every LLM is based on is almost certainly a slice of the Internet which has a very very specific lean to in terms of culture, language, tone, and availability.
Al-Khwarizmi
Not to argue with anything you said but just as an aside, the Bing habit you mention doesn't seem to be from the LLM itself, but from some censorship module that has been bolted in. In the first few days after release, it would get really confrontational (and sometimes emotional) with some users and this attracted bad press, so now it ends the conversation before getting into any remotely thorny issue. There's also another censorship module that sometimes deletes its output before it has finished.
famouswaffles
Ignoring users was something that was happening from the very beginning. It was just much less abrupt and more conversational/no obvious triggers that wouldn't already be present. You could even still send messages(but you'd be ignored)
Examples
https://www.reddit.com/r/ChatGPT/s/bXrDW2plxG (pic 5)
https://www.reddit.com/r/ChatGPT/s/tc3Iaf5GKc
At least here, it's clear that even if Bing wasn't actually receiving text and predicting a "no token" response then it was able to send an API request that cut off the chat.
May be implemented differently now though.
robertlagrant
> it would get really confrontational (and sometimes emotional) with some users
Nitpick: it was trained to respond with emotional responses by the safety people involved. It wasn't getting emotional itself!
FeepingCreature
So, if as a young child I've learnt to behave emotionally in order to participate in society, my emotions are ... not even "not real" but "not mine"? I mean, we can argue realness, but surely I'm still the one having them.
This feels like a "free will" debate, where if we can explain how a decision was made, that deprives the person of their agency. The model being trained to respond emotionally is why it is getting emotional itself.
YurgenJurgensen
"Tests like gold" is doing a lot of heavy lifting here. By any reasonable standard, LLMs do not 'test like gold'. They may sell like gold, but so do a lot of alloys which are mostly copper if you can find a naive enough buyer. Selling debased metals as gold is a con that's literally thousands of years old and has hundreds of variations, so maybe that metaphor is indeed apt.
You're the seller trying to offload a bunch of shiny yellow coins that are mostly lead telling people not to bother checking for density or ductility or conductivity, because what does it 'really' mean to be 'gold' anyway?
You can give them basic logic tasks and watch them fail abysmally. Who cares if they're "conscious" or "emotional" if they're idiots either way?
altruios
good metaphor. but extending it: do we have the equivalent tests for consciousness (I don't think so, right?)?
So by all tests (1 - visual inspection), it passes as gold...
We need better tests: to test consciousness... and better definitions to define it.
Seems like AL and AI are diverging on how to go about things. AI focusing on LLM's, and AL focusing heavily on CA: 'small parts => build great things' approach.
famouswaffles
Humans who are not formally trained on logic do poorly on logic tests. Not as poor as most LLMs but nothing to write home about.
There is no testable definition of general Intelligence that GPT-4 fails that a chunk of humans also wouldn't. They definitely test like gold lol.
mnky9800n
The point of this paper is that the authors realized they could write a nature paper about this and not get called out in any meaningful way.
coldtea
If you're going to be totally cynical about it, and are not even willing to entertain the idea that they did it to further scientific progress, then you have my upvote.
cs702
I upvoted the OP only because of your comment, so that more people can read it.
People outside the field do not realize that there is a sort of "cottage industry" of academics whose purpose in life appears to be to redefine "intelligence" as whatever they currently believe the machines cannot do. Their arguments tend to be wishy-washy.
Let me propose a simple test for detecting wishy-washy arguments:
1. Replace all references to AI with references to "a person."
2. Re-read the argument with fresh eyes.
3. If the argument no longer seems persuasive, it isn't.
somewhereoutth
Conversely there are many in the field who are happy to redefine intelligence to be whatever it is that the current crop of AIs can do.
However those same people are also trying to sell something, so their claims deserve extra scrutiny.
pixl97
'redefine intelligence'
The particular problem here is both groups are right.
Intelligence is a spectrum of behaviors everywhere from the actions of the lowliest single celled organism up to and exceeding human capabilities. The definition with this wide of range is unfortunately practically useless when it comes to narrowing down a multitude of specific behaviors, specifically around the median of human intelligence. It gets more tricky as nothing in the past really got close to human ability so we were never forced to really define what human intelligence is formally.
We're going to find the same issues here as we do defining the term life.
kelseyfrog
That's a reliable indication that currently intelligence is socially defined. Learning how social definitions replicate, and evolve is going to reveal the levers of change more than debating about definitions themselves. What I'm saying is that social engineering is 90% of the game here.
cs702
LOL. Oh yes, that's true: There are a lot of cheerleaders with shiny pom-poms, always chanting and dancing, trying to get others into joining the AI frenzy.
nsagent
Read my comment [1] for this one academic's take on intelligence in current LLMs. I feel it passes the spirit of the test you provide (doing it literally would be a bit nonsensical).
numpad0
So, this is tangential but, on scale of 0.0f to 1.0f, how likely do you/anyone think it is that AI needs a simulated intestine to proceed from here?
atherton33
> This shiny piece of yellow metal looks like gold, tests like gold, sells like gold but is not...real gold ?
This is exactly a thing said about lab vs found diamonds in jewelry.
I agree with your point, I just think this is an interesting similar effect.
MichaelMoser123
> Large Language Models are predictors. Not imitators, not simulators
How is it that an LLM can react to meta prompts like 'be brief' or 'Ensure responses are unique and without repetition' ?
Get the top HN stories in your inbox every day.
If you ever play Improv you soon realize that learning the basic rules of improv quickly lead to better scenes. You don't verbalize these while playing, but you keep them in your head and have a feel for them. I believe the authors greatly miss out on these internal representations by only focusing on the outcomes of what is being said.
Shameless plug, I made a GPT for playing improv: https://chat.openai.com/g/g-LkQhMxpvM-improv-theatre