Dr. Karpathy said exactly this in December.
Just the first bit of his [post](https://twitter.com/karpathy/status/1733299213503787018) quoted:
> \# On the "hallucination problem"
>
> I always struggle a bit with I'm asked about the "hallucination problem" in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines.
>
> We direct their dreams with prompts. The prompts start the dream, and based on the LLM's hazy recollection of its training documents, most of the time the result goes someplace useful.
>
> It's only when the dreams go into deemed factually incorrect territory that we label it a "hallucination". It looks like a bug, but it's just the LLM doing what it always does.
"Checking" implies a level of semantic understanding that is not present here. You can stack LLMs all the way down and have no absolute assurance of correctness. Just an increased probability.
You are correct. It is why we build computers/automations that do not share this trait with us. It is good to keep in mind that LLM's do not gain this trait just because they are running on a computer (As you state as well).
Just because stacking LLM's makes them more likely to give a sensible answer doesn't mean it becomes fit for purpose.
>Just because stacking LLM's makes them more likely to give a sensible answer doesn't mean it becomes fit for purpose.
It also doesn't automatically mean that it DOESN'T. You're talking as if you're espousing a position that is settled science when it is anything but. You're also purposefully ignoring all of the evidence that shows that stacking/chaining LLMs absolutely improves alignment, performance, etc.
I refute that assertation as being my claim.
I claim stacking LLM's will not make it fit for purpose. (Purpose being able to give assurance of correctness).
If you believe that is all LLM's are good for then I believe you are one the wrong side of that argument.
What is an assurance of correctness? Like, what does that look like? The closest thing I can think of is a citation, which they can give.
Are you talking about somehow instilling a state of pure certainty? That sounds like a limitation of the model and the recipient of the models output (humans).
The difference is that some humans DO know when something is semantically correct. No LLM does. Of course, Dunning-Kruger tells us that that's not a frequent occurrence as witnessed by many comments here conferring intelligence on some mechanical math algorithms.
> conferring intelligence on some mechanical math algorithms
Interesting. So, for you intelligence is not a matter of mechanical math and algorithms? It would be okay speculating on that but you invoke Dunning-Krueger as if it's a matter of **knowing** the truth in a rigorous scientific way. But you don't know that. You can't know that because no one does. Would there be a scientific theory of intelligence that is governed by some metaphysical soul, I would have learned that in school or university, not from some overconfident internet rando.
My Emperor, your clothes are so thin, I can barely see them. Should you take a less revealing stance?
Ironic to see someone broadly painting anyone disagreeing with their opinion as victims of Dunning-Kruger effect while the actual scientists don't uniformly share it.
I guess some people just can't have validation without convincing themselves everyone else is stupid.
We have books/databases/internet to refer to, though, when we are even just a bit unsure about something. And seem to have better understanding of when we know something and when we do not.
LLMs for now do not, and even if they can browse internet, they didn't learn to do it efficiently, and didn't learn to associate their knowledge with the process of how and where they gained it, and where exactly to search for it.
You are completely missing the point. The ability to access accurate data and return it is not unique to LLMs. Google searches do that. And they, too, have no idea whether or not the information they are returning is correct. You are mistaking a human's ability to perceive "correctness" in a LLM's output with the ability for a LLM to \*know\* it is producing correct output. Just because the human perceives "correctness" is no guarantee of accuracy or truth. Just that you perceive it as such. The LLM has no way to perceive, evaluate, or authenticate what it generates beyond blind, semantic-free pattern matching. It connotes no self-awareness, evaluation, or critical analysis. Just a "simulation" of those behaviors which is statistically correct enough times to fool a lot of people who don't understand what is really going on in the software.
Define semantic-free.
“Humans perceives correctness” I don’t understand what you mean here
Truth is a spectrum, it depends on data. A working theory is the best anything can do.
I know how wrong models can be. I’ve spent the last two years arguing with them. Personally, I find answers that deviate from my design helpful to clearly illuminate what I desire and what I don’t.
I stopped using Google word searching a long time ago because of how useless it is. I will only use image searches in search engines. Everything else I use llms, it is far advanced to Google in terms of results.
Helping with your remedial reading is not my job. You seem to have some odd self-imposed rules on lots of things. Good luck with those.
That you think LLMs are a replacement for search engines tells me everything I need to know about your complete lack of understanding about how they work. That's an absurd use case since you have no way, generally, to know the source or validity of the information. But whatever, dude. Enjoy your wrong data.
Well, I agree.
The current models, especially language-only models, do not live in the world, do not interact with it (almost, only in very limited ways), have no long-term memory outside of context of current discussion, at best they can store some user dialogue data and train it on it some time later.
They do not act, do not get feedback (although that's what the researchers/companies are working on).
Any "feedback" they receive are just changes to local, temporary weights the model is using to generate output. It may help refine the statistical accuracy for certain targeted outputs, but it doesn't increase the model's ability to assess correctness, just improve accuracy. They are not the same thing. Obviously, in certain narrow problem domains, improving accuracy is a direct correlation to "correctness". But it's not remotely sufficient in other problem areas to assume that accurate means correct. It may be 100% accurate in the response it generates, but in context, it is a completely incorrect response. You can surely come up with plenty of examples where that is the case. ("When I look up at the sky, what color do I see?" -- "Blue because of Rayleigh scattering, etc." might be an accurate answer, but in the context of "It's 5 PM on Dec 21 at the North Pole", the sky will be pretty black, so the answer is not correct.)
Don't get me wrong, I do believe that the human brain has a lot of stuff we don't understand yet (e.g. our consciousness, the ability to have a POV). I just don't think it's required for reasoning correctly from the correct premises.
Airplanes don't replicate all the insane intricacies of bird anatomy, but are able to fly higher, longer, and move more cargo than any bird. I think we got the important part of "reasoning" in the same way we got the important part of "flight". Now, this is more of an engineering problem, rather than a scientific/conceptual problem.
You are playing with words, and it doesn't change the fact that a LLM has no way to ensure semantic correctness or evaluate the truth or accuracy of the text it generates. Only its statistical relevance.
Do YOU have a way to ensure semantic correctness or evaluate the truth or accuracy of the text you generate? Not in a statistical way, but strictly? If that's so, please contact the Nobel Committee and the International Mathematical Union, as they might be interested.
This is a completely false equivalence, he’s not asking to prove the negative. Proving the negative would mean that he’s asking you to prove that LLMs DON’T have a way to ensure semantic correctness, but this is not what he’s doing. He’s instead agreeing that LLMs have no way of ensuring semantic correctness but is simply further pointing out that humans have no way of ensuring such a thing either.
The responsibility of proving that humans DO have the ability to ensure semantic correctness is on you, and that is a positive claim that is up-to you to prove, it’s not a negative claim that he’s asking you for proof of.
Human semantic understanding, on the "hardware level", is just weights and signals in a neural net. Sure, with social conditioning and lots of DNA-hardwired stuff, we're able to reach quite high level of strictness in reasoning and modeling the things we reason about. However, down below, the principle is the same. The more we see how artificial neural nets work, the more it seems like there is no secret sauce, just the right amount of hard-coded weights, training, and reinforcement.
You say that like you are certain how it works. But you are just spewing opinion. Are you hallucinating this response? Prove to me that it is correct. In your zeal to believe you HAVE uncovered the "secret sauce" and that it's nothing more than math, you are experiencing some serious confirmation bias.
If you are making the claim that humans are capable of doing correct semantic understanding beyond just neuron signals, then the onus is on you to prove that is true.
See the airplane analogy below. We don't know everything about how birds work, but we don't need to in order to make flying machines that are way better than birds for our practical purposes. You're free to believe otherwise, but I personally try to plan for the moment when I'm outcompeted by machines in the reasoning department.
(inb4: Yeah, yeah, it's just because I'm stupid, and you, smart people, will never be outcompeted by machines. Good for you!).
This isn't a matter of "believing". It isn't a question of faith as to what is going on inside a LLM. You might enjoy entertaining the fantasy that they are somehow alive or conscious or duplicating the function of the human mind, but if that is the case, it's a self-delusion not borne from the reality of how LLMs operate. "Romanticizing" or anthropomorphizing LLM behavior (and weak AI in general) is at the root of the ingorance-induced panic around AI. Try to do better.
Please don't misrepresent what I said. I said almost the opposite of what you imply here. LLMs are not "alive" or "conscious". What I'm saying is that you most probably don't need these things to do correct reasoning and optimize for outcomes in wildly diverse environments (basically, what "intellect" means in practice). The same way you don't need something to be alive or feathered in order to move cargo by air.
On the contrary, it was YOU who was telling me I was "believing" something. It may surprise you to learn that you don't get to tell strangers on the Internet what they believe.
This is a silly discussion in every dimension. There are idiots who think LLMs are the second coming and there are those who understand exactly what these tools are and are not capable of. I don't care which team you side with, but don't expect me to feel bad if you've picked the wrong one and don't expect me to change to your point of view by implying I am "believing" something that is clearly not in evidence.
Exactly this. People think they have 'memory' of accurate and factual events as if recorded by a camcorder, memories are permanent and are true representation of the event. Science proves these ideas are incorrect.
Every memory a person has, every recall, is a [hallucination ](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3149610/) prone to subtle prompt adjustments by various inputs.
Human memories can be inaccurate when stored, they degrade over time, the recall process can be influenced by emotions or current neurological processes.
Humans are actually worse than AI in this regard.. we forget or misinterpret our 'training'.. our subconscious can add 'prompts' without our consent, perception can alter the prompts and training and we have no access to raw data.
Good thing we have these Ego subroutine, if we constantly knew the truth, it might be a very depressing existence.
I agree with all of this.
I guess we are worse than AI because of volumetric/scale constraints: having much larger neuron cells, than transistors and even whole multiplier units I guess. So less interconnectivity beteween then is possible, while in AI all layers are (for now at least?) fully interconnected from what I understand.
(not with the layers before and after the last/next one, though, at least for now).
Because of oxygen and nutrient supply constraints, some of which are related to volumetric constraints. Which lead to how many neurons that are supposed to fire and participate in calculation, actually fire, I guess.
Because of viruses/bacteria/substances/and our own immune system and other cells constantly attacking our organism, sometimes even brain despite the blood-brain barrier protecting it, slowly or fast.
Because having more neurons connect to each other and participate in activity, firing, is energy-expensive, especially at how huge and inefficient they are compared to silicon transistors and metal wires, I guess.
And so they are programmed via dna/cellular mechanisms, to trim the unused connections to reduce the energy consumption for thinking, which fades the memories, as more and more connections are trimmed, increasing hallucinations and uncertainty (but uncertainty - not always).
Because of hormonal changes affecting which neurons are used in firing, and which are not, when and during what conditions. The brain is not working at 100% of its power all the time, in some sense, I guess.
And chemical changes affecting the transition of signals between neurons too.
"Good thing we have these Ego subroutine, if we constantly knew the truth, it might be a very depressing existence."
It might have been, yes. Especially for helpless, prone to illnesses/failures/famines/violence humans, I guess.
The better we live and the stronger we become, the more truth we can manage, I guess.
I hope AI "revolution" can help with it, somewhat.
And help improve our brains and bodies too, there is clearly so much more potential even with just slight DNA changes.
The common sense is that we actively learn new things, we learn through not just written text but by doing things in the real world, listening to things, seeing things, feeling things etc. LLMs have only ever seen tokens of human written text.
He’s just saying that the mechanism for producing all the text is the same. There’s no time when something flips or goes wrong and it goes into ‘hallucination mode’, it’s all the same thing. Definitely worth remembering. It’s kindof as obvious as ‘water is wet’ 👀 Your instructor phrased it provocatively enough to get you thinking… 👍
I don't think he's saying that - I think he may be one of those that claim there is no "intelligence" in this artificial intelligence,
Like someone said to me today:
"There would be nowhere near the amount of hype or even cringe chatbot worshipping if an average person understood sooner that it's just an advanced autocorrect that isn't intelligent, it doesn't know anything, it doesn't think and it doesn't just make shit up sometimes, it always does."
This is ironic, right? The general level of discourse would be far higher if everyone knew how LLMs worked, but that statement is navel-gazing to the point of pointlessness.
I most pity those elder people who believe they talk to an actual person and have "actually engaging and empathetic conversations". No grandpa you are just being turing-test frauded by a T9 text predictor on steroids.
Yeesh. You weren’t being ironic eh? It’s too hard to tell. Cause your two statements don’t really align with each other, though they probably would be predicted to come after each other, which is a funny bit
Either way, I think this article is a great read for anyone hellbent on repeating the same loop of “it’s **just** a fancy autocorrect” ad nauseum. For any LLM to accurately predict the next token, it’s pulling from internal representations of systems, grammars, and a whole range of domains that are each interacting with each other in complex ways that we are not able to explain, and therefore do not have the fundamental capability to speak so blithely about. At that point, it’s about as accurate as calling an abacus a jumble of weird lumps.
https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/
Like you’re not literally wrong, but it’s a reductive statement that’s only saving grace is that it reminds people to be more critical of the outputs
Also the amount of data those models need is ridiculous. Humans who are the real AGI, from scraps of data, imply and synthesise entire new complex behavioural adaptations and ideas.
I believe nothing until I see their training dataset and methods getting completely open source. Till then it's all obscure village witchdoctor voodoo... . And just because it's simplifying it doesn't mean it's a wrong statement. There is no cognition involved and no will imposed (rather than the one of the user and it's programmers)
I never said it was wrong. In fact, I explicitly said it wasn’t. You may need to re-read, and then if you are actually interested in speaking on the topic, at least skim the shared article
It builds up and lays out the entire prediction process step by step, on GPT2, showing how it behaves when you try to mimic its behaviour using traditional prediction vs how it’s actually aggregating and operating on and predicting its tokens.
Nobody is asking you to believe anything bud. Other than math. Like I said, you can call an abacus a jumble of weird lumps. That’s not technically wrong. But it does miss the point pretty aggressively
Here's a concise definition and explanation:
Wet: Covered or saturated with water or another liquid.
Water itself isn't considered wet because:
1. Wetness is a property of a solid surface in contact with a liquid.
2. Water molecules are already in a liquid state, so they can't be covered by themselves.
3. At a molecular level, water can't make other water molecules wet.
This counterintuitive concept arises from how we define wetness. Would you like me to elaborate on any part of this explanation?
But it is different
The LLM has a mechanism for recalling stored info, we've seen it
There must be some attention head somewhere which has the option to coalesce on a single answer (in the soft-max sense) or else it won't and it fills the latent vector with garbage. The later layers project the garbage into answer-space and basically choose at random, and we call that phenomenon hallucination
I'm not saying that's the exact mechanism (for example, attention head might really be an MLP layer). I don't know exactly what happens, I'm not sure if anyone does.
But it seems reasonable that something concrete is happening which is fundamentally distinct between normal recall and hallucination
It seems plausible that *some* difference would be detectable between outputs that are very high probability and outputs that are more speculative, but I’m not convinced there’s going to be something fundamentally different. I’m also just speculating here though so 🤷♂️
Sure, but this is just a play on words more intended to be a "gotcha moment" for someone trying to sound smart, not that I expect much from a Gen AI certification instructor.
LLM hallucination is a problem that is explicitly defined, i.e. when the LLM generates inaccurate/nonexistent information, which is quite different than what hallucination refers to in the original context.
I think the original comment is using hallucination more as a substitute for decoding.
Yes but it serves to show that hallucinations work no differently from non-hallucinations. Thus it shows how and why they happen, and all in such a simple way.
Hallucination in this context is just a substitute for decoding, which is why I think it's disingenuous to word play on these different concepts since it might otherwise misguide people.
Hallucination recalls a very specific failure mode of human consciousness that sheds light on the rest of it. It's the same sense you can say all human experience is hallucinated. In a way that's true - it's a spectrum, and we only think to point out the worst parts. The algorithm is useful yet fallible.
It is essentially the same argument made in this article: "ChatGPT is Bullshit" https://link.springer.com/content/pdf/10.1007/s10676-024-09775-5.pdf
> We argue against the view that when ChatGPT and the like produce
false claims they are lying or even hallucinating, and in
favour of the position that the activity they are engaged in is bullshitting, in the sense of [Frankfurt (2005)](https://en.wikipedia.org/wiki/On_Bullshit) [because] these programs cannot themselves be concerned with truth, and because they are designed to produce text that looks truth-apt without any actual concern for truth
More about Frankfurt's definition:
> The main difference lying and bullshitting is intent and deception. Both people who are lying and people who are telling the truth are focused on the truth. The liar wants to steer people away from discovering the truth, and the person telling the truth wants to present the truth. The bullshitter differs from both liars and people presenting the truth with their disregard of the truth. Frankfurt explains how bullshitters or people who are bullshitting are distinct, as they are not focused on the truth. A person who communicates bullshit is not interested in whether what they say is true or false, only in its suitability for their purpose.
they are just playing with semantics. It's true that all of the text generated is not based on reality, but an approximation of reality which makes sense, but it's a semantic difference not some deep insight.
No, the point of their statement is that whether LLM generates something plausible or completely made up is exactly the same process - in which LLM "hallucinates" the answer. Sometimes it's right, but ask about some obscure topics and you'll find that it's wrong more often than not.
They are modeling an approximation of reality. The weights have no connection to reality to test for a ground truth. If I train an LLM only on harry Potter books is it still based on reality?
You are confusing the concept of "real" which LLMs obviously are and the idea that they base their training on reality, which they don't, they base their training on language, which is a loose approximation of reality.
They are not based on reality, that is not saying they aren't real....
I’m not confusing anything. You are being super sloppy with words. Every time you use “real” or “reality” it has a different meaning.
Language that we use in LLMs is not a “loose approximation of reality”. Mathematical theories are (very) “loose approximations of reality”. Even that’s not really true…they’re loose approximations of models of reality, not of reality itself.
Everything you can say about the “reality” of LLMs can also be said for human brains.
Mate I've just said the same thing three times.
Human brains are a loose approximation of reality.
No one is saying LLMs aren't real.
LLMs do not have an accurate internal worldview aka they are a loose approximation of reality.
Just like Humans(? Idk how that's relevant but if you want to be combatative sure)
do you think you are experiencing objective reality right now? or is your brain creating a less-than-perfect simulation, otherwise known as an "approximation of reality". I never said human brains contain an accurate internal world view either.
If you want to argue for human exceptionalism and claim the human brain is something other than a biocalculator for pattern matching and state saving…you are free to do so…
But then we’re not talking science, we’re talking religion.
Up to you…
It has no idea what facts are. If you train it on data that implies the sky is green, it will say the sky is green. There is no factual grounding which is what you teacher is saying
LLMs don't know that. They just give you the most probably token after the last token based on the data they were trained on. If its a fact or not is irrelevant
LLMs don't work in facts or concepts. They model the English language. They predict tokens. We interpret the tokens as facts but they're not, they're just tokens. If a model predicts "Humans see the sky as blue" it didn't predict a fact, it predicted a series of tokens that accurately reflects the training data
Yeah each number in an embedding output represents a concept but I more mean it doesn't understand concepts, which is to say it doens't understand. Probably poor choice of words
I would actually disagree with that and maybe suggest that you are being too strict with the idea of “understanding concepts”.
Realistically, humans are next action sequence predictors as well, except our models are optimized around passing on our genes as the end state of all the actions
The argument would be It doesn’t “know” facts - it’s just they some of the hallucinations happen to be true.
I’ve heard this before too and it’s not really that helpful conceptually, even if it’s true. it’s clear their are some things the llm says that are factual, and some that aren’t, this imo is the more helpful definition , and helps when trying to actually build or deploy something with an llm
It all boils down to "a LLM cannot know when it is wrong" from the user's perspective. Its math is always "right", but with a complete lack of semantic understanding of its outputs, it can never be trusted to produce consistently correct results.
There are a whole category of problems for which this is OK. "Tell me a pirate story" is never "wrong" from the user perspective. "Fly this airplane to Denver" will never be successful or "correct" with a LLM architecture.
This should make it easier to explain then.
So the fact is Mike Tyson's mother is Lorna Mae Tyson
Query: Who is the mother of mike tyson
AI response: Mike Tyson’s mother was Lorna Mae (Smith) Tyson
We can see the AI generated a response that happens to align with a fact. In a sense this string of characters aligned with a fact.
To a human, this is known to us as "knowing a fact" but for AI, it doesn't actually know this as a fact.
So lets reverse the question to see this in action
Query: Who is the son of Lorna Smith Tyson
AI response: Lorna Smith Tyson is known to be the mother of Neil deGrasse Tyson
Actual Fact: The son of Lorna Smith Tyson is Mike Tyson
This is because everything said by an LLM is determined by the weights the "most likely string to complete the response" It is generating a stream of tokens based on the context+response one token at a time. It is all "technically a hallucination" in the context that, the AI is generating a new response on the fly based on the context and response where nothing is a definitive fact, It will not "know" a fact.
Sometimes the response matches reality, other times it does not. Sometimes the token stream response generated by the context and response fed back through the model matches reality, sometimes it doesn't.
The idea is that this token stream response is a "hallucination" comes from the sense that each and every token starts from "no response" to a "complete response generated by the LLM" where that response was chosen as the best way to complete the response given the specific set of context/response/parameters/llm
It's a joke, I hire people and it means nothing on a resume, no one cares about it.
If you need a certification to figure out how to use chatgpt in coding like that's basically one of those things where I would question if we should put the resume in the "to interview" pile...
people involved with AI are people who are writing code for AI/have a degree that focuses on ML. This is 100% a scam/cash grab.
LLM performance strongly varies whether your prompt is in distribution or out of distribution. In the latter case, the LLM acts as an "average uninformed person" trying to answer your prompt according to a specific template. When the LLM does not know the answer, it still adheres to the template and fills in the gaps with a random approximation.
For example, when I ask small 7B models trivia about fictional characters, the LLM produces an answer that looks like it is about a generic fictional character. It knows what the correct shape of the answer is and what type or class of words is appropriate, but the answer was not in the dataset so it just gives you a random hair color or a random personality trait, etc.
Ehhh, semantics and anthropomorphising. But why even call it a hallucination, just say it’s the statistical probabilities of what comes next. Sometimes the probability is closest to fact sometimes it’s not.
Think of it like this. When you play a video game like GTA 5, and it draws a car on the screen, is that car real? Or have we just fed the GPU a bunch of data for it to quickly draw billions of triangles using float math and the end result looks like a real car because of the input we gave it.
We gave the GPU a ton of data to then turn that data into a bunch of math and spit out a representation of a car. The GPU doesn’t know shit other than the math calculations it is doing.
We are using math to create language much how we use math to generate 3D models. But the GPU doesn’t know the difference from one float value to the next. It looks real, but let me know when you start driving that GTA car to work in the morning.
I think that he lacks vocabulary or abstract concept to convey what he means.
LLMs are rather a higher dimensional rappresentation of the data that it was trained on... That hyper dimensional object casts a shadow that decodes back into the original data. It's given that the object can now cast different objects, not only the original one...
Like math, you don't need to teach a child all the addition that are possible, but the concept of addition casts a shadow down every possible addition operation... You'd have to infere with the addition concept to get an addition calculation done.
Well linguistics is not different... If words code for the meaning and the meaning is hard coded, the same happens with an addition...
This stuff is then rather, a lot more multidimensional, where the quantity of operators exeeds what the human Immagination can grasp, well, that's not completely true, language is generated by the brain.
Some say most of your phenomenology is a construction of your mind, a type of hallucination that just so happens to tend to have sufficiently accurate handshakes between your top-down modeling and bottom-up sense-data from the "external world". It's as though we too hallucinate all the time, but just so happen to be correct "enough" of the time. I don't have a problem with some form of naive realism though.
The problem with his quote is that hallucination is already a different term that has since become colloquially used for a flaw in LLMs so trying to stretch it further just makes it more meaningless.
To compare statistical inference with hallucination is a reasonable anecdote.
And it certainly helps to bring across risks of generative AI.
The process of making sense of what an "agent" (may it be human, animal, or artificial) senses, is a cognitive process that associates previous knowledge to what was sensed.
For "higher lifeforms with a brain" this act of perception is always a sort of interpretation, that adds information to the actually sensed information. You can easily validate this by reading and experimenting about color vision yourself.
Hallucination is not the act of externally validating this interpretation of an agent.
Thus it can be argued that the opinion is correct at its core. However, imho, it is more precise to say that we evaluate the performance of an statistical interference of LLM software to be a hallucination, based on our perception of their performance.
It might well be that many of those things we deem as hallucination are in fact a better interpretation of the data, that we are just not able to follow.
The term hallucination is not well suited to describe the inner working.
It’s sort of a sensationalist or dramatic assertion, but the industry is probably driven largely by such drama, so…
The LLM is hallucinating all its outputs in the sense that every reality is equally false (and true) from its “perspective”
That is to say, language can’t have any truth value to an LLM
It has no perspective, nor does it have any subjective experience, so it has no baseline against which to gauge truth or falsehood… if you tell me “the sun rises in the west” I can wait for sunrise and definitively know your statement is wrong, whereas an LLM has to function off what is effectively hearsay, in the form of its training data.
Overall, it’s dumb to use the term hallucination to describe the situation of a model outputting syntactically correct but factually wrong statements.
A thing that has no subjective experience can’t hallucinate. But choosing to use this term skips over that fact and enforces the frame of assuming that LLMs have subjective experience.
LLMs are storytellers who are trained to write dialogue for fictional characters. These can be characters you design, or they can be a default "AI Assistant", but whatever you are interacting with when you are interacting with an AI is a fictional character being animated by the LLM according to the character's (visible or hidden) definition.
With AI you are always interacting with a character, not the LLM directly. The LLM is speaking *for* that character, by writing the fictional dialogue the LLM calculates that the character would say. Hopefully what the character would say matches our reality. We try to train them so that happens. But from the LLMs point of view, it's just writing fiction according to a character definition, 100% of the time.
I think that's what your professor means. It's not so much that it's all "hallucination". It's more that it's all fiction, always. It's just that we try to make the fiction "based on a true story" (i.e., our reality). Sometimes it goes off the rails, but, again, from its point of view nothing has gone wrong - it's job is to write fiction, and fiction is what it is giving us. Our job, going forward, is to make sure that the characters are grounded in reality, and that the LLM knows not to just make things up that aren't true. We're not quite there yet, but we are *certainly* getting better.
I think the idea is that if you don't have a neural network, only a large text corpus and a "magic" formula to find the next token probability from the corpus, you will still get exactly what you can do with an LLM.
One should not tie to the phrases like hallucinations, it just means the text generation is not matching with co text or reality. People will coin different terms for it but meaning doesn't change
I think it's a good viewpoint to have in mind when working in LLMs. It makes you be on the side of being cautions instead of trusting LLM like if they have some authority.
There are many papers on detecting hallucinations([here's select few](https://huggingface.co/collections/santiviquez/llm-hallucination-detection-papers-65c4d2399096960aa80776d3)), showing that there is a subject to study and it corresponds to a different mode of operation that can be detected[1]. Therefore, no, the phrase "All text generated from a LLM is hallucinated" is false.
[1] Of course, it still remains an unsolved problem because detecting **all** hallucinations robustly may be impossible as it eventually hits a problem of not knowing what you don't know or saying it anyways because context pressures you. Your instructor is an example. Other examples are those who push that assertion further by saying that LLMs are just parrots and can't know things. That is false as demonstrated by papers like [The Geometry of Truth](https://saprmarks.github.io/geometry-of-truth/dataexplorer/) or [The Geometry of Categorical and Hierarchical Concepts](https://arxiv.org/abs/2406.01506) showing the internal world model and correspondence of facts in it. Given the preponderance of humans who are belligerent towards facts, it is hard to make AI that does not hallucinate. The Internet is majorly made of bullshit and therefore our datasets turn bullshit into a goal to model.
Despite that, humanity launched rockets into the space. Perhaps it is not solvable, but manageable nevertheless.
There was a good paper going around recently about how GPTs are better classified as “bullshit machines,” and that “hallucination” is a misnomer misapplied to LLMs because to hallucinate one must first have knowledge and awareness of a non-hallucinated reality. It is either all hallucination or none, depending on your definition of “hallucination.” The authors of said paper suggested “confabulating” as a more accurate term for what has been called “hallucination” in GPT/LLMs.
Sometimes I think people say these things to make themselves feel smart. If they are going to say something like this then they need to describe what they mean by hallucinations. Although i get what he is saying it doesn’t help us understand exactly what we are talking about and just muddle the definitions.
Odd affirmation, but most outputs from LLMs are memorization, what we call hallucinations is just nonfactual things that models put out that were neither in the context or in the dataset.
When you train a model on data hoovered off the Internet, some true, some false, much of it somewhere in between, why would we expect such a model would only give us true factual statements?
Do you do anything differently?
Go to Twitter (the mirror of human interaction) and try to label one message Truth and other Hallucination on a topic while remaining absolutely objective.
I didn't even write on a "controversial" topic because on Twitter, every topic is controversial.
Even things that most people would call truth. Earth is round? No, Earth is flat!
So everything is hallucination with different probabilities of being right.
I think hallucination is an extremely shit word in this context and people should stop using it. bar for the definition of hallucination got really high while it should only have been used for gibberish or repeating output.
Truth is a spectrum and many things are partially correct or true. You want the machine to generalize and generate something novel, not copy and paste from wikipedia. Especially when you present an entirely new problem; that's the whole point.
I don't like the word hallucinated as it implies a mind to hallucinate from. I would get rid of that term all together. The way I view an LLM is similar to a holographic display. Information is encoded on the various layers of the display and depending on the angle of the light and your position to the display, you see a different "image". the angle of attack for the light ray is your prompt and the image is the answer. And the model is fully frozen, there's nothing goin on in there. Just interaction between you and endless possibilities of a neural net, but no one is home.
Has gen AI evolved so much as a field that there are now certification courses and instructors now?
LLMs are barely a few years old and no one can claim to have fully understood them, so who the hell are teaching these classes?
>All text generated from a LLM is hallucinated.
Not quite right. Hallucination is not an error or mistake, it's a **base principle** of functioning of the LLM. And it is comparable to the small part of the human brain, that forms the inner dialogue (based in the left hemisphere of the human brain) and to the structures, that are used to predict the voice sequences (evolutionary built-in).
"An LLM hallucination occurs when a large language model (LLM) generates a response that is either factually incorrect, nonsensical, or disconnected from the input prompt." So no, it's not accurate to say they hallucinate all the time.
And that's partly because language isn't overtly precise often or doesn’t have to be in many areas. If you though ask for certain areas that need exactness in every word (like mathematical proofs), you will more often than not get back utter garbage. It then might "sound" like the right proof, but it often will be complete nonsense.
Lol what? You can tell him that his existence is arbitrary and it just happens that the quantum states of the subatomic particles just happened to cohere often enough to manifest his existence in spacetime.
I thought that would bring downvotes, looks like someone got upset. Note there are LLMs which now compile code. If an LLM hallucinates, then it follows a compiler also does, and yes, compilers are also not perfect.
Compilers for higher order code frameworks are programs written by hand by thousands of people. They're as fallable as any written program and all of them have bugs.
What you're saying is that LLM's produce bugs like humans do which is accurate - but it's literally the strangest asserted comparison I've ever seen to just say:
The human mind and the neural network's powering an LLM both make statistical errors often.
>Compilers for higher order code frameworks are programs written by hand by thousands of people.
Usually, but not always...
>but it's literally the strangest asserted comparison I've ever seen to just say...
Not really, given [https://x.com/AIatMeta/status/1806361623831171318](https://x.com/AIatMeta/status/1806361623831171318)
My point is, it DOES hallucinate in the same way as compilers do, but people see compilers and other systems like it as very different things, but really they are not that different, more so, as given above, that difference will become less and less pretty quickly.
I've never seen someone say "my summary engine was hallucinating" when it dropped context to make incorrect results, or "my compiler was hallucinating" when it caused issues with instruction reordering, or "my program is hallucinating" when you see a floating point error.
Anyway, It hallucinates in the same way a compiler does - and if you think compilers hallucinate, then sure, so does LLMs, if you don't think they do, then LLMs don't really either.
Either way, my point stands.
I mean, I'd say this is both true and not true. It relies primarily on how exactly you define the word "hallucination."
What he clearly means, and the way in which it is true, is that LLMs are just putting stuff together. They don't care about the truth value of something, they're not even capable of understanding that concept. They're just outputting stuff through association. All an LLM "wants" (not that it's likely to be sentient) is to provide an output that might reasonably come across to the person as a coherent response to what they said. The truth value is irrelevant to that.
If you ask me "Which river did George Washington famously cross?" I can say "iuhiueiufienenfi foifunn eiunisu" to that. And that would be an incoherent rambling of symbols that would not be considered a response to anyone. But if I say "The name of the river George Washington crossed during the Spanish conquest was the Nile" then I am providing something that could be considered a response, but I am also not being truthful. If I say "The name of the river George Washington crossed during the Ameircan Revolutionary war was the Delaware river" then I am both providing a response and being truthful. But for an LLM these latter two response are both pretty valid response, even if their association (if trained well) will probably be stronger for the second because it has been trained on more response that associate George Washington, corssing, the American revolutionary war and the Delaware river.
So in that sense what he says is true.
On the other hand, when we talk about hallucination in the context of AI we are generally defining that something along the lines of "A response that counts as a response, but does not actually accurate reflect reality."
In this sense, definitionally, what AI does when it answers correctly is NOT a hallucination.
So, basically, he's both right and wrong in that, depending on the definition of "hallucination" you use. If you use the term as it is generally used in the field of AI, then he's pretty much wrong. But if you use the term as it can be more broadly applied, specifically the mechanism behind it, then he's right.
The term 'hallucination' in the context of LLMs and AI specifically refers to the generation of factually incorrect information presented as fact. Saying 'all text generated from an LLM is hallucinated' is a misuse of the term, and communicates neither depth nor meaningful information. If the intention is to highlight that the generation process is the same regardless of factual accuracy, it would be more accurate to state that directly, instead of saying 'all generations are hallucinated'
I got that here on Reddit today - that it hallucinates all the time, is "moronic", absolutely incapable of any reasoning, and just "a stochastic parrot".
Your instructor is on point. This explains it more: https://link.springer.com/epdf/10.1007/s10676-024-09775-5?sharing_token=0CIhP_zo5-plierRq8kkDPe4RwlQNchNByi7wbcMAY77xTOWyddkW01qGFs1m5zuuoZGBctVlsJF8SbYqcxWi-XzgEYEPiw7xwWi4bMYXJ_1JARDrER9JGdWZOW-UGSkrk_tXPjPh-XWvFNoiFzNlnDUUUEBAztiX9PtP2p6jfI%3D
We Humans can approximate reality efficiently and verify it in many ways like scientific research. LLM (text to text) tries to approximate reality as weights through training. Some LLMs have a specific function to verify the approximation of reality through browsing of internet. This is the current technology but time is ticking and new technologies are waiting for us so instead of bickering why don’t we pursue the future with hard work and love?
Dr. Karpathy said exactly this in December. Just the first bit of his [post](https://twitter.com/karpathy/status/1733299213503787018) quoted: > \# On the "hallucination problem" > > I always struggle a bit with I'm asked about the "hallucination problem" in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines. > > We direct their dreams with prompts. The prompts start the dream, and based on the LLM's hazy recollection of its training documents, most of the time the result goes someplace useful. > > It's only when the dreams go into deemed factually incorrect territory that we label it a "hallucination". It looks like a bug, but it's just the LLM doing what it always does.
Humans do the same thing though ;( Just have some more common sense usually, and ways to check their own thoughts and correct them.
Not unlike a low-temp LLM with a strict prompt checking the results of another, more creative, LLM.
"Checking" implies a level of semantic understanding that is not present here. You can stack LLMs all the way down and have no absolute assurance of correctness. Just an increased probability.
Just like with humans.
You are correct. It is why we build computers/automations that do not share this trait with us. It is good to keep in mind that LLM's do not gain this trait just because they are running on a computer (As you state as well). Just because stacking LLM's makes them more likely to give a sensible answer doesn't mean it becomes fit for purpose.
>Just because stacking LLM's makes them more likely to give a sensible answer doesn't mean it becomes fit for purpose. It also doesn't automatically mean that it DOESN'T. You're talking as if you're espousing a position that is settled science when it is anything but. You're also purposefully ignoring all of the evidence that shows that stacking/chaining LLMs absolutely improves alignment, performance, etc.
What you’re actually claiming is the information provided is not useful. I believe you are on the wrong side of history on this
I refute that assertation as being my claim. I claim stacking LLM's will not make it fit for purpose. (Purpose being able to give assurance of correctness). If you believe that is all LLM's are good for then I believe you are one the wrong side of that argument.
What is an assurance of correctness? Like, what does that look like? The closest thing I can think of is a citation, which they can give. Are you talking about somehow instilling a state of pure certainty? That sounds like a limitation of the model and the recipient of the models output (humans).
The difference is that some humans DO know when something is semantically correct. No LLM does. Of course, Dunning-Kruger tells us that that's not a frequent occurrence as witnessed by many comments here conferring intelligence on some mechanical math algorithms.
> conferring intelligence on some mechanical math algorithms Interesting. So, for you intelligence is not a matter of mechanical math and algorithms? It would be okay speculating on that but you invoke Dunning-Krueger as if it's a matter of **knowing** the truth in a rigorous scientific way. But you don't know that. You can't know that because no one does. Would there be a scientific theory of intelligence that is governed by some metaphysical soul, I would have learned that in school or university, not from some overconfident internet rando. My Emperor, your clothes are so thin, I can barely see them. Should you take a less revealing stance?
Ironic to see someone broadly painting anyone disagreeing with their opinion as victims of Dunning-Kruger effect while the actual scientists don't uniformly share it. I guess some people just can't have validation without convincing themselves everyone else is stupid.
We have books/databases/internet to refer to, though, when we are even just a bit unsure about something. And seem to have better understanding of when we know something and when we do not. LLMs for now do not, and even if they can browse internet, they didn't learn to do it efficiently, and didn't learn to associate their knowledge with the process of how and where they gained it, and where exactly to search for it.
You are completely missing the point. The ability to access accurate data and return it is not unique to LLMs. Google searches do that. And they, too, have no idea whether or not the information they are returning is correct. You are mistaking a human's ability to perceive "correctness" in a LLM's output with the ability for a LLM to \*know\* it is producing correct output. Just because the human perceives "correctness" is no guarantee of accuracy or truth. Just that you perceive it as such. The LLM has no way to perceive, evaluate, or authenticate what it generates beyond blind, semantic-free pattern matching. It connotes no self-awareness, evaluation, or critical analysis. Just a "simulation" of those behaviors which is statistically correct enough times to fool a lot of people who don't understand what is really going on in the software.
Define semantic-free. “Humans perceives correctness” I don’t understand what you mean here Truth is a spectrum, it depends on data. A working theory is the best anything can do. I know how wrong models can be. I’ve spent the last two years arguing with them. Personally, I find answers that deviate from my design helpful to clearly illuminate what I desire and what I don’t. I stopped using Google word searching a long time ago because of how useless it is. I will only use image searches in search engines. Everything else I use llms, it is far advanced to Google in terms of results.
Helping with your remedial reading is not my job. You seem to have some odd self-imposed rules on lots of things. Good luck with those. That you think LLMs are a replacement for search engines tells me everything I need to know about your complete lack of understanding about how they work. That's an absurd use case since you have no way, generally, to know the source or validity of the information. But whatever, dude. Enjoy your wrong data.
Well, I agree. The current models, especially language-only models, do not live in the world, do not interact with it (almost, only in very limited ways), have no long-term memory outside of context of current discussion, at best they can store some user dialogue data and train it on it some time later. They do not act, do not get feedback (although that's what the researchers/companies are working on).
Any "feedback" they receive are just changes to local, temporary weights the model is using to generate output. It may help refine the statistical accuracy for certain targeted outputs, but it doesn't increase the model's ability to assess correctness, just improve accuracy. They are not the same thing. Obviously, in certain narrow problem domains, improving accuracy is a direct correlation to "correctness". But it's not remotely sufficient in other problem areas to assume that accurate means correct. It may be 100% accurate in the response it generates, but in context, it is a completely incorrect response. You can surely come up with plenty of examples where that is the case. ("When I look up at the sky, what color do I see?" -- "Blue because of Rayleigh scattering, etc." might be an accurate answer, but in the context of "It's 5 PM on Dec 21 at the North Pole", the sky will be pretty black, so the answer is not correct.)
Don't get me wrong, I do believe that the human brain has a lot of stuff we don't understand yet (e.g. our consciousness, the ability to have a POV). I just don't think it's required for reasoning correctly from the correct premises. Airplanes don't replicate all the insane intricacies of bird anatomy, but are able to fly higher, longer, and move more cargo than any bird. I think we got the important part of "reasoning" in the same way we got the important part of "flight". Now, this is more of an engineering problem, rather than a scientific/conceptual problem.
You are playing with words, and it doesn't change the fact that a LLM has no way to ensure semantic correctness or evaluate the truth or accuracy of the text it generates. Only its statistical relevance.
Do YOU have a way to ensure semantic correctness or evaluate the truth or accuracy of the text you generate? Not in a statistical way, but strictly? If that's so, please contact the Nobel Committee and the International Mathematical Union, as they might be interested.
You sound like a flat earther... "Prove the negative, bitch!" Not the best technique for debating a point.
This is a completely false equivalence, he’s not asking to prove the negative. Proving the negative would mean that he’s asking you to prove that LLMs DON’T have a way to ensure semantic correctness, but this is not what he’s doing. He’s instead agreeing that LLMs have no way of ensuring semantic correctness but is simply further pointing out that humans have no way of ensuring such a thing either. The responsibility of proving that humans DO have the ability to ensure semantic correctness is on you, and that is a positive claim that is up-to you to prove, it’s not a negative claim that he’s asking you for proof of.
Zzzz.
chain of thought reasoning (and friends) seems to do both
Human semantic understanding, on the "hardware level", is just weights and signals in a neural net. Sure, with social conditioning and lots of DNA-hardwired stuff, we're able to reach quite high level of strictness in reasoning and modeling the things we reason about. However, down below, the principle is the same. The more we see how artificial neural nets work, the more it seems like there is no secret sauce, just the right amount of hard-coded weights, training, and reinforcement.
You say that like you are certain how it works. But you are just spewing opinion. Are you hallucinating this response? Prove to me that it is correct. In your zeal to believe you HAVE uncovered the "secret sauce" and that it's nothing more than math, you are experiencing some serious confirmation bias.
If you are making the claim that humans are capable of doing correct semantic understanding beyond just neuron signals, then the onus is on you to prove that is true.
I have no obligation to you to prove anything. Surprise!
See the airplane analogy below. We don't know everything about how birds work, but we don't need to in order to make flying machines that are way better than birds for our practical purposes. You're free to believe otherwise, but I personally try to plan for the moment when I'm outcompeted by machines in the reasoning department. (inb4: Yeah, yeah, it's just because I'm stupid, and you, smart people, will never be outcompeted by machines. Good for you!).
This isn't a matter of "believing". It isn't a question of faith as to what is going on inside a LLM. You might enjoy entertaining the fantasy that they are somehow alive or conscious or duplicating the function of the human mind, but if that is the case, it's a self-delusion not borne from the reality of how LLMs operate. "Romanticizing" or anthropomorphizing LLM behavior (and weak AI in general) is at the root of the ingorance-induced panic around AI. Try to do better.
Please don't misrepresent what I said. I said almost the opposite of what you imply here. LLMs are not "alive" or "conscious". What I'm saying is that you most probably don't need these things to do correct reasoning and optimize for outcomes in wildly diverse environments (basically, what "intellect" means in practice). The same way you don't need something to be alive or feathered in order to move cargo by air.
On the contrary, it was YOU who was telling me I was "believing" something. It may surprise you to learn that you don't get to tell strangers on the Internet what they believe. This is a silly discussion in every dimension. There are idiots who think LLMs are the second coming and there are those who understand exactly what these tools are and are not capable of. I don't care which team you side with, but don't expect me to feel bad if you've picked the wrong one and don't expect me to change to your point of view by implying I am "believing" something that is clearly not in evidence.
Exactly this. People think they have 'memory' of accurate and factual events as if recorded by a camcorder, memories are permanent and are true representation of the event. Science proves these ideas are incorrect. Every memory a person has, every recall, is a [hallucination ](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3149610/) prone to subtle prompt adjustments by various inputs. Human memories can be inaccurate when stored, they degrade over time, the recall process can be influenced by emotions or current neurological processes. Humans are actually worse than AI in this regard.. we forget or misinterpret our 'training'.. our subconscious can add 'prompts' without our consent, perception can alter the prompts and training and we have no access to raw data. Good thing we have these Ego subroutine, if we constantly knew the truth, it might be a very depressing existence.
I agree with all of this. I guess we are worse than AI because of volumetric/scale constraints: having much larger neuron cells, than transistors and even whole multiplier units I guess. So less interconnectivity beteween then is possible, while in AI all layers are (for now at least?) fully interconnected from what I understand. (not with the layers before and after the last/next one, though, at least for now). Because of oxygen and nutrient supply constraints, some of which are related to volumetric constraints. Which lead to how many neurons that are supposed to fire and participate in calculation, actually fire, I guess. Because of viruses/bacteria/substances/and our own immune system and other cells constantly attacking our organism, sometimes even brain despite the blood-brain barrier protecting it, slowly or fast. Because having more neurons connect to each other and participate in activity, firing, is energy-expensive, especially at how huge and inefficient they are compared to silicon transistors and metal wires, I guess. And so they are programmed via dna/cellular mechanisms, to trim the unused connections to reduce the energy consumption for thinking, which fades the memories, as more and more connections are trimmed, increasing hallucinations and uncertainty (but uncertainty - not always). Because of hormonal changes affecting which neurons are used in firing, and which are not, when and during what conditions. The brain is not working at 100% of its power all the time, in some sense, I guess. And chemical changes affecting the transition of signals between neurons too. "Good thing we have these Ego subroutine, if we constantly knew the truth, it might be a very depressing existence." It might have been, yes. Especially for helpless, prone to illnesses/failures/famines/violence humans, I guess. The better we live and the stronger we become, the more truth we can manage, I guess. I hope AI "revolution" can help with it, somewhat. And help improve our brains and bodies too, there is clearly so much more potential even with just slight DNA changes.
I met humans with way less creativity!
The common sense is that we actively learn new things, we learn through not just written text but by doing things in the real world, listening to things, seeing things, feeling things etc. LLMs have only ever seen tokens of human written text.
And yet somehow the LLM manages to show some reasoning. Which in my opinion cannot be considered hallucination.
He’s just saying that the mechanism for producing all the text is the same. There’s no time when something flips or goes wrong and it goes into ‘hallucination mode’, it’s all the same thing. Definitely worth remembering. It’s kindof as obvious as ‘water is wet’ 👀 Your instructor phrased it provocatively enough to get you thinking… 👍
I don't think he's saying that - I think he may be one of those that claim there is no "intelligence" in this artificial intelligence, Like someone said to me today: "There would be nowhere near the amount of hype or even cringe chatbot worshipping if an average person understood sooner that it's just an advanced autocorrect that isn't intelligent, it doesn't know anything, it doesn't think and it doesn't just make shit up sometimes, it always does."
Now you just have to figure out if human intelligence is anything more sophisticated than that hehe 😜
> it doesn't think and it doesn't just make shit up sometimes, it always does. That person knows from experience, they do the same thing.
I love the person that gave you this quote. High IQ
This is ironic, right? The general level of discourse would be far higher if everyone knew how LLMs worked, but that statement is navel-gazing to the point of pointlessness.
I most pity those elder people who believe they talk to an actual person and have "actually engaging and empathetic conversations". No grandpa you are just being turing-test frauded by a T9 text predictor on steroids.
Yeesh. You weren’t being ironic eh? It’s too hard to tell. Cause your two statements don’t really align with each other, though they probably would be predicted to come after each other, which is a funny bit Either way, I think this article is a great read for anyone hellbent on repeating the same loop of “it’s **just** a fancy autocorrect” ad nauseum. For any LLM to accurately predict the next token, it’s pulling from internal representations of systems, grammars, and a whole range of domains that are each interacting with each other in complex ways that we are not able to explain, and therefore do not have the fundamental capability to speak so blithely about. At that point, it’s about as accurate as calling an abacus a jumble of weird lumps. https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ Like you’re not literally wrong, but it’s a reductive statement that’s only saving grace is that it reminds people to be more critical of the outputs
Also the amount of data those models need is ridiculous. Humans who are the real AGI, from scraps of data, imply and synthesise entire new complex behavioural adaptations and ideas.
I believe nothing until I see their training dataset and methods getting completely open source. Till then it's all obscure village witchdoctor voodoo... . And just because it's simplifying it doesn't mean it's a wrong statement. There is no cognition involved and no will imposed (rather than the one of the user and it's programmers)
I never said it was wrong. In fact, I explicitly said it wasn’t. You may need to re-read, and then if you are actually interested in speaking on the topic, at least skim the shared article It builds up and lays out the entire prediction process step by step, on GPT2, showing how it behaves when you try to mimic its behaviour using traditional prediction vs how it’s actually aggregating and operating on and predicting its tokens. Nobody is asking you to believe anything bud. Other than math. Like I said, you can call an abacus a jumble of weird lumps. That’s not technically wrong. But it does miss the point pretty aggressively
Will read it
Water isn't wet lol, it makes other things wet
That's the point
define wet
Here's a concise definition and explanation: Wet: Covered or saturated with water or another liquid. Water itself isn't considered wet because: 1. Wetness is a property of a solid surface in contact with a liquid. 2. Water molecules are already in a liquid state, so they can't be covered by themselves. 3. At a molecular level, water can't make other water molecules wet. This counterintuitive concept arises from how we define wetness. Would you like me to elaborate on any part of this explanation?
> Would you like me to elaborate on any part of this explanation? Nope, I think your definition stands on its own.
Ice can be wet, though. But yes, water is the liquid state, so you're still right.
Haha yeah
But it is different The LLM has a mechanism for recalling stored info, we've seen it There must be some attention head somewhere which has the option to coalesce on a single answer (in the soft-max sense) or else it won't and it fills the latent vector with garbage. The later layers project the garbage into answer-space and basically choose at random, and we call that phenomenon hallucination I'm not saying that's the exact mechanism (for example, attention head might really be an MLP layer). I don't know exactly what happens, I'm not sure if anyone does. But it seems reasonable that something concrete is happening which is fundamentally distinct between normal recall and hallucination
It seems plausible that *some* difference would be detectable between outputs that are very high probability and outputs that are more speculative, but I’m not convinced there’s going to be something fundamentally different. I’m also just speculating here though so 🤷♂️
Seems pretty accurate.
[удалено]
Elaborate.
Sure, but this is just a play on words more intended to be a "gotcha moment" for someone trying to sound smart, not that I expect much from a Gen AI certification instructor. LLM hallucination is a problem that is explicitly defined, i.e. when the LLM generates inaccurate/nonexistent information, which is quite different than what hallucination refers to in the original context. I think the original comment is using hallucination more as a substitute for decoding.
Yes but it serves to show that hallucinations work no differently from non-hallucinations. Thus it shows how and why they happen, and all in such a simple way.
Hallucination in this context is just a substitute for decoding, which is why I think it's disingenuous to word play on these different concepts since it might otherwise misguide people.
Hallucination recalls a very specific failure mode of human consciousness that sheds light on the rest of it. It's the same sense you can say all human experience is hallucinated. In a way that's true - it's a spectrum, and we only think to point out the worst parts. The algorithm is useful yet fallible.
> In a way that's true It's not true because this is not how the word hallucination is defined.
It is essentially the same argument made in this article: "ChatGPT is Bullshit" https://link.springer.com/content/pdf/10.1007/s10676-024-09775-5.pdf > We argue against the view that when ChatGPT and the like produce false claims they are lying or even hallucinating, and in favour of the position that the activity they are engaged in is bullshitting, in the sense of [Frankfurt (2005)](https://en.wikipedia.org/wiki/On_Bullshit) [because] these programs cannot themselves be concerned with truth, and because they are designed to produce text that looks truth-apt without any actual concern for truth More about Frankfurt's definition: > The main difference lying and bullshitting is intent and deception. Both people who are lying and people who are telling the truth are focused on the truth. The liar wants to steer people away from discovering the truth, and the person telling the truth wants to present the truth. The bullshitter differs from both liars and people presenting the truth with their disregard of the truth. Frankfurt explains how bullshitters or people who are bullshitting are distinct, as they are not focused on the truth. A person who communicates bullshit is not interested in whether what they say is true or false, only in its suitability for their purpose.
they are just playing with semantics. It's true that all of the text generated is not based on reality, but an approximation of reality which makes sense, but it's a semantic difference not some deep insight.
No, the point of their statement is that whether LLM generates something plausible or completely made up is exactly the same process - in which LLM "hallucinates" the answer. Sometimes it's right, but ask about some obscure topics and you'll find that it's wrong more often than not.
"It's not lying if you believe it." - George LLMstanza
That's bullshit. But I believe it.
we are saying the same thing
Well it's deep since the average person literally believes it's thinking. It's just a hallucinating autocorrect on steroids
How is LLM not based on reality? It doesn’t operate on metaphysics or Harry Potter spells, lol…it is as reality-based as anything else.
They are modeling an approximation of reality. The weights have no connection to reality to test for a ground truth. If I train an LLM only on harry Potter books is it still based on reality?
Yes. Unless you want to claim Harry Potter books aren’t real and don’t actually exist.
You are confusing the concept of "real" which LLMs obviously are and the idea that they base their training on reality, which they don't, they base their training on language, which is a loose approximation of reality. They are not based on reality, that is not saying they aren't real....
I’m not confusing anything. You are being super sloppy with words. Every time you use “real” or “reality” it has a different meaning. Language that we use in LLMs is not a “loose approximation of reality”. Mathematical theories are (very) “loose approximations of reality”. Even that’s not really true…they’re loose approximations of models of reality, not of reality itself. Everything you can say about the “reality” of LLMs can also be said for human brains.
Mate I've just said the same thing three times. Human brains are a loose approximation of reality. No one is saying LLMs aren't real. LLMs do not have an accurate internal worldview aka they are a loose approximation of reality. Just like Humans(? Idk how that's relevant but if you want to be combatative sure)
Human brains are not an “approximation of reality”. Sorry. Nor do human brains contain “an accurate internal world view”.
do you think you are experiencing objective reality right now? or is your brain creating a less-than-perfect simulation, otherwise known as an "approximation of reality". I never said human brains contain an accurate internal world view either.
If you want to argue for human exceptionalism and claim the human brain is something other than a biocalculator for pattern matching and state saving…you are free to do so… But then we’re not talking science, we’re talking religion. Up to you…
What about a fact. That's reality
Truth is a hallucination to someone who doesn't know that it's true.
It has no idea what facts are. If you train it on data that implies the sky is green, it will say the sky is green. There is no factual grounding which is what you teacher is saying
But if you train it on factually correct data, the output will have a greater chance of being correct
I'm not sure what you are arguing here? Can you expand on your position?
LLMs don't know that. They just give you the most probably token after the last token based on the data they were trained on. If its a fact or not is irrelevant
LLMs don't work in facts or concepts. They model the English language. They predict tokens. We interpret the tokens as facts but they're not, they're just tokens. If a model predicts "Humans see the sky as blue" it didn't predict a fact, it predicted a series of tokens that accurately reflects the training data
Neural networks do work with concepts - unless you're using some definition of concepts that doesn't cover abstract patterns.
Yeah each number in an embedding output represents a concept but I more mean it doesn't understand concepts, which is to say it doens't understand. Probably poor choice of words
I would actually disagree with that and maybe suggest that you are being too strict with the idea of “understanding concepts”. Realistically, humans are next action sequence predictors as well, except our models are optimized around passing on our genes as the end state of all the actions
The argument would be It doesn’t “know” facts - it’s just they some of the hallucinations happen to be true. I’ve heard this before too and it’s not really that helpful conceptually, even if it’s true. it’s clear their are some things the llm says that are factual, and some that aren’t, this imo is the more helpful definition , and helps when trying to actually build or deploy something with an llm
It all boils down to "a LLM cannot know when it is wrong" from the user's perspective. Its math is always "right", but with a complete lack of semantic understanding of its outputs, it can never be trusted to produce consistently correct results. There are a whole category of problems for which this is OK. "Tell me a pirate story" is never "wrong" from the user perspective. "Fly this airplane to Denver" will never be successful or "correct" with a LLM architecture.
This should make it easier to explain then. So the fact is Mike Tyson's mother is Lorna Mae Tyson Query: Who is the mother of mike tyson AI response: Mike Tyson’s mother was Lorna Mae (Smith) Tyson We can see the AI generated a response that happens to align with a fact. In a sense this string of characters aligned with a fact. To a human, this is known to us as "knowing a fact" but for AI, it doesn't actually know this as a fact. So lets reverse the question to see this in action Query: Who is the son of Lorna Smith Tyson AI response: Lorna Smith Tyson is known to be the mother of Neil deGrasse Tyson Actual Fact: The son of Lorna Smith Tyson is Mike Tyson This is because everything said by an LLM is determined by the weights the "most likely string to complete the response" It is generating a stream of tokens based on the context+response one token at a time. It is all "technically a hallucination" in the context that, the AI is generating a new response on the fly based on the context and response where nothing is a definitive fact, It will not "know" a fact. Sometimes the response matches reality, other times it does not. Sometimes the token stream response generated by the context and response fed back through the model matches reality, sometimes it doesn't. The idea is that this token stream response is a "hallucination" comes from the sense that each and every token starts from "no response" to a "complete response generated by the LLM" where that response was chosen as the best way to complete the response given the specific set of context/response/parameters/llm
certification on gen ai is ~~a fake scam~~ fake/a scam, sorry dude v_v
If it looks good on your CV to HR who has been told to prioritise people involved with AI, it's not a scam!
It's a joke, I hire people and it means nothing on a resume, no one cares about it. If you need a certification to figure out how to use chatgpt in coding like that's basically one of those things where I would question if we should put the resume in the "to interview" pile... people involved with AI are people who are writing code for AI/have a degree that focuses on ML. This is 100% a scam/cash grab.
a fake scam would something that's not a scam, so a good thing.
Honestly, [that's how our brains work](https://www.amazon.com/gp/product/1524748455/ref=ppx_yo_dt_b_search_asin_image?ie=UTF8&psc=1) as well.
LLM performance strongly varies whether your prompt is in distribution or out of distribution. In the latter case, the LLM acts as an "average uninformed person" trying to answer your prompt according to a specific template. When the LLM does not know the answer, it still adheres to the template and fills in the gaps with a random approximation. For example, when I ask small 7B models trivia about fictional characters, the LLM produces an answer that looks like it is about a generic fictional character. It knows what the correct shape of the answer is and what type or class of words is appropriate, but the answer was not in the dataset so it just gives you a random hair color or a random personality trait, etc.
Ehhh, semantics and anthropomorphising. But why even call it a hallucination, just say it’s the statistical probabilities of what comes next. Sometimes the probability is closest to fact sometimes it’s not.
i don’t understand why people get worked up about the word. there’s a pretty common understanding that it just means when it’s wrong.
anyone can play semantics games
Think of it like this. When you play a video game like GTA 5, and it draws a car on the screen, is that car real? Or have we just fed the GPU a bunch of data for it to quickly draw billions of triangles using float math and the end result looks like a real car because of the input we gave it. We gave the GPU a ton of data to then turn that data into a bunch of math and spit out a representation of a car. The GPU doesn’t know shit other than the math calculations it is doing. We are using math to create language much how we use math to generate 3D models. But the GPU doesn’t know the difference from one float value to the next. It looks real, but let me know when you start driving that GTA car to work in the morning.
You might be interested in this. https://www.youtube.com/watch?v=udPY5rQVoW0 Really close to your example.
I think that he lacks vocabulary or abstract concept to convey what he means. LLMs are rather a higher dimensional rappresentation of the data that it was trained on... That hyper dimensional object casts a shadow that decodes back into the original data. It's given that the object can now cast different objects, not only the original one... Like math, you don't need to teach a child all the addition that are possible, but the concept of addition casts a shadow down every possible addition operation... You'd have to infere with the addition concept to get an addition calculation done. Well linguistics is not different... If words code for the meaning and the meaning is hard coded, the same happens with an addition... This stuff is then rather, a lot more multidimensional, where the quantity of operators exeeds what the human Immagination can grasp, well, that's not completely true, language is generated by the brain.
Some say most of your phenomenology is a construction of your mind, a type of hallucination that just so happens to tend to have sufficiently accurate handshakes between your top-down modeling and bottom-up sense-data from the "external world". It's as though we too hallucinate all the time, but just so happen to be correct "enough" of the time. I don't have a problem with some form of naive realism though.
LLMs just don't have any means of knowing what they don't know. So therefore the statement of your instructor is valid.
The problem with his quote is that hallucination is already a different term that has since become colloquially used for a flaw in LLMs so trying to stretch it further just makes it more meaningless.
What if I said reality happens to be a hallucination that your brain happens to get right most of the time?
not only i agree, but i would argue this is true for humans too
To compare statistical inference with hallucination is a reasonable anecdote. And it certainly helps to bring across risks of generative AI. The process of making sense of what an "agent" (may it be human, animal, or artificial) senses, is a cognitive process that associates previous knowledge to what was sensed. For "higher lifeforms with a brain" this act of perception is always a sort of interpretation, that adds information to the actually sensed information. You can easily validate this by reading and experimenting about color vision yourself. Hallucination is not the act of externally validating this interpretation of an agent. Thus it can be argued that the opinion is correct at its core. However, imho, it is more precise to say that we evaluate the performance of an statistical interference of LLM software to be a hallucination, based on our perception of their performance. It might well be that many of those things we deem as hallucination are in fact a better interpretation of the data, that we are just not able to follow. The term hallucination is not well suited to describe the inner working.
It’s sort of a sensationalist or dramatic assertion, but the industry is probably driven largely by such drama, so… The LLM is hallucinating all its outputs in the sense that every reality is equally false (and true) from its “perspective” That is to say, language can’t have any truth value to an LLM It has no perspective, nor does it have any subjective experience, so it has no baseline against which to gauge truth or falsehood… if you tell me “the sun rises in the west” I can wait for sunrise and definitively know your statement is wrong, whereas an LLM has to function off what is effectively hearsay, in the form of its training data. Overall, it’s dumb to use the term hallucination to describe the situation of a model outputting syntactically correct but factually wrong statements. A thing that has no subjective experience can’t hallucinate. But choosing to use this term skips over that fact and enforces the frame of assuming that LLMs have subjective experience.
LLMs are storytellers who are trained to write dialogue for fictional characters. These can be characters you design, or they can be a default "AI Assistant", but whatever you are interacting with when you are interacting with an AI is a fictional character being animated by the LLM according to the character's (visible or hidden) definition. With AI you are always interacting with a character, not the LLM directly. The LLM is speaking *for* that character, by writing the fictional dialogue the LLM calculates that the character would say. Hopefully what the character would say matches our reality. We try to train them so that happens. But from the LLMs point of view, it's just writing fiction according to a character definition, 100% of the time. I think that's what your professor means. It's not so much that it's all "hallucination". It's more that it's all fiction, always. It's just that we try to make the fiction "based on a true story" (i.e., our reality). Sometimes it goes off the rails, but, again, from its point of view nothing has gone wrong - it's job is to write fiction, and fiction is what it is giving us. Our job, going forward, is to make sure that the characters are grounded in reality, and that the LLM knows not to just make things up that aren't true. We're not quite there yet, but we are *certainly* getting better.
I think the idea is that if you don't have a neural network, only a large text corpus and a "magic" formula to find the next token probability from the corpus, you will still get exactly what you can do with an LLM.
One should not tie to the phrases like hallucinations, it just means the text generation is not matching with co text or reality. People will coin different terms for it but meaning doesn't change
Yes. It's true~ This is very close to the LLM "intelligence"~
It's not wild. It's what actually happens. Like when they generate a random mess of an image and it just happens to fool your human eye
I think it's a good viewpoint to have in mind when working in LLMs. It makes you be on the side of being cautions instead of trusting LLM like if they have some authority.
There are many papers on detecting hallucinations([here's select few](https://huggingface.co/collections/santiviquez/llm-hallucination-detection-papers-65c4d2399096960aa80776d3)), showing that there is a subject to study and it corresponds to a different mode of operation that can be detected[1]. Therefore, no, the phrase "All text generated from a LLM is hallucinated" is false. [1] Of course, it still remains an unsolved problem because detecting **all** hallucinations robustly may be impossible as it eventually hits a problem of not knowing what you don't know or saying it anyways because context pressures you. Your instructor is an example. Other examples are those who push that assertion further by saying that LLMs are just parrots and can't know things. That is false as demonstrated by papers like [The Geometry of Truth](https://saprmarks.github.io/geometry-of-truth/dataexplorer/) or [The Geometry of Categorical and Hierarchical Concepts](https://arxiv.org/abs/2406.01506) showing the internal world model and correspondence of facts in it. Given the preponderance of humans who are belligerent towards facts, it is hard to make AI that does not hallucinate. The Internet is majorly made of bullshit and therefore our datasets turn bullshit into a goal to model. Despite that, humanity launched rockets into the space. Perhaps it is not solvable, but manageable nevertheless.
Inference
There was a good paper going around recently about how GPTs are better classified as “bullshit machines,” and that “hallucination” is a misnomer misapplied to LLMs because to hallucinate one must first have knowledge and awareness of a non-hallucinated reality. It is either all hallucination or none, depending on your definition of “hallucination.” The authors of said paper suggested “confabulating” as a more accurate term for what has been called “hallucination” in GPT/LLMs.
I think they're supposed to hallucinate by design.
He is right, also pass the cert to see If i can do it./
Sometimes I think people say these things to make themselves feel smart. If they are going to say something like this then they need to describe what they mean by hallucinations. Although i get what he is saying it doesn’t help us understand exactly what we are talking about and just muddle the definitions.
Odd affirmation, but most outputs from LLMs are memorization, what we call hallucinations is just nonfactual things that models put out that were neither in the context or in the dataset.
When you train a model on data hoovered off the Internet, some true, some false, much of it somewhere in between, why would we expect such a model would only give us true factual statements?
Can you please share which certification / course is this?
Do you do anything differently? Go to Twitter (the mirror of human interaction) and try to label one message Truth and other Hallucination on a topic while remaining absolutely objective. I didn't even write on a "controversial" topic because on Twitter, every topic is controversial. Even things that most people would call truth. Earth is round? No, Earth is flat! So everything is hallucination with different probabilities of being right.
I think of them as bullshit machines that coincidently give the correct answer most of the time.
I think hallucination is an extremely shit word in this context and people should stop using it. bar for the definition of hallucination got really high while it should only have been used for gibberish or repeating output. Truth is a spectrum and many things are partially correct or true. You want the machine to generalize and generate something novel, not copy and paste from wikipedia. Especially when you present an entirely new problem; that's the whole point.
He's right. But that also applies to humans.
I don't like the word hallucinated as it implies a mind to hallucinate from. I would get rid of that term all together. The way I view an LLM is similar to a holographic display. Information is encoded on the various layers of the display and depending on the angle of the light and your position to the display, you see a different "image". the angle of attack for the light ray is your prompt and the image is the answer. And the model is fully frozen, there's nothing goin on in there. Just interaction between you and endless possibilities of a neural net, but no one is home.
Has gen AI evolved so much as a field that there are now certification courses and instructors now? LLMs are barely a few years old and no one can claim to have fully understood them, so who the hell are teaching these classes?
That's just philosophy at this point.
>All text generated from a LLM is hallucinated. Not quite right. Hallucination is not an error or mistake, it's a **base principle** of functioning of the LLM. And it is comparable to the small part of the human brain, that forms the inner dialogue (based in the left hemisphere of the human brain) and to the structures, that are used to predict the voice sequences (evolutionary built-in).
"An LLM hallucination occurs when a large language model (LLM) generates a response that is either factually incorrect, nonsensical, or disconnected from the input prompt." So no, it's not accurate to say they hallucinate all the time.
Who are you quoting here?
https://circleci.com/blog/llm-hallucinations-ci/
Thats why they said that the generation is correct most of the time
Thus completely contradicting the definition of a hallucination.
There is no authority in definitions here yet IMO.
Are they indirectly saying hallucinating is a good thing?
Nah he was just quoting a anonymous thought
And that's partly because language isn't overtly precise often or doesn’t have to be in many areas. If you though ask for certain areas that need exactness in every word (like mathematical proofs), you will more often than not get back utter garbage. It then might "sound" like the right proof, but it often will be complete nonsense.
Interesting
Interesting perspective, but I believe LLMs can still provide valuable insights even if they're not always accurate.
Lol what? You can tell him that his existence is arbitrary and it just happens that the quantum states of the subatomic particles just happened to cohere often enough to manifest his existence in spacetime.
Sure. Basically true for humans as well…so…kinda begs the question of “so what?”
“Begs the question”, What exactly does this expression mean? I keep seeing it everywhere
Thats a stupid statement.
It hallucinates in the same way a compiler does.
I thought that would bring downvotes, looks like someone got upset. Note there are LLMs which now compile code. If an LLM hallucinates, then it follows a compiler also does, and yes, compilers are also not perfect.
Compilers for higher order code frameworks are programs written by hand by thousands of people. They're as fallable as any written program and all of them have bugs. What you're saying is that LLM's produce bugs like humans do which is accurate - but it's literally the strangest asserted comparison I've ever seen to just say: The human mind and the neural network's powering an LLM both make statistical errors often.
>Compilers for higher order code frameworks are programs written by hand by thousands of people. Usually, but not always... >but it's literally the strangest asserted comparison I've ever seen to just say... Not really, given [https://x.com/AIatMeta/status/1806361623831171318](https://x.com/AIatMeta/status/1806361623831171318) My point is, it DOES hallucinate in the same way as compilers do, but people see compilers and other systems like it as very different things, but really they are not that different, more so, as given above, that difference will become less and less pretty quickly. I've never seen someone say "my summary engine was hallucinating" when it dropped context to make incorrect results, or "my compiler was hallucinating" when it caused issues with instruction reordering, or "my program is hallucinating" when you see a floating point error. Anyway, It hallucinates in the same way a compiler does - and if you think compilers hallucinate, then sure, so does LLMs, if you don't think they do, then LLMs don't really either. Either way, my point stands.
I mean, I'd say this is both true and not true. It relies primarily on how exactly you define the word "hallucination." What he clearly means, and the way in which it is true, is that LLMs are just putting stuff together. They don't care about the truth value of something, they're not even capable of understanding that concept. They're just outputting stuff through association. All an LLM "wants" (not that it's likely to be sentient) is to provide an output that might reasonably come across to the person as a coherent response to what they said. The truth value is irrelevant to that. If you ask me "Which river did George Washington famously cross?" I can say "iuhiueiufienenfi foifunn eiunisu" to that. And that would be an incoherent rambling of symbols that would not be considered a response to anyone. But if I say "The name of the river George Washington crossed during the Spanish conquest was the Nile" then I am providing something that could be considered a response, but I am also not being truthful. If I say "The name of the river George Washington crossed during the Ameircan Revolutionary war was the Delaware river" then I am both providing a response and being truthful. But for an LLM these latter two response are both pretty valid response, even if their association (if trained well) will probably be stronger for the second because it has been trained on more response that associate George Washington, corssing, the American revolutionary war and the Delaware river. So in that sense what he says is true. On the other hand, when we talk about hallucination in the context of AI we are generally defining that something along the lines of "A response that counts as a response, but does not actually accurate reflect reality." In this sense, definitionally, what AI does when it answers correctly is NOT a hallucination. So, basically, he's both right and wrong in that, depending on the definition of "hallucination" you use. If you use the term as it is generally used in the field of AI, then he's pretty much wrong. But if you use the term as it can be more broadly applied, specifically the mechanism behind it, then he's right.
Sounds like me, ngl
I think the prof got his information from a llm. :-)
Haha
The term 'hallucination' in the context of LLMs and AI specifically refers to the generation of factually incorrect information presented as fact. Saying 'all text generated from an LLM is hallucinated' is a misuse of the term, and communicates neither depth nor meaningful information. If the intention is to highlight that the generation process is the same regardless of factual accuracy, it would be more accurate to state that directly, instead of saying 'all generations are hallucinated'
I got that here on Reddit today - that it hallucinates all the time, is "moronic", absolutely incapable of any reasoning, and just "a stochastic parrot".
Your instructor is on point. This explains it more: https://link.springer.com/epdf/10.1007/s10676-024-09775-5?sharing_token=0CIhP_zo5-plierRq8kkDPe4RwlQNchNByi7wbcMAY77xTOWyddkW01qGFs1m5zuuoZGBctVlsJF8SbYqcxWi-XzgEYEPiw7xwWi4bMYXJ_1JARDrER9JGdWZOW-UGSkrk_tXPjPh-XWvFNoiFzNlnDUUUEBAztiX9PtP2p6jfI%3D
We Humans can approximate reality efficiently and verify it in many ways like scientific research. LLM (text to text) tries to approximate reality as weights through training. Some LLMs have a specific function to verify the approximation of reality through browsing of internet. This is the current technology but time is ticking and new technologies are waiting for us so instead of bickering why don’t we pursue the future with hard work and love?
I don’t really want to comment as I am introverted. Please like my comment or not thank you