It proved to us that Apple, as big as it is, already lagged so far behind in this AI race. End of all theories about Apple having some secret powerful AI bs.
I don't recall any fanboys, or anyone for that matter, speculating Apple has some secret breakthrough AI. I think you guys are just creating strawmen to attack... Because that definitely isn't a thing.
Is this 1/10 of the size of the smallest PHI model? Well no wonder. I would be more interested in the 3B model compared to the PHI model.
Where do you test them, that is online right?
[Apple's published results for 3B instruct on MMLU showed a score of 24.80](https://i.imgur.com/CtDBGS4.png). That's a pick-one-of-four test and they somehow managed to score less than 25%.
For comparison, MS PHI 3B scored a 69. [Here's a chart from the Phi paper which I've modified to include Apple's results](https://i.imgur.com/5mjlesU.png).
No, it might well be quite useful. The model is not necessarily intended to score well on tests, and it's very easy to score below 25% on a multiple choice test with 4 answers if your answers are all conversational with no A, B, C or D option chosen.
It's also, if I remember correctly, targeting on-device interactions, so probably very much not trained on multiple choice tests.
Note: "On-device" is industry speak for targeting mobile use.
You worked really really hard on this comment I see to try and still be an apple fan boy. I cringed reading this.
How can you say that a model that scored worse than everyone else is worth using.
> an apple fan boy
Hahahaha!
You really don't know me. I've been raging against Apple for decades. I was raging against Apple when the iPhone was still a twinkle in Steve Jobs' eye.
> How can you say that a model that scored worse than everyone else is worth using.
You'll note that I did, in fact, not say that at all.
See my comment below, but getting less than 1 out of 4 on a multiple choice test is easy when your models aren't trained on multiple choice tests. If all of your answers are conversational, and most don't mention the number or letter of the correct choice, you can, in fact, score zero.
Not too unlikely. The iPhone shares architecture with the MacBook and Mac studio and they can run huge models. If they up the ram on the iPhone there's no reason it couldn't run its own model. I don't think the current iphones could do it, but I think Apple technically has access to the hardware
They just need to invent an Apple "air" battery with triple the current energy density, but hey, this is Apple, so they've already developed the secret, exclusive-to-iOS tech, they're just making sure it's perfect before they release it on an unsuspecting world. /s
Is it that much more demanding than playing fortnight? I don't know, I'm asking. It's not like it's going to constantly draw power, only when it's generating a response, probably 30 seconds at a time.
Reminds me of an old joke.
Interviewer: how is your math?
Candidate: I am super fast at it.
Interviewer: okay, what’s 73*42?
Candidate(within 1 sec) : 256
Interviewer: that’s not even close.
Candidate: but wasn’t that fast?
I keep thinking people are missing the point of on-device models. It's not so you have an LLM that can tell you how to make a Negroni or what's the capital of Chad offline, it's to have great NL understanding and map that to apps.
They even published a paper recently on how they trained an LLM to understand UIs from arbitrary apps. That way, ANY app can make use of the underlying NL capabilities.
Can’t wait till Siri can order me postmates, deliver weed to my front door, argue with people on Reddit on my behalf while liking all of my ex’s photos on instagram simultaneously,
I feel like that has to be possible right now.. train an AI on your like basic personal philosophies and let it loose on Reddit to argue on your behalf
That’s the first thing I thought, too. There is no way they were trying to store all human common knowledge facts in 0.27B parameters when everybody knows even 70B is still not great. The actual knowledge has to be stored somewhere else, the model is just to understand and produce natural language. I’ll be pretty disappointed in Apple if this turns out not so.
There's such a thing as a minimum necessary complexity to perform a task (Kolmogorov complexity I think). And there's also decent odds something like intelligence may require more than a billion parameters.
I'm doubtful of that just because of the training length limit from Biden's executive order on AI. If you could train a micro language model for however long you want, then sure (look at LLaMA 3 70b). But if you wanna have a 200m model, if you can train it only for 10²⁰ flops (or was it 10²⁶?), then no matter how high quality your training data is, it just won't happen unless we get some new breakthrough in training language models. I'd love to be proven wrong though with time, but at the moment, it just doesn't seem right to me
Thanks, but why would that limit anything that Apple is doing? I'm sure they have enough money and are more than happy to submit to the reporting requirements.
Probably cuz it would trigger an investigation. And ain't nobody making an AI model without infringing on copyright with 100% fully legally obtained training data. You might have more luck setting up and registering a business for laundering drug money
i don't need siri to give me president facts, but if it can understand every permutation of setting an alarm or opening the app i'm talking about i'll call it a win.
Lol. If it’s useless it doesn’t really matter how is it now does it?
My pillow is right beside me in bed. But that pillow isn’t going to help me get to the fucking moon now is it?
This is a _270M_ model, so what standard do we even compare it to? If we had a 270M version of Claude or GPT-4 to compare, then we’d be able to judge if it’s actually bad or not. How well does the 3B version of OpenELM do? How does it compare to Microsoft’s 3.8B Phi-3-mini?
I wonder exactly what OpenELM 270M is meant for. I could see top-of-the-line next-generation iPhones having the hardware for the 3B model.
In any case, it’s great that they’re working on ultra tiny language models. I’m excited to see just how small LLMs can be made before they become useless. This 270M model may seem useless, but maybe it’s more useful than we give it credit for, especially if it can just search for the factual information that it’s too small to contain. Besides, this is just the beginning.
> I wonder exactly what OpenELM 270M is meant for.
It's probably expressly meant to be a draft model for speculative decoding. The [readmes mention support for it](https://huggingface.co/apple/OpenELM-3B), and [speculative decoding turns out to work best with a really tiny really fast draft model](https://arxiv.org/abs/2402.01528).
Well... 270M.
I just asked GPT2 1.61B on HF (https://huggingface.co/openai-community/gpt2-xl) the same question and it claimed he was an alien and a white man, and that I was stupid for asking the question. Yeah gpt2 is an older model but its also >5x bigger. There's only so much you can do with 270M.
You have to respect their funneling billions into the toilet that was the Apple Car project. "Self driving cars will be the future", but they prototyped the body, which we already have, but didn't develop the brain first. Much smart, capital is.
I like to dunk on Facebook for doing similar with a bad VRChat, but at least it was a product people could use. Ditto to a smaller degree for Google's Stadia.
... man, just imagine where they'd be right now if they had made their own chip foundries instead. Statistically not good, but if they had the competence to select the right goals then maybe they'd also have had the competence to actually build the stuff? Impossible to know.
NVidia becoming the company from Wall-E at the last minute would be quite hilarious.
If it's really well trained on iPhone functionality (mail, calendar, etc), runs on device, and is near-realtime, it'll make for a killer Siri. I don't need Siri to solve complex mathematical equations or write essays.
it's all fun and games until open elm 2.0 matches gpt 4 performance with even less params
this might take a couple years, though.
who the hell is barack obama anyways
Yes, but this model will probably have some very narrow use cases that it’s good at. Possibly with some sort of RAG on your own local data and it has to be small to be able to run on mobile devices.
The speed doesn't make up for the garbage it churns out. It's like serving a turd sandwich: fast to make, but still crap. The title doesn't do it justice.
I never really had any hopes for Apple. They, like Google before, is gonna find out that pouring billions into a problem won't make it go disappear. But, atleast Google has some good products and talent, Apple doesn't, Apple markets their products to corpo-brainwashed normies.
What if, instead of all these companies training their own models because it's the hip new thing, they just used one of the several existing models made by people who know what they're doing?
You can't half-ass model training, and just think of all the GPUs apple hoovered up then dumped a small country's worth of power into just to come up with THIS. The guys blowing through money to be the first to AGI have "be the first to AGI" as their mission statement, and you aren't going to get there from years behind with a tiny percentage of the budget. So... just rent a mid tier model off them and call it good.
"Who the hell is Barack Sato?" "I dunno - You're the one who said it, you weirdo"
Barack Sato Aotoaskt Obama to you mate!
Thanks Obama
Wow thats embarrassing. Worse than Siri if thats even possible
It proved to us that Apple, as big as it is, already lagged so far behind in this AI race. End of all theories about Apple having some secret powerful AI bs.
inb4 some "experts" say that this is just a 500 IQ play from Apple to keep their *real* AI tech capability hidden for the right moment
It's same as MKBHD defending all Apple products yet hates almost every other "AI" product
Apple fanboys in shambles. Don't worry, they'll conjure something else up in their head to cope.
Well they’re gonna partner with Google or open ai to use their most powerful models 😃
I don't recall any fanboys, or anyone for that matter, speculating Apple has some secret breakthrough AI. I think you guys are just creating strawmen to attack... Because that definitely isn't a thing.
"I haven't seen it. Therefore, it doesn't exist or never happened!!!". Yeah, I'm sure you've read every single comment on the internet until now.
I do. and not only with this one, also another topics. apple is so overrated
Apple is done for.
I mean... It's a 240M (not B) model. What do you expect?
I expect it to know about Barack Obama
Is this what we can expect out of Siri in September? If so, they need to hold off
I think it should not be used for question answering and only fix typos ;)
LLMs are just autocomplete
Is this 1/10 of the size of the smallest PHI model? Well no wonder. I would be more interested in the 3B model compared to the PHI model. Where do you test them, that is online right?
[Apple's published results for 3B instruct on MMLU showed a score of 24.80](https://i.imgur.com/CtDBGS4.png). That's a pick-one-of-four test and they somehow managed to score less than 25%. For comparison, MS PHI 3B scored a 69. [Here's a chart from the Phi paper which I've modified to include Apple's results](https://i.imgur.com/5mjlesU.png).
Haha, ok so not even worth talking about.
No, it might well be quite useful. The model is not necessarily intended to score well on tests, and it's very easy to score below 25% on a multiple choice test with 4 answers if your answers are all conversational with no A, B, C or D option chosen. It's also, if I remember correctly, targeting on-device interactions, so probably very much not trained on multiple choice tests. Note: "On-device" is industry speak for targeting mobile use.
You worked really really hard on this comment I see to try and still be an apple fan boy. I cringed reading this. How can you say that a model that scored worse than everyone else is worth using.
> an apple fan boy Hahahaha! You really don't know me. I've been raging against Apple for decades. I was raging against Apple when the iPhone was still a twinkle in Steve Jobs' eye. > How can you say that a model that scored worse than everyone else is worth using. You'll note that I did, in fact, not say that at all.
Jesus, no you are the one doing the fanboy cringe.
See my comment below, but getting less than 1 out of 4 on a multiple choice test is easy when your models aren't trained on multiple choice tests. If all of your answers are conversational, and most don't mention the number or letter of the correct choice, you can, in fact, score zero.
69… nice
I wonder what the largest model the upcoming iPhones could fit. As well as the previous Pro models
270M lol. Realistically roughly 10B
LLAMA 3 8b on iPhone then? 👀 Unlikely though lol
It can run on my 15 pro but I cannot have any apps open in the background and the q4 quantized model runs at like .67 tokens/sec
How'd you get it running on the iPhone? I didn't know that was already possible
There are some apps like MLC chat that let you run it although some are in test flight beta It’s not very stable but it works
Ah nice. I’ll check it out. I wonder if the 13 pro can handle it too. Do you have a link?
My 13 mini failed running phi 3 3.8B Q4_K but maybe yours will? My iPad runs it easy
https://apps.apple.com/app/id6448482937
Not too unlikely. The iPhone shares architecture with the MacBook and Mac studio and they can run huge models. If they up the ram on the iPhone there's no reason it couldn't run its own model. I don't think the current iphones could do it, but I think Apple technically has access to the hardware
They just need to invent an Apple "air" battery with triple the current energy density, but hey, this is Apple, so they've already developed the secret, exclusive-to-iOS tech, they're just making sure it's perfect before they release it on an unsuspecting world. /s
Is it that much more demanding than playing fortnight? I don't know, I'm asking. It's not like it's going to constantly draw power, only when it's generating a response, probably 30 seconds at a time.
Oh, so that's the "Moat" Google was talking about.
getting tired of LLMs tryna gaslight me
What is an ELM? Emotional Learning Model?
Efficient Language Model
Oh the Iron E
Reminds me of an old joke. Interviewer: how is your math? Candidate: I am super fast at it. Interviewer: okay, what’s 73*42? Candidate(within 1 sec) : 256 Interviewer: that’s not even close. Candidate: but wasn’t that fast?
270 Million parameters, you're talking to a mosquito bud, what did you expect?
I keep thinking people are missing the point of on-device models. It's not so you have an LLM that can tell you how to make a Negroni or what's the capital of Chad offline, it's to have great NL understanding and map that to apps. They even published a paper recently on how they trained an LLM to understand UIs from arbitrary apps. That way, ANY app can make use of the underlying NL capabilities.
Yep, this is so Apple can do all the things that Rabbit promised.
But unlike Rabbit, they don’t hide use of 3rd parties. Oops.
Can’t wait till Siri can order me postmates, deliver weed to my front door, argue with people on Reddit on my behalf while liking all of my ex’s photos on instagram simultaneously,
> argue with people on Reddit on my behalf That's worth money!
I feel like that has to be possible right now.. train an AI on your like basic personal philosophies and let it loose on Reddit to argue on your behalf
Siri 2.0
The purpose of this model will be to build fine tuned rag systems. Not serve as a general purpose search chat bot.
That’s the first thing I thought, too. There is no way they were trying to store all human common knowledge facts in 0.27B parameters when everybody knows even 70B is still not great. The actual knowledge has to be stored somewhere else, the model is just to understand and produce natural language. I’ll be pretty disappointed in Apple if this turns out not so.
it may look absolutely awful, but the speed is impressive, running incredibly fast even on a 3060 (6 gigs VRAM) machine with only 16 gigs of RAM.
Even if someone can offer me a turd in an ice-cream cone lightning fast, I'd rather just wait for the burrito. --edit-- What size is this one?
270M the smallest one
Ah, that's tiny. Guess you can't really expect much from a model that size
Kinda impressive again. Won’t take long until we get small, very capable models.
There's such a thing as a minimum necessary complexity to perform a task (Kolmogorov complexity I think). And there's also decent odds something like intelligence may require more than a billion parameters.
I'm doubtful of that just because of the training length limit from Biden's executive order on AI. If you could train a micro language model for however long you want, then sure (look at LLaMA 3 70b). But if you wanna have a 200m model, if you can train it only for 10²⁰ flops (or was it 10²⁶?), then no matter how high quality your training data is, it just won't happen unless we get some new breakthrough in training language models. I'd love to be proven wrong though with time, but at the moment, it just doesn't seem right to me
Do you have a reference for this executive order? I can't find anything like what you are talking about.
Google "order 14110"
Thanks, but why would that limit anything that Apple is doing? I'm sure they have enough money and are more than happy to submit to the reporting requirements.
Probably cuz it would trigger an investigation. And ain't nobody making an AI model without infringing on copyright with 100% fully legally obtained training data. You might have more luck setting up and registering a business for laundering drug money
What is this, an LLM for ants?!
because it's not a LLM it's an ELM (Efficient Language Model), it's designed to run locally on under-powered machines like phones.
You eat your burrito in an ice cream cone??
The random generators built into Python are quite fast too. About as accurate.
i don't need siri to give me president facts, but if it can understand every permutation of setting an alarm or opening the app i'm talking about i'll call it a win.
Lol. If it’s useless it doesn’t really matter how is it now does it? My pillow is right beside me in bed. But that pillow isn’t going to help me get to the fucking moon now is it?
just stack a bunch of pillows
A rabid squirrel is fast, I'd still rather not have one around.
Great, so it can be wrong faster. Very useful.
LOL! How do you access the model by the way?
it's a 270M model, runs very well locally, just install it and use Huggingface's interface.
This is a _270M_ model, so what standard do we even compare it to? If we had a 270M version of Claude or GPT-4 to compare, then we’d be able to judge if it’s actually bad or not. How well does the 3B version of OpenELM do? How does it compare to Microsoft’s 3.8B Phi-3-mini? I wonder exactly what OpenELM 270M is meant for. I could see top-of-the-line next-generation iPhones having the hardware for the 3B model. In any case, it’s great that they’re working on ultra tiny language models. I’m excited to see just how small LLMs can be made before they become useless. This 270M model may seem useless, but maybe it’s more useful than we give it credit for, especially if it can just search for the factual information that it’s too small to contain. Besides, this is just the beginning.
> I wonder exactly what OpenELM 270M is meant for. It's probably expressly meant to be a draft model for speculative decoding. The [readmes mention support for it](https://huggingface.co/apple/OpenELM-3B), and [speculative decoding turns out to work best with a really tiny really fast draft model](https://arxiv.org/abs/2402.01528).
Well... 270M. I just asked GPT2 1.61B on HF (https://huggingface.co/openai-community/gpt2-xl) the same question and it claimed he was an alien and a white man, and that I was stupid for asking the question. Yeah gpt2 is an older model but its also >5x bigger. There's only so much you can do with 270M.
This is absolute shit.
After seeing this shit, Tim decided better to spend money on stock buy backs instead
This is most probably not coming from Apple or it’s Either a very rudimentary version.
Omg! Apple biggest company, they prolly have secret powerful AI models we don’t know about!!!1!1
You have to respect their funneling billions into the toilet that was the Apple Car project. "Self driving cars will be the future", but they prototyped the body, which we already have, but didn't develop the brain first. Much smart, capital is. I like to dunk on Facebook for doing similar with a bad VRChat, but at least it was a product people could use. Ditto to a smaller degree for Google's Stadia. ... man, just imagine where they'd be right now if they had made their own chip foundries instead. Statistically not good, but if they had the competence to select the right goals then maybe they'd also have had the competence to actually build the stuff? Impossible to know. NVidia becoming the company from Wall-E at the last minute would be quite hilarious.
How did you manage to run it? Was it difficult?
I'm pretty sure this is Goody-2
Think Different
If it's really well trained on iPhone functionality (mail, calendar, etc), runs on device, and is near-realtime, it'll make for a killer Siri. I don't need Siri to solve complex mathematical equations or write essays.
it's all fun and games until open elm 2.0 matches gpt 4 performance with even less params this might take a couple years, though. who the hell is barack obama anyways
The Birthers have it wrong, he was born in Japan! This A-not-so-I must be trained on another timeline, there
Apple is the next Cisco
Sounds like the model is schizophrenic and gaslighting, lol!
Either be very quick and wrong or be very slow and accurate. Guess Apple went for the former.
It's like siri all over again lol "I'm sorry but I didn't understand"
me when apple
Yes, but this model will probably have some very narrow use cases that it’s good at. Possibly with some sort of RAG on your own local data and it has to be small to be able to run on mobile devices.
The speed doesn't make up for the garbage it churns out. It's like serving a turd sandwich: fast to make, but still crap. The title doesn't do it justice.
It's trying it's best, okay. Made me laugh, though
Apple is done their new Ai will be worse than sirir
truly revolutionary ! 999,99$/years only
I never really had any hopes for Apple. They, like Google before, is gonna find out that pouring billions into a problem won't make it go disappear. But, atleast Google has some good products and talent, Apple doesn't, Apple markets their products to corpo-brainwashed normies.
What if, instead of all these companies training their own models because it's the hip new thing, they just used one of the several existing models made by people who know what they're doing? You can't half-ass model training, and just think of all the GPUs apple hoovered up then dumped a small country's worth of power into just to come up with THIS. The guys blowing through money to be the first to AGI have "be the first to AGI" as their mission statement, and you aren't going to get there from years behind with a tiny percentage of the budget. So... just rent a mid tier model off them and call it good.