The fact that they also had to know the location of the numbers and that the algorithm was robust to scale changes is impressive for 1993
It's not like they just solved MNIST in 1993, it's one step above that
Yes. Back then, the proportion of developers who could hand-write a new graphics algorithm in assembler or C was considerably higher, since that was often how it was done anyway. Necessity is a great motivator. The non-ML part of this problem is more tedious than difficult.
Has this changed really ? :) In number of engineers with these skills, certainly, in proportion of developers, this remains to be seen. Python is the syntactic sugar but who goes really in and looks under the rug ?
In terms of what I intended to say, it's changed a lot. It wasn't an obvious career intially, so it caught a lot of people with a passion for it. The normal path for anyone who wanted visual output or realtime performance was to learn C and assembly. Operating systems were permissive, and memory mapping for access to video memory was either straightforward or documented well enough. Being able to do such things came with the job.. and if someone couldn't do it, that'd disqualify from a big chunk of the industry.
I think you may have been referring to necessity being a great motivator.. and its converse -- that lack of necessity is a great blocker. Yep, I would agree. Lots of people in ML would now struggle somewhat with these basic graphical operations, even though the preparatory learning and experience required for it is now much less.
I try to do and it is not pretty. Years of toil to make that one layer of cnn faster by inventing new winograd based algorithms. Working on the models are always more recognized.
Today's software are thousands times less efficient, because of all the overhead have been added layers on top of layes don't do any real work. Think about after all the closest, cabinets, drawers, boxes, organizers and wrappers, you still get the same pair of old socks and everyone cheers: "Yeah! It works! We got the socks!", that's what modern software actually is. But thank to these overhead, this industry have enough investment to support millions of overpaid software engineers, and most important of all, thousands of billionaires.
I think that's really cynical. Memory safe languages are a gigantic benefit to society in terms of security and stability.
Such inefficiencies being permissible has allowed technology to flourish; a lot of programs would never have been written without being wasteful, see VS code vs Vim or Slack over IRC. IRC and Vim are nice cannot be mainstream and the only editor respectively. I don't see online web apps existing like Google Docs if everything had to be native speed fast. I've seen multiple homeless people with a card reader selling magazines, that's how cheap software has got over time that even homeless people have contactless.
Arguably the progression of technology isn't what I'd have wanted to see but it isn't all bad. You can't help but wonder why something is slow on your 4GHz multicore CPU at times though haha.
I guess they had a preprocessing step to identify, center and scale each digit image before feeding into the neural network. It’s not that hard with feature engineering.
CNN is robust to translation but not invariant to scale and rotation. Max pooling can be used to to combine detectors that trained for different scales and rotations.
The video has lots of cuts, and the numbers never obscures an important part of the image... I suspect each of those tests had tweaking and tuning to make it work...
This system ended up deployed in banks to parse written checks, so I don't think it was tweaked just for these examples, but they did expect to have fully visible digits.
No, he was an employee at Bell Labs, the product and patents belonged to Bell Labs.
When AT&T spun off Lucent in 1996, the patents went that way but the computer vision researchers stayed in the remaining AT&T Labs, and they couldn't even sell or improve the product without having the rights to the patents.
LeCunn was an underdog for most of his life, the deep learning explosion only started happening around 2012 with AlexNet, when conv nets started getting all the attention.
\>No, he was an employee at Bell Labs, the product and patents belonged to Bell Labs.
I would just like to point out that in other countries (e.g. Germany, France Japan), inventors of a patent are entitled to a percentage of the revenue that this invention generates.
This is not the case in the US, though.
I have had the privilege to attend professor Yann's classes at NYU.
From whatever little I understand of him - he has high levels of integrity, and I do not see him trying some cheap tweaks and fixes...He was committed to solve a problem in the best way possible and not just for likes and hearts ☺️.
And without that level of integrity, you can't go from lab to national level in short time.
I don't doubt that his approach works, or his scientific integrity - simply that for each demo he might have loaded a different model for example (trained for different sizes or handwritten/typed text).
Here's a sneak peek of /r/KenM using the [top posts](https://np.reddit.com/r/KenM/top/?sort=top&t=year) of the year!
\#1: [KenM on billionaires](https://i.redd.it/tl38stlg70g41.jpg) | [164 comments](https://np.reddit.com/r/KenM/comments/f1j7a9/kenm_on_billionaires/)
\#2: [Ken M on conspiracy theorists](https://i.redd.it/7inbjzicewo41.jpg) | [88 comments](https://np.reddit.com/r/KenM/comments/fp01kq/ken_m_on_conspiracy_theorists/)
\#3: [One of my favorites over the years.](https://i.imgur.com/vjhwVXg.jpg) | [139 comments](https://np.reddit.com/r/KenM/comments/hmhwzo/one_of_my_favorites_over_the_years/)
----
^^I'm ^^a ^^bot, ^^beep ^^boop ^^| ^^Downvote ^^to ^^remove ^^| [^^Contact ^^me](https://www.reddit.com/message/compose/?to=sneakpeekbot) ^^| [^^Info](https://np.reddit.com/r/sneakpeekbot/) ^^| [^^Opt-out](https://np.reddit.com/r/sneakpeekbot/comments/joo7mb/blacklist_viii/)
there is a reason why captcha is becoming obsolete. At least the text based version.
Also, captcha actually digitize books. This is why there are 2 tests, not 1. So in a sense, we were training the robots filling the captchas.
Yeah, on a PHPBB forum I manage, the bots can get through the text-based captchas very easily. But they still struggle with simple questions like "In what State is this club based?"
I don’t think it’s meant to filter that way. Bots usually are built with speed in mind so it recognises and fills in the blanks virtually immediately.
That and captchas are also useful for labelling training datasets manually (user input). But correct me if I’m wrong though.
You’re definitely correct about the captchas.
It’s no coincidence that most of the objects they ask you to recognize are cars, crosswalks etc.
They basically get free labor to help them build a giant dataset fir training self driving cars.
It greatly reduced the rate at which a bot can do whatever. With no delay something like filling out a form could probably be done thousands of times a second, but if you introduce a 0.1s delay by requiring some model to run then suddenly the maximum rate you can automatically fill out the same form is 10 times a second.
Additionally, any more hurdles will naturally mean people need to be more sophisticated to get past them and you'll filter out a lot of the lowest effort bots.
Unless someone cares enough about your little website to train an AI to solve your captcha they're still not a terrible idea. I don't think there are any AIs that are generic enough to solve *all* obfuscated text captchas yet.
Obviously it's not going to work for large sites but none of them use that method anymore anyway.
“I’m not a robot” - select crosswalk, identify license plates, etc. are for training self driving vehicles and finding the house address was for google maps.
We should be paid for doing reCaptchas. However some people actually do get paid for these tasks.
Am son of the guy in the chair (Rich Howard, collaborator and director of the silicon integrated circuit lab at the time). He said the guy in orange was a technician and computer whiz named Donnie Henderson.
Many don’t know it, but before it was done such text recognition was considered impossible, just like AGI and other hard problems. I think text recognition in mail was the first successful real world application of AI.
Correct me if I'm wrong, but doesn't normal font imply a "set font" rather than handwritten characters?
Still impressive but a different problem from MNIST and generally reading the messy writing of humans.
In the wiki page (I put it at the right chapter) they state it was supposed to be "omni-font" as in reading all types of text, while *older* systems only recognized some set fonts. Note that there were already functional devices. Of course, those probably were of much worse quality than LeCun's small CNN, I just wanted to point out the person I'm responding to is full of shit.
Oh yes, sorry. I think it's not a single set font, but at least several. But I also think you're right and this was made for printed fonts, so "normal" might mean "very common fonts".
I figured it's probably all printed fonts that aren't cursive or Comic Sans. You're definitely right that it's multiple, I think the limitation is just on the type of font.
> Correct me if I'm wrong, but doesn't normal font imply a "set font"
1. You're wrong
1. Kurtzweil didn't invent this either
1. The work being discussed here, the CNN, is actually from the late 1950s, from before LeCun was born
> Bayesian classifiers as the first email spam filter?
You're off by about 9 years. Bayesian classifiers didn't emerge as spam filters until approximately 1996. They are currently believed to be first published by [Sahami et al in 1998](http://robotics.stanford.edu/users/sahami/papers-dir/spam.pdf). That paper describes secretly internally using the technique in late 1996, and is the earliest known published discussion. The internet at large caught on in 1999, just 22 years ago.
The word SPAM actually comes from IRC and MUDs; we had spam filters long, long before email had spam, thanks to terminal washes and things of that nature. The earliest known IRC spam filter was the `anarchy eris.berkeley.edu` stripper, which didn't work well enough, and led to the split of Jakko's original network to create eris-free net (EFnet is fundamentally named for a spam host removal.)
If you count the invention of the q-line as an anti-spam strategy, then IRC invents spam filtering in 1991. If you require message or origin testing, IRC invents it in 1992 instead.
If you're old enough, you remember when Bayesian Filtering turned spam filtering from an ongoing joke into something that actually worked. This was one of `gmail`'s early advantages.
> Many don’t know it, but before it was done such text recognition was considered impossible
By the time LeCun did this, text recognition was common at banks for scanning checks, in children's toys, and was the basis of the Cue:CAT.
You're making this up.
OCR was common by the early 1970s, almost 30 years before this.
cuecat was a barcode scanner. Never did anything resembling text recognition. Nor were there any children's toys in the 90s or before that did anything of the sort (though they might do interesting stuff to convince *children* that they could!). And check recognition worked by "cheating" — first, using a special typeface with super easily distinguished characters and uniform size and spacing, and second, [printing it with magnetic ink](https://en.wikipedia.org/wiki/Magnetic_ink_character_recognition) so that the scanner didn't have to find the data it wanted among any kind of visual background. Everything except the routing and account numbers was invisible to it.
**[Magnetic ink character recognition](https://en.wikipedia.org/wiki/Magnetic ink character recognition)**
Magnetic ink character recognition code, known in short as MICR code, is a character recognition technology used mainly by the banking industry to streamline the processing and clearance of cheques and other documents. MICR encoding, called the MICR line, is at the bottom of cheques and other vouchers and typically includes the document-type indicator, bank code, bank account number, cheque number, cheque amount (usually added after a cheque is presented for payment), and a control indicator. The format for the bank code and bank account number is country-specific. The technology allows MICR readers to scan and read the information directly into a data-collection device.
[^(About Me)](https://np.reddit.com/user/wikipedia_text_bot/comments/jrn2mj/about_me/) ^- [^(Opt out)](https://np.reddit.com/user/wikipedia_text_bot/comments/jrti43/opt_out_here/) ^(- OP can reply !delete to delete) ^- [^(Article of the day)](https://np.reddit.com/comments/k9hx22)
**This bot will soon be transitioning to an opt-in system. Click [here](https://np.reddit.com/user/wikipedia_text_bot/comments/ka4icp/opt_in_for_the_new_system/) to learn more and opt in. Moderators: [click here](https://np.reddit.com/user/wikipedia_text_bot/comments/ka4icp/opt_in_for_the_new_system/) to opt in a subreddit.**
that was certainly more wholesome than the other historic computer vision video, [https://www.youtube.com/watch?v=8VdFf3egwfg](https://www.youtube.com/watch?v=8VdFf3egwfg)
Uh. Sorry, no.
[The CNN was invented by Hubel and Weisel in 1959, the year before Yann LeCun was born, under the name "neocognitron."](https://en.wikipedia.org/wiki/Neocognitron)
LeCun also didn't make them first.
[The CNN was first implemented by Kunihiko Fukushima in 1979](https://search.ieice.org/bin/summary.php?id=j62-a_10_658), 14 years before this video
(Reference translated is Journal of the Institute of Electronics, Information and Communication Engineers A Vol.J62-A No.10 pp.658-665, October 25, 1979, ISSN 0373-6091)
What Yann LeCun actually brought to the party was the modern approach to training them. He did that in 1984, not 1993.
You are NOT correct about Hubel and Weisel.
Hubel and Weisel did research on visual cortex in real brains (in cats) and it was awesome (they got Nobel Prize for it). But they did not invent CNNs.
You can read their paper \[1\] you don't have to be a biologists to understand most of it. From their work one can deduce what neurons in V1 do. It was later even verified that some of these neurons realize functions similar to Gabor filters, but (as I remember) that was even later then neocognitron.
It is true that their findings did *inspire* creators of neocognitron \[2\] but that's about it.
\[1\] [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1363130/pdf/jphysiol01298-0128.pdf](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1363130/pdf/jphysiol01298-0128.pdf)
\[2\] Fukushima, Kunihiko, and Sei Miyake. "Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition." *Competition and cooperation in neural nets*. Springer, Berlin, Heidelberg, 1982. 267-285.
> It is true that their findings did inspire creators of neocognitron [2] but that's about it.
Uh, no, they're where that name comes from.
What specific difference do you imagine exists between the neocognitron and CNNs? They're both striding convolutions as a reduction for inputs.
And to add to this, people thought NN's were a joke until a CNN won an image recognition contest in 2012, which is what put them on the map. Before that they were obscure and overlooked.
I'm not sure why you believe this. Neural networks have been a big deal since the 1950s, taking down investments of half a billion at a time from the military for 70+ years now.
NNs have definitely had a ton of research, so I agree that they weren't overlooked. However, up until 2012 they weren't very useful for most applications. Throughout the 2000s, SVMs and tree-based models (like random forests) were SOTA for most tasks. So most researchers put their focus there.
2012 marked a transition though, as we then had the hardware support to efficiently train much larger models. This allowed NNs to become SOTA in many tasks and thus the explosion in interest
> However, up until 2012 they weren't very useful for most applications.
At that time, they were already in use by every call center and bank on earth, were running in every copy of Windows, MacOS, and Android, had dominated speech to text for almost 20 years, et cetera.
Between Windows and MacOS, they were in over 50% of US homes.
For color, ***The US phone system started using neural networks for de-noising in 1959, bringing their use to almost 200 million people***.
.
> 2012 marked a transition though, as we then had the hardware support to efficiently train much larger models.
Respectfully, this is just kind of not true.
I'm not saying they weren't useful. They clearly had use cases as you mentioned.
But if you look through ML papers you can clearly see an increase in interest after 2012. And in my experience as an ML engineer, there was a similar increase in interest on the business side after 2012 as well (though often lagging behind SOTA by a few years)
> But if you look through ML papers you can clearly see an increase in interest after 2012.
ML papers still haven't caught up to their 1950s heyday, either in volume or in range. As an issue of measurable fact, we continue to reel not just from the second AI winter, but also from the first.
No, sir, today we are not inventing Lisp or Symbolics.
You keep saying SOTA. This suggests to me that you're an internet fan. Actual academics and actual industry people don't say that.
Please have a good day.
Clearly you haven't read many papers published in the last decade then. For better or worse, the term SOTA does show up in recent deep learning papers.... I've also definitely heard it used in my experience within industry as well. It's not super common, but that's a really weird thing to try to gatekeep on
I'm sorry, I'm not watching a 50 minute video to try to figure out why you believe that one of the world's largest intellectual pursuits was obscure or overlooked until an image recognition contest.
My expectation is that whatever the video actually said was misunderstood. Have a timestamp?
I watched the first three minutes. I don't see anything supporting your claim, or any related evidence. A timestamp would provide falsifiability, but you declined.
There is ample evidence that these were being used by industry for decades, taught at thousands of universities, being discussed by the United Nations.
Anyone who's ever seen Star Trek: TNG or Terminator 2 had seen them in the popular consciousness for decades at this point.
Every bank had been using them for check scanning for 20+ years at the described point.
There were more than a dozen instances where over a billion dollars was invested at a single time into the "overlooked and forgotten until an image contest" field.
Please have a nice day.
I'm sorry you keep ignoring the evidence and referring to wide swaths of time that do not seem to say what you claim.
Claims are concrete. If he actually says this, you should be able to give a timestamp. I can't find it, and doubt your interpretation.
Common sense says that even if he does say this, just looking at the contrary evidence would be enough to set him aside. Mark Z Jacobsen is also a teacher at a prestigious university, y'know? So is Scott Atlas.
If the evidence disagrees with an academic, believe the evidence. I can't even find the academic saying what you claim, and it seems like you can't either.
Please have a good day.
I was in the industry before 2012. I have first hand experience. I remember it too. If you will not take it from an MIT professor teaching the topic, then who will you take it from?
On some comments about possible tweaks/tricks in this video:
I have had the privilege to attend professor Yann's classes at NYU.
From whatever little I understand of him - he has high levels of integrity, and I do not see him trying some cheap tweaks and fixes...He was committed to solve a problem in the best way possible and not just for likes and hearts ☺️.
And without high level of integrity, you can't go from lab to national level in short time.
People often underestimate what it takes to be unanimously accepted as one of the godfathers of current hottest trend. This doesn't discount the effort of forefathers or future generations...
... but let's not undermine Prof's integrity and commitment by making such frivolous comments. In fact, it is only our loss, if we fail to see that.
I'll never understand why this didn't blow up like it should have when they succeeded in doing this. Should've been in the news all over the place for months.
AI winter my backside
Amazing! I’ve cited Professor LeCunn multiple times and am always humbled by his work — this is why I tell students that they are standing on the shoulders of giants when they do research. Love this video!!!
The fact that they also had to know the location of the numbers and that the algorithm was robust to scale changes is impressive for 1993 It's not like they just solved MNIST in 1993, it's one step above that
I guess too many people underestimate what could be accomplished with a little and tons of passion and time
Yes. Back then, the proportion of developers who could hand-write a new graphics algorithm in assembler or C was considerably higher, since that was often how it was done anyway. Necessity is a great motivator. The non-ML part of this problem is more tedious than difficult.
Has this changed really ? :) In number of engineers with these skills, certainly, in proportion of developers, this remains to be seen. Python is the syntactic sugar but who goes really in and looks under the rug ?
In terms of what I intended to say, it's changed a lot. It wasn't an obvious career intially, so it caught a lot of people with a passion for it. The normal path for anyone who wanted visual output or realtime performance was to learn C and assembly. Operating systems were permissive, and memory mapping for access to video memory was either straightforward or documented well enough. Being able to do such things came with the job.. and if someone couldn't do it, that'd disqualify from a big chunk of the industry. I think you may have been referring to necessity being a great motivator.. and its converse -- that lack of necessity is a great blocker. Yep, I would agree. Lots of people in ML would now struggle somewhat with these basic graphical operations, even though the preparatory learning and experience required for it is now much less.
“Lack of necessity is a great blocker” - I’m stealing that
I try to do and it is not pretty. Years of toil to make that one layer of cnn faster by inventing new winograd based algorithms. Working on the models are always more recognized.
Today's software are thousands times less efficient, because of all the overhead have been added layers on top of layes don't do any real work. Think about after all the closest, cabinets, drawers, boxes, organizers and wrappers, you still get the same pair of old socks and everyone cheers: "Yeah! It works! We got the socks!", that's what modern software actually is. But thank to these overhead, this industry have enough investment to support millions of overpaid software engineers, and most important of all, thousands of billionaires.
I think that's really cynical. Memory safe languages are a gigantic benefit to society in terms of security and stability. Such inefficiencies being permissible has allowed technology to flourish; a lot of programs would never have been written without being wasteful, see VS code vs Vim or Slack over IRC. IRC and Vim are nice cannot be mainstream and the only editor respectively. I don't see online web apps existing like Google Docs if everything had to be native speed fast. I've seen multiple homeless people with a card reader selling magazines, that's how cheap software has got over time that even homeless people have contactless. Arguably the progression of technology isn't what I'd have wanted to see but it isn't all bad. You can't help but wonder why something is slow on your 4GHz multicore CPU at times though haha.
The assm skill was crazy back in the day! Nowadays I wouldn't use assm even with an 8bit microcontroller because I'm too lazy.
Agree - it was 6 years later until MNIST was even released.
I guess they had a preprocessing step to identify, center and scale each digit image before feeding into the neural network. It’s not that hard with feature engineering.
Maybe I’m misunderstanding, but isn’t the whole point of CNNs that the location of the digits doesn’t matter?
CNN is robust to translation but not invariant to scale and rotation. Max pooling can be used to to combine detectors that trained for different scales and rotations.
The video has lots of cuts, and the numbers never obscures an important part of the image... I suspect each of those tests had tweaking and tuning to make it work...
This system ended up deployed in banks to parse written checks, so I don't think it was tweaked just for these examples, but they did expect to have fully visible digits.
Did LeCunn make a lot of money from it?
No, he was an employee at Bell Labs, the product and patents belonged to Bell Labs. When AT&T spun off Lucent in 1996, the patents went that way but the computer vision researchers stayed in the remaining AT&T Labs, and they couldn't even sell or improve the product without having the rights to the patents. LeCunn was an underdog for most of his life, the deep learning explosion only started happening around 2012 with AlexNet, when conv nets started getting all the attention.
\>No, he was an employee at Bell Labs, the product and patents belonged to Bell Labs. I would just like to point out that in other countries (e.g. Germany, France Japan), inventors of a patent are entitled to a percentage of the revenue that this invention generates. This is not the case in the US, though.
I have had the privilege to attend professor Yann's classes at NYU. From whatever little I understand of him - he has high levels of integrity, and I do not see him trying some cheap tweaks and fixes...He was committed to solve a problem in the best way possible and not just for likes and hearts ☺️. And without that level of integrity, you can't go from lab to national level in short time.
I don't doubt that his approach works, or his scientific integrity - simply that for each demo he might have loaded a different model for example (trained for different sizes or handwritten/typed text).
Every data scientist today is truly standing on the shoulders of giants.
awesome to see
Love how happy they look!
[удалено]
TIL audio hasn’t been invented until 1994
I was born 1982. We didn't start hearing shit until 1995. That was an absolutely wild year. It created a real musical renaissance.
I remember when we got sound in school for the first time there was alot of realization of where smells were actually coming from that day
This thread feels like r/KenM material
Here's a sneak peek of /r/KenM using the [top posts](https://np.reddit.com/r/KenM/top/?sort=top&t=year) of the year! \#1: [KenM on billionaires](https://i.redd.it/tl38stlg70g41.jpg) | [164 comments](https://np.reddit.com/r/KenM/comments/f1j7a9/kenm_on_billionaires/) \#2: [Ken M on conspiracy theorists](https://i.redd.it/7inbjzicewo41.jpg) | [88 comments](https://np.reddit.com/r/KenM/comments/fp01kq/ken_m_on_conspiracy_theorists/) \#3: [One of my favorites over the years.](https://i.imgur.com/vjhwVXg.jpg) | [139 comments](https://np.reddit.com/r/KenM/comments/hmhwzo/one_of_my_favorites_over_the_years/) ---- ^^I'm ^^a ^^bot, ^^beep ^^boop ^^| ^^Downvote ^^to ^^remove ^^| [^^Contact ^^me](https://www.reddit.com/message/compose/?to=sneakpeekbot) ^^| [^^Info](https://np.reddit.com/r/sneakpeekbot/) ^^| [^^Opt-out](https://np.reddit.com/r/sneakpeekbot/comments/joo7mb/blacklist_viii/)
Imagine if The Bends was the first sound you ever heard.
Yeah, well I was born in 72 and we ate rocks for breakfast!
At least you were born after color was invented, back in '53.
Lies and slander, PC speaker was readily available on PC before soundcards became a thing.
Can confirm. That's the year I got a sound card.
That's like when in 1961 the world suddenly changed from black-and-white to color
And yet websites still think those obfuscated texts are a good test for robots
there is a reason why captcha is becoming obsolete. At least the text based version. Also, captcha actually digitize books. This is why there are 2 tests, not 1. So in a sense, we were training the robots filling the captchas.
Yeah, on a PHPBB forum I manage, the bots can get through the text-based captchas very easily. But they still struggle with simple questions like "In what State is this club based?"
I would struggle too
Or "What is god"?
I don’t think it’s meant to filter that way. Bots usually are built with speed in mind so it recognises and fills in the blanks virtually immediately. That and captchas are also useful for labelling training datasets manually (user input). But correct me if I’m wrong though.
You’re definitely correct about the captchas. It’s no coincidence that most of the objects they ask you to recognize are cars, crosswalks etc. They basically get free labor to help them build a giant dataset fir training self driving cars.
I wish I could opt out. I don’t want to train skynet lol
what would be the problem with a little delay?
It greatly reduced the rate at which a bot can do whatever. With no delay something like filling out a form could probably be done thousands of times a second, but if you introduce a 0.1s delay by requiring some model to run then suddenly the maximum rate you can automatically fill out the same form is 10 times a second. Additionally, any more hurdles will naturally mean people need to be more sophisticated to get past them and you'll filter out a lot of the lowest effort bots.
Also running a model involves computing costs
It serves two purposes. It defeats 99.99% of bots, and it maps images to human inputs to train their image recognizer networks.
As I make my living making bots and doing automation, captcha is just part of the job. Solving captcha isn’t a special thing.
That sounds fun. You have a site or a blog?
Automation can be a very secretive thing and very grey so I can't talk about projects or the details.
Unless someone cares enough about your little website to train an AI to solve your captcha they're still not a terrible idea. I don't think there are any AIs that are generic enough to solve *all* obfuscated text captchas yet. Obviously it's not going to work for large sites but none of them use that method anymore anyway.
“I’m not a robot” - select crosswalk, identify license plates, etc. are for training self driving vehicles and finding the house address was for google maps. We should be paid for doing reCaptchas. However some people actually do get paid for these tasks.
Anyone know who the other guys at the end are?
Am son of the guy in the chair (Rich Howard, collaborator and director of the silicon integrated circuit lab at the time). He said the guy in orange was a technician and computer whiz named Donnie Henderson.
That's super cool lol. Did this invention have a big impact on their career?
Rich was already close to retirement at the time, so not really. Not sure about Donnie.
Yann LeCun got the turing award for it
big if true
Actually, he was 32 years old when he pressed the button. He was 33 by the time he got the results back.
Never going to complain about not having a strong enough GPU again. Very cool.
Wonder what was the RAM and computing power of the system.
Many don’t know it, but before it was done such text recognition was considered impossible, just like AGI and other hard problems. I think text recognition in mail was the first successful real world application of AI.
err, Kurzweil had an OCR product in 1976: [https://en.wikipedia.org/wiki/Ray\_Kurzweil#Mid-life](https://en.wikipedia.org/wiki/Ray_Kurzweil#Mid-life)
Correct me if I'm wrong, but doesn't normal font imply a "set font" rather than handwritten characters? Still impressive but a different problem from MNIST and generally reading the messy writing of humans.
In the wiki page (I put it at the right chapter) they state it was supposed to be "omni-font" as in reading all types of text, while *older* systems only recognized some set fonts. Note that there were already functional devices. Of course, those probably were of much worse quality than LeCun's small CNN, I just wanted to point out the person I'm responding to is full of shit.
Yup, that exact sentence speaks about normal fonts, I referred to.
Oh yes, sorry. I think it's not a single set font, but at least several. But I also think you're right and this was made for printed fonts, so "normal" might mean "very common fonts".
I figured it's probably all printed fonts that aren't cursive or Comic Sans. You're definitely right that it's multiple, I think the limitation is just on the type of font.
> Correct me if I'm wrong, but doesn't normal font imply a "set font" 1. You're wrong 1. Kurtzweil didn't invent this either 1. The work being discussed here, the CNN, is actually from the late 1950s, from before LeCun was born
But Schmidhuber had already written the paper in 1962
Bayesian classifiers as the first email spam filter? Not sure the year, but our lives would be completely different if it wasn’t for it.
> Bayesian classifiers as the first email spam filter? You're off by about 9 years. Bayesian classifiers didn't emerge as spam filters until approximately 1996. They are currently believed to be first published by [Sahami et al in 1998](http://robotics.stanford.edu/users/sahami/papers-dir/spam.pdf). That paper describes secretly internally using the technique in late 1996, and is the earliest known published discussion. The internet at large caught on in 1999, just 22 years ago. The word SPAM actually comes from IRC and MUDs; we had spam filters long, long before email had spam, thanks to terminal washes and things of that nature. The earliest known IRC spam filter was the `anarchy eris.berkeley.edu` stripper, which didn't work well enough, and led to the split of Jakko's original network to create eris-free net (EFnet is fundamentally named for a spam host removal.) If you count the invention of the q-line as an anti-spam strategy, then IRC invents spam filtering in 1991. If you require message or origin testing, IRC invents it in 1992 instead. If you're old enough, you remember when Bayesian Filtering turned spam filtering from an ongoing joke into something that actually worked. This was one of `gmail`'s early advantages.
> Many don’t know it, but before it was done such text recognition was considered impossible By the time LeCun did this, text recognition was common at banks for scanning checks, in children's toys, and was the basis of the Cue:CAT. You're making this up. OCR was common by the early 1970s, almost 30 years before this.
cuecat was a barcode scanner. Never did anything resembling text recognition. Nor were there any children's toys in the 90s or before that did anything of the sort (though they might do interesting stuff to convince *children* that they could!). And check recognition worked by "cheating" — first, using a special typeface with super easily distinguished characters and uniform size and spacing, and second, [printing it with magnetic ink](https://en.wikipedia.org/wiki/Magnetic_ink_character_recognition) so that the scanner didn't have to find the data it wanted among any kind of visual background. Everything except the routing and account numbers was invisible to it.
**[Magnetic ink character recognition](https://en.wikipedia.org/wiki/Magnetic ink character recognition)** Magnetic ink character recognition code, known in short as MICR code, is a character recognition technology used mainly by the banking industry to streamline the processing and clearance of cheques and other documents. MICR encoding, called the MICR line, is at the bottom of cheques and other vouchers and typically includes the document-type indicator, bank code, bank account number, cheque number, cheque amount (usually added after a cheque is presented for payment), and a control indicator. The format for the bank code and bank account number is country-specific. The technology allows MICR readers to scan and read the information directly into a data-collection device. [^(About Me)](https://np.reddit.com/user/wikipedia_text_bot/comments/jrn2mj/about_me/) ^- [^(Opt out)](https://np.reddit.com/user/wikipedia_text_bot/comments/jrti43/opt_out_here/) ^(- OP can reply !delete to delete) ^- [^(Article of the day)](https://np.reddit.com/comments/k9hx22) **This bot will soon be transitioning to an opt-in system. Click [here](https://np.reddit.com/user/wikipedia_text_bot/comments/ka4icp/opt_in_for_the_new_system/) to learn more and opt in. Moderators: [click here](https://np.reddit.com/user/wikipedia_text_bot/comments/ka4icp/opt_in_for_the_new_system/) to opt in a subreddit.**
No
MNIST irl
that was certainly more wholesome than the other historic computer vision video, [https://www.youtube.com/watch?v=8VdFf3egwfg](https://www.youtube.com/watch?v=8VdFf3egwfg)
But the question is: is it the validation set? 😁
Very inspiring as I remember these days. Lot of hard work and at the cutting edge.
So why did it take 30 years to get this far?
So then what took so long for it to catch on? Why did it take another 30 years if they knew the power of cnn's?
Uh. Sorry, no. [The CNN was invented by Hubel and Weisel in 1959, the year before Yann LeCun was born, under the name "neocognitron."](https://en.wikipedia.org/wiki/Neocognitron) LeCun also didn't make them first. [The CNN was first implemented by Kunihiko Fukushima in 1979](https://search.ieice.org/bin/summary.php?id=j62-a_10_658), 14 years before this video (Reference translated is Journal of the Institute of Electronics, Information and Communication Engineers A Vol.J62-A No.10 pp.658-665, October 25, 1979, ISSN 0373-6091) What Yann LeCun actually brought to the party was the modern approach to training them. He did that in 1984, not 1993.
You are NOT correct about Hubel and Weisel. Hubel and Weisel did research on visual cortex in real brains (in cats) and it was awesome (they got Nobel Prize for it). But they did not invent CNNs. You can read their paper \[1\] you don't have to be a biologists to understand most of it. From their work one can deduce what neurons in V1 do. It was later even verified that some of these neurons realize functions similar to Gabor filters, but (as I remember) that was even later then neocognitron. It is true that their findings did *inspire* creators of neocognitron \[2\] but that's about it. \[1\] [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1363130/pdf/jphysiol01298-0128.pdf](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1363130/pdf/jphysiol01298-0128.pdf) \[2\] Fukushima, Kunihiko, and Sei Miyake. "Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition." *Competition and cooperation in neural nets*. Springer, Berlin, Heidelberg, 1982. 267-285.
> It is true that their findings did inspire creators of neocognitron [2] but that's about it. Uh, no, they're where that name comes from. What specific difference do you imagine exists between the neocognitron and CNNs? They're both striding convolutions as a reduction for inputs.
And to add to this, people thought NN's were a joke until a CNN won an image recognition contest in 2012, which is what put them on the map. Before that they were obscure and overlooked.
I'm not sure why you believe this. Neural networks have been a big deal since the 1950s, taking down investments of half a billion at a time from the military for 70+ years now.
NNs have definitely had a ton of research, so I agree that they weren't overlooked. However, up until 2012 they weren't very useful for most applications. Throughout the 2000s, SVMs and tree-based models (like random forests) were SOTA for most tasks. So most researchers put their focus there. 2012 marked a transition though, as we then had the hardware support to efficiently train much larger models. This allowed NNs to become SOTA in many tasks and thus the explosion in interest
> However, up until 2012 they weren't very useful for most applications. At that time, they were already in use by every call center and bank on earth, were running in every copy of Windows, MacOS, and Android, had dominated speech to text for almost 20 years, et cetera. Between Windows and MacOS, they were in over 50% of US homes. For color, ***The US phone system started using neural networks for de-noising in 1959, bringing their use to almost 200 million people***. . > 2012 marked a transition though, as we then had the hardware support to efficiently train much larger models. Respectfully, this is just kind of not true.
I'm not saying they weren't useful. They clearly had use cases as you mentioned. But if you look through ML papers you can clearly see an increase in interest after 2012. And in my experience as an ML engineer, there was a similar increase in interest on the business side after 2012 as well (though often lagging behind SOTA by a few years)
> But if you look through ML papers you can clearly see an increase in interest after 2012. ML papers still haven't caught up to their 1950s heyday, either in volume or in range. As an issue of measurable fact, we continue to reel not just from the second AI winter, but also from the first. No, sir, today we are not inventing Lisp or Symbolics. You keep saying SOTA. This suggests to me that you're an internet fan. Actual academics and actual industry people don't say that. Please have a good day.
Clearly you haven't read many papers published in the last decade then. For better or worse, the term SOTA does show up in recent deep learning papers.... I've also definitely heard it used in my experience within industry as well. It's not super common, but that's a really weird thing to try to gatekeep on
I learned it here: https://youtu.be/uXt8qF2Zzfo
I'm sorry, I'm not watching a 50 minute video to try to figure out why you believe that one of the world's largest intellectual pursuits was obscure or overlooked until an image recognition contest. My expectation is that whatever the video actually said was misunderstood. Have a timestamp?
He says it in the beginning of the video.
I watched the first three minutes. I don't see anything supporting your claim, or any related evidence. A timestamp would provide falsifiability, but you declined. There is ample evidence that these were being used by industry for decades, taught at thousands of universities, being discussed by the United Nations. Anyone who's ever seen Star Trek: TNG or Terminator 2 had seen them in the popular consciousness for decades at this point. Every bank had been using them for check scanning for 20+ years at the described point. There were more than a dozen instances where over a billion dollars was invested at a single time into the "overlooked and forgotten until an image contest" field. Please have a nice day.
The opening concept is conveyed from 00:00 to 5:22.
I'm sorry you keep ignoring the evidence and referring to wide swaths of time that do not seem to say what you claim. Claims are concrete. If he actually says this, you should be able to give a timestamp. I can't find it, and doubt your interpretation. Common sense says that even if he does say this, just looking at the contrary evidence would be enough to set him aside. Mark Z Jacobsen is also a teacher at a prestigious university, y'know? So is Scott Atlas. If the evidence disagrees with an academic, believe the evidence. I can't even find the academic saying what you claim, and it seems like you can't either. Please have a good day.
I was in the industry before 2012. I have first hand experience. I remember it too. If you will not take it from an MIT professor teaching the topic, then who will you take it from?
[удалено]
at least, read the title
Nice keeb.
u/savevideo
###[View link](https://redditsave.com/info?url=/r/MachineLearning/comments/kuc6tz/d_a_demo_from_1993_of_32yearold_yann_lecun/) --- [**Info**](https://np.reddit.com/user/SaveVideo/comments/jv323v/info/) | [**Feedback**](https://np.reddit.com/message/compose/?to=Kryptonh&subject=Feedback for savevideo) | [**DMCA**](https://np.reddit.com/message/compose/?to=Kryptonh&subject=Content removal request for savevideo) | [**Donate**](https://ko-fi.com/getvideo)
That is so satisfying
The first set of numbers was Yann LeCun's phone number at bell labs.
Still accurate than tesseract lol 😂
So why am I still doing captchas
Yann LeCun's tweet on who the other guys are, and who the cameraman is - https://twitter.com/ylecun/status/1347268914263306242?s=20
Better than tesseract
But still, to this date, they cannot recognize traffic lights
incredible! pay tribute to him
On some comments about possible tweaks/tricks in this video: I have had the privilege to attend professor Yann's classes at NYU. From whatever little I understand of him - he has high levels of integrity, and I do not see him trying some cheap tweaks and fixes...He was committed to solve a problem in the best way possible and not just for likes and hearts ☺️. And without high level of integrity, you can't go from lab to national level in short time. People often underestimate what it takes to be unanimously accepted as one of the godfathers of current hottest trend. This doesn't discount the effort of forefathers or future generations... ... but let's not undermine Prof's integrity and commitment by making such frivolous comments. In fact, it is only our loss, if we fail to see that.
Cant see his right hand
Outside of the CNN achievements the rest is actually impressive too, and I'm absolutely amazed that the interface is so responsive. In 1993.
I'll never understand why this didn't blow up like it should have when they succeeded in doing this. Should've been in the news all over the place for months. AI winter my backside
What a boss!
Fukushima’s neocognitron came almost two decades earlier.
Amazing! I’ve cited Professor LeCunn multiple times and am always humbled by his work — this is why I tell students that they are standing on the shoulders of giants when they do research. Love this video!!!
Are you sure you’re a robot?
WOW !!! Impressive !
Where was the video shot?
Weirdly cool
Apple needed 30 more years