The HLF Portraits: Yann LeCun
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 66 | |
| Autor | ||
| Mitwirkende | ||
| Lizenz | Keine Open-Access-Lizenz: Es gilt deutsches Urheberrecht. Der Film darf zum eigenen Gebrauch kostenfrei genutzt, aber nicht im Internet bereitgestellt oder an Außenstehende weitergegeben werden. | |
| Identifikatoren | 10.5446/18527 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
The HLF Portraits57 / 66
2
9
14
16
17
18
20
23
25
26
27
29
30
34
35
36
38
40
43
46
50
53
57
58
00:00
InformatikWort <Informatik>BereichsschätzungFormation <Mathematik>PhysikalismusGradientProzess <Informatik>NeuroinformatikComputerspielMereologieMathematikEndliche ModelltheorieLateinisches QuadratVerkehrsinformationQuick-SortVererbungshierarchieVersionsverwaltungAuswahlaxiomFlächeninhaltFamilie <Mathematik>Physikalisches SystemGüte der AnpassungSchlussregelPunktNatürliche ZahlSprachsyntheseDifferenteBetrag <Mathematik>BeobachtungsstudieProjektive EbeneKlassische PhysikMultiplikationsoperatorWeg <Topologie>WellenpaketPunktspektrumGebäude <Mathematik>MAPSichtenkonzeptBesprechung/Interview
09:22
ÄhnlichkeitsgeometrieNeuroinformatikKlasse <Mathematik>Quick-SortFastringGenerator <Informatik>ProgrammbibliothekLogiksyntheseHyperbelverfahrenStochastische AbhängigkeitWellenlehreBitProjektive EbeneKategorie <Mathematik>MathematikTelekommunikationFormation <Mathematik>Konfiguration <Informatik>Prozess <Informatik>ExpertensystemAnalogieschlussEvoluteMinkowski-MetrikEinsPhysikalisches SystemAssemblerInformatikFolge <Mathematik>MereologieNeuronales NetzInformationsverarbeitungRobotikRechter WinkelGrundsätze ordnungsmäßiger DatenverarbeitungFrequenzFormale SpracheDatenfeldVirtuelle MaschineMultiplikationsoperatorBeobachtungsstudieWissenstechnikMathematikerPerzeptronLesen <Datenverarbeitung>AuswahlaxiomResultanteTermAutomatische HandlungsplanungComputersimulationBesprechung/Interview
18:25
Algorithmische LerntheorieObjekt <Kategorie>Analoge SignalverarbeitungQuick-SortMathematikGeradeDigitaltechnikTermPhysikalische TheorieMultiplikationsoperatorÄquivalenzklasset-TestVirtuelle MaschineElement <Gruppentheorie>SoftwareProgrammierungNeuroinformatikBeobachtungsstudieInformatikApproximationBackpropagation-AlgorithmusInverser LimesDatenstrukturPhysikalisches SystemGebäude <Mathematik>Rechter WinkelGewicht <Ausgleichsrechnung>Orientierung <Mathematik>Office-PaketDatenfeldAlgorithmusMinimalgradMultiplikationEinsPunktFlächeninhaltKlassische PhysikTuring-TestAutorisierungVerzweigendes ProgrammStrategisches SpielParametersystemMomentenproblemRückkopplungBesprechung/Interview
27:28
AssemblerRechter WinkelDemoszene <Programmierung>Prozess <Informatik>VirtualisierungVirtuelle MaschineBoltzmann-KonstanteZweiSelbst organisierendes SystemWellenpaketGewicht <Ausgleichsrechnung>SprachsyntheseKognitionswissenschaftMultiplikationBell and HowellRichtungE-MailMinimalgradInformatikt-TestFigurierte ZahlDatenfeldPhysikerNeuroinformatikMathematikerKlasse <Mathematik>AlgorithmusBitEuler-WinkelMultiplikationsoperatorQuick-SortWort <Informatik>HilfesystemGlobale OptimierungKartesische KoordinatenLogischer SchlussInformationsverarbeitungPeer-to-Peer-NetzBesprechung/Interview
36:31
SchlussregelKontextbezogenes SystemWellenpaketFolge <Mathematik>MultifunktionQuick-SortEinfach zusammenhängender RaumKartesische KoordinatenPunktwolkeSchnittmengeBackpropagation-AlgorithmusAlgorithmusVersionsverwaltungPropagatorFunktionentheorieBell and HowellSprachsyntheseVirtuelle MaschineLeistung <Physik>Gewicht <Ausgleichsrechnung>NeuroinformatikWort <Informatik>Deskriptive StatistikFaserbündelShape <Informatik>MustererkennungDigitalisierungProgrammierungPerfekte GruppePhysikalisches SystemCodeSpeicherabzugGruppenoperationStatistische Hypotheset-TestTypentheorieGroße VereinheitlichungPunktTermTaskRechter WinkelAutomatische HandlungsplanungMomentenproblemBesprechung/Interview
44:53
SoftwarePrototypingFaltungsoperatorProgrammierungMessage-PassingMultiplikationsoperatorGewicht <Ausgleichsrechnung>DigitalisierungEndliche Modelltheoriet-TestSchreib-Lese-KopfAlgorithmusRandverteilungProjektive EbeneVirtuelle MaschineNeuroinformatikMustererkennungBell and HowellVersionsverwaltungInterpretiererComputerspielGruppenoperationAlgorithmische LerntheorieHeimcomputerPhysikalisches SystemBildgebendes VerfahrenFrequenzTermPunktNichtlinearer OperatorVererbungshierarchieDatenkompressionAggregatzustandURLKontrollstrukturMaschinenschreibenMereologieQuick-SortBesprechung/Interview
53:16
Elektronisches ForumBesprechung/InterviewComputeranimation
Transkript: Englisch(automatisch erzeugt)
00:17
I'd like to begin at the beginning.
00:22
When you came into the world, where were you? I was born near Paris. I grew up in a little town in the northwestern suburb of Paris, about 15 kilometers from Paris. My dad was an engineer, he was a mechanical engineer, he worked in the car industry
00:41
and then the aeronautical industry. Before we pursue that, which is of course critical in, I think, in your intellectual development, is the family a Parisian family? Does it think of itself as a Parisian family? My father's family came from Brittany, in fact my name is very, very typical. It's pronounced Yannes-le-Cain, I've given up on trying to tell people it's pronounced,
01:04
but the N is silent or it's nasalized. So that part is from Brittany and culturally I'm kind of close to that, so Celtic culture if you want. Celtic culture. On my other side, on my mother's side, her father was from Auvergne, the center of France.
01:21
Yes. And there's a stereotype in France where when you're from Auvergne, your job is to sell wine and he was selling wine. I barely knew him because he died when I was very young. And my grandmother on my mother's side was from Alsace, she was born in 1900 when Alsace was still Germany. And after World War I, she moved to Paris not speaking a word of French.
01:43
You knew her in your childhood? Yeah. Of course she was speaking French by then. That's right, but she lived until I was in my 20s and she always spoke with a very strong German accent. I know about, and we will talk about your father's work and one might even say intellectual
02:01
tradition, but what about your mother? Was she intellectually curious? Was she working or not? She was. She didn't study beyond high school, but was intellectually very curious and was reading a lot and was interested in music and all kinds of different topics.
02:23
So she was certainly very nurturing from the intellectual point of view. Is it the sort of household where there were books everywhere where you were taken to concerts or whatever? Was it a French intellectual household? So I wouldn't say it's very far on that spectrum.
02:44
I think it was more sort of fashioned by my dad who as an engineer liked to build things. So he had a workshop and he had hobbies, he taught himself electronics, he was building electronic widgets, he was building model airplanes, and I got into this because of him.
03:00
And so I learned engineering with him basically. As a boy? As a boy. You drank it in. That's right. Yes. I was fascinated by engineering and technology because of that. How many children in the family? Three. I'm the oldest one and then I have a younger sister, two years younger than me, and then
03:21
a young brother who is six years younger than me who happens to work at Google. He's also a computer scientist. Yes, so the atmosphere to drink in was one that influenced at least two of you in your career choices. Yeah, my sister is more of an artist and also a teacher.
03:45
I'm always interested in the mentors in life. Sometimes it's the parents, sometimes not, but I'm beginning to sense that your father was a mentor. You had one very early in your life. Yeah, certainly. Absolutely. He guided you consciously in certain interests or you just saw what interested him?
04:03
He was doing things and he was kind of trying to... It was not like pushing for me to get interested in the same thing he was doing, but he was welcoming and very helpful. And so my brother and I had a natural kind of interest for building things. It was natural, but there was no strong sense of ambition for you.
04:24
No. We hope you do this. We hope you do that. No, I think it was never pushy. The one time he was was when he told me that he regretted not to have been pushier with himself. And it was because he had very little...he didn't have a lot of self-confidence when
04:43
he was young. He studied...my grandfather, his dad was a doctor, and he liked the idea in France back in those days of classical education where you learn Latin and science is kind of a little secondary.
05:01
And my dad was really sort of a technology geek. He hated learning Latin and these kind of things. And so he went all the way to 12th grade and then there's a high school exam in France back at Oria. He failed it because he had, I think, zero in Latin.
05:23
And that means it doesn't matter what grades you get in other things, you fail. So a few months later, after the summer, he retook it, but he took the modern version without ever having studied. And he got top grades in math and physics. So as a result, he kind of lost confidence in himself and he did an engineering school
05:44
that wasn't quite an engineering school and he said, like, I made a mistake, I should have been more ambitious. So that's what he told me. That's a profound influence, however indirect. So we have to get you to school. Is it a good school system in the area you are?
06:01
Are you going to what we would call a public school? So I went to a public elementary school in our town, Suez-y-Souba-Mon-Sie. And then the middle school wasn't that great. And so my parents decided to put me in what I guess here you would call the charter school or something like this, which was sort of a Catholic school, but not really.
06:25
It's one of these strange school systems in France. And so I did all of my middle school and high school in that school. In this other system. So I, you know, I mean, it's the same. I mean, they're subject to the same broad educational rules.
06:40
Same education system. The teachers are the same type of people. And there is some sort of private way of managing the school. Can we find at this stage of your life another mentor? Is somebody noticing you or are you noticing more what you want to do before we get you
07:04
to university? I was noticed by my science teacher in seventh grade because we had this project we had to work on and I did this whole report on microscopic life or something like that. But the real, in other words, physics and math, you know, physics teachers later.
07:23
But the person I think who was probably the biggest mentor was a music teacher. Really? So, you know, we learned to play various instruments starting in sixth grade. And then some of the best students, he had kind of a band which was playing Renaissance
07:41
and Baroque music. And I got into this band when I was about 14. And it was completely delightful. I mean, I love music and I love practicing music. I'm not particularly good at it, but I really like it and I think I had a good sound. And so I got into that thing and I think it helped me.
08:02
I was kind of a shy, overweight, geeky boy and, you know, playing in public helps you sort of confront your fears, right? And so I think that helped me afterwards when I had to do public speeches and, you know, that kind of conference.
08:20
And rather more than a lot of your colleagues, you do that. I mean, you embrace that role in life. So we thank your music teacher for that. That's right. Again, I don't know the system enough to know, is there a counselor who then directs you in terms of university? How are your choices being made?
08:40
It was totally obvious to me that I was going to do something that had to do with science or engineering. Engineering specifically? Yeah, because I like building things. So, and I thought, I mean, there's sort of, you know, two tracks that I could take after high school. One would have been, you know, there's a system in France called the Grands Ecole, right? So you do two years of intense math and physics training and then you basically take
09:06
an exam and depending on your score and your ranking, you can get into one of the top schools. And I thought that was risky, first of all, because if you don't get into the school, you're interested in, you do stuff you're not interested in.
09:20
I was really interested in, you know, computer electronics, things like this. So I said, like, I'm just going to go to an engineering school that starts from high school. It's the same thing in the end. You do five years. And as I progressed in that school, we can talk about this later. So the school is called ESIWE, École superie en agénieur en électronique électrotechnique.
09:44
And as I progressed to that school, I realized I was more interested in science than engineering. And so a little bit like my father, I kind of regretted not doing the other option. But I kind of fixed that by doing a PhD afterwards. What we didn't yet bring into your childhood or adolescence is your encounter with computers.
10:04
Unlike many of the people I've interviewed, you're actually born into a computer generation. I mean, it's around. So how do you encounter the computer? What do you think about the experience? So I've been fascinated by the idea.
10:21
I think one of the things that kind of attracted me to this is that I was interested in the mystery of intelligence from a very young age. How does the brain work? What is it that caused human intelligence to appear? How different are we from apes? So I was very interested in human evolution for that reason.
10:43
And then other themes that kids are interested in, you know, astronomy and technology of various kinds, space flight. When I was nine years old, the movie 2001 A Space Odyssey came out, which talked about everything I was interested in. It talks about the emergence of human intelligence, artificial intelligence, intelligence in computers,
11:04
space travel, the future of humanity. So I found this movie absolutely fascinating. I was only nine years old, but it just opened my eyes. Well, it's a good age to have that experience. That's right. Early enough. Exactly. So I think that movie probably had a huge influence on me.
11:20
And sort of opened me to the idea that machines could be intelligent and perhaps a way to discover the mysteries of human intelligence is to build machines that are intelligent. So it's about how? It's about how. Yeah. I mean, how went bad, but you know. Well, and of course, we'll talk about that, because at least in popular journalism, machines
11:40
going bad is a big topic, but we'll take the positive side of it now. When you decide in engineering, you're going to switch later. But when you decide in engineering, is that where a computer-directed education would be in any case? So not really, because the school I went to is electrical engineering, and you could do
12:03
computer as a tool, or you could do computer science, but I didn't do computer science. But sort of pedaling back, in the mid-70s, it started to become possible to buy computers, They were too expensive for me, I didn't have any money.
12:21
But I was reading about it in various magazines, and I kind of learned about assembly programming and stuff like that, basically in sort of a platonic fashion, because I didn't have access to a computer. And then there was this trade show in Paris that was taking place every year, where you could walk around and see a bunch of computers that manufacturers were displaying, including
12:43
like little programmable calculators, they were too expensive for me again, but I played with them. I spent an entire day at that place, and the guy who was at the booth probably hated me at some point, and I was playing with those Texas Instruments or Hewlett-Packard computers and kind of programming them on the spot. And then I was interested in electronics, because I was interested in music, and the mid-70s
13:06
was when synthesizers came to the fore, and you could use electronics for music. And you could also use computers for music. You could not yet, but that was kind of a new thing that you could do. So my cousin, my older cousin, was an aspiring electronic musician, he had an analog synthesizer,
13:25
and I was modifying it for him, because I knew electronics. I taught myself electronics when I was in high school. I built sequencers for him, I built things like this, and I said, you know, we could use computers for like sequencing. So I got interested in computers partly because of this, partly because of my long-term interest
13:43
in AI and robotics. And I started playing with computers about a computer, a very simple computer. When I was 17, so in 12th grade, and I started kind of teaching myself. The schools are not providing you with access to computers at all?
14:01
Not yet, but at that time at all. No, we're talking 1977. So that's too early. Okay, I want to now make the switch with you from engineering to more formal scientific inquiry. You realized that you made the wrong choice in terms of a school or a...
14:23
It wasn't the wrong choice, it was just a little bit of a struggle, but I had mentors also at the engineering school, math professors, with whom I would, you know, some of them took me under the arm, I was interested in sort of independent projects with them,
14:40
so I did a couple research projects with them. Some of them had to do with neural nets, actually, because I got interested in this pretty early on, and I can tell you how I got interested in this later. So I did a few independent projects with math professors, and they kind of encouraged me to sort of continue, and that gave me the taste for research and scientific research.
15:02
So you graduate with a specialty in what? Two specialties. One was chip design, the other one's microelectronics. The other one was what we call automatics, but it's systems control, systems theory,
15:21
process control. Looking back now, and we're at an ongoing, extraordinary growth period in your field, but what was the state of the field at the time that you are graduating from university? So when I started my studies in 1978, AI was really sort of a backwater, not a huge
15:46
amount of work on this, but by the time I was completing my studies, there was a wave of interest in what people at the time called expert systems. Japan announced a big plan of fifth generation computers, which were supposed to be sort of the intelligent computers of the future, it was a complete failure.
16:03
But there was a big wave of interest, and there was this idea somehow that it was going to be a big category of jobs over the next decade of people who are knowledge engineers that would sit down with experts and then compile their knowledge into a knowledge base,
16:21
and then the computers would be able to kind of reproduce the expertise of that expert. There was not so much of a big success in the sense that this didn't lead to a way to build intelligent machines, but like a lot of things in AI, it led to a bunch of tools in computer science that are now part of the standard toolbox and are very useful.
16:45
But what happened to me was around 1980, I stumbled, I was really interested in cognitive science, epistemology, and things like this because of my interest in human intelligence, and I stumbled on the book, which was the transcripts of a debate between Noam Chomsky,
17:04
the linguist, and Jean Piaget, the developmental psychologist. Yes, my God. Okay, so the nature-nurture debate about language, is language learned, is language acquired or innate? Each of those guys brought their team of people to argue for their side,
17:24
and on the side of Jean Piaget was a gentleman by the name of Seymour Papert, who was at MIT at the time, he was a mathematician at MIT, and computer scientist. And he talks about a computer model called the perceptron that is capable of learning,
17:41
and so I'm reading this and it's a machine that can learn, that's fascinating. This is the first time I read about something like that, and I always thought that learning was, could not be really separated from intelligence, intelligence is the result of learning really, right? So I said, that's the trick, if we're ever going to build intelligent machines, they're going to have to be able to learn.
18:02
And you probably heard very similar stories from Geoff Hinton and Yoshua Nanjio. And so I started digging the literature, I spent, we didn't have class on Wednesday afternoons, so I would borrow my dad's car, drive to Rancourt-Concoure near Versailles, where there was a big library of Inria, Inria is kind of the national lab for computer
18:22
science in France, and they had a big computer science library. So I would spend my entire afternoon there, and dig the literature, and I quickly realized that all the work in what was not yet called neural nets, you know, started with the Perceptron in the 1950s, and then basically ended in the late 60s, and it ended because
18:44
of a book that Seymour Papert was a co-author of, which basically showed through mathematics the limitations of those models, and it kind of helped kind of make the whole field die, or not die, but go underground and change names.
19:01
This is, I think, the first winter, so to speak. It's the first winter of neural nets. So it was not the winter of AI, because other people, like Marvin Minsky, for example, and others, were pursuing a completely different approach to AI, which was based on logic, and search, and reasoning, which kind of went on for a few years, and then died a
19:20
little later, in the mid-70s, and then reappeared in the 1980s. So I saw this sort of branching out of sort of, you know, logic-based AI, world-based system, and, you know, that led to classical computer science, if you want, and then this other
19:40
branch based on, like, neural networks, there were the descendants of cybernetics, the field of, you know, studies of systems with feedback and things like this. And this was closer to, you know, the first branch was closer to computer science, so the second one closer to electrical engineering, and I was more of an electrical engineer.
20:02
So it was continuous math, the math is very similar to the one you study in circuit theory or signal processing, so that seemed more like, you know, I felt more like more at home. I was not a computer science student. And so I kind of looked at this literature and realized that, first of all, what people
20:22
were looking for at the time to go beyond the limitations of those systems was the ability to build neural nets with multiple layers and to train them that way. The Perceptron basically had only a few layers, but only the last layer was trainable. The other ones were kind of hardwired, if you want.
20:40
And it was pretty clear that having multiple layers would kind of lift a lot of the limitations. And so I looked for people who were thinking about those questions and basically didn't find anybody. You literally could not. There were people who sort of formulated that problem who said like, you know, this is where we need to do research, but there was no funding in this area anymore because
21:02
the whole field had been abandoned essentially. So there was no literature almost at all, except, you know, a few Japanese authors between the late 60s and the early 80s when I was looking into this. And you're not so pragmatic that you say, OK, I will go with the dominant thinking on this.
21:23
You're already captured by a particular problem or strategy. It's very simple, right? Intelligence is produced by learning. If you want to reproduce intelligence, you want to reproduce learning. This is something Turing already said in the 50s.
21:41
How do you do learning? So an interesting principle of learning that Perceptron used is the idea that learning basically tries to optimize some objective, right? There is an objective that measures the performance of the system, and you're basically adjusting the parameters of the system so that this objective is optimized. Yes, well said. And you can find this in textbooks. That was a very good principle, I thought.
22:02
And the sort of more logic-based approaches are basically incompatible with this. You can't, because it's all discrete and combinatorial. And it seemed to me that... To you, this was obvious. To me, it was completely obvious. The whole line that they were following didn't make sense in terms of the ultimate Well, it made sense for some purpose, not for what I was interested in, OK?
22:21
There were very, very few people who were actually interested in learning at the time. They were interested in building intelligent machines by typing rules and facts. And that may have good use. In fact, it has some use, but I didn't think that was the way to go forward. So learning was really crucial to me. So I started trying to figure out, like, how could we train multilayer neural nets?
22:45
And I came up with some idea, which you could think of as some sort of approximate ancestor to what we now call the backpropagation algorithm, around 1982 or so, I was still an undergrad.
23:00
But it took me a couple of years to kind of implement it. What is the structure of your inquiry? Are you now in graduate work? So I'm still an undergrad. You're still an undergrad. Right. The school I'm studying at has, for the time, fairly powerful computers. So I kind of experiment with various things. I finish my studies.
23:20
The last year, my study is some sort of long-term internship where I design a signal processing chip for a French company. But on my spare time, I work on this idea of learning. Because you have nobody teaching you who could be useful in this inquiry.
23:41
Other than those kind of well-meaning math professors who told me, yeah, they were encouraging me, but were not particularly expert. They made me meet people, and the people I met weren't working on this because they knew the failure of the perception from 10 years earlier.
24:01
And so they tried to talk me out of it. Is the problem, I'm just going to say, what encourages your inquiry and what is beside the point of your inquiry, is the problem not only the, we won't call it a fashion, but the moment in intellectual inquiry about these questions of learning or not?
24:20
Or is the problem being in France far away from those who are asking these questions? No, I don't think it's being far away. I think maybe being far away and outside of the mainstream system made it so that I didn't have a set path that I had to follow. I could just talk to a professor and he would tell me what to work on and blah, blah, blah. So that allowed me to basically follow my own ideas because it wasn't easy, right?
24:45
So maybe that's what made me a little original. And then at the end of my studies, I wanted to pursue, I had this idea of how you could train multiple systems. It wasn't quite working yet, but I think I could figure it out eventually. So I decided maybe the best way to do this is to do graduate studies.
25:06
And I stumbled on a second book in a bookstore, the French equivalent of Barnes and Noble when it still existed. And that book was, again, the transcript of some symposium by people who are interested
25:22
in self-organizing systems, the fact that you can have very simple elements that are interconnected and you will see emergent phenomena. Some people call this automata networks. And I realized a few of them are in Paris, so I called them up and said, I'm interested in this. Can I come see you? Yeah, sure, come.
25:41
So I'm talking to them and they tell me, well, you should sign up for a graduate program. And go to that school, I know the professor who runs the program, it's a bit late, but they'll take you. So I sign up. This is for a degree called a DOI, which at the time was like the first year of preliminary
26:06
to a PhD. Kind of a master's in our terms. Sort of a master's, but it's like oriented towards research. So the qualifying year for a PhD, the first year of PhD, we take courses. So I do this.
26:22
And in the course of that year, using the computers from my engineering school where I had graduated, but I had, they were nice enough to keep me in office there, I implemented this algorithm that I had the idea of before, and it does have to kind of work a little
26:40
bit. It was around 1983, 1984. So then, that's when you're programmed, so now I have to sign up for a PhD. And it's this curious thing in France, which is that to be the advisor of a PhD student, you have to be a full professor.
27:00
You have to be a full professor who had the ability to assume, which is the kind of thing that allows you to advise PhD students. And most of the people in this little lab I was talking to, it's kind of an independent lab called the Laboratoire de Dénamique des Resolvers, LDR. The people there were mostly not full professors.
27:21
There's only one who was full professor, but he was not working on machine learning at all. His name was Maurice Milgram. So I go talk to him, and I say like, I want to do a PhD. You are the only full professor. Can I sign up with you? You know, I won't need to ask you much because I have a scholarship from my engineering school. They give me an office. They give me a computer. You know, I can use their big computers.
27:42
I just need a bit of your advice. And he says, well, you need like a smart enough guy. I have no idea what you're working on, but I'm ready to sign the papers. So the one professor who would make this possible had a good attitude about what he didn't know and what you might learn. Exactly. He was excited?
28:01
He said, you know, I'll be helpful, but I can't help you technically. You can talk to the other members of the lab maybe who might be more helpful. So another member of the lab was Francois-Sueil Faujalman, who was an associate professor, so she could not take on PhD students, but she was more directly interested.
28:21
She understood the kind of inquiry you were interested in. Right. So she was the person I was interacting with the most at the time. And she helped me meet people from the US and other places that I wanted to meet. So in 1983, a paper came out written by Jeff Hinton and Terry Sinofsky, whose title was
28:47
Optimal Perceptual Inference. And the whole paper is kind of coded to hide the fact that it's a neural net, okay? Because otherwise it would never have passed the peer review process. And but I read this paper and I said, these guys get it.
29:03
Like they understand that the whole thing, first is learning, second of all this is this idea of training neural nets with, you know, they call them hidden units, you know, neural nets with multiple layers. I said, these are the people I want to meet the most in the world. Around 1984, 83, 84.
29:24
And so... You made direct contact with them? Well, not really. I couldn't. There was no email at the time. You had to write people, right? I mean, that only came a few years later. So I actually did an internship with Francois at Xerox PARC.
29:44
So he gave me like a first taste of research in the US. I went to Xerox PARC. I spent about a month there kind of doing research or whatever, you know, talking to people there. And I was really impressed with the way research was carried out in the US.
30:02
This is while you were a doctoral student. You haven't completed your degree at all. I was a first year. The first year, yeah. Yeah, it must have been 1984. And I didn't get to meet any of those people. I came back and then in 1985, early 1985, Francois and a couple of other people organized
30:25
what I think became kind of the first workshop in France about neural nets, broadly speaking, if you want. And they invited Terry Sinowski. They invited John Hopfield, which was a big figure of the field and a lot of famous neuroscientists and physicists and mathematicians and computer scientists and engineers.
30:44
And that was my first contact with the scientific community. And I had to give a talk in English and I was absolutely terrified. I was particularly terrified because in the room, there was two people in the room, one guy that looked nice enough.
31:01
And another guy who looked like a cowboy from Arizona. He had like a denim jacket and jeans and he had like big sideburns. And he was a young guy. He just came out with a PhD or something.
31:20
And anyone, he was a very senior person, would give a talk and at the end of the talk, he would raise his hand and ask a devastating question. Which was usually very well thought out, but truly devastating. And for me, in French culture, you don't do this, right?
31:41
I mean, in the U.S., yes, but this was... European culture in general, you don't do this. Right, even less in Asian culture, but okay. So I asked Francois, who are these guys? I say, oh, these guys are from Bell Labs. You know, Bell Labs, whenever you talk about something, it's already been done 10 years ago at Bell Labs, or it doesn't work.
32:02
It's like, okay. And the other Bell Labs guy gives a talk that I found really impressive on fabricating electronic neural nets. And it sort of merged the two things I was interested in, chips and neural nets. So I was fascinated by these guys and I talked to them at the end. And they listened to my talk, okay. So I give my talk in broken English.
32:20
Nobody understands a word. Not necessarily the English word that came out of my mouth, but the stuff I was telling people, you know, we need multi-layer nets. And this was completely new to the assembled... Mostly, yeah. Wow. Not bad for a young scientist. Anyway. Right. They didn't quite see the point, for most of them at least.
32:43
And you know, my talk, this guy, the cardboard guy... Tex, or whatever. He's from Arizona. You know, raises his hand. And I liquefy, literally, okay. You see this movie, Emily, right? There is this scene where, you know, her, you know, the guy she wants to talk to, you
33:06
know, comes into her bistro and she liquefies, literally. So that's what happened to me, essentially, but virtually. But not because of the question, but because he was raising his hand.
33:21
Yes. And he was about to probably do what he did to everybody else before, including a very senior scientist. Right, right. He's going to humiliate you. That's right. Prove that you're wrong. Right. And he says something nice. He says, oh, I finally understood something. Two years later, they were hiring me at Bell Labs.
33:43
This was the same group. So it was... Two years later, and I know this is very important. I want to talk about your Bell Labs experience. But you have to get your degree first, or they hire you before... No, no, I have to get my degree. And then I do a postdoc with Jeff Hinton's lab. And while I was a postdoc, they tell me, like, you know...
34:01
That's when you're invited. Is it important to talk about what your dissertation is, or is that not... OK. So what did you decide to write about? So the thing I talked about there was this learning algorithm for multilayer neural nets, which was not very sort of well thought out mathematically, but it had something to it.
34:23
It's equivalent to what we now call target props. So it's a class of algorithms for training neural nets, which is not very widely used, but you could think of it as kind of a predecessor of this. And it took me a long time to write the paper, because there was the meeting,
34:41
and then people were asked papers about it, and the book came out like a year later. In the meantime, though... So what happened is, Terry Sinofsky comes to that meeting, and he was not there for my talk. He arrived later. He gave a talk about Boltzmann machines, which was another example of such an algorithm.
35:00
And then I corner him at a break, and I tell him, like, I got something to tell you what I'm working on. And he listens patiently. So I explain to him what I'm doing. At the time, he was already working on using back prop, so the particular training algorithm for a particular application.
35:21
And back prop had been sort of proposed by Jeff, but Jeff made it work, but it was originally proposed by David Warmerhart at UC San Diego. And they were already working on it. They hadn't published it yet. And so he completely kept silent. I told him what I was working on. He didn't say he was working on very similar things. He went back to the U.S., and then told Jeff,
35:44
there's a kid in France who's working on the same stuff we're doing. A few months later comes a conference, like a bigger conference in Paris, which also is on cognitive science neural nets, cognitive in 1985.
36:02
And this time, Jeff Hinton is the keynote speaker. So Jeff arrives, gives his keynote speech about Boltzmann machines. And Boltzmann machines had become kind of a hot topic. This was June 1985. And so he's surrounded by 100 people who want to talk to him. And he's standing next to one of the organizers of the conference.
36:24
And he turns to the conference organizer, and he says, do you know guy called Yann LeCun? And I was far away, and I say, I'm here. And what happened is that he had read my paper in the proceedings, which was badly written in French,
36:42
and figured out this was similar to the stuff he was doing. And then he tells me, Terry told me there was a kid in France who was working on the same thing. Is it you? Say yes, it's me, because I talked to him a few months ago. So we kind of connected. So we met for lunch. We had couscous.
37:02
And he explained to me what he was working on. And I said, you don't have to explain to me. I know exactly what you're working on, because I'm doing the same thing. And so we realized we were basically completely aligned in the kind of stuff we were thinking of. And he said, well, I'm writing a paper. I'm going to cite yours. And I said, I was on cloud nine,
37:23
because that's the best thing that can happen to you. And it's very generous of him. It's very generous of him, yeah. And then he says, I'm organizing a summer school next summer. Why don't you come? It's going to be the start of a new community of neural nets. And I went there.
37:41
So that was summer 1986. Oh, before that, Terry Sinofsky uses back propagation to train a neural net to learn to read, so to basically translate English text into spoken words using a speech synthesizer. It's a difficult problem in English,
38:01
because it's hard to pronounce English, right? I mean, to know how to pronounce each sequence of letters. It's trivial in French, but it's very hard in English. And so he demonstrated that system. This is one of the first sort of compelling demonstrations that neural nets and also neural nets
38:21
were working very well. And he goes around the world, Europe, giving talks about it. He comes to Paris in the summer of 1986, or spring, and talks about it. And everybody is floored. And all of a sudden, everybody talks to me, because I'm the only person in France who knows how to train those networks.
38:41
You know, because I worked on them for a few years, and nobody was paying attention. But all of a sudden, everybody is paying attention. What is the core insight, I mean, spoken to a layman, that was demonstrated by what he was proving about speech? What, because this was a turning point in terms of people's acceptance.
39:02
Yeah. What confounded them or surprised them or turned them around about the value of the neural net? Well, so he was a task that is relatively complicated, which is very difficult to reduce into rules.
39:21
Like if you want to write a rule-based system that figures out how to pronounce English, you have to have lots of exceptions, you have to do parsing, and it's complicated. So it's a very complex rule. You have to take context into account to figure out how you pronounce each letter. And what he demonstrated was that with, you know, a reasonably small amount of training samples,
39:44
so you show a sequence of letters, and you tell the system, here is a sequence of phonemes that that corresponds to. You train the machine to map one to the other, and it works. Okay. It's a complex function they wouldn't be able to learn with sort of more traditional methods that were around before. So that showed the power of machine learning
40:01
into doing things that you would otherwise have done with rules and, you know, explicit programming. Perfect. So you're the only one who's working on this in France. Yeah, so I have a piece of code that can do this, and nobody else has it, right? Which nobody was interested in until then. Has this become essentially at the core of your dissertation?
40:22
Yeah, right. So in my dissertation, I talk about this backpropagation algorithm and various versions of it that I come up with earlier. Yes. The one that I published in 1985, among others, this target propagation. And then sort of various applications of this to various data sets. But it was very difficult to come by,
40:42
I mean, to obtain data sets at the time. So I experimented with simple shape recognition by drawing little digits with a computer mouse, basically. I had a data set that I obtained from a hospital
41:03
which was a description of symptoms of people who have belly pain, essentially, belly ache, together with the diagnosis, right? Is it appendicitis? Is it are your guts clogged?
41:20
Or something like that, right? Or there's 22 different diseases this could be caused by. And I trained a neural net to do this, and it was working decently well. And I tried a couple of the things, like detecting the transition between introns and exons in DNA sequences.
41:41
So this is things that people now do routinely in bioinformatics. So I put my hands on a few of those data sets and kind of showed that it could work. But it remained, I was never very good at writing papers,
42:01
particularly in English, and so I never actually published this as much as I should have. Really? Yeah. And I graduated, I broke my ankle just a couple of months before, so I had to defend my thesis sitting down, and then went to Toronto for a postdoc in summer 1987.
42:26
And I arrived in Toronto exactly three weeks after Jeff Hinton, who had just moved from Carnegie Mellon University. He'd become a full professor in Toronto, and so he just moved. And I worked in Jeff's lab for about a year
42:41
before going to Bell Labs. Now, in part, this is the biography of your curiosity, as I'd like to put it, but it's also the biography of your career. So at this point, are you thinking consciously, are you beginning to think in a professional context, do I want to work in the university, or do I want to work in industry,
43:02
or is it just that this invitation comes to you and that decides you? Invitation comes to me, and that's the obvious path, because I want to work with Jeff, right? I mean, he's the obvious person to work with. He invites me to do a postdoc with him,
43:22
because in 1986, I went to the summer school at Carnegie Mellon University. I met a whole bunch of students who had the same interests and were about my age. And there, Jeff says, I'm moving to Toronto next year. Would you like to do a postdoc with me? I get that part, but going to Bell Labs, with the invitation that follows
43:41
upon the two-year before discussion, isn't that in danger of taking you outside of an academic career, or at this point? So I never thought of myself as necessarily wanting to go to academia. Okay. Okay. This wasn't something that I'd thought about,
44:02
like, you know, do I want to be a professor? The type of, what it means to be a professor in France is not quite the same as in the US, and in France, that was not very attractive to me. And so I never thought of myself as, you know, going to teach. So there was no tortured moment
44:21
when the opportunity comes to go to Bell Labs. That's right. I mean, it was, you know, I didn't have to make a choice, right? They offered me a job, and I said, sure. So, but two things happened. Yes. In 1986, when I go to the summer school, somehow the people at Bell Labs, the same people at Bell Labs I met a year earlier,
44:42
learned that I'm in the US. And they called me up in Pittsburgh, another county in Maryland, and they say, we don't know what your plans are, but can you stop in New Jersey on your way back and come visit us? I said, sure. I planned to, like, spend a day in New York because I'd never been to New York.
45:01
And I said, you know, I'm canceling my visit to New York. I'm just going to go to New Jersey. So I went to Holmdel, New Jersey, to the Bell Labs location where they worked. And I spent the day there. I talked to them, and they seemed to be very impressed by my talk. And, you know, I said, that's great. Let's keep in touch.
45:21
And they want to offer me a job, but I tell them, I'm probably going to do a postdoc in Toronto. They say, okay. And they didn't tell me at the time, but they decided to hold off. They wanted to hire me. I mean, as soon as I graduated. But they sort of decided to hold off
45:40
so as not to poach universities from postdoc. So I go back to France. Another year passed. I graduate. I meet one of my closest friend and collaborators, a gentleman by the name of Leon Betou. So I was finishing my PhD. I was in the last semester. I was writing my thesis, basically.
46:02
And he calls me up. He's a student in the last year of École Polytechnique, which is one of the elite schools in France. He says, I'm interested in neural nets, and I have to do a project for my last year, and I want to work with you. I said, sure. He didn't tell his school that I didn't have a PhD yet,
46:23
because otherwise they would not have accepted. And we meet, and we realize we both use the same kind of obscure personal computer. So he says, well, it's going to be easy for us to communicate, because we can develop on the same machine. This was a Commodore Amiga, which was like the most amazing computer at the time,
46:42
personal computer. And I tell him, I'm writing a software, a neural net simulator, a new version of a neural net simulator, and I need someone to write a Lisp interpreter as the interacting language with that. He goes away, and he comes back three weeks later
47:02
with a completely implemented Lisp interpreter. I say, ooh, this guy's good. And it's time for me to go to Toronto, so I take his unfinished piece of software, I complete it when I'm in Toronto, and that piece of software gave him and me superpowers.
47:23
We were the only people who had the tool that allowed us to basically train neural nets in a particular way, and particularly to build what later became convolutional neural nets, which was the thing that everybody now uses for image recognition. So the tools you build for yourself as a scientist
47:40
give you superpowers. That's super important. And a lot of scientists you talk to will tell you this. Famous neuroscientists became famous because they invented a new technique for staining neurons or something. It's the same in art. Famous painters are also people who invented new pigments and new techniques for mixing pains, right? Yes, absolutely, no question.
48:01
So it's the same thing. So I developed this software while I was in Toronto, and Jeff was slightly annoyed with this because I was spending my time hacking and not actually doing science, right? But I knew what I was doing. I came up with a few interesting things, and then I joined Bell Labs, and I had this software ready
48:20
with prototypes of convolutional nets, which is the thing I was dreaming of implementing for a long time. And they have thousands of examples of handwritten digits that they collected from the US Postal Service. Within two months, I tried out, and I beat everything, like all previous methods that I tried on it
48:40
by a fairly large margin. Within two or three months of being there. So there's a lot of excitement. That was one of the most exciting times, I think, of my life in terms as a research scientist. At this point in Bell Labs, this is still the golden age. This is 1988.
49:01
So this is October, November, December 1988. This is four years after divestiture of AT&T. So four years after AT&T was broken up into the original Bell companies, and it became the long-distance company. But the way Bell Labs was operating was still very, very good, very interesting.
49:20
I mean, your career now will be at Bell Labs. I mean, we don't have a lot more time, but this is such an important period. How many years before you come to NYU? Essentially, you're there until you come to NYU. So I was a research scientist at Bell Labs until 1996. So I joined in end of 1988, until 1996.
49:42
In 1996, AT&T break itself up again. Yes. With Lucerne Technologies and others, other companies. And I stay with AT&T in the part of Bell Labs that became AT&T Labs. And I'm promoted. Because the projects I've worked on on handwriting recognition have become really successful,
50:01
so I'm promoted to department head. So now I have my own group. I can start new research projects. And this is just a time when the machine learning research community loses interest in neural nets. Exactly. It's another winter coming. It's the winter of neural nets. It's not a winter of machine learning or anything. Yeah, yeah, but neural nets. People lose interest in neural nets, partly because we were able to make them work
50:23
because we had this piece of software. We could not distribute it in open source. Nobody else has it. And so these neural nets have the reputation that only Yann LeCun can train them, which is not true, of course. But it's just because we had this whole system. I see. That we developed over years with Leon
50:40
and then other colleagues at Bell Labs. So I decide to work on something else. It's the new headates of the internet, 1996. And I work on something else, a system called DJVU, which is an image compression technology. And I lead this project for about five years.
51:01
And in the end of 2001, AT&T runs into financial trouble and decides it doesn't have the money to maintain a large research lab anymore. And so at the end of 2002, basically, they tell us, you're all fired. But we knew that was coming. So in the meantime, I decided to leave AT&T
51:21
and go to the NEC Research Institute in Princeton. So this belongs to the Japanese company, NEC. I only stayed there 18 months. I brought most of my team from AT&T, so Leon Betou, Vladimir Vapnik, and others. And I joined NYU in 2003. So we come to the end, although it's just
51:42
the beginning of your current work. As you come to NYU, I won't ask you about the present. You've been interviewed about that. But as you come to there, what is the status and state of the neural inquiry at this point?
52:01
Well, so not very good. Neural nets are still taboo to some extent. I'm very thankful to NYU and the NYU department, the chairwoman in particular, was a former colleague from Bell Labs. We didn't know each other very well. But NYU wanted to get into machine learning. And I was a senior person.
52:20
I could attract other people, despite what I was working on. I had some success that could be measurable. So then I decided to completely restart a research program on neural nets and basically changing the opinion of the community towards those models. And I get together with Jeff and Yoshua.
52:42
Jeff gets some money from a Canadian foundation called CIFAR. And the three of us decide that we should deliberately try to renew the interests of the community for those methods by showing that they work, by inventing new algorithms that are more powerful
53:01
and by showing that they work on real problems. And that succeeded beyond our wildest dreams. That will be the last word. Thank you very much. Thank you.
Empfehlungen
Serie mit 66 Medien