AMA: Yann LeCun

[u/ylecun]

My name is Yann LeCun. I am the Director of Facebook AI Research and a professor at New York University.

Much of my research has been focused on deep learning, convolutional nets, and related topics.

I joined Facebook in December to build and lead a research organization focused on AI. Our goal is to make significant advances in AI. I have answered some questions about Facebook AI Research (FAIR) in several press articles: Daily Beast, KDnuggets, Wired.

Until I joined Facebook, I was the founding director of NYU's Center for Data Science.

I will be answering questions Thursday 5/15 between 4:00 and 7:00 PM Eastern Time.

I am creating this thread in advance so people can post questions ahead of time. I will be announcing this AMA on my Facebook and Google+ feeds for verification.

[u/[deleted]]

What is your team at Facebook like?

How is it different then your team at NYU?

In your opinion, why have most renowned professors (eg. yourself, Geoff Hinton, Andrew Ng) in deep learning attached themselves to a company?

Can you please offer some advice to students who are involved with and/or interested in pursuing deep learning?

[u/ylecun → u/[deleted]]

My team at Facebook AI Research is fantastic. It currently has about 20 people split between Menlo Park and New York, and is growing quickly. The research activities focus on learning methods and algorithms (supervised and unsupervised), deep learning + structured prediction, deep learning with sequential/temporal signals, applications in image recognition, face recognition, natural language understanding. An important component is ML software platform and infrastructure. We are using Torch7 for many projects (as does Deep Mind and several groups at Google) and will be contributing to the public version.

My group at NYU used to work a lot on applications in vision/robotics/speech (and other domains) when the purpose was to convince the research community that deep learning actually works. Although we still work on vision, speech and robotics, now that deep learning has taken off, we are doing more work on theoretical stuff (e.g. optimization), new methods (e.g. unsupervised learning) and connections with computational neuroscience and visual psychophysics.

Geoff Hinton is at Google, I'm at Facebook, Yoshua Bengio has no intention of joining an industrial lab. The nature of projects in industry and academia is different. Nobody in academia will come to you and say "Create a research lab, hire a bunch of top scientists, and try to make significant progress towards AI", and no one in academia has nearly as much data as Facebook or Google. The mode of operation in academia is very different and complementary. The actual work is largely done by graduate students (who need to learn, and who need to publish papers to get their career on the right track), the motivations and reward mechanisms are different, the funding model is such that senior researchers have to spend quite a lot of time and energy raising money. The two systems are very complementary, and I feel very privileged to be able to maintain research activities within the two environments.

A note on Andrew Ng: Coursera keep him very busy. Coursera is a wonderful thing, but Andrew's activities in AI have taken a hit. He is no longer involved with Google.

Advice to students: if you are an undergrad, take as many math and physics course as you can, and learn to program. If you are an aspiring grad student: apply to schools where there is someone you want to work with. It's much more important that the ranking of the school (as long as the school is in the top 50). If your background is engineering, physics, or math, not CS, don't be scared. You can probably survive qualifiers in a CS PhD program. Also, a number of PhD programs in data science will be popping up in the next couple of years. These will be very welcoming to students with a math/physics/engineering background (who know continuous math), more welcoming than CS PhD programs.

Another advice: read, learn from on-line material, try things for yourself. As Feynman said: don't read everything about a topic before starting to work on it. Think about the problem for yourself, figure out what's important, then read the literature. This will allow you to interpret the literature and tell what's good from what's bad.

Yet another advice: don't get fooled by people who claim to have a solution to Artificial General Intelligence, who claim to have AI systems that work "just like the human brain", or who claim to have figured out how the brain works (well, except if it's Geoff Hinton making the claim). Ask them what error rate they get on MNIST or ImageNet.

[u/BeatLeJuce]

1. We have a lot of newcomers here at r/MachineLearning who have a general interest in ML and think of delving deeper into some topics (e.g. by doing a PhD). What areas do you think are most promising right now for people who are just starting out? (And please don't just mention Deep Learning ;) ).

2. What is one of the most-often overlooked things in ML that you wished more people would know about?

3. How satisfied are you with the ICLR peer review process? What was the hardest part in getting this set up/running?

4. In general, how do you see the ICLR going? Do you think it's an improvement over Snowbird?

5. Whatever happened to DJVU? Is this still something you pursue, or have you given up on it?

6. ML is getting increasingly popular and conferences nowadays having more visitors and contributors than ever. Do you think there is a risk of e.g. NIPS getting overrun with mediocre papers that manage to get through the review process due to all the stress the reviewers are under?

[u/ylecun → u/BeatLeJuce]

Question 2:

There are a few things:

• kernel methods are great for many purposes, but they are merely glorified template matching. Despite the beautiful math, a kernel machine is nothing more than one layer of template matchers (one per training sample) where the templates are the training samples, and one layer of linear combinations on top.

• there is nothing magical about margin maximization. It's just another way of saying "L2 regularization" (despite the cute math).

• there is no opposition between deep learning and graphical models. Many deep learning approaches can be seen as factor graphs. I posted about this in the past.

[u/ylecun → u/BeatLeJuce]

Question 6:

No danger of that. The main problem conferences had is not that they are overrun with mediocre papers. It is that the most innovative and interesting papers get rejected. Many of the papers that make it passed the review process are not mediocre. They are good. But they are often boring.

I have explained why our current reviewing processes are biased in favor of "boring" papers: papers that bring an improvement to a well-established technique. That's because reviewers are likely to know about the technique and to be interested in improvements of it. Truly innovative papers rarely make it, largely because reviewers are unlikely to understand the point or foresee the potential of it. This is not a critique of reviewers, but a consequence of the burden they have to carry.

An ICLR-like open review process would reduce the number of junk submissions and reduce the burden on reviewers. It would also reduce bias.

[u/somnophobiac]

How would you rank the real challenges/bottlenecks in engineering an intelligent 'OS' like the one demonstrated in the movie 'Her' ... given current challenges in audio processing, NLP, cognitive computing, machine learning, transfer learning, conversational AI, affective computing .. etc. (i don't even know if the bottlenecks are in these fields or something else completely). What are your thoughts?

[u/ylecun → u/somnophobiac]

Something like the intelligent agent in "Her" is totally out of reach of current technology. We will need to invent new concepts, new principles, new paradigms, new algorithms.

The agent in Her has a deep understanding of human behavior and human nature. It's going to take quite a while before we build machines that can do that.

I think that a major component we are missing is an engine (or a paradigm) that can learn to represent and understand the world, in ways that would allow it to predict what the world is going to look like following an event, an action, or the mere passage of time. Our brains are very good at learning to model the world and making predictions (or simulations). This may be what gives us 'common sense'.

If I say "John is walking out the door", we build a mental picture of the scene that allows us to say that John is no-longer in the room, that we are probably seeing his back, that we are in a room with a door, and that "walking out the door" doesn't mean the same thing as "walking out the dog". This mental picture of the world and the event is what allows us to reason, predict, answer questions, and hold intelligent dialogs.

One interesting aspect of the digital character in Her is emotions. I think emotions are an integral part of intelligence. Science fiction often depicts AI systems as devoid of emotions, but I don't think real AI is possible without emotions. Emotions are often the result of predicting a likely outcome. For example, fear comes when we are predicting that something bad (or unknown) is going to happen to us. Love is an emotion that evolution built into us because we are social animals and we need to reproduce and take care of each other. Future AI systems that interact with humans will have to have these emotions too.

[u/[deleted]]

Hi Dr. LeCun, thanks for taking the time!

1. You've been known to disagree with the long-term viability of kernel methods for reasons to do with generalization. Has this view changed in light of multiple kernel learning and/or metric learning in the kernel setting?

2. How do you actually decide on the dimensionality of learnt representations, and is this parameter also learnable from the data? Every talk I hear where this is a factor it's glossed over by something like, "representations are real-vectors in 100-150 dimensions * next slide *".

3. If you can be bothered, I would love to hear how you reflect on the task of ad click prediction; nothing in human history has been given as much time and effort by so many people as advertising, and I think it's safe to say that if you're a new human born into a geographical location selected at random, you have a higher probability of encountering the narrative of 'stuff you must consume' than any other narrative. Is this something we should be doing as a species?

Thank you so much for the time and for all your work!

[u/[deleted]]

What set of skills is your team seeking for and how can someone join your team?

Is your team focused on theoretical research, applied research or both? Can you give us some examples of what kind of problems you are working on?

[u/ylecun → u/[deleted]]

We are looking (mostly) for people with a PhD and a strong publication record in machine learning, AI, computer vision, natural language processing, applied mathematics, signal processing, and related fields.

We are also recruiting a small number of engineers (Master and PhD level) for technology development. Much of that (though not all of it) occurs through internal transfer within Facebook.

Facebook AI Research has activities that span the full spectrum from theory, applied mathematics (e.g. optimization, sparse modeling), to principles and methods, to algorithms, to applications, software tools, and software/hardware platforms.

Speaking of platform, much of our work is done with Torch7.

[u/dabunk]

What fields do you think are most in need of incorporating deep learning? Or in other words, where would you like to see your work in deep learning applied?

[u/ylecun → u/dabunk]

Deep learning has become the dominant method for acoustic modeling in speech recognition, and is quickly becoming the dominant method for several vision tasks such as object recognition, object detection, and semantic segmentation.

The next frontier for deep learning are language understanding, video, and control/planning (e.g. for robotics or dialog systems).

Integrating deep learning (or representation learning) with reasoning and making unsupervised learning actually work are two big challenges for the next several years.

[u/flyingdragon8]

1. Do you think there are any gains to be had in hardware-based (partially programmable and interconnectible) deep NN's?

2. How would you advise someone new to ML attempt to understand deep learning on an intuitive level? i.e. I understand generally that a deep net tries to learn a complex function through a sort of gradient descent to minimize error on a learning set. But it is not immediately intuitive to me why some problems might be amenable to a deep learning approach, why some layers are convolutional and some are fully connected, why a particular activation function is chosen, and just generally where the intuition is in designing neural nets (and whether to apply them at all in the first place).

[u/akkhong]

I am particularly interested in scaling up Bayesian methods to large datasets. Are you optimistic at all about the possibility of more widespread use of Bayesian methods for large datasets? Do you think that there is a place for MCMC sampling in the future of machine learning?

[u/[deleted]]

Hi! I have two questions at the moment.

1. What do you think are the biggest applications machine learning will see in the coming decade?

2. How has the recent attention "Big data" has gotten in the media affected the field? Do you ever feel like it might be overly optimistic or that some criticism is overly pessimistic?

[u/ylecun → u/[deleted]]

1. Natural language understanding and natural dialog systems. Self-driving cars. Robots (maintenance robots and such).

2. I like the joke about Big Data that compares it to teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it.

Seriously, I don't like the phrase "Big Data". I prefer "Data Science", which is the automatic (or semi-automatic) extraction of knowledge from data. That is here to stay, it's not a fad. The amount of data generated by our digital world is growing exponentially with high rate (at the same rate our hard-drives and communication networks are increasing their capacity). But the amount of human brain power in the world is not increasing nearly as fast. This means that now or in the near future most of the knowledge in the world will be extracted by machine and reside in machines. It's inevitable. En entire industry is building itself around this, and a new academic discipline is emerging.

[u/chcampb]

What do you believe is the most promising artificial general intelligence project in the works?

[u/ylecun → u/chcampb]

I don't think any of them is promising right now. AFAICT, no one has a proposal (let alone a prototype or a system) that has a good shot at succeeding today.

But a lot of smart people and well-funded companies are taking this on seriously. So things may change in the next few years. But don't hold your breath.

[u/LionSupremacist]

Do you think having a PhD is important if you want to work in a good research team in the industry?

[u/ylecun → u/LionSupremacist]

Yes.

[u/Dtag]

I actually have two questions:

1. When I heard about Deep Learning for the first time, it was in Andrew Ng's Google Tech Talk. He talked about unsupervised layer-wise training, forced sparsification of layers, noisy autoencoders etc., really making use of unsupervised training. A few others like Hinton argued for this approach and said that backprop suffers from gradient dilution, and the issue that theres simply not enough training data to ever constrain a neural net properly, and argued why backprop does not work.

At the time, that really felt like something different and new to use these unsupervised, layer-wise approaches, and I could see why these approaches work where others have failed in the past. As the research in that field intensified, people appeared to rediscover supervised approaches, and started using deep (convolutional) nets in a supervised way. It seems that most "Deep Learning" approaches nowadays fit in this class.

Am I missing something here? Is it really the case that you can "make backprop work" by just throwing huge amounts of data and processing power at the problem, despite problems like gradient dilution etc (mentioned above)? Why has the idea of unsupervised training not (really) taken off so far, despite the initial successes?

2) We presently use loss functions and some central learning algorithm for training neural networks. Do you have any intuition about how the human brain's learning algorithm works, and how it is able to train the net without a clear loss function or a central training algorithm?

[u/ylecun → u/Dtag]

1. You are not missing anything. The interest of the ML community in representation learning was rekindled by early results with unsupervised learning: stacked sparse auto-encoders, RBMs, etc. It is true that the recent practical success of deep learning in image and speech all use purely supervised backprop (mostly applied to convolutional nets). This success is largely due to dramatic increases in the size of datasets and the power of computers (brought about by GPU), which allowed us to train gigantic networks (often regularized with drop-out). Still, there are a few applications where unsupervised pre-training does bring an improvement over purely supervised learning. This tends to be for applications in which the amount of labeled data is small and/or the label set is weak. A good example from my lab is pedestrian detection. Our CVPR 2013 paper shows a big improvement in performance with ConvNets that unsupervised pre-training (convolutional sparse auto-encoders). The training set is relatively small (INRIA pedestrian dataset) and the label set is weak (pedestrian / non pedestrian). But everyone agrees that the future is in unsupervised learning. Unsupervised learning is believed to be essential for video and language. Few of us believe that we have found a good solution to unsupervised learning.

2. It's not at all clear whether the brain minimizes some sort of objective function. However, if it does, I can guarantee that this function is non convex. Otherwise, the order in which we learn things would not matter. Obviously, the order in which we learn things does matter (that's why pedagogy exists). The famous developmental psychologist Jean Piaget established that children learn simple concepts before learning more complex/abstract ones on top of them. We don't really know what "algorithm" or what "objective function" or even what principle the brain uses. We know that the "learning algorithm (or algorithms) of the cortex" plays with synapses, and we know that it sometimes looks like Hebbian learning of Spike-Timing Dependent Plasticity (i.e. a synapse is reinforced when the post-synaptic synapse fires right after the pre-synaptic synapse). But I think STDP is the side effect of a complex "algorithm" that we don't understand. Incidentally, backprop is probably not more "central" than what goes on in the brain. An apparently global effect can be the result of a local learning rule.

[u/ResHacker]

I always find it hard to perceive the general purpose of unsupervised learning. Do you think there exists a unified criterion to judge whether an unsupervised learning algorithm is effective? Is there any other way to objectively evaluate how good an unsupervised learning algorithm is, other than just using supervised learning to see how good the unsupervisedly learnt 'features' are?

[u/ylecun → u/ResHacker]

Interesting question. The fact that this question has no good answer is what kept me away from unsupervised learning until the mid 2000s.

I don't believe that there is a single criterion to measure the effectiveness of unsupervised learning.

Unsupervised learning is about discovering the internal structure of the data, discovering mutual dependencies between input variables, and disentangling the independent explanatory factors of variations. Generally, unsupervised learning is a means to an end.

There are four main uses for unsupervised learning: (1) learning features (or representations); (2) visualization/exploration; (3) compression; (4) synthesis. Only (1) is interesting to me (the other uses are interesting too, just not on my own radar screen).

If the features are to be used in some sort of predictive model (classification, regression, etc), then that's what we should use to measure the performance of our algorithm.

[u/urish]

Where would you place Facebook Research on the spectrum between pure academic-like research and building products for Facebook?

Would it be in the vein of Bell Labs and MSR, or more similar to Google Research, or perhaps something completely different?