r/MachineLearning • u/[deleted] • Feb 24 '14

AMA: Yoshua Bengio

[deleted]

202 Upvotes

98% Upvoted

208 comments sorted by

15

u/Sigmoid_Freud Feb 24 '14

Traditional (deep or non-deep) Neural Networks seem somewhat limited in the sense that they cannot keep any contextual information. Each datapoint/example is viewed in isolation. Recurrent Neural Networks overcome this, but they seem to be very hard to train and have been tried in a variety of designs with apparently relatively limited success.

Do you think RNNs will become more prevalent in the future? For which applications and using what designs?

Thank you very much for taking your time to do this!

15

u/yoshua_bengio Feb 26 '14

Recurrent or recursive nets are really useful tools for modelling all kinds of dependency structures on variable-sized objects. We have made progress on ways to train them and it is one of the important areas of current research in the deep learning community. Examples of applications: speech recognition (especially the language part), machine translation, sentiment analysis, speech synthesis, handwriting synthesis and recognition, etc.

2

u/freieschaf Feb 24 '14

Take a look at Schmidhuber's page on RNNs. There is quite a lot of info on them, and especially on LSTMNN, an architecture of RNN designed precisely for tackling the issue of vanishing gradient when training RNNs and so allowing them to keep track of a longer context.

0

u/[deleted] Feb 25 '14

10

u/yoshua_bengio Feb 26 '14 edited Feb 27 '14

Liquid state machines and echo state networks do not learn the recurrent weights, i.e., they do not learn the representation. Instead, learning good representations is the central purpose of deep learning. In a way, the echo-state / liquid state machines are like SVMs, in the sense that we put a linear predictor on top of a fixed set of features. The features are functions of the past sequence through the smartly initialized recurrent weights, in the case of echo state networks and liquid state machines. Those features are good, but they can be even better if you learn them!

3

u/rpascanu Feb 27 '14

I would add that ESNs or LSMs can provide insights in why certain things don't work or work for RNNs. So having a good grasp of them could definitely be useful for deep learning. An example is Ilya's work on initialization (jmlr.org/proceedings/papers/v28/sutskever13.pdf‎), where they show that an initialization based on the one proposed by Herbert Jaeger for ESNs is very useful for RNNs as well.

They also offer quite a strong baseline most of the time.

21

u/exellentpossum Feb 24 '14

When asked about sum product networks, one of the original Google Brain team members told me he's not interested in tractable models.

What's your opinion about sum product networks? They made a big splash at NIPS one year and now they've disappeared.

8

u/yoshua_bengio Feb 26 '14

There are many kinds of intractabilities that show up in different places with various learning algorithms. The more tractable the easier to deal with in general, but it should not be at the price of losing crucial expressive power. I don't have a sufficiently clear mental fix on the expressive power of SPNs to know who much we lose (if any) through this parametrization of a joint distribution. In any case, all the interesting models that I know of suffer from intractability of minimizing the training criterion wrt the parameters (i.e. training is fundamentally hard, at least in theory). SVMs and other related kernel machines do not suffer from that problem, but they may suffer from poor generalization unless you provide them with the right feature space (which is precisely what is hard, and what deep learning is trying to do).

3

u/celestec Feb 25 '14

Hi exellentpossum, I am studying some machine learning on my own, and have not yet come across "tractable models." What exactly is a tractable model? (Searching on my own didn't help much...) Sorry if this is a dumb question.

3

u/exellentpossum Feb 25 '14

In the context of sum product networks, it means that inference is tractable or doesn't suffer from the exponential growth in computational cost when you add more variables.

This comes at a price though, sum product networks can only represent certain types of distributions. More specifically, probability distributions where its parameterization can be expressed as a product of factors (when multiplied out this creates a much larger polynomial). I'm not sure of the exact scope of distributions this encompasses, but it does include hierarchical mixture models.

3

u/Scrofuloid Feb 26 '14 edited Feb 26 '14

Not quite. All graphical models can be represented as products of factors, and deep belief networks and such are special cases of graphical models. Inference in graphical models is usually considered intractable in the treewidth of the graph. So, in conventional graphical model wisdom, low-treewidth graphical models were considered 'tractable', and high-treewidth models were 'intractable', so you'd have to use MCMC or BP or other approximate algorithms to solve them.

Any graphical model can be compiled into an SPN-like structure (an arithmetic circuit, or AC). The problem is that in the worst-case, the resulting circuit can be exponentially large. So even though inference is still linear in the size of the circuit, it's potentially exponential in the size of the original graphical model. But it turns out certain high-treewidth graphical models can still be compiled into compact circuits, so you can still do efficient inference on them. This means that there are certain high-treewidth graphical models on which inference is tractable -- kind of a surprise to the graphical models community.

You can think of ACs and SPNs as a way to compactly represent context-specific independences. They can compactly represent distributions that would result in high-treewidth graphical models if you tried to represent them in the usual graphical models way. The difference between ACs and SPNs is that ACs are compiled from Bayesian networks, as a means of performing inference on them. SPNs directly use the circuit to represent a probability distribution. So instead of training a graphical model and hoping you can compile it into a compact circuit (AC), you directly learn a compact circuit that fits your training data (SPN).

1

u/exellentpossum Feb 26 '14

I agree, SPNs can represent any probability distribution. But there is a certain set which can be represented efficiently. Can you be more specific about this set of distributions which can take advantage the factorization property of SPNs (a distribution with a reasonably sized circuit)?

0

u/richardabrich Feb 25 '14

Disclaimer: I am not an expert.

If you're familiar with complexity theory, tractable means that the solution can be calculated in polynomial time. If not, this simply means that it's possible to do on a modern computer in a reasonable amount of time.

One of the requirements for tractability in e.g. Bayesian networks is that nodes be conditionally independent, which allows us to "explain away" the causes of all of the other nodes in the net for a given node. Sum-Product networks can be computed in O(n) time regardless of conditional independence, thereby making them tractable.

0

u/[deleted] Feb 24 '14

1

u/exellentpossum Feb 25 '14

But the architecture is fascinating, the way backpropagation is applied to a tree is a very interesting way to look at a deep network. But I suppose one caveat I remember from the discriminative sum product network is the first layer of feature learning is separated from the 'deep learning' part.

42

u/[deleted] Feb 24 '14

Hello Prof. Bengio, What motivates you to stay in academia? What do you think about corporate research labs in terms of productivity and innovation when compared to academic labs. Does research flexibility (doing what you want, more or less) play a large role in this decision?

29

u/yoshua_bengio Feb 26 '14

I like academia because I can choose what to work on, I can choose to work on long-term goals, I can work for the benefit of humanity rather than for a specific company, and I can talk about my work freely. Note that to different degrees, my esteemed colleagues in large industrial labs also enjoy some of that freedom.

31

u/alecradford Feb 24 '14 edited Feb 24 '14

Hi there! I'm an undergrad and your work combined with Hinton's is a huge inspiration to me! A bunch of questions, so feel free to answer all or none!

Hinton semi-recently offered an awesome MOOC on Coursera over NNs. The resources and lectures it provided are what allowed me and many others to build homebrew nets and really get into the field. It would be a great resource if another researcher at the forefront of the field offered their own take, do you have any plans for something like this?

As a leading professor in the field, how do you personally view the resurgence of interest in modern NN applications? Do you believe it's well deserved recognition, guilty of overhype, some mixture of the two, or something completely different! On a similar note, how do you feel about the portrayal of modern NN research in popular literature?

I'm interested in using unsupervised techniques to learn automated data augmentations/corruptions for increasing generalization performance, which I hope is a promising hybrid of supervised and unsupervised learning that's different from traditional pretraining. A lot of advances have been made using "simple" data augmentations/corruptions pioneered in your lab like gaussian noise corruption and what we now call input dropout in the context of DAEs. Preliminary results on MNIST seem successful (~0.8% permutation invariant) and I can send code if you are interested but admittedly I'm just an undergrad with no formal research experience. Do you see this as an area with potential and could you point me to any resources or papers that you are aware of - I've had a hard time finding them.

No one has a crystal ball, but what do you see as the most interesting areas of research for continuing to advance your work? The last few years has seen purely supervised techniques make a lot of headroom riding off the success of dropout, for instance.

Thank you so much for doing this AMA, it's great to have you here on r/MachineLearning!

23

u/yoshua_bengio Feb 27 '14

I have no clear plan for a MOOC but I might do one eventually. In the meantime, I write a new and more complete book on deep learning (with Ian Goodfellow and Aaron Courville). Some draft chapters should come out in the next few months and feedback from the community and students would be great. Note that Hugo Larochelle (formerly a PhD with me and a post-doc with Hinton) has great videos on deep learning http://www.youtube.com/playlist?list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH (and slides on his web page).

I believe that the recent surge of interest in NNets just means that the machine learning community wasted many years not exploring them, in the 1996-2006 decade, mostly. There is also hype, especially if you consider the media. That is unfortunate and dangerous, and will be exploited especially by companies trying to make a quick buck. The danger is to see another bust when wild promises are not followed by outstanding results. Science mostly moves by small steps and we should stay humble.

I have no crystal ball but I believe that improving our ability to model joint distributions (either in an unsupervised way or conditioned on some input, either explicitly or implicitly through learning of good representations) is going to be crucial for future progress of deep learning towards AI-level machine understanding of the world around us.

Another easy prediction is that we need to and will make progress towards efficiently training much larger models. This involves improvements in the way we train model (the numerical optimization involved), as well as in ways to do it computationally more efficiently (e.g. through parallelization and other tricks that avoid doing the computation associated with all the parts of the network for every example).

You can find out more in my arxiv paper on "looking forward": http://arxiv.org/abs/1305.0445

13

u/PasswordIsntHAMSTER Feb 24 '14

Hi Prof. Bengio, I'm an undergrad at McGill University doing research in type theory. Thank you for doing this AMA!

Questions:

  • My field is extremely concerned with formal proofs. Is there a significant focus on proofs in machine learning too? If not, how do you make sure to maintain scientific rigor?

  • Is there research being done about the use of deep learning for program generation? My intuition is that eventually we could use type theory to specify a program and deep learning to "search " for an instantiation of the specification, but I feel like we're quite far from that.

  • Can you give me examples of exotic data structure used in ML?

  • How would I get into deep learning starting from zero? I don't know what resources to look at, though if I develop some rudiments I would LOVE to apply for a research position on your team.

9

u/yoshua_bengio Feb 27 '14

There is a simple way that you get scientific rigor without proof, and it's used throughout science: it's called the scientific method, and it relies and experiments and hypothesis-testing ;-) Besides, math is getting into more deep learning papers. I have been interested for some time in proving properties of deep vs shallow architectures (see papers with Delalleau, and more recently with Pascanu). With Nicolas Le Roux I worked on the approximation properties of RBMs and DBNs. I encourage you to also look at the papers by Montufar. Fancy math there.

Deep learning from 0? there is lots of material out there, some listed in deeplearning.net:

2

u/PokerPirate Feb 24 '14

On a related note, I am doing research in probabalistic programming languages. Do you think there will ever be a "deep learning programming language" (whatever that means) that makes it easier for nonexperts to write deep learning models?

5

u/ian_goodfellow Feb 27 '14

I am one of Yoshua's graduate students and our lab develops a python package called Pylearn2 that makes it relatively easy for non-experts to do deep learning:

https://github.com/lisa-lab/pylearn2

You'll still need to have some idea of what the algorithms are meant to be doing, but at least you won't have to implement them yourself.

2

u/serge_cell Feb 24 '14

IMHO definitely should be. There are several open source packages with similar functionality right now, and different research papers refer to different packages for results reproduction. Would be great if one wouldn't have to install and learn new package to reproduce result, but just use ready made cfg or script in dl language. Would improve reproducibility too - results reproduced with different implementation are more relatable.

1

u/PokerPirate Feb 24 '14

There are several open source packages with similar functionality right now

links?

3

u/serge_cell Feb 25 '14

I mostly familiar with convolutional networks, so most of packages here are for CNN and autoencoders
Fastest:

  1. cuda-convnet - most used gpgpu implementation, used in other packages too
    https://code.google.com/p/cuda-convnet/ there are also several forks on github

  2. caffe
    https://github.com/BVLC/caffe

  3. NNforge
    http://milakov.github.io/nnForge/
    Based on cuda-convnet, but include more staff:

  4. pylearn2
    https://github.com/lisa-lab/pylearn2
    other staff:
    http://deeplearning.net/software_links/

2

u/polyguo Feb 25 '14

What probabilistic programming languages are you researching? Any experience with Church? I have an internship this summer with someone who does research using PPLs and it would be immensely useful to me if you could point me to resources that would allow me to get more familiar with the subject matter. Papers and actual code would be best.

1

u/PokerPirate Feb 25 '14

Have you been to http://probmods.org? It's a pretty thorough tutorial.

2

u/polyguo Feb 25 '14

I'm actually taking the probabilistic graphical models course in Coursera and i got a copy of Koller's book. I'm familiar with the theory, I've yet to see mature code written in PPLs.

And, yes, I've been to the site. I'm actually going to be working with one of the authors.

2

u/orwells1 Feb 27 '14

Can't see a reply so this might help:

“There is a strong oral tradition in training neural networks so if you read the papers it will be hard to understand how to do it, really the best thing is to just spend a couple of years next to someone who does it and ask them a lot of questions. Because there are a lot of those, so, to get results there are a lot of things you need to do and there are really boring and they are really hacky, and you don’t want to write them in your papers so you don’t, and so if you try and get into the field it can still be done, and people have done it but you need to be prepared for a lot of trial and error.”

Ilya Sutskever https://vimeo.com/77050653 2013, 1:05:13

1

u/dwf Feb 26 '14

Is there a significant focus on proofs in machine learning too?

Machine learning is a big field. The folks who submit to COLT would be big on proofs. Others, not as much. Empirical study counts for a lot.

0

u/[deleted] Feb 25 '14

Is there research being done about the use of deep learning for program generation? My intuition is that eventually we could use type theory to specify a program and deep learning to "search " for an instantiation of the specification, but I feel like we're quite far from that.

Fuckity. I thought of the same thing after reading... ahaha, you're not scooping me THIS TIME! Nah, it was after reading Joshua Tenenbaum's paper on Bayesian concept learning.

13

u/wardnath Feb 24 '14 edited Feb 25 '14

Dr. Bengio, In your paper Big Neural Networks Waste Capacity you suggest that gradient descent does not work as well with a lot of neurons as it does with fewer. (1) Why do the increased interactions create worse local minima? (2) Do you think hessian free methods like in (Martens 2010) are sufficient to overcome these issues?

Thank You!

Ref: Dauphin, Yann N., and Yoshua Bengio. "Big neural networks waste capacity." arXiv preprint arXiv:1301.3583 (2013).

Martens, James. "Deep learning via Hessian-free optimization." Proceedings of the 27th International Conference on Machine Learning (ICML-10). 2010.

8

u/dhammack Feb 24 '14

I think the answer to this one is that the increased interactions just lead to more curvature (off diagonal Hessian terms). Gradient descent, as a first-order technique, ignores curvature (it assumes the Hessian is the identity matrix). So what happens is that gradient descent is less effective in bigger nets because you tend to "bounce around" minima.

9

u/yoshua_bengio Feb 25 '14

This is essentially in agreement with my understanding of the issue. It's not clear that we are talking about local minima, but what I call 'effective local minima', because training gets stuck (they could also be saddle points or other kinds of flat regions). We also know that 2nd order methods don't do miracles, in many cases, so something else is going on that we do not understand yet.

12

u/ian_goodfellow Feb 24 '14

Verification post: https://plus.google.com/103174629363045094445/posts/2fqbkyYULAf

7

u/hf98hf43j2klhf9 Feb 25 '14

We should try to request Yann LeCunn as well, he seems to be open to the idea.

0

u/SnowLong Feb 27 '14

I would have hard time to justify to login to freaking Facebook for LeCunn AMA...

Don't get me wrong, nothing wrong with LeCunn, but FB is completely different story...

11

u/Megatron_McLargeHuge Feb 24 '14

With the recent success of maxout and hinge activations, how relevant is the older work on RBM pretraining using various contrastive divergence tweaks? What do you think is still worth investigating about stochastic models?

How biologically plausible is maxout, and should we care?

5

u/yoshua_bengio Feb 27 '14 edited Feb 27 '14

The older work on RBM and auto-encoders is certainly still worth further investigation, along with the construction of other novel unsupervised learning procedures.

For one thing, unsupervised procedures (and pre-training) remain a key ingredient to deal with the semi-supervised and transfer learning cases (and domain adaptation, and non-stationary data), when the number of labeled examples of the new classes (or of the changed distribution) is small. This is how we won the two 2011 transfer learning competitions (held at ICML and NIPS).

Furthermore, looking farther into the future, unsupervised learning is very appealing for other reasons:

  • take advantage of huge quantitities of unlabeled data

  • learn about the statistical dependencies between all the variables observed so that you can answer NEW questions (not seen during training) about any subset of variables given any other subset

  • it's a very powerful regularizer and can help the learner to disentangle the underlying factors of variation, making much easier to solve new tasks from very few examples

  • it can be used in the supervised case when the output variable (to be predicted) is a very high-dimensional composite object (like an image or a sentence), i.e., a so-called structured output

Maxout and other such pooling units do something that may be related to the local competition (often through inhibitory interneurons) between neighboring neurons in the same area of cortex.

4

u/ian_goodfellow Feb 27 '14

Right now pretraining does seem to be helpful for preventing overfitting in cases where there is very little labeled training data available. It now longer seems to be necessary as an optimization technique for deep networks, since we can just use the piecewise linear activation functions that are easy to optimize even for very deep networks.

Probabilistic models are still useful for tasks like classification with missing input (because they can reason about the missing inputs), or tasks where the goal is to repair damaged inputs (example: photo touchup) or infer the values of missing inputs, or where the task is just to generate realistic samples of data. It can also often be useful to have a probabilistic model that you use as part of a larger system. For example, if you want to use a neural net as part of an HMM, the HMM requires that its observation and transition models provide real probabilities.

Rectified linear units were partially motivated by biological plausibility concerns, because some neuroscientific evidence suggests that real neurons rarely operate in the regime where they reach their maximum firing rate.

I'm the grad student who came up with maxout, and I didn't have any biological plausibility concerns in mind when I came up with it. After I started using maxout for machine learning, another of Yoshua's grad students, Caglar Gulcehre, told me that there is some neuroscientific evidence for a function similar to maxout but with an absolute value being used in the deeper layers of the cortex. I don't know much about this myself. One thing about maxout that makes it a little bit difficult to explain in biological terms is the fact that maxout units can take on negative values. This is a bit awkward for a biological neurons since it's not possible to have a negative firing rate. But maybe biological neurons could use some average firing rate to indicate 0, and indicate negative values by firing less often than that.

My main interest is in engineering intelligent systems, not necessarily understanding how the human brain works. Because that's what my interest is, I am not very concerned with biological plausibility. Right now it seems easier to make progress in machine learning just by working from first principles than by reverse-engineering the brain. We don't have good enough sensor equipment to extract the kind of information from the brain that we would need to make reverse engineering it convenient.

11

u/[deleted] Feb 24 '14 edited Feb 24 '14

Hello Prof. Bengio, thank you for the AMA. What recommendations would you have for someone who is not a PHD in getting started with Deep Learning.

10

u/[deleted] Feb 24 '14

Dear Yoshua, thanks for doing this!

You are, to my knowledge, the only ML academic to publicly (and wonderfully!) speculate about the sociocultural perspectives afforded by the vantage of deep representation learning. In your fascinating article "Culture vs Local Minima" you touch on many important things, some of which I'm very curious about:

  • You describe how individuals learn by being immersed in culture. We both agree that they don't always learn very wholesome things. If you were king of the world, and you could prescribe a set of concepts that should be a part of every childhood learning trajectory, what would those be and to what end?

  • A corollary of "cultural immersion" is that the specific process of learning is not evident to the learner, the world simply "is" in a particular way. The author David Foster Wallace phrased this phenomenon as akin to fish having to figure out what water is. In your opinion, is this phenomenon an experiential byproduct of the neural architecture, or does it confer some learning benefit?

  • Why do you think that cultural trends become entrenched and cause their learners to fight to stay in (what could be argued to be) local optima - like e.g. the conflicts between various religious institutions and Enlightenment philosophy, or patriarchal society vs the suffragettes, etc.? Is this a case of very pernicious parameters, or is there some benefit to the learners in question?

  • Do you have an opinion on such concepts as mindfulness meditation, and if so, how do you think they relate to the exploration of "idea space"?

Again, thanks a lot for taking the time. In the space of human ideas you are a trailblazer, and we are immensely richer for your presence!

11

u/yoshua_bengio Feb 26 '14

I am not a social scientist or a psychologist, so my opinions on these subjects should be taken as such. My opinion is that many learners stay entrenched in their beliefs because these beliefs have become part of their identity, their definition of who they are, and it's harder and scary to change that. There may also be a more computational aspect related to the notion of effective local minima (the optimization getting stuck). I believe that a lot of what our brain does is try to bring coherence to all of our experience, in order to construct a better model of the world. Mathematically, this may be related to the problem of inference, by which a learner searches for plausible explanations (latent variables) of the observed data. In stochastic models, inference is done by a form of stochastic exploration of configurations (and a Markov chain really looks like a series of free associations). Meditation and other time spent not doing anything directed but just thinking may well be useful to help us explore in this way. Sometimes it clicks, i.e., we find an explanation that fits well with many things. This is also how scientific ideas often seem to emerge (for me at least).

9

u/marvinalone Feb 24 '14

What's your opinion of Solomonoff Induction and AIXI? I'm just starting to read up on the topic, and I can't quite decide whether it's serious work, or a fringe theory by a small group of people who all cite each other.

2

u/[deleted] Feb 25 '14

I am interested in this also.

2

u/[deleted] Feb 25 '14

Not Bengio, but reasonably well-versed in this specific topic.

It's serious work by theoreticians. You need a freaking Turing oracle to make those algorithms work, and all the relevant proofs are about global optimality in the presence of that Turing oracle, not about how good a learning/error rate you're going to get out of a finite sample with limited computing power (as you're going to need to build real algorithms).

That said, Schmidhuber and Hutter (who invented AIXI) have publication and competition records like nobody fucking else.

2

u/dwf Feb 27 '14

I'll just say that while the IDSIA group's competition record and benchmark results are impressive, it's important to compare apples to apples. Comparing a method that uses elastic distortions and other dataset augmentation strategies against a method that doesn't doesn't tell you anything about either method; it's been known for decades that more data helps, and that you can sometimes acquire more data by artificially augmenting a given training set with distortions. It's important to not conflate impressive engineering with scientific novelty.

9

u/EJBorey Feb 24 '14

We have all been hearing about the performance achievable via deep learning (in academic journals such as the New York Times, no less!). I've also heard that it's difficult for non-experts to get these techniques to work: Ilya Sutskever says that there is a weighty oral tradition about the design and training of deep networks and that the best way to learn how is to work for years with someone who is already an expert (source: http://vimeo.com/77050653).

I studied machine learning but not deep learning. Going back to grad school is not really an option for me. How can I learn how to design, build, and train deep neural networks without access to the oral tradition? Could you write it down for us somewhere?

2

u/[deleted] Feb 25 '14

Related to this: would it be possible to use a Bayesian approach to try and encode some of this folk-lore knowledge?

What is the road-map to making deep learning accessible to all?

Thank you.

9

u/yoshua_bengio Feb 27 '14

Hyper-parameter optimization has already been found to be a useful way to (partially) automate the search for good configurations in deep learning.

The idea is to automate the process of selecting the knobs, bells and whistles of machine learning algorithms, and especially of deep learning algorithms. We call such "knobs" hyper-parameters. They are different from the parameters that are learned during training, in that they are typically set by hand, by trial and error, or through a dumb and extensive exploration of all combinations of values (called "grid search"). Deep learning and neural networks in general involve many more such knobs to be tuned, and that was one of the reasons why many practitioners stayed far from neural networks in the past. It gave the impression of deep learning as a "black art", and it remains true that strong expertise helps a lot, but the research on hyper-parameter optimization is helping to move towards a more fully automated deep learning.

The idea of optimizing hyper-parameters is old, but had not had as much visible success until recently. One of the main early contributors to this line of work (before it was applied to machine learning hyper-parameter optimization) is Frank Hutter (along with collaborators), who devoted his PhD thesis (2009) to algorithms for optimizing knobs that are typically set by hand in general in software systems. My former PhD student James Bergstra and I worked on hyper-parameter optimization a couple of years ago and we first proposed a very simple alternative, called "random sampling" to standard methods (called "grid search"), which works very well and is very easy to implement.

http://jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf

We then proposed using for deep learning the kinds of algorithms Hutter had developed for other contexts, called sequential optimization and this was published at NIPS'2011, in collaboration with another PhD student who devoted his thesis to this work, Remi Bardenet, and his supervisor Balazs Kegl (previously a prof in my lab, now in France).

http://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf

This work has been followed up very successfully by researchers at U. Toronto, including Jasper Snoek (then a student of Geoff Hinton), Hugo Larochelle (who did his PhD with me) and Ryan Adams (now a faculty at Harvard) with a paper at NIPS'2012 where they showed that they could push the state-of-the-art on the ImageNet competition, helping to improve the same neural net that made Krizhevsky, Sutskever and Hinton famous for breaking records in object recognition.

http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips.pdf

Snoek et al put out a software that has since been used by many researchers, called 'spearmint', and I found out recently that Netflix has been using it in their new work aiming to take advantage of deep learning for movie recommendations:

http://techblog.netflix.com/2014/02/distributed-neural-networks-with-gpus.html

1

u/james_bergstra Mar 03 '14 edited Mar 03 '14

Plug for Bayesian Optimization and Hyperopt:

FWIW my take is that Bayesian Optimization + Experts designing the search spaces for SMBO algorithms is the way to deal with this: e.g. other post and ICML paper on tuning ConvNets

The Hyperopt Python package provides SMBO for ConvNets, NNets, and (soon) a range of classifiers from scikit-learn hyperopt-sklearn.

Sign up for Hyperopt-announce to get alerts about new stuff such as upcoming Gaussian-Process and regression-tree-based SMBO search algorithms similar to Jasper Snoek's Spearmint and Frank Hutter's SMAC software.

2

u/EJBorey Feb 25 '14

Actually, I wasn't asking about the Bayesian optimization work that Jasper Snoek et al. are doing, because I don't think it will be possible to automate away all human judgement in the design of these things. Rather, I wanted to know how to quickly acquire the necessary intuition without postdoc-ing in Bengio, Hinton, or LeCunn's labs.

Deep learning will never be practical if there's only 10 people on the planet who can get it to work! Is there a way to quickly become one of the savants?

1

u/orwells1 Feb 27 '14 edited Feb 27 '14

Hello, same here. I fit the bill of their intended phd students (according to Y. Lecun's page, awesome math + coder), but wanted to avoid more phd/post-docs. I went through a reasonable number of papers, but in most there are either explanations missing or later the authors comment online on the "human in the loop optimization"/"tricks of the trade"/"black magic". I'm not sure if I should be investing much more of my time alone, if the full knowledge is not there. Is it? Thanks a lot for doing this!

8

u/serge_cell Feb 24 '14

Hi Prof. Bengio, There were some work on applying "higher" math - algebraic/tropical geometry, category theory, to deep learning. Notably, John Healy several years ago claimed improving neural net (ART1) with category theory. What's your opinion on this approach? Will it be only toy model in foreseeable future, or there is some promise in this approach in your opinion?

4

u/yoshua_bengio Feb 27 '14

See the above suggestions http://www.reddit.com/r/MachineLearning/comments/1ysry1/ama_yoshua_bengio/cfq7a3s Regarding algebraic/tropical geometry, look at the work of Morton & Montufar.

2

u/polyguo Feb 25 '14

Source? I'm extremely interested in the intersection between Programming Language Theory and Machine Learning. This seems to be right there.

8

u/n_dimensional Feb 24 '14

Dear Prof. Bengio,

I am about to finish my PhD in computational neuroscience and I am very interested in the "gray area" between neuroscience and machine learning.

What aspects of brain computation do you think are (or will be) most relevant for machine learning?

If you could know the answer to one question about how the brain computes information, what would that be?

Thanks!

5

u/yoshua_bengio Feb 27 '14

Understanding how learning proceeds in brains is clearly the subject most relevant to machine learning. We don't have a clue of how brains can learn in the kinds of efficient ways that we are able to implement in artificial neural networks, so this could be really important, and a place where information could flow both ways between machine learning research and computational neuroscience.

7

u/BeatLeJuce Feb 24 '14

  1. Why do Deep Networks actually work better than shallow ones? We know a 1-Hidden-Layer Net is already an Universal Approximator (for better or worse), yet adding additional fully connected layer usually helps performance. Were there any theoretical or empirical investigations into this? Most papers I read just showed that they WERE better, but there were very few explanations as to why -- and if there was any explanation. then it was mostly speculation.. what is your view on the matter?

  2. What was your most interesting idea that you never managed to publish?

  3. What was funniest/weirdest/strangest paper you ever had to peer-review?

  4. If I read your homepage correctly, you teach your classes in French rather than English. Is this a personal preference or mandated by your University (or by other circumstances)?

5

u/yoshua_bengio Feb 27 '14

Being a universal approximator does not tell you how many hidden units you will need. For arbitrary functions, depth does not buy you anything. However, if your function has structure that can be expressed as a composition, then depth could help you save big, both in a statistical sense (less parameters can express a function that has a lot of variations, and so need less examples to be learned) and in a computational sense (less parameters = less computation, basically).

I teach in French because U. Montreal is a French-language university. However, three quarters of my graduate students are non-francophones, so it is not a big hurdle.

1

u/rpascanu Feb 27 '14

Regarding 1, there are some work in this direction. You can check out these papers:

http://arxiv.org/abs/1312.6098 (about rectifier deep MLPs),

http://arxiv.org/abs/1402.1869 (about deep MLPs with piecewise-linear activations),

RBM_Representational_Efficiency.pdf,

http://arxiv.org/abs/1303.7461.

Basically the universal approximator theorem says that a one layer MLP can approximate any function if you allow yourself an infinite number of hidden units which in practice one can not do. One advantage of deep models over shallow one is that they can be (exponentially) more efficient at representing certain family of functions (arguably the family of functions we actually care about).

6

u/shanwhiz Feb 24 '14

We have seen deep learning work really well for image/video/sound. Do you foresee it working for text classification as well? Most papers that have tried text/document classification using deep learning have not done better than the conventional SVM/Bayes. What are your thoughts on this?

10

u/yoshua_bengio Feb 26 '14

I predict that deep learning will have a big impact in natural language processing. It has already had an impact, in part due to an old idea of mine (from NIPS'2000 and a 2003 paper in JMLR): represent words by a learned vector of attributes, learned so as to model the probability distribution of sequences of words in natural language text. The current challenge is to learn distributed representations for sequences of words, phrases and sentences. Look at the work of Richard Socher, which is pretty impressive. Look at the work of Tomas Mikolov, who beat the state of the art in language models using recurrent networks and who found that these distributed representations magically capture some form of analogical relationships between words. For example, if you take the representation for Italy minus the representation for Rome, plus the representation for Paris, you get something close to the representation for France: Italy - Rome + Paris = France. Similarly, you get that King - Man + Woman = Queen, and so on. Since the model was not trained explicitly to do these things, this is really amazing.

10

u/hapagolucky Feb 24 '14

I see more and more pop media articles extolling deep learning as a panacea that will make AI a reality (Wired is especially guilty of this). Given the AI winters of the 1970's and 1980's that arose from overhyped expectations, what can deep learning and ML researchers and advocates do to mitigate this from happening again?

4

u/yoshua_bengio Feb 27 '14

Stick to the scientific ways of demonstrating advances (which often is lacking from companies branding themselves as doing deep learning). Avoid overselling. Stay humble while not using our motivation associated with the long-term vision that brought us here in the first place.

-1

u/[deleted] Feb 25 '14

That depends what you mean by "AI" ;-). Certainly deep learning makes it easier to get better ML performance out of large datasets of uncategorized stuff.

5

u/[deleted] Feb 24 '14

Hi Bengio. I'm a masters candidate in robotics, mostly doing reinforcement learning mushed together with some ML regression methods for the identification of interesting value functions and state space representations.

How is your work life balance? Do you have fun? What sorts of things do you do to unwind?

I'm considering doing a PhD, but I literally feel like just getting a part-time job and doing independent research, because the academic environment can be pretty stifling.

Also, Montreal seems really fun!

J

15

u/yoshua_bengio Feb 27 '14

Life balance. That is tough. Many prominent scientists will tell you the same story. My inclination is to work as much as I can: that is probably part of the reasons for my early success, but this may threaten my health and personal life. We live in an environment which puts so much pressure on us that it is easy to forget that we are humans and we need breaks and to take care of our body (I have some health issues that I cannot just ignore) and our relationships with other humans. Some kind of self-discipline helps, but I found that what works best is to cultivate what is rewarding and pleasurable and the same time is good for me and my physical and emotional well-being. For example I like very much to walk (many ideas come!), not to speak about eating healthily and enjoying a romantic relationship based on authenticity and where I can really be myself.

Oh, and yes, Montreal IS fun ;-)

The advantage of academia is that you can focus on research and that you can benefit enormously from the interactions with other researchers. Research is a collective enterprise. This is NOT like what you tend to see in science-fiction movies. Never forget that!

1

u/[deleted] Feb 27 '14

This is really refreshing to hear!

I have been struggling with balance as well. I think I should find my balanced way of being a scientist as well, and find a supervisor who wants to be my long term colleague and friend - not just a pedantic sort of guide and disciplinary figure. Perhaps giving up on academia is the easy way out. Perhaps what I really need to do is make more inspirational friends, and help join and build the community I want to be a part of.

Thanks so much for the candid response! It's very eye opening. I hope you keep being awesome and inspiring people like me! (but no so much that we keep losing so much sleep on our work :p)

5

u/[deleted] Feb 25 '14 edited Feb 25 '14

Dr Bengio,

I'd like to thank you for the amazing research and software(theano, pylearn2) that your lab has contributed.

What are your feelings on Hinton and LeCun moving to industry?

What about academia and publishing your research is more valuable than the floating point overflow of money you could make at private companies?

Are you nervous that machine learning will go the way of time-series analysis, where a lot of advanced research takes place behind closed doors because the intellectual property is so valuable?

Given the recent advancements in training discriminative neural networks, what role do you envision generative neural networks play in the future?

6

u/yoshua_bengio Feb 27 '14

I think that with Hinton & LeCun in industry, there will be more rapid advance in applying deep learning to really interesting and large-scale problems. The down side may be a temporarily reduced offer in terms of supervising new graduate students for deep learning. However, there are many young faculty who are at the forefront of deep learning research and who are eager to take new strong students. And the fact that deep learning is being used heavily in industry means that more students get to know about the field and are excited to jump into it.

Personally, I prefer the freedom of academia over more zeros in my salary. See also what I wrote above: http://www.reddit.com/r/MachineLearning/comments/1ysry1/ama_yoshua_bengio/cfpbc1g

I believe that a lot of research will continue to happen in academia and that in the large industrial labs the incentive to publish will remain high.

I think that generative networks are very important for the future. See what I wrote above about unsupervised learning (the two are not synonym, but often come together, especially since we found the generative interpretation of auto-encoders, see the work with Guillaume Alain, http://arxiv.org/pdf/1305.6663.pdf):

http://www.reddit.com/r/MachineLearning/comments/1ysry1/ama_yoshua_bengio/cfq7v4v

1

u/quaternion Feb 25 '14

Could you provide additional info on who and what you are referring to with time series analysis?

8

u/tryolabs_feco Feb 24 '14 edited Feb 25 '14

Hi Yoshua, very excited about this AMA, thank you for your time. I have a few questions:

  • What are the biggest challenges in ML nowadays?

  • What are the most interesting and/or creative ways you have seen people/businesses using ML?

  • What does the future of Machine Learning look like?

4

u/freieschaf Feb 24 '14

Last year I did my undergrad thesis on NLP using probabilistic models and neural networks partly inspired by your work. I became interested and at that point I considered doing further work on NLP. Currently I am pursuing an MSc degree taking several related courses.

But, after several months, I haven't found NLP to be as motivating as I was expecting it to be; research on this area seems to be a little stagnant, from my limited point of view. What do you think are some challenges that are making or going to make this field move forward?

Thanks for taking the time to answer some questions here!

9

u/yoshua_bengio Feb 27 '14

I believe that the really interesting challenge in NLP, which will be the key to actual "natural language understanding", is the design of learning algorithms that will be able to learn to represent meaning. For example, I am working on ways to model sequences of words (language modeling) or to translate a sentence in one language into a corresponding one in another language. In both of these cases we are trying to learn a representation of the meaning of a phrase or sentence (not just of a single word). In the case of translation, you can think of it like an auto-encoder: the encoder (that is specialized to French) can map a French sentence into its meaning representation (represented in a universal way), while a decoder (that is specialized to English) can map this to a probability distribution over English sentences that have the same meaning (ie. you can sample a plausible translation). With the same kind of tool you can obviously paraphrase, and with a bit of extra work, you can do question answering and other standard NLP tasks. We are not there yet, and the main challenges I see have to do with numerical optimization (it is difficult not to underfit neural networks, when they are trained on huge quantities of data). There are also more computational challenges: we need to be able to train much larger models (say 10000x bigger), and we can't afford to wait 10000x more time for training. And parallelizing is not simple but should help. All this will of course not be enough to get really good natural language understanding. To to this well would basically allow to pass some Turing test, and it would require the computer to understand a lot of things about how our world works. For this we will need to train such models with more than just text. The meaning representation for sequences of words can be combined with the meaning representation for images or video (or other modalities, but image and text seem the most important for humans). Again, you can think of the problem as translating from one modality to another, or of asking whether two representations are compatible (one expresses a subset of what the other expresses). In a simpler form, this is already how Google image search works. And traditional information retrieval also fits the same structure (replace "image" by "document").

1

u/[deleted] Feb 27 '14

I am not from academia, but ever since I have started following machine learning stuff, I keep getting interesting ideas/problems to solve. Here is one I got few years back.

You take simple math word problems, e.g. simple ratio/proportion, rate/motion, age, give/take etc. word problems, they can (have to) be translated to a bunch of constants, unkown(s) and math relations/concepts, eventually to find some unknown(s). And every one who understands the concepts, will come up with similar equations, and definitely one correct answer. You can view it as a NLP problem.. How to solve it? Well I don't know, may be trying to first extract basic concepts/relations from standard (and simple) word problems?

Thinking aloud - you may start by doing something like "part of (math) speech" tagging...or, get some labeled data ( problem -> math equation), and see if you can find some hidden factors/relations defining the translations...

4

u/deeperredder Feb 24 '14 edited Feb 24 '14

While deep nets have helped move the state of the art forward in natural language text understanding, the improvements there haven't really been significant. Where do you think significant progress can come from in that field?

3

u/yoshua_bengio Feb 27 '14

I do think that significant progress will come in the area of natural language processing, most importantly, natural language understanding. Progressively, though (because full understanding is essentially AI-level understanding of the world around us). See my previous answer:

http://www.reddit.com/r/MachineLearning/comments/1ysry1/ama_yoshua_bengio/cfpje92

10

u/CyberByte Feb 24 '14

What will be the role of deep neural nets in Artificial General Intelligence (AGI) / Strong AI?

Do you believe AGI can be achieved (solely) by further developing these networks? If so: how? If not: why not, and are they still suitable for part of the problem (e.g. perception)?

Thanks for doing this AMA!

4

u/davidscottkrueger Feb 27 '14

Hi! My name's David Krueger; I'm a Master's student in Bengio's lab (LISA).

My response is: it is not clear what their role will be. AGI may be theoretically achievable solely by developing NNs, (especially if we include RNNs), but this is not how it will actually take place.

What incompetentrobot said is literally false, but there is a kernel of truth, which is that Deep Learning (so far) just provides a set of methods for solving certain well-defined types of general Machine Learning problems (such as function approximation, density estimation, sampling from complex distributions, etc.).

So the point is that the contributions of the Deep Learning community haven't been about solving fundamentally new kinds of problems, but rather finding better ways to solve fundamental problems.

0

u/incompetentrobot Feb 25 '14

"Deep Learning" is just an umbrella term for a group of ad-hoc function approximators. It has no connection with AGI.

7

u/yoshua_bengio Feb 27 '14

I strongly disagree.

Deep learning is FAR from being limited to function approximation (which by the way can be a very powerful in its own right). For example, please consider all the work on generative and unsupervised learning with deep architectures (which I consider far more intellectually interesting).

3

u/[deleted] Feb 25 '14

Even if deep learning was 'just' a collection of ad-hoc function approximators, how would it follow that they had no connection to AGI?

3

u/Sigmoid_Freud Feb 25 '14

I disagree. The brain certainly uses hierarchical feature detectors (i.e. deep learning). The way that a deep convolutional NN models features is much more similar to how the visual cortex does it than the way a shallow NN does it.

It definitely seems like an important piece of the puzzle, though it is only one piece.

One large missing piece in regular feedforward deep NNs is the lack of feedback loops. The brain actually has MORE feedback connections than feedfoward connections, enabling it to detect sequences, time series and contextual information. This is one reason I'm curious about RNNs :) (see my question above)

1

u/rpascanu Feb 27 '14

Also, if you think of, e.g., an RNN, it does more than approximating a function. RNNs can be seen as approximators of arbitrary dynamical systems.

Mathematically speaking, these are two very different objects, so it seems you do have quite a bit of stuff outside your umbrella.

9

u/Should_I_say_this Feb 24 '14

Can you describe what you are currently researching, first by bringing us up to speed on the current techniques used and then what you are trying to do to advance that?

10

u/SnowLong Feb 24 '14

I think your question was answered by Yousua here:

Deep Learning of Representations: Looking Forward

Yoshua Bengio

arXiv:1305.0445v2 [cs.LG] 7 Jun 2013

1

u/Should_I_say_this Feb 24 '14

This is excellent thanks!

4

u/dwf Feb 27 '14

Following on work Ian and I did on maxout, I recently did some work empirically interrogating how and why dropout works, focusing on the rectified linear case. More recently I've been working on hyperparameter optimization.

3

u/exellentpossum Feb 24 '14

It would be cool if members from Bengio's group could also answer this (like Ian).

6

u/rpascanu Feb 27 '14

I've done some work lately on the theory side (showing that deep models can be more efficient than shallow ones):

I've been spending quite a bit of time on natural gradient, and I'm currently exploring variants of the algorithm, and I'm interested in how it addresses non-convex optimization specific problems.

And, of course, recurrent networks which have been the focus of my PhD since I started. Particularly I worked on understanding the difficulties of training them (http://arxiv.org/abs/1211.5063) and how depth can be added to RNNs (http://arxiv.org/abs/1312.6026).

6

u/caglargulcehre Feb 27 '14

Hi, My name is Caglar Gulcehre and I am PhD student at Lisa lab. You can access my academic page from here, http://www-etud.iro.umontreal.ca/~gulcehrc/.

I have done some works related to Yoshua Bengio's "Culture and Local Minima" paper, basically we focused on empirically validating the optimization difficulty on learning high level abstract problems: http://arxiv.org/abs/1301.4083

Recently I've started working on Recurrent neural networks and we have a joint work with Razvan Pascanu, Kyung Hyun Cho and Yoshua Bengio: http://arxiv.org/abs/1312.6026

I've also worked on a new kind of activation function in which we claim to be more efficient in terms of representing complicated functions compared to regular activation functions i.e, sigmoid, tanh,...etc:

http://arxiv.org/abs/1311.1780

Nowadays I am working on Statistical Machine Translation and learning&generating sequences using RNNs and what not. But I am still interested in optimization difficulty for learning high level(or abstract) tasks.

5

u/ian_goodfellow Feb 26 '14

I'm helping Yoshua write a textbook, and working on getting Pylearn2 into a cleaner and better documented state before I graduate.

→ More replies (2)

0

u/[deleted] Feb 24 '14

9

u/EJBorey Feb 24 '14

Here's an example where experts won a Kaggle contest: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st-place-interview/ And here, where they won the Netflix Prize: http://techblog.netflix.com/2012/04/netflix-recommendations-beyond-5-stars.html

But I think the reason why they don't work on the problems is that the bad ML researchers won't win and therefore not publish, while the good ones would get paid millions of dollars by companies to answer the same questions! Why do it for free?

→ More replies (1)
→ More replies (3)

0

u/[deleted] Feb 24 '14

9

u/yoshua_bengio Feb 27 '14

There is a constructed task on which all the traditional black-box machine learning that were tried failed, and where some deep learning variants work reasonably well (and where guiding the hidden representation completely nails the task, showing the importance of looking for algorithms that can discover good intermediate representations that disentangle the underlying factors). Note that many deep learning approaches also failed so this is interesting. See http://arxiv.org/abs/1301.4083. What's particular about this task is that it is the composition of two much easier tasks (detecting objects, performing a logical operation on the result), i.e., it intrinsically requires more depth than a simple object recognition task.

→ More replies (1)