Deep Learning has been amazingly successful in applications such as speech recognition, image and video analysis and machine translation. Yet, compared with the human brain it is still extremely inefficient, both in terms of data and power. In this talk we will discuss a number of directions to improve in both these dimensions. First I will discuss how symmetries in the data can be exploited to extract more information from each data point, through the use of group convolutional networks. Then we will discuss how a Bayesian view of deep learning can help us compress neural networks, sometimes by a very large amount, thus improving its power efficiency. Finally, we will discuss how spiking neural networks can improve the efficiency of deep learning in the temporal domain.