Self-driving cars have long been dreamed of by generations and they are expected to bring significant impact to our society.  Recent research in deep neural networks has achieved promising progress in various tasks in perception, control, planning, etc., which are core to autonomous driving. However, while recent research has been primarily focusing on improving accuracy, to actually deploy neural network models in autonomous vehicles, we also need to deal with other critical issues such as inference speed, energy efficiency, model size, etc. In this talk, we will focus on the task of image object detection and discuss how we designed efficient neural networks to address above problems.