Abstract: Image classification is a fundamental and important task in the field of computer vision and artificial intelligence. In recent years, image classification has made breakthrough progress based on deep learning on large-scale datasets. However, it still exits big challenges on small-sample image data. The main difficulty is that the deep neural network easily overfit small-sample data and has big variance. Ensemble learning is a good way to overcome overfitting and reduce the variance of the model; however, the existing ensemble methods based on deep neural network still could overfit on small-sample image data due to the big randomness of deep neural network. In this paper, we propose a new ensemble method for small-sample image classification tasks. The proposed method based on VGG16 network, we modified the structure of the VGG16 network to two branches, one branch is a classifier based on prototype learning, and the other is a classifier based on margin learning. The experimental results on two small-sample image datasets, the LabelMe dataset and the Caltech101 dataset, show that the proposed method has better performance and higher stability than other referred methods.