This paper introduces a new method to find the most important samples for classification in image sets to speed-up the classification phase and reduce the storage space for large-scale face recognition tasks that use image sets obtained from face videos. We approximate the image sets with the kernelized convex hulls and show that it is sufficient to use only the samples that participate to shape the image set boundaries in this setting. To find those important samples that form the image set boundaries in the feature space, we employed the kernelized Support Vector Data Description (SVDD) method which finds a compact hypersphere that fits the image set samples best. Then, we show that these kernelized hypersphere models can also be used to model image sets for classification purposes. Lastly, we introduce ESOGU-285 (ESkisehir OsmanGazi University) Face Videos database that includes 285 people since the most popular video datasets used for set based recognition methods include either a few amount of people or large amount of people with just a few (or single) video collections. The experimental results on small sized standard datasets and our new larger sized dataset show that the proposed method greatly improves the testing times of the classification system (we obtained speed-ups up to a factor of 10 in ESOGU Face Videos dataset) without a significant drop in accuracies.