The classifiers that return compact acceptance regions are crucial for the success in anomaly detection and open set recognition settings since we have to determine and reject the anomalies and samples coming from the unknown classes. This paper introduces novel methods that approximate the class acceptance regions with compact hypersphere models for anomaly detection and open set recognition. As opposed to the other deep hypersphere classifiers, we treat the hypersphere centers as learnable parameters and update them based on the changing deep feature representations. In addition, we propose novel loss terms that are more robust to the noisy labels within the outlier exposure and background datasets. The proposed methods bear similarity to the deep distance metric learning classifiers using the triplet loss function with the exception that the anchors are set to the hypersphere centers which are updated dynamically. The experimental results show that the proposed methods achieve the state-of-the-art accuracies on the majority of the tested datasets in the context of anomaly detection and open set recognition.