Label noise detection under Noise at Random model with ensemble filters

Abstract

Label noise detection has been widely studied in Machine Learning due to its importance to improve training data quality. Satisfactory noise detection has been achieved by adopting an ensemble of classifiers. In this approach, an instance is assigned as mislabeled if a high proportion of members in the pool misclassifies that instance. Previous authors have empirically evaluated this approach with results in accuracy, nevertheless, they mostly assumed that label noise is generated completely at random in a dataset. This is a strong assumption since there are other types of label noise which are feasible in practice and can influence noise detection results. This work investigates the performance of ensemble noise detection in two different noise models: the Noisy at Random (NAR), in which the probability of label noise depends on the instance class, in comparison to the Noisy Completely at Random model, in which the probability of label noise is completely independent. In this setting, we also investigate the effect of class distribution on noise detection performance, since it changes the total noise level observed in a dataset under the NAR assumption. Further, an evaluation of the ensemble vote threshold is carried out to contrast with the most common approaches in the literature. Finally, it is shown in a number of performed experiments that the choice of a noise generation model over another can lead to distinct results when taking into consideration aspects such as class imbalance and noise level ratio among different classes.