Robust handwritten signature representation with multi-task continual learning of synthetic data over predefined real feature space and contrastive fine-tuning

Abstract

In spite of recent advances in computer vision, the classic problem of offline handwritten signa- ture verification still remains challenging. The signature verification task has a high intra-class

variability because a given user often shows high variability between its samples. Besides, sig- nature verification is harder in the presence of skilled forgeries. Recently, in order to tackle

these challenges, the research community has investigated deep learning methods for learn- ing feature representations of handwritten signatures. When mapping signatures to a feature

space, it is desired to obtain dense clusters of signature’s representations, in order to deal with intra-class variability. Besides, not only dense clusters are required but also a larger separation between different user’s clusters in the feature space. Finally, it is also desired to move away feature representations of skilled forgeries in relation to the respective dense cluster of genuine representations. In this thesis, we concentrate our efforts on achieving these properties in the learning of handwritten signature representations, aiming at subsequent improvements in the verification time. The research reported in this thesis is developed in two moments. Initially,

we propose a framework based on the contrastive learning of handwritten signature represen- tations. Based on the observation of the experimental results of the first part of this research,

we extend this framework to handle synthetic and real signature data during training through

knowledge distillation, providing greater robustness of the learned representations in the verifi- cation of diverse datasets. In the first part of this thesis, we hypothesize that desired properties

can be achieved by means of a multi-task framework for learning handwritten signature feature representations based on deep contrastive learning. The proposed framework is composed of two objective-specific tasks. The first task aims to map signature examples of the same user closer within the feature space, while separating the feature representations of signatures of different users. The second task aims to adjust the skilled forgeries representations by adopting

contrastive losses with the ability to perform hard negative mining. Hard negatives are exam- ples from different classes with some degree of similarity that can be applied for training. We

demonstrate that our contrastive multi-task models perform better than using cross-entropy

loss, which experimentally indicates a signature verification improvement provided by the pro- posed framework. As successful results in deep learning models require a significant amount

of training data, in these experiments we resorted to the GPDSsynthetic dataset with syn- thetic signature data for training models. However, as a result of experimental observation,

we found that the intrinsic characteristics of real genuine signatures are not perfectly embed-

ded in the synthetic signature images. This way, we have found a difference in verification performance between models trained using the GPDS-960 and GPDSsynthetic datasets; and undesired effects introduced by excessively shifting the model distribution for any of these data sources. Given this problem, in the second part of this thesis, we resort to the SigNet

model pre-trained with GPDS-960 data to provide knowledge about real signatures. We com- plement the SigNet predefined real data feature space using synthetic data while minimizing

the divergence in distribution between the datasets. This is achieved through class-incremental

continual learning based on knowledge distillation with subsequent fine-tuning of the repre- sentation space adopting a contrastive objective. Furthermore, we propose a new distillation

dataset inverted from the pre-coded real distribution in SigNet. Our proposed method obtains improved writer-dependent verification and a more balanced writer-independent verification, producing more robust models in the verification of the diverse Western and non-Western script datasets GPDSsynthetic, GPDS-300, CEDAR, MCYT-75, BHSig-H, and BHSig-B. Finally, we demonstrate that the proposed method can be used to improve signature verification in the training of new large vision architectures when access to the GPDS-960 dataset is unavailable.