Beyond landscapes : an exemplar-based image colorization method

Abstract

Image colorization consists in, given a grayscale image, generating a plausible color version of this image, which can be performed as a manual/artistic process1 but also as a computer assisted or even fully automated process. Colorization is a underconstrained problem, which requires extra information in order to provide a unique solution. This dissertation focuses on exemplar-based colorization methods, in which the extra informa- tion comes from a user-selected color reference image with similar semantic content to the target. While the user selects the reference based on content similarity, the algorithms estimate similarity based on local descriptors of image regions. This difference in abstrac- tion between the user and algorithm perspective can lead to the algorithms not always being able to transfer colors between semantic corresponding elements in the image pair, specially in images in which the mapping between content/color and local descriptors is complex. Most exemplar-based methods in the literature display successful examples mostly limited to simple instances, such as landscapes, animals and simple buildings. Based on this observation, in this research we propose a new exemplar-based method that aims at generating plausible colorizations for a wider range of image pairs, including images of higher complexity. To that end, the proposed method features a two-stage clas- sification scheme that uses the available features in a more consistent manner and makes the initial color assignments more robust. It also includes an edge-aware relabeling method that enhances the spatial coherence and mitigates the impact of the multimodality, inher- ent to the colorization problem, over the method’s colorized outputs. In this dissertation, we present a broad review of the colorization literature introducing a taxonomy that cat- egorizes colorization techniques based on the source of prior information used to guide their color assignments. The proposed method pipeline is then described in details, and its key modules are validated through experiments. Moreover, a comparative analysis is performed which subjects the proposed and baseline methods to different source/target pairs to visually assess and compare their results. Experimental results indicate that the proposed method yields colorization results that are more coherent and of higher visual quality compared to two state-of-the-art exemplar-based colorization algorithms, both in simple and complex image sets. The results also indicate that exemplar-based methods can achieve results of comparable visual aspect to those of modern deep learning approaches while allowing more user control.