A novel method for content-based retrieval in art image collections utilizing color semantics

Abstract

The field of image retrieval has to overcome a major challenge: it needs to accommodate the obvious difference between the human vision system, which has evolved genetically over millenniums, and the digital technologies, which are limited within pixels capture and analysis. We have the hard task to develop appropriate machine algorithms to analyze the image. These algorithms are based on completely different logic and "instruments" compared to the human process of perception, but would give similar results in interpreting the input image. In the context of this thesis the challenges are even bigger because we focus our efforts on image analysis of the aesthetic and semantic content of art images. Naturally, the interpretation of what we – humans – see is hard to characterize, and even harder to teach to a machine. Yet, over the past decade, considerable progress has been made to make computers learn to understand, index and annotate pictures representing a wide range of concepts. Convenient image capture techniques, inexpensive storage, and widely available dissemination methods have made digital images a widespread information format. This increased availability of images is accompanied by a need for indexing and retrieval tools [Chen et al, 2005a]. The digital repositories of cultural heritage objects can employ techniques similar to the ones used in any general-purpose image processing environment in order to provide standard functionality for searching objects. However, cultural heritage objects are rich in content describing events, monuments, places, people; and they are distributed across different locations. The users can formulate queries using different modalities such as free text, similarity matching, or metadata; one important current trend is the use of linked data [Gradmann, 2010]. To be addressed properly, the specificity of these objects suggests that support should be provided for some additional tasks which need to be reflected in the image analysis methods and techniques used. In the area of art paintings retrieval the sensory, semantic and abstraction gaps are a predominant problem. Colour perception influences multiple aspects in image analysis of art colour ranging from the physical nature of light, to physiological specifics the human vision system, psychological peculiarities and socio-cultural ground, in which the artwork was created as well as the one of the receiver of the message which the artist expressed in his work.In the process of perception, figure-ground separation is the first cognitive step. Colour plays an important but secondary role; however colour responses are more connected to human emotions than to rational mind. This property itself makes the colours' influence on human perception pivotal. The presence of one or more colours in different proportions conveys different messages, which can increase or suppress the perception of the observed objects. In the field of image retrieval the ways of perceiving colours and colour combinations as similar or not similar is crucial when one has to extract images based on a criterion reflecting the level of emotional perception, or to search for any specific characteristics of the artist's expressiveness. The colour impact on people depends on multiple factors with physical laws and physiology being only the beginning. Further along this process psychological perception plays an important role; with both the particular psychological state and the socio-cultural environment in which a character of a person is composed playing a role. Perception of colour brings the whole emotional and mental identity of the artist as well as of the observer, joining their intelligence, memory, ideology, ethics, aesthetics and other sensations. The primary goal of this work is to make an analysis of a range of theories on the existing interconnections in colour combinations and to formalize them in order to allow for extracting them from digitized artworks. We use Itten's colour theory s a basis of our research. Global low-level features reflecting the quantitative presence of quantized colour characteristics are suggested as a helpful instrument for the formal definitions of harmonies/contrast descriptors.We propose a classification of colour harmonies and contrasts, which is consistent with the human perception of visual expression and conforms to the possibilities of automatic extraction of visual information from digitised copies of art images. The classification combines the three main colour characteristics which are closest to human perception – hue, saturation and lightness. We provide the formal definition of extracting such descriptors from images. The third group of descriptors, based on vector quantization of MPEG-7 descriptors over the partitioned images, are introduced in order to analyze what are the possibilities of capturing more detailed information for semantic and abstraction content of art images based on the MPEG-7 descriptors with significant dimensionality reduction. Further, we propose architecture of an experimental CBIR system, where the extracted visual features are combined with the extraction of textual metadata on the examined art images. The textual metadata are necessary in order to apply supervised learning methods. A designated software system "Art Painting Image Colour Aesthetics and Semantics" (APICAS) was developed as an appropriate environment for implementing the algorithms suggested and for conducting experiments.As a testing collection, we have created an experimental dataset of 600 paintings by 18 representative artists from different movements of the West- European fine arts from Renaissance, Baroque, Romanticism, Impressionism, Cubism, and Modern Art and Eastern Medieval Icons. Using the functionalities of APICAS we have conducted several kinds of experiments. The first set of experiments focused on the analysis of the colour distribution characteristics. We discover some specifics of given movements, which distinct them from the others. The examination of the features for all paintings revealed some common trends for art images. These results had been used later as a normalization factor in the process of defining the colour harmonies and contrast descriptors. The second set of experiments is focused on results of automatic annotation of the images with harmonies' and contrasts' descriptors. These high level features can be used not only in the processes of categorization focused on cultural influences and specific techniques, but also for extracting images with specific characteristics closely connected with the emotional responses evoked by an image. These features refer to corresponding elements of the abstract space of the image content. The third set of experiments is focused on examining the significance of local features, extracted by the proposed method, and more specifically on the type of underlying MPEG-7 descriptors; position of tiles; as well as number of clusters. They aim to decrease further examined features in order to accelerate the process without significant loss of accuracy in the classification tasks. A further set of experiments focuses on the evaluation of classification accuracy achieved by different types of classifiers. For the purposes of this work we conducted experiments with PGN classifier developed by our team as a specialised classification tool. The goal was to confirm the hypothesis that the PGN classifier has good representation for such kind of features compared with the other well known classifiers, such as OneR, JRip, and J48.To illustrate our hypothesis, the last group of experiments seeks to show the results of classification by PGN on the basis of painting periods in Goya's creative works. The work conducted in this thesis demonstrates the possibilities of narrowing the semantic gap using an appropriate set of defined features in combination with machine learning algorithms for upgrading the concepts. This paves the way for future research which may explore two directions: − Expanding the potential of the scope of examined visual features for defining higher-level concepts; − Widening the range of knowledge acquisition methods, which can be used for extracting such concepts.Work in these areas will be helpful in the transition from Web 2.0 to Web 3.0. In the era of Web 3.0 bridging the semantic gap is crucial. Finding appropriate combination of retrieval methods and techniques, which can lead to high quality image discovery, is a core problem in this domain and we hope that our work contributes to address part of these issues especially in the case of vast collections of digital images.