Journal of Advanced Research 17 (2019) 31–42 Contents lists available at ScienceDirect Journal of Advanced Research journal homepage: Review Analytical and mathematical methods for revealing hidden details in ancient manuscripts and paintings: A review Anna Tonazzinia, Emanuele Salernoa, Zienab A. Abdel-Salamb, Mohamed Abdel Harithb, Luciano Marrasc, Asia Bottod, Beatrice Campanellad, Stefano Legnaiolid, Stefano Pagnottad, Francesco Poggialinid, Vincenzo Palleschid,⇑ a National Research Council of Italy, Institute of Information Science and Technologies ‘‘Alessandro Faedo”, Via G. Moruzzi, 1, Pisa, Italy b National Institute of Laser Enhanced Sciences, Cairo University, Cairo, Egypt c Art Test Studio di Luciano Marras, via del Martello 14, 56121 Pisa, Italy d National Research Council of Italy, Applied and Laser Spectroscopy Laboratory, Institute of Chemistry of Organometallic Compounds, Via G. Moruzzi, 1, Pisa, Italy h i g h l i g h t s g r a p h i c a l a b s t r a c t Methods for revealing hidden details in ancient manuscripts and paintings are presented. Different experimental approaches are described. The most effective techniques of image analysis are introduced. Special attention is given to multi-spectral imaging and blind separation methods. Several case studies are presented. a r t i c l e i n f o a b s t r a c t Article history: Received 18 October 2018 Revised 7 January 2019 Accepted 8 January 2019 Available online 12 January 2019 Keywords: Image analysis Cultural heritage Archaeology Multispectral imaging Ancient manuscripts Blind separation techniques In this work, a critical review of the current nondestructive probing and image analysis approaches is pre-sented, to revealing otherwise invisible or hardly discernible details in manuscripts and paintings rele-vant to cultural heritage and archaeology. Multispectral imaging, X-ray fluorescence, Laser-Induced Breakdown Spectroscopy, Raman spectroscopy and Thermography are considered, as techniques for acquiring images and spectral image sets; statistical methods for the analysis of these images are then discussed, including blind separation and false colour techniques. Several case studies are presented, with particular attention dedicated to the approaches that appear most promising for future applications. Some of the techniques described herein are likely to replace, in the near future, classical digital photog-raphy in the study of ancient manuscripts and paintings. 2019 The Authors. Published by Elsevier B.V. on behalf of Cairo University. This is an open access article under the CC BY-NC-ND license ( Introduction Peer review under responsibility of Cairo University. ⇑ Corresponding author. E-mail address: (V. Palleschi). This review is focused on the analytical techniques and meth-ods that have been used to date and are likely to be used exten-sively in the near future to reveal hidden details in cultural heritage artefacts. Technically, all techniques used in archaeometry (the discipline that applies scientific methods to the study of cul- 2090-1232/ 2019 The Authors. Published by Elsevier B.V. on behalf of Cairo University. This is an open access article under the CC BY-NC-ND license ( 32 A. Tonazzini et al./Journal of Advanced Research 17 (2019) 31–42 tural heritage and archaeology) are aimed at revealing what is not evident and cannot be determined without the use, as a matter of fact, of specific analytical techniques and methods. To further define the scope of this paper, the discussion will be focused on the techniques that may help improve the interpretation and understanding of the manuscripts and paintings, not considering techniques such as radiography or X-ray tomography, which, although extremely interesting for their applications, are typically used for acquiring bulk information, well below the visible surface of the objects under study. These techniques would require, because of their complexity and importance, a full separate review. In the following, probing methods, instrumentation and digital processing techniques for the analysis of the artefact surface are described and discussed. Particular attention is devoted to spec-trally resolved imaging methods (reflectometry, fluorescence), although methods based on thermal or elemental analysis of arte-facts are also considered when functional to the recovery of surface information. Among the processing techniques, only those that operate on sets of images (separation techniques, false colour imaging, etc.), rather than those operating on single greyscale images (image enhancement technique, segmentation, etc.) are discussed. In the conclusion, a brief discussion of the most promis-ing approaches in the field is presented. Keywords, including image analysis, cultural heritage, archaeol-ogy, multispectral imaging, ancient manuscripts, and blind separa-tion techniques were searched through database ‘‘Scopus” from 1968 to 2018. Probing techniques Multispectral imaging – MSI Multispectral imaging is one of the most popular techniques for the study of cultural heritage and archaeological findings. One main advantage of MSI is that it is a non-invasive technique and therefore can be applied to any artwork, despite its possible fragi-lity. Although the spectral resolution of this type of analysis is, in general, limited (typical bandwidths are of the order of 50 nm or even larger in multispectral imaging and of the order of 10– 20 nm in so-called hyperspectral imaging), the amount of informa-tion that can be obtained is extremely high, considering the high spatial resolution of the images that can be obtained through very simple experimental setups. MSI, originally developed for remote sensing applications, began to be applied extensively in art conservation and art history in the early 1990s [1–7], as it can reveal information in an artwork that cannot be seen by the human eye. A multispectral image can be described as a set, or cube, of images of the same scene taken over different spectral ranges, i.e., at different wavelengths in the electromagnetic spectrum, including light outside of the visible range, such as infrared (IR) and ultraviolet (UV) light. Reflectance and fluorescence images can be independently acquired but treated simultaneously [8]. From an experimental point of view, an image in a multispectral cube (a channel) can either be isolated by specific filters [9] or using appropriate narrow-band illumination systems [10]. Scan-ning systems can also be used [11]. In the method’s simplest realization, four images of the subject under study are acquired in the blue, green, red and infrared spec-tral bands. In most cases, the infrared image is the one carrying the most information because of the unique ability of infrared radia-tion to penetrate the object surface, allowing for the visualization of otherwise invisible details such as underdrawings and penti-menti in canvas and panel paintings [11–14]. Infrared imaging is also important for other applications because of the possible enhancement of features deriving from the different infrared reflectivity of the subject’s constituent materials. The improve-ment of readability of degraded manuscripts in the infrared image was demonstrated, for example, in the recovery of the burnt Her-culaneum scrolls [15] and in revealing several hidden characters obscured by exposure to moisture in the Dead Sea Scrolls [16]. In other cases, e.g., in palimpsests or archaeological wall paintings, imaging in the UV spectral range often succeeds in providing addi-tional information [17,18]. In addition to highlighting hidden patterns, multispectral images and their further elaborations can also provide information on the materials used for the realization of a painting [19–22], on illumination conditions and pigment identification [23,24], and for monitoring the conservation of cultural heritage objects [25–27]. Grifoni et al. [28,29] recently proposed the use of spectrally resolved images as photogrammetric sources for building 3D mod-els of paintings that would carry information about the painted surface in depth structure (see Fig. 1). Multispectral and hyperspectral imaging, along with techniques for the digital processing of the acquired images, has been the focus of several national and international projects devoted to the study of precious artworks of great historical value. In most cases, dedicated imaging equipment has been devised and imple-mented. The study of ancient manuscripts and, among them, of palimpsests in particular, is one of the fields where multispectral imaging has demonstrated to give excellent results. The Archi-medes palimpsest project [30] has been one of the most important efforts in this field, aimed to the recovery from a XIII century prayer book of the erased and overwritten text of a earlier copy of two lost treatises of Archimedes. In the framework of this pro-ject, Easton et al. [31–33] introduced an MSI acquisition system that makes use of narrow-band LEDs. Other projects have been carried out regarding palimpsests in Europe, among which one of the most important and comprehen-sive has been the European Project ‘‘Digitale Palimpsestforschung‘‘ (2001–2004) [34]. The project was led by the University of Ham-burg and gathered the efforts of more than 50 partner institutions from 26 countries to study a large part of existing Greek palimpses-ts, with the help of newly developed digital technologies. From a technological perspective, new multispectral capture systems were among the results of this project, along with a set of basic image enhancement techniques and computer tools for document archiv-ing and cataloguing. In the project ‘‘Critical Edition of the New Sinaitic Glagolitic Euchology (Sacramentary) Fragments with the Aid of Modern Technologies” [35,36], a portable MSI system has been used to image the Sinaitic Glagolitic manuscripts. This system consists of two multispectral LED panels and two different cameras, a grey-scale camera with sensitivity from the UV to the near-infrared (NIR), and a traditional RGB camera utilized for UV fluorescence and visible-light imaging. Also in the field of manuscript analysis, Bianco et al. [37] described an MSI apparatus that uses a filter wheel consisting of eight different optical filters and a monochromatic camera for simultaneous 3D acquisition. Lettner et al. [38] introduced a similar MSI system with an extra single-lens reflex camera. Rapantzikos and Balas [39] used a system with optical filters for imaging over 34 narrow spectral bands. The efficacy of MSI for the analysis of texts was evaluated in [40]. X-ray fluorescence (XRF) X-ray fluorescence (XRF) can be used to support MSI for the non-destructive elemental analysis of those parts of the artwork in which MSI is ineffective. This technique consists of acquiring A. Tonazzini et al./Journal of Advanced Research 17 (2019) 31–42 33 Fig. 1. 3D multispectral reconstruction of the surface of a painting. (a) RGB; (b) UV–Vis fluorescence; (c) infrared [29]. the spatial distribution of the chemical elements [41,42] of large samples. When used to probe ancient manuscripts, XRF can distinguish among different types of iron-gall inks due to its high sensitivity to iron concentration and the impurities (typically of copper and zinc) that characterize different batches of ink or inks of different periods [43]. Experiments on the use of XRF for reading palimpsests have been conducted within a project carried out by the Centre for the Study of Manuscript Cultures at the University of Hamburg and the University library of Leipzig, in cooperation with the Hamburg synchrotron radiation laboratory (HASYLAB) and the German elec-tron synchrotron (DESY). Within that project, monochromatized, high-flux X-ray fluorescence techniques were employed [34]. Knox et al. [44] analysed the capabilities of MSI and XRF in revealing hidden characters in various types of damaged parch-ment manuscripts. The conclusions of this analysis were that the nature of the inks and the condition of the parchment would influ-ence what regions of the optical spectrum would reveal characters. As general rules, it was concluded that infrared illumination is good for revealing carbon-based ink on blackened parchment, ultraviolet fluorescence (and sometimes reflectance) can enhance erased characters, and finally, X-ray fluorescence can detect iron gall ink that is completely covered by optically opaque materials. Thermography Infrared thermography [45] can also be used effectively to reveal the presence of hidden patterns or structures in a large vari-ety of objects. Multispectral imaging normally detects the near-IR radiation emerging from the objects under test (0.75–1.4 lm wavelength range); the typical wavelengths used for thermogra-phy belong to the thermal IR range (3–15 lm). Techniques based on infrared thermography are capable of detecting subsurface fea-tures in the investigated object by mapping the temperature distri-bution at its surface and can be implemented in different experimental arrangements [46]. A first distinction can be made on the possible presence of an artificial illumination system: pas-sive techniques evaluate temperature differences naturally occur-ring at the investigated surface, whereas active techniques rely on the temporal evolution of surface temperature induced by suit- ably timed and filtered artificial heating systems (usually flash lamps). Both of these approaches have already been used to inves-tigate many classes of objects relevant to cultural heritage, such as historical stone and masonry artefacts [47–49], archaeological findings and ancient documents [46,50–52]. In particular, active pulsed thermography has been successfully applied to non-invasively highlight the presence of ancient texts in parchment book bindings, to characterize the status of conservation of painted decorations and to reveal the presence of possible pentimenti under the painted surfaces [46,52]. Raman and LIBS imaging The effectiveness of using micro-Raman imaging, a technique that provides information about the molecular structure of sur-faces, together with MSI, was evaluated by Maybury et al. [53] in an analysis of Armenian manuscripts. Deneckere et al. [54] used micro-Raman imaging coupled with the elemental technique of micro-XRF to acquire elemental and molecular images of a Belgian porcelain card. Bicchieri et al. [55] used MSI, FT-IR spectroscopy, micro-Raman and micro-XRF for the analysis of a degraded 18th century manuscript. Finally, Botteon at al. [56] used a variation of Raman microscopy called spatially offset Raman spectroscopy (SORS) to demonstrate the possibility of recovering painted images hidden by, for example, graffiti or other types of overpainting. In fact, any experimental technique capable of reconstructing spec-trally resolved images of the surface of cultural heritage artefacts can be used for recovering hidden information. Elemental images obtained using Laser-induced Breakdown Spectroscopy (LIBS), a micro-destructive spectroscopic technique, were reported in [57] and [58]. Among these, non-destructive approaches are obviously preferable, when applicable. Digital processing techniques Statistical analysis and source separation Among the image processing techniques typically explored using MSI data, statistical analysis and dimension reduction have proven to be powerful tools for further enhancing and detecting hidden patterns in artworks or removing unwanted interferences. Dimension reduction can be both unsupervised, as in blind source 34 A. Tonazzini et al./Journal of Advanced Research 17 (2019) 31–42 separation (BSS) techniques [59,60], and supervised, as in Fisher linear discriminant analysis (LDA) [61]. Indeed, unsupervised dimension reduction techniques, such as principal component analysis (PCA) and independent component analysis (ICA), linearly combine highly correlated spectral images to produce a different set of images that are uncorrelated and show decreasing variance. Furthermore, the output channels of ICA are statistically independent. Thus, the main principle underlying the enhancement capabilities of dimension reduction techniques is that, while the spectral components of an image are usually spa-tially correlated, the individual patterns (or classes, or sources) superposed onto the image are usually much less correlated. Hence, decorrelating the colour components gives a different rep-resentation, where the now orthogonal components of the image could coincide with single classes [62–66]. For example (Fig. 2), for palimpsests containing mixtures of two different texts and pos-sibly further information layers (parchment texture, mould, etc.), dimensionality reduction often results in images in which each shows a single layer separated from the others. Because the statis-tical independence requirement of ICA is a stronger condition than the assumption of uncorrelation of PCA, it is possible that signals that are not well segmented by PCA may be separable by ICA or by ICA applied to a set of principal components, as done in [67]. PCA, ICA and other orthogonalization methods, when applied to multispectral images, can increase the readability of degraded texts [68–70] or reveal hidden features not apparent in any of the individual input images, as in [66], in which a hidden text was shown to exist in a XVIII century painting, or in [71], in which the existence of many otherwise hidden details was demonstrated in wall paintings found in the Etruscan Tomb of the Monkey (Chiusi, Italy, 5th Century BCE). BSS techniques were particularly important in investigating the lost mural paintings in the Etruscan Tomb of Blue Demons (Tarquinia, Italy, 5th Century BCE), as reported in a recent paper by Adinolfi et al. [18]. In that study, a set of visible, infrared and fluorescence images was treated statistically by BSS algorithms, revealing a magnificent hunting scene with three hun-ters, a wild boar, a deer, a dog, and two felids, where the naked eye could perceive only a white wall (a detail of the scene, depicting the wild boar and the head of a hunter, is shown in Fig. 3). If the mutual independence assumption is not tenable, the ICA-based strategies for source separation may fail. One option in this case could be to rely on dependent component analysis (DCA), a class of model-based techniques that exploit other possible proper-ties of sources or mixtures to reach their goal [72,73]. This type of technique has been applied extensively to fields such as remote sensing [74,75] and medical imaging [76]. Although some DCA approaches could be employed to analyse different types of cul-tural heritage-related images, only one proposal is present in the open literature in which a DCA approach is used for the digital restoration of colour images of double-sided documents [77]. Fisher linear discriminant analysis (LDA) can also be applied to reduce the dimensions of multispectral scans and to enhance degraded writings. Because Fisher LDA is a supervised dimension reduction tool, it is necessary to label a subset of multispectral data. To this end, in [78], a semi-automated label generation step was introduced based on an automated detection of text lines. This approach is thus based not only on spectral information, as in PCA and ICA, but also on spatial information and, when tested on two Slavonic manuscripts, has yielded better performance compared with that of unsupervised techniques. Fig. 2. Folio 16v-17r of the Archimedes palimpsest. (a) RGB image under strobe lamp illumination. (b) Second component output (contrast-enhanced) from the 2 2 PCA of the red and blue colour channels, revealing the underwritten text and drawings ( The owner of the Archimedes Palimpsest, licensed for use under creative Commons Attribution 3.0 Unported Access Rights. Image processing: The Institute of Information Science and Technologies, National Research Council of Italy). A. Tonazzini et al./Journal of Advanced Research 17 (2019) 31–42 35 Fig. 3. Detail of the hunting scene (a wild boar, running from right to left) recovered using BSS in the Tomb of Blue Demons in Tarquinia, using MSI and BSS. On the right, the visible image of the wall. Note the improvement in readability of the wild boar (muzzle with ear and fang is evidenced in the yellow circle) and of the head of one of the hunters (red circle) and vegetation at his right. The self-organizing maps (SOMs) method, introduced by Koho-nen at the end of last century [79], represents a completely differ-ent approach to the blind separation problem. SOMs are artificial neural networks that achieve separation through the similarity of the (optical) properties of materials, which are represented in an n-dimensional space by the coordinates of the corresponding (hyper)-colours. Unlike in BSS, the number of images that can thus be extracted by a multispectral set can be greater than the number of images in the original set. No hypothesis is made on the linearity of the model, and the information layers are separated through an iterative, competitive process between the neurons that ‘‘move” in the hypercolour space arriving, after convergence, to assume the coordinates of the centroid of the corresponding cluster. This method requires the definition of a metric that determines the similarity of the hypercolours defining different materials (Eucli-dean, Angular, Manhattan, etc. [80]). The number of neurons is also left to the decision of the operator, based on the expected number of different materials/optical responses in the physical object [65]. An important advantage of the SOM approach is that no dimen-sional reduction must be performed for the classification of mate-rials; the position of the neurons in the hyperspace represents the ‘‘prototype” of the optical properties of the corresponding material. The hypercolour associated with each pixel in the MSI image may have components corresponding to visible and infrared reflectivity, fluorescence, and elemental or molecular information. Applica-tions of SOMs to elemental images obtained by the LIBS technique were reported by Pagnotta et al. [57,58] (Fig. 4). In addition to BSS, SOM and LDA, non-blind spectral unmixing has proven useful in text analysis. This approach, popular in remote-sensed hyperspectral image analysis, is based on the avail-ability of a dictionary containing the typical spectral signatures of the materials of interest and unmixing strategies such as spectral angle mapping (SAM) [81] capable of labelling the different sensed regions as belonging to specified classes [82,83]. In document image analysis, pixel regions belonging to specific object classes, e.g., parchment, mould, overwriting, or erased text, are first identi-fied by the user. An algorithm then computes the class member-ship of each pixel in the image based on the similarity of its spectrum to each of the specified classes. Although intensive both in terms of human interaction and computation time, this method was applied with success to the Archimedes Palimpsest [84–86]. Spectral unmixing for document image analysis can be particularly useful in situations in which different feature spectra are known or can be determined a priori, as in remote sensing for earth observa-tion, where the spectra are known from field or laboratory measurements. Pseudocolour imaging A simple approach for enhancing hidden features in an artwork when appropriate non-visible bands are available is a rendering technique called false colour or pseudocolour. Because only three spectral bands can be displayed in a colour image, three suitable images are selected from the multispectral set and superimposed in the form of a (false) colour image. The most common combina-tion of the multispectral images is infrared, red and green (IrRG), although the combination infrared, green and blue (IrGB) [87] is also used. The procedure implies that one of the visible colour channels is discarded (the blue band in IrRG false colour imaging or the red band in IrGB imaging), and the information it contains is not present in the pseudocolour image (Fig. 5). Pseudocolour imaging can be generalized in several ways. For example, to render the image data used in the study of the Archi-medes Palimpsest, images captured through a blue filter under ultraviolet illumination, where the underwriting was mostly visi-ble, and through a red filter under tungsten light, where the under-writing had nearly disappeared, were combined to render the overwriting in black and the underwriting in a reddish tint. In the resulting pseudocolour image, the two texts were then percep-tually well separated because they featured highly contrasting col-ours, enabling the reader to distinguish between them [88]. When using data reduction methods, if layers are perfectly sep-arated, each feature class would dominate the greyscale range in the related output channel, while pixels belonging to the other fea-ture classes would exhibit the same grey value and thus merge with the background in that channel. More realistically, PCA or ICA may not succeed in separating features if the different feature patterns are not truly orthogonal or independent. Thus, in palimpsests, traces of overwriting usually appear in those channels in which the erased underwriting is most visible, and the variation in statistics across the scene, for example to variations in erasures, makes the erased text often appear in more than one output chan-nel with varying intensity. This fact can, however, be exploited to generate pseudocolour rendering of extracted component images, 36 A. Tonazzini et al./Journal of Advanced Research 17 (2019) 31–42 Fig. 4. SOM segmentation of a set of elemental images obtained on a Roman mortar sample using l-LIBS [57]. The yellow square in the figure indicates the zone analysed. Fig. 5. Schematic representation of the procedure used for building IrRG and IrGB false colour images from a multispectral set of images. where small variations in grey value may appear as large changes in colour, thus further improving the readability of the erased text [89]. A similar effect can be obtained by changing the pseudocolour rendering by varying the hue angle or by creating weighted combi-nations of images, including results from ICA and PCA and possibly original image bands. Legnaioli et al. [71] introduced a false colour imaging technique called chromatic derivative imaging (ChromaDI), which exploits the subtraction of consecutive couples of 4 consecutive spectral images, namely, G-B, R-G and IR-R. This method was developed with the intent of building a false colour image that would take into account the information from all multispectral images acquired, without excluding a priori one of the four images in the multispectral set. The ChromaDI image provides information on the changes in reflectivity of an object with wavelength. With respect to the canonical false colour image, the differences between the optical behaviour of various pigments are enhanced, taking into account the changes occurring while passing from short wavelengths (blue band, which is more sensitive to surface details) to longer ones (green and red bands) in the visible image (see Fig. 6). ChromaDI has been successfully applied to images of a Roman painted sarcophagus, III century A.D., and to images of a mural A. Tonazzini et al./Journal of Advanced Research 17 (2019) 31–42 37 Fig. 6. Schematic representation of the procedure used to build a ChromaDI image. painting of an Etruscan tomb in Chiusi (Siena, Italy), among arte-facts. The method can easily be generalized to multispectral sets containing more than four images. For instance, in palimpsests, ChromaDI images could include one or more channels of UV fluorescence. Another false colour imaging method, only experimented on paintings to date, aims at producing chromatically faithful pseudo-colour images, which maintain good readability of the information contained in the infrared band. Examples of the application of this technique include the multispectral images acquired for the Pietà of Agnolo Bronzino (1569, Florence) and the analysis and visualiza-tion of the multispectral data obtained from Etruscan mural paint-ings (Tomb of the Monkey, Siena, Italy, V century B.C.) [90]. The method is called gradient transfer and, through a regularization strategy, merges the information from the IR band into the RGB image, preserving at best the chromatic similarity with the visible image (Fig. 7). A similar approach for image inpainting exploiting infrared information has been recently proposed by Calatroni et al. [91] for removing overpaintings in the visible image in the analysis of illuminated manuscripts and by Peng et al. [92] for mining patterns of painted cultural relics in ancient pottery and murals. In the context of ancient manuscripts, e.g., palimpsests, the IR band could be substituted by the blue band of the UV fluorescence, where, presumably, the underwriting is best visible.