Heritage and Repeat Photography: Techniques, Management, Applications, and Publications

Abstract

Repeat photography, the process of locating the camera position of an old photo and taking a new photo from this position, is utilized to protect our natural heritage, by measuring landscape changes between old and new image, and to conserve and recover our architectural and archaeological heritage, by observing changes between images taken before, during, and after conservation efforts or destructive events. Most importantly, repeat photography—in short, rephotography—is connecting us to our heritage on all levels from personal to global: while rephotographing, we connect with the old image and its creator and—when viewing and comparing the old image and the rephotograph—we connect with the tangible, intangible, and natural heritage shown in the images by observing changes and continuities. In this paper, we review publications and applications employing rephotography. We discuss the organization, retrieval, and presentation of rephotographic compilations, their generation by different image registration techniques, and their conservation as rephotographic heritage. Further, we describe challenges in rephotography, discuss how they differ from standard challenges in photography, computer vision, and image registration and present both analog and digital solutions and tools used in practice by rephotographers to address said challenges.

1. Introduction

Repeat photography describes the process of photographing a scene, of which a previous image exists, again from the camera position of the previous image. This results in an image pair showing the same scene from the same position at different points in time, allowing to examine changes in the scene between these two points in time. Figure 1 shows an example; ref. [1] shows more.

To create a repeat photograph—in short, rephotograph—the following steps have to be completed: (1) An old image, which will be rephotographed, is retrieved or created. (2) The location and possible the recording time of this old image is determined. (3) The location is visited and the precise camera pose is found. (4) The new image is taken. (5) The new image is registered, i.e., aligned, to the old image. (6) The images are presented in a suitable format or measurements are taken from them.

The motives to create rephotographs are manifold: Traditionally, rephotographs are used in natural heritage to photogrammetrically measure changes in a landscape scene. They are used to tell a story themselves, strongly visualize a fact, and as a starting point for a story, raising the question, why and how the changes visible between the old and new image occurred. They are used to show two frozen points in time but may require additional information to tell the journey from one point to the other. Rephotographs can also help to connect with the creator of the old image, by standing at the same position and viewing the same scene, while viewing the old image or recording the rephotograph. They embed intangible heritage, like historic events, in the present by showing which events took place at locations such that one might know and recognize in the new image. They are personal narratives, by visualizing the human aging process, weight losses, and the history of one’s birthplace.

With this paper, we aim to give an overview of the broad field of rephotography and its applications. Furthermore, we motivate why and how it can serve as useful tool in many fields related to heritage, highlight difficulties, and discuss which techniques were and are used to overcome them. Moreover, we aim to provide sources for active rephotographers as well as academics considering to utilize existing rephotography.

The paper is structured as follows: After presenting different definitions of rephotography and vocabulary in Section 2, we discuss the origins of rephotography in Section 3. In Section 4, we present applications of rephotography with the goal to visualize the popularization of rephotography from its origin in art and glaciology to mainstream applications in personal heritage. We begin with applications in natural heritage and then move to other scientific applications in tangible and intangible heritage. This is followed by an overview on nonscientific applications both published in print, e.g., as illustrated books, and online, as project websites or on social media. The section is completed by an overview of related fields.

In Section 5, we present online portals and management software for rephotography and identify different options to collect, organize, and retrieve rephotographs across continents and applications. Rephotographs are hard to retrieve, since no specialized search engines exist. This motivates the creation of Web portals for sharing and retrieval of rephotographs.

In Section 6, we present challenges in rephotography, addressing the determination of the correct camera location and time, the recording of the new image, and the registration of the new image to the old image, after recording. Here, we show that rephotography is not as straightforward as it might seem and indicates caveats that a potential rephotographer has to be aware of.

We offer solutions to the previously presented challenges in Section 7. We follow a typical rephotographic process, discussing the acquisition of historic images, georeferencing these images, finding the camera position, recording the new images, and registering and postprocessing these new images. Here, we compare traditional analog solutions with digital solutions. We examine how algorithms from the field of computer vision can be applied to rephotography and how the challenges in rephotography differ from standard computer vision challenges. This part concludes with a comparison of different presentation methods of rephotographic compilations.

Last, we summarize the current state of rephotography and derive incentives for future research in Section 8 and Section 9.

2. Vocabulary and Definitions

Traditionally, rephotography is defined as photographing a geolocatable landscape scene, of which a historic image exist, again from the camera position of said historic image. This ideally results in a perfectly registered pair of images. In registered images, the same or corresponding objects are at the same image coordinates in both images. This in turn allows to quantitatively measure changes in the scene between the two images using photogrammetry.

This definition of rephotography can be generalized by encompassing non geolocatable scenes, including objects or people, which do not necessarily need to be rephotgraphed at the same location. The definition can also be generalized by omitting the condition of perfect registration. This is often sufficient for visualization but inhibits comparative photogrammetry.

In a very general sociological definition, rephotography tracks changes in activities, functions, and processes, not limited to the same locations, subjects, and objects. This means that the images may show completely different scenes, but the photographed activities have the same goal or outcome.

Normally, a slightly generalized definition is used, requiring rough image registration and the presence of at least one discernible object being visible in both images, allowing viewers to connect the two images and to ensure that the same scene is shown in both images. Being able to find these connecting objects and being able to analyze how they changed is one of the core aspects in making rephotographs interesting.

Vocabulary is diverse. The term rephotography is seldom used itself, especially in nonacademic contexts. In academic contexts, the following terms are used: Multitemporal Imagery generally refers to images taken at different points in time, Photomonitoring refers to change monitoring in a scene with several images taken over time. The terms Bitemporal Change Detection in Pairs of Images and Temporal Trajectories are used as well.

In other contexts, even broader terms are used and titles of rephotographic publications contain phrases like Then and Now or Now and Then, X through Time, X Then and Again, X Revisited, or Looking into the Past, or Past and Present. Other keywords are Leaps in Time, Window to the Past, and Link to the Past.

The names for the old and new images vary as well. They may be called historic and contemporary image, first and second image, and, coming from the field of image registration, fixed and moving image. The later naming scheme indicates which image is kept immovable and which images are transformed or warped during registration to match the fixed image. In landscape change research, the old image is sometimes called benchmark. The old image can also be called template, for which a new image, the rephotograph, is recorded. The pair of new and old image as a whole is called (rephotographic) compilation, diptych, or image pair.

In the following, we will use the terms old and new image to refer to single images and the term compilation to refer to the pair of old and corresponding new image. Generally, a rephotographic compilation encompasses two images, but compilations with three or more images exist as well.

3. Origins of Repeat Photography

The first rephotographs are nearly as old as photography itself. The 1838 photograph Boulevard du Temple [2] by L. Daguerre is considered as the first photograph featuring people. It was rephotographed on the same day [3]. Other examples are the photographs of the Sphinx and the Great Pyramids taken by F. Frith in 1857 [4] and 1858 [5] from nearly the same position. However, these first rephotographs were likely unconscious or unintentional rephotographs. New images of the same scene were taken to check the camera or film, to substitute broken or deteriorated glass negatives, and to reflect changes in the scene, without the intention to display old and new images together. Other reasons are experiments with different lighting conditions, camera parameters, and positions. For mass reproduction of printed matter like postcards, more original negatives and backups might also have been desired. For example, the different versions of Boulevard du Temple were likely taken to examine the influence of different lighting conditions (morning vs. midday) on the newly engineered daguerreotype process [6].

The first intentional rephotographs and rephotographic series, titled Anatomy of Foliage, were created by E. Fox between 1865 and 1866 and encompass compilations of eight trees taken both in summer and winter. These were taken from the same position “enabling the student to trace the limbs when hidden by the masses of foliage” [7].

The end of the 19th century marks the beginning of scientific rephotography: S. Finsterwalder recorded images of the glaciers of the Alps to photogrametrically measure changes in the alpine glaciers using colored stones and landmarks as markers [8]. Already then, Finsterwalder formulated the idea of citizen scientists by asking other photographers to rephotograph the Alpine glaciers [9].

From European glaciology, scientific rephotography spread to other fields and continents [10,11]. Today, rephotography is no longer limited to a scientific context with publications in books or papers but extends to art, causal uses, and is adopted in digital and online formats. Nonetheless, the photographs from the 19th century are still used for rephotography and glaciology today [12,13].

4. Applications and Publications

Rephotographs are used and published in scientific papers, illustrated books, exhibitions, videos, and online resources. In this section, we will give examples of these different publication and application forms, to illustrate how diverse and global rephotography has become.

4.1. Scientific Publications

Most of the scientific publications employ rephotography to measure geological or ecological change. Other works measure social changes, which may be reflected in landscape changes, use rephotography as form of art, and employ it as a teaching aid. Last, more reflective works discuss rephotography from a history of art or psychophilosophical point of view. In the following, we will present exemplary work from the different fields.

4.1.1. Landscape Change

A wealth of publications employs rephotography to monitor landscape changes. We will exemplary present recent publications from each continent to elucidate the spatial range and topical diversity of these publications. R. Webb et al. give a thorough view on the applications of rephotography in landscape change research [14]. Publications addressing landscape change mainly in the Western United States of America up to the year 1984 are compiled in a bibliography by G. Rogers et al. [15]. This bibliography contains over 450 entries, each employing rephotography in some way.

In Africa, C. Kull et al. employed rephotography to identify trends in land use research in highland Madagascar [16]. They found that rephotography provides deeper historical reach and higher-resolution data compared to satellite and aerial imagery. Moreover, they found rephotography to be time- and cost-efficient if combined with other field work.

In Antarctica, J. Putkonen et al. used aerial and terrestrial rephotography together with soil traps and measured pebble transport distances to examine regolith transport [17]. Transport of regolith, a layer of loose dust, soil, and broken rock covering solid rock, eradicates records of geologic and climatic history, present in the Dry Hills of Antarctica. The combination of different measurement methods provide lower and upper bounds for topographic diffusivity: Rephotographs provide a lower bound, since movement of fine sediment cannot be measured.

In Asia, P. Chand et al. examined glacier fluctuations of the Bara Shigri Glacier and their link to climate change after the Little Ice Age (LIA) [13]. They used a multidata integrative analysis (MDIA) including rephotographs. They conclude, that combining rephotography with field observation, historical archives, documents, and satellite imagery into a MDIA for long-term glacier fluctuation reconstruction is useful.

In Australia, B. Brook et al. used georeferenced aerial rephotography to understand landscape change, demonstrating an expansion of closed forest in the Australian monsoon tropics [18]. Moreover, they found that a retrospective habitat suitability index (HSI) model based on historic images correctly predicted forest expansion. They found that aerial rephotography offers precise time control, has high historical depth, and offers high resolutions over wide areas. However, historical coverage is patchy, from irregular points in time. Moreover, flight attitudes as well as film material and cameras change.

In Europe, M. Peichl et al. used rephotography for phenology and climate research: They monitored vegetation phenology in a boreal mire in northern Sweden [19]. Furthermore, they found that rephotography is a valuable tool to measure both mire phenology and carbon dioxide exchange.

In North America, J. Jenkins et al. rephotographed historic panoramic images using stereoscopic imagery and visualized the panoramic compilations in a Wide–Area Visualization environment (WAVE) room [20]. Old images were taken by Lester Moe as part of the 1930s New Deal programs with a unidirectional panoramic camera along the southwestern edge of Yosemite National Park. New images where taken with an omnidirectional stereoscopic camera. The old smaller images are embedded in the new full 360° images, opposed to traditional rephotography, where both image share the same view. Jenkins et al. argue that this changes ones baseline of change perception to the now and raises the question “what has let us here?”.

In South America, M. Masiokas et al. documented significant glacier recession and hydroclimatic changes in northwest Pantagonia [12]. They developed a climatic index, which predicts a negative trend in glacier mass, agreeing with the glacier recession visible in the rephotographs. They found that rephotography gives powerful baseline information on local glacier long-term behavior, especially since the area was visited early by explorers and researchers. They provided photographs along with maps and site description, allowing for a good registration of old and new image.

4.1.2. Ecology

Rephotography of animals can be used to track them and study migration behavior. The happywhale project collects georeferenced photographs of marine mammals taken by citizen scientists. They identify individuals and publish the data on their webpage, allowing to track individual marine mammals around the globe [21]. However, animals are seldom the direct subject of rephotography. More often, changes in their habitat are investigated with rephotography (see [14], Part IV). Many rephotographs of animals, which are not used in academic publications, exist, but they illustrate the success of diets, medical procedures, rescue efforts, and cleaning of oil-covered sea birds.

4.1.3. Art and Culture

This section presents exemplary artistic and photojournalistic works of rephotography. Although photo series and exhibitions employing rephotography are not per se scientific publications, they have an intellectual claim and are thus presented here. Note that there are no clear boundaries between these publications and illustrated books, discussed in Section 4.2.

In his photo series Ausencias, G. Germano employs rephotography to draw attention to the abductions and murders by dictatorial regimes in South America from the 1960s to the 1980s [22]. Germano rephotographed family snapshots of intact families before the regimes. In the new images, the victims of the regimes are absent, leaving a void in the family (portrait). M. Miles discusses Germanos work focusing on a shift of an objective past to personal narratives [23].

Photojournalist L. Snow created compilations of three images, showing soldiers before, during, and after deployment to Afghanistan and combined these images with quotes, given by these soldiers also before, during, and after deployment [24]. This series, called “We Are The Not Dead”, visualizes the manifestations of the experiences of war in the faces of these soldiers. Snow’s series was preceded by Marked, a similar series by C. Felicie of Netherlandish soldiers before, during, and after deployment to Afghanistan [25].

A. Bamji combines art, history, and medicine with a book collecting compilations of World War I (WWI) soldiers incurring facial injuries [26]. The book contains images taken before and after plastic surgery on the soldiers’ faces and analyzes the development of plastic surgery and its impact on the life of the soldiers. For more information on the relation of rephotography and medicine, see Section 4.5.4.

Published in 1938, 20 years after World War I (WWI) a book containing images of the Western Front taken during and twenty years after WWI is one of the oldest rephotographic books [27]. It contains old images from the Imperial War Museum in London and the Archives Photographiques in Paris, which were then rephotographed after WWI. Notably, this is one of the few rephotographic works on war; it offers a positive outlook showing the successful reconstruction efforts in cities as well as recovering landscapes.

J. Langer used rephotography to visualize the human aging process [28]. He combined image from family albums with new portraits of Czechs, who are more than 100 years old.

B. Notteboom et al. collected quadruplets of images taken at the beginning of the 19th century by botanist J. Massart, and in the years 1980, 2004, and 2014 of the Flanders region of Belgium [29]. The first series of images aimed, to visualize connections between natural vegetation and its landscape context as well as between agriculture and geography. The second series visualized the impoverishment of the natural environment. These first two series were displayed in a traveling exhibition. Notably, in the book, the quadruplets are combined with a description text and a map section. The later making rephotographing the scenes again straightforward. This is one of the few works featuring compilations of more than three images with two being the standard.

British photographer J. Southam analyzed landscape change in his series The Shape of Time: Rockfalls, Rivermouths, Ponds and Once. Again. Photographs in Series rephotographing coastal cliffs in Great Britain and France [30]. In archaeology, rephotography is used to visualize change in historic sides. (War) damage can be visualized, restoration and reconstruction works can be supported, and their success can be evaluated with rephotography. For example, the demolition of the Buddhas of Bamiyan [31], the restoration of the prehistoric passage tomb Newgrange [32], and heritage sites in Malta [33] are recorded with rephotography.

Iconic photographs are subject to rephotography as well. Examples are the photographs of H. Cartier–Bresson, capturing the decisive moment. His works are re-enacted in place [34] and are rephotographed in the studio with construction blocks toys and minifigures [35]. In the later, rephotography becomes more a homage or citation. Here, detecting change is no longer the focus.

For two iconic photographs, Moonrise, Hernandez, New Mexico and Moon and Half Dome by A. Adams, great effort was made to correctly compute time and date from the position of the moon or other celestial bodies in the old image not only to create a rephotograph with the celestial bodies at the correct positions but also to “connect” with Adams by standing at the same position at the same time of the year or after a certain number of Metonic cycles [36,37].

4.1.4. Education

Rephotography can be used in two ways in education: (1) Students can be confronted with complete compilations or (2) can be tasked to create rephotographs themselves, using either given templates or having to create these also themselves.

K. Lemmons et al. examine the effect of rephotography on cultural understanding and interaction in short stays abroad [38]. Students abroad were tasked to rephotograph given templates, of which the exact location is unknown. They claim that using four steps—(1) asking the motivation behind the old image, (2) reflecting on one’s own view and influences while taking the new image, (3) reflecting on the compilation, and (4) forming an understanding of the changes in the (cultural) landscape—students would become active participants. They would be more comfortable in and better understand the host’s culture. Experiments showed that students did not relate rephotography to culture by themselves but found it to be a medium of communication without knowing the host’s language. Interviews revealed that a historic context and not only an old image is required for understanding, that having to search for the correct location forces one to actively see the surroundings or even ask locals for help, and that a certain amount of comfort with the host community and vice versa is required for communication.

G. McLeod finds it promising to use rephotography in class but argues that further research is required to develop strategies [39]. Although he does not answer his initial question, whether rephotography can deepen critical visual skills in the classroom, he argues that it can encourage students to think about image creation and usage.

McLeod published further results, examining students responses on a rephotography task, focusing on their rephotography accuracy and the depth of the response to the compilation [40]. He states that students use rephotography more as a tool, than seeing it as being intrinsic when examining subjects over time.

A. Metcalfe argues that rephotography can not only be a tool in education, but can be employed in educational research. Rephotography can be used to display changes in educational spaces and places and to base educational policies and practices on these observations [41].

4.1.5. Social Science

C. Burton et al. link landscape change research with social conditions, while examining the recovery operations after Hurricane Katrina with rephotographs [42]. Photographs of 131 scenes were taken every 6 months over 3 years. For each photograph a recovery score was given and for each of the six points in time a recovery surface was computed visualizing the recovery amount over the whole study area. They found rephotography promising for postdisaster recovery assessment, even though flexibility and ancillary data is required.

J. Walker et al. combine landscape and social change as well, while rephotographing the works of R. West and J. Parson of the Topia Road in Mexico [43]. The mining road connects remote communities influenced by mining, migration, and drug cultivation and trafficking. Methodological, they found that taking the old images into the field encouraged locals to participate in the project. Moreover, the rephotographs feature only a small part of the occurring changes, but act as a catalyst for discussion.

B. Ganzel rephotographs the scenes from the images taken for the Farm Security Administration (FSA) during the Great Depression [44] in the Great Plains. He tracks down the subjects of the old images or their relatives and rephotographs them. For example, he rephotographs F. Thompson and her daughters, the subjects of D. Langes iconic photograph Migrant Mother, resulting in a compilation of the same subjects in the same pose but in contrasting scenes, situations, and attires.

The concept of rephotographing iconic portraits is not only used by Ganzel: K. Phúc the main subject of N. Uts photograph The Terror of War, was asked to re-enact the scene with her family and then rephotographed [45]. S. Gula, known as the Afghan Girl from the cover of the National Geographic, was also searched for, identified, and rephotographed [46].

J. Rieger uses the term rephotography in a more general sense. For him the repeating factor is neither the scene nor the camera position but the goal and outcome of human actions and methods. He uses tree felling as an example: one image of his compilation shows the historic method, using a chainsaw, and the other the modern method using a feller buncher [47].

4.1.6. Reflections

In this Section, we present publications reflecting on rephotography from a history of art or psychophilosophical perspective. P. Moore discusses rephotography with a focus on Scotland and relates his own personal experiences with rephotography fieldwork in Scotland [48]. He examines how rephotography constructs visual narratives and how these narratives can be used to understand changes in people and places over time. N. Kumar links rephotography to the psychoanalytical concept of deferred action, claiming that repetition in rephotography generates remembrance through this concept [49]. Similarly, J. Kalin argues that rephotography is not a form of memory but is a practice which actively generates memories, positioning them as lever to the present [50]. L. Munteán on the other hand examines connections between rephotography and the semiotic phenomenon of indexicality and compares different presentation formats (see Section 7.7) with respect to ones engagement with the past generated by them [51]. The observations of S. Lyons are based on an unorthodox definition of rephotography, defining photographic appropriation art, i.e., the reproduction or copying of photographs of another artist, as rephotography. She examines photographic appropriation and originality in online photographic platforms as well as legal and cultural views on them [52]. T. Smith reflects on rephotography from a visual anthropological point of view and sees it as an ethnographic strategy joining anthropology and visual art, aiding understanding of humans in the world. He examines, why humans view and photograph landscape repeatedly from the same vantage point and what knowledge over the place, humans and their relationship can be constructed from revisiting said vantage points [53].

4.2. Illustrated Books

A large number of illustrated books on rephotography exist. The first publications date back to the seventies. Both publishers and authors specialized in rephotographic series exist. Here we present English and German language books. These illustrated books can be divided in the following classes:

1.

Books which address a certain subject most often a region or a town. They may be published by the town, by a local heritage society, by interested citizens, or as part of a repeat photography series. While some address tourists or residents of the area, covered in the book, others have a more scientific scope and provide detailed historic information in written form. Still others are more artistic and less documentary. Quality ranges from cheap paperbacks with generic layout to high quality coffee table books.

2.

Books which follow the tracks of one or more iconic photographers or expedition. In these books all the old images are taken by the same photographer and are rephotographed by one or more contemporary photographers. Since these books often follow expeditions or journeys, they are limited to a certain region. They are of better quality and not published as part of a series.

Obviously, these classes can overlap, but there are clearly two (coexisting) motivators: interest in an area and interest in a photographer or expedition. Overall, there are at least 2000 illustrated English-language rephotography books. They are difficult to find, since they do not carry rephotography related terms in their titles or cover texts, and are self-published with a small number of copies.

4.2.1. Illustrated Books Addressing a Certain Subject

One of the first examples was published in 1976. It contains 83 rephotographs of Manhatten [54]. This book is set in the classical layout of illustrated rephotography books. Each compilation is presented on a spread, with the old image on the left and the new image on the right page and a short caption under each image.

Examples of local history books are the books of the German municipalities Großenkneten [55] and Emlichheim [56]. These books describe the development of the respective villages and substantiate the text with rephotographs. In these books the rephotographs do not necessarily register perfectly and are not always arranged next to each other or as clearly recognizable entity. Whereas the first book is published by a local resident, the second is published by a heritage society.

In the United States historical societies publish rephotography books, too. For example, the Alburquerque Historical Society published a book [57] juxtaposing old and new photographs of Alburquerque. They present a notable presentation technique by cutting out rectangular parts of the old image and positioning them over the new image without blending.

Other scientific publications are the work by G. Gruell, who visually documented vegetational and cultural changes and the influence of wildfire in the Northern Rocky Mountains [58] as well as the earlier publication by W. Phillips. He documented vegetation changes in the Northern Great Plains [59]. While Phillips work is more descriptive, giving location, recording time, and a description of the scene, Gruell also analyzes the changes emphasizing the influence of fire.

Notable are also two books by R. Turner et al. and J. Hastings et al., who rephotographed the Sonoran Desert region both in 1965 [60] and before 2003 [61]. While the old book contains photographs from the late 1800s and early 1900s and their rephotographs, the new book contains triplets of the old photographs, old rephotographs and new rephotographs. The books visualize vegetation change and analyze causes for these changes.

4.2.2. Illustrated Books with Historic Images of a Single Photographer/Expedition

The rephotographic survey project covered the American West and was published in 1984 [62]. M. Klett et al. followed the tracks of the expeditionary photographers of the nineteenth century, among others, W. Jackson, T. O’Sullivan, and W. Bell. These photographers conducted government surveys and photographed mostly characteristic landmarks. Klett et al. extended this project in 2004, revisiting the places already rephotographed in the seventies [63]. Two images of a compilation are displayed on one spread. The third image is displayed on the previous or following spread together with a description. All images are in black and white. The book is accompanied by a DVD, see Section 4.4. The recording of the third views is described in a travelogue [64]. Klett also (co)authored, among others works, publications on the San Francisco Earthquake of 1906 [65], the Grand Canyon [66], and the Yosemite National Park [67]. To the second category belongs also a book by M. Amundson from 2014, following the tracks of J. Stimson [68] and combining continuous text with rephotographs. Amudson published a book on the same topic in 1991 [69] and in the new book, the plates of Stimson are displayed together with Amundson’s old and new rephotographs. The newest rephotographs are in colour, all others in black and white. J. Sherow et al. followed the tracks of the Union Pacific Railway. The construction of the railway was documented by A. Gardner, whose photographs were rephotographed [70]. The book depicts the transformation away from native land use. B. Wolfe and S. Brady followed E. Muybridge, who, after being acquitted of murder of his wife’s lover, photographed the Central American Pacific Coast [71]. R. Webb followed R. Stanton, who made photographs of the Colorado River corridor every one to two miles in 1889 and 1890 in order to plan a railroad. Webb rephotographed the 455 plates and a selection of these photographs was published [72]. In Great Britain G. Diprose and J. Robbins rephotographed works of Victorian photographer H. Taunt, featuring scenes of the River Thames [73].

4.2.3. Series on Rephotography

Amberley Publishing publishes the Through Time series encompassing over 900 books by various authors of rephotography of British cities and places, starting from Aberdeen City Centre [74] to Yorkshire Windmills [75]. Pavilion Books publishes the Then and Now series with over 100 books combining rephotographs of world capitals like Beijing [76] or Sydney [77] and other tourist attractions like Hollywood [78] with texts about local history. Westcliff Publications publishes a Then and Now series with mostly the U.S. states as topic, e.g., Florida [79] or Texas [80]. Rephotographer M. Hesh self-publishes his Time After Time City Books, typical coffee table books containing rephotographs of US–American cities [81]. The German Sutton Verlag publishes the series Gestern & Heute with repeat photography books about German towns and villages from Anklam [82] to Worms [83] as well as books on niche local history, like train stations in the County of Hohenzollern [84]. The book about Worms is of special interest since it does not compare historic with contemporary photographs, but photographs taken before the destruction of the town in World War II (WWII) and directly after. Since 1973, the Battle of Britain International Limited publishes After The Battle, a quarterly magazine, displaying rephotographs of various battles of (WWII) [85]. From the same publisher are several books dealing with other battles, e.g., battles of (WWI) [86] or the Zulu War [87].

4.3. Online Media

Single persons or groups publish rephotographs on dedicated pages and on social media. Online news portals use rephotography extensively. However, online services for the management of user generated compilations or for the collection of meta information about the field of rephotography are scarce. They are presented in Section 5.

4.3.1. Project Pages

In this section we present exemplary online resources harnessing rephotography for a certain topic. They may be founded by single or groups of persons, charities, or government agencies. Projects, calling for audience participation, are presented in Section 5.

S. Larenkov publishes his own rephotographs on a blog [88]. Main subjects are images taken during WWII in Eastern Europe and Germany. He employs the technique of feathering where only parts of the old images are blended into the new image resulting in a new image in which parts are replaced with the old image. Similarly, K. Zoltán publishes rephotographs of Budapest on Flickr [89]. M. Klett shares rephotographs on his page [90] and together with B. Wolfe [91].

Museums employ rephotography on their websites as well. For example, the Museum of London uses rephotography to visualize the abandonment or concealment of London’s waterways [92].

High Life Scotland, a Scottish charity, hosts Ambaile, a website dedicated to the culture and history of the Scottish Highlands [93]. On this site a series of mostly urban rephotographs is published.

Other project pages exists as well, few use the term rephotography and not all are maintained over longer periods of time. This inherent ephemerality is naturally problematic for the preservation of the rephotographic heritage.

4.3.2. Social Media

Rephotography can be present in four ways on social media:

(1) Single compilations can be posted using existing social media services, and can spark large discussions on the content and implications of these compilations. The topics of these compilations are diverse: Compilations can deal with politics or can be used as propaganda: This is, e.g., the case for a compilation depicting a park scene in Afghanistan. The old image from the 1970s shows an intact park, the new image from the 2000s depicts the same park barren and the infrastructure destroyed [94]. Searching for images using terms like before US after US results in more compilations questioning the interventions of the USA in the Middle East. Social media is also subject to direct rephotographic manipulation (see Section 4.5.6). Persons and animals are the subjects of compilations, documenting diets, pregnancies, and adolescence. Old images taken from the family photographic album are rephotographed, resulting in humorous scenes of adults using toys like tricycles or trying to fit in cramped spaces, easily occupied in their childhood. Compilations on social media are also devoted to objects and processes, e.g., the detailing of a car and the growth and pruning of a tree.

(2) Rephotography groups can be established on existing social media sites. They may address a specific topic or encompass rephotography as a whole. They may allow for the contribution of templates, rephotographs, and compilations from their members and for discussions of the posted works. In particular, groups dealing with more specific topics are difficult to find, because rephotography or related terms are rarely used. On Flickr an example for a more general group is the Looking into the Past group, collecting compilations of urban and landscape scenes from mostly Europe and North America [95]. Users experiment with different presentation techniques, but are limited in that they can upload only one non animated image per compilation. An example for a more specific group is the Vancouver “Then and Now” group comprising around 2800 compilations of Vancouver [96]. On Facebook an example for a more general group is the Rephotography group collecting compilations from all over the world and presenting other rephotography projects [97]. Examples for more specific groups are the Sydney Then and Now group [98] and the In the footsteps of Kurt Hielscher group, rephotographing the works of the German photographer Kurt Hielscher [99]. On reddit, e.g., the subreddit Old photos in real life [100] addresses traditional rephotography. For rephotography of humans another subreddit exists [101].

(3) Users or agencies can also post rephotography albums on their own social media page. They post, for example, compilations of Aleppo before and after the Syrian civil war posted by a Syrian restaurant on Facebook [102] and rephotographs of iconic movie scenes [103].

(4) Lastly, special social media sites or groups on rephotography can be created. They may be addressing a certain topic, give templates to which users can add rephotographs, and have a comment and discussion section. One example would be the SNS Challenger project, A Social Network site following the research voyage of the HMS Challenger from 1872–1876 [104]. On this site templates taken during the voyage are uploaded to which users can upload rephotographs resulting in a second virtual voyage. Additionally, the site offers a blog, commentary section, and personal pages. Observation and results were published as a paper [105].

Hashtags for rephotography are, e.g., #thenandnow, #beforeandafter, or #transformation. Under the hashtag #10yearschallenge users of Facebook and Instagram posted compilations of themselves with a ten year distance between the old and the new image. The later sparked a discussion, whether these facial rephotographs are used by social media companies to create more age robust face recognition software [106].

4.3.3. Online News Portals

Newspapers, magazines, and television channels use rephotography on their websites. Often a series of compilations, addressing a certain topic, together with some background information is published. Rephotography may also stand on its own as photo series or is employed together with other media to enhance a written article.

Images taken shortly before and after a natural disaster are used to visualize damage. The Los Angeles Times uses animated compilations of satellite images to visualize the land movement caused by the 2019 Ridgecrest earthquakes [107]. The BBC uses rephotography to visualize the destructive power of war, showing compilations of Aleppo on their website [108]. Similar compilations are again published during the 2022 Russian invasion of Ukraine [109]. Compilations feature locations in the distribution area of local newspapers: The New York Times with a series featuring New York [110], the Berliner Morgenpost and the rbb with series featuring Berlin [111,112], and the Westdeutsche Allgemeine Zeitung with a series featuring Essen [113]. Compilations feature also other locations, e.g., the Frankfurter Allgemeine with a series featuring New York [114]. Even prominent figures, e.g., boxers [115] or celebrities [116] are targets for rephotography. Some publishes host overview pages dedicated to their rephotography publications, e.g., The Guardian [117].

Overall, rephotography is used for a wide range of topics, including war, disasters, and (regional) history, in online news portal but is seldom called rephotography. It is used alongside with videos, photo series, and other media.

4.4. Other Media

Rephotography is used in video, film, computer games, children’s books, and as in place rephotography at the location of the historic image.

4.4.1. In Place Rephotography

Passersby can be engaged by providing a historic image in place of its correct camera position. Passersby can also be tasked to take contemporary images as discussed in Section 7.1. Duplicates of historic photographs or other artworks can be placed at the position of the artist or camera, allowing direct comparison of the historic image with the live contemporary scene. An example of this technique is shown in Figure 2. Embedding the historic artwork in the contemporary scene gives context and extends the view. It helps to understand the motives and emotions of the artist and allows a mutual state of mind with the artist. Normally, passersby would not know that an artwork was created from their vantage point, even if it is a known masterpiece. Placing the artwork into its scene allows to make this connection between historic image and their current position and view.

Devices similar to tourist telescopes can be used as well. Here, a scope is positioned at the old camera position pointing into the old camera direction. A transparent print of the old image is inserted into the scope at the correct scale allowing a viewer to see the old image superimposed to the live scene through the scope. An example of this method is shown in Figure 3. A glass plate can be used as well: On it the reconstructed wire frame model of a ruined building can be printed. Standing at the correct position in front of the glass, allows passersby to view the model, correctly positioned over the ruin [118].

4.4.2. Rephotography Collections

Only few physical collections of rephotographs exist. The The United States Geological Survey (USGS) Southwest Repeat Photography Collection [119], formerly named Desert Laboratory Repeat Photography Collection, and the archive of the Santa Rita Experimental Range [120] are example collections limited to specific areas and research interests. Encompassing rephotography collections are difficult to create, since the old images, when they were not taken for a rephotographic project, are often already part of another collection. For digital online collections see Section 5.

4.4.3. Interactive Media

Interactive DVDs are another form to present rephotography. The book [63], presenting the compilations of the Third View project, is accompanied by a DVD. The DVD contains the compilations from the book, additional written information and maps, allowing users to select rephotography sides and to follow the journey made by Klett et al.

In the London Street Museum project of the Museum of London rephotography was used to augment reality. Users were able to download a mobile app, highlighting different locations of London on a map. Visiting these locations in reality, allowed for the overlay of a registered historic photograph over the live camera preview of the smartphone [121]. The functionally and impact was presented in a video [122].

ReFotografie is a mobile app collecting compilations of Łódź, the third-largest city in Poland. The app offers a map view, exact address of the rephotographed buildings and descriptions. Compilations are well registered and displayed as animation [123].

Mobile apps aiding the creation of rephotographs are presented in Section 7.3.2.

4.4.4. Video

Documentaries and news can use rephotography as well. G. Divossen, for example, uses rephotography in his documentary, published on DVD, of the German town Beuel [124] and NBC News featured a video of compilations of satellite images showing the destruction caused by Hurricane Dorian in the Bahamas [125].

Documentations on rephotography exist, too: BBC Scotland produced an episode of the television series Landward accompanying landscape photographer C. Prior, while rephotographing in Scotland [126] and another German TV report presented rephotography in general [127].

4.4.5. Revideography

Repeated recording is not limited to still images, but can be extended to animated still images and even videos. A simple animation is alpha blending (see Section 7.7), in more complex animations parts of the new still image are successively blended into the old still image or vice versa. Even more complex is the blending of a new moving image with an old still image or vice versa. Digital effects, like zoom, can be added to revideographic compilations as well. A further possibility is to revideograph a historic video, carefully replicating the movement of the camera and of objects in the scene. These two videos can then be played simultaneously next to each other. Almost all variations of revideography are applied by S. Brajtera [128].

4.4.6. Renderings and Animations

Renderings can be used instead of the historic or contemporary photograph. NeoMam Studios used historic photographs of metropoleis and combined them with both contemporary photographs and renderings depicting how those cities could be shaped in the future [129]. Renderings can also be used to animate, e.g., an imaginary reconstruction process of historic sides starting with a contemporary image on which first a drawing is superimposed followed by an animation of the reconstruction of the site [130,131].

4.4.7. Video Games

Rephotographs can be used in video games with a similar motive as in the real world, providing a link to the past but also to provide background to the story. In the game Horizon Zero Dawn players can search for and visited vantage points where the gaming environment is overlaid with a hologram depicting the same scene before an extinction event [132].

4.4.8. Children’s Books

The essential idea of rephotography manifests also in children’s books. Although, they do seldom use photographs but illustrations, the idea to make changes over time visible is apparent in the following children’s books: Outstanding is a book by Noon and Millard containing 14 illustrations showing the same street from the Stone Age over the Roman and Middle Ages to the Presence [133]. Through these images, the development of, e.g., architecture, agriculture, or industrialisation becomes graspable. Rephotography of the broadest definition is used in children’s books to illustrate the history of agriculture [134,135] and in a narrower definition to illustrate the development of a street and a house in the last 100 years [136], as well as of children’s everyday life from 100 years ago till today [137].

4.5. Related Work

4.5.1. Face Movies

I. Kemelmacher–Shlizerman et al. developed software to register a series of portrait photographs of the same person and combine these into an animation [138]. These face movies visualize the development of a person from infancy to adulthood. Notable is the use of more than two images and the completely automated registration process for faces.

4.5.2. Street View

In Google Street View users select different recording times of the displayed street view scene with a slider. Depending on location several points in time can be selected and compared [139]. Although, the images are not perfectly registered and sometimes of bad quality, they offer a time series of the same scene. Images date only back to 2007 and are limited to legally and infrastructurally accessible areas. In the covered areas however, they offer a straightforward way of rephotography.

4.5.3. Satellite Data

Related to rephotography are Satellite Image Time Series (SITS), compilations of satellite data of the same location from two or more points in time. Compared to rephotography, satellite data for the last decades is abundant. Moreover, satellite data is georeferenced and thus images of the same location from different points in time are already registered.

SITS can be used for precision agriculture, crop yield prediction, or disaster relief. The U.S. National Geodetic Survey (NGS) for example publishes emergency response imagery after natural disasters consisting of satellite images from before, during and after the disaster [140], allowing damage analysis and planning of disaster relief but also later scientific analysis of the disaster. In this case satellite data from each point in time is displayed as separate layer, whose visibility can be toggled, on a map. A more traditional publication is for example a website by the National Aeronautics and Space Administration (NASA) consisting of compilations of satellite images illustrating glacial decline [141]. The Earthshots project by the USGS employs SITS to monitor environmental changes [142]. Similar to rephotography websites on the site of this project a map is displayed, with markers at positions of SITS. Each time series is accompanied with a description of the visible changes and their causes.

Summarized, the field of SITS does not share the same challenges and processes with rephotography but can have similar applications and forms of visualization. In the standard definitions rephotography is often limited to ground-based cameras.

4.5.4. Medical Photography

In medicine procedures similar to rephotography are used. Multiple images are taken to monitor healing processes after the transplantation of tissue [143], nutritional intake of nursing home residents [144], and success of orthodontic treatment [145].

However, these procedures are normally not considered rephotography. Reasons might be the distance between landscape change research, the origin field of rephotography, and medicine, shorter time periods in medical photography, and simply different content and scenes, resulting in different challenges and solutions: In medical photography the environment is controlled and often consists only of the foreground object and a neutral background. The position of the sensor is fixed or well known, simplifying the capturing of the rephotograph.

4.5.5. Object Rephotography

In this branch of rephotography an object is photographed repeatedly in different environments. Z. Kerényi uses a photo camera as reoccurring object. The old image is a close-up of a girl using the camera. This old image is superimposed over the window display of an antique camera shop such that the camera is partially visible on display for sale and partially used by the girl [146].

4.5.6. Rephotographic Manipulation

Rephotographic compilations are manipulated by not disclosing which image was taken first. Two images of the Crimean War, depicting a road in a valley with cannon balls covering the ground in one image but not in the other, spark discussions on the order of the two images. Either the photographer placed the cannonballs there for a dramatic effect or they were removed to be fired again [147].

The old image can be manipulated into a rephotograph, never recorded in reality. One example compilation shows in the first image a battalion of soldiers before (WWI) in front of Edinburgh Castle [148] and in the second image the surviving soldiers after the war in the same location. The second image is likely a manipulated version of the first, best visible on the exact same treeline, which should have changed in the five years, allegedly elapsed between the two images [149,150].

4.6. Conclusions

In this chapter, we followed the rise of rephotography from first occurrences in art during the 19th century to the onset of its scientific use in glaciology and landscape change research at the turn of the century. From there, rephotography grew to other natural sciences and other sciences, education, art, and culture. This resulted not only in scientific publications but also in rephotographic series, exhibitions, and illustrated books. In the 20th century, digital rephotography in social and other media is no longer limited to a specific applications or publication form. It is used in videos, videogames, animations, and news, depicting landscapes, urban scenes, humans, and animals. The goal of rephotography, however, remains unchanged. It is used to visualize changes in its subject and to connect to the past.

5. Management Software and Online Resources

Online Services, addressing rephotography, can be divided into two overlapping categories:

1.

Services for the management of rephotographs themselves.

2.

Services which collect information about books, persons, etc. related to rephotography.

The re.photos project [151,152,153] is the most notable member of the category, offering online methods for interactive creation, browsing, searching, locating, rating, sharing, and discussing of rephotographs. Users can add old images, for which other users or they themselves can record a new image and publish them as a compilation. An interactive registration method allows control over the registration process and with a map based browsing mode users can select compilations and templates on a map. re.photos currently hosts over 2900 rephotographs and has over 2300 users.

Geneanet combines an online collection and a mobile app for the recording of rephotographs [154]. The collection is browsable via a map, on which users can select and view compilations. The old images are mainly postcards, which are rephotographed via the app. The rePhoto [155] project combines a homepage and an app. Compilations can be organized into projects with a dedicated project site, featuring a specific area and topic, e.g., animals or trees. The site seems to be dormant. Timera was another web project to collect historic photos and create rephotographs from them. An archived copy is accessible [156]. Here, parts of the old images are feathered into the new images. Users were provided with a mobile app and were able to upload old images, photograph new images, and create compilations from them. Timera was succeeded by The Vintage News a popular history site, which published articles illustrated with compilations but declined into a news aggregator [157] no longer offering any functionality related to rephotography. F. Branco et al. proposed another online tool for rephotography [158] aiming for automatic feature based registration of compilations consisting of two or more images. They claim that their tool, Evolapse, enables feature based registration and visualization and use it to create a timeline feature for the online project Lx Conventos, examining the development of religious orders and their buildings in Lisbon [159]. The timeline feature allows to select registered images and view an overlay of them. However, the feature seems to be no longer available.

Photo tourism [160] is an online project and aims to create a browsable 3D model of a scene using contemporary images only. Keypoint based image registration is used to locate the images in 3D space and users can navigate this space by moving and zooming. Doing so results in the selection, transformation, and display of the image, whose real camera position is nearest to the virtual camera position in the 3D model. The project could be extended from contemporary images to include also historic images. However, for these images automatic registration is not possible. Photo tourism resulted in the Photosynth app, allowing users to create 3D models from photographs. It was discontinued in 2017 but ported to Microsoft Pix [161].

The PhotoCity game [162], discontinued in 2011, aimed to collect images from their users. These images were used for 3D modelling, especially for the Photo tourism project. Teams of users competed in capturing enough photographs for the successful creation of a 3D model of a certain building. Reward was given in form of markers, displayed on a map indicating locations and completed buildings by team colour. Gamification could also be applied to standard rephotography to animate users to record rephotographs, although, in this case not the quantity and variance in perspective is relevant but exact reproduction of the old image. Similarly, the HistStadt4D project aims to map historic images on a 3D model of a city, visualizing the result in a browser based mobile VR allowing to move through the scene and time, the fourth dimension, from one image to the next [163,164].

The rePhotoSA project is dedicated to the landscapes of South Africa, displaying rephotographs of this landscape and examining landscape change [165]. The site provides a map interface allowing to view compilations or templates of a select area. The side calls for participation by providing old photographs and providing rephotographs. Rephotographs from the Black Hills are collected under the Where in the Hills project of the Black Hills State University [166]. The site hosts an archive of around 2000 historic photographs and asks for participation by rephotographing these old images. A provided kml file, which can be processed with virtual globe applications like Google Earth, overlays images from the project over the data from the virtual globe application. The Alpine of the Americas Project aimed to understand and record landscape changes in alpine environments in North America with rephotography. The site is no longer available. Their Facebook page [167], an instruction video on how to take rephotographs [168], and a video of the rephotographic journey of the project members [169] remains.

To the second category belongs the project RePhotography: A Survey of Projects Rethinking Photographs [170], listing rephotography projects and creators. Entries date from 2012 till 2015. This project is accessible via the Internet Archive. The Forest History Society maintains a mixed category page providing information, a bibliography, and rephotographs related to the forests of the United States [171]. BBC Scotland maintained a—now archived—rephotography website, too [172]. They provided templates and tips how to take rephotographs. Compilations were combined with location and background information. A similar mixed category page is maintained by the USGS, addressing climate change research and glacial decline in Glacier National Park, MT [173]. A second page by the USGS calls for volunteer rephotographers to rephotograph old images of glaciers in Glacier National Park and provides templates and information where and how to rephotograph them [174]. H. Basgic organizes The Glacier RePhoto Project with a database of around 280 rephotography locations of glaciers in the American West and over 1300 images taken at these stations [175]. The Knight Lab offers JuxtaposeJS, a tool for the presentation of compilations [176]. Users can provide links to an old and new image, from which a slider presentation (see Section 7.7) is generated, which in turn can be embedded into other websites.

The discontinued projects above serve as examples of a general problem of online rephotography: Project pages vanish over time, hyperlinks break, and images are no longer available. Sometimes data can be recovered from the Internet Archive or other sources. However, this is not a future-oriented solution. Cooperation with other projects aiming to preserve our digital heritage is necessary for the field of rephotography.

6. Challenges in Rephotography

Rephotography and image registration of the old and the new image is challenging for humans and computers. The challenges are presented here, solutions in the next chapters. While we focus on the registration process, A. Pupo–Correia et al. focus on methodology and camera position [177].

6.1. Methodological Challenges

Old images are spatially biased: They are taken from easily accessible places, like the side of the road, and feature pleasing or interesting scenes. Old images, not specifically created for a rephotographic study, do not represent the subject of the study objectively. They over represent human influence, when they were taken from easily accessible places, and under represent the normal, when the artist focused on irregularities. This is especially problematic for ecological research and when human impact is measured.

6.2. Missing Location Data

Often metadata or unique landmarks are missing in the old image, making it difficult or impossible to locate the old image on a continental or regional scale. Finding the correct camera position and orientation with submeter accuracy is another problem (see below).

6.3. Missing Time Data

If strong shadows or celestial bodies are present in the old images, they can be replicated in the new image. For shadows the correct time of day can be sufficient. Exactly the same shadows are only cast at a pair of two days in the year symmetrically distanced from the solstices. Rephotographs with perfect shadows are only possible at the correct date or the other day of that pair. Foliage and other landscape features may be highly different in the pair, leaving only one day in a year to take the rephotograph. To rephotograph images featuring celestial bodies like the moon the correct year may have to be determined as well (comp. with Section 4.1.3 A. Adams).

6.4. Missing and Masked Objects

The following points are challenges for image registration. Objects may be no longer visible in one of the images. They may be missing entirely in one of the images (fallen trees, demolished houses, passing vehicles) or are occluded by other objects (grown trees, newly erected house in front of the object). These objects can no longer be used to register the images. If the invisible object was the main subject, the compilation becomes unusable, except the rephotograph was taken to show exactly that the object is now missing.

6.5. Changed Objects

Objects may change significantly by nature (forestation, erosion), by season (snow coverage, foliage), date and time (illumination, shadow length and direction), weather (illumination, shadows), or by humans (buildings are painted, get additional floors, get a new type of roof).

6.6. Irrelevant Objects

Especially automatic registration algorithms suffer from irrelevant objects, which cannot be registered. The sky is the most prominent object of this category: The clouds in the old and new image cannot be registered meaningfully. Other examples are water and waves, changing foliage, and generally non fixed objects. This challenge is aggravated by its subjectivity: In one compilation the cars in front of a house might be irrelevant and disruptive, in another compilation, a car might be the main subject. Determining the (ir)relevance of an object is easy for the rephotographer—knowing her or his intentions—but likely impossible for an automatic registration algorithm.

6.7. Changed Recording Methods

Although, humans can cope well with different recording methods when viewing images, registration is heavily influenced by large differences in recording methods. While the new images are often high quality digital colour images, old images can be damaged, noisy, blurry, and monochrome. Old images can be paintings, drawings, or prints, like postcards, resulting in more irrelevant differences between the old and new image.

6.8. Unknown Camera Parameters

Misgauging the focal length of the lens with which an old image was recorded can lead to the selection of a wrong camera position, yielding in notable parallax (see Section 6.10) and a different size relation of foreground and background objects between the old and new image. This allows only for a partial registration. Other camera parameters, like lens distortions, have a lesser influence on registration quality and are ignored in practice. While camera parameters of the new image are widely available for lens corrections, camera parameters for the old image are almost always unknown and could only be estimated.

6.9. Camera Position

One of the main challenges of rephotography is the original camera position: (1) The original location is no longer accessible, e.g., the building from which the old image was taken is demolished. (2) Reaching the original camera position is expensive because it is remote, distant, and blocked, e.g., by traffic. (3) The view from the original camera position is blocked, e.g., by trees or new buildings. (4) The original position cannot be found precisely, e.g., due to time constraints, wrong technique, human error, and not having the old image on hand. When the two camera positions do not match, image registration can become difficult, as shown in Figure 4.

6.10. Parallax

Different camera positions for the old and new image result in parallax, a different overlap of foreground and background objects in the two images as shown in Figure 5. This is especially problematic if the distance between foreground objects to background objects is large compared to the distance between camera and foreground. If the distance between the two camera positions is large, image registration is only partially possible: Rigid registration of images taken from different positions allows only one plane in the old image to be registered perfectly to one plane in the new image. All objects on this plane are registered, all other objects are not; with the quality of the registration decreasing with the distance from the plane. For example, only the front but not the sides of a house can be registered as shown in Figure 4.

7. Techniques

For scenes and objects that are place-based, i.e., locatable on a map, the following steps have to be completed to create a rephotograph:

(1) Old images of the area, scene, or photographer of interest are found. Ideally, together with metadata, containing place and date information. Alternatively, the old image has to be recorded and its camera position marked.

(2) The location of each old image is found on a grander scale. This can be done with landmarks, metadata, typical vegetation, etc.

(3) If necessary, the date and time of the recording of the old image is determined. Ideally, the new image should be recorded at a similar date of the year and at similar time of the day such that shadows, vegetation, snow coverage, etc. are similar in both images.

(4) The estimated location is visited at the estimated date and time and the exact vantage point is searched for.

(5) The new image is taken from the same point, with the same view direction, and preferable with a lens of similar focal length.

(6) Back in the lab, exact image registration can be done in the darkroom or digitally on a computer. In most cases only this step allows for detailed comparison and aesthetically pleasing results, but this step is not always carried out.

(7) Compilations can be analyzed and be prepared for printed or digital display choosing an applicable presentation method.

For unlocatable scenes and objects, e.g., portrait photographs, the procedure is different. Here, the correct person or object has to be found. Then the recording conditions of the old image, e.g., lighting, background, camera parameters and position, are determined and can be replicated if desired. The new image is taken followed by step 6 and 7 from above. If the goal is not, or cannot be, a perfectly registered rephotograph, steps can of course be skipped. In the following, we present solutions to the challenges from Section 6, following the steps from above.

7.1. Sources and Types of Old and New Images

Both old and new image are not necessarily photographs but can be other visual art forms. They can be retrieved from many sources:

For ground level photographs private or public collections can be used, which may address a certain topic or photographer. Sometimes the complete or large parts of the collection are rephotographed and the compilations published (compare with Section 4.2). The FSA collection is one example collection for which rephotographs were created [44]. It encompasses photographs from the 1930s and 1940s focusing on resettlement, migratory labourers, and later patriotism as well as war production in the United States. Over 175,000 of these images are available from the Library of Congress [178], most of them are not copyright protected.

If one intends to go back further back in time, photographs no longer exist. Then, other visual art forms, for example paintings or drawings can be retrieved from archives, e.g., for watercolours from The Watercolour World [179], a collection of (roughly) georeferenced documentary watercolours painted before 1900. Postcards can substitute photographs, too, and can be retrieved from various archives like the Newberry Collection [180] or the Bildpostkartensammlung der Universität Osnabrück (photo postcard collections of the Osnabrueck University) [181].

Historic maps together with contemporary maps or satellite or aerial imagery can be used for rephotography as well. Historic maps can be retrieved, e.g., from the David Rumsey Map Collection [182]. An overview of online historical map collections is compiled by the University of Texas Libraries [183]. An example application of historic maps and postcards would be the work of P. Bannon et al., who examines the use of historic photo postcards for coastal geomorphological analysis [184]. They state that the quality of historic maps can be assessed with these photographs and that a compilation of several old and new images allows for a better dating of coastal changes compared to the long time periods between mapping surveys.

Aerial imagery, taken from airplanes or today by unmanned aerial vehicle (UAV), without combination with historic maps can be used for rephotography to monitor, e.g., glacial calving [185] and archaeological sites [186].

In Photopointmonitoring (PPM) the old image is yet to be taken. Hence, the camera position and orientation is known and can be marked both in the real world as well as in a data form or journal. Rephotographs from the marked position can then be taken by scientists and the general public, e.g., hikers. F. Hall wrote a practical handbook on PPM, also applicable to rephotography in general [187,188]. PPM is mainly used in ecological research but can also be used to monitor the condition of historic sites [189].

Citizen scientists, tourists, and passersby cannot only help at clearly marked photopoints but also photograph animal sightings (see Section 4.1.2) or send in photographs of a certain scene. Although, the later might not result in a collection of perfect rephotographs they still allow to examine changes in the scene over time.

7.2. Georeferencing

The first problem to overcome after an old image is acquired is to georeference the image, i.e., to determine the geographic coordinates of the camera position from which the image was taken.

Titles and metadata, written, e.g., below or on the back of the photograph as well as landmarks in the photograph and their alignment can be used together with local knowledge of the rephotographer or locals. Comparison with other photographs, maps, or field records is also an option. This step becomes difficult if metadata or unique landmarks are missing.

If the old image is recent and of suitable quality, the coordinates can be estimated algorithmically. Neural networks can be trained to output global location probabilities of a given input image [190,191]. For input images for which similar images exist in the training data, i.e., input images of regularly photographed scenes, the error can be in the range of few meters. However, for input images of seldom or never photographed scenes the error can be large. These networks were trained with recent digital images only, hence, the estimation error is often large for historic photographs and especially for historic postcards and other visual art forms.

If a large number of images of the area in which the old image was taken exists, not only the camera position can be estimated but also the orientation. However, the neural network needs to be trained specifically for this area [192]. This was for example done for the centre of Cambridge, UK.

T. Hanks et al. describe the process of virtual repeat photography (VRP), in which the camera position is searched in a Digital Elevation Model (DEM). Views of the DEM can be compared with the old image until the approximate vantage point is found [193].

If the old images were or are taken with the intend to rephotograph them at a later point in time, markers and discernible foreground objects can be employed. This process was already described and applied, e.g., by K. Gilbert for glacier rephotography by building cairns and relating the moving ice to these fixed foreground objects [194]. H. Malde argues that camera locations should be marked with steel rods, cameras should be level and aimed to a conspicuous object, and that meta information should be stored with the photograph, making it a easily rephotographable geologic benchmark [10]. Nowadays, coordinates can be directly recorded with Global Positioning System (GPS).

7.3. Camera Relocalization

After the preliminary work is done, the exact camera position and angle have to be found in field. This camera relocalization can be done either with traditional methods or with a laptop or smartphone.

7.3.1. Analogue Camera Relocalization

M. Klett describes the process of comparing measurements, taken on the old image, with measurements, taken on instant pictures, to verify the correct vantage point. Further, he describes, how these methods can be transferred into the digital age by taking laptops, battery operated printers, and digital cameras into the field [62,195].

H. Malde developed a parallax based method in which middleground and background objects are aligned on the centre line. The camera is moved along this line until peripheral objects are aligned. The camera height is then found with the help of foreground objects [10].

A. Harrison argues that Malde’s method requires the presence of discernible objects on the central axis and presents a method in which a transparent plastic overlay, on which edges of landmarks are printed, is held in front of the eye of the rephotographer while the vantage point is searched [196]. He further presents a method to verify the accuracy of the vantage point based on the ratio of lengths between landscape points.

7.3.2. Digital Camera Relocalization and Mobile Apps

Several mobile apps and computer applications for the relocalization of the camera position and orientation of the old image exist. Functionality and complexity of these apps differ. Positioning apps can help with finding a known camera position using GPS whereas pose apps guide users to the position and orientation of the camera of the old image using correspondences between life camera preview and old image. Often, an app combines both functionalities. For the complete automation of pose relocalization, the corresponding feature points between the two images have to be detected automatically, i.e., for these mobile apps, the automatic image registration problem described in Section 7.5 has to be solved first. All current apps, which rely on corresponding points between the old image and the camera preview, require that the users marks these corresponding points by hand.

Simple pose apps overlay the life preview of the smartphone camera with the old image. Users can then move the smartphone until the old image and camera preview register. With small displays and under harsh sunlight, registration is difficult to archive with this simple alpha blending. Using a laptop or tablet with larger screen improves the situation with the cost of bulkier equipment. Using augmented reality (glasses) is also an option, yet to be investigated.

More complex apps overlay features, like edges or corners, generated from the old image and the life preview. Features are simpler to register but do not work well for all image pairs. Straight lines, for example, are suitable features for architectural scenes but are insufficient for most natural scenes. Moreover, complex apps, targeted at landscape and cityscape rephotography, feature a map or show old images near the current location of the user.

The above apps use the smartphone camera to record the new image. This can be disadvantageous since the image quality and resolution can be low. The focal length is in most cases fixed and cannot be matched to the focal length of the old image. Camera parameters like aperture to match the depth of field (DOF) can often not be adapted. S. Bae et al. combined a laptop with a digital single lens reflex camera (DSLR camera) [197]. Once corresponding points between the DSLR camera live preview shown on the laptop screen and the old images are determined by the user, the laptop computes the transformation required to move from the current pose to the pose of the old image. The result is displayed as arrows on the laptops display, indicating the required direction of movement to register camera preview and old image. Albeit, this was done due to lack of computing powers on smartphones, combining a laptop—or currently, a smartphone—with a DSLR camera can solve the problems linked to the smartphone camera.

Bae et al. tested different guiding methods and found that a blend of old and new image or relative pose information in 3D was hard to interpret by users. However, users were able to follow two 2D arrows, one pointing on the horizontal plane and the other on the plane perpendicular to the optical axis.

The app by Y. Shi et al. does not require the manual selection of correspondences but uses a combination of feature and flow based matching to compute the relation between old image and a set of key frames as well as the current frame of the camera preview [198]. However, they only tested robustness against illumination change. Old and new image where taken only days apart, limiting the changes in the scene. Two navigational rectangles, which have to be registered, are used to guide users to the pose of the old image.

Related is the task of active camera relocalization. It combines the task of finding the correct camera pose and the task of visually manoeuvring a robot operated camera to this position. The task was first studied theoretical [199] and later on a practical level for archaeological rephotography to monitor changes in ancient murals, for which the original camera pose was recovered from one old image [200]. Here, a camera mounted on a motion platform is relocated precisely to the camera position of a previous image making computational registration of the resulting image unnecessary and allowing to detect changes in the murals on a 0.1 mm level. Changes in the scene were minimal and quality of the old image high.

7.4. Recording the Rephotograph

After the camera position is determined, the camera parameters and the optimal recording time have to be determined:

7.4.1. Focal Length

Counterintuitively, selecting the correct focal length, which corresponds to the field of view, is not that important. It is important that the same or a bigger part of the scene relative to the historic image is recorded, but the correct size of the contemporary image can be archived by scaling and cropping. Focal length, contrary to camera movements, does not affect the visibility relations of objects like parallax. This is visualized in Figure 5. However, upscaling and cropping decreases the image quality.

7.4.2. Lighting and Timing

Lighting and especially shadowing is important to make the images comparable. Natural lighting depends not only on the weather and the time of day, but also the time of the year. At two points in the year the lighting can be replicated [195]. However, the landscape may differ at these two points in time. Thus, the contemporary image should best be recorded at the same time of day and the same time of year as the historic image. This is especially important if one is to measure variables depending on the time of year—examining changes in glacier or snowfield size between images taken in summer or winter does not allow for prediction on long-term behavior.

7.4.3. High-Dynamic-Range Imaging

K. Gilbert suggested in 1904 to take two photographs with different exposure from the same camera position to record the complete dynamic range of the scene, especially addressing glacier rephotography, in which details in white ice and snow as well as in dark rocks are relevant [194]. This could now be archived with high-dynamic-range imaging (HDRI) photography if the range of today’s digital sensors is not sufficient.

7.5. Image Registration

While analogue image registration is possible in the darkroom (cropping, scaling, and to a very limited extend perspective transformations), nowadays, it is best done digitally. Digital image registration is one of the main fields of Computer Vision and by no means limited to rephotography [201,202,203]. The registration methods can be organized by different parameters (grade of automation, basis (intensity, statistics, feature), and transformation method). In the following, we discuss how a variation of these parameters influences the suitability of the registration method for rephotography and present methods specifically developed with rephotography in mind.

7.5.1. Automation

The automation grade of the registration process varies from manual registration to completely automatic registration.

Manual registration of compilations can be done with a standard raster graphic editor. The old and new image are imported as separate layers of a single image. The opacity of the top layer is reduced, such that both images are visible. The rephotographer then uses transformation tools like translation, scaling, or perspective transformation to register both layers. However, unspecialized tool makes the process tedious and unintuitive.

In interactive or semiautomatic registration specialized tools are used. While it requires user interaction, it allows to register also difficult rephotographic compilations. Users can control which areas are to be registered, facilitating the registration of images taken from different vantage points. Another approach to interactive registration is the use of automatic registration algorithms applied on masked images. Users mark areas and only these areas are regarded by the registration algorithm. This may only work for images in which the marked areas are similar. Interactive registration is a feasible compromise between manual registration and automatic registration.

Automatic Registration requires no further input aside from the two images. Automatic registration works well for many tasks with similar images taken shortly after each other from the same position and with the same camera, e.g., panorama stitching [204], HDRI [205], or focus stacking [206]. The challenges in rephotography (Section 6) hinder all standard automatic registration algorithms. To the best of our knowledge, no method for the automatic registration of rephotographic compilations exists.

7.5.2. Basis

Registration can be directly based on image intensity, on image statistics, and on classical or deep features.

Image intensity/pixel based direct methods compute an error between pixel values of the two images and select the transformation for which the error is lowest. Different error functions exist, all of them fail for rephotography, since almost all old and new images are highly dissimilar on a pixel level.

Image statistic based methods compute measures like correlation or mutual information on all or a subset of the pixels [207,208]. This approach seems promising for rephotography with similar images in an interactive setting, in which users select the relevant areas to be registered. To the best of our knowledge no study on statistic based methods for rephotography exist.

Feature based methods extract features from both images, try to find corresponding features, and compute a transformation, which minimizes the location error between corresponding feature points. Most of these methods fail for rephotography because of the large differences between the images, because they are not designed for historic photographs or other forms of art, and because features are extracted and matched in irrelevant areas like the sky.

7.5.3. Transformation Method

The allowed transformations used for registration can vary from simple affine transformations, over projective transformations, to more complex non-rigid transformations. While affine transformations preserve lines and parallelism, projective transformations only preserve lines. More complex transformations may deform lines and act locally on the image. Simpler transformation methods are computationally fast and can be predictable—an advantage for interactive registration. Especially for urban rephotography, the preservation of straight lines is an additional advantage of the simpler methods.

Complex methods on the other hand, allow for the registration of difficult compilations but are slower, counterintuitive, and can introduce artefacts. Since complex methods deform the image locally, they can register several objects. This can also be a disadvantage. A declining glacier, for example, can be registered together with the surrounding mountains, giving the impression that the glacier kept its size. Typically, for rephotography registration methods of varying complexity are used, starting from simple translations, over rotation and scaling to projective transformations.

7.5.4. Specialized Rephotography Registration Methods

In the interactive registration tool of the re.photos project users are able to select point pairs consisting of a point in each image. From these pairs a transformation matrix is computed registering the two images [152]. Usability is increased with a magnifier for the area around the mouse cursor and a snap to corners functionality, in which points are snapped to object corners present in the image. Interactivity is increased by directly computing a new transformation when a point pair is selected. Another tool allows for the alternating moving and stapling of the two overlaying images: The top image is moved with the mouse and then stapled to the bottom image at the current mouse position. The process is then completed for more and more complex rigid transformation modes [209]. The tool can be extended by replacing one or both images with edge images containing only the prominent edges of the images.

As affine as possible (aaap) is a feature based method for the semiautomatic nonrigid image registration [210], well suitable for cityscapes. It aims to keep straight lines, like building edges, straight, but requires user generated pairs of corresponding lines and can introduce artefacts at crossing or neighbouring lines. It works best if the input images are already roughly registered via a less complex method.

For specific areas of rephotography specialized registration algorithms exist: I. Cléry et al. presented a method to register historic aerial images with a topographic database using line features [211]. I. Aicardi et al. register multitemporal image blocks taken by an unmanned aerial vehicle (UAV) without standardground control points (GCPs) but with image intrinsic information of the unchanged area [212]. D. Hauagge et al. presented a method for architectural scenes based on local symmetries [213]. Although they use their method for image matching, application to image registration is possible. Special combinations of feature detectors, descriptors, and filters exist as well and yield better results for rephotography [214]. These are made-to-measure for specific scene types.

At the moment interactive, point-based registration methods applying a projective transformation seem to yield the best results for general rephotography. This may change in the future with deep learning based methods, e.g., D2-net [215], DISK [216], pixloc [217], R2D2 [218], and superglue [219].

7.6. Postprocessing

Registered compilations cannot only been directly viewed and analyzed by humans, but automatic change detection or photogrammetric methods can be applied as well.

7.6.1. Photogrammetry

Analogue [220] or digital [221] photogrammetry, i.e., the process of taking accurate measurements of physical objects and the environment from photographs, can be applied to compilations.

H. Malde describes how to measure the exact focal length of an analogue camera, how fiducial marks can be imprinted on film, and how these can be used to make distance measurements from the analogue negative [10]. He distinguishes between graphical and analytical methods and covers also analog stereophotography and grammetry.

Compilations consisting of multitemporal UAV images are used for photogrammetry as well. F. Rinaudo et al. create orthophoto maps from UAV images taken along a planned flight over an archaeological site [186]. They argue that the process is faster and more automated compared to traditional monitoring methods. J. Ryan et al. used stereophotogrammetry to extract a DEM from UAV images of the calving outlet of Store Glacier, Greenland [185]. Feature tracking across repeated flights allowed to measure ice flow speed and resistive forces in the glacier.

Colour photogrammetry techniques can be applied to compilations as well. O. Sonnentag et al. examined the applicability of rephotography for phenological research [222]. They focused on colour space transformations, especially excess green and green chromatic coordinate, of multitemporal imagery to track canopy development.

7.6.2. Automatic Change Detection

Detecting changes in the rephotographed scene is often the motivation of taking the new image. Hence, automatic change detection is a vital part of post processing. An overview of change detection methods used for ecosystem monitoring is presented by P. Coppin et al. [223]. They present methods not only for 3-channel but also for multispectral imagery and group detection methods in nine classes starting from simple methods like difference or ratio images to image regression. For ecosystem monitoring, change detection is only the first step. It is followed by change classification, i.e., classification of change patterns, and change modeling, i.e., prediction of when and why changes occur based on a model—both outside of the scope of this paper.

7.7. Presentation Formats

Several possibilities exist to present the old and new photograph together. In print, presentation forms are limited compared to an interactive presentation on screen. An overview is given in

Table 1. Overview of possible presentation formats of rephotographs. Print indicates whether the given method is suitable for printing, Cost indicates the required work to create the registered image pair, and Quality of Registration indicates the required registration quality for this method.

Name	Description	Print	Cost	Quality of Registration
Images Besides	The two images are displayed next to each other	Yes	Lowest	Low
Overlay Selections	(Rectangular) parts of one image are inserted into the other image	Yes	Low	Medium
Feather Selections	Arbitrary parts of one image are blended into the other image	Limited	Medium	Medium
Photomontage	Content aware precise cut out and insertion of objects	Yes	High	Case dependent
Chequerboard Pattern	Images are divided into chequerboard fields. Black fields belong to one white fields to the other image	Yes	Medium	High
Holding	Holding the old image into the scene of the new image	Yes	Low	Very low
Embedding	One or more old images are overlaid over a new (panoramic) photograph	Yes	Medium	Medium
Slider	A vertical slider separates the two images. By moving it more of one or the other image is revealed	No	High	High
Alpha Blending	The $\mathrm{displayed}\mathrm{image}$ = $\alpha ·\mathrm{old}\mathrm{image}$ + $(1-\alpha )·\mathrm{new}\mathrm{image}$	Limited	High	High
Animation	Quick alternation or slow transformation repeatedly from old to new image and back	No	High	High
Time Reveal	First image is overlaid over second image around mouse cursor	No	High	High

.

The classical approach to represent rephotographs in print is to present the images besides each other either on a page or a spread. This allows for easy global comparison. Comparing local details, e.g., how far a glacier declined is more difficult. However, this format is easy to implement and images do not have to be perfectly registered. If more than two images are to be compared this methods is of limited use, since switching pages to compare images results in a negative user experience.

Another possibility in print is to overlay selections of the images. One or more—most often rectangular—patches of one image are cut out and laid over the other image at the correct positions. This allows for a good comparison at the transition between old and new image. However, parts of the old image are masked by the new image or vice versa. If only the compilation is available, information is lost. This format is not as easy to create, as to display the images beside each other, because the registration has to be good at the transition edges.

Yet another possibility, to feather selections of one image into the second, is an extension of the overlay format: Instead of a hard rectangular cut, a—most often oval—selection of one image is positioned at the correct position in the other image and the edge is softly blended, resulting in a smooth transition from one image into the other. The selection has not to be oval, it can also encompass a certain object. Most often an object of the old image is selected and feathered into the new image. This format is more complicated to create. However, registration does not need to be perfect—only the transition part between the two images has to be aligned. The method is less obtrusive, but, similar to the cutout and overlay formats, information from both images is missing due to overlap.

Instead of feathering the images a photomontage can be employed. In this case single objects are cut out precisely from the new image and inserted into the old image or vice versa. This process allows to place present objects into the past and past objects into the present. Placing a modern streetcar onto a historic street would be one example [224]. The effort to create this presentation format is high, since objects have to be selected, cut out, and placed correctly in the other image.

Using a chequerboard pattern is a method less used fur actual presentation but to evaluate the quality of the registration. With this format an imaginary chequerboard is laid over each image. The images are then combined by taking the parts, overlaid by white fields of the imaginary chequerboard, from one image and the parts, overlaid by the black fields, from the other image. This results in a presentation with many hard transition between the two images, allowing to check registration at these transitions. By scaling and translating the chequerboard pattern relative to the images the number and position of the transitions can be varied. Other patterns such as vertical or horizontal stripes are an option, too.

Holding the new image as a (photographic) print or displayed on a smartphone into the scene while taking the new image is a straightforward method of rephotography. The rephotographers hand together with the old image is visible in the new image. This format can also be achieved with image manipulation techniques, after the new image was taken. The comparison between the two images is difficult, but the images need not to be registered. Often only the compilation is presented, thus the information of the area in the contemporary image over which the historic image is superimposed is lost. This technique is sometimes called Photofit [225].

Embeddings of one or several old photographs into a new (panoramic) photograph allow to compare several images and give an understanding, how images not taken from the same vantage point or with the same view relate to each other. In this way, the stories of different historic photographers can be connected.

Similar to moving or pop-up books, e.g., the ones of E. Nister [226,227], mechanical transformations of images printed on cardboard can be used. Both images are printed on stripes of cardboard and placed on top of each other. Manually pulling on a pull tab moves the stripes of the top image behind the stripes of the bottom image, revealing the bottom image [228].

For onscreen view, the above methods can be extended and other methods can be employed. Although images can be displayed beside each other on screen, this method is seldom used. For more than two images, this method can be made interactive by selecting which two images are displayed in large, based, e.g., on thumbnails.

The feather selections method is used on screen as well. It can be extended into an interactive version. In the time reveal method mouse input is used: In the area around the mouse cursor the first image is visible, while the second image is visible in the reaming area. When moving the mouse, different parts of the first image become revealed. The area can be either a sharp rectangle [63] or can be circular and feathered. For this method registration has to be good. This method can either use special software or can be included into webpages using JavaScript (JS) [209].

The most used on screen format is an interactive vertical slider to either reveal more of the old or the new image. In the beginning, the left half of one image is displayed on the left side of the slider and the right part of the other image is displayed on the right with a line, which is the slider, dividing the images. Users can grab the slider with the mouse and move it to either side, enlarging the visible area of one image while simultaneously reducing the visible area of the other image. This method can also be animated by moving the slider automatically from side to side. JS—code for this method is freely available [176].

On screen alpha blending is another format. The registered images are stacked over each other and the user can select the visibility of both images. Either only one image is visible, both images are visible to equal or unequal parts. While this method requires very good registration, it allows for global and local comparisons between the two images. This method can be extended to more than two images by either user selection of alpha values for each image separately or setting $\alpha$ of all images to $\frac{1}{\mathrm{number}\mathrm{of}\mathrm{images}}$. More complex blending methods can be used as well. For example black and white images can be colourized with complimentary colours, e.g., one image is coloured red and the other cyan. This can be used to check registration quality: For cases in which similarity of grey values corresponds to similarity of objects registration is good, when no cyan or red borders around object edges exist.

Using an animation in form of a Graphics Interchange Format (GIF) is another possibility. Here, the old and new image are alternating in the same image frame or the transformation from old to new is animated with several steps blending one image into the other. In the first case creating the animation requires less work, compared to the second case. In both cases images have to be well registered. Animations catch the eye and require no user interaction. A quick alternation makes changes graspable, a slow transformation allows to understand relationships. Normally, users cannot change the speed of the animation. Animations are not limited to simple transformations but can become more complex up to revideography (see Section 4.4.5).

7.8. Computational Rephotography

Rephotographers can be assisted by computers not only in finding the exact camera position and (semi)automatic image registration (see Section 7.5). 3D models, too, can be used in several ways: (1) From a 3D model, generated from multiple new images, a new image can be generated, which is registered to an old image. (2) New or old images can be embedded into a 3D scene, giving context and allowing to move between different old and new photographs of the same scene or object.

7.8.1. Virtual Rephotography

One possible solution to the problem of an unknown or unreachable original camera position (see Section 6.9) is the creation of a virtual rephotograph from a collection of images. K. Lee et al. create a virtual point cloud from a collection of images, either taken by the rephotographer or collected from the internet. The rephotographer creates correspondences between the old image and the point cloud. Then a rephotograph is computed from the point cloud [229].

This idea, to construct a model of a scene from a collection of images/multiple views and synthesize new images/novel views from the model, is not limited to the works by Lee et al. but a reoccurring theme in the field of computer graphic [230,231]. Novel views can be generated from single images as well [232,233,234]. Waechter et al. present a metric to evaluate the quality of the novel view or rephotograph [235].

This approach seems promising for rephotography, but further work is required. However, while this may result in visually pleasing rephotographs, these photographs are computed and do not necessarily reflect the real view. This is especially true for 3D scenes and virtual rephotographs which were generated from images taken at many different points in time. This can be extended to a mapping of (re)photographs into a 3D scene in which users can navigate from image to image (see Section 5). for browser based implementations.

7.8.2. Mapping of (Repeat) Photographs onto a Digital Elevation Model (DEM)

P. Chippendale et al. map georeferenced images of mountain scenes onto a DEM [236]. They create a virtual 360° panorama from the DEM centred at the camera position of the image. From both the panorama and the image mountain profiles are extracted and mapped to each other. Although, this method is limited to areas with significant elevation changes and requires the exact original camera position, it could be applied to the rephotography by mapping both the old and new image to the DEM.

8. Future Research

While semiautomatic registration of rephotographs is now possible in the web browser, further automation is desirable. Using deep local features [215] seems to be a promising approach. Feature matching could be limited to areas marked by the user allowing to register highly different images.

When it is easy to register rephotographs, it is easy as well to register a camera stream with historic images to help find the correct camera position. This in turn makes mobile apps for the recording of rephotographs more feasible. For these apps further research in Human Computer Interaction (HCI) is required to guide the user to the correct camera position in an intuitive way—potentially with Augmented reality (AR).

Another open question is how much similarity between the old and new image is required to make a connection between the two. While there exists compilations from (WWI) with no similarity except the horizon line [27], one cannot make a connection between these images without external resources proofing that these two images show indeed the same scene. It seems that at least one clearly visible object has to be present in both images to make a connection.

An entirely different question is how the heritage of rephotography can be preserved and made accessible. Up till now no all-encompassing physical rephotographic collections exist. The old images of rephotographic compilations are often already part of other collections and many online resources vanish with time. Creating a persistent online archive for rephotographs could be a valid option.

9. Conclusions

We followed the growth of rephotography from its birth in glaciology at the end of the 19th Century to other sciences and to all continents. We showed many different scientific applications and publications from the fields of natural heritage, art and culture, education, and sociology. Further, we presented popular scientific and belletristic publications mostly dealing with natural and more personal tangible heritage.

Following this literature review, we presented online management software and other online resources, noting that many online resources cease to exist after rather short periods of time and that many resources addressing a niche topic of rephotography exist, but only few rephotography portals for the creation, storage, and retrieval of rephotographs exist. Here, we concluded that work is required to preserve the rephotographic heritage itself.

In the next chapter, we presented the challenges in rephotography, finding the rough area, in which the old image was taken, finding the exact recording position and time, and registering the old and new image, such that objects are positioned at the same image coordinates in both images. We noted that different recording positions lead to parallax and that changes in the scene can make reaching the correct recording position and the registration difficult or impossible.

Lastly, we presented analogue and digital techniques to tackle the previously presented challenges. We discussed how the camera position can be found with the help of instant pictures, transparent film, and parallax or digitally, by registering the preview of a digital camera with the old image. Further, we discussed digital methods to register rephotographs, noting that no complete automatic method exists. We compared different analogue and digital presentation formats, identifying the trend that digital methods can be more sophisticated and interactive but require a higher quality registration.

Overall, we saw that rephotography—not always under this name—is widely used in scientific, artistic, and recreational ways, to connect with, measure, and conserve our heritage, and that the already existing work on camera relocalization, image registration, and preservation of the rephotographic heritage needs to continue.