Smart and Sentient Retail High Streets

Abstract

Here, we examine the extension of smart retailing from the indoor confines of stores, outward to high streets. We explore how several technologies at the union of retail intelligence and smart city monitoring could coalesce into retail high streets that are both smart and sentient. We examine the new vantages that smart and sentient retail high streets provide on the customer journey, and how they could transform retailers’ sway over customer experience with new reach to the public spaces around shops. In doing so, we pursue a three-way consideration of these issues, examining the technology that underpins smart retailing, new advances in artificial intelligence and machine learning that beget a level of street-side sentience, and opportunities for retailers to map the knowledge that those technologies provide to individual customer journeys in outdoor settings. Our exploration of these issues takes form as a review of the literature and the introduction of our own research to prototype smart and sentient retail systems for high streets. The topic of enhancing retailers’ acuity on high streets has significant currency, as many high street stores have recently been struggling to sustain custom. However, the production and application of smart and sentient technologies at hyper-local resolution of the streetscape conjures some sobering considerations about shoppers’ and pedestrians’ rights to privacy in public.

1. Introduction

“People in the streets/Please/People in the streets/Please”

[1]

In this paper, we examine the ongoing transformation of retail high streets from their traditional roles as corridors of urban activity in downtowns and we portray them as newly smartened, technology-saturated environments that are increasingly cyber-physical in form and function (Figure 1). In particular, we examine retailers’ increasing aptitude for establishing a degree of sentience—through sensing, knowledge discovery, and contextual awareness—of the activities and intentions of would-be customers as they move along high streets. We trace the precursors for these capabilities in traditional smart city functions that are rooted in Internet and Communications Technologies (ICTs) and monitoring of streetscapes, and we then explore their fusion with knowledge discovery tools that were pioneered for retail computing of customer experience. We examine the tendency for both smart technologies and machine sentience to migrate inexorably outward, from the cloistered setting of retail stores and onto the streetscapes beyond, where they come into contact and synergy with the physical functions of high streets as a substrate for embodied pedestrian activity. We will follow the contention that various conditions permit some of the monitoring and communications capabilities of smart cities to be distilled to very hyper-local resolutions of the streetscape, where they can then be attenuated with contextual (and often automated) intelligence that reaches to individuals on sidewalks. In particular, we examine the role of the customer journey framework, originally developed for customer experience operations, in enabling sentient technologies to sense—and make sense of—customer behavior on retail high streets. Importantly, we reach a conclusion that retailers’ insight of outdoor behavior can, in some cases, match the resolution that they have honed in-store and on electronic commerce (e-commerce) platforms, with potentially positive implications for building smart communities of retailers and citizens, albeit tempered by possibly adverse influence on existing expectations of privacy.

The term smart retailing is perhaps quite difficult to pin down, as is the case for many “smart” phenomena [2]. Generally, smart retailing refers to the use of digital technologies to manage touchpoints between retailers and customers. Touchpoints may manifest in many different ways in retailing, either at the site of pre-purchase (e.g., advertising, comparison shopping, price-matching); product evaluation (e.g., embellishing product displays, suggesting product pairings, up-selling related product services); at the point of sale (e.g., contactless payment, harvesting on-transaction customer demographics, generating coupons); or in after-sale relationships (e.g., collecting customer reviews, managing warranties, informing marketing campaigns).

Our discussion initially expands upon the touchpoint-centered conceptualization of smart retailing, to also consider a combined physical and technological approach to managing the entire broader customer journey, which in the case of high streets notably and necessarily includes new interpretation of outdoor servicescapes and non-customer pedestrians. We will treat the customer journey in more detail in Section 5.6, but here we offer a concise definition. Customer journeys represent retailers’ codification of customers’ (and would-be customers’) experiences while shopping, as paths through retail servicescapes and the touchpoints that the retailer and the customer might co-create while engaged in those paths. One significant consideration is that this codification is now routinely entrusted to machine learning that operate on advanced sensing capabilities [3]. This is an important point that is worth emphasizing: the customer journey is at its essence a management lever for the retailer. One may usually consider servicescapes as the physical environment for retailing, composed on high streets as store fronts, retail objects and infrastructure, and palpable advertising. Increasingly, servicescapes are additionally (and perhaps simultaneously) information spaces: of branding, e-commerce, and digital applications that can be used to bring customers to the high street to shop and to deliver customer experiences to them while they are there. We consider touchpoints as being opportunities for customer action, reaction, interaction, and transaction to be embedded and embodied [4] within those servicescapes. And, critically for this paper, we focus largely on the high street as the setting for that servicescape. We therefore exclude several “back-of-house” components of smart retailing that relate, for example, to inventory management and warehousing, supply logistics, and dynamic pricing. We also skirt the role of smart technologies in e-commerce and related issues of customer relationship management.

A chief thesis of this paper is that we might envisage sentient retailing as a developmental offshoot of smart retailing. We will advance this idea more fully in the remainder of this paper (particularly in Section 4), but here we offer the definition of sentient as meaning the ability of retailers, through automated technology that can generate empirical data and knowledge, to not only monitor retail high streets through various means of observation and sensing, but also to map those observations relative to an extensible knowledge framework. One important point that we raise in this discussion is that unlike smart cities, for which urban science provides the most convenient structure for knowledge discovery, on retail high streets it is the individual behavior of customers and would-be customers that provide the ingredients for establishing a knowledge base. Indeed, we will argue that the long-held and now quite robust use of the customer journey framework from customer experience analysis provides a firm footing for retailers to build streetside sentience and to match it to their operations.

2. Retail High Streets

High streets manifest from the set of activity sites, building façades, pedestrian infrastructure, and visual media that form along urban arterials in many cities. We might consider high streets as a special form of streetscape, distinguished by a centralized and highly-connected location in an urban area, or by their ability to offer core urban services such as transit, municipal administration, and tourist attractions. In some cases, high streets are pedestrianized, but in others they also include curbsides with roadways and transit infrastructure such as bus and tram stops and taxi ranks. We have a number of reasons for considering high streets as the target for our review. A key component of high streets is the large volumes of varied pedestrians that they attract. High streets are, perhaps above all other considerations, characterized as usually being full of people. High streets are thus often considered and examined as a critical social component of many cities, driving different facets of local economics, culture, and well-being [5,6]. In many cases, large communities are anchored to high streets, with nearby residents depending on them for services, jobs, and transportation. The constituents and relative vitality of high streets have therefore become a significant concern in consideration of urban development and community welfare [7].

Retail high streets are generally characterized by relatively highly-traversed shopping, dining, and entertainment opportunities that are densely interspersed among other service establishments (often with complementary relationships, e.g., transit stations, banking, libraries, and post offices). Many retail high streets are themselves regarded as significant places within cities, commonly with historical longevity in that role, e.g., Grafton Street in Dublin, Oxford Street in London, Rodeo Drive in Los Angeles, Ginza in Tokyo, and Strøget in Copenhagen. Retailers are commonly drawn to high street locations because of the spatial advantages that they afford [8]. The business model for retail high streets is in many ways quite straightforward: groups of vendors co-locate on a streetscape to offer goods for sale. In return for sharing their locational advantage on a high street, retailers gain a share of aggregate footfall to the area. Many retailers can also draw secondary benefits of complementarity from proximity to related retail activities (tailors next to haberdashers, for example). Fixity in location generally assures retailers on a high street continued and privileged access to its benefits. However, colocation also implies two additional important points. First, retailers are essentially in competition for the same foot traffic that courses past their establishments. Second, any diminution in the spatial advantage afforded by the high street can have broad fallout that is shared across all vendors. This is because locations are economically sticky, in the sense that it is very difficult (and costly) for a retail enterprise to move its front-of-house operations from one site to another. This stickiness is due to various factors, including the investment inertia in fixed costs of leasing (or buying) a physical store, the effort required to furnish retail sites, the tendency for retailers to adorn high streets with advertisements and branding, and the staunch connections that form between notions of place [9] and the experiences that those places can fashion between customers, their activities, and local urban geography of the high street. For many retail high streets, significant neighborhood advantages may be impossible to recreate outside the high street, including access to transit and the footfall that it supplies, ingress and egress geographies for vendors and suppliers, and local catchment for employee labor. As a consequence, issues of general decline in the popularity and visit frequency of retail high streets weigh heavily in the concerns of many retailers [10].

In recent years, vendors on retail high streets have witnessed general abatement in the volumes of footfall past their establishments [11], which has put pressure on their businesses collectively, while also elevating their internecine competition for the attention of individual customers. There are overlapping causes for the drop in visitor traffic. Aspects of the decline have been associated with a relative downturn in the general popularity of downtowns and other central locations in cities [12,13,14,15,16,17,18,19,20]: essentially, central locations in many cities have stopped being popular activity sites. Other facets are retail-specific and relate to the relative uptick in use of e-commerce platforms as a substitute for in-person shopping and the zero-sum outcomes of that competition for tangible stores. An extreme example of this is apparent in the phenomenon of “showrooming” [21], whereby a would-be customer might visit a tangible high street store to browse and assess products, only to make the purchase by e-commerce from another vendor. These dips (of downtown attractiveness and favoritism for physical shopping) have been abruptly accelerated due to the sudden drop in footfall during the COVID-19 pandemic, with the consequence that many retailers (particularly those providing non-essential goods and services) have shuttered their operations altogether. Nevertheless, the problem of retail high street decline has long endured as a larger concern for urban studies, because erosion of the overall vitality of the retail high street can have serious knock-on implications for the communities that are anchored to them [16,17]. Many residents rely on retail high streets for day-to-day services and goods. When those opportunities disappear from reach, phenomena such as food deserts can begin to take hold [22]. Similarly, many retail high streets provide a source of employment for their local communities, both from retailing jobs themselves, but also from allied services that rely on retailing. The shuttering of large swaths of stores can also negatively impact street culture, and this may also be the case for closure of individual businesses, particularly when those businesses provide nîche services to particular sectors of the community, as in the case of specialty food stores or health goods providers, for example.

To address these issues, many retailers have embraced technology as a solution to varying degrees. At face value, an obvious adjustment has been for high street retailers to adopt e-commerce platforms, shifting essentially to a hybrid form of online/offline service provision, albeit often with higher costs than purely e-commerce players for carrying prime-location leases and staffing physical shops when those services are fixed to high street locations. This shift, toward what is called “omnichannel” retailing, allows customers to engage in retail services in one channel (e.g., online reviews and pricing searches) with freedom to move back and forth to another channel (e.g., physical inspection of the product and collection of a purchase) [23,24,25]. For the most part, the high street retail omnichannel is a by-product of developments in database and Web technologies. We will address those developments in this paper, but they are not a main focus of our narrative. Alternatively, we are interested, in this exposition, in uncovering a parallel transformation of high street retailing: the perhaps more subtle and hidden emergence of computing technologies that are smartening retail high streets in ways that resemble existing developments in urban computing, but in shrinking format and application. Alongside the omnichannel, then, we see the rise of smart and sentient retail high streets, perhaps coexisting with the traditional view of the high street, but now in cyber-physical form.

A logical question is how smart and sentient retailing differs from smart cities, and where we might usefully draw parallels between the concepts? Both smarts and sentience are sourced in data. Dealing with data is obviously a huge component of smart retailing. Data are one of the levers that retailers have to drive action in their business models [26]. Through data, we see one of the pronounced connections between smart retailing and smart cities. Smart cities are, at their core, data-rich environments and part of the smartening of the city has involved making urban data easier to acquire, organize, analyze, and convey [27,28]. Retail data tend to be much more rapidly acquired and updated, more high-resolution, and possess much more significant fidelity than the traditional data products associated with smart city research. Retail data are also often unique to a given retail operation; they are generally proprietary and private. Indeed, few retail data are open to public or academic view, save some crumbs that fall from customer review systems that are public-facing on the Web or social media applications. This contrasts significantly with smart city data, which are perhaps more likely to be open in access and free in availability than retail data can be [29,30].

We contend that high streets might conveniently present as a point of fusion between smart cities and smart retailing. As the information that smart technologies generate increasingly become pivotal in brokering interactions, there are perhaps opportunities to do more with the burgeoning systems that have been developed by retailers for their transactions. In this paper, we especially focus on the capacity for retail high streets to provide the functions of a smart community, initially as an informational subset of the smart city, but perhaps in the longer term as a platform for local interactions between pedestrians, shoppers, businesses, residents, and the everyday and moment-to-moment transactions that can bring them together in information spaces. That these developments might lead to smart communities is a thread that we pick up in Section 5.4. At this point in the paper, let us simply say that high streets could present as a natural intermediary between city-scale structures (intra-urban areas, districts, neighborhoods) and establishment-scale structures (buildings, frontages, display elements). Pedestrians and shoppers alike rely upon the high street as a central structure in organizing their activities in many urban environments [31]. We will make the argument that smart technologies now provide high-street-level data to connect cities, establishments, and individuals, at least in information space. We will also highlight that a range of technologies are perhaps capable of fomenting a burgeoning sentience across these elements. Moreover, retailers’ frameworks for building that sentience atop their ideas about the customer journey could provide a broader conceptual model for localizing the smart city relative to individual customers and pedestrians.

In the next section, we will review the broad capabilities that smart retail high street technologies support. Our emphasis, in doing so will be to emphasize how traditional forms of smart technology—chiefly those developed as ICTs—have found their way from stores, outside onto high streets, and what implications may have resulted. Following this, we will review the technological developments in computing—especially via sensing, communications, knowledge discovery, and the varied forms of automated inference that they enable—that have catalyzed the emergence of sentient retail high streets. In particular, we will structure this discussion around how smart and sentient technologies match to the framework of the customer journey [32]. The break from “smart” to “sentient” is not discrete; in other words, at some indeterminate stage, retail technology shifts from ICTs to artificial intelligence (AI) and it is not always straightforward (or necessary) to draw a hard distinction between the two [33]. Finally, we note that our thesis is not one of unchecked technological determinism; rather, we couch the benefits of smart and sentient technologies for retailers with the very real pitfalls that those same technologies might wreak for people’s experiences of privacy and control in public spaces.

3. The Technological Substrate of the Smart Retail High Street

Defining what might be considered a “smart” technology, and what may not, is always difficult. Generally put, one might consider smart retail technology as the apparatus for collecting data about retail activities and for communicating those data at scales that can drive retail decisions at scales of space and time that outpace “non-smart” technologies. We might emphasize the central role of automation in smart technology: the ability for smart systems to tirelessly and inexorably pore over the minutia of retail facilities and harvest data.

A point that we wish to raise is that smart retail technology is increasingly driving retailers’ insight into the customer journey. Traditionally, significant introductions of technology to retail operations have led to entirely new touchpoints between customers and the retail servicescape, or they have brought existing touchpoints into closer alignment. Currently, the ongoing refinement of the contactless store serves an extreme example (which we address in more detail in Section 5.10). Contactless retailing has been successful in rendering touchpoints both seamless and almost invisible to the customer, by folding them almost wholesale into technology. At its nucleus, perhaps, contactless retailing has managed to obviate the distinction between technology and customer experience. In this way, smart retailing is at the forefront of the classic disappearance trick [34,35] that ubiquitous computing is so apt to perform. This vanishing act offers a significant advantage for retailers to manage operations against the customer journey: at their core, contactless and frictionless touchpoints help retailers to build even stronger connections (and leverage) from their servicescapes to customers.

We might distinguish sentient retail technology as something that is different than smart retailing, yet nevertheless related (and reliant). In this paper, we consider sentient retail high streets as chiefly shaped by the systems and devices for sensing and making sense of retail environments that sit on top of smart technology. If smart retail high streets are predominantly a by-product of ICTs, sentient retail high streets are an outgrowth of the machine learning and AI that sprang up from the large troves of data that they produced. (We discuss sentience more fully in Section 4).

In this section of the paper, we seek to explore the impacts of technology in extending the reach of smart retailing from the indoor settings of stores and the shop floor to the outdoor high street. An important point that we make is that while moving through streetscapes as pedestrians, people also fall within the purview and potential catchment of smart retailing, where these technologies can be brought to bear in mapping them to (retail-sided) customer journey frameworks. Retailers refer to this as co-creation of the customer experience [36,37] with technology. While co-creation may traditionally have been a component of in-person sales and branding, increasingly retailers are relying on what others have termed to be a digital-physical servicescape [38], i.e., accentuation of physical stores and products with digital counterparts, so that retailing becomes cyber-physical [39]. For retailers, the benefits of cyber-physical servicescapes over purely physical settings are pretty straightforward: they help physical retailers to compete with e-commerce platforms by fundamentally lowering long-term costs by substituting technology in roles that would traditionally be staffed. Significant informational by-products can also be monetized, as cyber-physical retailing often allow retailers to harvest vast troves of very high-fidelity and fine-specificity data on customers and their customs. In a relatively recent development, however, retailers’ notion of cyber-physicality now additionally includes forms of computing and computation that have more similarities with AI than with traditional data-centric considerations. In some cases, facets of retail technology can work in self-reinforcing loops, with live data training AI routines that dynamically hone retailers’ sales approach. We might therefore consider that the smart retail high street is giving way to sentient retail high streets. That the means of that sentience are also now routinely hidden from view, while also operating in open public spaces of the high street, raises some cause for concern.

In the review that follows, we first address traditionally-considered “smart” technologies and their impact on high street retailing. This includes Linked Data, wireless communications, near-field communications (NFCs), the Internet of Things, and location-aware technologies. We then expand that consideration to encompass technology that is either designed for sentience, or that has seen applications to sentient functionality. We especially highlight the impact of wearables (as contextually-aware devices), cameras and computer vision, and edge computing in advancing retailers’ (machine-based) sensing and perception of customer journeys on retail high streets. It is via edge computing, especially, that sentient retailing could be considered as coming into operational contact with smart cities, and we examine how several of the technological components of retail high streets might coalesce on the edge.

3.1. Linked Data

High streets provide a set of natural structures for data. The customer journey (particularly a phased consideration of the customer journey as a procession through touchpoint events) provides further structure [32]. Several smart city technologies can provide high-precision identifiers that unveil anchors in that structure (location, time, customer identification, product identification), which allow formal linkages to be established. Together, identifiers and links support broader semantic operations on data, i.e., the ability to infer relationships context-free by relying on the data itself [40]. This gets us some of the way toward building knowledge bases for retail data, usually for retail transactions. However, a semantic challenge often presents for retail data in ways that are not concerning to smart cities: because of their proprietary nature, retail data-sets are often isolated as “islands” among different retailers, suppliers, and data-collection modalities. Moreover, these islands are also often incongruous with one another, with different ontology, making it difficult to build semantics among them. Linked Data provide mechanisms to build semantic interconnections between islands [41], and to do so with high flexibility. Linked Data are generally associated with efforts to build and leverage structure between data and documents (media) on the Web, in the variety of formats in which they appear. Specifically, Linked Data allow typed links (a link as well as information about the nature of that link) between arbitrary data, and do so in ways that facilitate formal definitions, that provide for external links, and that can be interpreted by machines [42].

When applied to customer records, Linked Data can potentially reveal components within the customer journey, with a number of advantages. First, Linked Data support standardization of vocabularies across different retailers and among different retail operations. Yet, Linked Data also support malleability to add new terms and meanings to those vocabularies, as needed. This is important in the (often dynamic) product environment of retailing. Second, Linked Data allow for connections between data and documents from very different domains, e.g., connecting data on retail products, supply chain information systems, and social media campaigns. Third, Linked Data can facilitate rapid indexing, so that semantics can keep pace with high-velocity data. This is important for matching the pace of retail transactions. Fourth, Linked Data can scale: they are built on top of (and for interoperability with) existing Web architectures, with the ability to handle very large volumes of data and documents, whether across large inventory systems, massive troves of customer records, or potentially the entire Web. Fifth, the semantic functions associated with Linked Data may be automated, with the result that retailing enterprises may build smart views on their operations (potentially) in real-time.

In aggregate, these five capabilities set the stage for massive inference within and among the big data stockpiles that can be collected on retail high streets. As we will elaborate in Section 5.7, with tethers to the sorts of Web-sourced data that the retail omnichannel provides and also to real-time data feeds that could come from street-side sensing, Linked Data, in particular, set the stage for the development of context-aware intelligence on retail high streets. We may also consider that Linked Data can support fusion between retail operations and broader rhythms and motifs of high streets, so that data on customer touchpoints could potentially be linked to data on pedestrian trajectories, for example, so that potential patrons to a store might be identified within broader streams of crowd flow on streetscapes [32].

3.2. Wireless Communications

Wireless communications technologies have been an important catalyst in transforming streets into smart landscapes [43]. Wireless communications are a relatively old technology, dating to the start of the Twentieth Century with Fessenden’s tests of wireless telephony [44,45]. However, the use of wireless technologies to transmit digital data have had a profound influence on smart cities, beginning with the use of cellular phones, and continuing with the development of Wi-Fi protocols [46]. Researchers in urban science and smart cities studies have long used wireless communications as a way to examine relatively coarse (intra-urban) as well as fine-scale (individual communications users) patterns of presence and movement, often as ingredients for building understanding of street-scale urban phenomena [47,48,49]. Innovation in the development of wireless technology now proceeds at a fantastic pace, and a set of new technologies based around narrow spectrum communications, in particular, are beginning to see applications to retailing and to streetscape applications. These include the development of fifth generation (5G) and even sixth generation (6G) communications, which provide broadcast ranges of around 300 m (over less distance than 3G and 2G, but with much greater bandwidth for data). 5G, for example, supports bandwidths of up to 1 Gbps, compared to up to 40 kbps for 2G, 144 kbps for early 3G, and 100 Mbps for 4G with Long Term Evolution (LTE) standards. 5G systems, in particular, adopt some smart approaches to mitigating latency issues, through use of beam-forming technologies [50]. These developments can support the delivery of high-definition video to mobile consumer devices, as augmented reality (AR), for example, with the ability to personalize marketing content to specific people in particular locations.

For retail high streets, the mobility in communications that wireless supports has fostered significant developments in transportation and delivery logistics, especially for the portions of logistics in which transportation nears retail premises, over so-called “last mile” distances that are generally situated along retail high streets. In particular, wireless communications have enabled retail information systems to exchange inventory and order data with transportation providers directly, greatly speeding-up the movement of retail products between stores and the high street. This is now evidenced by the plethora of just-in-time ordering and delivery systems that have emerged in most cities.

Large webs of wireless communications are routinely used for analysis of relative pedestrian density and crowd flow in smart cities frameworks. This includes research on the “pulse” of cities as evidenced from call detail records (CDRs) that are polled by cell towers, for example [51,52]. When accessible to researchers, CDR data can be useful in isolating patterns of assembly and relative movement of pedestrians within cities, with resolutions that can reach to intra-urban areas [47,48]. More fine-grained analyses of wireless communications can additionally be used to build dynamic profiles of activity on high streets, with the implication that these data could be used to assess foot traffic, customer surge, and even to model the behavior of pedestrians on sidewalks and around stores [53,54,55]. A range of companies (see [56], for example) have developed systems for determining the number of customers that visit stores, as well as the time that they spend in those visits, automatically from cellphone data.

3.3. Near-Field Communications

Many of the wireless technology applications that we discussed in Section 3.2 also support retailing in close form, down to the level of individual transactions with products. These include hyper-local communications schemes such as Bluetooth (which can broadcast within a range of about ten meters) and NFCs (which can broadcast within a range of about a few centimeters) [46]. In particular, NFCs support the exchange of information between consumers and retail objects, generally over very small windows of space and time, and usually using simple artefacts (rather than the smart devices we discussed in Section 3.2). The resolution of many NFC systems allow for incredibly high-resolution retail touchpoints to be characterized, down to the scale of a card’s motion on a retail counter. Many near-field systems have relatively low power requirements, which enables customers to generate interactions and transactions with retail infrastructure by using tapping and waving motions with credit cards, loyalty cards, and identity badges. NFCs are distinct from EMV-based systems (Europay, Mastercard, and Visa) that allow for a card chip to be read when inserted into a scanner [57,58]. NFCs are used to broker the broadcast of data, but do so over very small-range distances. This means that the NFC object does not have to come into contact with the reader; rather it simply has to come into proximity with it. The most prevalent applications of near-field technologies have been in support of expediting payment within indoor store environments [59], e.g., tapping a payment card at the point of sale, using in-wallet cards at kiosks and turnstiles to gain entry to retail experiences [60], and automating a tally of goods to be purchased as they are added to smart shopping baskets and carts [61]. NFCs have also been used by retailers to provide dynamic stock-taking: Resatsch et al. [62] showed that NFCs embedded in shelves can provide dynamic data on product levels on the shop floor. Karjol et al. [63] have even demonstrated that NFCs can support the exchange of information between shopping carts in a store. It is notable, in particular, that many aspects of retailers’ customer relationship management (CRM) systems may also piggyback on near-field transactions, so that hard connections between the tangible act of interacting with retail objects and digital records of the product and the customer can be established at the point of communication.

Near field technologies are also routinely used on the streetscape outside stores, for example, to support tap-in entry between sidewalks and buildings, for inventory management during stock loading and unloading at the curbside, and for theft prevention at the point of store exit. This suggests that broader linkages between the indoor dynamics of customer experience could be extended outdoors by following NFCs as they progress between the store and the high street. Work by Basili et al. [60] has shown, for example, that the development of a NFC-enabled “smart tourist card” can be used to couple information on tourists’ movement and transactions with retail objects and payment systems. In urban science, a significant volume of work has been built up by Batty and colleagues [29,64,65], to perform spatial analysis on the troves of smart card data that are produced, for example, by transit NFC payments. This work is expanding the resolution of analysis of general patterns of transit and commuter flow to high street transport interchanges, which often feature large associated retail footprints [66].

3.4. The Internet of Things

The Internet of Things (IoT) refers to the networks and network protocols that allow a range of machines (usually devices or networked objects) to communicate by exchanging information. IoT is broadly used in smart city applications [67,68,69], providing the protocols on which many of the wireless communications technologies that we discussed in Section 3.2 and Section 3.3 take place. Dourish [70] and Hudson-Smith et al. [71] have suggested that IoT is so pervasive in smart city applications that we might even have cause to distinguish an independent “Internet of Urban Things”. Spandonidis et al. [72] have also recently introduced a novel concept for the “Internet of Vehicles”.

IoT is an extension of the general-purpose Internet, primarily distinguishable by its intended use. IoT traffic is usually generated by devices for other devices, compared to the World Wide Web, which is designed to be human-facing. IoT data is therefore characterized by data such as sensor readings, state transitions, and location broadcasts. For example, a large volume of the traffic over IoT may be comprised by state notification data, with devices continually broadcasting and receiving status updates from other devices in a highly dynamic fashion. In many instances, the devices that form IoT networks produce and exchange these data themselves, rather than relying on formal server-side exchange hardware. This is important as IoT devices are often small in size and limited in power consumption, and one of the goals of IoT is to support the exchange of data between them using sparse resources. Devices may rely on dedicated protocols to achieve this on IoT networks. The use of Message Queue (MQ) Telemetry Transport (MQTT) is an example [73]: a protocol designed to consume a relatively small bandwidth in communication among devices such as microcontrollers. IoT can, in many instances, function synergistically with other Internet technologies: Shahrour and Xie [74], for example, have shown how a wide variety of IoT data from smart cities can interoperate with cloud resources.

IoT is something of a backbone technology for smart cities [67,75,76,77]. It has been used, for example, to promote crowd-sourcing of data collection and services in smart cities [74]; to accentuate the study and development of smart mobility services in cities [78]; and to assist in fostering smart communities within smart cities, through public access to IoT data and services, particularly those on the network edge [79]. In some instances, IoT provides a similar backbone in retailing. For example, Zualkernan et al. [80] implemented an IoT system for Bluetooth Low Energy (BLE) beacons in smart coffee shops, designed to facilitate “geo-marketing” (location-aware marketing), with MQTT as the protocol for data exchange. Their system was capable of generating event data for customer entry and exit, order data, and the position of customers and staff within the store. Using MQTT-mediated event data, they designed add-on systems that facilitated the assignment of tables to customers, access to customer purchase history for re-ordering, as well as a range of customer analysis functions, such as automated generation of information on customer stays, areas of the shop that customers visited, and coarse traces of customer movement paths. Potdar and Torrens [81] introduced an IoT framework using MQTT to transfer data on an edge computing system for monitoring pedestrian traffic in a retail district in New York.

3.5. Location-Aware Technologies

Most cities are now blanketed in omnipresent clouds of digital communications (Figure 2) that come from the wireless technologies we discussed in Section 3.2 and Section 3.3 [46]. The ability to triangulate the physical location of users and objects within these clouds (with details of their position along retail high streets), using communications between wireless devices or between devices and base stations, also presents a significant by-product of wireless technologies [82]. Retailers have used communications triangulation to develop estimates of customer footfall past stores based on communications data that are freely available within Wi-Fi clouds [83,84], as well as to develop new schemes for the delivery of geographically-curated advertising and promotions [85].

A range of location-based services (LBS) have also been pioneered to piggyback on the localization capabilities of wireless technologies. These include Radio Frequency Identity (RFID) systems that allow tags to be placed on retail products and for the position of those tags to be identified and traced through stores and outdoor environments. Melià-Seguí et al. [86] discussed a range of applications of RFID-based positioning within stores, including management of stock and the introduction of enhanced product display on mirrors in fitting rooms. Li et al. [87] introduced the “DeepTrack” system, which scans RFIDs for periods of spatiotemporal colocation in store settings. Ali et al. [88] developed the “TagSee” system, which ingeniously examines the interruption of RFID signals on shelf products to infer the presence and outline of customers adjacent to products. Outdoors, Cha et al. [89] demonstrated a system for embedding RFID base stations in street lights and then using that system to track mobile objects.

Concurrently, the cost and form factor for Geographic Positioning Systems (GPS) chips have steadily shrunk to a size that allows them to be routinely included in consumer devices, such as cellular phones, navigation systems, and cameras. Consumer-grade GPS can provide reliable positioning at resolutions of around five meters when coupled with wireless triangulation for so-called assisted GPS (aGPS) [90]. The proliferation of GPS and aGPS have supported the development of a range of LBS that are tied to high street retailing. These include the introduction of just-in-time curbside logistics for order pick-up [91], mobile customer targeting [92], location-based couponing [85], and location-based recommendation services [93].

3.6. Sensors

The proliferation of a range of sensors on high streets has been instrumental in shaping the smart city. These include sensors that are deployed along streetscapes to monitor dynamics that take place on them, as well as sensors that are placed on buildings to capture routine ambient features. Increasingly, sensors are also being deployed in a mobile fashion, on vehicles as a component of advanced driver awareness systems (ADAS), and on the devices that individual pedestrians carry as part of the smart functionality of those devices and their applications [94]. We might usefully classify the range of sensors on high streets by the things that they sense.

Environmental sensors are broadly used to survey the presence of specific compounds and particulate matter on streets, as part of environmental monitoring efforts to assess pedestrian exposure to emissions of matter in public spaces [95,96]. This may include the use of sensors to evaluate relative presence of volatile organic compounds [97], particle pollution [98], specific gases [99], or exposure to street-level heat [100], for different configurations of roadside geography or urban design along high streets [101].

Hyperspectral and multispectral imaging sensors, originally pioneered for remote sensing, have been used for the detection and tracking of pedestrians on streetscapes. Hyperspectral imaging is capable of detecting signatures across different wavelengths of light, while multispectral can peer within wavelengths. There are now a wide variety of applications of both to the study of streetscapes in general terms, as sites for the presence of environmental features that can be detected in different wavelengths [102,103,104,105,106]. In other examples, the sensing is targeted specifically at pedestrians on streets. For example, see the review by Negied et al. [107], discussing how pedestrians can be detected in complex scenes through analysis of thermal bands. Li et al. [108] also introduced a scheme for combining thermal and color bands using machine learning for multispectral detection of pedestrians.

Thermal sensors, in particular, have been used to study pedestrians. In most cases, these applications are multispectral. Thermal imaging has been used in some instances in indoor retail environments, for the detection of customer presence relative to displays (for example, the “IRLYNX” retail system [109]). Thermal imaging is commonly used to map the heatscape of street material, where issues of micro-scale urban heat islands, albedo, impacts of vegetation, and relative heat performance of different building compounds can be assessed by sensing. Work by Zhao et al. [110], for example, introduced three-dimensional thermal imaging to assess the heat characteristics of different pedestrian spaces on streets. Other approaches use thermal sensing to identify pedestrians (by their temperature signatures) within thermal scenes [111], and a range of research has shown that heat sensing can be used to trace pedestrian movement outdoors. This can be achieved with fixed [112,113] or airborne sensors [114]. When combined with machine learning, detection can be achieved with resolution capable of isolating individual pedestrian silhouettes [115].

Audio sensors may be used to measure the general volume of noise along a streetscape [116], or to monitor the presence of very specific sounds. Boles and Hayward [117] showed that street noise has varied impact on pedestrian behavior while walking. Work by Echevarria Sanchez et al. [118] has tied pedestrian exposure to noise directly to the morphology of building façades along streets. Often, particular sounds are associated with the presence of specific events on streetscapes. Using cues from bird song, research by Camacho et al. [119] examined what tones and volumes of pedestrian signal crossing alerts are most appropriate for visually-impaired road-crossers, for example. Schmidt et al. [120] argued that cheap audio sensors (microphones) in pedestrians’ phones could be used to build contextual awareness from measures of sound volume, base frequency, and comparisons to background noise. de Godoy et al. [121] introduced the PAWS wearable acoustic system for monitoring pedestrian safety around car noises using pedestrians’ own smart phones. More sophisticated audio sensing is routinely used for vehicles along high streets [122], to assess potential collisions between cars and people and objects, using vehicle-mounted ultrasound as a form of radar [123]. Lateral ultrasound sensing [124], in particular, has applicability to sensing of high street sidewalks, where pedestrians can often be rendered detectable from the roadway [125].

Motion sensors can be used to detect pedestrians on streets. The reader may be familiar with the popular use of automatic doors for retail store entry and exit [126]. The doors usually rely on pressure sensors (in the ground) or passive infrared or microwave sensors in the door to detect motion of an entering or exiting customer. Generally, little information on streetscape activity is available from these sensors, beyond the volume, timing, or frequency of movement past the sensed location. However, researchers have deployed depth sensors (usually using a mixture of infrared and red, green, and blue (RGB) light sensing, i.e., RGB+D) to study pedestrians (see an example from our research in Figure 3). Work by Charreyron et al. [127], using the Microsoft Kinect sensor has shown that pedestrian counts and speed can be sensed on outdoor streetscapes by examining motion in depth data. Similar schemes were used by the “Motionloft” commercial platform at various streetscape sites around the United States. Significant characteristics of pedestrian motion can be derived from a suite of sensors that are routinely available on the phones and watches that people carry while mobile, as well as via fitness trackers. A variety of features of motion (speed, periods of rest, effort, cadence, activity type) can be resolved by analyzing the data generated by inertial measurement units (IMUs) [125,128,129,130,131]. These insights are available chiefly from accelerometer (which measures acceleration as the derivative of velocity) and gyroscope (which measures angular velocity as the derivative of orientation) data.

Vibration sensors can be tasked in indoor environments with detecting the presence and movement features of walkers [132], e.g., for indoor occupancy sensing and measurement [133]. The technique is also deployed in health care settings, where it can additionally be put to use in monitoring falls (using seismic sensing) [134]. Vibration monitoring of footfall often relies on piezoelectric sensing, i.e., the electrical signal yielded by mechanical effects on a sensing surface. Piezoelectric sensors are usually implemented in pressure-sensing mats or tiles. Researchers have shown that high-detail features of pedestrians can be obtained from vibration sensors, including details of walkers’ direction, when the data are subjected to machine learning [135]. While most applications are to indoor environments, there has been some discussion about whether the technology could be applied to outdoor settings, on streets for example, where it has been proposed that the energy generated by high street footfall might be enough to power street lighting [136]. This points to a potential future source of information about pedestrian activity on outdoor streets, as ancillary data on the nature and the frequency of footfall could yield high-fidelity measures of individual and crowd flow past stores.

Light detection and ranging (LiDAR) sensors use lasers (light amplification by stimulated emission of radiation) and time of return of the beams to calculate three-dimensional positioning of scenes, objects, and people in indoor and outdoor environments. LiDAR may be used to build partial volumetric scenes as point clouds of localized laser–object interactions. Additionally, the amount of laser returned to sensors can yield information about properties of the object that the beams come into contact with. Premebida et al. [137] showed that LiDAR data, along with feature extraction routines, can be used to detect pedestrians from a moving golf cart. Wang et al. [138] showed that pedestrian identification and tracking can be achieved from a moving car. In both cases, the LiDAR data was sufficient enough to reveal individual features of pedestrians, such as height and body silhouette, in partial (occluded) three-dimensional fidelity. Zhao et al. [139] demonstrated that a wide array of streetscape features can be picked up from LiDAR sweeps of high streets from mobile vehicles, including the presence of pedestrians in a scene, as well as their location/position, and direction and velocity of their travel. In Figure 4 and Figure 5, we show some of our own work to deploy LiDAR sensing of retail high streets in Brooklyn, NY, USA.

4. An Argument for Sentient Retail High Streets

The smart technology discussed in Section 3 can provide a rich description of the what, where, when, and with whom of retail high street activity. This includes explanation of what is on retail high streets, where those things are in the larger and local geography of an intra-urban area, when they appear and disappear, and how objects and events might be associated with each other. Indeed, many retailers’ information systems now function on a substrate of this information, or rely on portions of it to organize their operations. In the case of mobile e-commerce (m-commerce), retailers may even rely almost exclusively on these smart technologies. Nevertheless, we might point out that significant components that would provide the “why” to match what, where, when, and with whom may remain largely unaddressed, at least outdoors. Recent developments in sensing technology have significantly expanded retail systems’ viewpoints on everyday activity as it takes place on high streets. In what follows, we discuss the role of wearable computing, cameras and computer vision, and edge computing as catalysts for these developments. Machine learning and AI are important considerations in each of these discussions, as they enable sentient systems to peer into the customer journey, as it unfolds naturally along high streets, and to reason about the “why” behind people’s behavior in the fleeting spaces and moments of context that the high street provides.

4.1. Wearables

Retail information systems are increasingly apt to draw inferences from wearable computing that many people now adopt [140]: particularly when shoppers use smart watches and fitness tracking devices to pay. Several wearable technologies also include the sensing technologies that we discussed in Section 4. A key feature of wearable computing is its focus on ego-sensing (leading to a consideration that wearables provide insight into the “quantified self” [141] of their users), including sensors designed for detecting touch and temperature, biometrics for measuring heart rate and blood oxygen, accelerometers for detecting velocity and linear acceleration, and gyroscopes for detecting orientation and angular velocity. Importantly, combinations of these readings can be used to build a range of indices of user activity and status as precursors in explaining user behavior. Other wearable devices, including headphones, object-tracking tiles, wearable cameras (Figure 6), and smart jewelry in the form of glasses and rings additionally add sensing functionality that can automatically measure users’ immediate ambient context. This includes light, sound, and the status of nearby devices (including arrays of IoT data).

Some work has been published on the use of wearables to study retailing, and aspects of this existing research have relevance to our consideration of smart retail sentience on high streets. For example, in a short paper, Vaidogaite and Sun [142] pondered potential futures for the use of wearable technology in fashion retailing. Öksüz and Maass [143] showed that it is possible to predict the stress of shoppers using accelerometer and heart rate data. El Mawass and Kanjo [144] showed a technique for building stress maps of shoppers in grocery stores, using data from a fitness tracker. Mehmood et al. [77] developed the “In-store Retail Insights on Shopper” (IRIS) framework to convert sequences of shopper micro-activity into what they termed to be “shopping episodes” within grocery stores (i.e., a series of hierarchically-considered individual interactions/transactions with the servicescape). They achieved this by indexing sequences of shopper gestures as polled from smart watches and smart phones.

4.2. Cameras and Computer Vision

The omnipresence of video cameras on streetscapes have lent considerable support to smart high streets. Video feeds are commonly deployed as closed-circuit television (CCTV) or network-accessible (Internet Protocol, or IP) streams from vistas on the outside walls, entrances/exits, and rooftops of retail stores for security purposes, usually with wide vantages of sidewalks and roadways from multiple overlapping perspectives. Similarly, CCTV and IP cameras are routinely deployed by cities and towns in streetscape infrastructure such as traffic lights and municipal building to additionally monitor traffic or to inform tolls. Increasingly, of course, pedestrian shoppers themselves capture and store huge volumes of video and image data of high street activity using the cameras on their cellphones and other portable devices, captured as they come and go from store or travel along the high street as residents, commuters, or tourists. This latter set of camera data, crowd-sourced from the throngs of people that populate high streets with their journeys, is particularly significant because the images and videos are often cross-indexed with large amounts of contextual data. This context comes from a wide degree of sources, including locations, placenames, inertial measurement readings, event data, transactions, multimedia messages, QR codes, review data, and social media tags that users actively add to video and image content as context. Importantly, much of these data are usually straightforward to cross-reference to retail indices (including as Linked Data).

A significant array of computer vision and machine learning techniques are also available to pore over camera data, often in real-time, and they have broad ability to extract meaningful information from images. A rather convincing argument arises from the perspective of computer vision using cameras: that the “why” of retail behavior is likely accessible in plain sight in many of these data streams. And, the many viewpoints that are potentially accessible from cameras create the possibility that street-side behavior might be interrogated over wide swaths of the city. Consider, for example, that computer vision and image processing include robust methods for cross-registering images from different sources, views, locations, and times so that large composite mosaics of streetscapes can be built [145,146]. This is perhaps most popularly exemplified by the use of Google’s Street View data-sets [147], which are presented as visual composites taken from successive imaging of streetscapes from (well-georeferenced) mobile vehicles [see 102 for an overview]. Significant characteristics of customer journeys could be drawn from these data. For example, Yin et al. [148] and Chen et al. [149] used Google Street View imagery to measure the volume of pedestrian footfall on streets. Hara et al. [150] accessed Google Street View to classify the accessibility of public transit for blind users, and curb ramp access for pedestrians with mobility challenges [151]. Rundle et al. [152] and Kelly et al. [153] used Google Street View composites to audit different urban neighborhoods, and Wang et al. [154] employed a similar approach to assess whether streetscapes could be deemed as “healthy” or not. Li et al. [155] relied on Google Street View to estimate building age from the appearance of streetscape façades as presented in imagery. Mooney et al. [156] used Google Street View to assess the potential environmental contributions to pedestrian injury.

Another popular source of camera-based computer vision research has developed atop archives of people’s photographs, often uploaded to cloud-based photo-sharing platforms. In some cases, these images can be used in aggregate to generate three-dimensional models of landscapes and streetscapes [157]. Using the Structure from Motion (SfM) algorithm [146], for example, Agarwal and Snavely and colleagues [158,159,160] showed how three-dimensional reconstructions of popular tourist spots in urban areas could be developed from collections of tourist photographs of those sites. Recent work in the development of Neural Radiance Fields (NeRFs) is extending this to support three-dimensional reconstruction, but also viewpoint synthesis, from images. Rematas et al. [161] demonstrated how “urban radiance fields” can be fashioned with NeRFs combined with LiDAR data to reconstruct very realistic-appearing (and geometrically accurate) models of streetscapes from around the world.

These examples are from urban science applications, but it is perhaps straightforward for the reader to imagine how they might be adapted to provide retail-specific insight on high streets, particularly to index customer behaviors to features of the outdoor servicescape, such as proximity to advertising, footfall past window displays, automated classifications of high street value platforms from visual elements of the streetscape, and so on. Indeed, significant signals of pedestrian action, interaction, and transaction can be extracted from camera imagery, often with the ability to frame the visual behaviors associated with them. This opens-up the possibility that computer vision and machine learning could be used to automatically build insight on customer behavior on high streets, where cameras have often unrestricted license to image natural scenes of everyday life. For example, computer vision has already been used to detect and label human poses [162], to classify streetscape scenes [163,164], to infer action and activity categories [165], to identify objects that people carry [166], and to track human motion [167].

4.3. Edge Computing

Traditionally, the large volume of data that has been captured in retail environments has been either stored locally for management and analysis at the site of collection, and/or it has been transferred to off-site locations, including cloud storage and computing facilities [168]. Edge computing has been suggested as an alternative scheme, designed to marshal low-power and low-cost computing to locations that are at or very close to the location of data capture, with the idea that the edge device can perform hyper-local computing, as data are being collected. In this way, edge computing leverages the automatic nature of sensing to additionally convey sense-making at a matching pace. In some instances, blisteringly quick-turnaround analyses can be performed directly on the edge device, including complex computer vision, machine learning, and AI operations. This leaves open the possibility that analyses of behavior could be performed on the edge device (which is often also at the edge of the streetscape) with return of insight at speeds that near real-time. Many edge devices rely on containerized software, i.e., small-footprint firmware and software libraries that contain slimmed-down operating systems, file systems, protocols, and analysis routines, which can be customized and pushed quickly to edge devices. Platforms such as “Docker” [169], “Kubernetes” [170], and “Singularity” [171] are popular container examples.

In many instances, the containers for edge devices are specifically designed to optimize a particular sensor modality that is connected to the edge platform, e.g., containers for computer vision that focus exclusively on pedestrian identification and action recognition on data streamed from a connected video camera [81]. Systems for Edge artificial intelligence (Edge AI) [172] are developing from these special cases. Many Edge AI solutions could work in hand-in-glove with Linked Data, with the edge device providing hyper-local semantics from their own particular vantage, but still capable of docking with more centralized information systems. For example, only salient information may need to be communicated to cloud resources, reducing cost and latency. In some instances, privacy-preserving schemes can also be embedded directly on the edge device so that the data are protected near the instance of acquisition [173].

Thus far, edge computing for retail applications has been largely proprietary to retailers (e.g., for their in-house and on-premises information systems, cloud-based computing platforms, and communications protocols), with the result that edge retailing systems and are not generally well-covered in academic literature. However, as we will discuss in Section 5.2, there are some significant research threads developing from edge AI of high streets.

5. Capabilities for Smart and Sentient Retail High Streets

In this section of the paper, we wish to explore how smart retailing is giving way to sentient retailing. In particular, we will focus on retail capabilities of geo-targeting, the wireless edge (Wi-Edge), augmented and extended reality, deep learning on streetscapes, the burgeoning idea of community as a platform, developing customer journey information systems (CJIS), new opportunities for retail advertising exchanges, context-aware intelligence, smart advertising and displays, and contactless stores. We also address the myriad privacy and ethics concerns that these capabilities foster when considered relative to studying behavior in public spaces.

5.1. Geo-Targeting

The field of geodemographics [174,175,176] was largely developed for retailing [177,178], as a method for classifying and segmenting customer and market catchment areas. Geodemographics initially grew from market analysis research [179] that was based in the geographical sciences [180], particularly from efforts to build social [181,182] and spatial [183,184] analyses of markets from census data [185], using Geographic Information Systems (GIS) [186,187]. In its essential form, geodemographics provides a set of techniques for typifying customers based on where they live. These classifications may then be used to site retail stores in particular market areas [188] and to design advertising campaigns that target desired customer profiles [189].

Traditionally, geodemographics had been relatively coarse (usually intra-urban in resolution) in their inquiry, due in part to an original focus on applied market analysis, but also to their initial reliance upon census data that were only accessible for large areal agglomerations of housing units, e.g., postal codes. However, the advent of smart technologies has established new, higher-resolution, platforms for geodemographics, with the prospect that the science can be honed to represent individual customers [190]. This creates new opportunities for advancing geodemographics from essentially static market research to more dynamic targeting, with the result that passersby might be individually influenced by advertising offers, or even that their shopping behaviors might be indexed to those of other customers in socio-spatial flows of foot traffic along streets. Generally, geodemographics intended to reach individuals are considered under the moniker of “geo-targeting”. In essence, geo-targeting uses geodemographic classifications, alongside fine-scale estimates of a customer’s location, to pass a bundled index of a customer’s likely value platform, purchase intent, lifestyle taxonomy, etc. to an advertising service or piece of software. This service can tailor content, such as product suggestions or coupons [191], that targets the would-be customer for patronage. Geo-targeting is usually carried out over m-commerce platforms and considerations, with the possibility that advertising exchanges can also be spun-up between the targeting and service delivery, i.e., very rapid online auctions among potential bidders for the customer’s attention via digital pop-up advertisements and in-application content [192]. We show an example in Figure 7, from our own work to develop geotargeting schemes that can ally signals of customer body language to very small geographies of shelf space in retail stores, using JavaScript Object Notation (JSON) metadata to translate the connection between retail space and customer body space into product information and inferred customer activity and attitude.

Two relevant extensions of geo-targeting are worth mentioning here, because of their relevance to smart and sentient retail high streets. The first is the practice of geo-fencing [193], under which targeted advertising is pushed to all amenable devices that fall within the (usually short distance) reach of a particular store. This is a component of “traffic conversion”, i.e., persuading passersby to frequent a store. Increasingly, individuals in a stream of high street foot traffic are amenable to conversion through information that is sent directly to their devices [193], often while they are also physically embodied in the retail landscape of the high street, with commensurate exposure to product displays, social influence of ambient pedestrians, signs on storefronts, etc. The second concept is geo-conquesting [194,195], in which competitors of a given store may broadcast targeted promotions to the devices of pedestrians as they near a rival store. In essence, geo-conquesting is designed to siphon foot traffic and custom from a competitor, in an attempt to drive that traffic and custom to your store instead.

Extended versions of geo-targeting can often be straightforwardly accomplished by switching the medium of touchpoints from analog to digital form. Use of digital (and often location-aware) coupons is an early example, as is the use of Quick Response (QR) codes in high street advertising displays and push-based mobile coupons [196] to accentuate location-based advertising [197] via geo-targeting [198]. Sentient geo-targeting is also increasingly possible through various forms of machine-learning designed for targeted marketing, particularly for display advertisements on e-commerce and m-commerce platforms [194,199], often on data produced from smart retail high streets. For example, Provost et al. [200] proposed the concept of the “geo-similarity network” (GSN) that would, essentially, segment potential customers based purely on the location of their devices, on the basis that similar devices that appear in like locations may be indicative of shared customer profiles. Provost et al. [200] showed that, for Online advertisement auctions, segmentation of this kind is possible from a single shared visit. The GSN is based on their earlier research to develop schemes for network-based classifications for marketing [201], as well as work by Crandall et al. [202] (among others), which showed statistical connections between social networks [203] and geographies of movement [204] through large databases of space-time patterning in social media platforms.

Work by Cezar and Raghunathan [205] has shown that geo-targeting can be extended from locations to trajectories. Similar results have been reported in simulation research by Crooks et al. [206]. The topic of trajectories is perhaps quite salient to our discussion here, if one considers the possibility that geo-targeting could be fitted to customer journeys directly, distinguishing, say, between the window-shopping phase of a customer journey and post-purchase phases to assist in delivery of promotional materials and customer loyalty content respectively. Lian et al. [198] discussed how geo-targeting can be enhanced using temporal targeting and behavioral targeting [207] (using techniques to build these insights from smart devices) to develop contextual targeting that has high degrees of “situational relevance” to the potential customer (p. 30). They demonstrated the utility of contextual targeting in the development of smart device display advertisements for restaurant retailing. The issue of building context from smart retail high streets is something that we turn to in Section 5.7.

Schweidel et al. [208] have discussed how consumers often willingly transmit signals of their behavior, using smart devices on smart high streets, and that collectively these signals present significant opportunities for studying (and monetizing) the customer journey. Customers may even do this when other users may contrarily be seeking to mask as many signals of their behavior as possible, due to concerns about their privacy. This presents something of a conundrum if we consider the original intent of geodemographics, which allows for users to be associated to other users simply due to their locations. In other words, while one set of users may seek to opt-out of geo-targeting, the geographical analysis schemes that geo-targeting invokes could possibly establish ways for retailers to identify and possibly target them anyway.

5.2. Wi-Edge

The interoperability between edge devices and new forms of communications that we discussed in Section 3, such as 5G and IoT have facilitated compound developments such as software-defined radio (SDR) that are leading to further advances in edge computing. We refer to this as “Wi-Edge”, to evoke the bundle of technologies that allow the coupled sensing and analytical capabilities of edge computing to reach out onto retail high streets where they may be brought to bear in building sentient understanding of events, phenomena, and things that are in their proximity.

In particular, Wi-Edge advances the notion of IoT beyond its traditional focus on networking, adding the ability to also perform computation (as well as communication) among “things”, and specifically supporting AI that can help those things build real-time context [209]. The data science and machine learning capabilities of Edge AI can now be broadcast in hyper-local range from edge devices. For example, Matveev et al. [210] have shown that rather sophisticated deep learning for object and feature detection on streetscape scenes can be performed on edge devices (with sufficient resolution to identify height and width data), with results communicated over IoT networks. Wi-Edge may also provide support for mobile forms of edge computing: so-termed “Mobile Edge Computing” (MEC) [211].

The use of Edge AI in autonomous vehicles is perhaps emblematic of these developments. As autonomous or semi-autonomous vehicles move along streetscapes, a range of sensors commonly sweep high street scenes [212]. For the most part, sensing in these cases is used to assist with sensorimotor control of the vehicle, or for ADAS. Increasingly, the data streams provided by vehicle sensors (e.g., RGB cameras, LiDAR, forward-looking infrared (FLIR), and ultrasound-based radar) are being considered as consumable by edge devices for rapid analysis, because vehicles require fast-term results and because most vehicles cannot host large-platform computing and data storage hardware. The result is that vehicles are, essentially, becoming omnipresent remote sensing platforms [213] at the high street edge, with very high-resolution, high-fidelity, high-frequency, and high-intelligence vistas on retail high streets. Additionally, ADAS-capable vehicles spend considerable time computing what they see: segmenting high street scenes; identifying people, things, and objects; tracking the movement of dynamic obstacles; geo-referencing them in ego-centric (car-to-thing) and allo-centric (thing-to-thing) frames; building volumetric maps of the environment form point clouds; and performing map-matching to determine their location within those maps [214]. Wi-Edge technologies such as millimeter-wave (mmWave) communications allow for relatively large amounts of data to be broadcast (~7 Gbps), which means that massive volumes of sensor data can be exchanged from edge-enabled vehicles. The communications themselves can also be used to perform novel sensing. Gu et al. [215], for example, have shown that mmWave can be deployed for the detection of people at individual scale, from the directionality, impenetrability, reflection, and scattering of the signal [215]. Spandonidis et al. [72] introduced the “ODOS2020” system, which allows for communications of street data directly to vehicles. It is conceivable that much of the smart and sentient information that we propose for retail high streets could feasibly be communicated by a similar system, perhaps in a two-way fashion.

We might note, also, that edge devices can be (and usually are) also connected to centralized computing resources and to cloud computing, with the results that those resources may also be brought to bear on edge scenarios. Okazaki and Peng [216], in their consideration of the potential impacts of 5G telephony on mobile advertising, discussed that the increased bandwidth of 5G could allow for much fine-granularity in the targeting and the delivery of advertising to smart devices while consumers traverse retail high streets. Key in doing so, they argued, is the adaptability in targeting that is made possible by 5G’s connections to real-time processing, which they represented as likely to be accomplished by AI. New technologies such as cloud-based Radio Access Networks (cloud RAN) may support this at the level of communication protocols, while new forms of radios such as cognitive radios [217] that can intelligently seek-out small swaths of radio spectrum for on-demand connections provide a level of agility that make dynamic and mobile connections between Wi-Edge and the cloud actionable. The synergies between edge computing and cloud computing, for example, are well-illustrated by federated learning [218,219]. Federated learning can assist in connecting the local AI of single edge devices, via cloud computing, into assemblies of training and learning that collectively provide comprehensive, above-local, intelligence. This could be particularly useful for computing and sentience on retail high streets, where individual sensors and caches of data may only have a partial (but privileged) view of broader scenes of pedestrian traffic and street dynamics.

Elsewhere, we have demonstrated that Wi-Edge can be used to analyze individual pedestrian actions in the context of crossing infrastructure on retail high streets, in real-time, using edge-based system on module (SOM) computing and containerized deep learning [81]. The system that we developed is capable of identifying pedestrians in video scenes of high streets, classifying their poses relative to likely shopping actions, segmenting the scene to determine streetscape features, and identifying objects (such as shopping bags) that pedestrians are carrying. All of these data can be streamed directly to a set of Key Performance Indicators (KPIs), which could potentially be components of store’s retail information systems or of community information platforms. The system works outdoors, as well as indoors (Figure 7 and Figure 8).

5.3. High Street Retail Recommender Systems

Individual stores (or franchises of stores) are usually the chief benefactor of information derived from smart and sentient retailing, in the sense that the produced insight is often designed for use by stores themselves and serves as a competitive advantage for an individual store relative to its competition. However, this isolation of benefits does not necessarily hold true for analysis of customer flow along retail high streets, where many retailers may share economies of scale, and where drawing customers to the retail area is a collective concern among different retail enterprises [220]. There are benefits for high street retailers to share information from their smart and sentient systems, and to develop communal systems [38].

Recommender systems Lu et al. [221] are a well-developed example of shared information platforms that allow different stakeholders to glean communal benefits. For example, most major taxi-hailing services and ride share systems employ street-level recommender systems commercially (which may also have linkages to retailers such as restaurants that offer delivery services). Yuan et al. [222], for example, introduced the “T-Finder” system for matching vacant taxis to passengers, built largely around taxi GPS trace data. Di Martino and Rossi [68] described a recommender system for planning multi-modal vehicle trips in a smart city framework that accounts for private trips, parking, and transfers to transit. Mishra et al. [66] and Sarker et al. [223] presented a recommender system for multi-modal transit, including rail, light rail, bus, and subway. Gavalas et al. [224] described a variety of mobile recommender systems that are used in tourism, including recommenders for points of interest (POIs) (which, we note, could be retail high streets), tourism services, social networking, and route and tour suggestions. Sasaki and Takama [225] introduced a walking route recommender system that makes use of smart city functionality to provide personalized suggestions on route amenity, safety, and walkability. Gavalas et al. [224] discussed the role of context-awareness and context inference (context of the user, as well as of the environment in which they find themselves) in recommender systems (p. 330), which is an aspect of retail high streets that we will discuss in Section 5.3.

Betzing et al. [226] introduced a field experiment in Paderborn, Germany to test a recommender system that is designed to leverage the geography of retail high streets in particular. They discussed previous efforts to build geographic recommendation systems within shopping malls. The application developed by Betzing et al. [226], instead, makes use of NFCs (see Section 3.3), specifically BLE beacons to gather spatial trajectories of consumers on a high street. These trajectories formed the ingredients for a set of clustering analysis (using DBSCAN [227]) designed to filter customers and match them to recommendations. (Their system makes use of the “Smartmarket” platform, which we will discuss in the next section). Betzing et al. [226] listed a set of hypotheses that could be tested using geo-recommendation. It is not difficult to envisage that these same hypotheses could be shaped into KPIs for high streets or for individual stores. The hypotheses include the strength of geo-recommendations in influencing customers’ propensity for visiting new retailers, as well as their likelihood of returning to previously-visited stores; correlation between geo-recommendation and duration of high street customer journeys and the number of stores visited over those journeys (ibid. p. 7); as well as influence between geo-recommenders and customers’ perception of utility of high streets, the relative attractiveness of the high street, and connections between those perceptions and visit intention (ibid. p. 8).

5.4. Community as a Platform

Recently, the idea of harnessing the communities that form on and within retail high streets as a platform for retail information systems, and perhaps other applications, has been raised. The relationship between the sorts of data that smart cities produce and evolving interpretations of the communities that can be interpreted from those data is particularly well-discussed in [228]. The notion of community as a platform extends from observations in retail science of communities (of consumers and retailers) that have developed through coupled physical ties and virtual community via social media. Advertisers, for example, use those communities to co-create customer experience in two-way channels between the information systems that stores can straightforwardly assemble to collect customer interests and reviews, and the word-of-mouth and social networks that customers may develop together. These can be tied to customer journeys via product touchpoints and servicescape branding.

The issue of how communities within information systems might convey into communities of people and establishments [229] is a topic that is worthy of significant interest. Choe et al. [230] showed that the economic fates of retailing and local communities are often intertwined. Lund et al. [231] documented how retailers’ engagement with the community can enhance their relationships with consumers. Peters and Bodkin [232] showed that community engagement is associated with consumers’ trust in store staff, their enjoyment of the shopping experience, sense of satisfaction with individual stores, and sense of commitment to particular stores. Connections to information may not be too far flung. Lyu and Kim [233], for example, detailed the correlations between psychological sense of community that younger-demographic consumers develop through connections between social media and brand interactivity.

Bartelheimer et al. [37] introduced a dedicated system to facilitate smart communities; it is connected to their associated work on mapping customer journeys on retail high streets [36,38]. They refer to the concept as “Smartmarket”, designed to instantiate community-wide platforms for smart high street retailing [37]. Smartmarket is intended to bridge connections between (centralized) retail information systems, information systems that facilitate store operations, and the smart devices (phones and tablets) of would-be customers that pass by on the high street. It is planned as a lateral network that could also potentially involve local government, forming what Bartelheimer et al. [37] refer to as a “service ecosystem” that draws different high street stakeholders into a role in which they can co-create retail value. Bartelheimer et al. [37] envisaged that Smartmarket could support shared promotional campaigns across retailers, and foster recommendation systems to be localized to specific geographies of the high street. Key in achieving this was the design of Smartmarket as a “digital multi-sided platform” [37] (p. 5), akin to the large information systems that many chain retailers have developed to manage their operations across stores, but instead focused on local ecosystems of small and medium-sized retailers within a high street community. Multi-sided platforms facilitate connections between (what are often third-party) actors, working to provide services for exchange [234]. In the context of a high street and the goal of instantiating it as a community platform, different actors (retailers, local government, chambers of commerce, advertisers, event coordinators, customers, would-be customers) could be drawn together—through the shared platform of a tangible high street [26] and the virtual platform of e-commerce applications or social networking applications [235]—to exchange recommendations and reviews, to purchase goods and services, and to engage in high street experiences. Much of the smart technology discussed in Section 3 is readily available to make various aspects of this platform realizable.

5.5. High Street Advertising Exchanges

Keegan et al. [236] described a range of recommender systems that were designed to support ubiquitous commerce (u-commerce). U-commerce is considered as facilitating an uninterrupted exchange of commerce data and interactions between seller and customer, as an extension of m-commerce (to support transactions while a user is moving) and traditional forms of e-commerce. These exchanges operate as a sort of informational marketplace in which advertisers may compete for placement on retail recommender platforms. Keegan et al. [236] introduced their own recommender system, “Easishop”, designed to collect data from would-be customers and to transmit those data to exchange systems that, in essence, enter into an auction-type framework to bid for customers’ attention or patronage. In the process, quite a lot of detailed information about customers and would-be customers is collected and organized. A similar system, “Shopper’s Eye” [237], was designed to collate data on customers’ shopping goals, location, retail preferences, and transaction histories. These data were bundled and exchanged with stores that match customer criteria, appearing as notifications that stores could then reply to with incentives and offers [236]. Shopper’s Eye would then handle the delivery of these incentives to the customer on their device. Another system in this tradition, “Impulse” [238] was designed for customers: users may enter a list of goods that they are interested in, as well as specific terms for those goods (e.g., availability of the product, price, seller review criteria, timing of purchase, and warranty details). As the would-be customer travels around different locations, this bundle of desired products and product terms is exchanged with stores that fall within the vicinity of the user’s location. The system treats retail geographies that range from individual stores, to shopping malls, and “shopping zones” [236]. On the Impulse system, merchants may participate in a background information exchange, to essentially haggle for a given customer’s patronage. Their consideration of how to bid for that patronage considers a bundle of factors, including the availability of goods in their stores and stocks, the age and shelf-life of those goods, loyalty of the customer, and likelihood of a sale [236]. Additionally, the data derived in the exchange could be used to refine how the retailer targets customers in future interactions. Easishop [236] was designed specifically for use on high streets. As with Impulse, Easishop relies on users populating a shopping list on their device. Upon moving into a high street, the shopping list of the would-be customer is passed to nearby retailers and a negotiation for that custom is brokered using software agents.

If one considers how much data smart cities and smart devices could potentially glean from customers, and the increasing sentience that those devices are capable of establishing relative to users’ behaviors, it is perhaps readily apparent that similar information exchanges could be built to cover a wide variety of retail transactions on high streets.

Migration of these functions into existing advertising exchanges is likely a next step. For example, several e-commerce platforms for examining in-content and pop-up advertising options on user devices, and bidding for those opportunities, are now available. These include “Google Ad Exchange (AdX)” [239]. These platforms are usually open to real-time bidding (RTB) [240] in auction-type frameworks [241], which raises the possibility that a wide swath of user information could be considered, beyond the content of the Online page in which the advertising is being considered for display. This information could, for example, study preferences and routine details from users (as conveyed by their devices and their history of interaction and transaction on that device), their data shadow in cyberspace and social media [242], as well as sensed information from smart retail high streets or even from the user’s device itself. Initially, these data could come from hyper-local geo-targeting (see Section 5.1). Additionally, Wi-Edge infrastructures could potentially provide the medium for conveying information in real-time (see Section 5.2). Alas, the prospect of huge amounts of very personal and personalized data being exchanged in software-mediated marketplaces, perhaps automatically gleaned and gathered as pedestrians simply walk around a high street, or that are designed to target them based on a location that they happen to occupy briefly in the course of their routine activity, opens-up a large swath of privacy and ethical concerns. We discuss these in more detail in Section 5.11.

5.6. Customer Journey Information Systems

Here, we consider a new structure for organizing data pertaining to high street retailing: the “customer journey information system” (CJIS). We consider that high street geo-targeting, retail recommendation systems, as well as advertising exchanges could feasibly be based around the structure provided by CJIS, and that smart and sentient retail high streets could supply the systems with replenishing feeds of information and context.

The customer journey framework [243,244,245] originated in customer management as a way to organize and drive customer experiences [32]. The framework has four main components. First, customers’ experiences are considered as a journey through retail servicescapes. The term servicescape is a riff on the idea of a landscape, and generally invokes a consideration of store layout, product displays, marketing and branding, and atmospherics such as lighting, music, and aroma (originally in indoor retail settings). Aspects of the servicescape may also consider staffing considerations, e.g., the placement of greeters and concierges at store entries. The servicescape can also include factors of loss prevention such as the placement of cameras, security staff, and product tagging. Particular instances of a servicescape are often referred to as service blueprints. Customers’ journeys through the retail servicescape can involve a physical journey, i.e., the path that a customer takes from entry to the facility, through shopping, checkout, and exit. Increasingly, the journey takes hold in different retail channels: beginning with e-commerce browsing, and perhaps ending with a visit to the tangible product in the store, for example. When the customer journey can shift nimbly between tangible and digital retail channels, it takes on copresence in virtual and physical form, as omnichannel retailing. Thus, the journey may also be considered as a path through a decision graph, for example in visiting different choice options in a hierarchy of products and categories. Customer journeys may be differentiated by their stage, e.g., relative to the traditional sequence of a sales funnel, the sequencing of the purchasing process, or the geography of a store. Connections between the customer and service offerings, the customer and shopping behavior typologies (window shopping as opposed to entering a store, for example), and the customer and events along the journey (contact with sales staff, accepting a flyer on a street) are considered as touchpoints. Touchpoints can be passive or active interactions and transactions, and they are usually localized to discrete bundles of space and time. Increasingly, touchpoints are being considered in diverse ways, including connections with digital technologies via non-physical retail channels (e.g., on the Web, on social media applications, in advertising media). One of the main tenets of omnichannel retailing, for example, is that customers may have multiple (overlapping) touchpoints across different retail channels.

Collectively, components of the customer journey allow for structured planning and management of retail operations. Importantly, the customer journey also provides a framework for data collection and for computing. Indeed, we would argue that the customer journey provides one of the main platforms for smartening retailing. Because large parts of the customer journey take place in phases outside stores, on retail high streets, we also contend that the customer journey can be used a framework for smart and sentient high street retailing. Berendes [38], for example, posited the question of whether the behavior of shoppers on high streets might be identifiable directly from analyses of customer journey trajectories.

In urban studies, as well as retail studies, quite a lot of research has already empirically investigated the nature of pedestrians’ and shoppers’ journeys along high streets, and has tied those journeys to retail KPIs (see [246] and [247] for a review in pedestrian studies; and [26] for an overview of the state of the art in retailing). In many cases, customer journeys can be isolated to individual pedestrians and to their potential shopping motivation and intent on the high street. For example, Babin et al. [248] used survey data to develop a measurement scale for shopping motivation, which ranged from utilitarian to hedonic in the value that consumers ascribe to shopping. Titus and Everett [249] expanded on this idea, examining how shopping value connected to consumers’ engagement in movement as they search for goods in stores. Titus and Everett [249] distinguished between what they referred to as “epistemic” retail search behavior (combing through retail environments as a necessary activity to find products that are organized by store architecture or geography, for example) and hedonic behavior (in which the act of searching is regarded as an enjoyable activity or experience). Titus and Everett [249] tied these shopping values to different forms of movement along the customer journey. Epistemic search, for example, was found to be associated with utilitarian movement to promote efficiency, such as backtracking and revisiting store areas for comparison shopping, or adhering to fixed pathways in a store layout such as product aisles (ibid. p. 112). Hedonic search, by comparison, was connected to more leisurely movement that was relatively slow and prone to frequent stops (ibid. p. 112). Berendes [38] examined the epistemic/hedonic scale and its usefulness in discerning among customer journeys. Berendes [38] found that hedonic shopping, in particular, could be associated with a range of journey types, including adventure shopping (e.g., excursions), exploration shopping (e.g., perusal), and idea shopping (motivated by fashion) (p. 315). Millonig and Gartner [250] examined similar shopping value motivations, but for consumers in high street settings. In addition to validating the utilitarian/hedonic scale proposed by Titus and Everett [249], Millonig and Gartner [250] identified three other customer journey classes that manifest on high streets: convenience (long duration and longer length journeys, with few stops at stores); discerning (relatively meticulous, seeking-out very particular shopping opportunities); and swift (goal-driven movement in search of basic and necessity-type retail products). Millonig and Gartner [250] used a combination of ethnographic research methods (shadowing and observing shoppers on high streets) and smart technologies (traces from GPS signals in shoppers’ devices) to build these classifications.

It is these sorts of insights into shopper and pedestrian behavior, gleaned from outdoor observations of activity, that we suggest as indicative of burgeoning retail high street sentience. In particular, work by Millonig and Gartner [250] gets us very close to answering “why” types questions (Section 4) from data that could be regarded as routinely accessible from smart retail technologies.

Various authors have suggested that key elements of the customer journey—path, servicescape, and touchpoints—could be used as the basis for informing existing retail information systems. For example, Lee et al. [251] examined customer journeys for indoor (shopping mall) environments, using the estimated position of shoppers’ Wi-Fi traces to build trajectories of their paths past stores and mall amenities. They concluded that customer journeys could be used within recommender frameworks, for example, to identify cohorts of like-acting shoppers, based solely on the geography of their positioning in shopping malls. Similarly, they explained that stores could develop schemes to identify complementary retailers (or rivals) using the same approach. This latter point, in essence, proposes aspects of the geo-conquesting that we discussed in Section 5.1. The Smartmarket system developed by Bartelheimer et al. [37] considered service encounters along the customer journey as an opportunity to match customer preference data with retail knowledge bases. Berendes et al. [36] made the observation that existing methods for notating customer experience (e.g., schemes such as the service blueprinting tradition of Shostack [252], or the shadowing methods used by Underhill [253,254]) are not well-equipped to deal with data from digital servicescapes [36] (p. 218), which Berendes et al. [36] consider to be a missed opportunity, given the wealth of customer journey data that are potentially available from smart retail systems and from the omnichannel (ibid. p. 219).

Berendes et al. [36] proposed a markup scheme that could facilitate the construction of customer journeys directly from digital data on high streets. This is based on their idea for a “Domain Specific Modeling Language” (DSML) that could be built from the event logs that are produced by existing retail information systems and applications [36] (p. 218). Berendes et al. [36] thus floated the idea of a “High Street Journey Modeling Language” (HSJML) for structuring customer journeys (in both e-commerce and tangible commerce) relative to high streets, specifically through journey analysis, mapping, and prediction (p. 218). However, the HSJML lacks treatment of the decisions of customers along those journeys (ibid. p. 221); rather, it frames the customer journey within the servicescape of the high street, essentially building the data structures to map customer events as touchpoints to data. The scheme also requires manual coding to match event logs to customer journeys (ibid. p. 226). Nevertheless, HSJML is very close to a knowledge discovery structure and hints at the sorts of substrate that smart retailing can provide in support of sentient retailing.

We contend that the customer journey may be useful as an information system in its own right, and that the data to build a CJIS could be produced automatically through smart sensing in retail environments. We will describe some of our existing work in sensing customer journeys and building information from machine learning in Section 5.9. In Figure 9, we show our preliminary work to build digital twins for CJIS in indoor retail settings. Our digital twins are useful in animating very high-resolution representations of shoppers in detailed store environments, where we can examine different (single) customer trajectories among varying customer demographics, shopping trips, and shopping values, while also examining those trajectories relative to configurations and characteristics of the servicescape and customers’ touchpoints. These are bundled as fused data records (exported as JSON data and alerts) that can be produced automatically, individually, and dynamically as the customer journey unfolds. The system is based on an architecture for pedestrian modeling [3,54,55,255,256] and indoor [53] and outdoor trajectory analysis [257], with the result that it could, potentially, provide complete street-and-store systems for customer journey analysis.

5.7. Context-Aware Retail Intelligence

Recent developments in context-aware computing are now being applied to smart cities, with particular usefulness, for example, in the development of smart health systems [258]. The general concept behind context-aware computing is to rely on ambience as the key channel for determining meaning in exchanges between people and the devices that they use to effect actions, interactions, and transactions. Many authors (see [259,260]) have discussed the role that location in the physical environment may play in providing context for mobile computing. We might logically consider how this could extend to high street retail information. Schmidt et al. [120], for example, have suggested that additional contextual factors beyond location (and location data) can be brought into consideration of mobility, e.g., the physical states of a user and of the environment at a given location. Indeed, Schmidt et al. [120] suggest a range of the sensing modalities that we discussed in Section 4 could provide this context, including indicators of attention and emotion (p. 894). By mapping these states to time (e.g., as a customer journey progresses), Schmidt et al. [120] suggest that contextual composites such as habits, goals, spontaneity, and group dynamics and sociality can be derived (p. 895). This shift in consideration has potentially significant implications for how one might consider sentient retail high streets if we consider, for example, that much of the sensing that might reveal user context and situational context is embedded in high streets [26,246].

In a related concept, Augusto et al. [261] introduced the idea of considering “Intelligent Environments” (IEs) as the basis for examining context. The idea for IEs shares some principles from smart environments, chiefly the dominance of autonomous systems against a backdrop of pervasive sensing, computing, and communications capabilities and the by-products of big data that they enable. However, IEs differ from smart environments, importantly, in their focus on software agents [262,263] as the main mechanism for providing intelligence (and AI). The original idea for IEs also places computer-human interaction (CHI), particularly multi-modal CHI based on natural language processing and body language, as a central interaction scheme for human use of IEs [261]. Augusto et al. [261] considered that CHI could be derived from “ubiquitous contextual information” (p. 2), which hopefully the reader can envisage could be provided by the sorts of technology that we discussed in Section 3.3, Section 3.5 and Section 4.2. For example, Augusto et al. [261] noted that software agents, acting on high-quality contextual information, might be well-poised to support adaptive responses in smart systems, with the goal of developing personalized instances of IEs for groups of users or even for individual users. Satoh [264], for example, has recently shown a working context-aware system for retailing that is based on context-sensing and agents within the IE framework. Satoh [264] designed a system for sentient advertising in indoor retail settings, in which product signs develop computational awareness of approaching users based on their proximity to displays (via RFID tags) and can spin-up instances of software agents to retrieve product information, as well as to control physical aspects of the signage (turning them on, lighting, playing audio and video). The software agents may migrate (carrying aspects of the sensed context) through the retailer’s system, to interact with related components, such as warehousing. While developed for in-store applications, it is straightforward to envisage how Satoh’s [264] system could be deployed outdoors, extending sentience to the ambient conditions around sidewalk and storefront signs.

5.8. Augmented and Extended Reality

AR makes use of both virtual reality (VR) and head-up display (HUD) to create the blended appearance of tangible and virtual contact and interaction within a user’s field of view as mediated by camera devices (in phones, in smart glasses, and in product displays) [94,265]. AR may be used in conjunction with controller devices (e.g., sensor-assisted gloves and handheld gaming controllers), through couplings to sensors in objects and devices (through inertial measurement, for example), or simple tagging of objects (using light-emitting diodes (LEDs) or readable QR codes, for example) to provide CHI between an object or device and an AR display [266]. Often, AR is coupled with location-aware technologies to create the illusion of one-to-one mappings between real physical spaces and objects and virtual representations of those things [267]. When these mappings are designed to augment larger environments (often with the goal of enhancing users’ immersion in a space such as an entire store or a streetscape) the concept is generally referred to as “extended reality” (XR). In some cases, AR/XR content can be delivered to location-aware technologies so that the AR/XR system becomes mobile.

Almost by necessity, AR/VR needs to be sentient relative to ambient conditions to function. As a user invokes their device to interrogate their surroundings and the objects that they encounter, virtual representations and annotations to those objects can be displayed. For example, holograms of entirely virtual artefacts can be inserted into the environment, and in some cases the user may be able to interrogate and annotate the object’s virtual representation. This sentient functionality has quite broad potential application to retailing. Ameen et al. [268] discussed the utility of VR in creating customer experiences within stores: by augmenting shoppers’ perception of the retail environment, VR may enhance customers’ senses of interactivity with products. Martínez-Navarro et al. [269] and Pizzi et al. [270] performed testing to establish what mechanisms of VR might be responsible for this. In both cases, they concluded that VR (across a variety of devices and formats in the testing performed by Martínez-Navarro et al. [269]) influences customers’ emotional connections to a sense of presence, and that VR environments influence brand recall. Pizzi et al. [270] concluded that users of VR retail environments reported a greater degree of sense of presence than visitors to tangible retail environments did, which Pizzi et al. [270] reasoned would translate to positive effects on value perceptions for VR users.

Using surveys, McLean and Wilson [271] examined similar considerations for AR usage in retailing, and they arrived at quite similar conclusions: if users have a positive experience when using AR technology, that positivity can transfer to the branded content beyond the experience. Yim et al. [272] examined customers’ use of an AR application for viewing jewelry and sunglasses products (with a virtual product superimposed on camera-supported video of their fingers and hands or faces in real-time). They compared AR-based shopping to traditional Web-type e-commerce and found that AR applications supported greater propensity for positive customer experience, and ultimately for purchase intention, than Web counterparts.

Javornik [273], Barhorst et al. [274], and Arghashi and Yuksel [275] attributed the relative positivity of AR customer experiences to the concept of “flow”. The notion of flow was initially discussed in the literature for Web-based e-commerce [276,277]. Javornik [273] considered its extension to AR, as the combined sense of interaction and immersion that AR supports through different channels of connection (e.g., social, product-based, technology use) with products, brands, and retail experiences that allow customers to become absorbed in the retail experience. Pedestrians’ experiences on streetscapes likely also involve aspects of flow, and there may be broad opportunity to augment their experiences on retail high streets using AR and XR. For example, de Ruyter et al. [278] discussed the potential for AR to assist in tailoring mass marketing to the unique contextual settings of customers as they are situated and embodied in their local environment [279], viewing that environment through the mediated rendering of AR. de Ruyter et al. [278] envisaged AR as an advertising tool that effectively produces “digital affordances” that can sway how customers behave in physical settings (p. 111), i.e., AR creates new mechanisms for advertisers to take the physical environment and embed it into their advertising content and campaigns. For example, de Ruyter et al. [278] referenced an advertising campaign by a global fast-food brand in Brazil that used AR to enable users to point their phone camera at a competitor’s advertisement (e.g., a poster or billboard visible from a streetscape), and upon doing so to generate a destructive animation that replaces the advertisement with a coupon for their restaurants [280]. de Ruyter et al. [278] also discussed the effectiveness of a campaign by a major beverage manufacturer to create an AR bus stop on a central high street in London, replete with animated UFOs, robots, and animals [281]. Heller et al. [282] designed a series of five experiments over tangible and e-commerce retail experiences to evaluate the possible mechanisms by which AR might influence customer behavior. They concluded that AR assists in customers’ formation of mental imagery (largely through its support for spatial presence [283] and other forms of situated cognition [284]) and their relative comfort in decision-making [285], particularly when a customer considers and evaluates the contextual use or value of retail products [282] (p. 102). Heller et al. [282] related this to the notion of “processing fluency” (p. 97), or the ability of AR to reduce cognitive load (thereby increasing decision comfort in some situations) by essentially painting, animating, and annotating context directly for the consumer, so that they do not have to build the mental imagery themselves.

In some instances, there may be opportunities to transfer functions of retail e-commerce directly to high street AR. For example, the “Monocle” project by Online review aggregator Yelp [286] allows users to essentially drop reviews and ratings of stores directly on AR-mediated streetscapes that align to the stores’ physical location on a high street. Users of Monocle that are walking down that high street can view those ratings in real-time via their camera-enabled devices as augmented annotations to their view. The broad potential for fusion between the social dynamics of real retail high streets and AR presents several possible research and development opportunities. Our own work in this area, for example, has shown that real human users react with authentic movement, time geography, and cognitive signals when engaging with virtual characters in synthetic high streets as they do in real-world contexts [287] (Figure 10). This suggests that it may be possible to build dynamics such as peer influence into AR advertising campaigns for high streets.

5.9. Deep Streetscapes

The development in sophistication of computer vision has supported the introduction of a new class of machine-learning that is designed to pore over the data produced by sensors and to (automatically) build interpretations from those data. Deep learning, for example, is a form of AI designed to approximate human sentience by mimicking the ways in which humans perceive and interpret their surroundings. Increasingly, deep learning is being used to build AI-driven analyses of streetscapes [26], with the motivation that various features of those settings could be identified, traced, contextualized, interpreted, and possibly even be explained by sentient-like computer vision. These technologies could see very widespread use for retailing, essentially enabling the lens of a camera to become a smart interpreter of activities that pass within its field of view (Figure 11).

Deep learning often uses a form of representation-learning [288]: a methodology that endows computers with the ability to interpret raw data (such as video data from retail high streets) and to automatically generate candidate representations for detecting, identifying, and classifying the content contained in those data [289]. The “deep” in deep learning comes from the use of many (relatively simple in isolation, but often complex in unison) layers of representation that work together to process the raw data toward a result. The use of these layers is learned, dynamically, rather than being set and applied a priori. This layer-and-learn approach is usually accomplished through the use of artificial neural networks (ANNs) [290], specifically through deep neural networks (DNNs) [291]. Several DNNs have been developed for general use in processing imagery and video, with the ability to build representations from views of streets and street life. These include convolutional neural networks (CNNs) [292,293], Generative Adversarial Networks (GANs) [294], Long Short Term Memory Networks (LSTMs) [295], Radial Basis Function Networks (RBFNs) [296], and Recurrent Neural Networks (RNNs) [297].

Deep learning is now widely used in retailing, both inside and outside stores. For indoor retail environments, deep learning has been applied to general product identification [298] as well as product scanning for smart checkouts [299] and smart kiosks [300]; in addition to retail planogram-product checking [301]. Deep learning is also employed for automation of customer analysis. Applications have included the use of deep learning to perform customer density mapping within stores [302], customer tracking in video [303], detection of customer gaze upon products on shelves [304,305], detection of shopper demographics and emotion recognition from facial expression [306], and even association of product detection with customer pose detection to determine customer purchase behavior [307]. The use of deep learning outdoors, on retail high streets, is less prevalent in the existing literature; however, a number of applications have been introduced. These include the deployment of deep learning to detect retail store fronts along streetscapes [308], as well as to detect and classify advertising on building façades along streets [309]. We show examples from our own work to apply deep learning to New York City streetscapes in Figure 11, Figure 12 and Figure 13.

5.10. Contactless and Frictionless Shopping

In many of the applications that we have discussed in Section 5, it is assumed that the technology for smart retail high streets might sit veiled from view, perhaps indistinguishable from the urban infrastructure itself. The introduction of contactless and frictionless stores, popularized perhaps by cashierless Amazon Go premises [310], represents a maturation of this disappearing act, in which the store itself is instrumented (largely through computer vision) to evaluate users’ customer journeys as they interact with goods, place them in their baskets, purchase them, and leave the store. Shopping, in these cases, becomes purely about the customer experience by omitting overt financial transactions. In particular, the purchase phase (which may give shoppers cause to pause and rethink their decision) is rendered relatively friction-free within the customer journey.

Contactless shopping has long existed in many other retail forms. Through the COVID-19 pandemic, for example, many shoppers elected to use hybrids of e-commerce and curbside collection of goods to minimize their contact with other shoppers [311]. This represented an extension of the long-existing drive-through model by which customers would interact with retail kiosks from their vehicle [312], but instead the retailer would designate the curbside of a high street as a pickup area or deploy the assistance of a staff member in loading purchased goods to a vehicle without directly interacting with the customer [313,314]. In these cases, retail high streets essentially serve as contactless environments for shopping [26]. Unlike delivery services, shoppers will still travel to the high street and to a store. They still have touchpoint interactions with the retail servicescape (passing by store fronts, branding, and advertising; showing a confirmation code on their phone as proof of a purchase; connecting with store information systems to announce arrival and parking location, for example), but not tangibly with store staff. This is a rather unique development, as it stands in contrast to typical consideration of the high street as a place where people would go to both shop and to physically immerse themselves among crowds of other shoppers. This was an often-necessary adjustment to pandemic conditions that emphasized reduction in human contact [315,316,317]. Verhoef et al. [318] have argued that many of these new forms of retailing are likely here to stay, while Díaz-Martín et al. [319] contended that contactless shopping may well be a new normal for retailing. Pantano and Willems [320] referred to this as the continued “phygitalization” of retailing, i.e., the convergence between Online and physical shopping. Sahinaslan et al. [321] provided a comprehensive review of the wide array of retailing steps that contactless technologies facilitate, including reducing or eliminating the need for queuing, staff assistance, interaction with cashiers, and handling of payment. Sahinaslan et al. [321] detailed a large list of potential information that retailers can collect and derive in contactless retail settings, including security, customer detection and recognition, customer localization within the store, motion and gesture detection near products, product identification and tracking, stock-taking, connection with customer campaigns, and payment details. However, work by Eriksson et al. [322] concludes that segments of customers will likely resist contactless retail due to distrust and unfamiliarity with the systems.

Based on our review of the literature, there has been relatively little academic consideration of how contactless and frictionless stores might fit within the idea of the smart and sentient streetscape or retail high street. In theory, the relatively massive sensing capabilities of “just walk in and walk out” type stores that are epitomized by the Amazon Go concept suggest that customer journeys within those IEs could be exquisitely identified, tracked, contextualized, and even modeled [32]. That such stores sit on a high street that is also smart and potentially sentient suggests that in-store customer journeys could be straightforwardly connected to high street journeys of would-be customers or of passing pedestrians. Indeed, many of the same technologies that enable contactless retailing within store premises are identical to those that researchers use to study pedestrian traffic on sidewalks. Moreover, the Online journey of customers via retail shopping applications and through to tangible store visits is already well-developed, opening-up the possibility that the entire journey of shoppers on streets outside stores might be accessible to retailers, as well as within store premises, with vistas from every angle of the retail omnichannel. Indeed, systems for indoor/outdoor positioning that make use of hand-offs between different forms of location-aware technologies (from GPS to Bluetooth to NFCs; for example, see Section 3.2, Section 3.3 and Section 3.5), provide a convenient structure for building CJIS that could unify customer journeys as a seamless customer experience that flits between the high street and stores.

5.11. New Landscapes for Public Privacy

Thus far, much of our discussion has cheered the sophistication in capabilities for retail intelligence that are possible on smart and sentient high streets. Nevertheless, we do not mean to imply that we regard this intelligence with an unbounded technological determinism [323]. Rather, as we near the conclusions of the paper, we think it necessary to highlight the broad range of privacy concerns that piggyback on the technological advances that we have discussed.

At the core of our concern is that as smart and sentient technologies leak from the indoor spaces of retail stores and out onto the streetscapes that surround them, passersby almost inevitably (and unwittingly) get caught up in their analysis and reasoning routines. Customer journeys through smart and sentient retail high streets are subject to perhaps all five of the privacy dimensions discussed by Martinez-Balleste et al. [324]: identity, owner, query, location, and footprint (p. 140). Unlike entry to a physical store, which patrons may choose to engage or not, most pedestrians that move along a streetscape cannot opt out of their journeys; nor would we reasonably expect them to. In many cities, there may be few rights or entitlements of privacy in public spaces such as sidewalks and the curbside. This stands in contrast to General Data Protection Regulation (GDPR) of data that are often generated by journeys through e-commerce, which at least in the European Union yield shoppers some rights to dictate how data on their customer journeys are captured and retained [325]. Even when pedestrians on high streets do opt-in to smart retail services, for example when using mobile shopping, they may be unaware that the application is also building insight about their ambient context and movement behavior. Customers may appreciate that their devices are smart, but may be less aware that they are sentient. Indeed, collection of supra-retail information may be one of the desired products that smart applications seek to collect, as part of what Pal and Crowcroft [326] have termed “surveillance capitalism” and what Elnahla and Neilson [327] have referred to as “retaillance” (a portmanteau of retailing and surveillance). Thus, issues of the growing connections between retail intelligence and surveillance of public spaces and public behavior form part of a long-standing broader conversation about the surveillance capabilities of smart cities more generally (see, for example, [328,329,330,331]).

In response to these privacy concerns, a suite of research is developing to explore how privacy can be baked (by design) directly into smart cities generally, and much of that work has some bearing on our discussion of sentience on retail high streets, where we now may additionally need to consider how privacy factors into AI. The use of keys and tokens [332] have been widely suggested as ways to privatize information that is generated by LBS. Other schemes have proposed what are, essentially, third-party brokering services that mediate user queries and responses in ways that preserve privacy in the exchange of location-based information [333]. Privacy masking schemes [334] for journey paths that also preserve the geography of those journeys [335,336] have been explored in Geographic Information Science. Recent developments in network protocols that are privacy-preserving seek to address the issue at the communications layer of smart cities [337]. Jiang et al. [338] have shown that these can be woven into edge computing, for example. Fitwi et al. [339] have also proposed that privacy could be baked into machine-learning schemes on video, as a mechanism to identify data that infringes on privacy, which could then feasibly be subjected to masking. The applications of protocols and privacy masking as privacy tools operate essentially by obfuscating the data that smart systems produce. As discussed in [26], however, retailers would likely have little incentive to obfuscate the data that they collect when they can realize significant value from collecting individual detail, and when the matching of that detail to the customer journey is an important part of delivering customer experiences. Perhaps, then, the most steadfast approach to privacy concerns is to raise awareness among high street shoppers of the existence of surveillance systems and their capabilities, and the potential that those systems create for function creep beyond the most immediately obvious terms of service. Consider, for example, that retailers have been shown already to be highly sensitive to customer opinions and perceptions of their data privacy [325,340,341,342].

6. Conclusions

In this paper, we examined developments in retail technologies through the lens of the developing concept of the smart and sentient high street. Our main conclusion is that retailing and sentience are increasingly considered as mutually-reinforcing at hyper-local scales of the city, often down to the resolution of individual people and their behavior along the customer journey. Several technologies that have been honed for smart city applications have found their way into retail use (and vice-versa), where they are perhaps rapidly also becoming sentient. Notably, the technologies that have steadily converted indoor retail environments into smart and sentient information spaces are beginning to leak into the streetscapes and social environments that surround stores.

We introduced the notion of a smart and sentient retail high street, which we argue is currently coming into focus at the intersection of a range of related developments in data science, computer science, electrical and computer engineering, behavioral science, design, retail science, and marketing. Critically, pedestrians in urban settings increasingly pass through and over smart and sentient retail high streets as they engage in routine and quotidian activities. A range of rather sophisticated retail systems have developed to study would-be shoppers from among this pedestrian traffic, with the ability to identify and target their demographics, as well as to infer their shopping intent and susceptibility to retail operations. The reach of these analyses can touch individual characteristics and behaviors of would-be shoppers, which potentially offers tremendous new insight into how, where, why, when, and with whom people shop. Nevertheless, this insight comes with quite serious potential erosion of privacy for pedestrians in public spaces.

We discussed the technologies that are making connections between retail intelligence and smart city functions possible. These include Linked Data, wireless communications, NFCs, location-aware technologies, and the very wide range of sensing modalities that are available to peer on high streets. We also reviewed the suite of sentient capabilities that are being developed on this backbone, especially cameras and the computer vision and AI schemes that may be run atop them, as well as new forms of edge computing and Wi-Edge.

Throughout the paper, we introduced some of our own research efforts to investigate and reveal the emerging character of smart and sentient retail high streets. These include detecting and mapping the otherwise invisible geography of wireless communications that pulse through high streets, with the capability of revealing details of individual users and groups of users that rely on wireless communications for everyday tasks as well as for e-commerce and m-commerce. We also discussed our research to examine what details of individual pedestrians and shoppers can be revealed by high street sensors. This includes details of individuals—their demographics, their body language, and their activities—from RGB+D sensors, as well as very high-resolution volumetric mapping of entire high street scenes that are now feasibly and automatically derived from LiDAR. Would-be shoppers increasingly use smart and sentient devices of their own, including wearable computer technology such as smart watches and cameras, which we showed can reveal a huge amount of detail regarding their activity. Fusion between different sensing technologies and the underlying communications capabilities of smart high streets is increasingly possible, and we discussed our ideas for CJIS and Wi-Edge as a dedicated infrastructure that brings advances in computing and AI directly to the high street, where they can be trained on everyday scenes of retail activity in public spaces. Our own work, for example, has revealed how Wi-Edge can be used to perform customer identification and action recognition in real-time, by customer segmentation, location tracking, pose detection, context inference, and labelling. Our preliminary work has shown that the application of this technology is mutable between indoor and outdoor environments. This opens up the possibility that adaptive CJIS can be built. We showed such a system for very high-resolution geographies of shopping, with the capability of allying store shelf geography, products on those shelves, and the body language of would-be patrons that pass by the shelf infrastructure, with contextual detail provided by their physical interactions with merchandise as touchpoints, as well as the fine space-time details of their movement and trajectories through store aisles and brand spaces. We are continuing to examine these ideas in store environments and on high streets, using observation, as well as simulation. Our initial work has revealed, for example, that connections between observations, models, and what-if simulation are also possible, through VR, AR, and XR. As a result, high resolution details of servicescape configurations for smart retail high streets can be experimented with in synthetic realities ahead of deployment in tangible realities, often as digital twins of high streets and the people that populate them.

Increasingly, a suite of machine-learning systems, based on Edge AI, can be marshalled to sense and to “make sense” of retail high streets, automatically as a by-product of their smart functioning. As a result, mappings from planned servicescapes or would-be customer journeys to the realities of physical customers and high street spaces become open to investigation and management in sophisticated new ways. We showed several examples of how this is possible through straightforward visual imagery captured on cameras, run through deep learning to automatically perform pedestrian detection and tracking, pose detection, object detection, dynamic mapping of streetscape and high street features, and classification of entire retail high street scenes in real-time. Increasingly, three-dimensional information regarding these scenes is interpretable from multiple views of two-dimensional imagery, with the result that machine-learned data could straightforwardly be transferred into virtual geographic environments [343].

A new set of capabilities for examining pedestrians as shoppers and would-be customers on high streets, before and after they connect with retail experiences, is now broadly feasible. Our central argument, in this paper, is that many of these capabilities can be understood within the customer journey framework developed in retail science. Ideas about the customer journey and the retail omnichannel that it supports were initially developed for in-store retailing, but are now relatively easily extended outdoors to the high street. In particular, this expansion of the customer journey, made possible with the support of smart retail high street technology, presents new opportunities for sentient geo-targeting at unprecedentedly high resolutions of insight. Consideration of customer journeys on high streets could also inform the development of automated AI schemes on the evolving Wi-Edge. There are adjacent commercial value platforms that are perhaps straightforwardly realized on Wi-Edge, including new forms of high street recommender systems and new inputs to high street advertising exchanges. Moreover, the development of sophisticated CJIS and context-aware retail intelligence could greatly inform the behavioral underpinnings of retailing. If we consider the broad potential for high-bandwidth and low-latency communications and computation that Wi-Edge supports, it is perhaps realistic to imagine foundations for future AR and XR systems that could support high street retailing and extend support for contactless and frictionless shopping. Similarly, we might feasibly consider that a natural extension of the emergence of deep streetscapes could be the spillage of the contactless and frictionless retailing concept beyond stores, into high streets outside.

This all sounds promising for revealing new retail insight and growing customer experiences, but it is also rather alarming in unveiling the now myriad ways in which smart and sentient retail high streets could automatically and tirelessly make judgements about shoppers and would-be shoppers that move through high street settings. Around all of these capabilities, a new landscape for public privacy is emerging, one in which any high street pedestrian could be identified, scanned, sorted, classified, and measured in a bewildering new suite of ways that are quite unprecedented (and relatively unregulated). The potential diminution of public privacy on the smart and sentient retail high street could therefore be an important focus for research and scholarship as technologies and applications develop. Indeed, there are broad opportunities to examine how customer privacy could be featured as a central design consideration for smart and sentient retail high streets.

Retail privacy issues echo those already being considered by researchers of smart cities [328,344], although it is perhaps worth concluding by restating that, unlike many municipal systems in smart cities, retailing is largely cloistered to the private and proprietary interests of stores and their holding companies. Including shoppers’ input into the design and application of smart and sentient retail high street systems is perhaps more daunting a task than might otherwise be the case for municipal systems. Nevertheless, significant developments in open and transparent systems, based around the concept of community platforms, are underway and we have discussed several such approaches in this paper. Also, we might reiterate that retailers are highly sensitive to shoppers’ opinions, with the implication that would-be shoppers’ discomfort about retail intelligence technologies also then features as an important consideration in stores’ design of the customer experience and the customer journey. Raising awareness of how and where retail high streets are becoming smart and increasingly sentient, and what AI capabilities are available while shoppers pass through public spaces, could be significant in letting customers know what retailers’ views on their activity and intent might be, so that retailers and customers can co-create future customer journey information systems that work for both parties.