Skyfall: Signal Fusion from a Smartphone Falling from the Stratosphere

Abstract

A smartphone plummeted from a stratospheric height of 36 km, providing a near-real-time record of its rapid descent and ground impact. An app recorded and streamed useful internal multi-sensor data at high sample rates. Signal fusion with external and internal sensor systems permitted a more detailed reconstruction of the Skyfall chronology, including its descent speed, rotation rate, and impact deceleration. Our results reinforce the potential of smartphones as an agile and versatile geophysical data collection system for environmental and disaster monitoring IoT applications. We discuss mobile environmental sensing capabilities and present a flexible data model to record and stream signals of interest. The Skyfall case study can be used as a guide to smartphone signal processing methods that are transportable to other hardware platforms and operating systems.

1. Introduction

The rapidly expanding capabilities of sensor systems in smartphones makes them viable Internet of Things (IoT) platforms [1] and has fueled their adoption as data collection systems. These mobile commercial off-the-shelf (COTS) cyber-physical systems [2] are well suited for constructing low size, weight, power, and cost (low SWaP-C) distributed sensor networks [3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]. Early studies [3] concentrated on possibilities, whereas more recent studies focus on specific use cases. Some applications center on biosignals [4,5,6,7], improvements in geolocation [8,9,10], and educational or citizen science [11,12,13,14,15]. Our work contributes to the growing environmental and disaster monitoring IoT [1] at regional and global scales [16,17,18,19,20,21,22,23,24,25]. All of these efforts have been aided by the rapid increase in processor speeds and dynamic range of on-board sensors, which have improved signal quality and reduced signal latency.

One of the key disaster monitoring IoT applications is explosion monitoring. The basic concept of operation is that smartphones with optimized settings could use their internal sensors to provide useful information about the environmental impact of an explosive blast [20]. We present smartphones as an efficient technical solution for exploratory studies and to provide additional information for observational gaps. A variety of infrasound and geoacoustic monitoring scenarios have been evaluated since 2014 [20], with successful collection and analysis of explosive signals varying from small-yield surface detonations (e.g., [21]) to the 2022 multi-megaton eruption blast from the Hunga Tonga volcano [22].

This paper reports the results of an experiment designed to record surface explosions from high-altitude balloons. This is a viable concept of operation for remote environmental monitoring on Earth and Venus (e.g., [24]). The balloon flew up to ~36 km into the stratosphere, about 3.6 times the nominal 10 km cruising altitude of commercial aircraft. Despite the ambient low pressure and subzero temperatures, the smartphone acoustic and acceleration sensors picked up the explosion signal [23], and all sensors recorded the rapid, turbulent descent and ground impact. The very survivability of a smartphone operating and falling from stratospheric heights is noteworthy and a testament to the present-day ruggedness of these platforms. We present the first published smartphone measurements in the stratosphere; the first smartphone observations of the dynamics of a sensor system falling through the stratosphere and troposphere and impacting the ground; and the first airborne smartphone multimodal sensor fusion and validation with external geospatial and barometric sensor systems. At the time of writing, the explosion signal data are embargoed. However, the data from the balloon’s descent through the stratosphere and troposphere (the Skyfall), and its impact with the ground, are openly available and have much higher signal diversity.

Although the key application is disaster (e.g., explosion) monitoring IoT, one of our main motivations is educational. Various students, engineers, and researchers have requested practical advice on how to access, process, and integrate smartphone multi-sensor data. This paper addresses this knowledge gap by describing the Skyfall stage in detail and providing open-source methods to perform multi-sensor data fusion to reconstruct the descent timeline. Our intended audience are researchers, engineers, early-career, and citizen scientists interested in a hands-on guide to the present data-gathering capabilities of smartphones and the resulting types of data structures. Due to the breadth of the intended audience, we have made substantial efforts to make this paper accessible. The next section provides a summary of the systems, sensors, and signals that define this work. Section 3 presents the materials and methods available to the reader to reproduce our results. Section 4 discusses the signal fusion process in some detail, where the combination of the information inferred from the different sensor signals allows the reconstruction of a cohesive Skyfall narrative. Section 5 summarizes the results in Section 4 and provides some recommendations on how to approach and interpret multimodal signals. The concluding remarks place this work in the context of emerging IoT information systems.

When arriving at Section 4, or if jumping to the Section 5 summary, we invite the reader to imagine being an observer—perchance part of a disaster rescue team—in a comparable airborne platform the moment it lost lift and propulsion. A pre-configured smartphone in your pocket would have provided near-real-time situational awareness to support teams, sent a prompt recovery location, and permitted a detailed reconstruction of the descent and impact.

2. Summary of Systems, Sensors, and Signals

On 27 October 2020, a Samsung Galaxy S10 [26] smartphone was hoisted by a helium-filled mylar balloon over southern Nevada to capture blast signatures from a surface chemical explosion [23]. This S10 model has been extensively tested and validated operationally in nuclear reactor monitoring studies (e.g., [25]). The S10 smartphone collected geophysical and state-of-health information using multiple on-board sensors. Additional location and barometric pressure information was provided by lightweight external sensors specifically designed for high-altitude balloon deployment and payload recovery. The balloon reached an altitude of 36.3 km before the payload was released; its descent through the stratosphere and troposphere was slowed by a parachute. Although the landing was abrupt—as expected from the non-steerable parachute canopy—the smartphone was recovered intact and has been deployed in subsequent missions.

We focus on the environmental and geophysical signals recorded by the smartphone during its rapid descent until its ground impact ~30 min later. The smartphone was running the Android version of the RedVox application available through the Google Play Store [27] (an Apple version is also available through the App Store [28]). Signals from the surface explosion are explicitly excluded in this work as they are part of separate studies [21,23].

The choice of smartphones as our low SWaP-C system platform is worth a brief note. We presently experiment with single-board computers; after adding a wireless communication system, various external sensors, tuning the analogue-to-digital parameters, adding a touch screen for navigation, and calibrating and bench testing the system, we end up with a bulky stand-alone digital data acquisition system with a price point comparable to a smartphone. In contrast, anybody with a reasonably modern smartphone can download an app and start recording useful environmental data.

Some of the advantages of the smartphone platforms are their ubiquity, accessibility, and immediacy. However, the types of sensors and processors on board a smartphone vary widely and evolve rapidly. All smartphones have a microphone [29] and most of them use micro-electromechanical systems (MEMS) [30,31]. Original equipment manufacturers (OEMs) often obscure the specifications and configurations of microphones to protect their supply lines and retain their competitive advantage. With tens of thousands of smartphone models available in the wild, standardizing calibrations is a challenging task. The microphones are designed for the recording and transmission of speech, which requires an enhanced response across the 350–4800 Hz passband.

Wagner and Fick [30] described the pressure reciprocity calibration of MEMS microphones commonly used in smartphones across 100–10,000 Hz. Their measurements showed the MEMS microphone responses to be within manufacturer specifications. Brown and Evans [29] measured 1/3-octave sound pressure levels and reverberation time with iPhone applications (apps) and compared them to a Brüel & Kjaer (B&K) Hand Held Sound Level Meter Type 2250. This study found that iPhone and B&K sound pressure levels were typically within 5 dB of each other, and that iPhone internal microphones had a limited dynamic and frequency range; namely, the iPhone internal microphone had a poor response below 200 Hz due to high noise floor. Further literature describing the frequency magnitude response of smartphone microphones is limited.

Examination of various smartphone sound apps by Kardous and Shaw [32] found some of them accurate and reliable in occupational noise measurements. In follow-up studies [33,34], the accuracy of these apps when used with two different external calibrated microphones was examined. The authors showed that the accuracy and precision of smartphone measurements can be significantly enhanced with the use of external microphones. Extension of calibration methods to the infrasound range were performed by Asmar [35,36] on an ensemble of Samsung S8 smartphones.

Model-by-model calibration of smartphone sensor systems is a daunting task. By the time a study is completed and published (e.g., [35,36]), new models, apps, and operating system updates can render the results obsolete. It is a Sisyphean task to defend a given model’s sensor stack; we instead focus on the signals that are produced by the sensor systems and develop robust data models and transportable signal processing methods that can be implemented across diverse platforms.

Even without precise calibrations, smartphones can produce useful data. The embedded accelerometers and Global Navigation Satellite System (GNSS) receivers of smartphones have proven useful in earthquake monitoring studies. Minson et al. [16] confirmed the capability of smartphones in detecting displacements from moderate and larger earthquakes, and proposed the use of crowdsourcing with smartphone data to achieve Earthquake Early Warning Systems. The MyShake app [17,18] developed for the collection and analysis of earthquake data has matured and integrated into operational early warning systems [19].

Shortly after the infrasound app concept was proposed and developed [20] to extend the smartphone microphones to the infrasound range, it was applied to volcano monitoring [37], near-shore underwater and surf observations [37,38], and nuclear reactor monitoring [25,39,40]. Extension to the audio range and the inclusion of additional sensor channels allowed improved surface explosion characterization [21,41,42], as well as the collection of moving aircraft [43], rocket [44,45], and space craft [46] signatures in microphone, barometer, accelerometer, and gyroscope sensors. The 2022 Tonga eruption pressure signal was clearly observed by smartphone barometers in Hawaii and elsewhere at distances of thousands of kilometers [22], opening new possibilities for global monitoring applications.

Digital data collections in high-altitude balloons are relatively new and are considered viable platforms for the exploration of Venus (e.g., [24,47]), where the ground is too hot for surface deployments. High-altitude balloons present an extreme environment outside the design parameters of most consumer electronics, and rapid technical improvements in recent years [23,24,47,48,49] have made it possible to observe a wider range of signals.

In the next section we present the geophysical and environmental data recorded by the RedVox Android application on a S10 smartphone during Skyfall. As mentioned in the Introduction, one of our educational goals is to demonstrate how to access and process the heterogeneous multimodal signals typically available in a smartphone. To this end, raw data and Python open-source software are provided in [50,51,52,53,54,55,56,57].

The raw data are provided as an event report in the RedVox Cloud interface at redvox.io [50]. The methods and Python code used to produce the figures are provided as a GitHub repository [51]. The RedVox app’s Application Programming Interface (API) is available online as an open repository [52]. The Python Software Development Kit (SDK) for the RedVox app is also openly available [53] to facilitate the loading and inspection of the raw data. The open-source Redpandas framework [54] builds on the Pandas [55] serialized data structures and is used to process the Skyfall smartphone data. Standardized time-frequency representations use a quantized Gabor atom framework [56] and incorporate a multiresolution weighted wavelet z-transform [57]. Various digital signal processing algorithms and geophysical methods are documented in these open-source repositories, and the interested reader is invited to look into the functions imported to process the Skyfall data. With the raw data [50] and open-source software [51,52,53,54,55,56,57], a reader with minimal Python experience will be able to reproduce the results and use the paper as a guide.

Since smartphones and their kindred cyber-physical platforms are rapidly evolving, the data are a snapshot of the technology’s capabilities at the end of the year 2020. The RedVox SDK converts API 900 fields used in 2020 to the current API 1000 (API M) data protocol released in June 2021 [52]. At the time of writing, data collection and transmission under API M has been implemented in Android, iOS, Linux, and Mac OS operating systems on both Intel and ARM platforms.

3. Materials and Methods

We present signals collected during the Skyfall plunge, starting at a height of ~36 km a few seconds before the initiation of descent and ending on the ground. Although human-readable local time is more readily accessible in daily life, our computations use Unix time (also referred to as the epoch time) UTC, the number of seconds since 1 January 1970 relative to the Greenwich meridian. Although our highest sample rate was only 8 kHz, present-day smartphone audio systems are able to record at 192 kHz, corresponding to a sampling interval of 5.2 microseconds. Throughout this work, the native unit of time for computations is represented as the epoch time in microseconds.

The tables in this section are intended to highlight the most important systems specifications and API fields needed to verify the observations and results.

3.1. Balloon Platform

The flight system consisted of a 4 kg Totex [58] weather balloon carrying a 2.25 kg instrumentation package (Figure 1). The flight line was a 10 m paracord tether from the balloon nozzle to a Balloon Ascent Technologies HAB Bounder Balloon Cut-Down Device [59]. The Bounder was geofenced to automatically terminate the flight if the balloon approached the Las Vegas metro area. The cut-down device was connected to a fishing swivel tied to the top loop of a 1.21 m Rocketman parachute [60]. The descent rate for the 1.21 m standard parachute at 2.3 kg is specified at 6 m/s (~21 km/h). The fishing swivel allows the payload to rotate independently of the balloon, which tends to prevent violent spinning during ascent. A High-Altitude Science Stratotrack APRS (Automated Packet Reporting System) [61] transmitter beneath the parachute was intended to provide one-minute balloon location and ambient temperature readings during flight.

The key elements of the payload box were a PolarTech insulated shipping carton containing a SPOT Trace [62] asset tracker to report the landing location and a Samsung Galaxy S10 [26] in a protective case taped screen-down to the bottom of the box. The dimensions of the payload box were 38.1 cm long, 33 cm wide, and 25.4 cm high; the rectangular shape of the payload enclosure may have helped stabilize the descent, as the vertical axis remained relatively stable until impact. The payload box was sealed with duct tape prior to launch, but a hole in the side allowed a venting tube to pass air between the exterior and interior of the box.

There were four independent instruments for estimating location; however, two of the four instruments only provided sparse estimates intended for recovery. The APRS and SPOT Trace were primarily intended to provide redundancy for the payload recovery. Only the Bounder and the S10 smartphone had sufficiently high sample rates to reconstruct the details of the rapid descent trajectory, and both of these instruments had a barometer to provide secondary height estimates. The Bounder relies exclusively on the GNSS for position, whereas the S10 uses both GNSS and cellular location when available. Only the S10 used a cellular network to transmit data in real time when an adequate signal from a cellular network was available.

The balloon was inflated in a hangar at Desert Rock Airport, then launched at 12:16 UTC on 27 October 2020 into a brisk wind blowing to the south/southwest. The APRS tracker provided the balloon’s location every minute until it became too cold during the early dawn ascent. It resumed reporting positions once the sun had risen above the horizon and the tracker had risen high enough in the stratosphere for ambient temperatures to come back into range.

The balloon burst at 13:47:50 UTC at an altitude of 36,296 m, landing at 14:13:25 UTC at an elevation of 1038 m. Landing was first indicated by the RedVox S10, which reported that it was no longer moving and that the ambient temperature was rising as the payload box warmed up after landfall. The SPOT Trace then confirmed the landing site. The payload was recovered at 17:00 UTC on the same day. The payload box was upside down, and large amounts of balloon material were tangled on the flight line and the parachute (Figure 2). This suggests that the parachute may not have deployed properly and that the payload may have descended faster than planned. Although the box was externally scraped during the hard landing, the S10 was undamaged and has been redeployed in subsequent missions.

3.2. Bounder

We use the Balloon Ascent Technologies HAB Bounder^TM Balloon Cut-Down Device [59] as a primary reference for the balloon trajectory. Unlike smartphones, the Bounder is designed for high-altitude balloons (HAB) so its barometer specifications are better matched to the 36 km balloon height. Furthermore, the Bounder provides a continuous record, whereas the smartphone location has a gap at the higher altitudes.

The Bounder exports balloon GNSS location and barometric pressure data collected at a rate of ~1 Hz to a text file. Per the documentation sheet, the pressure rating on the Bounder barometer is 8 to 1200 hPa (0.8–120 kPa). The Bounder trajectory fields and observations for the event of interest are presented in

Table 1. Bounder Event Information.

Description	Value
Start Date Time	2020-10-27 13:45:00
Start Epoch s	1603806300
Start Latitude degrees	36.07616
Start Longitude degrees	−115.62339
Start Altitude m (WGS-84)	35538
Stop Date Time	2020-10-27 14:16:00
Start Epoch s	1603808160
Stop Latitude degrees	35.83728
Stop Longitude degrees	−115.57234

and

Table 2. Bounder Location Sensor.

Description	Field	Units
Pres	Atmospheric pressure	hPa = mbar
Lon /Lat	GNSS longitude and latitude	degree decimal
Alt	GNSS altitude above ellipsoid	meters
Date	GNSS date	YYYYMMDD
Time	GNSS time	HH:MM:SS
Sample Interval	1.004, Derived from Time	seconds
Interval St Dev	0.061, Derived from Time	seconds

and shown in Figure 3; note the rapid descent after the peak height shown to the right of the red arrow in Figure 3. The Bounder’s peak recorded elevation at 13:47:50 GMT was 36,296 m above the WGS84 ellipsoid, and the balloon was ascending at the moment of the payload release. The Bounder data are provided in the Skyfall repository [51]. The values in the tables are representative of the original data precision and are not intended to represent the sensor accuracy.

3.3. Smartphone

All modern-day COTS smartphones come loaded with a suite of advanced low SWaP-C sensors. All mobile phones contain a microphone (mic) for voice communication. Starting with flip phones and continuing with smartphones, cameras shortly followed, along with accelerometers, gyroscopes, barometers, and other more esoteric sensor modalities. All operating systems (OSs) have a native application programming interface (API) to access data from internal sensors. An API is a formal definition of inputs and outputs for a given computational system; ideally, it provides a contract and documentation to a developer for how to interact with a given system. For example, the Google Android Platform API [63] describes what fields or methods are available, the data and types of data that can be accepted, and the data and types of data that can be returned. However, many OEMs interpret the Android API as recommendations and deviate from the ideal contract. The Google Android API provides access to all the possible elements in a smartphone. This is too much information for most applications. In contrast, the RedVox API [52] is much lighter and is used to organize and transport selected smartphone data.

RedVox APIs are tailored for collecting and standardizing a range of heterogeneous sensor data on diverse OSs. Google’s Protocol Buffers [64] provide a common serialization format and allow data to be accessed from multiple programming languages and environments. We store both device-specific metadata as well as user-specified sensor data. This includes sensors with stable, (mostly) evenly spaced sampling rates such as audio, sensors with unevenly spaced sampling rates, single channel sensors, and multi-channel sensors. Metadata includes station identification information, state-of-health metrics, and a summary of the station’s hardware and configuration. Enhanced metadata includes location, timing, timing correction information, and edge-computed event streams.

Data are quantized into individual files called packets. The duration of a packet is determined by the requested number of audio samples stored in the packet for a given sampling rate. Packets are compressed for both transport and storage using the LZ4 compression algorithm. The packets are always stored on the device and are also streamed to the RedVox Cloud Client [65] hosted by Amazon Web Services (AWS) if either cellular or Wi-Fi communications are available. Data packets collected during the Skyfall were transmitted over the cellular network in flight and after landing.

Associated with the API is a software developer kit (SDK) [53] which facilitates access to the data fields. The purpose of the SDK is to provide a wide range of tools to acquire, process, and analyze the data. The SDK functions on a variety of configurations of hardware and software, and should produce a consistent result when run across each of the configurations.

The components of the SDK critical to Skyfall are the Station and DataWindow objects. A Station object represents a single device. Each Station is capable of holding a variety of data from various sensors (e.g., audio, barometer, location, accelerometer, gyroscope, and magnetometer). A DataWindow object is a grouping of Station objects with a defined start and end datetime and is created by loading and processing the raw data stored in the packets [50]. DataWindow is intended for export to more traditional geophysical data formats such as miniseed or SAC, or converted to other data structures (i.e., a Pandas DataFrame) for analysis by data scientists.

The high-level specifications returned by the SDK from the S10 station are provided in

Table 3. Smartphone Station Specifications.

Description	Field	Value
Station ID	station.id()	1637610021
Make	station.metadata().make	samsung
Model	station.metadata().model	SM-G973U1
OS	station.metadata().os	ANDROID
OS Version	station.metadata().os_version	10
App Version	station.metadata().app_version	2.6.20
SDK Version	datawindow.sdk_version	3.0.0rc37

. There are many possible field names in the SDK and its hierarchical structure can be initially confusing. In this table and the ones that follow, the full SDK field name is provided so that the reader can readily reproduce the results.

At the time of writing, the Android RedVox application has over 10,000 downloads and is supported by >10,000 different Android smartphone and tablet models running OS7 Nougat and higher, not including Android virtual machines.

For the purposes of the Skyfall event, there are three defining timescales of importance: the start of the clock (station start date), the start of the event, and the end of the event (

Table 4. Station and Event Date.

Description	Field	Epoch s	Human UTC
Station Start Date 1	station.start_date()	1603719281	2020-10-26 13:34:41
Event Start Date	station.first_data_timestamp()	1603806314	2020-10-27 13:45:14
Event End Date	station.last_data_timestamp()	1603808114	2020-10-27 14:15:14

¹ Not available before v.2.6.3.

). Although all timestamp precisions are in microseconds, only the corresponding epoch times in seconds are shown in Table 4. The station clock resets every time the app is stopped and started, and the station recording parameters can only be changed if the app recording is stopped. Therefore, it is possible to identify a station and its data collection settings during a recording session by its Station ID (Table 3) and its Station Start Date (Table 4).

3.3.1. Microphone

Since sound quality is a key competitive feature between makes and models, OEMs are protective of their microphone specifications and configurations. It is neither possible nor practical to characterize the response of the >10,000 different models of smartphones in the market, although some traditional methods have been pursued [31,32]. We work with the raw digital, uncalibrated output and concentrate on the signal-to-noise ratio (SNR) of the observed signal. The nominal and observed sample rate of the primary internal front-facing smartphone microphone are presented in

Table 5. Audio Sensor.

Description	Field	Value
Sensor Name	station.audio_sensor().name	I/INTERNAL MIC
Nominal Rate Hz	station.audio_sample_rate_nominal_hz()	8000.0
Sample Rate Hz	station.audio_sensor().sample_rate_hz()	8000.043426
Sample Interval s	station.audio_sensor().sample_interval_s()	0.000124999
Interval Dev s	station.audio_sensor().sample_interval_std_s()	0.000000052

. The microphone is one of the few internal sensors that are designed to be evenly sampled. When initiating a recording session, we request a nominal sample rate. However, the sample rate is locked to the smartphone’s crystal oscillator that is used for timing, which can drift with time. We compute the effective sample rate from the data window duration. The difference between the nominal and effective sample rate is small, and correctable. The float64 precision of the SDK is presented in the tables as a representative value for the selected data window so the reader can easily reproduce and compare the results.

3.3.2. Barometer

The specifications for the station barometer obtained through the Android framework are listed in

Table 6. Barometer Sensor.

Description	Field	Value
Sensor Name	station.barometer_sensor().name	LPS22HH Barometer
Sample Rate Hz	station.barometer_sensor().sample_rate_hz()	24.40309971
Sample Interval s	station.barometer_sensor().sample_interval_s()	0.040978401
Interval Dev s	station.barometer_sensor().sample_interval_std_s()	0.006281194

. The barometer, accelerometer, gyroscope, and magnetometer sensors are sampled unevenly. The sampling strategy is to poll the sensor for changes, and only record if there is change. The primary metric is the sample interval between data points, which is averaged over the Skyfall data window to obtain a mean sample interval with its standard deviation. This can then be converted to a mean sample rate in Hz (Table 6). The barometer magnitude at sea level should have a static steady-state value of ~101 kPa. The expected static DC and AC offsets of the various on-board sensors can serve as a first-order quality check for a sensor’s state of health.

The pressure range of STMicroelectronics LPS22HH [66] is 260-1260 hPa (26–126 kPa), in contrast to the 0.8–120 kPa Bounder range [59]. The LPS22HH accuracy is 0.05 kPa, with a low-pressure sensor noise of 0.65 Pa. The digital output is specified as 24 bit and its sensitivity as 4096 LSB/hPa.

It is known that the height above the ellipsoid estimated from GNSS constellations has relatively large errors, and we can use the barometer as a secondary estimate for height. We expect the Bounder barometer to perform much better at high elevation as it can go down to 0.8 kPa. Barometric pressure is transitory and depends strongly on the specific time and location. Since the Bounder provides independent measures of height and pressure, we can construct an empirical function to compute height from pressure for the balloon height profile. Since pressure is expected to drop as the exponential of height, we performed a polynomial fit to the natural logarithm of scaled pressure. The method for the polynomial fit is provided in the redvox-pandas repository [54] as the function bounder_model_height_from_pressure.

Figure 4 compares the Bounder and S10 barometer data, showing a very good match below 20 km but divergence above 20 km heights. We postulate the Bounder barometer will be the most accurate of the two at high altitudes (>20 km) as its low-pressure specifications are better matched to the ~36 km height of this case study. Note the sample rate of the Bounder is ~1 Hz, whereas the sample rate of the S10 barometer is ~24 Hz.

3.3.3. Accelerometer and Gyroscope

The Accelerometer and Gyroscope specifications obtained through the Android framework under the RedVox API are listed in

Table 7. Accelerometer Sensor.

Description	Field	Value
Sensor Name	station.accelerometer_sensor().name	LSM6DSO Accelerometer
Sample Rate Hz	station.accelerometer_sensor().sample_rate_hz()	401.0575129
Sample Interval s	station.accelerometer_sensor().sample_interval_s()	0.002493408
Interval Dev s	station.accelerometer_sensor().sample_interval_std_s()	0.001007224

and

Table 8. Gyroscope Sensor.

Description	Field	Value
Sensor Name	station.gyroscope_sensor().name	LSM6DSO Gyroscope
Sample Rate Hz	station.gyroscope_sensor().sample_rate_hz()	401.057526
Sample Interval s	station.gyroscope_sensor().sample_interval_s()	0.002493408
Interval Dev s	station.gyroscope_sensor().sample_interval_std_s()	0.000992285

. These sensors are in synch and are routinely used in tandem in gaming applications to correct or compensate for each other. Although their average sample interval over the data window (and their sample rates in Hz) are identical, their standard deviation is out by ~10 microseconds. This suggests the cumulative accuracy of the timing and methods for these six data channels may be in the order of a hundredth of a millisecond. The final accuracy will be dependent on the make, model, crystal oscillator, and sensor data polling latency, and results will vary. We present the native float64 precision of the measurements so that we can evaluate if hardware and software hardware upgrades and updates can improve accuracy.

The accelerometer magnitude has a static steady-state value of 9.8 m/s², corresponding to Earth’s gravity. In contrast, a static gyroscope should have no DC offset.

The documentation sheet for the ST Microelectronics LSM6DSO [67] 3D always-on, low-power accelerometer and gyroscope is thorough. This component has many possible configurations; the specific S10 configuration parameters are not saved under API 900, and a high level of detail is not available under the Android API and framework. In other words, some sensor configurations and settings are not transparent or readily recoverable, and will be at the discretion of the OEM.

3.3.4. Magnetometer

The magnetometer specifications obtained through the Android framework are listed in

Table 9. Magnetometer Sensor.

Description	Field	Value
Sensor Name	station.magnetometer_sensor().name	AK09918 Magnetometer
Sample Rate Hz	station.magnetometer_sensor().sample_rate_hz()	100.1370283
Sample Interval s	station.magnetometer_sensor().sample_interval_s()	0.009986316
Interval Dev s	station.magnetometer_sensor().sample_interval_std_s()	0.001235985

. The magnetometer is generally used for orientation relative to Earth’s magnetic pole, which has a steady-state magnitude of up to ~65 microTeslas.

The specification sheet for the Asahi Kasei AK09918 3-axis Electronic Compass [68] provides a nominal sensitivity of 0.15 microTesla/LSB and states the digital output is 16-bit per axis. The magnetometer can also measure rotation relative to the magnetic field vector at a given location. In the Results section, we will fuse the gyroscope and magnetometer channels to improve the reconstruction of the Skyfall chronology.

3.3.5. Station State of Health Information

The state of health (SOH) information supported by the API 900 framework that we used for Skyfall is minimal, and consists of the station internal temperature and remaining battery percentage. This information is useful for maintaining station uptime, as sustained temperatures below 0 °C and exceeding 40 °C impact mobile devices’ batteries and battery-charging capabilities. The device’s battery status and temperature data are collected once per packet, which is 32.8 s at 8000 Hz audio sample rate (2¹⁸ points per packet). Although useful, the coarse temporal granularity of the SOH information under API 900 is marginal for SWaP-C optimization.

To address this deficiency, API M expanded the SOH recording capabilities to include network type, network strength, data transmission rate, temperature, cell service state, battery remaining, battery current, available RAM, available disk, CPU utilization, power state, and Wi-Fi wake lock state, when available, at a rate of up to once per second. Not all hardware and operating systems make these sensors accessible.

3.4. Station Timing, Orientation, and Location

When, where, and how a recording station is deployed is captured by various sensors, and with a high level of redundancy. In the simplest scenario, the station is not moving and connected to a stable communication network. For Skyfall, the station only stops moving after the payload hits the ground, rolls over, and is dragged by the parachute.

3.4.1. Timing

There are various timelines available to a smartphone. When first booted, a smartphone internal clock does not know the time and would by default utilize the Unix epoch time zero of 1 January 1970 00:00 UTC. It will try to find the time via GNSS, cell, or Wi-Fi. The operating system time (OS time) will use any and all available combinations to update its internal clock. OS time will jump back and forth as needed to keep up with the various clocks at its disposal. GNSS time will also jump about depending on which satellites are available. A third timeline, the machine time, is initialized when the unit is booted. Although the machine time is monotonic, it is only as accurate as its internal oscillator. Accurately estimating true time is beyond the scope of this work; our aim is to obtain an estimate of timing errors for a given network configuration relative to a reference clock. In Skyfall, the reference clock is at Amazon Web Services and is only available when there is either cellular or Wi-Fi connectivity.

The application uses well-established tri-message exchange algorithms [69,70] for synchronization and relies on three assumptions: that the message propagation delay is nearly symmetrical, that errors of the timestamp itself have been mostly compensated, and that the clock’s skew is negligible during the message exchange. The synchronization process starts when the reference server sends a message of its current timestamp to the device. The device then records the time of receipt and, after a delay, returns the message with its current timestamp. The server records the time of receipt, completing the synchronization message exchange. Using the timestamps, we can find the average latency to calculate the offset between the clocks. The best-fit offset is selected by finding the best minimal statistically significant latency to compute the offset.

Timing correction is built into the RedVox SDK, and is presently only possible when communication is available to either a reference synchronization clock or GNSS satellites during data collection. Potential timing improvements (e.g., [71]) are under consideration.

3.4.2. Station Orientation

The S10 coordinate frame follows the Android and iOS smartphone-centered cartesian XYZ axis, with Z positive away from the screen (Figure 5). A smartphone accelerometer laying screen up on a flat, level surface has a constant value of −9.8 m/s² in the Z-axis. If the long horizontal axis (Y) is pointing towards the north, the magnetometer X-axis should be zero and the Z-axis will be the projection of the magnetic line on the vertical. The gyroscope rotation should report zeros in repose, and will respond to rotation around the specified axis, positive counterclockwise (right-hand rule), and shown by rotation arrows in Figure 5.

3.4.3. Smartphone Location

As with timing, a smartphone will attempt to determine its location via GNSS, cell, or Wi-Fi. Skyfall represents an extreme high-altitude collection event, and the location framework did not perform well above 20 km height (twice aircraft cruising altitudes) as there is a data gap. The Skyfall API900 did not collect sufficient location information to disambiguate between cell, Wi-Fi, or GNSS location.

Table 10. Smartphone Location Sensor.

Description	Field	Value
Sensor Name	station.location_sensor().name	GNSS
Sample Rate Hz	station.location_sensor().sample_rate_hz()	0.709595563
Sample Interval s	station.location_sensor().sample_interval_s()	1.409253456
Interval Dev s	station.location_sensor().sample_interval_std_s()	12.22592513

provides some representative values averaged over the Skyfall data window. The Sensor Name is the generic older name; the actual values are derived from both GNSS and cell. This omission is corrected under API M.

The location data that was obtained from the S10 matched the Bounder location (Figure 6) within the vertical scale of descent (km). As shown in Figure 7, synchronization via cell did not begin until the S10 dropped below 10 km. The spike just before 800 s probably corresponds to communications from a different cell tower. This suggests the height measurements above 8 km were from GNSS.

As previously noted, data streams and backfills (last in, first out) as soon as there is a communication link available. Figure 7 shows some high-latency connections above 10km; not much data would be successfully transmitted during those brief links. Low-latency cell communication was established below 8 km and data began to stream to the RedVox Cloud. The S10 was the first positioning system to report its final resting location, and all the valuable experimental data had been saved to the Cloud before the payload was recovered. The in-flight backfill capability is particularly valuable for balloon deployments, as landing zones can be unpredictable and it is not always possible to recover instrumentation in a timely manner.

4. Results

Each sensor modality provides useful and complementary information on the descent chronology, revealing their capabilities and limitations. Fused together, the signal ensemble permits a cohesive reconstruction of the Skyfall. Without detailed information, all we know is that the balloon payload fell from 36 km high and was slowed down by a parachute before it hit the ground. From the S10 signals, we learn it descended in a fast flat spin, slowed down when it reached the troposphere, and impacted the ground with a sudden (but non-deadly) deceleration before it flipped over. In the introduction we invited the reader to imagine being an observer in an airborne platform the moment it lost lift and propulsion. The S10 would have provided near-real-time situational awareness to recovery teams during the critical ground approach and sent a prompt landing location with enough information to estimate a survival probability.

This section has some redundancy in the narrative to highlight how each individual signal modality reinforces the others to provide a higher level of confidence in the Skyfall chronology. A more abstract and concise interpretation of the results is provided in Section 5, with some recommendations for interpreting multi-sensor signals.

4.1. Bounder Location and Estimated Speed

The Bounder platform provided the most complete record of the payload position, although it does not contain information about its orientation or its rate of rotation. The Bounder position is converted to cartesian East North Up (ENU) XYZ coordinates in Figure 8 to show its trajectory with time relative to the end-of-flight terminus. A scalar speed is computed from the derivative of the cartesian coordinates in Figure 8 and is shown in Figure 9. The speed direction follows the trajectory. The balloon was initially moving briskly (~50 m/s) towards the southeast in the upper stratosphere (above 25 km), but swerved to the southwest in the troposphere (below 10 km). The fastest descent was at the onset of Skyfall, when it hit a top speed of ~80 m/s. It slowed down as the parachute deployed in the thin, cold stratosphere. The effects of the parachute seem clear in the lower stratosphere when it slowed down to ~20m/s. When it reached the troposphere, it was struck by a shearing wind that reversed its eastward journey. Surprisingly, it accelerated again in the upper troposphere to ~40 m/s; whether this was due to a parachute malfunction or strong winds is unclear from the Bounder location record.

4.2. Smartphone Location and Estimated Speed

The S10 smartphone location failed to provide location data in the upper stratosphere (Figure 10). This may be due to a phone’s tendency to turn off location logging when the screen goes off unless the “Always On” setting is enforced; this deficiency has been addressed in later app versions. However, vigorous movement tends to wake up a smartphone. The S10 began to record its location in the lower stratosphere, after its maximum descent speed of ~80m/s. Although the available S10 location and speed values are comparable to the Bounder, the S10 location data gap at height remains an unexpected surprise. As inferred from the synchronization data, cell position was only available below ~8 km. In summary, below 20 km and up to ground impact, the S10 location framework performed adequately (but not exceptionally) well, and it was the first sensor to report its position upon landing. Both the Bounder and the S10 show two regions of slow, near-vertical descent in the lower stratosphere (~10–20 km) where horizontal winds were low and the parachute performed as specified. As shown in Figure 10 with the higher-resolution smartphone data, the payload descent accelerated to ~40m/s in the troposphere, suggesting the parachute malfunctioned when it was hit by shearing winds. Fortunately, the parachute recovered, stabilized, and reached its nominal 6 m/s descent rate in the lower troposphere just before landfall, when it mattered most.

4.3. SOH Power and Temperature

The station power and temperature were coarsely sampled at intervals of ~30s and are shown in Figure 11. Not surprisingly, the internal battery was discharging during data collection. The S10 internal temperature during Skyfall did not rise above −4 °C; the external temperature would have been substantially lower. It is likely the station would have shut down and/or drained power rapidly had such low temperatures persisted. The temperature drop during descent is surprising; it should have been coldest at height. Advective cooling may have played a role. The reported internal temperature of the station dropped below −8 °C at ~8 km height after falling at relatively high speeds (>20 m/s). Temperature begins to rise again in the lower troposphere when the descent speed slows down to ~20m/s and, more predictably, continues to rise after ground impact.

4.4. Acoustic Pressure, Barometric Pressure, and Vertical Acceleration

The microphone, barometer, and vertical accelerometer sensors provide useful insight into Skyfall. These three channels are often complementary during ground-based measurements and are discussed together. The audio channel, raw vertical acceleration, and height inferred from the barometer data are presented in Figure 12.

As mentioned in Section 3.3.2, the Bounder barometer was designed for high altitudes and provided independent measures of height and pressure. Since barometric pressure is transitory and depends strongly on time and location, we constructed an empirical function to compute height from pressure for the balloon height profile, and performed a polynomial fit to the natural logarithm of scaled pressure. The method for the polynomial fit is provided in the redvox-pandas repository [54]. We then fused the Bounder and S10 barometer pressure data (Figure 4) to obtain a finer temporal resolution for the pressure and height time series to reduce temporal uncertainty. The height is labeled as Bar Z Height (km) in Figure 11 and was later sampled at 24.4 samples/second, in contrast to the Bounder pressure, which was sampled at 1 sample/second. The substantial decrease in the sample intervals reduces timing and position uncertainty.

The raw accelerometer and microphone time series are dominated by the shock of the ground impact at the end of the record. The microphone has no substantial DC offset, as expected, and there is little audio signal during the initial release. Acoustic flow noise increased during descent, with a burst of increased noise when it reached the tropopause (~10 km) halfway through the record (~900 s), and ground impact evident at ~1700 s. Although the initial experiment was designed for audio exploration and succeeded in its primary mission [23], the acoustics are less prominent during Skyfall. It is not because there are no signals, but rather because their interpretation will be nuanced and beyond the scope of this paper. For example, there are interesting pressure perturbation around the time of tropospheric penetration (~950s into the record) that could be the topic of a separate study. Since the microphone data are sampled at 8 kHz, it can be converted to an audio file and code is provided in the Skyfall repository to perform this function [51]. It is a long (~30 min) listening session of audio noise, but with careful audio editing it is possible to hear whistling in some segments, supporting the advective cooling hypothesis postulated in Section 4.3.

The vertical acceleration signal in Figure 12 shows that the S10 was lying screen down, with gravity pointing up relative to the screen (Figure 5) before release (+9.8 m/s²). The release was a turbulent event with some tumbling, as suggested by the oscillations and reversals of sign in the vertical acceleration. At the terminus, the payload enclosure flipped over, reversing the sign of gravity (−9.8 m/s²). The microphone (mic) channel also clearly shows the impact.

High-pass filtering the signal above 100 Hz (Figure 13) enhances the fine-scale variability in the barometer and vertical acceleration sensors. The sudden stop at the end produced an overpressure of ~400 Pa in the barometer and, unsurprisingly, clipped the microphone.

Although the deceleration was abrupt, it was less than 10 times gravity (10G), and within the lower tolerance level survivable by humans [72]. The most informative signals for the Skyfall impact stage are the accelerometer channels when fused with the position and the height inferred from the barometer, as they pin down and verify the impact time and location. The accelerometer reports a rough but typically non-lethal landing acceleration magnitude of <8G, although the reported tumble after landing could be injurious.

4.5. Three-Component Sensors: Accelerometer, Gyroscope, and Magnetometer

The three-component (3C) sensors provide insight into changes in the orientation of the payload. Accelerometers are included in almost all smartphones to detect when the device is picked up and for gaming apps. Gyroscopes are often bundled in with accelerometers to correct for a smartphone’s rotational orientation relative to gravity (e.g., screen orientation) and magnetometers are used for navigation. All the 3C channels are sampled unevenly, and all show three clear stages of descent: the release of the payload with a freefall in the low-density stratosphere, the transition to the denser atmosphere in the tropopause at ~900 s, and the impact with the ground. The accelerometer suggests the phone was deployed screen down and remained so through most of the turbulent descent. All (X–Y–Z) channels in Figure 14 show sign changes on release, consistent with an initial tumble during the initial stages of descent. The horizontal (X–Y) channels show high accelerations during the stratospheric descent and stabilized penetration into the troposphere. Since the DC offset of vertical acceleration was relatively stable at +9.8 m/s² during most of Skyfall, we can describe the rotation about the Z-axis as a flat spin.

All three gyroscope channels in Figure 15 indicate a stable payload while tethered to the balloon with the initiation of rapid rotation after release. This reinforces the inferences made from the accelerometer channels that the payload initially tumbled after release. The gyration about the horizontal X and Y axes (panels (c) and (b) in the figure, respectively) along the base of the payload box are dominated by fluctuations around the value of 0 rad/s during the descent, while the Z channel (panel (a)) appears to show continuous and rapid rotation of the device about the Z-axis from approximately 300 to 900 s into the descent through the stratosphere. The vertical components of the gyroscope and accelerometer support the description of the primary rotation motion as a flat spin. The maximum rotation rate is 13.43 radians/second (2.14 rotations/s) or ~128 revolutions per minute (RPMs) sustained for ~100 s. A lower but more sustained rotation of ~10 radians/second (~95 RPMs) lasted for ~200 s.

After the 900 s mark (after entry into the troposphere), the gyroscope readings drop and fluctuate about 0 rad/s. This is also consistent with the accelerometer channels. All channels spike at ground impact, with the polarity changes also indicating a tumble.

Data collected by the magnetometer along the three axes of the device are shown in Figure 16. The magnetometer is also a good rotation sensor. A slow rotation about the Z-axis is evident in the X–Y components before release while tethered to the balloon, while the magnetic line component along the Z-axis is stable. All magnetometer channels switched polarity on initial release: the payload tumbled. In the X and Y directions (shown in panels (b) and (c), respectively), readings oscillate around 0 µT during descent. For both the X and Y channels, the frequency is markedly higher from about 350 to 950 s into the descent, slowing down thereafter. The amplitudes, however, are similar throughout the descent for both the X and Y channels. The data in the Z direction are shown in panel (a). Unlike those of the other channels, the readings in the Z direction do not oscillate around the value of 0 µT, but instead remain near −5 µT after the initial moments of the descent, indicating that the compass direction of the Z-axis remained relatively steady during the rapid descent, flipping only upon impact as inferred from the vertical accelerometer channel.

High-passed filtered channels are shown in Figure 17, with their corresponding estimated short-term Fourier transforms in Figure 18. The waveform plot amplitudes are normalized relative to the maximum values. The three main stages of the Skyfall are clear in all channels: initiation, transition to troposphere, and ground impact. The accelerometer, gyroscope, and magnetometer provide a cumulative nine channels of data and can be integrated with the high-resolution height estimates from the barometer to provide a cohesive Skyfall narrative with a high level of confidence. The signal fusion reveals that throughout the flight and descent, the payload was in a flat spin along the vertical axis, with different rates of rotation at different stages: slow before descent, frantic (>10 rad/s) in the stratosphere, and slowing down in the thermosphere. The payload only tumbled twice: at the onset of Skyfall and after ground impact. The impact was abrupt but relatively moderate at <10G due to the parachute reaching stability before landfall. All channels show an interesting transition as the S10 enters the troposphere, where it is hit by strong shear winds and where the parachute may have malfunctioned. Although it would be interesting to zoom into the tropopause penetration stage around the 950s timeline, it will substantially increase the length of the manuscript to no clear advantage. The reader is welcome to explore the data and tools at their disposal.

The tonal structure in the spectra in the rotation-sensitive channels (Figure 18) is a measure of the flat-spin rotation frequency, and there is no evidence for tumbling outside the initial and terminal states. The spectra also support the narrative that the payload tumbles only briefly after release and after impact.

The gyroscope spectra reinforce the time-domain representation, as expected. More interestingly, the X and Y components of the magnetometer spectrograms confirm the ~2 Hz maximum revolution rate of the payload in the stratosphere obtained from the gyroscope. The magnetometer was sampled at ~100 Hz, with a Nyquist frequency at 50 Hz. The base-2 logarithm (bits) [56] of the spectral magnitude of the X component of the magnetometer is shown in Figure 19. The peak spectral magnitude envelope matches the vertical component of the gyroscope in Figure 15, with a peak revolution rate of ~ 2 Hz. The flat spin loses its speed and stability after 950s when it enters the troposphere, as evidenced by the smearing of the spectral energy to lower frequencies. It would be possible to further decrease the uncertainties in rotation speeds with signal fusion in the frequency domain if desired. For example, standardized multiresolution time-frequency representations [56] would increase the temporal resolution of the lower frequency bands and permit direct comparison between signal modalities.

Raw data and Python code are provided to reconstruct the results in this paper, as well as produce additional results not presented herein. The reader is welcome to explore these openly available data and software to search within the various intriguing spectral patterns for each channel.

5. Skyfall Summary

Fusion of the Bounder and smartphone signals permitted the reconstruction of a cohesive Skyfall chronology. At the beginning of the record, the balloon is ascending through the stratosphere with the payload slowly spinning about its vertical axis. The Skyfall event begins with the dropping of the payload and substantial movement and rotation along all axes. After the initial tumble, the payload settles into a flat spin. The fastest descent occurred during the initial freefall through the thin stratosphere, where the parachute has little to work with. The second Skyfall stage corresponds to penetration into the troposphere and a reduction in rotational motion of the payload enclosure. The Skyfall terminates with impact on the ground, with high transient amplitudes on all channels. The final minutes of the record show that the payload enclosure flipped upside down and remained static, awaiting recovery and redeployment.

In the design and interpretation of multi-sensor experiments, it is useful to consider separately the absolute and differential responses of the sensor systems. The absolute response can provide useful, complementary information of a station position relative to Earth’s natural coordinate frame; e.g., gravity (up-down), magnetic north (left-right), and pressure (height). The absolute response can be readily used to verify sensor performance before and after deployment, and it grounds the signals to the physical world. The differential response yields useful information about rates of change, and strongly depends on the signal’s sampling rate. Interpretation of differential signals requires more knowledge of digital signal processing and filter implementation. For example, if a station is not moving, the absolute response of a sensor could be removed by detrending. However, if a station is moving, it is preferrable to high-pass filter the signal, with special care taken near the edge of the record. Changes in the absolute (DC) value of the signal will appear as transients, which should be interpreted with care. Rotation rate can be inferred from accelerometer, magnetometer, and gyroscopes, and should provide complementary results in the time and frequency (Fourier) domains. Vertical oscillations would be observable in accelerometers, barometers, and, if sufficiently fast, microphones. Persistent, stable fluctuations in differential (high-passed) signals may be clearer in the frequency domain. These useful signal processing methods are provided as examples in the Skyfall repository [51].

6. Concluding Remarks

The multi-sensor data models we presented are intended to facilitate collection and engagement from professionals and citizen scientists that wish to use and develop smartphone IoT capabilities. This paper shows how to use an SDK to access API fields and process the heterogeneous multimodal signals typically available in a smartphone. Raw data and Python open-source software are available in [50,51,52,53,54,55,56,57] to reproduce the results.

This first-of-kind smartphone deployment in a high-altitude balloon and parachute to the ground provided useful and verifiable signals [23] that can be used for explosion monitoring IoT. This is the first scientific study of an extreme smartphone data collection and drop test from stratospheric altitudes, and it presents smartphones as viable IoT platforms for the collection of geophysical and environmental data.

There is an abundance of fine structure within the Skyfall records and their spectra that can be explored by the interested reader, as all data and software are openly available. Ongoing and future research will use these frameworks and methods to study signals of interest on Earth and beyond.