Multimodal Analysis of Gravitational Wave Signals and Gamma-Ray Bursts from Binary Neutron Star Mergers

Abstract

A major boost in the understanding of the universe was given by the revelation of the first coalescence event of two neutron stars (GW170817) and the observation of the same event across the entire electromagnetic spectrum. With third-generation gravitational wave detectors and the new astronomical facilities, we expect many multi-messenger events of the same type. We anticipate the need to analyse the data provided to us by such events not only to fulfil the requirements of real-time analysis, but also in order to decipher the event in its entirety through the information emitted in the different messengers using machine learning. We propose a change in the paradigm in the way we do multi-messenger astronomy, simultaneously using the complete information generated by violent phenomena in the Universe. What we propose is the application of a multimodal machine learning approach to characterize these events.

1. Introduction

The detection of gravitational waves (GWs) from the inspiral phase and coalescence of a pair of neutron stars (NS) on 17 August 2017 [1] and the following observations of the event in its electromagnetic (EM) counterparts (see [2] and references therein) marked the beginning of multi-messenger astronomy with GWs.

For the first time, we observed the coalescence of two NSs through GWs and EM radiation across the entire electromagnetic spectrum, thanks to the participation of more than 70 astronomical observatories to the EM follow-up campaign. Multi-messenger astronomy opens up not only new scenarios for the observation of the Universe and new perspectives for the investigations of astronomical objects, but also new challenges for the way of extracting all information that these astrophysical events bring with them. The synergy between the information that only GWs can provide and the concomitant observations through other detectors of the EM and neutrino counterparts can strongly accelerate our knowledge of the Universe. It is clear that multi-messenger astronomy discloses the need for new paradigms for data analysis and introduces new challenges for real-time analysis, and there are many efforts ongoing to face them that involve the use of machine learning techniques (see, e.g., [3,4,5,6,7,8,9,10,11,12]). Multimodal machine learning MMML analysis is efficiently applied in many fields of data analysis for the more inclusive interpretation of events where several modalities are concurrent, such as in a video with audio; images with captions; or images, text, and sound [13]. To our knowledge, these techniques have never been applied in the interpretation of astrophysical data, where signals of different nature can be almost simultaneous. In this paper we introduce, for the first time, a multimodal machine learning analysis applied to astrophysical transient signals such as the case of GWs and gamma-ray bursts (GRBs).

In Section 2, we will describe the importance of multi-messenger astronomy; in Section 3, we will introduce the multimodal analysis and some examples of application; in Section 4, we will describe how we can implement the multimodal analysis for multi-messenger events; and in Section 5, we will report a proof of concept for the the application of MMML to GW and GRB data. In the conclusion, we will discuss other possible applications.

2. Multi-Messenger Observations as a Powerful Tool to Investigate the Extreme Universe

The joint observation of GW170817 and its EM counterparts clearly demonstrated the enormous informative power of multi-messenger astronomy with GWs. For instance, the joint detection of GW170817 and GRB 170817A represents the first direct proof that NS-NS mergers are progenitors of short GRBs [14]. In addition to this, the observed time delay between the GW and the gamma-ray signal (∼1.7 s) allowed us to put constraints on the difference between the speed of gravity and the speed of light, which has been estimated to be between −3 $\times {10}^{-15}$ and $+7\times {10}^{-16}$ times the speed of light [14]. After the joint detection of GW170817 and GRB 170817A and the release of a three-detector, a well-constrained GW skymap has been key for the identification of other EM counterparts [2]: this allowed us to obtain more insights into the physics of the source. For instance, the multi-wavelength EM observations associated with GW170817 allowed us to infer some basic properties of short GRB jets. The temporal evolution of the X-ray [15] and radio [16] light curves, together with the very low gamma-ray luminosity of GRB 170817A, suggested two possible scenarios: an off-axis GRB with a relativistic, structured jet or a “cocoon” emission from the relativistic jet shocking its surrounding non-relativistic material [14]. Subsequent very long baseline interferometry observations have been crucial to discriminate between these two scenarios: they allowed astrophysicists to put constraints on the size of the source and on its displacement, which were found to be consistent with a structured, relativistic jet [17,18].

The detection of the optical/NIR counterpart to GW170817 (AT2017gf), first reported by [19] and later by other teams (see [2] and references therein), allowed for the first time to identify the host galaxy of a GW event and led to the first spectroscopic identification of a kilonova [20,21], thus expanding our knowledge of heavy element nucleosynthesis in the Universe. The joint GW and kilonova observation also allowed us to investigate in more detail the neutron star equation of state (EOS). For instance, [22] found the lower bound on the tidal deformability parameter through the interpretation of the UV/optical/IR counterpart of GW170817 with kilonova models, combined with new numerical relativity results; by combining this result with the constraints obtained with GW data alone, they have shown that both extremely stiff and soft EOSs are tentatively ruled out. In addition, Ref. [23] presented a Bayesian parameter estimation combining information from GW170817, GRB170817A, and AT2017gfo, and with this analysis, they were able to obtain multi-messenger constraints on the EOS and on the binary properties. More recently, [24] performed a joint analysis of GW170817, GRB170817A, and AT2017gfo, together with GW190425, and they combined these with previous measurements of pulsars using X-ray and radio observations, as well as and nuclear-theory computations, to put constraints on the EOS. A deeper knowledge of the EOS is also fundamental to understand which is the outcome of coalescing binary systems and therefore to constrain the short GRB central engine (see, e.g., [25]).

Finally, the estimate of the luminosity distance with GWs, together with the estimate of the redshift obtained from the host galaxy, allowed us to estimate the Hubble constant with a totally new approach, independent from previous measurements [26]. Joint observations of GW and EM are thus a remarkable instrument for unveiling the physics of some of the most extreme phenomena in the Universe.

3. Artificial Intelligence via Multimodal Inputs

Multimodal machine learning (MML) is a multidisciplinary research area that addresses some of the main objectives of artificial intelligence (AI) by incorporating and creating models that can process and link information from multiple modal inputs, differing in the representation (text, images, etc.), dimensions (1D, 2D, etc.), and input data sources. By considering data from multiple modalities, it is possible to take into account the complementary information among them, which in turn leads to more robust predictions that reflect patterns not available when working with individual modalities.

The multimodal approach is already used in a wide variety of artificial intelligence problems, such as use for virtual assistance, image captioning, question answering, etc. One must consider the advantages in obtaining multiple input data characterizing the same event, as well as features extracted in different domain decompositions. In this way, we have the ability to capture otherwise unidentifiable details [13]. In Figure 1, we reported a schematic view of the general idea underlying the multimodal analysis. The input samples contain different kinds of signals and representations which can be encoded in a deep network analysis and concatenated, and the later stage can be used for classification/regression analysis.

Machine learning (ML) and deep learning (DL) techniques, which have changed the way data are processed in recent years, have already been implemented in the gravitational wave astronomy community [27], thanks to the computational resources available to us in recent years and the implementation of algorithms that allow the effective use of graphical processing units (GPUs). Recently, DL has even been successfully applied to multimodal machine learning problems, with the aim of learning useful joint representations in data fusion applications [28]. In [29], we introduced a first approach to MML, based on the merging of deep learning pipeline outputs on two different kinds of inputs: time series and images in the scheme, which is called late fusion multimodal [30].

The challenge will also be to apply these techniques to data from different instruments with different outputs that characterize a multi-messenger event. We want to show how MML could be used to process a 1D strain as well as 2D spectrograms from GW detectors with sparse light-curves collected by astronomical telescopes in order to infer astrophysical information from the common sources. At the same time, even the ability to caption GW data with an associated GRB event could help in quickly identifying source parameters.

4. From Multi-Messenger Observations to Multimodal Analysis

In the next years, second-generation GW interferometers (Advanced LIGO [31], Advanced Virgo [32], and KAGRA [33,34]) will take data with increased sensitivity, and third-generation GW detectors (such as the Einstein Telescope [35]) will become operative; furthermore, many new telescopes will start taking data (e.g., CTA [36] and LSST [37]). We therefore expect an increase in the data rates and in the data complexity. In order to maximize the scientific return of future multi-messenger observations, there is a need to develop new approaches to analyse large streams of EM, GW, and neutrino data, taking into account the differences in instrument sensitivities, spatial and temporal coverage, data formats, etc.; furthermore, novel tools are needed to combine information in an efficient way from multi-messenger observations, allowing us to infer the properties of the astrophysical sources and their environment. For instance, joint GW and EM observations can be used to put more stringent constraints on the tidal deformability parameter (Λ) and therefore on the EOS of NS (see, e.g., [22,38]; see also Section 2). To do this, we need to perform a GW parameter estimation with an accurate family of template waveforms, as well as a detailed comparison between the EM observations and the existing kilonova theoretical models; information obtained from the two messengers should then be combined in a consistent way to put constraints on the EOS. Such constraints, together with the estimates of the masses of the two NSs as obtained with GWs, could also help us to obtain insight into the outcome of the coalescence (that can be an NS or a BH) [24]. In the case of a coincident short GRB observation, the additional characterization of the X-ray afterglow emission will allow us to eventually probe the GRB-magnetar model, based on which magnetars are the GRB central engine; such a model has revealed itself very successful in reproducing the observed properties for the sub-class of short GRBs showing an X-ray plateau [39] and/or an extended emission [40,41], but a direct proof of this connection is still missing.

In this work, we propose a new paradigm for analysing the multi-messenger data we will collect with next generation instruments.

In Figure 2, we sketched an example of multimodal analysis for multi-messenger events. We can approach the information of the diverse messengers through a dedicated pipeline, representing them in the best format for feature extraction. We can analyse these representations using the best suited machine learning workflow to maximize the capability of prediction and, at a final stage, combine the output of the extracted features in a multimodal model. We are depicting here a future vision, where, in an open access environment, we can analyse shared data using shared software on cloud systems [42].

5. Application to Astrophysical Sources: The Case of Binaries of Compact Objects

We decided to test our idea on a set of simulated short GRB light curves and associated GW events, with a focus on NS-NS mergers. Specifically, we divided this task into three steps. (1) We simulated a sample of NS-NS merging systems populating the universe volume that can be explored with next-generation GW interferometers. Specifically, we assigned to each NS-NS system a luminosity distance that is randomly extracted from a uniform distribution in the range between 1 and 500 Mpc to cover a realistic range of the matched filter signal-to-noise ratio (SNR), varying from 4 to 20. We then assumed that both components of the binary systems have a mass distribution equal to a uniform distribution between 1 and 2.5 M_⊙; the two distributions were assumed to be uncorrelated. The inclination angle (${\theta }_i$) of the systems was chosen taking into account that the GRB emission is collimated, and these sources are typically detected when they are on-axis, i.e., with the jet pointing towards the observer. According to the few estimates currently available, the jet opening angle ${\theta }_j$ ranges from $3^{°}$ to $8^{°}$ (see, e.g., [43] and references therein); we therefore restricted the range of possible values of ${\theta }_i$ to take this observation into account. (2) Following the approach presented in [44], we assumed that all the NS-NS mergers are associated with a short GRB, and we simulated the associated high-energy afterglow light curves that could be observed by the LAT instrument onboard the Fermi satellite [45], using GRB 090510 [46] as a template. (3) We simulated the GW signal associated with the NS-NS mergers using the TaylorF2 waveform model [47]. We simulated the noise data for a GW detector such as the Einstein Telescope [48], where we injected the NS-NS merger GW signals. The simulations were performed using the pyCBC library [49].

We propose an MML pipeline consisting of two convolutional neural networks (CNN) concatenated at the output in order to estimate the redshift of the GRB and GW sources. To convert luminosity distances in redshifts, we use the cosmological parameters in [50]. A 2D CNN takes the time-frequency image representation of the GW signal as input. The images are built from the detector strain time series containing the injected NS-NS GW signals. As a first step, the strain is whitened in the time domain by means of an auto-regressive (AR) model [51] to remove the stationary noise component. The 60 s long segments containing the chirp signals are then converted into a time-frequency representation based on the continuous wavelet transform, using Morlet wavelets (using ssqueezepy library (July, 2021)1) (see [52,53,54]). Finally, the images are compressed to dimensions in pixels of $128\times 256$. For GRB lightcurves, we used the data in the time domain. The length of the GRB simulated data was kept up to 1000 points, where most of the information was encoded2. Examples of the chosen representations are shown in Figure 3: on the left side the image for the GW signal, and on the right side, the time domain light curve data.

The CNN processing GW data consists of 5 layers with the following number of filters: 64, 32, 16, 16, and 32, and kernels of dimensions ($3,3$). After every convolutional layer, we applied maxpooling with kernels ($2,2$). The 1D CNN processing the light curves consists of 3 layers with the following number of filters: 80, 40, and 40, and kernels 5, 3, and 3. Additionally, in this case, after every convolutional layer, we applied maxpooling with kernel 2. After the last layers, both CNNs were flattened and concatenated as one. The concatenated output is fed to a fully connected layer which outputs the prediction for the source redshift. All layers have a ReLU activation function, with the exception of the final layer with a linear activation. The MML model was trained on the training data set using the mean squared error (MSE) loss function and Adam optimizer with a learning rate of $\alpha =0.001$ and a learning rate decay of 0.066667. The model is summarised in Figure 4. For training and evaluation, the dataset was divided with the following scheme: $70\%$ training set, $20\%$ test set, and $10\%$ validation set. Training was carried out with a batch size of 16 samples for 100 epochs. The total number of trainable parameters is 61,881. As a pre-processing step, we applied minmax scaling to the inputs. Shuffling is also applied.

In Figure 5, we report the mean squared error loss evolution over the training epochs for both the train and validation sets. It can be observed that the MSE converged well over the 100 epochs algorithm and that there is no overfitting.

In Figure 6, we show the comparison between the predicted redshifts and the true redshifts of the simulated sources on the test data set. It can be seen that for the lowest values of redshifts, the predictions are in good agreement with the real values, while for the highest redshifts, the scatter around the line $y=x$ increases, and the predicted values are typically lower than the true ones. This is partially due to the fact that for the most distant sources, the GW signal has a low SNR value. Additionally, we used a limited dataset, and this could have affected our results. However, with this work, we only wanted to investigate the feasibility of multi-modal ML analysis; a more detailed study, including a larger simulated dataset and possibly real EM and GW data, will be presented elsewhere.

In Figure 7, we show the histogram of the the relative difference between the predicted redshifts and the true redshifts. It can be seen that the histogram has a peak at zero, meaning that for the majority of the simulated sources, the algorithm allowed for correctly estimating the redshift.

6. Outlook and Perspective

Our team’s innovative multimodal machine learning-based approach consists of analysing each multi-messenger event simultaneously in its entirety through the information we can gather from GWs or electromagnetic or neutrino emissions, which will be the inputs for a single MMML pipeline. This allows us to increase the information we can extract, thanks to the concomitant analysis of every piece of information. Just as it is easier for us to understand language when we associate sound to video, or to understand people’s moods by analysing text and images, we will apply multimodal analysis techniques to astrophysical transient events.

This is a new paradigm for analysing the data we will collect in future astro and particle experiments. We tested the idea on a data set we created simulating both GW and GRB emissions, just to set up a proof of concept. We built an MML pipeline taking as input the images obtained through a time frequency representation of whitened GW data for the Einstein Telescope detector and the simulated GRB light curves for a Fermi-like detector. We want to underline that the work here presented is just an example of a basic application of multimodal analysis for multi-messenger events, based only on GW strain data and gamma-ray GRB light curves, with a simple neural network architecture. The results, even if preliminary, are very encouraging, and we plan to continue with our approach on real data or including other input messengers. We will implement a more sophisticated and comprehensive analysis by adding data in different formats, at different levels (from raw data to high-level data), and also related to other messengers (e.g., neutrinos and photons at other wavelengths). For instance, we want to include in our framework raw data from Imaging Atmospheric Cherenkov Telescopes (IACTs) such as CTA: 2D images that represent the tracks left in the telescope’s camera by showers of particles that can be induced, for example, by very high-energy (VHE, E > 100 GeV) gamma rays emitted by GRBs. At the same time, we plan to optimize the analysis workflows by using feature extraction and engineering, data balance, and augmentation for less represented classes of events and by working with deeper networks. It is worth emphasizing that this approach we are proposing represents a new challenge for data scientists and astrophysicists, even in anticipation of the larger number of events we will be able to detect with third-generation GW detectors.