Regime Tracking in Markets with Markov Switching

: The object of the investigation is a model of the incomplete financial market. It includes a bank deposit with a known interest rate and basic risky securities. The instant interest rate and volatility are governed by a hidden market regime, represented by some finite-state Markov jump process. The paper presents a solution to two problems. The first one consists of the characterization of the derivatives based on the existing market securities, which are valid to complete the considered market. It is determined that for the market completion, it is sufficient to add the number of derivatives equal to the number of possible market regimes. A generalization of the classic Black– Scholes equation, describing the evolution of the fair derivative price, is obtained along with the structure of a self-financing portfolio, replicating an arbitrary contingent claim in the market. The second problem consists of the online estimation of the market regime, given the observations of both the underlying and derivative prices. The available observations are either a combination of the time-discretized risky security prices or some high-frequency multivariate point processes associated with these prices. The paper presents the numerical algorithms of the market regime tracking for both observation types. The comparative numerical experiments illustrate the high quality of the proposed estimates.


Introduction
The continuous-time or discrete-time stochastic models of incomplete markets represent the required instruments of financial mathematics [1][2][3].Possessing a reasonable degree of adequacy, they assist in the proper formulation of the whole variety of system analysis problems arising in finance: - The analysis itself, for example, the derivative pricing, the market price of risk (MPR) calculation, modeling the term structure of interest rates, etc. [4,5]; - The state estimation, parameter identification, and statistical inferences in the market models given the heterogeneous a priori and statistical information [6][7][8]; - The optimization and stochastic control with complete or incomplete information, for example, the minimal hedge problem, the optimal investment problem [1,3,9], etc.
Usually, the market incompleteness is caused by some hidden external stochastic processes in the dynamics.Various stochastic volatility models [10,11] are typical examples of this situation.The monitoring of the hidden governing processes, and stochastic volatility, in particular, is an object of a sufficient number of papers [12][13][14][15][16]. Choosing one model or another, the authors transform this problem into a state-filtering one, given heterogeneous observations.The obtained results demonstrate the varying degree of applicability, from some clarification of the optimal nonlinear filtering problem to the well-elaborated optimal or suboptimal/robust algorithms.
The investigation object of this paper is a market model governed by a Markov regimeswitching process.Although it is simpler than the corresponding switching versions of the Heston, Hull-White, Vasicek, and other models [17][18][19][20], it has some advantages.First, this class retains the property of the market incompleteness.Second, it represents an efficient instrument to simulate market phenomena, like the unpredictable changes in the market evolution scenario.Third, reasonable model simplification allows us to obtain more advanced results.
The paper aims to reduce a priori statistical uncertainty in the market caused by the hidden regime.There are many ways for this, and the paper presents just two of them.The first one implies market completion by some additional securities.The authors of [21] propose to make the market regime tradable.By contrast, we suggest security enlargement by the derivatives with underlying assets already traded in the market.For this, it is necessary to derive the equations describing the evolution of the fair derivative price and formulate sufficient conditions for the derivative collection to complete the market.
Another way does not imply market completion.The goal is to estimate the hidden market regime given the various complexes of the statistical trading information.
The paper has the following organization.Section 2 contains a detailed description of the market model under investigation.Section 2.1 presents a stochastic differential system (SDS) with the martingales on the right-hand side, which describes the evolution of the basic security prices.The key feature is that both the interest rate and volatility are functions of a hidden market regime, representing a finite-state Markov jump process (MJP).Its presence in the model forms a natural reason for the online estimation of this hidden process using statistical information concerning security prices.Section 2.2 contains the mathematical statement of the optimal filtering problem, given the observations with the multiplicative noises.The section also contains the theoretical solution to the problem: the optimal estimate is the unique solution to a closed finite-dimensional SDS.
Section 3 is devoted to the completion problem of the considered market.We introduce some "natural" derivatives with the underlying securities already presented in the market.The fair derivative price is a function of the current underlying security price and the market regime.Section 3.1 introduces a system of partial differential equations, describing this function and generalizing the classic Black-Scholes equation.According to the assertion in Section 3.2, one can complete the market by a finite number of derivatives, and this number coincides with the number of possible regimes.
Section 4 is directed to the algorithmic support of the above problems.Primarily, it argues against the availability of the continuous-time noiseless observations of the underlying and derivative prices in practice.Section 4.1 presents a semi-analytical algorithm of the solution to the system of partial differential equations from Section 3.1.It is applied in the derivative price simulation and market regime filtering.The further material is about two suboptimal filtering algorithms of the market regime based on heterogeneous observations.In Section 4.2, the available observations include the noiseless underlying prices and the noisy derivative ones.All the data come at discrete nonrandom instants.In Section 4.3, the observations represent some multivariate point processes (MPP).The distribution of the MPP inter-arrival times and noise in the derivative observations depends on the current market regime.The key feature of the observations is their high frequency.It admits the application of the central limit theorem (CLT) for the generalized regenerative processes [22] to the specially preprocessed observations with the subsequent use in the filtering procedure.
Section 5 presents the comparative numerical analysis of the proposed filtering algorithms.The market model is identical for all experiments, but the observation complexes differ.The experiments demonstrate enhancement of the estimate quality by using additional observations of the derivative prices.Sections 5.1 and 5.2 are devoted to the performance analysis of the filtering algorithms introduced in Sections 4.2 and 4.3, respectively.Section 6 contains the concluding remarks.

Market Description and Arising Problems
With the probability triplet with filtration (Ω, F, P, {F t } t∈[0,T] ), we consider a financial market that contains a risk-free bank deposit B t = exp t 0 r u du and N basic risky securities.The deposit interest rate r t is non-random and known.The price S t ≜ col(S 1 t , . . ., S N t ) of the basic risky securities is the unique strong solution to the SDS where -S 0 is an F 0 -measurable initial condition with the distribution function P S 0 (•);  In (1), Z t = col(Z 1 t , . . ., Z L t ) ∈ S L is an MJP with the state set formed by all unit coordinate vectors of the Euclidean space R L : S L ≜ {e 1 , . . ., e L }.This process has a known transition rate matrix (TRM) Λ(•) and an initial distribution p Z 0 = col(p Z1 0 , . . ., p ZL 0 ).The MJP Z t is a unique strong solution to the following SDS [23]: where We are interested in two problems related to the above market.The hidden, unobservable sudden change of the exogenous factors, generating the market regime switching, is often connected with jumps in the macroeconomic situation [24,25].Its exhaustive mathematical model is not a subject of this paper.Nevertheless, the formal description of its specific phenomena and restoration of some hidden parameters, given a bulk of the financial statistics, look prospective and real.In this context, the process Z t represents a scenario of the market evolution.So, the first problem is the tracking of the regime-switching process Z t (or, shortly, the market regime) by the observable prices S t of the underlying securities and their potential derivatives.
It is known that the market (1) and ( 2) is incomplete due to stochastic volatility.Obviously, it admits various approaches to pricing and hedging (see, for example, [18]).Another approach implies the market completion by some derivatives.The authors of [21] suggest to use the jump derivatives associated with the market regime transitions.These securities look slightly artificial.First, their prices can be negative.Second, the underlying "securities" for them are the random flows of the regime transitions to one or another value.This implies that the transitions are directly observable and contradicts reality.So, the second problem is the construction of some "natural" derivatives, using the existing basic securities to complete the market.
We make the following assumptions, clarifying the details of the SDS (1) and (2): i. Z t is a cádlág process [26].ii.Without loss of generality, F t = σ{Z u , w u : 0 ⩽ u ⩽ t}.This condition also guarantees that the filtration {F t } is continuous from the right.iii.The TRM Λ(•) consists of cádlág elements on [0, T].All off-diagonal elements of Λ(•) are strictly positive, i.e., min Λ ij (t) > 0. All elements of the initial distribution p Z 0 are also strictly positive.
iv. Z t ∈ S L ; hence, the functions a and b have the form where {a ℓ (t)} ℓ=1,L and {b ℓ (t)} ℓ=1,L are the sets of the known nonrandom functions a ℓ (t) ≜ a(t, e ℓ ) and b ℓ (t) ≜ b(t, e ℓ ) (alphabets).All components of a ℓ (t) and b ℓ (t) are cádlág piecewise smooth functions.The random disturbances in (1) are uniformly non-degenerate, i.e., b ℓ (t) ⩾ α 2 I > 0 for all ℓ = 1, L and t ∈ [0, T] (here and below, I denotes the identity matrix of an appropriate dimensionality).v.
The conditions above seem non-restrictive and guarantee both solution to the regimetracking problem [27] and incompleteness of the market [21].They legitimize all inferences made below to present the fair derivative pricing and filtering algorithms, given the various complexes of the available observations.

Market Regime Tracking as Optimal Filtering Problem
In this subsection, we introduce a proper statement of the regime-tracking problem, given the observations of the security prices, as a particular case of the optimal filtering of a semimartingale [26].Further, we present some transformation of the observations and a theoretic solution to the problem: the optimal filtering estimate is the unique strong solution to a finite-dimensional closed SDS.
At first glance, the state Z t tracking represents the calculation of its conditional distribution, given the natural filtration {S t } generated by the security prices: S t ≜ σ{S u : 0 ⩽ u ⩽ t}.However, there are two issues preventing the usage of the advanced framework of stochastic analysis.First, {S t } is not continuous from the right [27,28].Second, the observations S t contain state-dependent noises [29,30], which hinder the use of the Girsanov measure transform.To overcome these obstacles, we treat Z t tracking as the calculation of Z t ≜ E{Z t |S t+ }.This theoretical passage is standard [26], and its degree of realism is discussed below in the subsection.
The details of the solution to the optimal filtering problem can be found in [27].We split the initial observations into the following N-dimensional fractions: - The process U t with P a.s.continuous trajectories; - The process C t with counting components; - The piecewise constant process D t with jumps, occurred at nonrandom instants.
We then use the obtained transformations to construct the optimal filtering estimate as a solution to some SDS.
The process of logarithmic prices where The continuous process U t has the form and represents an S t -adapted standard Wiener process.The counting observations C t and discrete ones D t are the results of the quadratic characteristic ⟨Y, Y⟩ t transformation.They are closely related to the S t+ -adapted process where (L × L)-dimensional non-random matrix-valued function Here, I A (x) denotes the indicator function of the set A. For any t ∈ [0, T), there exists a transformation T (t) such that the matrix T (t)K(t+) is trapezoidal with orthogonal rows, and the components are from the set {0, 1}.
Hence, the values of the process Z t (7) lie in the set S L with probability 1, and the matrix-valued non-random function J(t) has only cádlág components with values from {0, 1}.Denoting the discontinuity set of the function J(t) by J , we have the following decomposition: The first summand D t describes the indirect discrete-time observations of Z t , generated by the non-random jumps of the observation matrix J(t): The second summand R t accumulates the random transitions of Z t , observable through the quadratic characteristic ⟨Y, Y⟩ t .Finally, we have to transform V t into the process C t with components, counting the random transitions of V t in each specific value e ℓ ∈ S L : To present the filtering equations in a compact form, we use the following notations: A + is a Moore-Penrose pseudoinverse matrix; -1 is a row vector of an appropriate dimensionality, formed by units: - Proposition 1.Under conditions (i)-(v), the coincidence S t+ ≡ σ{S 0 , U s , C s , D s : s ⩽ t} holds for all t ∈ [0, T).
The proof of Proposition 1 is analogous to Lemma 1 from [27].
Proposition 2 ([27], Theorem 1).The optimal estimate Z t is a strong solution to the SDS The solution is unique within the class of non-negative piecewise continuous S t+ -adapted processes with a discontinuity set lying in V.
The process C t is an indirect observation of the MJP Z t transitions.One can expect the existence of some identifiability conditions for the observation system (2) and ( 4), which guarantees the exact restoration of Z t .Proposition 3 ([27], Corollary 1).If for any i ̸ = j (i, j = 1, L) the inequalities b i (t) ̸ = b j (t) hold almost everywhere on [0, T], then Z t = Z t P a.s., and Z t is the solution to SDS (14).
Let us discuss the contribution of the present result in the context of financial mathematics.The observation transform allows to obtain the optimal estimate by the closed finite-dimensional filter.The inclusion S t+ ⊆ F t is obvious, but under the identifiability conditions, it transforms into the coincidence S t+ ≡ F t .These conditions are not restrictive: they mean the almost constant distinction of all elements of the volatility alphabet.Otherwise, one can restore only some subsets of regimes with the coincidental volatility functions.
If the identifiability conditions hold, the market regime can be estimated precisely, and the market becomes complete.Nevertheless, this restoration of the market regime is possible for the natural filtration generated by the original observations and then closed from the right.The closure operation is routine in abstract mathematics but is crucial in practice.The results of the mathematical operations, including a limit passage, during the numerical realization, are replaced by some pre-limit values.In the context of the MJP state filtering, this means that the estimates of Z t (at the moment t) are the functions of the observation trajectory S [0,t+δ] with a short time lag δ.Hence, in the numerical realization, the initial filtering problem is replaced by smoothing with the fixed short delay.When the market regime estimation is the final goal, such relaxation of the estimation problem seems admissible.However, in the context of subsequent market completion and hedging, this action is inappropriate.In the complete market, one can replicate any contingent claim by some self-financing portfolio via a non-anticipating strategy.By contrast, in the case of the considered relaxation, a portfolio strategy depends on the future security prices.The filtration {S t } is not continuous from the right, but one can provide this continuity artificially by extending it with the MJP Z t transitions.The authors of [21] complete the market with some artificial securities generated just by these transitions.
The market would be complete if we construct some "natural" observer of the regime transitions.In this case, the filtration generated by the new observations would be continuous from the right.Below, we enlarge a market by the derivatives of the securities already traded in the market.First, these derivatives represent the natural way to complete the market.Second, additional observations of the derivative prices give a chance to raise the quality of the market regime estimates.

Fair Derivative Price
For the system (2) and ( 4), we presume the arbitrage absence [31]; hence, on the measurable space (Ω, F), there exists a prevailing martingale measure Q (Q ∼ P) [13], which ensures the fulfillment of the conditions below: 1.
The process M t is a martingale with respect to Q.

2.
Under Q, the price process S t is the unique strong solution to the SDS where w Q t ∈ R N is an F t -adapted standard Wiener process.According to the Girsanov theorem [32], the Wiener process w Q t is connected with w t : where θ t ∈ R N is an unobservable F t -adapted process of MPR [13,33].Keeping in mind the positivity of S t , formulas (1), (3), and ( 15), we obtain the relation between a ℓ , b ℓ and θ t : Hence, the MPR takes the form To complete the market, we enlarge it with some derivatives.Without loss of generality, we consider the case of a single derivative with the expiration date T, defined by the claim H(S T ).The goal of this subsection is to obtain the equation describing the evolution of the fair price F(t, S t , Z t ) of the introduced derivative.It is the discounted conditional expectation of H(S T ), with respect to the measure Q: Note, that G t ≜ E Q {H(S T )|F t }, being a martingale under Q, admits the representation We suppose that F ℓ is smooth enough, and the process G t has the following stochastic differential (here and below in the subsection, the dependency on the arguments is omitted for simplicity): We consider the second and third summands in the parentheses of the latter expression: where . Using formulas ( 16) and (18) in the third summand, and the notation ∇ s F ℓ ≜ row F ℓ s 1 , . . ., F ℓ s N , we obtain: The process G admits the stochastic differential The first summand is the differential of a Q-martingale, and the second one is the differential of a function with a finite variation.The martingale property of {G t } under Q and positivity of p Z t components on [0, T] allow to determine the price process {F ℓ (t, s)} as a solution to the following Kolmogorov system [34]: The system (20) relates to an analogue in [35], where only the volatility is switchable, and coincides with the system in [36].
Note that the derivative price and the differential with respect to the "real world" measure P
To complete the market, we enlarge it by L derivatives, which correspond to the contingent claims H(S T ) ≜ col(H 1 (S T ), . . ., H L (S T )) (T is an expiration date common for all derivatives).We group the derivative prices into the vector process The claims should satisfy the following condition: vi.The matrix The kth row of F(t, s), (F k1 (t, s), F k2 (t, s), . . ., F kL (t, s)), is a solution to the system (20) with the terminal condition F kℓ (T, s) ≡ H k (s) (ℓ = 1, L), and this row characterizes the price of the kth derivative.
It is easy to verify that under condition (vi), the process Z t is S t ∨ F t -adapted (here i.e., under conditions (i)-(vi), the coincidence F t ≡ S t ∨ F t holds for all t ∈ [0, T].Hence, the stochastic differential of Q-martingale w Q has the form Further, the price of the kth derivative F k t admits the stochastic differential The evolution of the derivative prices is the unique strong solution to the following SDS (here Γ t = col(Γ 1 t , . . ., Γ L t )): Condition (vi) allows to express the martingale M t , which generates the MJP Z t : Let the martingale µ t (23) represent an arbitrary contingent claim µ t = E{H(S T )|F t }.We construct a self-financing portfolio (π t , Π t , ϖ t ), which replicates the claim.Here, the vector π t ≜ row(π 1 t , . . ., π N t ) describes the fractions of underlying securities in the portfolio, Π t ≜ row(Π 1 t , . . ., Π L t ) plays the same role for the derivatives, and ϖ t is a portfolio fraction, invested in the deposit.The choice provides the replication of µ t .Actually, if C t is a current value of the portfolio, then We consider the gain process associated with the portfolio, and use ( 25), ( 28), ( 29) and integration by parts: where It is easy to verify that the choice of the fractions provides the self-financing property for the portfolio: Thus, we have proved the following.
Theorem 1.Under conditions (i)-(vi), the market with Markov switching, augmented by the set of L derivative securities, is complete.
As in the case of the market completion by the additional Markov jump securities [21], one needs to enlarge the market by L derivatives: this is the number of possible regimes in the market.Nevertheless, the completion method suggested here has some advantages.First, the additional securities represent "the routine derivatives" of the underlying securities already traded in the market.Second, the provided derivatives do not possess such artificial properties as a possibility to have negative prices.

Algorithms of Markov Regime Tracking
The results of the previous section are academic.If a trader has continuous-time observations of the basic security prices, they can restore the hidden market regime precisely or significantly reduce its statistical uncertainty, having an arbitrarily short time delay.Moreover, if the continuous-time noiseless observations of the derivative prices are available, then the online restoration of the market regime would be possible without the application of the sophisticated filtering formulas (10)-( 14) but using the elementary algebraic transformation (24).
The principal assumption contradicting the reality is the continuous-time character of the price observations.Hence, the first assumption, which makes the considering market model closer to reality, is that all security prices are observable only at some discrete instants.The exact prices of the underlying securities are known to all traders at these moments.By contrast, the derivative prices are unavailable for the noiseless observation.Otherwise, having direct discrete-time observations of the derivative prices, the traders could restore the market regime or its subset at the observation time moments, which contradicts the reality.So, the second assumption is that only noisy observations of the fair derivative prices are available for the traders.All traders can calculate the set of possible derivative prices {F ℓ (t, S t )} ℓ=1,L but they do not know the prevailing variant, chosen by the current market regime.The traders place the bid and ask orders with prices not coinciding with the prevailing ones.The reasons for this tactic are mistakes in planning or some advanced trading policies directed to an extra profit.
Thus, the observations registered at some nonrandom instants [8] look more realistic than the continuous-time noiseless data.Another possible procedure to collect the price statistics is a routine registration of a random flow of the security trades [15].The section presents suboptimal filtering algorithms of the market regime, given the observations of both types.We specify the observation models in detail in the corresponding subsections.Additionally, the section contains an algorithm of the numerical solution to the system (20) related to the realization of the provided filtering algorithms.

Algorithm of Numerical Solution to Generalized Black-Scholes Equation for Markov Regime Switching Market
The system (20), an extension of the classic Black-Scholes equation, does not provide an analytical solution, even in the case of time-invariant coefficients r, a, b, and Λ.In this subsection, we present a semi-analytic algorithm to solve (20), which can be considered some version of the splitting method [38].
To simplify the presentation, we make the following assumptions.
A1.All coefficients in (20) are time invariant: The condition is non-restrictive: the coefficients could be piecewise constant on the time steps.Actually, if we normalize the time to one year, treating it as 250 trading days per 8 h each, then the time increment 0.0005 would correspond to 1 h, and r, a ℓ n , b ℓ nm , and Λ ℓk would look like the constants on on the time steps.A2.Each contingent claim refers only to single underlying security, i.e., H ℓ (s 1 , . . ., s N ) = H ℓ (s n ) for some 1 ⩽ n ⩽ N.This condition excludes the case of the compound contingent claims.
Let us consider a single derivative and a single underlying security.In this case, the system (20) takes the form By setting one can rewrite the Cauchy problem (31) in the operator form Its solution has the form F(t) = L(t, T)H, where L(t, T) is a transition operator on the interval [t, T], corresponding to (L 1 + L 2 ).The Hadamard principle [38] is valid for this equation; hence, and it is a base for the calculation of F over the time grid {t j } j=1,J : t j = jh, h = T/J: It is impossible to express the operator L(t, T) in the analytic form, and it can be approximated by many ways [39].In this paper, we propose to use the splitting method [38].For this, we consider two auxiliary Cauchy problems with the solutions expressed via the corresponding transition operators L 1 and L 2 and Hadamard's principle: We approximate the solution to (34) {F(t j )} j=0,J , replacing L by the sequential composition of L 1 and L 2 : Note that the transition operators L 1 and L 2 can be found explicitly.
All equations in the system (35) are disjoined and can be separately transformed to the standard heat equation as the classic Black-Scholes equation [2].We consider one of these Cauchy problems (the indices n and ℓ are omitted for simplicity): and solve it by the consecutive replacements: 1. R is replaced by V: V(t) ≜ e r(T−t) R(t).The Cauchy problem (39) for V takes the form 2.
We introduce the new variable τ(t) = T − t and function U(τ(t), s) ≜ V(t, s).The Cauchy problem (40) for U takes the form 3.
We introduce the new variable x(τ, s) = ln s + (r − a − b 2 )τ and G(τ, x(τ, s)) ≜ U(τ, s).The Cauchy problem (41) for G takes the form The solution to ( 42) is known: where Making all reverse substitutions, we can write the explicit component-wise version of the transition operator L 1 = col(L 1  1 , . . ., L L 1 ) for the single time step: System (36) consists of homogeneous differential equations, and the transition operator L 2 over the single time step is expressed via the matrix exponential Finally, the recursive procedure (38) of the layer-by-layer approximate calculation of F can be written as where The integral in (46) cannot be calculated analytically, and here we calculate it approximately, using the composite trapezoid scheme with the space step ∆.The integration area [0, +∞) is replaced by a finite interval [s, s], which satisfies the condition E S T I (0;s)∪(s;+∞) (S T ) ⩽ ∆ 2 .According to [38], one can conclude that the approximation F (46) provides the local accuracy for a single layer as O(h 2 + ∆ 2 ).
So, we can calculate the approximate solution to the system (31) by the recursive procedure (46), using the matrix algebra operations, which have the effectively optimized realization in the contemporary software libraries.However, we should mention some natural issues related to the procedure of derivative price simulation or their usage for market regime filtering.The formula (46) represents the backward-time recursion.By contrast, the price simulation and filtering algorithms imply the forward-time recursive process.Hence, all values of F previously calculated on the time grid should be stored in the considerable volume of computer memory for consecutive utilization in the simulation or estimation procedures.

Algorithm of Regime Tracking by Discrete Time Observations
Let us consider the market (1), ( 2) and ( 22) as a continuous-time stochastic dynamic system with a compound hidden state col(S t , Z t , F t ).The market model satisfies Assumptions A1 and A2 of Section 4.1.The observations, which are available at the discrete instants t i = ih, i = 1, I (I = T/h), consist of the following: -Noiseless prices of N underlying securities -Indirect noisy observations of M derivative security prices The discrete-time nature of the available observations (47) prevents the use of traditional approximating schemes, like the Euler-Maruyama or Milstein methods [40], for the numerical solution to the SDS (14).The reason is that the system includes the transformed observations (U t , C t , D t ) instead of the original ones S t .The main property of this transformation is the use of two subsequent limit passages to obtain the quadratic characteristics ⟨Y, Y⟩ t and its derivative.The replacement of the continuous-time observations S t by their time discretization S i with some fixed non-vanishing time step h makes these limit operations impossible.Hence, we propose to transform the filtering problem.The aim is to estimate the market regime Z only at the observation moments t i .We treat the observations S i and F i as some noisy functionals of the regime trajectory {Z t } t∈[t i−1 ,t i ] .The solution to the new filtering problem represents a version of the Bayes formula.We treat its numerical realization both as an applied algorithm of market regime tracking and approximation of the solution to SDS (14).
To describe the observations properly, we introduce the following filtrations: -O i ≜ σ{S j , F j : 0 ⩽ j ⩽ i} are σ algebras generated by all available observations obtained till the moment t i ; -G i ≜ σ{F j : 0 ⩽ j ⩽ i} are σ algebras, generated only by observations of the derivative prices; -H i ≜ σ{S j , Z t j : 0 ⩽ j ⩽ i} are σ algebras generated by the underlying security and market regime, available on the time grid till the moment t i .
For the observable sequence {F i }, we admit the Markov property given the pair (S, Z), i.e., for any B ∈ B(R M ) and i = 1, I, the following equalities are P a.s.valid: where {ρ i } 1,I is a family of known nonrandom measures, and {ϕ i } 1,I are the corresponding densities.The variant of (49) for the initial observation F 0 has the form The filtering problem for the market regime is to find We introduce the discrete-time logarithmic prices , . . ., ln According to the Ito rule, each component of Y i has the form It is easy to verify the identity of the σ-algebras σ{S 0 , S 1 , . . ., S i } ≡ σ{S 0 , Y 1 , . . ., Y i }, hence one can use transformed observations instead of the original ones.
The observations {Y i } have a probability density function (pdf), and the mutual distribution of the pair (Z t i , Y i ), given Z t i−1 , can be expressed via the pdfs {ξ jk i (v)} i=1,I, j,k=1,L : Proposition 4. The optimal filtering estimate Z i is determined by the recursive procedure with the initial condition The proof of Proposition 4 follows from the Bayes formula.
The main issue in the numerical realization of the recursive procedure (52) lies in the calculation of the pdfs ξ kℓ i (v).From (51), it follows that the pdf of Y i is a mixture of some Gaussians.The mixing distribution depends on the duration of the MJP Z in each possible value on the intervals [t i−1 , t i ] given the fixed starting and ending points.In the case of a time-invariant market (see Assumption A1 in the previous subsection), ξ kℓ i (v) can be approximated by the composite midpoint rectangle rule [41].We use the following notations: -G(v, M, K) is the Gaussian pdf with the mean M and non-degenerate covariance matrix K; ) are the midpoints of the smaller intervals of the length h 1+α , m = 1, [h −α ], 0 < α ⩽ 1 (here and below [a] is an integer part of a); With these, we approximate the function ξ kℓ i (•) as where δ kℓ is the Kronecker delta.
If we replace recursive procedure (52) by the following version: then the global error of this approximating scheme has the order α [41], i.e., for some C > 0 and all i = 1, I.

Algorithm of Regime Tracking by High-Frequency Multivariate Point Observations
In this subsection, we present a suboptimal filtering algorithm of the market regime given the observations of the MPPs.The problem is to estimate the MJP Z t at the points t i = ih, where h > 0 is a time increment.It is essential that h ≪ min 1⩽ℓ⩽L |Λ ℓℓ | −1 , i.e., the probability of Z t to have a jump on some time discretization interval [t i−1 , t i ] is small enough.The key feature of the available observations is their high frequency: the number of the observations that occurred at [t i−1 , t i ] is large enough to apply the CLT for the generalized regenerative processes [22] to the original observations or their transformations.
To simplify the presentation of the filtering algorithm, we consider the particular case of the market (1), ( 2) and ( 22) with a single underlying security and a single derivative one.Assumptions A1 and A2 hold, and the compound market state (S t , Z t , F t ) is unavailable for the direct noiseless continuous-time observation.
The discrete observations have the following structure: {(τ S j , S j )} j∈N: is an MPP of the underlying security price observations, is an MPP of the derivative price observations.We assume that the observations satisfy the properties below: 1. S j ≜ S τ S j , i.e., at the increasing sequence of the random moments τ S j traders observe the exact price of the basic security.

2.
Given the market regime Z, the random inter-arrival times δ S j ≜ τ S j − τ S j−1 are mutually independent.The distribution of δ S j depends on the regime state Z τ S j−1 with the known conditional moments 3.
, where {v k } are multiplicative random errors, which are conditionally independent, given the market regime Z.This means that the noisy observations of the derivatives are available to the traders at the increasing sequence of the random moments τ F k .The distribution of v k depends on the regime state with the known conditional moments

4.
Given the market regime Z, the random inter-arrival times and has the known conditional moments

5.
Given the market regime Z, the sequences {δ S j } j , {δ F k } k , and {v k } k are mutually independent.

6.
The mean values {m S ℓ } ℓ and {m F ℓ } ℓ are much less than the time step h: Let us introduce the family of σ-algebras generated by the observations obtained till the moment In the case of the completely known mutual distribution of the system state and observations, we can calculate the optimal estimate of the MJP state Z i ≜ E{Z t i |O i }, similarly to [42].However, the optimal estimate is sensitive to the uncertainty in the MPP distribution and costly from the computational point of view.We propose a suboptimal robust filtering algorithm [43], which uses the observations transformed to some generalized regenerative processes with the distributions modulated by the MJP Z t .To apply the algorithm, we need to know only the moment characteristics from items 2-4 given above.The theoretical background of the algorithm is a version of the CLT for the generalized regenerative processes [22].
On each time step [t i−1 , t i ], we sample the original observations in the following way: To obtain the weight functions in the filtering algorithm, we have to reconstruct the asymptotic distribution of W i under each of the conditions Note that under a fixed MJP trajectory, the subvectors i ) are independent.Furthermore, the random numbers of summands in the first and second subvectors differ.From the Ito rule, ln where {u j } is some standard Gaussian discrete white noise.The block components W ′ i and W ′′ i are the random sums of the vectors Then, the vectors According to the CLT for the generalized regenerative processes [22], there exists the following weak convergence as h → ∞: .
Hence, we can conclude that the distribution of 1 Finally, the suboptimal filtering algorithm takes the following form.

1.
The initial condition: The prediction step: The correction step: There is an issue in the realization of the algorithm.In (58), one needs to know S τ F k , i.e., the underlying price at the moment of the derivative observation.This price is unobservable at this moment, and we suggest replacing it with the earlier available observation of the underlying price, obtained at the moment, closest to τ F k , i.e.,

Regime Tracking by Time-Discretized Continuous Observations
This example illustrates the impact of the mutual processing of the underlying and derivative prices on the performance of the market regime estimates.We consider the time interval [0; 1], which corresponds to 1 year with 250 trading sessions at 8 h each, and simulate the evolution of one underlying security and one derivative, a European call-option with the strike 1.1.Both the instant interest rate and volatility are governed by the Markov regime-switching process with four possible states "growth-epoch before panic-panic-recession".The deposit rate r = 0.05.
All observations are synchronous with the time step h = 0.0001 (this corresponds to 12 min).We compare the performance of the filters, calculated with three complexes of the available observations: C1.The precise security price {S i } is obtained with the time step h.C2.The combination of {S i } and the option price is corrupted by a multiplicative noise Here, Γ m = ∥Γ jkm ∥ j,k , (j, k, m = 1, L) are matrix exponentials Γ m (h) = exp(hΥ m ), calculated by the following TRMs: The distribution of the errors in the derivative observations depends on the time step h.
Let us argue for the choice of the parameters in the numerical experiment.First, the deposit interest rate corresponds to the values in the actual banks.Second, the number and description of the possible market evolution scenarios look reasonable.The author of [25] also suggests four macroeconomic market regimes.In this paper, we, in some way, transformed them into microeconomic regimes.For all of them, the MPR is non-negative and equals 0 in the fourth regime (recession).Third, we choose the volatility values close to each other to hamper the regime estimation.Fourth, in complex C2, the noise parameters are the same for all regimes.This choice also complicates the estimation procedure.If the noise parameters were different, the filtering accuracy would only increase: in this case, the noise would act as some useful signal, carrying information concerning the market regime.Thus, we choose the combination of the observation system parameters, which is "non-friendly" for the regime estimation.The additional statistical information concerning the derivative prices allows us to improve the estimation quality in this situation.Obviously, the prices do not allow to identify the market regime visually.
Although the option price observations in the observation complexes C2 and C3 look different, they are the particular cases of the model introduced in Section 4.2.To emphasize the difference between these observations, we present their errors in the additive form: 2 presents the plots of these errors.The oscillations of the errors ∆ LN t , corresponding to complex C2 with multiplicative noise, look like non-stationary white noise.By contrast, the errors ∆ MC t , corresponding to complex C3, are obviously dependent and demonstrate the bursting character.Analyzing the calculation results, we can make the following conclusions.First, based only on the underlying price observations, one can identify only regime No. 1 (growth) with acceptable reliability.The duration of other regimes is too short, and the availability of the underlying prices only is insufficient for the admissible precision of the estimates.Second, the usage of the noisy derivative price observations significantly raises the estimation quality.To confirm this thesis formally, we consider the following estimation performance index: Obviously, one can consider the (unconditional) mathematical expectation of the estimated MJP Z t as its trivial estimate.The proposed performance index is related to the determination coefficient [45] and represents a ratio of the sample variance of the considered filtering estimate to the theoretical variance of the trivial estimate.The proximity of the index to 0 indicates the high accuracy of the estimate.If the index value is slightly less than 1, then the advantage of the proposed estimate is insignificant compared with the trivial estimate.If the index value is greater than 1, then the proposed estimate is worse than the trivial one and should be excluded from the consideration.Obviously, this exclusion is reasonable, if we consider the mean square error criterion as the estimation performance index.In the considered numerical example C Z S = 0.135, C Z LN = 0.016 and C Z MC = 0.102.So, involving the observations of the derivative prices significantly improves the estimate precision.

Regime Tracking by High-Frequency Multivariate Point Observations
In this numerical experiment, we investigate the same market model as in the previous subsection but with different observation complexes.
C4.There are only the noiseless observations {(τ S j , S j )} of the underlying security, received at the random instants τ S j .Given the fixed regime-switching trajectory, the inter-arrival times δ S j = τ S j − τ S j−1 are mutually independent exponentially distributed values [46].The distribution parameter depends on the current market regime and is set by the vector col(100,000, 95,000, 80,000, 90,000).C5.In addition to the underlying prices {(τ S j , S j )}, there are available noisy option price observations {(τ F k , F k )}, received at the random instants τ F k .As well as {δ S j }, the inter-arrival times δ F k = τ F k − τ F k−1 between the option observations are mutually independent exponentially distributed values, given the fixed regime-switching trajectory.The distribution parameter depends on the current market regime and is set by the vector col(90,000, 88,000, 85,000, 89,000).The multiplicative noise {v k } in {F k } has the lognormal distribution with the mean 0 and the variance 0.5, which are common for all market regimes.The time step between the regime estimation procedure h = 0.0001 corresponds to the 12 min of the operating time.
Figure 4 presents the observable MPPs {(τ S j , S j )} and {(τ F k , F k )} obtained during step h and used for further preprocessing into the sampled observations W i .One can see that the interval h contains the number of observations sufficient for the CLT asymptotics "to start working".It includes the plots of the following: -The transformed inter-arrival times between the underlying price observations W It is impossible to restore the current market regime Z t using the visual analysis of the observations.One can also note a non-stationary behavior of the preprocessed option prices.
Figure 6 illustrates the legitimacy of the CLT use in the proposed algorithm.Let us consider two first components {(W 1 i , W 2 i )} of the preprocessed observations {W i }.Under the condition of the constant regime Z t ≡ e ℓ , t ∈ [t i−1 , t i ], the components {(W 1 i , W 2 i )} are the increments of the generalized regenerative processes, and the CLT admits an approximation of the pdf by the Gaussian G(x, m ℓ , K ℓ ).The homogeneous MJP Z t has stationary distribution p ≜ col(p 1 , . . ., p L ); hence, by the law of total probability, "the theoretical" unconditional pdf of the pair (W 1 , W 2 ) takes the form p W (w) = ∑ L ℓ=1 p ℓ G(w, m ℓ , K ℓ ).
The plots demonstrate a remarkable similarity.Note that the theoretical pdf is not a single Gaussian but is a mixture of them.The point is that the distribution parameters of the different modes are very close to each other, so the Gaussian modes visually differ little.Figure 6 demonstrates some asymmetry in the histogram and the theoretic density.Despite the similarity of the parameters, the proposed filtering algorithm provides high estimation performance.
Figure 7 presents the estimation results: -The exact regime state Z t ; - The regime-filtering estimate Z SM t , calculated by observation complex C4; -The regime-filtering estimate Z SFM t , calculated by observation complex C5.The analysis of the numerical results leads to the following conclusions.First, the filtering algorithm based only on the underlying security observations provides the acceptable estimation quality for regimes No. 1 (growth) and No. 4 (recession).The reason is again in the too short duration of regimes No. 2 ("epoch before panic") and No. 3 ("panic").Second, the option price observations in the filtering procedure significantly raises the estimation quality.The performance index (62) of the estimate calculated by the observation complex C4 equals C Z SM = 0.106; meanwhile, the corresponding value for observation complex C5 equals C Z SFM = 0.012.Third, the random inter-arrival times can serve as informative observations when their distribution parameters are distinctive for the different market regimes.

Conclusions
The investigation object of this work is a model of the incomplete market.Meanwhile, the deposit rate is known and non-random; both the interest rate and volatility of the risky securities depend on an outer uncontrolled market regime, an MJP with a finite state set.The paper contains a statement and solution for two interconnected problems.First, it gives a positive answer regarding whether the market can be completed with a finite set of derivatives based on the existing securities.The paper introduces a system of partial differential equations, describing the fair price of these derivatives and representing an extension of the classic Black-Scholes equation.It is determined that for the completion of the market with L possible regimes, it is sufficient to use L additional derivatives.The paper presents a self-financing portfolio replicating an arbitrary contingent claim built from the market securities.
Second, the work transforms the tracking of the market regime to a state-filtering problem in the stochastic dynamic system, given heterogeneous observations.It also places the arguments for the impossibility in the market to have direct continuous-time noiseless observations of both the derivative prices and their underlyings.Furthermore, the paper contains the regime-filtering algorithms corresponding to two different complexes of the available observations: -Discrete-time noiseless observations of the basic securities and noisy observations of the derivatives; - The observations of underlying and derivative prices in the form of the MPPs.
The comparative numerical study confirms the high quality of the proposed regime estimates and the significant enhancement of the estimation performance after including the derivative price observations in the filtering procedure.The application of the proposed estimation algorithms for market regime tracking has natural limitations.Actually, the estimation quality is sensitive to the a priori uncertainty of the observation system parameters.Besides the design of the procedures for the mutual parameter identification and state filtering, the development of the stable versions of the proposed filtering algorithms, which are robust to the imprecise knowledge of the system parameters and the outliers in the observations, looks real.Another limitation preventing the application of the proposed filtering algorithms is the low intensity of the observable MPPs.One can neutralize the problem by modifying the filtering algorithm to process the multi-scale counting observations.The high-frequency part can be processed by the proposed algorithm based on the CLT.The observation components with the low frequency can be processed directly, without preliminary sampling, similar to the approach suggested in [23,42].
The results of the investigations seem accomplished.However, they can be a starting point for the research directions.
The considered model looks simple but is flexible enough to describe the price evolution in the contemporary financial markets.Hence, the first area of the prospective investigation could be the parameter identification in the model ( 1), ( 2), (20), and (22), given the actual financial data by the use of the corresponding identification algorithms [23,47].
The second research area includes the sensitivity analysis of the hedging strategies relating to the choice of the derivative ensemble, which completes the market.Actually, the essential condition permitting the market completion is the non-degeneracy of the matrix F(•), corresponding to the chosen derivative set (condition (vi) in Section 3.2).Depending on this set, F(•) may be an ill-conditioned matrix, and this fact could lead to computational problems during the reconfiguration of the hedging portfolio.Another issue is that F(t, S t ) represents a matrix-valued random process.The matrix F(t, S t ) can be close to degenerate on some time intervals and fixed trajectory S t .
Keeping in mind the arguments for the discrete-time nature of the available observations, we can treat the market (1), ( 2), (20), and ( 22) as a controllable stochastic dynamic system with the discrete-time observation complex ( 47)-(48), or (56)-(57).The considered market stays incomplete.The third research area includes the solution to the hedging problems as the optimal or robust control in the stochastic dynamic systems with incomplete information.The filtering estimates proposed in the paper prompt us to apply the separation principle in the solution to these control problems.
adapted standard Wiener process; -a(•, •) and b(•, •) are (N × 1)-and (N × N)-dimensional functions of the instant interest rate and volatility (here b(•) is a symmetric non-negative matrix-valued function, and the notation b 1/2 where {ε i } is a sequence of independent identically distributed lognormal random values with the parameters a LN (h) = 0 and σ LN (h) = 0.5h.C3.The combination of {S i } and indirect observations of the option price {F MC i }.The latter ones represent a chain with values in the set of the possible option prices {F ℓ (t i , S i )}.The observations possess the Markov property, given the trajectory Z.The conditional transition matrices are formed by the probabilities P F MC i

Figure 1
contains the plots of the following: -The precise price of the underlying security S t (indicated on the left axis); -The precise option price F t (indicated on the left axis); -The number of the market regime State No. (indicated on the right axis).

Figure 1 .
Figure 1.Exact prices of underlying and derivative securities.

Figure 2 .
Figure 2. Additive errors of derivative price observations.

Figure 3
Figure 3 presents the estimation results in the component-wise form:-The exact regime state Z t ; -The regime filtering estimate Z S t , calculated by observation complex C1;

Figure 5
Figure 5 contains the plots of the transformed observations.

Figure 6 .
Figure 6.The "theoretical" pdf p W (w) and real histogram.

Figure 6
Figure 6 contains the plot of the pdf p W (w) in comparison with the 3D histogram of the actual observations{(W 1 i , W 2 i )}.The plots demonstrate a remarkable similarity.Note that the theoretical pdf is not a single Gaussian but is a mixture of them.The point is that the distribution parameters of the different modes are very close to each other, so the Gaussian modes visually differ little.Figure6demonstrates some asymmetry in the histogram and the theoretic density.Despite the similarity of the parameters, the proposed filtering algorithm provides high estimation performance.Figure7presents the estimation results:

Figure 7 .
Figure 7. Filtering results, obtained by use of multivariate point observations.

Funding:
The paper was published with the partial financial support of the Ministry of Education and Science of the Russian Federation as part of the program of the Moscow Center for Fundamental and Applied Mathematics under the agreement No. 075-15-2022-284.Data Availability Statement: Data are contained within the article.