Quasi-Lower C² Functions and Their Application to Nonconvex Variational Problems

Abstract

This study presents a novel category of nonconvex functions in Banach spaces, referred to as quasi-lower $C^2$ functions on nonempty closed sets. We establish the existence of solutions for nonconvex variational problems involving quasi-lower $C^2$ functions defined in Banach spaces. To illustrate the applicability of our findings, an example is provided in $L^p$ spaces.

1. Introduction

Let $f:X\to \mathbb{R}\cup \{+\infty \}$ be an extended real-valued function defined on a Banach space X. Function f is classified as convex if, for any $x,y\in domf:=\{x\in X:f(x)<+\infty \}$ and $t\in [0,1]$, the following inequality holds:

$$f(tx+(1-t\left)y\right)\le tf\left(x\right)+(1-t)f\left(y\right).$$

Similarly, a nonempty subset $S\subseteq X$ is convex if, for any $x,y\in S$ and $t\in [0,1]$, the relation $tx+(1-t)y\in S$ is satisfied.

It is straightforward to confirm that if f is a convex function, then all its level sets, defined as $[f\le \alpha ]:=\{x\in X:f(x)\le \alpha \}$ for any $\alpha \in \mathbb{R}$, are convex. This property of convex level sets has significant applications in optimization and economics, as detailed in [1,2].

Interestingly, some nonconvex functions also exhibit convex level sets. For example, consider the function $f\left(x\right)=\sqrt{\left|x\right|}$ on $X=\mathbb{R}$. While f is not convex on X, all its level sets $[f\le \alpha ]$ (for $\alpha \in \mathbb{R}$) are convex in X.

This observation inspired the definition of a broader class of functions known as quasi-convex functions (previously referred to as generalized convexity; see [2,3]). Formally, a function $f:X\to \mathbb{R}\cup \{+\infty \}$ is quasi-convex if, for any $x,y\in domf$ and $t\in [0,1]$, the following condition holds:

$$f(tx+(1-t\left)y\right)\le max\left\{f\right(x),f(y\left)\right\}.$$

This class has been extensively studied, with numerous papers and books dedicated to it; some notable references include [3,4,5,6,7,8,9,10]. Another notable extension of the convexity concept is the class of lower $C^2$ functions, initially introduced in finite-dimensional spaces by Rockafellar [11]. A function f defined on $X={\mathbb{R}}^n$ is called lower $C^2$ on an open set $\mathcal{O}$ if, for any point $\overline{x}\in \mathcal{O}$, there exists a neighborhood $\mathcal{V}$ of $\overline{x}$ where f can be expressed as the pointwise maximum of $C^2$ functions $\left(f_t\right)$ (with $t\in T$, a compact index set), i.e., $f\left(x\right)={max}_{t\in T}\left\{f_t\left(x\right)\right\}$ for all $x\in \mathcal{V}$. Moreover, the functions $f_t\left(x\right)$ and their gradients $\nabla f_t\left(x\right)$ must depend continuously on $(t,x)\in T\times \mathcal{V}$. A simple example is $f\left(x\right)=max\{f_1\left(x\right),f_2\left(x\right),\dots ,f_m\left(x\right)\}$, where each $f_i\left(x\right)$ belongs to the $C^2$ class.

A key characterization of lower $C^2$ functions, established in [11], states that a function f is lower $C^2$ on $\mathcal{O}$ if and only if, for every $\overline{x}\in \mathcal{O}$, there exist $\rho \ge 0$ and an open convex neighborhood $\mathrm{\Omega }$ of $\overline{x}$ such that the function $f(·)+{\displaystyle \frac{\rho }{2}}{\parallel ·\parallel }^2$ is convex on $\mathrm{\Omega }$.

This concept has been extended to infinite-dimensional Hilbert spaces (see [12,13]) using the same characterization. Specifically, a function $f:\mathbb{H}\to \mathbb{R}$ defined on a Hilbert space $\mathbb{H}$ is lower $C^2$ around $\overline{x}\in \mathbb{H}$ if there exist $\rho \ge 0$ and an open convex neighborhood $\mathrm{\Omega }$ of $\overline{x}$ such that $f(·)+{\displaystyle \frac{\rho }{2}}{\parallel ·\parallel }^2$ is convex on $\mathrm{\Omega }$. To incorporate uniformity over a set, the definition becomes the following: f is uniformly lower $C^2$ on an open convex set $\mathrm{\Omega }$ if there exists $\rho \ge 0$ such that $f(·)+{\displaystyle \frac{\rho }{2}}{\parallel ·\parallel }^2$ is convex on $\mathrm{\Omega }$.

This uniformly lower $C^2$ class includes a broad spectrum of functions. For instance, all lower semicontinuous convex functions belong to this class. It also includes the distance function $d_S$ associated with a uniformly V-prox-regular set S (see [14]). To adapt this framework to Banach spaces for nonconvex variational inequality problems, we redefine the concept using the functional V. A function f is said to be uniformly lower $C^2$ on an open convex set $\mathrm{\Omega }$ if there exists $\rho \ge 0$ such that $f(·)+{\displaystyle \frac{\rho }{2}}V(x^{\ast };·)$ is convex on $\mathrm{\Omega }$ for any $x^{\ast }\in X^{\ast }$. This definition aligns with the Hilbert space case when $V(x^{\ast };·)={\parallel ·\parallel }^2$ (see Remark 1).

Replacing convexity with quasi-convexity in this definition leads to a new class of nonconvex functions. Specifically, for an open convex set $\mathrm{\Omega }\subseteq X$, a function $f:X\to \mathbb{R}$ is said to be uniformly quasi-lower $C^2$ on $\mathrm{\Omega }$ if there exists $\rho \ge 0$ such that $f(·)+{\displaystyle \frac{\rho }{2}}V(x^{\ast };·)$ is quasi-convex on $\mathrm{\Omega }$ for any $x^{\ast }\in X^{\ast }$.

This new class includes all lower semicontinuous convex or quasi-convex functions, which are uniformly quasi-lower $C^2$ on any open convex subset of their domain with $\rho =0$. Similarly, all uniformly lower $C^2$ functions are uniformly quasi-lower $C^2$ with the same $\rho$. Further exploration of this class and its properties will be discussed in future works. The current paper focuses on investigating the existence of solutions to nonconvex variational problems involving uniformly quasi-lower $C^2$ functions.

The paper is organized as follows: Section 2 reviews foundational concepts and results necessary for the main proofs. Section 3 presents the primary results, and Section 4 demonstrates their application in $L^p$ spaces.

2. Preliminaries

In this paper, X will always refer to a Banach space, with $X^{\ast }$ as its dual space, and $⟨·,·⟩$ denoting the duality pairing between X and $X^{\ast }$. Denote the closed unit balls in X and $X^{\ast }$ by $\mathbb{B}$ and ${\mathbb{B}}_{\ast }$, respectively. The normalized duality mapping $J:X\to 2^{X^{\ast }}$ is defined as

$$J\left(x\right)=\{x^{\ast }\in X^{\ast }:⟨x^{\ast },x⟩=\parallel x{\parallel }^2=\parallel x^{\ast }{\parallel }^2\},$$

where $\parallel ·\parallel$ represents the norm on both X and $X^{\ast }$. Similarly, we define $J^{\ast }$ on $X^{\ast }$. The properties of J and $J^{\ast }$ are well-documented; for further details, refer to [15,16].

Definition 1. 

For a fixed closed subset $S\subseteq X$, a function $f:S\to \mathbb{R}\cup \{+\infty \}$, and a given $\lambda >0$, we define the functional $G_{\lambda ,f}^V:X^{\ast }\times S\to \mathbb{R}\cup \{+\infty \}$ by

$$G_{\lambda ,f}^V(x^{\ast },x)=f\left(x\right)+{\displaystyle \frac{1}{2\lambda }}V(x^{\ast },x),\forall x^{\ast }\in X^{\ast },x\in S,$$

where $V(x^{\ast },x)=\parallel x^{\ast }{\parallel }^2-2⟨x^{\ast },x⟩+{\parallel x\parallel }^2$.

In the special case where X is a Hilbert space (i.e., $X^{\ast }=X$), the functional V simplifies to

$$V(x^{\ast },x)={\parallel x^{\ast }-x\parallel }^2.$$

This highlights the utility of using V, as the square of the norm cannot generally be expressed in the form of V in Banach spaces.

Using $G_{\lambda ,f}^V$, we define the V-Moreau envelope of f associated with S (see [17]) as

$$e_{\lambda ,S}^{V}f\left(x^{\ast }\right):=\underset{s\in S}{inf}{G}_{\lambda ,f}^V(x^{\ast },s),\forall x^{\ast }\in X^{\ast }.$$

The study of V-Moreau envelopes has been the subject of a recent paper by the author (see [17]). We define the generalized $(f,\lambda )$-projection of $x^{\ast }\in X^{\ast }$ on S as

$${\pi }_S^{f,\lambda }\left(x^{\ast }\right):=arg\underset{s\in S}{min}{G}_{\lambda ,f}^V(x^{\ast },s),$$

where $G_{\lambda ,f}^V(x^{\ast },s)$ is as defined previously, and ${\pi }_S^{f,\lambda }\left(x^{\ast }\right)$ represents the set of all points in S that minimize the functional $G_{\lambda ,f}^V$.

We recall from [17] the definition of $N_{f,\lambda }^{\pi }(S;x)$ for a given closed set S, $\lambda >0$, and a given lower semicontinuous function f, as follows:

$$N_{f,\lambda }^{\pi }(S;\overline{x}):=\{x^{\ast }\in X^{\ast }:\mathrm{such}\mathrm{that}\overline{x}\in {\pi }_S^{f,\lambda }(J\left(\overline{x}\right)+\lambda x^{\ast })\}.$$

(1)

We also recall many useful concepts and definitions as follows:

Definition 2. 

• V-Proximal Subdifferential:
  Let $f:X\to \mathbb{R}\cup \{+\infty \}$ be a lower semicontinuous function, and $x\in X$ such that f is finite. The V-proximal subdifferential ${\partial }^{\pi }f$ of f at x (see [18]) is defined as $x^{\ast }\in {\partial }^{\pi }f\left(x\right)$ if there exist $\sigma >0$ and $\delta >0$:

  $$⟨x^{\ast },x^{\prime }-x⟩\le f\left(x^{\prime }\right)-f\left(x\right)+\sigma V(J\left(x\right),x^{\prime }),\forall x^{\prime }\in x+\delta \mathbb{B}.$$

  (2)
  If f is locally Lipschitz at $\overline{x}$, then ${\partial }^{\pi }f\left(\overline{x}\right)\subset L{\mathbb{B}}_{\ast }$ (see [19]).
• V-Proximal Normal Cone: For a nonempty closed subset S in X at $x\in S$, the V-proximal normal cone is defined as the V-proximal subdifferential of the indicator function ${\psi }_S$:

  $$N^{\pi }(S;x)={\partial }^{\pi }{\psi }_S\left(x\right).$$

  (3)
  Thus, $x^{\ast }\in N^{\pi }(S;x)$ if there exist $\sigma >0$ and $\delta >0$ such that the following is true:

  $$⟨x^{\ast },x^{\prime }-x⟩\le \sigma V(J\left(x\right),x^{\prime }),\forall x^{\prime }\in (x+\delta \mathbb{B})\cap S.$$

  (4)
  A global characterization of $N^{\pi }(S;x)$ (see [18]) is given as follows:

  $$x^{\ast }\in N^{\pi }(S;x)\iff \exists \sigma >0\mathit{such}\mathit{that}⟨x^{\ast },x^{\prime }-x⟩\le \sigma V(J\left(x\right),x^{\prime }),\forall x^{\prime }\in S.$$

  (5)
  Also, note that $N^{\pi }(S;x)$ is characterized (see [19]) via ${\pi }_S$ as follows:

  $$x^{\ast }\in N^{\pi }(S;\overline{x})\iff \exists \alpha >0,\mathit{such}\mathit{that}\overline{x}\in {\pi }_S(J\overline{x}+\alpha x^{\ast }).$$
• Limiting V-Proximal Subdifferential: The limiting V-proximal subdifferential ${\partial }^{L\pi }f$ of f at $\overline{x}\in domf$ is defined as follows:

  $${\partial }^{L\pi }f\left(\overline{x}\right)=\left\{w-\underset{n}{lim}{x}_n^{\ast }:x_n^{\ast }\in {\partial }^{\pi }f\left(x_n\right),x_n{\to }^f\overline{x},f\left(x_n\right)\to f\left(\overline{x}\right)\right\}.$$

  (6)
• Limiting V-Proximal Normal Cone: The limiting V-proximal normal cone is defined as follows:

  $$N^{L\pi }(S;\overline{x})=\{w-\underset{n}{lim}{x}_n^{\ast }:x_n^{\ast }\in N^{\pi }(S;x_n),x_n{\to }^S\overline{x}\}.$$

  (7)
• V-Proximal Trustworthy Spaces: A Banach space X is called a V-proximal trustworthy space if for any $ϵ>0$, any two functions $g,h:X\to \mathbb{R}\cup \{\infty \}$, and any $z\in X$ such that g is lower semicontinuous and h is locally Lipschitz at z, the following fuzzy sum rule holds:

  $${\partial }^{\pi }(g+h)\left(z\right)\subset \bigcup \left\{{\partial }^{\pi }g\left(v\right)+{\partial }^{\pi }h\left(w\right):v\in U_g(z,ϵ),w\in U_h(z,ϵ)\right\}+ϵ{\mathbb{B}}_{\ast },$$

  (8)

  where $U_g(z,ϵ)=\{v\in z+ϵ\mathbb{B}||g\left(v\right)-g\left(z\right)|<ϵ\}$ and $U_h(z,ϵ)=\{w\in z+ϵ\mathbb{B}||h\left(w\right)-h\left(z\right)|<ϵ\}$. It has been shown (see [18]) that all $L^p$ spaces (with $p\in (1,\infty )$) are V-proximal and trustworthy.
• Uniformly V-Prox-Regular Functions: A lower semicontinuous function $f:X\to \mathbb{R}\cup \{\infty \}$ is uniformly V-prox-regular over a nonempty set $S\subset domf$ (see [17]) if there exists $r>0$ such that for any $x\in S$ and $x^{\ast }\in {\partial }^{L\pi }f\left(x\right)$ the following is true:

  $$⟨x^{\ast },x^{\prime }-x⟩\le f\left(x^{\prime }\right)-f\left(x\right)+{\displaystyle \frac{1}{2r}}V(J\left(x\right),x^{\prime }),\forall x^{\prime }\in S.$$

  (9)
• Uniformly V-Prox-Regular Sets: A nonempty closed subset S in a reflexive smooth Banach space X is uniformly V-prox-regular if there exists $r>0$ such that for all $x\in S$ and $x^{\ast }\in N^{\pi }(S;x)$ (with $x^{\ast }\ne 0$):

  $$x\in {\pi }_S\left(J\left(x\right)+r{\displaystyle \frac{x^{\ast }}{\parallel x^{\ast }\parallel }}\right).$$

  (10)

We observe some important cases of the set $N_{f,\lambda }^{\pi }(S;x)$:

• When $f=0$, S is any closed subset in X, and $\lambda >0$ is any positive real number, we have the following:

  $$N_{f,\lambda }^{\pi }(S;x)\subset N_S^{\pi }\left(x\right).$$
  Its proof stems straightforwardly from Proposition 2 and the global characterization of $N_S^{\pi }\left(x\right)$ in (5).
• When S is assumed to be closed convex, and f is supposed to be lower semicontinuous convex over S, then the set $N_{f,\lambda }^{\pi }(S;x)$ does not depend on the parameter $\lambda$ and it has the following simpler form:

  $$\begin{array}{ccc}\hfill N_{f,\lambda }^{\pi }(S;\overline{x})& =& \{x^{\ast }\in X^{\ast }:⟨x^{\ast },x-\overline{x}⟩+f\left(\overline{x}\right)-f\left(x\right)\le 0,\forall x\in S\}.\hfill \end{array}$$
  Indeed, the set on right side is always a subset of $N_{f,\lambda }^{\pi }(S;\overline{x})$. So, we have to check the reverse inclusion. Let $x^{\ast }\in N_{f,\lambda }^{\pi }(S;\overline{x})$. By Proposition 2 we have

  $$\begin{array}{c}\hfill ⟨x^{\ast },x-\overline{x}⟩\le f\left(x\right)-f\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right);x),\forall x\in S.\end{array}$$

  (11)
  For any $z\in S$ and any $\delta \in (0,1)$, we set $z_{\delta }:=\overline{x}+\delta (z-\overline{x})\in S$ (by convexity of S). Thus, by convexity of f, we have by (11),

  $$\begin{array}{ccc}\hfill ⟨x^{\ast },\delta (z-\overline{x})⟩& \le & f(\overline{x}+\delta (z-\overline{x}))-f\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right);\overline{x}+\delta (z-\overline{x}))\hfill \\ \\ & \le & (1-\delta )f\left(\overline{x}\right)+\delta f\left(z\right)-f\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right),\overline{x}+\delta (z-\overline{x}))\hfill \\ \\ & \le & \delta [f\left(z\right)-f\left(\overline{x}\right)]+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right),\overline{x}+\delta (z-\overline{x})).\hfill \end{array}$$
  Applying division by $\delta$ to the last inequality and taking the limit when $\delta \downarrow 0$ yields

  $$\begin{array}{ccc}\hfill ⟨x^{\ast };z-\overline{x}⟩& \le & f\left(z\right)-f\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}\underset{\delta \downarrow 0}{lim}{\displaystyle \frac{V(J\left(\overline{x}\right),\overline{x}+\delta (z-\overline{x}))}{\delta }}.\hfill \end{array}$$
  Using the fact that V is differentiable with respect to the 2^nd variable with ${\nabla }_{x}V(z^{\ast },z)=2(J\left(z\right)-z^{\ast })$ and the fact that $V(J\left(\overline{x}\right),\overline{x}))=0$, we can write

  $$\begin{array}{ccc}\hfill \underset{\delta \downarrow 0}{lim}{\displaystyle \frac{V(J\left(\overline{x}\right),\overline{x}+\delta (z-\overline{x}))}{\delta }}& =& \underset{\delta \downarrow 0}{lim}{\displaystyle \frac{V(J\left(\overline{x}\right),\overline{x}+\delta (z-\overline{x}))-V(J\left(\overline{x}\right),\overline{x}))}{\delta }}\hfill \\ \\ & =& ⟨{\nabla }_{x}V(J\left(\overline{x}\right),\overline{x});z-\overline{x}⟩\hfill \\ \\ & =& ⟨2(J\left(\overline{x}\right)-J\left(\overline{x}\right);z-\overline{x}⟩=0.\hfill \end{array}$$
  Therefore, we get

  $$\begin{array}{c}\hfill ⟨x^{\ast };z-\overline{x}⟩\le f\left(z\right)-f\left(\overline{x}\right),\forall z\in S;\end{array}$$

  that is, we have proved the inclusion

  $$N_{f,\lambda }^{\pi }(S;\overline{x})\subset \{x^{\ast }\in X^{\ast }:⟨x^{\ast },x-\overline{x}⟩+f\left(\overline{x}\right)-f\left(x\right)\le 0,\forall x\in S\}.$$

To wrap up this section, we state, without proof, two important results in the following proposition that are needed for our present study. Their proofs are given in [18,20], respectively.

Proposition 1. 

• Suppose that X is V-proximal and trustworthy, $f_1$ is lower semicontinuous at $x_0\in dom{f}_1$, and $f_2$ is locally Lipschitz at $x_0$. Then, we have the exact sum rule for the limiting V-proximal subdifferential:

  $${\partial }^{L\pi }(f_1+f_2)\left(x_0\right)\subset {\partial }^{L\pi }{f}_1\left(x_0\right)+{\partial }^{L\pi }{f}_2\left(x_0\right).$$
• Suppose that S is a bounded set in a q-uniformly convex space X. Then, there is a positive constant $\alpha >0$ depending on S such that

  $${\parallel y-x\parallel }^q\le \alpha V(J\left(x\right),y),foranyx,y\in S.$$

3. Main Results

Variational inequality theory is an important field of mathematical science that focuses on solving general equilibrium problems, with broad applications in operations research, economic problems, industry, as well as in engineering and physical sciences. Recently, numerous research papers have explored both the theoretical aspects and practical applications of this branch. Significant links have been established with major areas in both pure and applied sciences (see, for instance, [21,22,23,24] and related references).

A common formulation of the main problem in variational inequality theory often encountered in literature is as follows:

$$\begin{array}{c}\hfill \mathrm{Find}\overline{x}\in S\mathrm{and}\overline{y}\in F\left(\overline{x}\right)\mathrm{such}\mathrm{that}⟨\overline{y},x-\overline{x}⟩\ge 0,\forall x\in S,\end{array}$$

(12)

where S is a nonempty closed convex subset of a Hilbert space $\mathrm{ℍ}$ and $F:\mathrm{ℍ}\to 2^{\mathrm{ℍ}}$ is a set-valued mapping.

In Bounkhel et al. [25], the authors extended (12) from the convex case to the nonconvex case in Hilbert spaces by reformulating (12) in the following inclusion form:

$$\begin{array}{c}\hfill \mathrm{Find}\overline{x}\in S\mathrm{so}\mathrm{that}F\left(\overline{x}\right)\cap \left[-N^{conv.}(S;\overline{x})\right]\ne \varnothing ,\end{array}$$

(13)

where $N^{conv.}(S;\overline{x})$ denotes the normal cone of S at $\overline{x}$ in the sense of convex analysis. This equivalent form of (12) permits us to extend it from the convex case to the nonconvex case by taking any nonconvex normal cone (for instance: Fréchet, Clarke, proximal, …) instead of $N^{conv.}(S;x)$. Also, for its extension from Hilbert spaces to Banach spaces, this variational problem (13) has been used (see [17,19]).

The existence of solutions for (12) has been proved in Hilbert spaces (see for instance [12]) and in Banach spaces (see for instance [26,27,28,29,30]). Also, the construction of numerical schemes converging to solutions of (12) has been studied in the Hilbert space setting in many papers (see for instance [25,31,32,33,34,35,36,37]). As far as we know, no research has been done on the existence of solutions of variational problems (13) in the nonconvex case on Banach spaces.

In the present work, we intend to prove the existence of solutions for the following more general nonconvex variational problems:

$$\mathrm{Find}\overline{x}\in S\mathrm{so}\mathrm{that}:F\left(\overline{x}\right)\cap -N_{f,\lambda }^{\pi }(S;\overline{x})\ne \varnothing ,$$

(14)

where f is a lower semicontinuous function and $\lambda >0$ and $N_{f,\lambda }^{\pi }(S;\overline{x})$ are defined in (1). This problem was introduced and partially studied in [17]. It covers the following important cases:

• When $f=0$ and X represent the Hilbert space, the problem (14) coincides with the problems studied in [12,19].
• When f and S are both convex, the problem (14) coincides with the problems studied in [38,39].
• When $f=0$, S is a nonconvex closed set, and X is a reflexive Banach space, the problem (14) coincides with the problems studied in [17].

We start our study by proving the following characterization of $N_{f,\lambda }^{\pi }(S;\overline{x})$.

Proposition 2. 

Assume that f is a lower semicontinuous function over a nonempty closed set S. Then, for any $\overline{x}\in S$ and any $\lambda >0$, we have $x^{\ast }\in N_{f,\lambda }^{\pi }(S;\overline{x})$ if and only if

$$\begin{array}{c}\hfill ⟨x^{\ast },x-\overline{x}⟩\le f\left(x\right)-f\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right);x),forallx\in S.\end{array}$$

Proof. 

Let $x^{\ast }\in N_{f,\lambda }^{\pi }(S;\overline{x})$. By definition, we have $\overline{x}\in {\pi }_S^{f,\lambda }(J\left(\overline{x}\right)+\lambda x^{\ast })$, and so by definition of ${\pi }_S^{f,\lambda }$, we have

$$e_{\lambda ,S}^{V}f(J\left(\overline{x}\right)+\lambda x^{\ast })=G_{\lambda ,f}^V(J\left(\overline{x}\right)+\lambda x^{\ast };\overline{x});\mathrm{that}\mathrm{is},$$

$$\begin{array}{ccc}\hfill f\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right)+\lambda x^{\ast };\overline{x})& \le & f\left(x\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right)+\lambda x^{\ast };x),\forall x\in S.\hfill \end{array}$$

Hence,

$$\begin{array}{ccc}\hfill f\left(\overline{x}\right)-f\left(x\right)& \le & {\displaystyle \frac{1}{2\lambda }}\left[V(J\left(\overline{x}\right)+\lambda x^{\ast };x)-V(J\left(\overline{x}\right)+\lambda x^{\ast };\overline{x})\right]\hfill \\ \\ & \le & {\displaystyle \frac{1}{2\lambda }}\left[{\parallel x\parallel }^2-{\parallel \overline{x}\parallel }^2-2⟨J\left(\overline{x}\right)+\lambda x^{\ast };x-\overline{x}⟩\right]\hfill \\ \\ & \le & {\displaystyle \frac{1}{2\lambda }}\left[{\parallel x\parallel }^2-{\parallel \overline{x}\parallel }^2-2⟨J\left(\overline{x}\right);x⟩+2⟨\lambda x^{\ast };x-\overline{x}⟩\right]\hfill \\ \\ & \le & {\displaystyle \frac{1}{2\lambda }}\left[V(J\left(\overline{x}\right);x)-2⟨\lambda x^{\ast };x-\overline{x}⟩\right].\hfill \end{array}$$

Therefore,

$$\begin{array}{ccc}\hfill ⟨x^{\ast };x-\overline{x}⟩& \le & f\left(x\right)-f\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right);x),\forall x\in S,\hfill \end{array}$$

and so the first direction of the equivalence is proven.

• Conversely, we assume that $x^{\ast }$ satisfies the last inequality. Then,

$$\begin{array}{ccc}\hfill ⟨x^{\ast };x-\overline{x}⟩& \le & f\left(x\right)-f\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right);x),\forall x\in S.\hfill \end{array}$$

Using the fact that

$$V(J\left(\overline{x}\right);x)=V(J\left(\overline{x}\right)+\lambda x^{\ast };x)-V(J\left(\overline{x}\right)+\lambda x^{\ast };\overline{x})+2⟨\lambda x^{\ast };x-\overline{x}⟩,$$

we obtain

$$\begin{array}{ccc}\hfill ⟨x^{\ast };x-\overline{x}⟩+f\left(\overline{x}\right)-f\left(x\right)& \le & {\displaystyle \frac{1}{2\lambda }}[V(J\left(\overline{x}\right)+\lambda x^{\ast };x)-V(J\left(\overline{x}\right)+\lambda x^{\ast };\overline{x})\hfill \\ \\ & +& 2⟨\lambda x^{\ast };x-\overline{x}⟩],\forall x\in S.\hfill \end{array}$$

Hence,

$$\begin{array}{ccc}\hfill f\left(\overline{x}\right)-f\left(x\right)& \le & {\displaystyle \frac{1}{2\lambda }}\left[V(J\left(\overline{x}\right)+\lambda x^{\ast };x)-V(J\left(\overline{x}\right)+\lambda x^{\ast };\overline{x})\right]\forall x\in S,\hfill \end{array}$$

and so

$$\begin{array}{c}\hfill f\left(x\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right)+\lambda x^{\ast };x)\ge f\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right)+\lambda x^{\ast };\overline{x}),\forall x\in S.\end{array}$$

Thus,

$$\begin{array}{ccc}\hfill G_{\lambda ,f}^V(J\left(\overline{x}\right)+\lambda x^{\ast };x)& =& f\left(x\right)+G{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right)+\lambda x^{\ast };x)\hfill \\ \\ & \ge & f\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right)+\lambda x^{\ast };\overline{x})\hfill \\ \\ & =& G_{\lambda ,f}^V(J\left(\overline{x}\right)+\lambda x^{\ast };\overline{x}),\forall x\in S;\hfill \end{array}$$

that is,

$$\begin{array}{ccc}\hfill G_{\lambda ,f}^V(J\left(\overline{x}\right)+\lambda x^{\ast };\overline{x})& =& \underset{x\in S}{inf}{G}_{\lambda ,f}^V(J\left(\overline{x}\right)+\lambda x^{\ast };x)=e_{\lambda ,S}^{V}f(J\left(\overline{x}\right)+\lambda x^{\ast }).\hfill \end{array}$$

This means that $\overline{x}\in {\pi }_S^{f,\lambda }(J\left(\overline{x}\right)+\lambda x^{\ast })$ and, equivalently, it means that $x^{\ast }\in N_{f,\lambda }^{\pi }(S;\overline{x})$; therefore, the proof is complete. □

We use this proposition two prove the following characterizations of (14).

Proposition 3. 

Suppose that f is a lower semicontinuous function over a nonempty closed set S and let $\overline{x}\in S$. Then, for any $\lambda >0$ we determine the equivalence between the following assertions:

(i) 

$\overline{x}$ is a solution of (14);

(ii) 

there exists some $x^{\ast }\in F\left(\overline{x}\right)$ such that $\overline{x}\in {\pi }_S^{f,\lambda }(J\left(\overline{x}\right)-\lambda x^{\ast })$;

(iii) 

there exists some $x^{\ast }\in F\left(\overline{x}\right)$ such that

$$\begin{array}{c}\hfill ⟨x^{\ast },\overline{x}-x⟩\le f\left(x\right)-f\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right);x),forallx\in S.\end{array}$$

Proof. 

The proof is a combination between the definition of $N_{f,\lambda }^{\pi }(S;\overline{x})$ and Proposition 2. □

Assume now that f is a uniformly V-prox-regular function over S with $r_1>0$ and that the closed set S is a uniformly V-prox-regular with $r_2>0$. We establish the following characterization of $N_{f,\lambda }^{\pi }(S;\overline{x})$ under the uniform V-prox-regularity of both f and S.

Proposition 4. 

Assume that f is a lower semicontinuous uniformly V-prox-regular function over S with constant $r_1>0$ and that S is closed and uniformly V-prox-regular with constant $r_2>0$. Then, for any $\overline{x}\in S$ and any $\lambda \in (0,{\displaystyle \frac{r_1{r}_2}{r_1+r_2}})$, and the set $N_{f,\lambda }^{\pi }(S;\overline{x})$ is independent of λ and has the following form

$$\begin{array}{c}\hfill N_{f,\lambda }^{\pi }(S;\overline{x})={\partial }^{L\pi }f\left(\overline{x}\right)+N^{L\pi }(S;\overline{x}).\end{array}$$

Proof. 

Let $x^{\ast }\in N_{f,\lambda }^{\pi }(S;\overline{x})$. Then, $\overline{x}\in {\pi }_S^{f,\lambda }(J\left(\overline{x}\right)+\lambda x^{\ast })$, and so by the definition of ${\pi }_S^{f,\lambda }$, we have

$$f\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right)+\lambda x^{\ast },\overline{x})\le f\left(x\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right)+\lambda x^{\ast },x),\forall x\in S.$$

Hence,

$$\begin{array}{ccc}\hfill f\left(x\right)-f\left(\overline{x}\right)& \ge & {\displaystyle \frac{1}{2\lambda }}\left[V(J\left(\overline{x}\right)+\lambda x^{\ast },\overline{x})-V(J\left(\overline{x}\right)+\lambda x^{\ast },x)\right]\hfill \end{array}$$

$$\begin{array}{ccc}& \ge & {\displaystyle \frac{1}{2\lambda }}\left[2⟨J\left(\overline{x}\right)+\lambda x^{\ast };x-\overline{x}⟩+\parallel \overline{x}{\parallel }^2-{\parallel x\parallel }^2\right]\hfill \end{array}$$

$$\begin{array}{ccc}& \ge & {\displaystyle \frac{1}{2\lambda }}\left[2⟨J\left(\overline{x}\right);x-\overline{x}⟩+\parallel \overline{x}{\parallel }^2-{\parallel x\parallel }^2\right]+⟨x^{\ast };x-\overline{x}⟩.\hfill \end{array}$$

(15)

Observe that

$$\begin{array}{ccc}\hfill 2⟨J\left(\overline{x}\right);x-\overline{x}⟩+\parallel \overline{x}{\parallel }^2-{\parallel x\parallel }^2& =& 2⟨J\left(\overline{x}\right);x⟩+\parallel \overline{x}{\parallel }^2-{\parallel x\parallel }^2-2⟨J\left(\overline{x}\right);\overline{x}⟩\hfill \\ \\ & =& 2⟨J\left(\overline{x}\right);x⟩+\parallel \overline{x}{\parallel }^2-{\parallel x\parallel }^2-2{\parallel \overline{x}\parallel }^2\hfill \\ \\ & =& 2⟨J\left(\overline{x}\right);x⟩-\parallel \overline{x}{\parallel }^2-{\parallel x\parallel }^2\hfill \\ \\ & =& -\left[\parallel \overline{x}{\parallel }^2-2⟨J\left(\overline{x}\right);x⟩+{\parallel x\parallel }^2\right]\hfill \\ \\ & =& -V(J\left(\overline{x}\right),x).\hfill \end{array}$$

Hence, the inequality (15) becomes

$$\begin{array}{ccc}\hfill f\left(x\right)-f\left(\overline{x}\right)& \ge & -{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right),x)+⟨x^{\ast };x-\overline{x}⟩,\forall x\in S.\hfill \end{array}$$

Thus,

$$⟨x^{\ast };x-\overline{x}⟩\le [f+{\psi }_S]\left(x\right)-[f+{\psi }_S]\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right),x),\forall x\in X.$$

This ensures by definition of ${\partial }^{\pi }$ and the exact sum rule of the limit generalized proximal subdifferential that

$$\begin{array}{ccc}\hfill x^{\ast }& \in & {\partial }^{\pi }[f+{\psi }_S]\left(x\right)\subset {\partial }^{L\pi }[f+{\psi }_S]\left(x\right)\hfill \\ \\ & \subset & {\partial }^{L\pi }f\left(x\right)+{\partial }^{L\pi }{\psi }_S\left(x\right)\subset {\partial }^{L\pi }f\left(x\right)+N^{L\pi }(S;x);\hfill \end{array}$$

hence, $x^{\ast }\in {\partial }^{L\pi }f\left(x\right)+N^{L\pi }(S;x)$. Conversely, let $x^{\ast }\in {\partial }^{L\pi }f\left(x\right)+N^{L\pi }(S;x)$. There exists $x_1^{\ast }\in {\partial }^{L\pi }f\left(x\right)$ and $x_2^{\ast }\in N^{L\pi }(S;x)$ such that $x^{\ast }=x_1^{\ast }+x_2^{\ast }$. We use the uniform V-prox-regularity of f over S with constant $r_1>0$ and the uniform V-prox-regularity of S with constant $r_2>0$ to write

$$\begin{array}{c}\hfill ⟨x_1^{\ast },x-\overline{x}⟩\le f\left(x\right)-f\left(\overline{x}\right)+{\displaystyle \frac{1}{2r_1}}V(J\left(\overline{x}\right);x),\mathrm{for}\mathrm{all}x\in S,\end{array}$$

(16)

and

$$\begin{array}{c}\hfill ⟨x_2^{\ast },x-\overline{x}⟩\le {\displaystyle \frac{1}{2r_2}}V(J\left(\overline{x}\right);x),\mathrm{for}\mathrm{all}x\in S.\end{array}$$

(17)

Adding these two inequalities and combining them with the assumption $\lambda \in (0,{\displaystyle \frac{r_1{r}_2}{r_1+r_2}})$ yields

$$\begin{array}{ccc}\hfill ⟨x^{\ast };x-\overline{x}⟩=⟨x_1^{\ast }+x_2^{\ast };x-\overline{x}⟩& \le & f\left(x\right)-f\left(\overline{x}\right)+{\displaystyle \frac{1}{2r_1}}V(J\left(\overline{x}\right);x)+{\displaystyle \frac{1}{2r_2}}V(J\left(\overline{x}\right);x)\hfill \\ \\ & \le & f\left(x\right)-f\left(\overline{x}\right)+{\displaystyle \frac{r_1+r_2}{2r_1{r}_2}}V(J\left(\overline{x}\right);x)\hfill \\ \\ & \le & f\left(x\right)-f\left(\overline{x}\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right);x),\mathrm{for}\mathrm{all}x\in S.\hfill \end{array}$$

Using Proposition 2, this ensures that $x^{\ast }\in N_{f,\lambda }^{\pi }(S;\overline{x})$. This completes the proof of the proposition. □

An examination of our proof in the previous proposition reveals that the inclusion $N_{f,\lambda }^{\pi }(S;\overline{x})\subset {\partial }^{L\pi }f\left(x\right)+N^{L\pi }(S;x)$ is always true without any additional regularity assumptions and it is true for any $\lambda >0$. However, the reverse inclusion ${\partial }^{L\pi }f\left(x\right)+N^{L\pi }(S;x)\subset N_{f,\lambda }^{\pi }(S;\overline{x})$ needs the uniform V-prox-regularity of f and S, and it is true only for the values of $\lambda$ in the interval $(0,{\displaystyle \frac{r_1{r}_2}{r_1+r_2}})$. Using this characterization of $N_{f,\lambda }^{\pi }(S;\overline{x})$ under the uniform V-prox-regularity of both f and f, we add one more characterization of (14) to the assertions in Proposition 3.

Proposition 5. 

Suppose that f is a lower semicontinuous uniformly V-prox-regular function over S with $r_1>0$ and that S is closed and uniformly V-prox-regular with $r_2>0$. Then, for any $\overline{x}\in S$ and any $\lambda \in (0,{\displaystyle \frac{r_1{r}_2}{r_1+r_2}})$, the following assertions are equivalent:

(i) 

$\overline{x}$ is a solution of (14);

(ii) 

$0\in F\left(\overline{x}\right)+{\partial }^{L\pi }f\left(\overline{x}\right)+N^{L\pi }(S;\overline{x})$.

The first corollary of Proposition 3 is the following.

Corollary 1. 

Suppose that S is a nonempty closed convex set and that f is a uniformly V-prox-regular function over S with $r_1>0$. Then, for any $\lambda \in (0,r_1)$, we have $\overline{x}$ as a solution of (14), which is equivalent to $0\in F\left(\overline{x}\right)+{\partial }^{L\pi }f\left(\overline{x}\right)+N^{Conv.}(S;\overline{x})$, and both are equivalent to the variational inequality: there exists $x^{\ast }\in F\left(\overline{x}\right)$ such that

$$⟨x^{\ast },x-\overline{x}⟩+f\left(x\right)-f\left(\overline{x}\right)\ge {\displaystyle \frac{1}{2\lambda }}V(J\left(\overline{x}\right);x),forallx\in S.$$

(18)

Proof. 

It is directly derived from the fact that the convexity of S is equivalent to the uniform V-prox-regularity with $r_2=+\infty$; so, ${\displaystyle \frac{1}{r_2}}=0$, and then, we are done. □

By taking both data f and S to be both convex, we get the following corollary proven in Proposition 4.1 in Bounkhel [17].

Corollary 2. 

Suppose that f is a lower semicontinuous convex over a closed convex set S. Then, for any $\lambda >0$, we have $\overline{x}$ a a solution of (14), which is equivalent to $0\in F\left(\overline{x}\right)+{\partial }^{Conv.}f\left(\overline{x}\right)+N^{Conv.}(S;\overline{x})$, and both are equivalent to the convex variational inequality: there is $x^{\ast }\in F\left(\overline{x}\right)$ so that

$$⟨x^{\ast },x-\overline{x}⟩+f\left(x\right)-f\left(\overline{x}\right)\ge 0,forallx\in S.$$

Here, ${\partial }^{Conv.}f\left(\overline{x}\right)$ denotes the convex subdifferential of f at $\overline{x}$ (see [12]).

Proof. 

It follows from the fact that the convexity of S is equivalent to the uniform V-prox regularity with $r_2=+\infty$, and the convexity of f is equivalent to $r_1=+\infty$ so ${\displaystyle \frac{1}{r_2}}+{\displaystyle \frac{1}{r_2}}=0$. Then, we are done. □

Now, we introduce our main class of nonconvex functions that will be utilized to establish the existence of solutions of the generalized nonconvex variational problem (14).

Definition 3. 

Let S be a closed convex subset of a Banach space X. A real-valued function $f:X\to \mathbb{R}$ is called uniformly quasi-lower $C^2$ with ratio $\rho \ge 0$ provided that $f(·)+{\displaystyle \frac{\rho }{2}}V(x^{\ast },·)$ is quasi-convex over S for any $x^{\ast }\in X^{\ast }$.

The local concept of uniform quasi-lower $C^2$ can be written as follows.

Definition 4. 

Let $\overline{x}$ be a point in a Banach space X. A real-valued function $f:X\to \mathbb{R}$ is called quasi-lower $C^2$ around $\overline{x}$ if there is a constant $\rho \ge 0$ and an open convex neighborhood ${\mathcal{U}}_{\overline{x}}$ of $\overline{x}$ on which $f(·)+{\displaystyle \frac{\rho }{2}}V(x^{\ast },·)$ is finite and quasi-convex for any $x^{\ast }\in X^{\ast }$.

The first example of this class is the lower $C^2$ functions introduced and studied in Rockafellar [11], as the next proposition shows.

Proposition 6. 

Let S be a closed convex subset of a Banach space X. Suppose that $f:X\to \mathbb{R}$ is lower $C^2$ around $\overline{x}\in X$. Then, f is quasi-lower $C^2$ around $\overline{x}\in X$.

Proof. 

By definition of lower $C^2$ around $\overline{x}\in X$, there is a constant $\rho \ge 0$ and an open convex neighborhood ${\mathcal{U}}_{\overline{x}}$ of $\overline{x}$, on which $f(·)+{\displaystyle \frac{\rho }{2}}{\parallel ·\parallel }^2$ is finite and convex. Fix now any $x^{\ast }\in X^{\ast }$ and observe that

$$f\left(x\right)+{\displaystyle \frac{\rho }{2}}V(x^{\ast },x)=f\left(x\right)+{\displaystyle \frac{\rho }{2}}{\parallel x\parallel }^2+{\displaystyle \frac{\rho }{2}}\left[\parallel x^{\ast }{\parallel }^2-2⟨x^{\ast };x⟩\right].$$

Hence, the convexity of the function $f(·)+{\displaystyle \frac{\rho }{2}}{\parallel ·\parallel }^2$ on ${\mathcal{U}}_{\overline{x}}$ ensures the convexity of $f(·)+{\displaystyle \frac{\rho }{2}}V(x^{\ast },·)$ on ${\mathcal{U}}_{\overline{x}}$, and by consequence, its quasi-convexity on ${\mathcal{U}}_{\overline{x}}$, and so we are done. □

Remark 1. 

We notice that the convexity of $f(·)+{\displaystyle \frac{\rho }{2}}{\parallel ·\parallel }^2$ is equivalent to the convexity of $f(·)+{\displaystyle \frac{\rho }{2}}V(x^{\ast },·)$, which is due to the nice property of the convexity which is its stability under addition. However, it is not the case for quasi-convex functions. Taking into account this observation, we used the function $f(·)+{\displaystyle \frac{\rho }{2}}V(x^{\ast },·)$ in the definition of our new class instead of $f(·)+{\displaystyle \frac{\rho }{2}}{\parallel ·\parallel }^2$. It is worth mentioning that the deep study of both concepts of quasi-lower-$C^2$ (local and uniform) will be the subject of future works. We focus here on its application to nonconvex variational problems.

We state the following nonconvex version of Fan-KKM Lemma from [40].

Lemma 1. 

Let K be any nonempty closed set in a topological vector space Y and let $F:K\to 2^Y$ represent closed valued mapping. Suppose that the two conditions are fulfilled:

• For any finite subset $\{x_1,x_2,\cdots ,x_n\}$ in K we have

  $$co\{x_1,x_2,\cdots ,x_n\}\subset \bigcup _{i=1}^{n}F\left(x_i\right);$$
• For at least some $x_0\in S$, the set $F\left(x_0\right)$ is compact.

Then,

$$\bigcap _{x\in K}F\left(x\right)\ne \varnothing .$$

Theorem 1. 

Let S be a closed convex subset of a Banach space X. Let $F:X\to 2^{X^{\ast }}$ be a set-valued mapping. Let $f:X\to \mathbb{R}$ be a lower semicontinuous uniformly quasi-lower $C^2$ over S and bounded from below on S. Assume that there exists some $y_0\in S$ such that the set

$$\{z\in S:\exists y^{\ast }\in F\left(z\right)sothat2⟨J\left(z\right)-\lambda y^{\ast };y_0-z⟩+f\left(z\right)+\parallel z{\parallel }^2\le f\left(y_0\right)+\parallel y_0{\parallel }^2\}$$

is a compact subset in S. Then, the generalized nonconvex variational problem (14) has at least one solution.

Proof. 

Using Proposition 3, we only need to prove the existence of $\overline{x}\in S$ and $x^{\ast }\in F\left(\overline{x}\right)$ so that

$$x\in {\pi }_S^{f,\lambda }(J\left(x\right)-\lambda x^{\ast }).$$

We introduce set-valued mapping $W:S\to 2^S$ defined by the following:

$$W\left(y\right)=\{x\in S:\exists y^{\ast }\in F\left(x\right)\mathrm{s}.\mathrm{t}.G_{\lambda ,f}^V(J\left(x\right)-\lambda y^{\ast };x)\le G_{\lambda ,f}^V(J\left(x\right)-\lambda y^{\ast };y)\}.$$

(19)

Clearly, for any $y\in S$, we have $y\in W\left(y\right)$; so, $W\left(y\right)\ne \varnothing$.

• We shall demonstrate that $W\left(y\right)$ is a closed subset in S for any $y\in S$. Suppose ${\left(x_n\right)}_n$ is a given sequence in $W\left(y\right)$ that converges to an element $x_0$. We are going to show that $x_0\in W\left(y\right)$. By definition of $W\left(y\right)$, there exists $y_n^{\ast }\in F\left(x_n\right)$ such that

$$G_{\lambda ,f}^V(J\left(x_n\right)-\lambda y_n^{\ast };x_n)\le G_{\lambda ,f}^V(J\left(x_n\right)-\lambda y_n^{\ast };y)$$

and so

$$f\left(x_n\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(x_n\right)-\lambda y_n^{\ast };x_n)\le f\left(y\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(x_n\right)-\lambda y_n^{\ast };y).$$

Using the continuity of F, there is $y_0^{\ast }\in F\left(x_0\right)$ such that $y_n^{\ast }\to y_0^{\ast }$. Using the continuity of J and V and the lower semicontinuity of f, by taking $n\to +\infty$, we get the following:

$$f\left(x_0\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(x_0\right)-\lambda y_0^{\ast };x_0)\le f\left(y\right)+{\displaystyle \frac{1}{2\lambda }}V(J\left(x_0\right)-\lambda y_0^{\ast };y),$$

which implies that $x_0\in W\left(y\right)$.

Next, we prove that W satisfies the assumptions of the Fan-KKM Lemma. Let $y_1,y_2,\cdots ,y_n$ be a finite number of elements in S and let ${\left({\rho }_i\right)}_{i=1,\cdots ,n}\subset (0,1]$ with ${\sum }_{i=1}^n{\rho }_i=1$. Set $v:={\sum }_{i=1}^n{\rho }_i{y}_i$ and let $v^{\ast }\in F\left(v\right)$ and $y^{\ast }:=J\left(v\right)-\lambda v^{\ast }$. Then,

$$\begin{array}{ccc}\hfill G_{\lambda ,f}^V(y^{\ast };v)& =& f\left(v\right)+{\displaystyle \frac{1}{2\lambda }}V(y^{\ast };v)\hfill \\ \\ & \le & f\left(\sum _{i=1}^n{\rho }_i{y}_i\right)+{\displaystyle \frac{1}{2\lambda }}V(y^{\ast };\sum _{i=1}^n{\rho }_i{y}_i)\hfill \\ \\ & \le & \underset{i=1,\cdots ,n}{max}\{f\left(y_i\right)+{\displaystyle \frac{1}{2\lambda }}V(y^{\ast };y_i)\}.\hfill \end{array}$$

Hence, there exists some $i_0\in \{1,\cdots ,n\}$ such that

$$\begin{array}{ccc}\hfill G_{\lambda ,f}^V(y^{\ast };v)& \le & f\left(y_{i_0}\right)+{\displaystyle \frac{1}{2\lambda }}V(y^{\ast };y_{i_0})=G_{\lambda ,f}^V(y^{\ast };y_{i_0}),\hfill \end{array}$$

which ensures by definition of the set-valued mapping W that $v\in W\left(y_{i_0}\right)$. So, the first assumption of the Fan-KKM Lemma is satisfied. Let us check the second one. By our assumptions of this theorem, there exists some $y_0\in S$ for which the subset $\mathcal{A}:=\{z\in S:\exists y^{\ast }\in F\left(z\right)\mathrm{s}.\mathrm{t}.2⟨J\left(z\right)-\lambda y^{\ast };y_0-z⟩+f\left(z\right)+\parallel z{\parallel }^2\le f\left(y_0\right)+\parallel y_0{\parallel }^2\}$ is compact in S. Observe that

$$\begin{array}{ccc}\hfill 2⟨J\left(z\right)-\lambda y^{\ast };y_0-z⟩+f\left(z\right)+{\parallel z\parallel }^2\le f\left(y_0\right)+{\parallel y_0\parallel }^2& \iff & \\ \\ \hfill f\left(z\right)+{\parallel z\parallel }^2-2⟨J\left(z\right)-\lambda y^{\ast };z⟩\le f\left(y_0\right)+{\parallel y_0\parallel }^2-2⟨J\left(z\right)-\lambda y^{\ast };y_0⟩& \iff & \\ \\ \hfill f\left(z\right)+V(J\left(z\right)-\lambda y^{\ast };z)\le f\left(y_0\right)+V(J\left(z\right)-\lambda y^{\ast };y_0)& \iff & \\ \\ \hfill G_{\lambda ,f}^V(J\left(z\right)-\lambda y^{\ast };z)\le G_{\lambda ,f}^V(J\left(z\right)-\lambda y^{\ast };y_0).\end{array}$$

This allows us to rewrite the set $\mathcal{A}$ as follows:

$$\mathcal{A}=\{z\in S:\exists y^{\ast }\in F\left(z\right)\mathrm{s}.\mathrm{t}.G_{\lambda ,f}^V(J\left(z\right)-\lambda y^{\ast };z)\le G_{\lambda ,f}^V(J\left(z\right)-\lambda y^{\ast };y_0)\}=W\left(y_0\right).$$

Thus, there exists some $y_0\in S$ for which $W\left(y_0\right)$ is compact in S. Therefore, all the assumptions of the Fan-KKM Lemma are fulfilled and we obtain

$$\bigcap _{y\in S}W\left(y\right)\ne \varnothing .$$

This guarantees the existence of at least some ${\displaystyle \overline{x}\in \bigcap _{y\in S}W\left(y\right)}$, i.e., $\overline{x}\in S$ and $x^{\ast }\in F\left(\overline{x}\right)$, satisfying

$$G_{\lambda ,f}^V(J\left(\overline{x}\right)-\lambda x^{\ast };\overline{x})\le G_{\lambda ,f}^V(J\left(\overline{x}\right)-\lambda x^{\ast };y),\forall y\in S.$$

This confirms that

$$G_{\lambda ,f}^V(J\left(\overline{x}\right)-\lambda x^{\ast };\overline{x})=\underset{y\in S}{inf}{G}_{\lambda ,f}^V(J\left(\overline{x}\right)-\lambda x^{\ast };y)=e_{\lambda ,S}^{V}f(J\left(\overline{x}\right)-\lambda x^{\ast });$$

that is, $\overline{x}\in {\pi }_S^{f,\lambda }(J\left(\overline{x}\right)-\lambda x^{\ast })$. Thus, the proof is achieved. □

We state our first corollary of Theorem 1, which has been proven in Theorem 3.1 and in [38]. Its proof follows on straightforwardly from our Theorem 1.

Corollary 3. 

Let S be a nonempty closed convex subset of Banach space X. Let $A:X\to X^{\ast }$ be an arbitrary nonlinear operator and let ${\xi }^{\ast }\in X^{\ast }$. Let $f:X\to \mathbb{R}$ be a lower semicontinuous convex function on S and bounded from below on S. Suppose that there is some $y_0\in S$, such that the set

$$\{z\in S:2⟨J\left(z\right)-{\displaystyle \frac{1}{2}}Az;y_0-z⟩+f\left(z\right)+\parallel z{\parallel }^2\le f\left(y_0\right)+\parallel y_0{\parallel }^2\}$$

is a compact subset of S. Then, we obtain the following convex variational inequality:

$$Find\overline{x}\in Ssothat⟨A\overline{x};x^{\prime }-\overline{x}⟩+f\left(x^{\prime }\right)-f\left(\overline{x}\right)\ge 0,\forall x^{\prime }\in S,$$

has at least one solution.

Our second corollary is the following:

Corollary 4. 

Let X be a Hilbert space and S be a closed convex subset in X. Let $F:X\to 2^{X^{\ast }}$ be a set-valued mapping. Let $f:X\to \mathbb{R}$ be a lower semicontinuous uniformly quasi-lower $C^2$ over S and bounded from below on S. Assume that there exists some $y_0\in S$, such that the set

$$\{x\in S:\exists y^{\ast }\in F\left(x\right)suchthat2⟨x-\lambda y^{\ast };y_0-x⟩+f\left(x\right)+\parallel x{\parallel }^2\le f\left(y_0\right)+\parallel y_0{\parallel }^2\}$$

is a compact subset in S. Then, we obtain the following nonconvex variational inequality:

$$Find\overline{x}\in Sand{x}^{\ast }\in F\left(\overline{x}\right)suchthat⟨x^{\ast };x-\overline{x}⟩+f\left(x\right)-f\left(\overline{x}\right)\ge -{\displaystyle \frac{1}{2\lambda }}{\parallel x-\overline{x}\parallel }^2,\forall x\in S,$$

has at least one solution.

Proof. 

It is derived straightforwardly from Theorem 1 using the fact that, in Hilbert spaces, mapping J coincides with the identity operator on X, along with the equality $V(J\left(x\right);y)={\parallel y-x\parallel }^2$, for all $x,y\in X$. □

4. Applications

This section focuses on applying the main existence result from Theorem 1 to prove the existence of solutions of a nonconvex variational problem in $L^p$ ($p\in (1,2]$) spaces, which are Banach spaces that do not necessarily have a Hilbert space structure. To achieve this, we must first establish the next theorem.

Theorem 2. 

(1) 

If f is Lipschitz and uniformly V-prox-regular over a compact convex S for some $r>0$, then $f+{\displaystyle \frac{1}{2r}}{\parallel ·\parallel }^2$ is finite and convex over S.

(2) 

Let $X=L^p([0,1],\mathbb{R})$ with $p\in (1,2]$ and let S be a compact convex set in X. Every $C^2$ function on X is uniformly V-prox-regular over S with constant

$$r:={\left[max\{\parallel {\nabla }^{2}f\left(z\right)\parallel :z\in S\}max\{\parallel z\parallel :z\in S\}\right]}^{-1}.$$

(3) 

Assume that X is a V-proximal trustworthy space. If f and g are uniformly V-prox-regular over a closed nonempty set S, then the sum $f+g$ is also uniformly V-prox-regular over S.

Proof. 

(1) Suppose that f is Lipschitz and uniformly V-prox-regular over a closed convex S for some $r>0$, that is, for any $v\in S$ and any $v^{\ast }\in {\partial }^{L\pi }f\left(v\right)$, we have

$$\begin{array}{cccc}\hfill ⟨v^{\ast };u-v⟩& \le & f\left(u\right)-f\left(v\right)+{\displaystyle \frac{1}{2r}}V(J\left(v\right),u),\hfill & \end{array}$$

$$\begin{array}{ccc}& \le & f\left(u\right)-f\left(v\right)+{\displaystyle \frac{1}{2r}}{[\parallel v\parallel }^2+{\parallel u\parallel }^2-2⟨J\left(v\right),u⟩],\forall u\in S.\hfill \end{array}$$

(20)

Then, for any $u\in S$

$$\begin{array}{cccc}\hfill f\left(u\right)+{\displaystyle \frac{1}{2r}}{\parallel u\parallel }^2& \ge & f\left(v\right)-{\displaystyle \frac{1}{2r}}{\parallel v\parallel }^2+{\displaystyle \frac{1}{r}}⟨J\left(v\right);u⟩+⟨v^{\ast };u-v⟩\hfill & \end{array}$$

$$\begin{array}{ccc}& \ge & h(v,v^{\ast },u),\forall v\in S,\forall v^{\ast }\in {\partial }^{L\pi }f\left(v\right),\hfill \end{array}$$

(21)

where $h(v,v^{\ast },u):=f\left(v\right)-{\displaystyle \frac{1}{2r}}{\parallel v\parallel }^2-⟨v^{\ast };v⟩+⟨{\displaystyle \frac{1}{r}}J\left(v\right)+v^{\ast };u⟩$. Set ${\displaystyle \mathrm{\Gamma }:=\bigcup _{s\in S}{\partial }^{L\pi }f\left(s\right)}$. Since S is compact and f is Lipschitz over S, the set $\mathrm{\Gamma }$ is a weak compact in $X^{\ast }$ by Theorem II-25 in [41]. Therefore, the inequality (21) ensures that

$$f\left(u\right)+{\displaystyle \frac{1}{2r}}{\parallel u\parallel }^2=g\left(u\right)=:\underset{(v,v^{\ast })\in S\times \mathrm{\Gamma }}{max}h(v,v^{\ast },u).$$

Since g is obviously convex by construction, we obtain that $f(·)+{\displaystyle \frac{r}{2}}{\parallel ·\parallel }^2$ is finite and convex over S; therefore, we achieve our proof of (1).

(2) Suppose that f is a $C^2$ function on $L^p([0,1],\mathbb{R})$ and S is a compact convex set. First, we recall that, for such functions, all nonsmooth subdifferentials included in the Clarke subdifferential coincide and are equal to the singleton $\{\nabla f(x\left)\right\}$. Consequently, ${\partial }^{L\pi }f\left(z\right)=\{\nabla f\left(z\right)\}$. Now, we utilize the second order Taylor expansion for $C^2$ function to write

$$f\left(u\right)=f\left(v\right)+⟨\nabla f\left(v\right);u-v⟩+{\displaystyle \frac{1}{2}}⟨{\nabla }^{2}f\left(z\right)(u-v);u-v⟩$$

for any $u,v\in S$ and for some $z\in [u,v]:=\{su+(1-s)v:s\in [0,1\left]\right\}$. Hence,

$$\begin{array}{cccc}\hfill ⟨\nabla f\left(u\right);u-v⟩& =& f\left(u\right)-f\left(v\right)-{\displaystyle \frac{1}{2}}⟨{\nabla }^{2}f\left(z\right)(u-v);u-v⟩\hfill & \end{array}$$

$$\begin{array}{cccc}& \le & f\left(u\right)-f\left(v\right)+{\displaystyle \frac{1}{2}}\parallel {\nabla }^2{f\left(z\right)\parallel \parallel (u-v)\parallel }^2\hfill & \end{array}$$

$$\begin{array}{ccc}& \le & f\left(u\right)-f\left(v\right)+{\displaystyle \frac{1}{2}}sup\{\parallel {\nabla }^2{f\left(z\right)\parallel :z\in S\}\parallel (u-v)\parallel }^2.\hfill \end{array}$$

(22)

Since X is 2-uniformly convex, using Part (2) in Proposition 1, we obtain

$${\parallel u-v\parallel }^2\le \alpha V(J\left(u\right),v),\forall u,v\in S,$$

where $\alpha :=max\{\parallel z\parallel :z\in S\}<\infty$. Set $M:=sup\{\parallel {\nabla }^{2}f\left(z\right)\parallel :z\in S\}<\infty$. Then, the inequality (22) gives

$$\begin{array}{cccc}\hfill ⟨\nabla f\left(v\right);u-v⟩& \le & f\left(u\right)-f\left(v\right)+{\displaystyle \frac{1}{2}}M{\parallel (u-v)\parallel }^2\hfill & \end{array}$$

$$\begin{array}{ccc}& \le & f\left(u\right)-f\left(v\right)+{\displaystyle \frac{1}{2}}M\alpha V(J\left(u\right),v),\forall u,v\in S.\hfill \end{array}$$

(23)

Thus, for any $v\in S$ and any $x^{\ast }\in {\partial }^{L\pi }f\left(v\right)=\{\nabla f\left(v\right)\}$, we have

$$\begin{array}{ccc}\hfill ⟨\nabla f\left(v\right);u-v⟩& \le & f\left(u\right)-f\left(v\right)+{\displaystyle \frac{1}{2r}}V(J\left(v\right),u),\forall u\in S,\hfill \end{array}$$

(24)

with $r:={\displaystyle \frac{1}{M\alpha }}$. This ensures that f is uniformly V-prox-regular over S with constant $r:={\displaystyle \frac{1}{M\alpha }}$.

(3) Assume now that X is V-proximal and trustworthy. Let f and g be two uniformly V-prox-regular over a nonempty closed set $S\subset X$ for positive constants $r_1>0$ and $r_2>0$, respectively. Then, for any $v\in S$, any $v^{\ast }\in {\partial }^{L\pi }f\left(v\right)$, and any $w^{\ast }\in {\partial }^{L\pi }g\left(v\right)$, we have, for any $u\in S$,

$$\begin{array}{ccc}\hfill ⟨v^{\ast };y-x⟩& \le & f\left(y\right)-f\left(x\right)+{\displaystyle \frac{1}{2r_1}}V(J\left(x\right),y),\hfill \end{array}$$

$$\begin{array}{ccc}\hfill ⟨w^{\ast };y-x⟩& \le & g\left(y\right)-g\left(x\right)+{\displaystyle \frac{1}{2r_2}}V(J\left(x\right),y).\hfill \end{array}$$

(25)

Then, for any $v\in S$ and any $z^{\ast }\in {\partial }^{L\pi }(f+g)\left(v\right)$, by the exact sum rule for the limiting generalized proximal subdifferential stated in Part (1) in Proposition 1, there exist $v^{\ast }\in {\partial }^{L\pi }f\left(v\right)$ and $w^{\ast }\in {\partial }^{L\pi }g\left(v\right)$ such that $z^{\ast }=v^{\ast }+w^{\ast }$. Therefore,

$$\begin{array}{cccc}\hfill ⟨z^{\ast };u-v⟩& =& ⟨v^{\ast }+w^{\ast };u-v⟩=⟨v^{\ast };u-v⟩+⟨w^{\ast };u-v⟩\hfill & \end{array}$$

$$\begin{array}{ccc}& \le & (f+g)\left(u\right)-(f+g)\left(v\right)+\left[{\displaystyle \frac{1}{2r_1}}+{\displaystyle \frac{1}{2r_2}}\right]V(J\left(v\right),u),\hfill \end{array}$$

$$\begin{array}{cccc}& \le & (f+g)\left(u\right)-(f+g)\left(v\right)+{\displaystyle \frac{1}{2\frac{r_1{r}_2}{r_1+r_2}}}V(J\left(v\right),u)\hfill & \end{array}$$

(26)

for any $u\in S$. This ensures by Definition 2 that $f+g$ is uniformly V-prox-regular over S with constant ${\displaystyle \frac{r_1{r}_2}{r_1+r_2}}$; hence, the proof is finished. □

Now, we are ready to present an example of application of Theorem 1.

Example 1. 

Assume that $X=L^p([0,1],\mathbb{R})$ with $p\in (1,2]$ and let S be a compact convex set in X. Let $f_1$ be a nonconvex $C^2$ function on X and $f_2$ be a Lipschitz uniformly V-prox-regular over S with constant $r_2>0$, and let $f_3$ be a Lipschitz convex function on X. Then, by Parts (2) and (3) in Theorem 2, the function $f:=f_1+f_2+f_3$ is Lipschitz and uniformly V-prox-regular over S with constant $r={\displaystyle \frac{r_2}{1+2r_{2}M}}$. Let $\lambda >0$ $F:X\to 2^{X^{\ast }}$ be continuous set-valued mapping $F:X\to 2^{X^{\ast }}$. We associate F, f, and S with the nonconvex variational problem (14). Since f is a Lipschitz uniformly V-prox-regular over S, using Part (1) in Theorem 2, the function f is lower $C^2$ over S and so it is quasi-lower $C^2$ over S. The compactness condition on the set S ensures the compactness assumption in Theorem 1. Also, the continuity of f over the compact set S ensures that f is bounded from below on S. Therefore, all the assumptions of Theorem 1 are fulfilled, and consequently, the noncovex variational problem (14) has at least one solution.

Remark 2. 

It is worth mentioning that the existence of a solution for (14) cannot be established using existing results in the literature, owing to the nonconvexity of function f, even if S is convex. Furthermore, this existence result is, to the best of our knowledge, a novel contribution, even in Hilbert spaces.

5. Conclusions

In the present study, we introduced a new concept called quasi-lower $C^2$ functions in Banach space in Definition 3. This class of nonconvex functions expands upon the idea of lower $C^2$ functions, providing a broader class to work with. We also extended the traditional convex variational inequalities (12), which deal with convex data (sets and functions) and nonconvex variational problems (14) in order to include nonconvex data (sets and functions). Many characterizations of the proposed nonconvex variational problems (14) were established in Proposition 3.

Using this new class of quasi-lower $C^2$ functions, we demonstrated the existence of solutions for nonconvex variational problems (14) associated with this new class of nonconvex functions. This is an important step forward in the field of nonconvex variational inequalities, as it addresses the challenge of finding solutions in nonconvex settings.

Additionally, in the last section, we applied our main abstract results to a specific example in $L^p$ spaces. This example illustrates the practical importance of our work and shows its potential for broader applications.

For our upcoming research, we aim to explore the key characteristics of quasi-lower $C^2$ functions. We also intend to investigate their applications in nonconvex challenges, including nonconvex variational inequalities and nonconvex complementarity problems within Banach spaces.