With the constant evolution of cellular mobile communication systems, traditional cellular monolayer networks have failed to satisfy the requirement of transmission speed, and are beset by such issues as imperfect indoor coverage effect and insufficient capacity for outdoor hotspot areas [

1]. Hence, the heterogeneous network (HetNet) has become a key technology for the next generation of communication networks. Given that over 70% mobile data is produced indoors at present [

2,

3], the femtocell can effectively solve relevant issues about indoor communication due to its flexibility of deployment. Therefore, deploying femtocells in current macro-cellular networks to form macro-femto bilayer HetNet can transfer the load of the macro base station to femtocells, thus improving system performance. However, the introduction of femtocells will bring cross-layer interference between macro base stations and femtocells. The macro users located around femtocells tend to be highly subject to the strong interference of femtocells, thereby lowering the overall performance of system, especially the throughput [

4]. In order to solve the above problems within academia, there are some cases using resources allocation method.

In recent years, most research concerning resource allocation algorithms have introduced game theory [

5,

6]. A game of resource allocation problem in HetNets involves a set of players, strategies, and payoffs. The players can be defined as base stations and users. Strategies can be the choice of transmission power level or subcarrier allocation. The payoff is the evaluation for a player of all possible outcomes represented by the utility function. Based on these settings, the interferences in wireless networks can then be considered as a result of the frequency resources among stations and users. Due to the strong transmitting power and wide coverage scope, the macro base station plays a more important role than femtocells in the formation of cross-layer interference, so it should first satisfy the requirement of frequency spectrum of macro users. Moreover, players cannot know strategy at the current moment, which means that it is a game with incomplete information. In previous literature, games with incomplete information—known as Bayesian games—are employed to address these kinds of issues. These games often assume that the players act independently according to some complex strategies for optimizing the setting of various resources of the network. With respect to HetNet, the macro plays a centralized role and can communicate with the low power node (LPNs) through the X2 interface. Thus, the topology matches a game with the macro base station being leader and the femtocells acting as followers. With this understanding, the Stackelberg game is a strategic game in economics in which the leader firm moves first and then the follower firms move sequentially. In the Stackelberg model, the leader can make prior decisions, and followers will choose a reasonable resources allocation scheme based on these decisions. Therefore, the Stackelberg game theory is more appropriate to analyze the resources allocation of HetNet based on cross-layer interference suppression.

There are already some researchers studying this area. The Stackelberg game theory was used to solve the resource allocation problem both in power and in spectrum [

7,

8,

9,

10,

11,

12,

13,

14,

15,

16,

17,

18,

19,

20]. In the two-tier network scenario of single macro community, reference [

8] proposed a supermodel game-based femtocell downlink power control algorithm in the double network of single community. The proposed algorithm maximized the network capacity through reducing femtocells’ cross-layer interference upon neighboring macro users, yet without considering the interference of the macro base station upon femtocell users or the macro base station in game process. In the same network scenario, [

10] proposed a power control algorithm with self-adapting utility based on Signal to Interference plus Noise Ratio (SINR). This algorithm could be realized through distributed ways and gradually reduce femtocell transmitting power causing strong interference, so as to effectively hinder cross-layer interference. The Stackelberg game is introduced to address numerous problems such as power control [

8,

10], but there are only a few algorithms attempted to optimize the spectrum sharing between macro and femtocells. In particular, reference [

11] proposed a price based resource allocation strategy to handle the spectrum sharing problem, where the macro base station acts as a leader and protects itself by pricing the interference from femtocell users. In the system model given by [

12], the leaders in the Stackelberg game could make priority decisions, while the followers should choose a reasonable resource allocation scheme based on the decisions. Considering both macro base station utility and femtocell utility, reference [

14] proposed the Stackelberg game based on an uplink power distribution framework. Under the condition of ensuring Quality of Service (QoS) of macro users, it did not take into account the uplink interference of macro users upon femtocell. Besides, when transmitting power of macro base station was given, the macro base station had enough information to predict response of base station, which would bring large amount of signaling overhead to network. A kind of spectrum leasing framework with one substep to obtain the optimal algorithm of Stackelberg equilibrium was proposed in [

16]. Based on the above analysis, a kind of spectrum leasing framework was also proposed in [

18], where femtocells lease spectrum from their co-existing macro base stations to serve femtocell users and allow the dynamic access of macro base station users. The above references all assume that each base station in HetNet is an independent individual in game model that attempts to seek maximal energy self-sufficiency without considering the feelings of other individuals in the network, so the optimal overall performance of the community cannot be realized through adjusting self-possessed spectrum resources. Pricing policy considering both economic income and spectrum income was proposed in [

19,

20]. When femtocells bought spectrum from a macro base station, the price would be determined by the improvement brought by system throughput. Meanwhile, if femtocells can provide macro base station users under severe interference with spectrum and access service, the price will be reduced to certain degree. However, setting a spectrum price will lead to high signaling overhead. At the same time, none of the above methods has provided the solution scheme when the value assignment scope for the Nash equilibrium is continuous. The Nash equilibrium is a solution concept of a game involving two or more players in which each player is assumed to know the equilibrium strategies of others, and a player has nothing to gain by just changing their own strategy. Summarizing the results of the above research, some problems exist in current algorithm studies. On the one hand, most algorithms focus on the performance of the macro base station but neglect the performance of the overall system; on the other hand, most of the existing algorithms adopt utility function using earnings to define both game playing parties. Pecuniary transactions are not only impractical but also the cause of high signaling overheads [

21]. Moreover, most of the algorithms are Stackelberg game featuring finite discrete perfect information, whose Nash equilibrium solution can be obtained through simple backward induction. However, when the range of participant features have continuum value, game playing will become continuously expansive and dynamic. Equilibrium is never actually reached since the equilibrium they could pick from was discrete. It would drop to the Bertrand equilibrium. Letting the players learn an optimal payoff function, it can be found that automated learning gives consistent results. The optimum for the payoff function corresponds to the optimal price in an equilibrium. Therefore, an effective and intelligent Nash equilibrium solution algorithm is needed.

In this paper, the Stackelberg game model is adopted to cope with modeling resources allocation. First, we build a spectrum allocation scheme with the concept similar to community expansion, where macro base station pushes partial users under severe cross-layer interference to femtocells nearby and contributes partial spectrum as return. Then considering the interest of both the macro base station and femtocells, one kind of utility function without using pricing strategy is built to improve the overall system throughput capacity. Finally, given that optimal strategy value range is continuous, which renders the existing algorithms impossible to work out Nash equilibrium, a gradient algorithm is introduced for the first time to solve Nash equilibrium, based on which an effective and intelligent gradient algorithm is proposed. Simulation results show that this scheme has improved the performance of the system to a certain extent.

The rest of the paper is organized as follows.

Section 2 presents the system model.

Section 3 explains the utility functions and optimization objects analysis.

Section 4 introduces the iterative algorithm by adopting the gradient descent algorithm. In

Section 5, the simulation is given to evaluate the improved performance of the proposed algorithm. Finally, we show the conclusion of the paper.