#### 5.1. Workload

The values of variables

${f}_{lt}$ and

${c}_{lt}$ (fog/strict and cloud/flexible workload demands) were taken from the dataset in [

21]. Every time a mobile user required services from a telecommunications provider, a Call Detail Record was recorded in the metropolitan area of Milan during a two-month period. The geographical area was divided into a 100 × 100 grid, in which each cell has information on the Short Message Service (SMS) messages received and sent, phone calls made and received, and Internet usage. These data were aggregated into 10-minute intervals. In this paper, the Internet usage information models the workload demands since it represents a variety of mobile applications, different from calls and SMS. This dataset was chosen since it provides real records of a city accounting for user mobility.

In the dataset [

21], demands are separated into geographical cells, but users actually request services from a base station, which may not be in the cell area. Correspondingly, some base stations serve a larger number of cells (larger areas). The set of locations

$\mathcal{L}$ is, therefore, the set of areas covered by the antennas, the location of which was determined by the OpenCellId project [

24], an open database containing information about base stations worldwide collected by mobile users. This database has comprehensive data and has been employed in previous work reported in the literature [

25]. The location of all base stations was obtained by filtering the existing base stations in the period of the Milan dataset [

21]. The workload of each cell was mapped to the closest base station, as in [

20]. In the case of multiple base stations inside a cell, the workload is equally balanced on these BSs. Thus,

$\mathcal{L}$ is the set of base stations obtained from the OpenCellId project, and the workload of each cell from [

21] is associated with the closest base station to define the values of

${f}_{lt}$ and

${c}_{lt}$. In this paper, a complete fog-cloud infrastructure is designed, so that all locations in

$\mathcal{L}$ are considered, thus evaluating the complete metropolitan area of Milan. Although the solution can be evaluated on a smaller scale, results are presented for all locations.

The input to the problem consisted of

N,

R,

$\mathcal{L}$,

$\mathcal{T}$,

${f}_{lt}$, and

${c}_{lt}$. The capacity

R of a server was fixed, and

N was varied to evaluate solutions obtained under different budget constraints. The number of locations in

$\mathcal{L}$ was determined using the OpenCellId dataset as explained above. Since the dataset [

21] has data for two months,

T was also varied to evaluate the solution under different lengths of planning intervals, from 1 h to 24 h. The proportion of fog and cloud requests was varied using three scenarios, namely

$P25$,

$P50$, and

$P75$. In

$P25$, 25% of the workload for an antenna was strict and 75% flexible. In the

$P50$ scenario, the proportion was 50% for each type of request, and, in

$P75$, the workload is 75% strict and 25% flexible.

Table 2 summarizes the input values and the adopted scenarios.

#### 5.3. Numerical Results

In this subsection, the performance of the proposed solution is assessed. This evaluation showed how the solution improves fog service, reducing costs and dealing with the two types of workload. Furthermore, several scenarios with different traffic patterns and budget constraints were used to evaluate the efficiency of the solution under various conditions. First, the results produced by

$OPT$ using different planning intervals are discussed. Then, the results obtained under degradation are presented. Finally, different scenarios of traffic patterns (

$P25$,

$P50$, and

$P75$) were evaluated. Three metrics were considered: acceptance ratio of strict latency workload, acceptance ratio of flexible latency workload in the fog, and the number of deployed servers. A 95% confidence interval is used in the graphs. In this section,

$STRX$ is used to refer to all solutions that allow degradation of the objective function in Equation (

1) and

$SERX$ to all solutions that allow it for the objective function in Equation (

2). Graphs of the strict latency acceptance ratio are in a logarithmic scale.

$OPT$ results are discussed for the

$P50$ scenario in

Figure 3. The larger is the number of available servers, the larger is the number of servers utilized (

Figure 3b). This is a result of the main goal of the solution to serve the maximum number of strict workloads possible, which leads to more servers being used in the solution. For

$1\le N\le 1024$, the available servers cannot cope with the entire strict latency workload since most of the available servers (

N servers) are used. This causes the overlap of the curves for all planning intervals. For

$N\ge 2048$, the available capacity is greater than the total demand, so the entire demand is met (

Figure 3a), requiring between 1480 and 1710 servers. The number of required servers varies according to the planning interval: short planning intervals may not contain periods during which a location is crowded. Consequently, for longer intervals, a large number of periods of peak demand is present for several locations, which requires the deployment of a larger number of servers. Results for

$N>2048$ are the same as those for

$N=2048$ since the multi-level programming approach optimizes the entire served demand in Equation (

1); hence a larger number of servers does not lead to any improvement in the strict latency workload service.

The ratio between flexible requests served in the fog and the total flexible workload is shown in

Figure 3c. The extra capacity of fog nodes can be used to host the flexible workload, thus, when nearby 80% of the strict workload is served (

$N=512$), more than 30% of the flexible workload can be executed in the fog, which improves the latency of end users as well as allows more flexibility in the energy management of the cloud data center.

$OPT$ maximizes flexible requests utilization of fog nodes (the objective function in Equation (

3)) only after satisfying the objective functions in Equations (

1) and (

2). As a result, no new fog servers will be deployed to host only flexible workloads. Thus, for

$N=2048$, between 60% and 80% of the flexible workload is hosted in the fog and the remainder in the cloud. If the order of objective functions in Equations (

2) and (

3) were reversed in the multi-level optimization, flexible workload allocation would be prioritized in the fog, but at a higher server deployment cost than that was in the original order.

The order of the curves changes in the interval

$1024\le N\le 2048$ in

Figure 3c due to the availability of a larger number of servers for

$N=2048$ and longer planning intervals. For

$N\le 1024$, the available resources do not meet the full demands of the strict workload. Conversely, when

$N=2048$, all strict workloads are processed, and powerful fog nodes tailored to the peak demands are produced. Consequently, during periods when strict demands are low, fog servers are used to host the flexible workload. Thus, solutions for larger planning intervals are capable of hosting more flexible workloads, explaining the difference in the order of the curves in

Figure 3c for

N between 1024 and 2048.

In the remainder of this section, results for 24 h planning intervals are shown. Using a larger interval results in more variation in demands in the considered locations. Hence, larger intervals are useful in planning long-term infrastructures. A comparison of

$OPT$ and the solutions which allow degradation in one of the objective functions is presented in

Figure 4 for the

$P50$ scenario. The acceptance ratio of strict latency workloads is shown in

Figure 4a. The curves for

$OPT$ and all solutions that allow degradation in the objective function in Equation (

2) overlap since it is optimized after the objective function in Equation (

1). Curves corresponding to

$STRX$ are parallel to

$OPT$ in the log scale according to the allowed degradation, from 5% to 20%.

Figure 4b shows the number of employed servers as a function of

N.

$SERX$ deploys a larger number of servers than

$OPT$ and

$STRX$. Since all servers are used for

$1\le N\le 1024$, differences in the values obtained by

$OPT$ and

$SERX$ appear only for

$N=2048$, when there is more capacity than that required for the strict workloads. For

$SERX$, extra servers are employed to host more flexible latency workloads in the fog, as shown in

Figure 4c. Notice, however, that an increase in the number of servers less than 15% results in a minimal increase in the flexible latency service in the fog. This is due to the distribution of demand across different locations. To explain this trend better,

Figure 5 presents results for flexible latency workload in the fog for all values of the planning intervals considered (1 h, 3 h, 6 h, 12 h, and 24 h) and

$N=2048$. For 1 h planning, an increase in the number of servers increases the acceptance ratio of flexible workloads in the fog. However, results for longer intervals show that small gains are obtained for degradation smaller than 15%. For small intervals, users are less mobile, which makes demands more uniform over all locations. Longer intervals, however, present peak demand periods on a larger number of locations. Thus, given that servers cannot be moved from one fog node to another, serving the total flexible demand in the fog requires a larger number of servers in many fog nodes, making the employment of

$SERX$ effective only when high degradation is allowed.

One important effect of

$STR5$ is noticed for

$N=2048$, where it reduces more than 400 servers in the solution in relation to

$OPT$, which accounts to about 30% of savings in server costs (

Figure 4b). This is due to the fact that the removal of one or two servers from each fog node does not lead to great blocking. Serving strict workloads is the main goal of optimization, hence most servers process mainly this type of workload. However, to fully process the demand, a fog node may have servers that remain idle or process only a small number of strict workloads. For example, during an interval facing peak demand, a fog node may need five servers to process all the strict demand, while most of the time only three or four servers would be sufficient. Thus, even if degradation in the objective function in Equation (

1) is small, high infrastructure costs can be avoided if the fog node capacity is not tailored to the peak demands in the fog area. If the blocking of a small number of requests is acceptable,

$STRX$ becomes a viable solution.

Results for

$P25$ and

$P75$ scenarios are presented, yet for

$OPT$,

$STRX$, and

$SERX$ solutions and 24 h planning intervals. These results for the acceptance of strict latency workload are displayed in

Figure 6. Results follow the same pattern of those under the

$P50$ scenario. All solutions result in greater acceptance of strict latency workload under

$P25$ than for those of

$P50$, and less than those of

$P75$. The former is explained by the reduction of the strict workload, making the available servers sufficient for dealing with a larger part of the strict demand. The opposite situation happens when there are more strict workloads, when the strict demands are harder to serve.

The acceptance of flexible latency workloads in the fog and the number of employed servers are shown in

Figure 7 and

Figure 8, respectively, for both

$P25$ and

$P75$ scenarios. In the

$P25$, there are less strict workloads. Accordingly, the total number of employed servers is reduced (

Figure 8a), which also reduces the capacity available for hosting flexible workloads (

Figure 7a). For

$P75$, there is much more strict workload, which requires about 1700 servers (

Figure 8b). The reduced demand for flexible latency workloads (

Figure 7b) allows almost 100% processing of this demand in the fog nodes for

$OPT$, and the employment of

$SERX$ under these circumstances leads to few gains. Finally, applying

$STR5$ instead of

$OPT$ leads to savings about 30% for both the

$P25$ and

$P75$, as shown for

$P50$. When the strict workload demand is high (

$P75$), the absolute number of servers is higher, thus

$STR5$ can reduce costs considerably with the infrastructure.

All results in this section were obtained using the Gurobi Optimizer solver. The execution time depends on the input size, mainly affected by N and the planning interval length. Scenarios with the largest inputs, high N and 24 h intervals, took less than 350 s, which is less than 1% of the planning interval length. Therefore, the proposed solution is feasible and, in the case of changes of demands, the location of fog nodes can be quickly recalculated.

This section has presented an evaluation of the results produced by the multicriteria optimization formulation employing multi-level programming proposed. Solutions considered hierarchical objectives with and without the allowance of degradation in one of the objective functions to optimize the others. The deployment of a fog infrastructure requires an analysis of all locations. Moreover, mobility of end users causes different regions to have demand peaks at different times, thus, in addition to the locations, the variation of demands must also be considered in the choice of the location. Given the priority of the multiple objectives, $OPT$ represents the ideal solution. However, results for the other solutions produce interesting results: if the provider accepts the blockage of some users, the employment of $STR5$ leads to large savings with physical servers in the infrastructure, which leads to a potentially useful trade-off between service and deployment costs for the provider. The employment of $SERX$, on the other hand, is seldom useful since the number of servers in each fog node is fixed, although the demands are variable and distributed. All results can be obtained in a reasonable time using the proposed formulation.