Data Management and Integration of Low Power Consumption Embedded Devices IoT for Transforming Smart Agriculture into Actionable Knowledge

Abstract

Smart agriculture today uses a wide range of wireless communication technologies. Low Power Consumption Embedded Devices (LPCED), such as the Internet of Things (IoT) and Wireless Sensor Networks, make it possible to work over great distances at a reduced cost but with limited transferable data volumes. However, data management (DM) in intelligent agriculture is still not well understood due to the fact that there are not enough scientific publications available on this. Though data management (DM) benefits are factual and substantial, many challenges must be addressed in order to fully realize the DM’s potential. The main difficulties are data integration complexities, the lack of skilled personnel and sufficient resources, inadequate infrastructure, and insignificant data warehouse architecture. This work proposes a comprehensive architecture that includes big data technologies, IoT components, and knowledge-based systems. We proposed an AI-based architecture for smart farming. This architecture called, Smart Farming Oriented Big-Data Architecture (SFOBA), is designed to guarantee the system’s durability and the data modeling in order to transform the business needs for smart farming into analytics. Furthermore, the proposed solution is built on a pre-defined big data architecture that includes an abstraction layer of the data lake that handles data quality, following a data migration strategy in order to ensure the data’s insights.

1. Introduction

The term “agriculture 4.0” was coined by analogy to the buzzword “industry 4.0”. The main features involved in the Industrial Revolution were technological, socioeconomic, and cultural. In the 1880s, it was transformed with the help of steam engines, which was industry 1.0. The introduction of electricity on assembly lines corresponds to industry 2.0. The third phase of the industrial transformation is characterized by informatics and the automation of processes on assembly lines. Industry 4.0 brings together the latest changes made thanks to digital technology and the possibility of interacting and communicating with different equipment. Agriculture has undergone the same changes [1,2].

Today all machine tools and the entire production chain environment can transmit real-time information on their condition and performance. This information, centralized in the factory, makes it possible to control the various machines, considering the other devices’ conditions. In this way, it is possible to automate and robotize a complete production chain comprising robots working on the same product simultaneously and in a coordinated manner, which saves time and, therefore, increases productivity [3].

Agriculture has encountered the same transformations as other industries, with more powerful engines and machinery. Today, all agricultural machinery integrates electronic controls and has entered the digital air, allowing them to be informed of their condition and performance.

Besides, electronics have become more democratic, with sensors and drones collecting data on the weather, plots, animals, and overall farm life. Currently, one can “take the pulse” of any activity on the farm and translate it into performance.

This trend is global and affects all sectors. The development of electronics, software, and databases makes it possible to process a lot of information quickly in order to assist a decision-maker in their choices.

This digitization helps us to understand a situation better and to automate tasks that are thankless or difficult to perform because they are exact, fast, or repetitive. It will be possible, for example, to identify a weed in the process of emerging and control a weed-killer robot working in masked time 24 h a day so that it treats the seedling [4].

These advances will facilitate the understanding of the various agricultural issues, decision making, and improved performance via robots, drones, or IoT sensors. Therefore, the farmer is free to use them according to their needs, or not.

Once the information has been capitalized and understood, agriculture 4.0 will allow, for example, the provision of fertilizers in a localized manner or to ensure a geolocalized treatment. In addition, it will determine the ideal time for harvest based on the weather, agricultural market prices, and farm characteristics, thus optimizing the selling prices. Furthermore, with agriculture 4.0, a variety of sensors can be found in the barn, e.g., for animal identification, animal location, heat detection, or barn climate [5,6].

Of course, changes are underway, but agriculture should benefit. To convince ourselves of this, let’s go back to agriculture 3.0, before the arrival of tractors and mechanization. At that time, our grandfathers saw this wave of modernity arrive, often with suspicion. It has profoundly changed the farmer’s work, yet no one today would want to go back, even those most resistant to modernity. Agriculture 4.0 opens a transition of the same magnitude [7].

The management of agricultural technologies has an important impact in leading the development of agriculture as a sector. There are different technologies, but farmers’ behavior and data characteristics are crucial to choosing suitable technology. Given this fact, the use of data has become heavily required. If the needed information is acquired, it will help to improve the production process. For example, agricultural management can record a whole crop-growth process and monitor the weather, allowing an inclusive intervention in order to manage the damage of the natural environment [8,9].

As big data evolves, organizations are trending toward having multiple data lakes instead of one. These additional data lakes are built for several reasons. They may serve as backup or disaster-recovery purposes for an existing production data lake, or perhaps they may replicate the contents of one data lake to another geographic location [10].

This paper will be organized as follows: Section 2 will focus on the literature review and architecture challenges. The importance of data in agriculture is given in Section 3. Section 4 presents the technical challenges and architectural requirements. An overview of the proposed solution is given in Section 5. The data sources that should be considered in smart farming are given in Section 6. Section 7 broaches the importance of data quality and data governance. A description of the data generated by the machine is given in Section 8. Section 9 explores the differences between the most important data modeling methodologies, as well as determining which one fits better to the smarting farming analytics and in which cases. Section 10 demonstrates the data science part and what we propose as algorithms for smart farming analysis and predictive maintenance. Finally, Section 11 presents our conclusions.

2. Agriculture Challenges

A modern concept of farming management has appeared. Recently, precision agriculture has been introduced in order to optimize and monitor production processes [11].

The emergence of advanced data analytics and big data has offered new business opportunities by breaking down borders and reshaping agricultural activities’ structure.

The benefits to the farmer and the environment as a whole are enormous, by minimizing the use of sprayers, fertilizers, wastewater, and, at the same time, increasing crop yields. This saves time and money and is environmentally friendly. Drones can be used in agriculture by integrating, for example, remote sensing imaging technology with multi-spectral cameras. Spectral imaging also allows the extraction of additional information that the human eye cannot capture. Invisible wavebands reveal the agronomic characteristics of the soil, plants, and crops. This sensor technology can be used throughout the crop cycle. Whether used during planting, irrigation, fertilization, or harvesting, drones providing multi-spectral imaging can be used at every stage, allowing farmers to manage their crops very efficiently in every season. The reflectance properties of the vegetation are used to calculate vegetation indices. The most popular index is the NDVI (normalized difference vegetation index). The difference in reflectance observed in the red and near-infrared provide, in particular, information on the vegetation cycle in poorly- or not instrumented areas.

The IoT sensors, local weather forecasts, and robotics (e.g., aerial imagery by drone) have been revealed in agriculture due to the consistency of the advanced analytics and big data to inspect the farming processes, farmers’ experiences, and farm equipment behaviors [12]. From this perspective, a promising opportunity for agriculture has appeared. The intelligent farming development focuses on using communication and information technologies in the cyber-physical farm. While precision farming (or precision agriculture) considers the variability in the field, smart farming goes further than that by basing management tasks on the data and not only on the location.

Thus, in order to strengthen and reinforce agriculture, we have to focus on operations that pursue resource consumption. Technological innovation and agricultural-related techniques lead to intensive and sustainable development [13]. We must be aware of the challenges and find the right and less costly procedures to remedy them.

In rural areas, the broader employment sector is heavily represented by smallholder farmers. Those farmers are indispensable contributors and donors in global food production [14]. Family farms represent more than 90% of the world’s farms. They contribute to over 80% of food production and operate 75% of the farmland [15,16].

Small farms have less and less of a future in a market where competition is only based on price. If the small businesses are as good as the big ones, there will be no problems continuing into the next generation. However, the transfer could remain problematic. They will have to multiply the alliances that affect their daily activities, such as machinery or labor cooperatives, in order to stay competitive.

It was estimated that food production would have challenges in the future, as shown in Figure 1, which focuses on the significant challenges based on the World Resources Institute [16].

From 2010 to 2050, and to feed nearly 10 billion people, we will need 56% more food, making sure not to use more land [16].

We are currently using around 50% of the vegetated land for agriculture. Therefore, we need to prevent that from expanding, in order to be able to save at least a forested area nearly “two times” the size of India [16]. Simultaneously, we limit emissions by −67% using innovative technologies [16], such as IoT, big data, and artificial intelligence (AI).

By 2050 it will get hotter and hotter. Estimates show that climate change might reduce global agriculture productivity by 17% by 2050.

The quality of water used to irrigate land could become a limiting factor in some parts of the world [17,18,19].

Maize is the basis for food security in some of the world’s poorest regions in Africa, Asia, and Latin America. For example, it is expected that some African areas might face a lack of rainfed maize yields, represented by a decrease of 25% in 2050, compared to 2000. In addition, it was also estimated that agricultural withdrawal would attend 80% by 2020 in Morocco [20].

Water consumption increases gradually every year, and irrigation can be considered a significant factor. Figure 2 represents the Moroccan national water resource affectation in 2015 for 13 billion, where irrigation represents 90% [21].

Moreover, vegetable consumption has increased, which will increase agricultural pesticides. The World Health Organization (WHO) defines pesticide residue as a specified substance on food, animal feed, water, or other environmental media and agricultural commodities resulting from pesticides.

The Malaysian Ministry of Health regulations defines a safe maximum residue level (MRL) of vegetables and fruits, guaranteeing acceptable agricultural practices in this field [12].

The ‘maximum residue level’(MRL) is defined as the upper legal level of concentration for a pesticide residue in or on food following regulations, based on good agricultural practice and the lowest consumer exposure necessary to protect vulnerable consumers.

Using accurate data analytics techniques and calculations can help farmers decrease the pesticide residue and drive decisions that farmers will perform during production. The European Union method can be adapted in order to estimate the potential MRL values on certain vegetables or fruits for which their MRL has not been established. Below is the equation.

Table 1. Variable’s explanation of the equation of the MRL estimation.

Variable	Explanation
R	The mean of the HR
HR	The highest residue after pre-harvest interval. PHI from each of the field-trial.
K	The one-sided tolerance-factor for normal distributions with a confidence level of 95%.
S	The standard-deviation of HR after PHI.

contains the explanation of the variables.

Estimated MRL = R + KS

(1)

3. IoT and the Importance of Data in Agriculture

The IoT is based on the ubiquitous presence around people and of objects that are capable of measuring, understanding, and even modifying their environment.

The main consequence of IoT is, without a doubt, its impact on the daily life of potential users. IoT has remarkable effects both in the home and in the workplace. It will play a decisive role in the near future (health, intelligent transport, home automation, assisted living, etc.). Businesses also expect significant benefits (freight transport, security, logistics, industrial automation, etc.). Based on these considerations, the U.S. National Intelligence Council has declared the IoT one of six technologies that will potentially impact U.S. interests by 2025. In 2018, the number of interconnected devices was estimated to be 30 billion. Furthermore, this number is expected to reach the value of 75 billion by 2025. These numbers suggest that the IoT will be one of the significant sources of volumetric data.

Sensors and actuators form the key elements of the Internet of Things. They track the state of their environment, obtain information about temperature, motion, position, etc. Moreover, they include a network that is generally composed of a potentially large number of nodes. As a result, these sensors face many communication issues, such as security and confidentiality, mobility, short-range reliability, robustness, scalability, and resources (energy, limited storage, processing storage, processing capacity, bandwidth, etc.).

For innovative agriculture applications, the IoT is used to monitor the moisture and temperature levels in the hay, straw, etc., to prevent fungus and other microbial contaminants and weather stations, which study the weather conditions in the field in order to predict ice formation, rain, drought, snow, or changes in the weather.

Agriculture is becoming increasingly complex and interconnected. Onfarm application facilitates its management. Contextual information, such as mapping, location, soil moisture, telemetry, and weather, are used for effective real-time decision making. For example, the bumblebee application monitors drone life by collecting and processing visual, audio, light, weather, and temperature information. It automatically reports the current status of the colony and its well-being.

On the other hand, an application, such as Hydropoint, retrieves contextual information through meteorological stations. It automatically schedules irrigation according to local weather conditions and needs, thus reducing the water bill. Microstrain is a wireless environmental sensing system that monitors key growth episodes in vineyards. Information, such as soil and leaf moisture, solar radiation, and temperature, are collected and merged in order to monitor vineyards remotely and alert growers to critical situations.

The technical aspect manifested in data analytics becomes an essential factor to consider in smart farming [22,23,24,25]. The power of the data no longer needs to be proven. Instead, it is the key technical aspect that we can take advantage of to manage and operate the farming processes and maintain the farm equipment effectively and intelligently. Based on the result, we can strategically seek sustainable agriculture where the farms are operated and managed smartly, mostly automatically, limiting the manual human intervention.

From a technical perspective, to remedy the agricultural challenges mentioned above, big data analytics remains the most promising technique (and approach) in the field of intelligent farming [26]. For this approach, and from a global perspective, many applications exist that can improve.

Making subsidy schemes, intervention, and appropriate legislation led by policymakers to address farming processes based on understanding the underperformance causes of some crops.

Farmers must regularly update themselves and vary their sources of information. Reading magazines outside of agriculture can help them learn a lot about what people care about when it comes to food. For example, finding suitable ingredients can help find new ways of growing food. The Internet provides an unimaginable mass of information.

Big data analytics services are potentially skillful at answering how to deal with obstructing constraints.

Table 2. Big data analytics services from a business point of view.

Services	Role	Perspective
Financial	Supporting the activities of agriculture including micro-finance, subsidy schemes, and banking services.	Managing insurance. Managing credit. Managing subsidies.
Production (related services)	Assisting farmers in extracting the valuable information from their assets, as well as combating disease or pest that put the harvest in danger.	Managing farm diagnostics and records.
Trade and Market	Easing access to marketing and also supporting farmers in getting the most benefits (prices) for their products (commodities).	Accessing markets and customers. Implementing trade and markets for farms.
Registration	Surrounding farmers-groups and cooperatives services with their members, including communication and membership-management.	Managing identification. Managing profiles. Managing admission. Managing registration processes.

shows how those services are represented.

Extensive data analytics services are potentially skillful at dealing with obstructing constraints. Table 2 shows how those services are represented.

Thus, our perspective in this research is to deliver tailored, actionable farming-based information and services. The proposed architecture uses a big data approach relevant to smallholder farmers. The architecture is especially suitable for developing countries to provide a solution that enables farmers to make different analytics. These analyses are related to (1) spatial distribution, (2) water management, and (3) mechanical system maintenance. It also allows agronomists and bioinformatics scientists to store and effectively handle their data, based on a data-migration strategy on top of a refined data architecture dedicated to intelligent farming analytics, the SFOBA [6].

4. Technical Challenges and Architectural Requirements

The smart farm is supposed to explore intelligent agricultural processes and managements routines in order to handle the agricultural requirements. From a data analytics perspective, the system is composed of the following three main components:

• Data sources: including databases and IoT and smart-devices [26];
• Data migration strategy: including data collection from the source, data ingestion based on physical and business data models, and data architecture that meets all smart-farming-analytics requirements [27];
• Data wisdom: or the smart farm’s servomotor. It will conduct all of the innovative farming processes and management based on the data analysis insights, predictions, and predictive maintenance reporting.

A farming process should include the following six intelligence levels:

• Readjusting (e.g., a process);
• Sensing (e.g., anomalies, equipment behavior);
• Inferring (e.g., the key performances);
• Learning (from the data);
• Anticipating (e.g., predictive maintenance);
• Self-organizing (e.g., smart processes conducted by algorithms).

The incorporation of IoT technologies and devices is increasing gradually so that the number of devices should communicate with the host. As a result, each become unpredictably huge [28,29].

With these requirements, especially for smart farming, the 5G may not provide highly effective connectivity of network that handles the large number of users/farmers and devices that results in the network congestion [30].

With the integration of AI, the 6G is supposed to overcome the network’s manual configuration and optimization, enabling the smart farm to scale up to an ultra-dense network that will be “complicated” and dynamic [31].

Also, Blockchain technologies have driven the development of performance-based communication applications, such as software-defined networks and the Internet of Vehicles (IoV).

The main constraints that 6G networks are supposed to handle are as follows:

• Seamless connectivity;
• Quality-of-Services (QoS) requirements, especially for large-scale devices (including IoT);
• Massive amount of data produced by these devices should be collected and analyzed.

This research will build an abstraction layer on top of a big data architecture (SFOB) to process and analyze these collected data. By including AI techniques in the 6G system, all of these challenges are solved using AI analysis, high learning, and classification.

The 6G networks automatically implement processes for knowledge discovery through complicated decision making.

Given the fact, and based on the above requirements, it is essential to note that more IoT devices and data sources generate more data, and more AI analysis has led to more data processing. Therefore, building a sustainable smart-farming analytics system is significant for making a data architecture that meets all of these requirements and technical constraints.

This comprehensive data architecture will have the following capabilities:

• Handling small files in the Hadoop environment;
• Handling data compression and archiving without impacting the computation performance;
• Determining the suitable parameters and constraints that should be considered in developing job scheduling algorithms for the MapReduce framework;
• Handling data quality;
• Handling security;
• What suitable data modeling methodologies should be followed (from a business intelligence perspective) and when;
• Finding the right technical components;
• Finding the suitable data models (from a data science perspective) and algorithms;
• Building a data architecture (SFOBA) that handles all of the above constraints and requirements.

5. Overview of the Proposed Solution

In order to tackle our research questions, we studied literature considering research about big data, machine learning, and IoT in articles published from “Scopus” and “Web of Science.” In addition, we focused on studies related to intelligent farming (and agriculture in general). Many research articles were considered to compare the solutions proposed for smart-farming analytics and determine what is missing from a data management perspective. Finally, we listed the technical challenges in order to find an appropriate solution.

In order to analyze and visualize the bioinformatics data, the author of [32] aimed to find an appropriate design to make biological data available in a single repository to the specialist, considering both primary-database and secondary-database as data collected from the literature. Big data provide a framework for processing and storing the data, and a platform for bioinformatics analysis tools, as shown in Figure 3.

The proposed solution was a high-level design that considers the following three major segments:

• Data sources;
• Files system and programming framework;
• Data insights and tools.

Given this fact, we list some of the essential works conducted. It might share the same “main opinion” and conclusion regarding technicity with other research types, from which we drove the decisions to go further in our study.

This section focuses only on the research related to the big-data-based analytics systems that come as a solution to the traditional data processing and data storage systems, data warehouse [6].

For example, we can cite the work of Vadivu et al. [33]. The authors used big data techniques to conduct spatial distribution-based analytics in order to study crop concentration in their work. However, the authors also evaluated the impact of spatial distribution-based analytics on pre-green and post-green revolutions without considering the culture seasons. Thus, our research attempted to focus on different seasons. At the same time, our research determines a data ingestion strategy beginning with data sources with the Hadoop data repository [34].

Our research aims to propose a framework that tackles all of the challenges and requirements mentioned above, favoring the facilitation of agricultural practices. Our primary purpose is to deliver the big data application to the end-user (i.e., the farmers). These outcomes can be accessed via laptop, phone, tablet, or any smart device. The information will react based on the rules defined by machine learning algorithms, guaranteeing the smart-farming analytics system’s sustainability.

6. Data Sources and Low Power Consumption Embedded Devices

Many traditional farming apps have been innovative by adding devices, tools, and resources. In general, however, these added technologies require a wireless solution that allows a much longer range since they must transmit the signal over the large distances usually covered by farms or agricultural estates. In addition, given the remoteness of the sites where they are used, these long-range transmission devices are often powered by battery or solar panels and therefore have to consume little energy. Thus, the ideal technological solution for these intelligent farming operations is to use an LPWAN (low-power wide-area network) and other Low Power Consumption Embedded Devices.

Nowadays, IoT has become one of the most promising technologies to deal with modern “smart” life systems, organizations, environments, industries, and farms [35]. IoT is heavily present in agriculture management processes and is deployed to increase productivity [36]. These IoT applications generate massive datasets [37]. Thus, the importance of data is gradually growing in IoT. As a result, a new concept has appeared called “Data of Things (DoT),” which refers to any technique that manages data within a process that is based on IoT technologies or includes one of the IoT technologies [38]. This research determines some big data management techniques within agricultural processes, including IoT technologies.

IoT technologies promote a way to deliver agriculture services at scale [39]. However, the accuracy of these services’ insights, actionable information, and content depends on their capacity and ability to aggregate different data sources and process all of the insightful data. From the same perspective, Bill Gates, the former chairman and CEO of Microsoft, said, “More Data, Better Farms.”

Data is a critical component that provides more support to farmers and enables them to access financial, extension, and trade services. As shown in Figure 3, from an administrative point of view, these services can improve the handling of personal, geographical, financial, production, business, farms-details, and field information data, then driving insights and decisions.

Recently, data in agriculture has significantly increased, and the data has become more and more available. Governments strongly advocate open data to authorize farmers and scientists to benefit from these massive generated data and the desire to be more transparent. The private sector and the research institutions generate a critical amount of data, and they are ready to share it as a standard benefit. The practical part of this research is based on open data, mostly from governmental websites.

Generally, there are three types of data generated in agriculture, as follows:

• Data Mediated by Process (DMP);
• Data Generated by Machine (DGM);
• Data from Human Origin (DHO).

This section below will define each type of generated agriculture data, and we will also describe the data used in this research.

7. Data Mediated by Process (DMP)

The DMP is the data generated during agriculture, such as applying fertilizers, customer complaint submissions, etc. and it is stored automatically in an information system (IS) or manually in files (e.g., Excel files). This data is mainly generated from IS, e.g., CRM, ERP, Laboratory Information Management System (LIMS). The data could be stored in a relational database system. The relational database concept links two-dimensional tables containing columns and rows, as shown in Figure 4 (example from India). Both rainfall and temperature tables were used in this research.

From a smart-farming analytics perspective, these RDBMS will be considered as a data sources. It will not be used for the decision-making process because they scale vertically. The data cannot be distributed through different clusters to handle the huge amount of data that is generated from IoT-devices and smart-machines.

It is necessary to mention the following points:

• Relational-programming is non-procedural and can operate on a set of rows at a time;
• The content of a row can be referred to as a record;
• The column can be referred to as a field;
• The database schema, which is a logical organizational unit inside the database, is where the tables are stored;
• This relational approach has lent itself to the structured query language (SQL), which was defined initially by an IBM study, then introduced by Oracle Corporation in 1979. SQL can be used in the following ways:

  (1)

  For querying using SELECT statement;

  (2)

  As a DML or data manipulation language to INSERT, UPDATE, and DELETE tables;

  (3)

  As DDL or data definition language to CREATE or DROP;

  (4)

  To GRANT or REVOKE while setting privileges for users or groups.

In a relational database, the column relationships between different tables are designated using a key, which is effectuated through referential integrity constraints and their supporting-indexes.

We can establish a link between the rainfall and temperature table based on the column Rainfall_ID; in this case, we assumed that a unique key on the rainfall table exists as a foreign key on the temperature table.

Based on the functional requirements, we can build the link between tables, depending on standard fields. In our case, we can have the concatenation of the four columns (year, month, country, state, and district) as standard key in the form of a list that a function can process to check whether we can join the records or not, demonstrated as follows:

• Constants
• n ∈ N and a ∈ R₊
• Variables
• B = [List[a], …, List[n]] and C = [List[a], …, List[n]]
• currentTemperatureRecord ∈ R and currentRainfallRecord ∈ R
• bypass: Boolean
• Begin
• currentRainfallRecord := 1;
• C := rainfallRecordsProcessing(currentRainfallRecord);
• bypass := 0;
• for eachI → C do
• currentTemperatureRecord := 1;
• B := temperatureRecordsProcessing(currentTemperatureRecord);
• for eachJ → B do
• ifC[I] !=B[J] then
• bypass = 1;
• exit();
• ifbypass == 0 then
• joinRecord(C, B)
• whiletrue do
• currentTemperatureRecord += 1;
• b := temperatureRecordsProcessing(currentTemperatureRecord);
• for each J → B do
• ifC[I] !=B[J] then
• bypass = 1;
• exit();
• joinRecord(C, B)

In this work, to put in practice the theoretical solution and simulate the agricultural process at the data management’s level, an Oracle database was used to store rainfall, temperature, and drought status in their raw format, as described by farmers. Excel files were used to fill in the crop statistics data.

Figure 5 shows the rainfall and temperature tables’ description that contains the rainfall rate for each month for the years between 2001 and 2016 (each column represents a month, and each row represents a year). Figure 5 also shows the description of the Excel file that contains the crops statistics. It is necessary to mention that a REST API is also used to gather pressure data (details in the data-ingestion section) from a governmental website for a specific state, district, and year. Therefore, it will be stored in its raw format with the same description as rainfall and temperature (as described in Figure 5). Based on these parameters, we use a formula (described in the data-ingestion section) to label the drought status, as illustrated in Figure 5.

Figure 6 shows both tables related to the mechanical system maintenance (available detail in the DGM section below). This type of analysis is associated with the industrial part of smart farming. The farmers store their data to have visibility on their interventions (the status and the cause of failure) conducted on their equipment. Therefore, this DMP is mainly stored on RDBMS to be linked with DGM for a data warehousing perspective.

8. Data Generated by Machine (DGM)

Nowadays, we are using sensors, smart devices, and satellite imagery to collect data related to agricultural practices. This data is mostly generated from IoT devices, “smart” machines, and intelligent robots or drones [40,41,42]. This DGM mainly comes in streaming, micro-batches, and small-batches of data that can be collected and processed in real-time. In this research, we defined the primary DGM sources and the technical challenges related to collecting and processing this DGM and exploring the data storage techniques.

IoT and Wireless Connection: The world society is expected to be heading towards more digitization by 2030. Data will drive decisions and stimulate intelligent systems and processes that need a near-instant, ubiquitous, wireless connectivity [43,44].

With the IoT support of the sensor’s interconnectivity, smart farming could benefit from numerous real-time monitoring and analysis opportunities. IoT-based solutions are investigated in order to help drive decisions in many agricultural areas, mainly water management and agriculture monitoring. These solutions can collect data from the farm’s environment using networking protocols and standards, such as RFID, 4G/5G, LoWPAN, IEEE 802.15.x, and the future 6G [45,46,47,48,49,50,51,52].

The main objective behind using IoT in intelligent farming is interconnecting common physical objects and devices through the internet. Several IoT devices can be used for water management and intelligent irrigation analytics [53]. For example, we can cite pH sensors that measure the soil’s nutrient content, moisture sensors that measure the ground’s dielectric constant, and temperature sensors that measure both farm and equipment temperature [54,55]. Internet-experts predicted that IoT’s extension and growth would be recorded around 50 billion physical objects and more by 2025.

9. Data Modeling

The success of data-driven analytics for smart farming is related to how organized the data management is. Therefore, we should develop actions, policies, and architectures to manage the fundamental requirements of the data life cycle of the farm properly. The development of data architecture for smart farming involves accurate rules (for governing the data collection), data models, data storage, data processing, and integration. Therefore, a well-designed data model should be built in order to achieve efficient smart-farming analytics.

For example, the relation between the various data should be well defined. Therefore, the farm databases/sources (DMP, DGM, and DHO) should be well integrated.

Once the data has been gathered and prepared for analysis, various data science and advanced analytics disciplines can be applied for smart-farming analytics.

When it comes to big-data modeling, other criteria should be considered. For example, integrating the existing farming data with the new data, on top of a Hadoop architecture, can be completed while considering the different types of data formats. In addition, other features, such as data compression, capacity, availability, reliability, data integrity, and data security, are all critical features that should be considered.

It is necessary to remember that the farm’s data is available in different sources (mostly RDBMS, IoT, and Excel files) and is probably stored with or without schema. Additionally, for smart farming analytics, we need to model the farm’s data so that it will be stored in one “source of truth,” which can be considered a centralized data warehouse of the farm inside the data lake.

As we described in the previous section (i.e., Data sources), the relational data model was proposed by IBM to represent data in the form of tables in order to reduce the complexity of the “previous” hierarchical-model and deliver an understandable overview of the data. In the same context, the transactional data model can be built to store discrete data of the “farm” business transactions, such as inventory, purchase orders, sales, etc. It is mainly how the DMP are modeled in their source system, throughout the procedure of normalizing the large raw tables and splitting them into smaller ones, then creating relationships between these tables using reference integrity constraints and respecting the three normal forms (3NF).

The problem with this approach in smart-farming analytics is area/business-specific. Each database stores only the data related to a particular functional area (e.g., the productivity database stores only production- and productivity-related data).

On the other hand, smart-farming analytics necessitates storing transactional data for multiple areas, such as spatial distribution (including climate, productivity, and global-issues data), water management, and mechanical system maintenance, in a single “target” source (data lake). The second problem is that transactional databases use primary key.

Still, many critical DGM and DHO for farming business logic come without a primary key in smart farming. As a result, we will have duplicates in our tables, significantly impacting the analytics and decisions.

The third problem is, retrieving data from a normalized database will involve several joins across these many split tables where various pieces of the farm data are stored. But for smart farming, the analysis and the analytics should be conducted and refreshed continuously. Thus, denormalizing the tables will significantly enhance the data retrieval by adding redundancy (copying values between tables) to the farm’s data lake and reducing the number of joins on the queries. This denormalization will also provide aggregation and calculation capabilities to the analysis and help to extend the calculated values for smart-farming analytics, which requires a lot of time and slowing-down of the query’s execution, calculating them on the fly in a normalized database that does not contain these values.

10. Data Science for Smart Farming

Data science is the study of large quantities of disparate data and extracting insights and knowledge to aid the organization in making tactical, strategic, timely, and correct decisions. This process involves statistical analysis, mathematics, machine learning, data visualization, etc.

In agriculture, farmers have always been collecting (manually or automatically) a massive amount of data for each agriculture season, e.g., applied inputs, harvested crops, planted deeds, etc.

The growth of new digital technologies (satellites, remote-sensors, and drones) has caused the rise of digital farming, including temperature, utilization of nitrogen, and soil conditions [56,57].

Here, data science can help farmers to personalize their decisions regarding each farming process, using AI tools to speed up analyzing data and bringing back valuable insights to farmers. Data science is a generic term that includes the complete data processing methodology, where AI encompasses each tool that enables computers to learn problem-solving and decision-making skills.

Machine learning is a subset of AI that makes accurate decisions, uses algorithms to analyze the data, and learns from this process without programming it explicitly. Machine-learning models use large datasets for training and learning as things progress from these examples, but not from defined rules (unlike the rule-based algorithms).

These techniques can be used for predictive analysis, fraud detection, market basket analysis, recommendations, etc. in order to define a decision rule from which the strategic decisions will be conducted.

11. Conclusions and Perspectives

The digitalization of agriculture is based on developing and introducing new tools and machines in production.

The future smart farming revolution (or agriculture 4.0) will use advanced data-acquisition technologies combined with network-communication technologies.

Smart farming will also incorporate intelligent processing technologies. All of these outlooks involve data management in a certain way. As we described in this research, smart-data management can provide a wave of solutions to facilitate the farmers’ pain when dealing with their daily processes by employing one. We proposed a Smart Farming Oriented Big-Data Architecture that provides a platform for all of the intelligent data management requirements and manages data processes in different layers that execute the tasks accordingly, allowing for more flexibility and agility.

In this research, we have proposed some of the main technical constraints, a data-quality process, to model the farm’s data based on business logic and compare machine-learning algorithms. Our future perspective is to build an abstraction layer on top of the SFOBA that automatically handles them based on rules defined in the system, without adding them as actions on the directed acyclic graph of the ingestion workflows. The project management aspect, DataOps methodology, and the agile methodology, give an enterprise tincture to the smart farm data management.

At length, our research’s central perspective is to ensure agricultural sustainability and environmental protection, secure the energy consumption, and reach a reasonable level of quality, productivity, and high efficiency.