Evaluation of a Smart System for the Optimization of Logistics Performance of a Pruning Biomass Value Chain

Featured Application: This work can be used as benchmark for performance evaluation of tools to improve supply chain management in general and solid biofuel supply chain in particular. The potential application is for integrated goods distribution management that promotes efﬁcient resource utilization and sustainability. Abstract: The paper presents a report on the performance evaluation of a newly developed smart logistics system (SLS). Field tests were conducted in Spain, Germany, and Sweden. The evaluation focused on the performance of a smart box tool (used to capture information during biomass transport) and a web-based information platform (used to monitor the ﬂow of agricultural pruning from farms to end users and associated information ﬂow). The tests were performed following a product usability testing approach, considering both qualitative and quantitative parameters. The detailed performance evaluation included the following: systematic analysis of 41 recordable parameters (stored in a spreadsheet database), analysis of feedback and problems encountered during the tests, and overall quality analysis applying the product quality model adapted from ISO/IEC FDIS 9126-1 standard. The data recording and storage and the capability to support product traceability and supply chain management were found to be very satisfactory, while assembly of smart box components (mainly the associated cables), data transferring intervals, and manageability could be improved. From the data retrieved during test activities, in more than 95% of the parameters within 41 columns, the expected values were displayed correctly. Some errors were observed, which might have been caused mainly by barriers that could hinder proper data recording and transfer from the smart box to the central database. These problems can be counteracted and the performance of the SLS can be improved so that it can be upgraded to be a marketable tool that can promote sustainable biomass-to-energy value chains. three evaluate the of similar features of the with different smart data triangulation test approach to test capability of central information platform by analyzing data sourced from testing with three different


Introduction
The use of pruning biomass for renewable energy production is one of the renewable energy uses being promoted in Europe. This study was part of an EU EuroPruning project, "Development and implementation of a new and non-existent logistics chain for biomass from pruning", which was aimed at developing improved logistics chains for biomass from agricultural pruning residues [1]. This includes the development of a new decision tool for a pruning biomass trading system and logistics management (see Figures 1 and 2). To fulfill this, a smart logistics system (SLS) was developed as described by Gebresenbet et al. [2]. Computer technology has changed the historical face-to-face communication in trading systems [3], and the SLS enables utilization of recent technology.
Understanding and managing the material and information flow (see Figure 1) enables efficient and effective utilization of resources. The SLS developed for this purpose has been reported in detail in [2]. But there are associated research questions: What is the performance of the SLS tool in real conditions? What is the feedback of end users of the tool?
The objective of this study was to evaluate the performance of the SLS under field conditions. This is important, as the SLS is a newly developed tool and evaluating its performance is important. This report describes the methodology applied and the results obtained from applying the SLS tool in real conditions. This is important for replication of the study and improvement of the SLS tool.
Figures 1 and 2 describe activities at different stages of a logistics chain. Pruning fruit trees (removing top or unwanted branches) (see Figure 2a) is done by farmers on an annual or biennial basis to maintain the desired tree form and structure and to increase the productivity of fruit trees. Pruning harvesting is often integrated with chipping or baling (see Figure 2b). The harvested product can be stored in the form of bales or chips (see Figure 2c). Storage of biomass is related to seasonal variability and logistics operation cost, and storage locations should be efficient to reduce transportation and operation costs. The location of biomass storage can vary depending on the nature of the biomass supply chain [4]: on-field storage, intermediate storage (between the fields and power plants), and storage facility at the biomass power plant. Biomass transportation is an important activity in biomass-to-energy systems. This includes both on-farm transportation and main transportation (from farm to storage or power plant).

Smart Logistics System and Its Utilization by Different Actors
The SLS has four major components, as indicated in Figure 3 and presented in detail by Gebresenbet et al. [2]. The smart box is a sensor unit for measuring parameters such as relative humidity, temperature, geographic position, and route tracking, and information associated with quick read (QR) codes. The onboard control unit is used for planning transport routes and monitoring recordings by Cargolog. The information platform is used for documentation and data sharing, and to facilitate biomass trading and management of the pruning supply chain and traceability. The central control unit is a point of administration of the biomass trading and logistics system and links the information platform and onboard control unit.

Smart Logistics System and Its Utilization by Different Actors
The SLS has four major components, as indicated in Figure 3 and presented in detail by Gebresenbet et al. [2]. The smart box is a sensor unit for measuring parameters such as relative humidity, temperature, geographic position, and route tracking, and information associated with quick read (QR) codes. The onboard control unit is used for planning transport routes and monitoring recordings by Cargolog. The information platform is used for documentation and data sharing, and to facilitate biomass trading and management of the pruning supply chain and traceability. The central control unit is a point of administration of the biomass trading and logistics system and links the information platform and onboard control unit.
The two components of the SLS (the smart box and information platform) have been integrated so that the smart box is functionally connected with the web address of the information platform [5]. The smart box records and transmits data to the central information platform. The information platform is designed to facilitate interactions among biomass supply chain actors and data collection, as well as the management of the entire logistics of the pruning biomass supply chain (see Figure 2 and Table 1). Table 1 presents how different actors use the platform to interact regarding the flow of materials and related information along the pruning biomass supply chain. The two components of the SLS (the smart box and information platform) have been integrated so that the smart box is functionally connected with the web address of the information platform [5]. The smart box records and transmits data to the central information platform. The information platform is designed to facilitate interactions among biomass supply chain actors and data collection, as well as the management of the entire logistics of the pruning biomass supply chain (see Figure 2 and Table 1). Table 1 presents how different actors use the platform to interact regarding the flow of materials and related information along the pruning biomass supply chain. The SLS will be used to improve the performance of the biomass supply chain focusing on pruning biomass. In evaluating the performance of supply chain management, it is important to identify the performance measurement metrics within the context of the supply chain under evaluation [6]. The overall performance of a supply chain depends on the role of each stakeholder in

Action
Actor Description of Actions to Be Performed

Add new user Administrator
Logistics manager who administers the smart system can add new users to the system. New label codes will be assigned to new actors.
Add new pruning Farmers Farmers have to input pruning quality parameters required by end users.

Edit pruning Farmers
Farmers can update input information such as pruning quality parameters while it is in "pending" status.
Delete pruning Farmers Farmers can delete pruning as long as it is still pending and has not been flagged as "ready".

Make ready Farmers
Farmers can flag pruning as "ready" so that consumers can view that pruning and order it.
Generate QR code Farmers Farmers can generate a quick read (QR) code for "pending", "ready", and "ordered" pruning.

Order pruning Consumers
Consumers order pruning based on quality parameters. Consumers select a transporter from a list registered in the central system.
Add to shipment Transporters Transporters select orders to be included in one shipment in order to plan the route between locations of chosen orders.

Plan route Transporters
Transporters generate driving instructions for source and destination locations for all orders included in one shipment.

Add to pruning Traders
Traders use the tool to select which orders are part of a new treated pruning offer. Trader can add new pruning quality parameters, e.g., if bales are chipped.
The SLS will be used to improve the performance of the biomass supply chain focusing on pruning biomass. In evaluating the performance of supply chain management, it is important to identify the performance measurement metrics within the context of the supply chain under evaluation [6]. The overall performance of a supply chain depends on the role of each stakeholder in the chain [6,7]. Shashi et al. [6] discussed that stakeholders' interest, value addition, and partners' performance play important roles in the overall performance of a supply chain.
The rest of this paper is structured as follows. Section 2 describes the testing of the SLS tool and evaluation methodology. Section 3 presents the performance evaluation results and discussion, while Section 4 presents the conclusion.

Usability Testing
Usability testing is an evaluation method that has the largest impact on product [8]. It is one method of evaluating the learning and use of new products. In this study, the usability testing approach was used considering six basic characteristics: system to evaluate, focus, participants, tasks, data, and results [8]. These characteristics were applied as explained in Table 2. Both qualitative and quantitative data analyses were used in the evaluation of SLS. Table 2. Usability testing characteristics.

System to evaluate
Used to evaluate the SLS. Usability testing is an evaluation approach that can be applied to evaluate almost any product or technology; for example, software for database management, network management tools, early-stage prototypes, and related help manuals.

Focus
The focus was on usability of the SLS. The test was intended to validate the first release of the smart system (smart box and information platform) prototype, with less consideration of marketability of the product.

Participants
The intended main participants were (potential) end users of the smart system who were actors of the pruning supply chain (farmers, pruning traders, transporters, power plants). Test participants and administrators interacted during tests. Training and instruction guides were provided before starting. During this testing, research centers such as Leibniz Institute for Agricultural Engineering Potsdam-Bornim (ATB) in Germany, Research Centre for Energy Resources and Consumption (CIRCE) in Spain, and the Swedish University of Agricultural Sciences (SLU) as well as Gruyser (transport dealer) participated actively.

Tasks
The smart system has different functional features. Each participant considers the specific functionality of the system that best fits him/her. Test participants who act as farmers perform tasks that the pruning producers perform, while traders perform what processors and traders of biomass perform using the system. Similarly, transporters and consumers test features of the system that serve the transport company and end consumers of the pruning biomass.

Data
Pruning biomass-related data were recorded, stored in a database, and analyzed. Both problems and positive aspects noticed during tests were analyzed. Data collected with three prototypes were used to evaluate the functionality of similar features of the system but with different smart box prototypes. Accordingly, a data triangulation test approach was used, especially to test data storage and display capability of the central information platform by analyzing data sourced from testing with three different prototypes.

Results
Test results were used to improve the smart system and communicate the research outputs. These analysis results are documented, archived, and used to identify what problems surfaced and how to solve them.

Metrics for Performance Evaluation
The SLS deals with data recording, transfer, and storage in a centralized database through the integrated action of a smart box and a web-based information platform (see Figure 3). The functional features of the smart box tool and information platform were integrated using Cargolog PC software, Cargolog FAT90 V2 (Mobitron, Huskvarna, Sweden). In this report, as part of the evaluation of the smart system, metrics were selected and adapted from international standards for systems and software quality requirements and evaluation (ISO/IEC 25010:2011) as described in [9] and Table 3. Table 3. Quality attributes considered for performance evaluation of the SLS.

Functionality
How easy it is for the system to integrate its functional units while maintaining the security and accuracy of service provided Reliability How well the smart system can provide service with required precision Usability How easy it is for the smart system to learn, operate, and analyze the data and make decisions Efficiency Amount of resources and time required by the system to perform its intended function

Maintainability
How easy it is for the system to identify and fix an error Portability How easy it is to move the smart box from one place to another and to move the application platform from one server to another

Product Quality Model
The general product quality model for internal and external quality evaluation of information technology (software product) described in ISO/IEC FDIS 9126-1 (see Figure 4) was adapted for evaluation of the smart system [10]. The evaluation is based on the definitions given in international standards for information technology-based product quality (ISO/IEC FDIS 9126-1) as indicated in Table 3 and Figure 4. The detailed analysis results are presented in Table 9.

Metrics for Performance Evaluation
The SLS deals with data recording, transfer, and storage in a centralized database through the integrated action of a smart box and a web-based information platform (see Figure 3). The functional features of the smart box tool and information platform were integrated using Cargolog PC software, Cargolog FAT90 V2 (Mobitron, Huskvarna, Sweden). In this report, as part of the evaluation of the smart system, metrics were selected and adapted from international standards for systems and software quality requirements and evaluation (ISO/IEC 25010:2011) as described in [9] and Table 3. Table 3. Quality attributes considered for performance evaluation of the SLS.

Functionality
How easy it is for the system to integrate its functional units while maintaining the security and accuracy of service provided Reliability How well the smart system can provide service with required precision Usability How easy it is for the smart system to learn, operate, and analyze the data and make decisions Efficiency Amount of resources and time required by the system to perform its intended function Maintainability How easy it is for the system to identify and fix an error Portability How easy it is to move the smart box from one place to another and to move the application platform from one server to another

Product Quality Model
The general product quality model for internal and external quality evaluation of information technology (software product) described in ISO/IEC FDIS 9126-1 (see Figure 4) was adapted for evaluation of the smart system [10]. The evaluation is based on the definitions given in international standards for information technology-based product quality (ISO/IEC FDIS 9126-1) as indicated in Table 3 and Figure 4. The detailed analysis results are presented in Table 9.

Smart Box Installation and Training
Before starting the actual testing in Sweden, Spain, and Germany, training/guidance was provided on proper installation and operation of the smart system. In addition to this training, two user guide documents were prepared and provided to actors involved in testing the tool. The first user guide enables smart box users to install Cargolog PC software on computers for onboard monitoring purposes and to mount the smart box tool (with metal case) appropriately on the truck (see Figure 5). The second guide enables users to use the information platform efficiently.
Before starting the actual testing in Sweden, Spain, and Germany, training/guidance was provided on proper installation and operation of the smart system. In addition to this training, two user guide documents were prepared and provided to actors involved in testing the tool. The first user guide enables smart box users to install Cargolog PC software on computers for onboard monitoring purposes and to mount the smart box tool (with metal case) appropriately on the truck (see Figure 5). The second guide enables users to use the information platform efficiently.

Smart System Field Testing
In this study, three prototypes were developed. The testing and evaluation activities were done with all the three prototypes during 2015-2016. Although testing of the smart system was done during different stages of the development of the system, this report presents only the testing and evaluation results of the final smart system prototype used by end users during final testing activities. The smart boxes were used at farm-to-storage and storage-to-consumer transport stages.

Data Retrieved from Central Database of the Smart System
In the spreadsheet-based central database, the data are stored in 41 columns (see Table 4 and Appendix A). Data entered by actors directly on the platform and data transferred from the smart box are stored mainly in the 41 columns, while some additional information can be displayed and

Smart System Field Testing
In this study, three prototypes were developed. The testing and evaluation activities were done with all the three prototypes during 2015-2016. Although testing of the smart system was done during different stages of the development of the system, this report presents only the testing and evaluation results of the final smart system prototype used by end users during final testing activities. The smart boxes were used at farm-to-storage and storage-to-consumer transport stages.

Data Retrieved from Central Database of the Smart System
In the spreadsheet-based central database, the data are stored in 41 columns (see Table 4 and Appendix A). Data entered by actors directly on the platform and data transferred from the smart box are stored mainly in the 41 columns, while some additional information can be displayed and visualized on the platform. It is important to test the performance of the smart system by considering how correctly the recorded values and information are displayed in each of the 41 columns. This enables us to understand the performance of the smart system and identify the columns where incorrect information might be displayed. In general, during this test, about 104 records of product lines (with specific lot numbers) were retrieved from the SLS database. This in turn enabled us to identify functional features of the SLS linked to errors in displayed values and to figure out what further improvement would be needed to upgrade the system.

Data on Product Delivery Information
Using the SLS tool, the system administrator can easily identify products entered into the system, products for which transport routes are planned, and products that have been delivered to end users. Table 5 indicates that, out of the recorded 104 products, route planning was tested for 54%, while actual product delivery of 16% was confirmed according to the data captured by the smart system. * Percentage of delivered products does not necessarily indicate efficiency, because some registered products may not be ordered by traders/consumers; for some products, route may be planned but they may not be delivered to consumers. NA, not available.

Data Recorded and Visualized Directly on Platform
The feedback from users regarding the attractiveness and user-friendliness of the functional features of the smart system is very important to improve the system. The feedback may be on utilization of functional features to perform specific actions and/or visualize the results.
Samples of visualizations of recorded data are presented in Figures 6-9, which describe information gathered with smart box 2015085119 (see Figure 6) at a test site in Germany during transport of pruning products registered as Field3, lot number 151008B03R1.          The relative humidity data retrieved from the testing activity in Germany are presented in Table  6, together with the moisture content of the biomass at the storage site. It must be noted that relative humidity indicates the humidity in the air inside the truck box after loading the woodchips. In the initial moment, the moisture measured was just the air moisture at the site. Once the probe (see Figure  5a) was inserted into the truck box, the air in the internal void volume among the woodchips The relative humidity data retrieved from the testing activity in Germany are presented in Table 6, together with the moisture content of the biomass at the storage site. It must be noted that relative humidity indicates the humidity in the air inside the truck box after loading the woodchips. In the initial moment, the moisture measured was just the air moisture at the site. Once the probe (see Figure 5a) was inserted into the truck box, the air in the internal void volume among the woodchips increased its moisture content until it reached equilibrium with the moisture of the woodchips.

Information Flow and Product Traceability Performance of the Smart System
The information gathered using the smart system was rich enough to construct an effective traceability map of each registered product along the pruning supply chain [11]. Detailed pruning quality parameters and the product traceability system are provided in [12,13]. The SLS was integrated with the biomass traceability system, with pruning quality and related traceability parameters. The usability testing indicated that all of the traceability-related parameters were implemented on the information platform of the SLS (see Figure 3). The identification of producers (provider ID), transporters (transporter ID), and consumers (consumer ID), and the specific name (title) and production lot number assigned to each registered product are mandatory parameters to trace product movement along the pruning supply chain (see Table 4). Table 7 indicates that except for smart box number (Cargolog serial no.), delivery code, pickup date, and delivery date, all important parameters are recorded correctly. The errors in records of Cargolog serial no., pickup date, and delivery date could be mainly due to barriers in data transfer from the smart box to the central database. In the case of Spain, the Cargolog serial no., pickup date, and delivery date were not recorded at all in the central database, indicating that the information was not properly transferred from the smart box tool to the central database.
For illustration purposes, a product from the test in Germany is considered here. For a product registered with the title "Field7" and lot number 151008B05R1, the producer, transporter, and consumer are identified as FDEU020, DDEU012, and CDEU013, respectively (see Appendix A). The corresponding complete product identification code can be constructed as FDEU020-DDEU012-CDEU013. Whenever a product traceability issue is initiated, further detailed traceability-related information can be retrieved from gathered data for each stage of the logistics chain. This can be done by an experienced administrator of the smart system who can interpret all recorded values in each of the 41 columns of the spreadsheet-based central database. The data barriers in delivery date (column 38) and Cargolog serial number (column 7) may hinder the system administrator from confirming the final delivery of the product to the intended consumer. In such cases, the administrator should find additional information by telephone or internet conversation, for instance, by introducing an option (on the platform) for the end consumer to confirm the product delivery. This enables the system administrator to easily construct the product delivery identification code and strengthen the traceability and management of the entire pruning supply [10]. The smart system was developed to effectively gather, monitor, store, and analyze data to improve pruning logistics management (even though the tool could be utilized for any biomass supply chain). As indicated in Table 8, this integrated system enables improved performance of pruning supply chain management and product traceability by reducing biomass loss (in terms of quality and quantity), increasing the quality of solid fuel and delivery service, and reducing logistics and transaction costs [2,11,14]. Therefore, evaluation results indicate that the SLS is important tool for management of the biomass value chain and trading system. It enables tracing and tracking of the product, controls product quality, facilitates information flow, reduces management cost, and enables managers to manage the entire supply chain. Table 8 indicates that most of the major functional features performed satisfactorily. System administrator, consumer √ * This analysis was done assuming that the system will be used by trained actors and system managers. 1 = poor; 2 = fair; 3 = good; 4 = satisfactory; 5 = very satisfactory.

Evaluation of Smart System Using Product Quality Model
In this section, the product quality model is used to analyze the performance of the smart system. The feedback from the testing was systematically used as input information for the product quality model. This input includes observations of the experts participating in testing the tool (see Table 9). Table 9. Evaluation of the SLS using product quality model for internal and external quality assessment Table 9. Cont.

Attribute Smart Logistics System Performance Description
Security Confidentiality: Personal information of registered actors is confidential, as it can only be accessed by system administrators (of the information platform). Integrity: The system well integrates data recorded by the smart box and related data provided by pruning biomass producers. Accountability: All registered users are accountable for the information they provide to intended users of the information platform. The system enables pinpointing damages that could happen at any stage along the logistics chain through the pruning traceability system integrated in the smart system. Authenticity: All registered actors have specific codes and access to and recognition of data and information they provided.

Modularity:
The smart box components can be disassembled and reassembled by the prototype developer. This facilitates maintenance service. Reusability: The system can be maintained and reused. Maintenance service may be required on average once a year, and some parts of the smart box may need to be replaced.

Modifiability:
The information platform can be modified based on the interest of users or if additional service is required. Testability: Both the smart box and the information platform are testable. After appropriate maintenance, the system will be well tested before it is used.

Portability
Adaptability: The possibility of adapting to logistics of biomass other than pruning residues, such as forest wood products, was taken into consideration during the development of the smart logistics system.

Installability:
The smart box has a metal case with a magnetic foot for easy and appropriate mounting on a truck. The metal case is used to protect the smart box from mechanical damage, aggressive gases, liquids, and humidity. Packaging: The smart box needs to be shipped from its manufacturing place to end users. The packaging system for this should be hard enough for protection from damage due to impact. Improved packaging is recommended for this purpose.

Major Recommended Improvements
When the SLS was conceived, it was designed for use in supporting and improving biomass logistics, not only for pruning biomass, but for any type of biomass, such as forestry woodchips and herbaceous agrarian residues. For this purpose, further improvements are suggested so that the SLS can be effectively adapted for biomass-to-energy businesses. The performance test results indicate that the SLS performs data collection and storage in the central database satisfactorily. The stored data and other relevant information enable the tool to support product traceability and supply chain management very effectively. For further improvement and service quality, assembly of the smart box components (mainly the associated cables) and data transfer intervals as well as column arrangement in the spreadsheet format for data storage could be improved more.
In general, each batch of pruning biomass data entered into the online platform directly by actors (farmers, traders, etc.) was found to be well documented in the central database and easily visualized by intended users. However, some errors were noticed regarding data on parameters measured by the smart box (such as relative humidity, temperature, Global Positioning System (GPS) coordinates during biomass transport) and transmitted to the database. These errors may be caused by improper initialization of the smart box, unplugging the power cables before data transmission is complete, or unexpected barriers that could interrupt data transmission from the smart box to the online platform.
The smart system enables users to determine the transport distance and truck speed at the route planning stage. However, the real routes followed during transport are displayed only on maps. The actual driving distance and time (or speed) are not determined and displayed along the route maps. Through further improvement, these values can be either displayed on the maps or documented in the spreadsheet-based central database. To increase effective utilization of the information platform, additional functional options should be created (on the platform interface) for transporters and end consumers where they can confirm product delivery and acceptance, respectively. In addition, a billing system could be included as a component to facilitate trading of biomass products.

Conclusions
The testing and evaluation of a smart logistics system was carried out in order to identify functional limitations and important improvements to be implemented. The functional features of the smart box and centralized information platform were considered during testing. The features were designed to enable users to have an effective information acquisition, monitoring, and utilization system while promoting sustainable pruning for the energy value chain. All three smart box prototypes were tested together with the information platform. Based on tests done in Spain, Germany, and Sweden, the performance evaluation of the smart system was carried out by systematically analyzing: Appl. Sci. 2018, 8, x FOR PEER REVIEW 16 of 20

Conclusions
The testing and evaluation of a smart logistics system was carried out in order to identify functional limitations and important improvements to be implemented. The functional features of the smart box and centralized information platform were considered during testing. The features were designed to enable users to have an effective information acquisition, monitoring, and utilization system while promoting sustainable pruning for the energy value chain. All three smart box prototypes were tested together with the information platform. Based on tests done in Spain, Germany, and Sweden, the performance evaluation of the smart system was carried out by systematically analyzing:  values of parameters recorded and stored in the spreadsheet database with 41 columns (for each pruning product registered with a specific lot number),  feedback and problems encountered during the test, and  selected performance metrics in relation to product quality model adapted from ISO/IEC FDIS 9126-1 standard.
The performance test results indicate that the smart system satisfactorily performed data collection and storage in the central database. The stored data and other relevant information enabled the tool to support product traceability and supply chain management very effectively.
In general, for each batch of pruning biomass (with specific lot number) entering and leaving the biomass supply chain, data entered into the online platform directly by actors (farmers, traders, etc.) were well documented in the central database and easily visualized by intended users. The information platform enabled users to display the locations of pruning farms, biomass storage and processing sites, energy plants, and the planned and/or actual delivery routes according to the needs of different actors, such as raw materials producers, traders, and consumers. However, some errors were noticed regarding the smart box identification numbers (expected to be transferred from the smart box to the central database during product pickup and delivery), product delivery codes (expected to be generated by the smart system), and product pickup and delivery times. These errors might have been caused by improper initialization (while triggering the smart box with a power scan), unplugging the power cables before data transmission was complete, or unexpected barriers that interrupted data transmission from the smart box to the online platform. To counteract these problems and increase information continuity along the pruning supply chain as well as effective utilization of the smart system, additional functional features should be created so that transporters and consumers can confirm product delivery and acceptance, respectively. This option enables retrieval of important information that could be missed due to problems encountered by the smart box during data transfer to the central database and increases system performance so that it can be upgraded to a marketable tool promoting sustainable biomass development for renewable energy generation.

Conclusions
The testing and evaluation of a smart logistics system was carried out in order to identify functional limitations and important improvements to be implemented. The functional features of the smart box and centralized information platform were considered during testing. The features were designed to enable users to have an effective information acquisition, monitoring, and utilization system while promoting sustainable pruning for the energy value chain. All three smart box prototypes were tested together with the information platform. Based on tests done in Spain, Germany, and Sweden, the performance evaluation of the smart system was carried out by systematically analyzing:  values of parameters recorded and stored in the spreadsheet database with 41 columns (for each pruning product registered with a specific lot number),  feedback and problems encountered during the test, and  selected performance metrics in relation to product quality model adapted from ISO/IEC FDIS 9126-1 standard.
The performance test results indicate that the smart system satisfactorily performed data collection and storage in the central database. The stored data and other relevant information enabled the tool to support product traceability and supply chain management very effectively.
In general, for each batch of pruning biomass (with specific lot number) entering and leaving the biomass supply chain, data entered into the online platform directly by actors (farmers, traders, etc.) were well documented in the central database and easily visualized by intended users. The information platform enabled users to display the locations of pruning farms, biomass storage and processing sites, energy plants, and the planned and/or actual delivery routes according to the needs of different actors, such as raw materials producers, traders, and consumers. However, some errors were noticed regarding the smart box identification numbers (expected to be transferred from the smart box to the central database during product pickup and delivery), product delivery codes (expected to be generated by the smart system), and product pickup and delivery times. These errors might have been caused by improper initialization (while triggering the smart box with a power scan), unplugging the power cables before data transmission was complete, or unexpected barriers that interrupted data transmission from the smart box to the online platform. To counteract these problems and increase information continuity along the pruning supply chain as well as effective utilization of the smart system, additional functional features should be created so that transporters and consumers can confirm product delivery and acceptance, respectively. This option enables retrieval of important information that could be missed due to problems encountered by the smart box during data transfer to the central database and increases system performance so that it can be upgraded to a marketable tool promoting sustainable biomass development for renewable energy generation.

Conclusions
The testing and evaluation of a smart logistics system was carried out in order to identify functional limitations and important improvements to be implemented. The functional features of the smart box and centralized information platform were considered during testing. The features were designed to enable users to have an effective information acquisition, monitoring, and utilization system while promoting sustainable pruning for the energy value chain. All three smart box prototypes were tested together with the information platform. Based on tests done in Spain, Germany, and Sweden, the performance evaluation of the smart system was carried out by systematically analyzing:  values of parameters recorded and stored in the spreadsheet database with 41 columns (for each pruning product registered with a specific lot number),  feedback and problems encountered during the test, and  selected performance metrics in relation to product quality model adapted from ISO/IEC FDIS 9126-1 standard.
The performance test results indicate that the smart system satisfactorily performed data collection and storage in the central database. The stored data and other relevant information enabled the tool to support product traceability and supply chain management very effectively.
In general, for each batch of pruning biomass (with specific lot number) entering and leaving the biomass supply chain, data entered into the online platform directly by actors (farmers, traders, etc.) were well documented in the central database and easily visualized by intended users. The information platform enabled users to display the locations of pruning farms, biomass storage and processing sites, energy plants, and the planned and/or actual delivery routes according to the needs of different actors, such as raw materials producers, traders, and consumers. However, some errors were noticed regarding the smart box identification numbers (expected to be transferred from the smart box to the central database during product pickup and delivery), product delivery codes (expected to be generated by the smart system), and product pickup and delivery times. These errors might have been caused by improper initialization (while triggering the smart box with a power scan), unplugging the power cables before data transmission was complete, or unexpected barriers that interrupted data transmission from the smart box to the online platform. To counteract these problems and increase information continuity along the pruning supply chain as well as effective utilization of the smart system, additional functional features should be created so that transporters and consumers can confirm product delivery and acceptance, respectively. This option enables retrieval of important information that could be missed due to problems encountered by the smart box during data transfer to the central database and increases system performance so that it can be upgraded to a marketable tool promoting sustainable biomass development for renewable energy generation. The performance test results indicate that the smart system satisfactorily performed data collection and storage in the central database. The stored data and other relevant information enabled the tool to support product traceability and supply chain management very effectively.
In general, for each batch of pruning biomass (with specific lot number) entering and leaving the biomass supply chain, data entered into the online platform directly by actors (farmers, traders, etc.) were well documented in the central database and easily visualized by intended users. The information platform enabled users to display the locations of pruning farms, biomass storage and processing sites, energy plants, and the planned and/or actual delivery routes according to the needs of different actors, such as raw materials producers, traders, and consumers. However, some errors were noticed regarding the smart box identification numbers (expected to be transferred from the smart box to the central database during product pickup and delivery), product delivery codes (expected to be generated by the smart system), and product pickup and delivery times. These errors might have been caused by improper initialization (while triggering the smart box with a power scan), unplugging the power cables before data transmission was complete, or unexpected barriers that interrupted data transmission from the smart box to the online platform. To counteract these problems and increase information continuity along the pruning supply chain as well as effective utilization of the smart system, additional functional features should be created so that transporters and consumers can confirm product delivery and acceptance, respectively. This option enables retrieval of important information that could be missed due to problems encountered by the smart box during data transfer to the central database and increases system performance so that it can be upgraded to a marketable tool promoting sustainable biomass development for renewable energy generation.