5.1. Flow Velocity Determination
The first goal of this work was to test crowdsourcing as a valid method for analyzing experimental radiography data, supporting the determination of meaningful process parameters. Therefore, a quantitative scenario for calculating the velocity of each marked particle based on the position extracted from the system output was designed. A pilot study was conducted with two domain experts and up to
distributed participants processing the assigned fragments as crowd workers. The workers were presented with a series of chunks taken from experimental datasets showing silo flows. The lengths of the chunks ranged from 10–140 frames, but typically contained at least 40 frames. For calculations, we used the superimposed positions of between seven and 19 individual trace particles, overlapping on separate frames. In order to compare flow for different conditions (dense, loose) and for different heights, particles located at the center of the silo were chosen. This approach reduced the displacement of particles from the main path of the flowing particle. The following four tables present calculated velocity results for each silo zone for initially dense (
Table 1 and
Table 2), as well as loose (
Table 3 and
Table 4) packing density conditions.
Table 1 and
Table 3 give an overview of the obtained numbers in pixels per frame, while
Table 2 and
Table 4 give the actual calculated velocity in mm per s. Intermediate state results are shown in pixels in order to give the reader a deeper understanding of the consecutive steps in the algorithmic procedure of using crowdsourcing for velocity determination and to demonstrate its consistency.
Table 1, which presents initially dense packing conditions, shows the four components of velocity: Vx: horizontal Velocity (upper left corner), Vy: vertical Velocity (upper right corner), Vt: total Velocity (bottom left corner), Vp: pointer index Velocity (bottom right corner); while
Table 2 gives a simplified view of Vx and Vy only, but given in mm/s.
Table 3 and
Table 4 give similar results for initially loose packing conditions, accordingly.
As anticipated, the higher overall velocity was observed in the center zones of each row (B, E, H) than in the side zones (A, D, G, C, F, I). The highest velocity was observed at the bottom of the hopper component of the silo: Zone H [
34,
49]. The comparison of dense and initially loose packing density flow, based on velocity component analysis (
Table 2 and
Table 4), produced a significant difference in Zone H, up to 50% in the overall velocity values. While the central zones for both conditions were similar (Zones E, B), the side zones (Zones D, F, A, C) differed (by more than 65% for Zones C and F), since the character and shape of the funnel varied. The increasing differences between the initially dense and initially loose packing conditions may be explained primarily by the varying size of the flow area (funnel area; see
Figure 2). The wider the funnel, especially in the upper part of the silo (Zones A, B, C), the greater the difference (assuming the silo outlet is the same size). These differences were also visible in the values of horizontal component velocity, where in the case of initially loose packing density, the absolute values inside of side zones were generally higher than for initially dense packing density (Zones A, C, D, F).
Figure 8 shows a velocity distribution map derived on the basis of the results obtained. Groups of similar velocity vectors were arranged in circles. It can be noted that similar results were obtained for the three main vertical Zones, A, D, and G (left of the funnel), C, F, and I (right of the funnel), and finally, B and E (upper center of the funnel). These results were consistent regardless of the condition, i.e., they were similarly situated on the velocity map for both loose and dense initial packing densities. The only exception was the velocity for H, which was the lowest funnel zone, just above the outlet where the particles gained the highest velocity. Therefore, given that the results for
(dense) and
(loose) were still within a moderate range of values conforming to theoretical expectations, the efficacy of the method was proven.
The results presented here were for the single-particle method, but the accuracy and precision calculated for all three methods were satisfactory and did not differ significantly. First of all, we considered accuracy compared to the ground truth baseline prepared by two experts. All the methods achieved comparable results, with variations of no more than 2%. Precision taken as the repeatability of results also reached 98%. Next, we examined the duration, i.e., the length of time required to complete the task by workers using different methods.
Table 5 shows aggregated times taken by workers to complete the tasks. The columns show the results for classical, zone, and single-particle methods, respectively. The rows show average results for processing a single frame, 10 consecutive frames, or 100 frames (from top to bottom, respectively). SD indicates the Standard Deviation. The single-particle targeted method was the fastest, as was to some extent expected. However, it should be noted that it consumed approximately 10-times less time than the classical method and almost four-times less time than the zone particle method. These factors were greater than anticipated, since there were no cases in which there were 10 trace particles; the average maximum number oscillated around four or five for most of the populated zones. The zone particle method was more than twice as fast as the classical method in all cases. Given the ability of crowdsourcing to parallelize jobs, it may be possible to speed up the entire process significantly. It is also worth noting that the SD was significant, since some frames or frame sequences were much more difficult to process (or simply required more time to process).
Table 6 shows average processing times for different zones. The most important zones were the central funnel flow zones, i.e., B, E, and H. However, no significant differences were visible, since both the processing time and SD remained close to the average values.
By quantitative analysis of radiographic images with the aid of the crowdsourcing system, it is possible to obtain a profile of the granular material velocity during the silo discharging process. The results provided additional knowledge about granular flows, making detailed comparative analysis of flow dynamics possible. Such analysis can be conducted on the basis of the calculated velocity profile derived from X-ray imaging data. The results obtained in our study are in agreement with previously-reported data [
34,
49].
The proposed crowdsourcing system enabled the distribution of the imaging data (image sequences) for different flow fragments (see
Figure 6 for the task allocation algorithm). The results showed better quality particle detection for frames pre-marked based on previous images in the sequence. More details on the development of the crowdsourcing method and system were given in [
11,
48]. In contrast, workers reported decreasing efficiency due to rising fatigue related to physical and cognitive workload demands over time when they worked with longer fragments of image sequences. Therefore, in the future, it would be interesting to investigate whether it would be beneficial to work with the system at random times chosen by the workers, of limited durations, possibly adjusted to the specific needs of the workers. Further development of the system itself, as well as of the crowdsourcing methodology for tomographic imaging analysis will be continued in the next stages of this research.
5.2. Qualitative Assessment: NASA TLX
In order to assess the workload of the participants, we performed NASA Task Load Index (TLX) tests. The participants completed a self-assessment rubric, in which they evaluated six main factors related to the given tasks, namely mental, physical, and temporal demand, how they perceived their performance in terms of quality and effort, and finally, the level of frustration induced by the task. These factors approximated to some extent the measurement of task complexity, the user experience, and the usability of the proposed approach, all in relation to the background of individual workers.
Diagrams of the TLX results are presented in
Figure 9, separately for the two conditions for the processed X-ray measurement results, i.e., dense silo filling (on the left-hand side) and loose silo filling (on the right-hand side). The colored bars for each TLX category represent results obtained using the three methods. The blue bar (always on the left in each group) shows the performance of the baseline crowdsourcing method, i.e., results achieved by crowd workers annotating all the trace particles in each frame. The orange bar (always in the middle) illustrates the performance of the zone tracking method, i.e., results achieved by crowd workers annotating trace particles bounded by a single area, as depicted in
Figure 4. The green bar (always on the right) illustrates the performance of the single-particle tracking method, i.e., results achieved by crowd workers annotating only a single particle of their choice, taken from a particular, indicated area.
The NASA TLX index tests showed a significant decrease in mental and temporal demand, as well as a drop in job frustration for both conditions (dense and loose silo filling), for both proposed methods compared to the classical crowdsourcing method. The results were better (a larger drop) for the single-particle tracking method than for the zone-tracking targeted method. However, the decrease in physical demand was slightly more prominent for the single-particle method and was significant for both methods only in the case of the loose filling condition. A different effect can be observed in the category of effort. Effort was reported to decrease significantly, mainly for the single-particle method (a drop of more than 50%) compared to the classical baseline method. Performance went up significantly, by more than 10%, but only for the single-particle method. Interestingly, performance increased by more than 10% only for the single-particle method. Performance was not perceived to be significantly different for the zone-tracking method, no matter the condition, yet it was perceived to be worse than the classical method in the case of dense filling.