Development, Implementation, and Assessment of a Creative Additive Manufacturing Design Assignment: Interpreting Improvements in Student Performance

Abstract

The purpose of this article is to present a design for additive manufacturing assignment focused on creativity rather than functionality and to analyze its results (N = 70) acquired during five years. The assignment teaches the unique advantages of additive manufacturing to engineering students and encourages learning from failure to achieve designs that are possible to manufacture. The students of the course assignment were in their fourth year of studies and pursued master’s degrees in mechanical engineering. The article presents the design for additive manufacturing course assignment in enough detail for it to be applied by educators in the sphere of additive manufacturing. The result assessment is performed with a numerical method and a jury method. The statistical significance of the correlation of the numerical approach with the jury approach is evaluated. The study conducts a multi-point creativity assessment on a large sample of parts created by students acquired over five years with the support of 10 jury members. This assessment process gives insight on how creativity in design for additive manufacturing can be quantified and can be readily applied by educators. The data of the jury evaluation are verified with an interrater reliability evaluation. Our results indicate that conducting the course assignment for multiple years increases the quality of the student work. The improvement of the results is theorized to be partly due to students gaining inspiration from an increasing number of high-quality parts from previous years of the assignment. The numerical method of result assessment can be used for evaluation when resources are scarce; however, the jury method should be used if possible.

1. Introduction

The past decade has seen rapid development of additive manufacturing (AM) technologies, with the costs of AM equipment decreasing radically [1]. Due to the increased spread and awareness of AM, its unique ability to produce complicated parts and artistic objects has drawn great interest from the public [2,3,4]. It is now common for universities, libraries, and schools of all levels to own AM machines.

The origins of AM are in rapid prototyping and it was later popularized as a tool for artists and tinkerers to create unique objects. These days AM has an increasing role in the production of components that are installed in advanced applications, such as airplanes and high-performance vehicles [5,6]. The incentive to use AM in engineering applications is in the ability to create components with much more intricate shapes than previously possible with conventional manufacturing technologies. This situation has created a need for engineers that understand how to leverage the unique design benefits of AM while conforming to its limitations [7,8,9].

Creating a workforce capable of designing for AM requires incorporation of AM teaching at universities, universities of applied science, and other institutions responsible for training mechanical engineers. This poses a challenge because the foundation of mechanical engineering education has long been in designing for traditional manufacturing processes, which have a very different set of design requirements than AM [10,11,12]. There is, therefore, an immediate need for guiding principles in setting up education in design for AM (DfAM) [13,14,15,16].

Another challenge in teaching DfAM is that it is a very recent concept and it is therefore relatively poorly known outside of the active participants in the scientific community and designers of high-performance machinery. University teachers and lecturers not previously familiar with DfAM can, therefore, have difficulties understanding what exactly they must teach.

The impetus to create better AM education can be seen in how heavily it is publicized. There are currently different practices for teaching DfAM to different interest groups depending on their level of knowledge of computer-aided design (CAD) and the level of AM machinery to which they have access [17]. Universities are at the forefront of organizing AM education and several articles have been published describing how AM courses of various formats have been set up. Most presented courses use problem-based learning and are focused on creating and manufacturing new designs as solutions to functional problems [18,19,20,21], while some courses extend the practice to assembling or creating AM machines [22,23]. AM is also used as a tool to teach other subjects in engineering education, which helps with the level of familiarity towards the technology among students [24,25,26,27]. In industry, AM service providers and machine manufacturers offer workshops and webinars for client companies to improve the competence of their research and development units [28,29]. Some universities advocate courses organized between universities and industrial companies as the best method to organize AM education [30].

Outside of the engineering domain, there is growing public interest in AM even though the applications may be less professional. High schools and public libraries often host makerspaces and FabLabs that organize short introductory courses in the concepts of AM and how to use it on a basic level [31,32,33,34,35].

To better communicate the nature of DfAM, Laverne et al. proposed a naming scheme of opportunistic and restrictive DfAM. Opportunistic DfAM exists to instruct designers to create components that leverage the design freedom of AM. Restrictive DfAM guides users to implement the new limitations of AM, such as minimum wall thickness and minimum feature size so that the component can be reliably manufactured [36].

Research has been carried out on which aspects of DfAM are most important to teach to students. Prabhu et al. (2018) conducted an experiment in which three groups of students were taught different aspects of DfAM before being asked to complete a design task. The first group was taught only restrictive aspects of DfAM, the second group restrictive and opportunistic aspects, and the third group was not taught DfAM. The findings were that students are commonly exposed to the restrictive side of DfAM and grasp it easily, while design opportunities of DfAM are more obscure and require more teaching [9,37]. These findings are in line with those of Bornasal et al. (2018), who reported that engineers tend to identify constraints before applying concepts [38].

The practices of teaching DfAM are numerous but due to the relatively short period of practice, there has not been much research concentrating on the cumulative effects of offering DfAM education [39]. To fill this gap in literature, this article aims to provide a longitudinal reference by presenting the assignment part of an annual course given to fourth-year mechanical engineering students between 2014 and 2018. The educational background of the students and the amount of CAD and manufacturing technology education given to them during their studies did not change during the years. The students were introduced to Siemens Solid Edge during their first year of studies and PTC Creo during their third and fourth years. Therefore, the starting level of the students regarding DfAM was similar in all years of the course. The course consists of a lecture series to teach students the theories and applications of AM, as well as a course-long assignment, in which the students learn to design for AM and to use entry-level machinery.

The course assignment addresses the mechanical engineering program goals of preparing the students for changing practices that come with emerging technologies. It forces mechanical engineering students to experiment with innovative geometries and teaches them to make them compliant with AM. The course is particularly focused on challenging the preconceptions that mechanical engineering students have regarding manufacturability of objects, as they are taught to design components for conventional manufacturing for three years before taking this course. The aim of the assignment is to encourage students to experiment with complex structures and use their imaginations without being restricted by functionality, with the expectation that they will use the learned skills in consequent projects in problem-based product design courses and their careers as engineers. The assignment is intended to supplement rather than replace problem-based learning assignments focusing on functionality, as those have been verified as indispensable methods to teach mechanical engineering students to design viable products.

The course assignment differs from the problem-based learning assignments presented in previous literature by focusing on creativity instead of functionality. In the assignment, function of the designed part does not play a role and students are encouraged to design their parts with a large variety of beneficial aspects of AM. The students are asked to iterate and learn from their failures to deliver a part that demonstrates benefits of AM while still possible to produce with AM machines.

As the assignment has proven to be very popular among the mechanical engineering students, it is presented in enough detail that it can be replicated by other institutions. Apart from demonstrating a way of organizing DfAM teaching, the study of this article aims to investigate how to quantify and judge the results of the assignment and to observe whether there are incremental improvements between the results from year to year. Unlike assignments with clearly quantifiable goals, such as the one presented by Stern et al. (2019), the assignment of this course is open-ended and its results difficult to quantify [20]. The research questions the article aims to answer are, therefore:

• RQ1: How can the student performance of a creative DfAM assignment be reliably evaluated?
• RQ2: Can incremental improvement in the results between years be detected and how can it be explained?

Structure of the Article

This article first presents a hands-on course assignment in which students are asked to design and manufacture complex AM components. Two methods of evaluation are then presented. The first is a fast method based on the complexity of the existing data of the 3D models of the parts. Three definitions of complexity are used for this method and their merits and disadvantages are discussed. The second method of evaluating the results is based on jury assessment, which is more time-consuming but gives a holistic appraisal of the work of the students. After presenting the assignment and methods used in its assessment, the article presents the results, evaluates the usability of the assessment methods, and discusses the trends in the results of the assignment.

2. Materials and Methods

The course in question aimed to teach engineering students DfAM and to integrate AM in their studies. It consists of a series of lectures and a practical assignment teaching DfAM and machine handling. In the first lecture, the assignment brief was delivered and student work from previous years was presented along with general examples of features that are only possible to manufacture with AM, such as lattice structures, internal channels, and complex surfaces. The rest of the lectures focused on the technical details of machines, materials, and applications of each of the AM technology categories defined by the ISO/ASTM 52900 standard.

During the assignment, students learned to design parts for AM in such a way that leverages the design advantages of AM. The parts were manufactured using low-cost, user-friendly AM machines. Although low-cost machines are not entirely representative of industrial-grade machinery that the students will likely use in their future work, the fundamental ideas behind DfAM for both types of machinery are the same. The machines available for the students to use were Bits from Bytes 3000 in 2014 and Ultimaker 2 from 2015. Both machines are based on the material extrusion method, in which plastic filament is melted and deposited in the x-y-direction on a table that moves in the z-direction [40]. In addition to being relatively cheap, AM machines at this price point are uncomplicated to operate and their software is simple to learn. Although low-cost material extrusion machines broadly represent industrial 3-axis machines, such as metal laser powder bed fusion, they cannot be used to effectively teach designing for machines that have more axes, such as robotic arms with directed energy deposition equipment.

The course assignment was given relatively unchanged to the students between 2014 and 2018. The most significant difference was the increase in the popularity of the course from 2016 to 2017 due to positive word-of-mouth among the students. Although there were more students during the latter years, their educational background remained unchanged.

2.1. Assignment Design

The assignment for the AM course consisted of students in groups of three designing and producing a part that demonstrates unique advantages of AM. The assignment was carried out in groups for two reasons. First, teaching groups requires considerably less machinery, materials, and course staff resources. Second, working in groups leads to more creative solutions to problems, which is the desired goal of the assignment [41,42].

A lecture with the instructions for the assignment was given at the beginning of the course, during which selected sample parts from previous students were presented to the students of the current year. A description of what should be taken into consideration while designing the part was given to the students prior to the assignment. The designed parts were to demonstrate the complexity made possible with AM and should leverage some of the following features: Lattice structures, conformal channels, moving parts, and topology optimization. The students were told that diversity of AM benefits used in the design and the quality of the final part are the main criteria upon which the assignment is graded.

The functionality of the designed part was not considered in this task since the focus was purely on teaching new opportunities and limitations of AM. As such, no interfaces, load conditions, or other mechanical requirements were given. Support structures and post-processing of the parts were not allowed in order to keep the focus on design for AM and to remove any individual manual skills of students from the submitted results. A second reason for the limitation was to teach the students to minimize the amount of support structures and post-processing through creative design solutions. The size of the parts was limited to 6 × 6 × 10 cm to keep the build times reasonable. Together, these limitations created a controlled environment that eliminated factors influencing the result, such as the craftmanship of individuals, although it limited the freedom of design and set a ceiling of the complexity of the geometry of components.

During the weeks after the first lecture, the students were required to take part in an AM machine use workshop at the laboratory to get them acquainted with the machines and software workflows they would use to manufacture their parts. The number of students per workshop was limited to four per machine to make sure everyone would have an opportunity to operate the machine. For flexibility reasons, the students were not required to attend the workshop in the groups formed for the assignment. After attending the workshop, the student groups were free to use any CAD suite to design their part.

Before starting production, each group submitted their final design for feedback to the course assistant. This process provided suggestions for modification in case the design clearly faced critical challenges in production or violated the provided design rules. After acceptance, students were free to manufacture the component with the supplied AM machine. As the number of groups was high and the number of machines limited, the students were required to reserve slots from the calendar and observe the entire manufacturing process in case of error to not waste any machine time. The students operated the machines entirely independently and had to set the parameters, such as speed, fill rate, and layer thickness, by themselves.

Finally, the student groups submitted a report detailing the design considerations, the success of the manufacturing process, as well as comparing the final parts to the nominal design. The parts and reports were then graded by the course assistant and the responsible professor at the end of each course. The process of the assignment is shown as a flowchart in Figure 1.

As there were no complications in the organization of the assignment and because the students rated the assignment very highly, the assignment was not modified after its inception. Altogether, 70 parts were designed and manufactured by student groups for the course between 2014 and 2018. A selection of the parts of differing quality of design and execution can be seen in Figure 2.

2.2. Assessment of the Assignment

The parts were graded following the same rules at the end of the course each year by the course assistant and the professor in charge. However, they were subjected to a new assessment for this study to eliminate the potential annual variation in the subjective judgment and to provide comparability among the results through the years. The retrospective assessment of the assignment focused on the demonstration of the unique advantages of AM and innovation of ideas, while also considering the execution of the process and future potential of the demonstrated ideas. The quality of the group assignments was quantified and then compared between each year of the course.

The assessment was done with two methods. In the first method, a numerical value was acquired for each part based on a set of definitions of complexity. This type of objective quick grading of creative tasks is very time efficient and thus valuable if proven to work [43]. The second method was based on a commonly used method in creativity measurement, in which external judges are used to evaluate the results [44,45]. In this approach, the physical parts manufactured by the student groups were randomly organized and graded based on design, execution, and potential by a jury of 10 experts in the field of AM.

To quantify the quality of the assignment results from an objective point of view, the criteria needed to be established against which the results were to be measured. Because the main advantage of DfAM is producing complex designs that cannot be manufactured with other methods, the complexity of the design was taken as a criterion to use to evaluate the results. However, a straightforward definition of complexity is not provided in DfAM literature and therefore needed to be defined prior to being used as a criterion.

2.3. Complexity Assessment

The most commonly cited definition of complexity in AM given by Rosen et al. (2007) states that the three levels of complexity are shape complexity (possibility to build any shape), material complexity (multiple materials can be mixed), and hierarchical complexity (a part can have microscopic and macroscopic features) [46]. Later, this framework was augmented with functional complexity where functional parts could be produced directly from an AM machine [47]. The course assignment focused on shape complexity because the students were limited to a single material and entry-level AM machines with limited accuracy.

To define shape complexity as a calculable quantity, the total surface area of a part is used as the simplest definition of complexity. To use a more articulate method, a definition based on the Shape Complexity Index proposed by Pradel et al. is used [48]. The Shape Complexity Index is defined as the ratio between the component surface area and the volume of the design space of the part, which is the volume of the part, which can be modified without affecting the functionality of the part. However, this approach of measuring complexity is aimed at the redesign of components rather than the design of new components and must be adapted [49]. As the design space for the assignment given to the students encompasses the entire bounding box of the part, the second definition was chosen to be the ratio of the surface area to the bounding box volume.

A weakness of using the bounding box as a definition of complexity is that it is exaggerated for parts that protrude slightly in one direction. Because most of the parts created by the students were very complex, this was a common occurrence. Another quantification of complexity is the ratio of surface area to the volume of the part. While this does not account for the maximum extent of parts, it solves the issue of exaggerated bounding box volumes.

In summary, the following definitions of quantified complexity were used in this study: The surface area, the ratio of the surface area to the bounding box, and the ratio of the surface area to the part volume. The complexity was calculated for the 70 parts by using CAD software to obtain the surface area, volume, and bounding boxes for each part.

2.4. Jury Assessment

The strengths of the numerical complexity assessment are its speed and its objectivity as the results are not affected by the persons doing the assessment. However, complexity assessment is quite superficial on its own as it can potentially omit important factors that may influence the quality of the final part. Therefore, a second part of the assessment, the jury assessment, was devised. In the jury assessment, all parts were laid out in a randomized order and jurors were asked to evaluate each of them based on three criteria. The jury was not made aware of the separate complexity assessment to not introduce any bias. The 10 members of the jury were selected from among university researchers who had many years of experience in designing components for AM. On average, the jury members had five years of experience in AM research and all held at least a master’s level degree in mechanical engineering. The jury members were presented with a randomized set of the 70 parts so that the year of manufacturing could not be discerned from their order. The randomization was done with the Microsoft Excel randomize function and the parts were set on a large table according to their given number. The arrangement of the parts is shown in Figure 3.

The jury members were asked to individually grade each part. To facilitate the grading process, a three-factor multi-point creativity assessment (MPCA) method was employed. MPCA has been previously shown to be an appropriate tool for evaluating learning outcomes [50]. The three factors of the MPCA were chosen to be design, execution, and potential. The jurors were given a form with an 18-point Likert scale for each factor of each part. An adjective corresponding to the extreme ends of the scale was placed on each side of the scale. In total, each juror evaluated 210 aspects. To ensure that the results would be comparable, the jurors were instructed using the same guidelines on the three factors. The factors, their adjectives, and elaborations are provided in

Table 1. The factors of the multi-point creativity assessment (MPCA), the adjectives of the Likert-scale questionnaire, and the elaborations of the factors given to the jurors.

Factor	Adjectives	Elaboration
Design	Surprising vs. Expected	Is the part designed for conventional manufacturing methods or does it demonstrate new capabilities of AM?
Execution	Well-made vs. Crude	Are there any defects and failures? (e.g., stringing or oozing, layer shifting, layer delamination, curling)
Potential	Promising vs. Valueless	Does the part show promise in potential industrial applications?

.

There was no time limitation for completing the survey assessment; however, most jurors finished it within 40 min. The results of each jury member were transferred from paper to Microsoft Excel for analysis.

2.5. Jury Assessment Data Reliability

To verify the value of the data acquired with the jury method and to establish the approach as a reliable method, the data needed to be subjected to reliability analysis. A common way to calculate reliability is to implement one of the various inter-rater agreement analysis methods [51]. In this study, IBM SPSS Statistics 25 software was used to calculate the inter-rater agreement using the Intraclass Correlation Coefficient (ICC). The two-way mixed model was chosen for the analysis and the type of analysis was chosen to be consistency because the interest was in whether the scores of all jurors had a linear relationship rather than the absolute value of the parts. The confidence interval of 95% was also obtained for each result.

2.6. Aggregation of the Data

The values for the design, execution, and potential factors of the 10 jurors were used to calculate an average for each factor for each submitted part. Once all factor values were calculated, their average was used to calculate the general score for each part. The values of each factor and general score for each part were aggregated by year of submission for comparison.

2.7. Correlation and Statistical Significance between Complexity Assessment and Jury Assessment

The correlation between the two assessment methods was analyzed to understand which, if any, complexity analysis approach is a sufficiently accurate method to measure the quality of the parts made by students. The correlation and statistical significance analysis were conducted on the three criteria of the jury assessment separately. As with inter-rater reliability, the software used to calculate the statistical significance was IBM SPSS Statistics 25.

3. Results

This section presents the results of each of the assessment methods. The methods are subsequently compared and the statistical significance between them is analyzed.

3.1. Results of the Complexity Assessment

The average values of the complexity assessments for parts produced in each year are presented in

Table 2. Average complexity factors of the parts according to the year they were submitted.

Year	Area (cm2)	Area/Volume (cm−1)	Area/Bounding Box (cm−1)
2014	207.2	6.2	0.9
2015	373.1	6.1	1.7
2016	465.6	7.8	1.9
2017	371.2	9.7	2.3
2018	294.9	9.8	1.8

. The three approaches were normalized between 0 and 1 for clearer visual presentation between each year. The normalized average grade of the complexity of the parts based on the surface area, the surface to bounding box ratio, and surface to volume ratio is shown in Figure 4.

As can be seen from normalized complexity scores, the three definitions of complexity were not completely dependent on each other and warrant individual examination in comparison to the jury assessment.

3.2. Results of the Jury Assessment

The results of the jury assessment organized by year are presented in Figure 5.

The interrater reliability for the jury assessment is presented in

Table 3. Interrater reliability for jury assessment.

Factor	Intraclass Correlation	95% Confidence Interval, Lower	95% Confidence Interval, Upper	Data Reliability
Design	0.829	0.763	0.883	Good
Execution	0.812	0.739	0.872	Moderate to good
Potential	0.825	0.756	0.880	Good

.

When interpreting the reliability of intraclass correlation, correlation values of less than 0.5 indicate poor reliability, values between 0.5 and 0.75 indicate moderate reliability, values over 0.75 indicate good reliability, and values over 0.9 indicate excellent reliability. It is recommended that the reliability interpretation be done to the range between the lower and upper 95% confidence levels rather than the true ICC [52].

The lower 95% confidence levels of all three factors were very close to the threshold of 0.75, with design and potential factors being just over and the execution factor being just under. The upper 95% confidence level of the factors was above the 0.75 threshold and close to the 0.9 threshold of excellent reliability. Therefore, even though the reliability of the Execution factor must be reported as moderate to good, the overall reliability of the data was comfortably on the higher end of the spectrum of good reliability. This means that the jury assessment approach was consistent and can be reliably used as a method for evaluating the results of the assignment.

3.3. Correlation between the Complexity Assessment and the Jury Assessment

The results of the statistical significance analysis between the complexity assessment and the jury assessment were carried out. The Pearson Correlation and the two-tailed statistical significance were obtained for each of the correlation combinations in

Table 4. Statistical significance between complexity assessment and jury assessment.

Complexity Approach	Measure	Design	Execution	Potential
Area	Pearson correlation	0.201	0.250	0.197
	Significance	0.095	0.037	0.102
Area/Volume ratio	Pearson correlation	0.263	0.006	0.357
	Significance	0.028	0.958	0.002
Area/Bounding box ratio	Pearson correlation	0.196	0.108	0.319
	Significance	0.104	0.372	0.007

. The alpha used in the analysis is 0.05.

Because the alpha used in the analysis was 0.05, values under 0.05 indicate that there is statistical significance between the complexity approach and the jury score factor. According to the analysis, the correlation between area as the definition of complexity with the execution factor was statistically significant while there was no significant correlation between the other factors. This is the opposite of area per volume ratio, whose correlations with the design and potential factors were significant, while the correlation with the execution factor was very low. The only statistically significant correlation of area per bounding box ratio was with the potential factor.

4. Discussion

The results of this study suggest that there is a trend that the quality of the students’ parts improved year after year during 2014–2018. This indicates that the students’ average learning outcome of this course has been continuously improving even though the curriculum and course content have remained the same.

It should be noted that while the design and potential factors of the jury assessment show signs of continuous improvement, the level of the execution factor has remained on a similar level for the last four years of the assignment. The lower score of the execution in the first year of the assignment can be largely attributed to the use of inferior machinery. The implication is that DfAM knowledge, and not the mastership of machines, has improved among the students with each year.

As the effect of improving student work was noticed in 2018, there is no direct evidence from earlier years and all possible explanations for the effect are hypothetical to a degree. To understand the reasons and factors behind the improvement from 2014 to 2018, the influencing factors must be considered individually. As the jury and complexity assessments were devised after all the parts of the study were submitted, the students could not be aware of the criteria of these evaluations at the time of designing the parts and could therefore not optimize their designs for a higher score.

A major annually changing factor in the five years of teaching is the growing selection of student-made high-quality parts shown during the assignment instruction session. It is easy to understand that the more and better student-made parts were demonstrated to the new participants, the higher the ambition of the students became. This phenomenon was also noted by Watchke et al. (2017), who concluded that having access to physical AM models supports inspirational ideation [16].

The inspiration from previous years can be seen in echoed design choices and trends through the years. The features were imitated, evolved, and combined to create more complex designs. To demonstrate the effect of previous designs inspiring following ones, and to illustrate the spectrum of quality of the designs, Figure 6 presents five parts from each year of the assignment. The complexity and jury scores rise with each year for the presented results. While the improvement in design quality in the first years are clearly visible, it is much more difficult to see in the student designs of the latter years, which is in line with the jury assessment.

The students of the first year of the assignment did not have examples from previous years to influence them. As such, there were no clear design trends among the first-year results and all designs were unique. Although some of the designs were impressive, most were quite simple. Rudimentary lattice structures and simple internal channels were the most prominent features in 2014. The stand-out designs with most interesting features in the first year were cylindrical, which influenced the works in all subsequent years.

The designs of 2015 took on board features seen in the previous year but improved upon them and introduced some new features. All designs in 2015 were cylindrical. The designs included slightly more complex lattice structures, much more advanced internal channels, and the first designs with moving parts were created. The designs also had more complex surfaces than in the previous year. In the first two years of the assignment, most designs relied on a single AM feature and did not combine them.

The results of 2016 improved slightly on the previous year with more advanced shape complexity, but the major improvement was combining several AM features in one design. This can be seen in hollow interlocking parts with internal channels and lattice structures that are used in combination with complex internal channel systems. The first non-cylindrical parts since 2014 appeared.

In 2017, most designs combined several features of AM. Lattices became more complex and designs with moving parts became more intricate. The results in 2018 were again improved from the previous year with more complex and better integrated feature combinations.

Although some designs are clearly inspired by specific earlier designs, the students innovated enough on the designs to not simply copy the designs of previous years but make features more complex and include their own ideas. Each year also introduced new ideas, which were necessary to avoid stagnation of the designs.

While having examples from previous years is valuable and can help students to explore complexity of AM in more depth, it may also have the effect of guiding students onto specific paths, unless they are actively encouraged to break the design trends. As an example of breaking the trend in a positive way, in 2018, a student group realized for the first time that the largest length of the bounding box did not necessarily have to be the height. Combined with the fact that the part did not have to be cylindrical, they thus created a component with linear motion, which had not been seen before in the assignment.

In addition to the performance improvements on the group level, the amount of silent knowledge in the community has tangibly increased. The act of performing the assignment annually has fostered a healthy base of knowledge in the university laboratory. Because students are free to use the laboratory after the assignment, it can be theorized that being part of a supportive community influences the students to create better parts. Additionally, the increase of scores can be partially attributed to a general increase of recognition of AM in the general society and in media. The increased grouping of scores in the later years can be attributed to the increased number of groups and therefore parts. The increased number of groups offsets the effect of poor and excellent performers, which may be the reason behind the reduced variety in scores.

The improvement in student performance could be hypothesized to be linked to the improvement of teaching staff. However, this is unlikely because the feedback given to the students was purposefully minimal and staff only intervened when the designs were clearly problematic or outside of the assignment brief. The feedback collected from students each year after the course also suggests that the source of the improvement of student work is not in the delivery of the assignment. The students graded the teaching methods of the course on a scale from 1 to 5, and the grades (average of each year between 3.30 and 3.87) show no clear trend. In the written part of the feedback, the students emphasized that it was very hard to design parts without any help and that they learned most from their own failures. This reinforces the fact that course staff did not have a big influence on the improvement of performance.

The jury assessment appears to be a statistically very reliable method of evaluating the design, execution, and potential factors of an open-ended assignment in which students are asked to design and manufacture complex AM components without a set functional requirement. It can, therefore, be used by the evaluators if they need reliable results and can dedicate the time required. The applications of the systematic jury assessment are not limited to the course assignment presented in this article. It can be used to assess any student project that has a qualitative component that cannot be directly measured and compared to a numerical ideal. However, the jury assessment criteria need to be individually adjusted to each different project where jury assessment is used.

The complexity assessment using area per volume ratio was shown to have a statistically significant correlation with the design and potential factors of the jury assessment. The area per volume ratio can, therefore, be used as a decent indicator of quality by the evaluators or the students. It should, however, be noted that the execution factor cannot be predicted with this approach and the jury assessment provides a more accurate and holistic evaluation. It is possible that the area per volume ratio corresponds closely to the jury assessment because this definition of complexity is intuitively perceived by humans on a psycho-physiological level. However, further experiments are necessary to confirm this claim.

The area per bounding box ratio is similar to the area per volume ratio in terms of correlation with the jury assessment, with slightly lower statistical significance. While it may have applications outside of education, it is not recommended to use the area to bounding box ratio to evaluate the results of student assignments.

Simple surface area of 3D models as the definition of complexity does not correlate strongly with the jury assessment. Nevertheless, the statistical significance of its correlation to the execution factor is an interesting finding and bears consideration. The correlation indicates that the higher the surface area of a part, the higher its execution factor is judged. A likely explanation behind this phenomenon is that parts with large, simple features leading to large areas are easier to manufacture and are therefore more likely to be free of defects, earning higher scores. While this leads to parts that are well-produced, it is at the cost of the design aspect and should not be encouraged in a course that aims to teach innovative design. The use of surface area as the definition of complexity is therefore not recommended for assignment evaluation.

Other measures of using 3D data of the designs could conceivably be used as definitions of complexity in future studies. For example, the area of layers and their deviation throughout the part could be used as a measure, the average gradient of the surface of the part could be used to represent curviness of the object, and the total length of contour lines of layers of the part could be considered in lieu of total surface.

The fact that the course assignment does not account for the functionality of the parts neither in assignment description nor in evaluation is a known weakness from the point of view of teaching the students practical engineering skills. However, focusing the task away from functionality and solely towards shape complexity is done on purpose as it has been identified that teaching opportunistic AM design is a particularly challenging facet of DfAM education [37].

Although appropriate support structure generation and post-processing are vital aspects of many AM technologies, and are taught separately for each AM technology type in the lectures of the course, their implementation is prohibited in the assignment. The students are not allowed to use automatically generated support structures or post-process their parts with tools. The purpose of the restriction is to teach students to optimize their designs to minimize supports and post-processing steps. However, if the goal of the assignment were to teach support generation, the assignment brief could be adjusted to allow external supports and removing them after manufacturing, while still prohibiting internal supports.

The study of the article evaluated the learning of students in an indirect way by examining the quality of the handed in assignments. Although this is an adequately precise method to evaluate the learning performance of entire classes of students when their starting level is similar, it does not provide sufficient information to evaluate the learning outcomes of groups of individuals. Therefore, more effort will be put into the evaluation of the starting level of individuals and their learning outcomes in the future iterations of this study.

A major limitation of the jury assessment was that it was a long process and jurors experienced fatigue towards the end, which possibly influenced their scores. However, since the parts were in a randomized order, this affects the scores of all years equally. There is also the risk that the jurors understood the descriptions of the factors of the assessment differently. However, the data reliability analysis shows that the data are reliable. Nevertheless, because the scores of the design and potential factors of the jury assessment were very close to each other, another factor could be used to replace the potential factor in the rubric, or it could be removed entirely, in which case the weighting of the factors should be re-evaluated. The factors were originally weighed equally, but as the focus of the assignment is primarily on the design of parts, the execution factor should be kept at a smaller importance than design. The weighing ratio could therefore be 2/1 in favor of design if only two factors are used.

The study observed that the scores of the parts produced by students increased annually but the rate of improvement is decreasing. This might signify a possible ceiling for the quality of the designs. The assignment will be kept under evaluation throughout the following years to see if there is further improvement in the parts created by students. As the course assignment is foreseen to continue in the following years, a follow-up jury assessment should be conducted after the results become available to see if the increase in quality is indeed decelerating.

5. Conclusions

This article reported the execution and evaluation activities of a well-received university course assignment teaching DfAM to mechanical engineering students. The data presented in the article are extensive with 70 parts produced by students over five years. The first research question that this study set out to answer was how creative DfAM assignments without functional requirements can be properly evaluated. The second research question that the study aimed to answer was whether there has been improvement between the years and, if there is, how it could be explained.

The study answers the first research question by presenting two methods of assessing the parts created by students: The objective evaluation of the complexity and the subjective jury assessment. The result of the correlation analysis between the complexity and jury assessments shows that while the former is a decent tool to get an approximate score for the design of parts, the latter should be used when a more holistic evaluation is needed.

The second question is answered by the results of the jury assessment showing a significant improvement in scores over the years. The cause behind the improvement was examined in the discussion of this article by considering the changes that have happened during the years the assignment has been conducted. In summary, the most influential changes have been the fostering of an active experimenting community at the university laboratory, a greater general presence of AM in media, and the annually growing physical part library providing the students with a better baseline from which to build.

The DfAM course assignment has been a great success in communicating skills and ideas that will prove to be useful to the students in their work careers. We urge any school, university, or the educational arm of a company to adapt the course assignment presented in this article to their needs. While the assessment of the results of the assignments is not paramount from the learning perspective of the students, it can also be used to measure if the fostering of a knowledge base is occurring and is therefore highly recommended.