1. Introduction
Nowadays, teaching music at Primary School in Spain is a complicated task. The most relevant learning outcome pursued in the Music subject is to learn how to play the flute. In this sense, students do not receive proper support for self-learning while performing flute pieces at home with the current learning methodology. Usually, they end up frustrated with the music lessons.
Music performance involves a closer contact between students and the instruments to transmit feelings, emotions and ideas. This helps to develop the student’s creative capacity, as well as agility and psychomotor coordination.
Presently, learning flute is a serious inconvenience for students, due to the lack of a sequenced and systematic methodology to be followed up during group lessons at primary school. In fact, the adequate atmosphere to guarantee favorable associations between students and instruments, which was highlighted in the preceding paragraph, is not properly achieved in the classroom. This usually means that the majority of the group cannot even play the complete flute piece during class time; or that the teacher cannot focus on correcting common and particular errors identified for a specific flute piece.
These facts lead to a deficient acquisition of flute technique with consequences such as insecure students while playing the instrument at home with many difficulties, or students performing nervously in front of classmates and teachers in the classroom.
Our proposal is based on the creation of a methodology to learn flute, including the acquisition of general aspects of Musical Theory. As a result of the implementation of this methodology, the development of a mobile application (for smartphone or tablet) to support learning flute in a guided manner will become reality. Apart from providing customized help to the students, through metrics that evaluate their interpretations and enforce motivation and individualized attention, our aim is to provide feedback to the teachers so that they may follow the improvements in the interpretation of a piece and the effort invested by each student at home. This makes possible a detailed continuous evaluation that is incompatible with the traditional methodology, carried out with 45 min of weekly group lessons.
Additionally, the application will draw on different resources and musical concepts, such as Tablature, Articulation and Tempo, among others, which will be introduced in the proposed methodology (
Section 3.1) and which will help students during the learning process. Technologies to support these resources by the mobile application will be presented in
Section 3.2 and
Section 3.3. The most important are the MIDI standard protocol to manage symbolic musical notation closer to Musical Theory than digital audio representation in terms of sampling and byte quantization; MusicXML [
1] as a standard open format for exchanging digital sheet music using XML markup language; MusicXML rendering APIs to display verified MusicXML content in smartphone or tablet scalable displays and Pure Data, which is an open source visual programming environment to efficiently implement required algorithms, such as a pitch tracker to identify tones of monophonic flute melodies.
2. Related Works
There are many approaches related to the use of ICTs for learning music. In [
2], authors propose tutorials regarding aspects such as rhythm, notation, etc. However, this approach is no more than passive tutorials. In [
3], Orff’s methodology goes beyond the use of ICT tools in the classroom, promoting active student participation in music lessons. It is about following a pedagogical strategy for learning and teaching music in which vocal exercises, the use of simple musical instruments (e.g., flutes, drums, xylophones, etc.), combined with body percussion and body expression enable students to feel the music as a real living experience, instead of focusing only on Musical Theory. Despite ICT tools may fit perfectly to support Orff’ strategy in the classroom, they do not consider self-learning at home.
Reynolds [
4] investigates different ways of demonstrating student’s musical development by using ICT tools, according to the students’ ages. Particularly, software intended for data gathering and analysis is used to quantitatively monitor children’s approaches to music composition through advanced music software.
In the same vein, McDowall [
5] has collaborated with primary school teachers in designing music learning experiences, based on ICT tools. As part of her research, children have used a range of computer-based music technologies, with particular emphasis on applications that fostered musical creativity. Unlike [
4], qualitative analysis techniques are applied in this case, including interviews with teachers and children, direct observation, and analysis of children’s compositions. The research concludes that children have participated in relatively sophisticated creative music processes that would not have been possible using conventional music resources instead of computer-aided.
Regarding the study of a musical instrument, in [
6] authors propose the use of simple graphs that help students to understand and familiarize with a particular instrument. Other educational songbooks, focused on flute study [
7,
8,
9,
10], rely on methods based on performing small musical pieces that gradually increase the level of difficulty. None of these methods proposes an auditory or visual support to the students other than a pentagram. In [
11], authors develop a tool for supporting the flute study by means of a software application and sensors placed on the flute. This means that it is only possible to control plugging or releasing flute holes for the correct interpretation of each piece. This method is unable to consider all the sound characteristics, i.e., intonation or sound intensity. In [
12,
13] authors encourage us to embed technologies into musical instruments by means of the concept of
hyper-instrument as “musical instrument designed or adapted to be used with electronic sensors whose output controls the computerized generation or transformation of the sound.” This fact allows, by means of the development of ubiquitous technologies, to improve the interactions between students and musical instruments [
14,
15]. Finally, the
augmented flute concept is considered in some researches. In [
16], the computer has the role of a virtual extension of the flute, providing a real-time composition model. In it, the student acts as composer and performer. In [
17], a
virtually real flute is presented through the use of embedded sensors to control a real-time synthesis model that extend traditional flute sounds. This model provides parameters that can be measured while the instrument is being used, with the goal of helping the flautist while learning new playing techniques.
Traditional educational songbooks implicitly limit self-learning capacity in instrumental practice, partly due to limitations found in Musical Theory knowledge when facing pentagram notation. With the resources provided by ICTs and, particularly, by means of the Virtual Teacher Metaphor, the aim is to eliminate these limitations, providing more learning autonomy to the students at home. They can listen to a segment interpretation of the piece under study that is considered well performed, then, they can try to play it using the recording tool that will be integrated in the mobile application. At this point, they will receive automated feedback with mistakes made and points of improvements.
An important aspect in Musicology is the music segmentation. Through it, dividing a score into units such as sections or phrases, makes possible a simpler analysis that helps students to memorize pieces or understanding authors’ compositional strategies. In [
18] a segmentation model assisted by computer simulation is presented.
In addition, another relevant aspect arises, that is, the
Computational Modeling of Music Cognition studying the analogy between the human mind and computer programs [
19]. This concept can be covered by the Music Information Retrieval (MIR). This discipline is defined as a “research endeavor that strives to develop innovative, searching schemes, novel interfaces, and evolving networked delivery mechanisms in an effort to make the world’s vast store of music accessible to all” [
20]. In this context, depending on the application, there are some approaches in the literature. Through the use of start and end markers within the music segments some authors support music navigation, editing and synchronization [
21,
22,
23,
24]. Others develop applications for computer-assisted composition and improvisation [
25,
26,
27,
28]. In [
29] authors study patterns and metrics characterizing musical aspects such as pitch, timbre, and loudness in a specific musical corpus. Accordingly, these patterns can help us to analyze the flute performance and to offer students advices in order to improve their interpretations.
To our knowledge, there is no other proposal in the literature about developing a methodology and a digital mobile application to support it, both focused on learning flute and considering aspects such as the Virtual Teacher Metaphor and the Continuous Evaluation of students. In this sense, we have found only one contribution that uses melodic similarity measures for automatic scoring of singing voice [
30]. In this work, feedback is returned after the comparison between the user performance and a reference melody. To some extent, this can be considered a partial implementation of the Virtual Teacher Metaphor, since it provides mechanisms to find dissimilarities between interpretations and it explicitly guides the user through these mistakes.
3. A Proposal for Learning Flute
3.1. Educational Methodology
Our proposal for an educational methodology to support learning flute at Primary School revolves around three main cyclical stages: (1) Repertoire; (2) Virtual Teacher support; and (3) Continuous Evaluation.
Repertoire: A key point within our pedagogical strategy consists in determining some dynamic/active criteria to take into account during the selection of musical pieces for a particular group of students. These are as follow:
Musical Theory difficulty. Increasing the level of difficulty of the pieces selected for study in a cyclical manner. It is important to periodically evaluate the learning progression of the student group in terms of Musical Theory knowledge, adjusting the repertoire when necessary.
Technical Difficulty. It is crucial to adjust the repertoire and the following pieces under study based on the psycho-evolutionary development of the students. It also important to recommend playing flute daily [
31].
Segmentation. Dividing each piece into blocks/passages in an adequate manner proposed by the teacher. The divisions have to be appropriate to strengthen the performance step by step, taking difficulty into account (in both dimensions, Musical Theory and technical when playing the passage).
Contextualization. Choosing a repertoire closer to the students to awaken their interest and motivation.
Virtual Teacher support: This metaphor includes mechanisms to guide students during their self-learning process at home. In order to collect a representative set of resources in this context, a survey has been carried out to Primary School music teachers by means of the Google Forms platform. The survey helps us to define a taxonomy of elementary mechanisms to be included as part of the Virtual Teacher functionality. Two kinds of mechanisms have been categorized:
Preset visual guidelines. Some of the mechanisms serve as visual marks to try to enhance musical performance by displaying helpful and persuasive information at appropriate beats. Complementary visual marks may appear dynamically while the student is playing the piece. This kind of help is set up previously by the teacher, when he/she prepares the flute piece to be distributed to the students. The following mechanisms fall into this category:
- ○
Fingering: By showing schemes to reflect proper finger positions during note transitions.
- ○
Articulation: By displaying symbols on the musical score that indicate characteristics about transition or continuity on a single note or between multiple notes within a musical phrase.
- ○
Tablature: By providing an alternative diagram to the pentagram similar to the one used in guitar.
Mechanisms for identification of dissimilarities/mistakes. This category includes mechanisms to find dissimilarities between a reference performance of a flute piece (considered well played) and one or more consecutive passages that have just been played by the student. Dissimilarities can be directly mapped into performance mistakes of different types. This may provide automatic feedback so that the student can check what they did wrong. The following dissimilarities/mistakes were considered in the survey:
- ○
Tempo: It is important to have a mechanism to estimate the overall speed at which each student plays one or more consecutive passages. Suggestions about increasing/decreasing speed should be offered according to the recommended tempo in the original flute piece.
- ○
Duration mistakes: It refers to dissimilarities in the duration of singles music notes that even can break beat constraints. In such cases, apart from indicating wrong notes (duration), some reviews of Musical Theory are necessary. Additionally, the performing of a sequence of exercises for specific metrics may be demanded.
- ○
Articulation mistakes: It refers to identify dissimilarities in the performance that affect articulations within a musical phrase. The definition of articulation is provided in the previous Preset visual guidelines category.
- ○
Tone mistakes (Intonation): It makes reference to dissimilarities found in pitch estimation of musical notes in one or more consecutive passages. As feedback, tablature and pentagram notation diagrams with highlighted musical notes at their correct tone heights should be provided. Also, the possibility of listening to the corrected passage/s.
- ○
Intensity/volume mistakes: Supporting mechanisms to identify parts during one or more passages in the flute piece performance that need to be played at a higher/lower volume. As in the previous case, it should include the possibility of listening to the passage/s before and/or after volume correction.
Continuous Evaluation: Every individual interaction of the students with the Virtual Teacher is gathered. Students’ performances are recorded and temporarily signed while practicing at home. In addition, dissimilarities/mistakes found by the mechanisms proposed in the Virtual Teacher support are also linked to their performances. This information is grouped by student and sent to the teacher as a log. In this manner, weekly evolution of each person is available to the teacher in a transparent manner.
By understanding student individual efforts at home, the teacher can analyze their learning evolution and know about their particular problems for the music piece that is being studied. Moreover, statistics may be provided to the teacher to summarize the learning progression at both, individual and collective levels. On an individual level, the teacher can offer customized help messages to a particular student about how to deal with a specific mistake. This will maximize his/her practicing time and boost the learning process at home.
On a collective level, broadcast messages can be sent to the group to remind them there is little time left for the next lesson, recommending daily flute practice, among other kinds of group messages. Furthermore, through the use of the proposed methodology, the teacher has a broader perspective of the problems and mistakes that students have faced in the period between in-person lessons. Thus, the teacher knows in advance common and particular problems to be solved and he/she can better plan the upcoming lesson to be more effective in the classroom.
Lastly, continuous evaluation helps to assess the repertoire convenience. The teacher can find scheduled pieces that are difficult for the current students’ level and opt to change the piece for another that strengthens detected learning issues. More generally, continuous evaluation offers valuable knowledge for improving lessons, repertoire, effective tips and suggestions, in combination with other educational complements that are not available through the current traditional method.
Figure 1 illustrates an overview of the proposed methodology to support learning flute at Primary School.
3.2. Digital Tool to Support the Methodology
The proposed methodology in
Section 3.1 needs support from a digital tool to implement some of its functional requirements. Smartphones and tablets provide Natural User Interfaces (NUIs) with simple and intuitive gestures that fit well with the navigation actions required to move forward or backward through musical passages, pentagrams and tablatures. Likewise, multi-touch screens make it easier to start/stop performance recordings by tapping on a button with one finger; or to enable navigation through the mistakes identified by the Virtual Teacher during individual practice at home.
In addition, smartphone and tablet devices are affordable and nowadays, more and more commonly available in the classroom. Therefore, these reasons push us towards the implementation of a mobile client application to support methodology aspects, instead of a traditional desktop program.
Besides the mobile client application running on the students’ smartphone or tablet, a RESTful API to manage a set of essential web services hosted in the Cloud is required. Through API requests made by the mobile client application, relative services responses will be received to cover several of the methodology steps. Relevant functionalities of these services are listed in
Table 1, including the main methodology stages and the participant roles involved. Main methodology stages are ordered by level of engagement to accomplish the functionality. Parentheses indicate that the methodology stage may interfere or not in the current functionality.
As indicated before, the mobile client application, through its user interfaces, will make requests to the different services listed in
Table 1, depending on the related methodology stage.
Figure 2 illustrates the mockup design for the main user interface of the mobile client application. The main container is intended to host the music score of the proposed piece (one musical passage at a time); or the resulting music score of a student’s performance (after post-processing to identify the melody and visually mark mistakes on it). Pentagram or tablature notations can be chosen for representation. Musical passages have assumed to be conveniently sectioned by the teacher, according to the segmentation criteria established for that piece (
Section 3.1).
Below the main container, the MIDI playback and Record buttons are on the left part of the bottom container. Tablature/Pentagram, Fingering and Articulation show/hide buttons are displayed on the right part. Tempo selection control is also on the right. MIDI playback controls may work in three different operation modes: playing the entire piece, playing a selection of musical passages or even a selection of beats to be listened at the chosen Tempo.
Regarding the Tablature, if this type of notation is selected, the correct positions of the fingers on the flute can be shown on the screen. If Fingering show/hide button is enabled, simplified flute hole diagrams are displayed in particularly difficult transitions between musical notes (previously identified by the teacher). Also, when Articulation control is enabled the appropriate details on the interpretation corresponding to the Articulation of musical phrases are shown (e.g., staccato or accent symbols).
There are two gesture icons at the middle left and right sides of the main container that show two navigation drawers. The left navigation drawer will make possible to navigate and select one or multiple musical passages from the current flute piece under study. On the other hand, the right navigation drawer will be used to list types of mistakes found for the displayed musical passage.
We propose a simple and usable main user interface. Thus, and because the application will run on a smartphone or tablet, there are only eight user-friendly controls for listening, recording, enable/disable Tablature or Pentagram notation, Fingering, Articulation and for Tempo selection.
Moreover, the navigation across musical passages and type of mistakes/dissimilarities is handled by one or two-finger swiping gestures. At this point, it is important to highlight the limitations found due to screen dimensions. Only a few beats may be enclosed in a musical passage to fit in a smartphone screen. Regarding tablet devices, space constraints are less critical because the screen dimensions are more adequate for the flute pieces used in Primary School lessons.
3.3. Matching Dissimilarities/Mistakes between Musical Passages and Feedback Generation
One of the web services highlighted in
Table 1 is related to the upload of individual performance logs with identified dissimilarities to each student’s profile. In order to accomplish this service, we have to determine all the required steps in the processing pipeline that will contribute to generate the feedback related to the mistakes made by the student. In this respect, a matching algorithm that compares and finds dissimilarities between the score of a musical passage (reference) and the score generated from the passage recorded by the student is necessary.
Before introducing the model of the processing pipeline to match the dissimilarities and generate the adequate feedback, we distinguish two types of mistakes according to their scope within the musical passage (local or global).
Table 2 groups dissimilarities/mistakes depending on these types. The local ones are subcategorized as mistakes with a single note scope, beat scope, or musical phrasing scope whether they affect a complete musical phrasing. For its part, global mistakes impact the full musical passage.
Articulation errors may affect one or more beats that make up a musical phrase. Duration mistakes can break single beat constraints. While intonation and intensity/volume mistakes affect each musical note individually. On the other hand, tempo has impact on the full musical passage.
The diagram in
Figure 3 illustrates the processing pipeline to match dissimilarities and generate the adequate feedback. Every time the student plays a flute passage he/she records the performance through the record control in the mobile client application (process 1). Recording functionality will be accomplished by using Pure Data [
32] open source visual programming environment. A pitch tracker algorithm, particularly the Specially Normalized AutoCorrelation (SNAC) algorithm [
33], will be implemented as a routine in Pure Data to identify tone changes (process 2). More details about the SNAC algorithm are provided in
Section 4.
From the pitch tracker output (process 2), three different post-processing step will be applied according to the scope reached by each analyzed characteristic (
Table 2). The first post-processing step (post-processing step 3.1) will estimate the duration of the identified music notes and their intensity. The second post-processing step (post-processing step 3.2) will determine articulation hints, such as attacks and articulation ligatures, etc. The third post-processing step (post-processing step 3.3) will estimate tempo of the recorded passage. These three post-processing steps will be implemented as routines in Pure Data. The outcome of the post-processing will be a MIDI [
34] standard file which is a synthesized version of the student’s recording.
Process 4 of the processing pipeline will consist in transforming the generated MIDI file into MusicXML markup language. This will represent the music score of the sudent’s recording. Having both MusicXML files (MusicXML of the reference passage & MusicXML of the sudent’s recording), the matching algorithm (process 5) may be executed to identify dissimilarities in intonation, intensity, duration, articulation and tempo.
Finally, detected mistakes from process 5 will be packaged in a feedback log for the student’s profile (process 6). Then, the resultant MusicXML of the sudent’s recording will be rendered on the smartphone or tablet screen with the corresponding marked mistakes (process 7). Feedback logs will be used to generated automatic summaries for the teacher at individual and collective levels (process 8).
4. Specially Normalized AutoCorrelation algorithm (SNAC)
A monophonic pitch tracking algorithm is needed to identify tones in flute melodies. As illustrated in
Figure 3, pitch tracking is the first required step towards dissimilarity matching between flute performances. An error-free pitch tracking capable of analyzing a flute signal stream live is a complicated task. In this sense, processing in frequency domain of signal chunks may work to identify their fundamental frequency, however, this kind of analysis does not provide proper pitch estimations in zero-latency applications that require very small signal chunks. In addition, the spectrum will show a lot of peaks and dips in the presence of pink or Brownian noises which are quite common in acoustic and analog electronic sources. In this sense, confidence/fidelity indicators are required and must be implemented additionally to decide whether a pitch report from spectral analysis is accepted as a periodic sound with an estimated pitch or discarded as an aperiodic mostly noisy signal.
Alternatively, pitch can be tracked in time domain, as a function of period length. The method involves finding a repetitive pattern in the waveform. Autocorrelation is a popular transformation of a periodic waveform that stands out its pattern repetition, making the period length better recognizable as pronounced peaks within the autocorrelation signal (autocorrelation coefficients). Specifically, the separation between the two most prominent peaks in the autocorrelation signal marks the period of the original signal.
In the autocorrelation method, all samples in the signal chunk under study are multiplied with samples of time-shifted versions of the same signal chunk and summed to compose the autocorrelation coefficients. Equation (1) shows the mathematical formula of a short-time discrete autocorrelation.
is the signal chunk and
is the autocorrelation function.
Figure 4 graphically illustrates the time shifting of the signal chunk. When periods in the
signal coincide with periods in the shifted version (
), autocorrelation coefficients have the highest value.
The main problem of the autocorrelation method can be inferred from
Figure 5. While increasing shift, the number of samples used to compute the autocorrelation decreases and it influences the real correlation value between the shifted waveforms. This can be directly observed in the computed autocorrelation signal, marked in red. Any short-time autocorrelation is affected by this effect, known as bias, producing a progressive reduction in the amplitude of the autocorrelation coefficients which may lead in some cases to errors in identifying the autocorrelation peaks that mark the signal period.
In order to correct the bias effect and raise the correlation coefficients to their proper values regardless their position, the autocorrelation function is usually ‘
unbiased’ dividing it by a dynamic factor that counteracts the reduction in the amplitude due to the decrease in the number of samples used to compute the autocorrelation. McLeod [
33] introduces the Specially Normalized AutoCorrelation (SNAC) which creates a dynamic factor based on the actual signal values. This divisor factor is inversely proportional to the bias reduction mitigating its effect. As a result, autocorrelation maxima for a periodic signal will steadily keep their position, no matter the shift.
The mathematical formulation of the normalization factor is shown in Equation (2).
In addition, this normalization encloses the amplitude of the autocorrelation coefficients in the −1 and 1 range. Perfect periodic signal chunks have correlation maxima at value 1, quasi-periodic signal chunks have lower maxima, i.e., in the presence of noise or if the fundamental frequency suddenly changes within the chunk. Therefore, correlation maxima can be used as a confidence or fidelity index of the period estimation quality.
According to the biased short-time discrete autocorrelation formula in Equation (1) and the normalization factor in Equation (2), the Specially Normalized AutoCorrelation (SNAC) can be calculated as:
where
is the biased short-time discrete autocorrelation and
the normalization factor.
The evaluation of SNAC presented in McLeod’s thesis [
33] is promising. The results obtained indicate much better pitch estimation accuracy than regular unbiased autocorrelation, especially when the signal chunk encloses only few periods for analysis. This encourages us to use it for the pitch tracking stage in the processing pipeline used to match dissimilarities between flute performances (
Figure 3).
5. Conclusions
This paper is a work in progress in which a new methodology for learning flute at Primary School is proposed. The methodology describes three main stages complemented by a digital tool (mobile client application & RESTful API). By using the mobile client application, students will practice at home, playing selected flute pieces from the repertoire and receiving adequate automatic feedback from the Virtual Teacher. Furthermore, the teacher will send individual and collective messages, when necessary, to guide the learning process throughout the week.
From a technical point of view, using Pure Data in the mobile client application will accomplish recordings of flute performances. The transformation of the recorded real instrument melody into a synthesized MIDI version will be performed through a pitch tracker algorithm, also implemented in Pure Data. The pitch tracker will identify tones of monophonic flute melodies from each recording. Post-processing steps will be required to determine music note durations, intensity and articulations for the estimated tones. The resultant synthesized MIDI will be used for playback. Both, the students and the teacher will be able to listen the MIDIs from the different performances of the flute piece and compare them with the one considered as the reference.
Generated MIDI files will be exported to the MusicXML markup language with a twofold aim. First, running a matching algorithm to find dissimilarities (mistakes) between one or more musical passages of the original score and the MusicXML score generated from the student’s performance. Secondly, rendering the MusicXML score from the student’s performance on the smartphone/tablet screen. A rendering API, which accepts MusicXML format, will be used to display the score and visually mark the mistakes made.
Thanks to the digital support, the teacher will have access to individual evaluations, becoming aware of individualized and collective errors. This will have a positive impact on weekly lessons. The teacher can design more effective and smarter classes because of the previous knowledge of the learning progress, for instance, grouping students with common errors, promoting alternative exercises for Musical Theory reinforcement, modifying selected repertoire dynamically according to the group level for the upcoming lessons, etc.
In essence, the continuous evaluation promoted by the proposed methodology and the active support provided during the practice at home will improve the class evolution. These aspects are the most important contribution of this proposal because there are no approaches combining these faculties in the literature of Music Education.