Next Article in Journal
Exploring Knowledge of the Concept of Physical Literacy among Rehabilitation Professionals, Students and Coaches Practicing in a Pediatric Setting
Previous Article in Journal
Exploring the Quality of Life of People with Incomplete Spinal Cord Injury Who Can Ambulate
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Disabilities
Volume 3
Issue 4
10.3390/disabilities3040030
disabilities-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Assessing the Tool-Use Learning Process in Persons with Profound Intellectual and Multiple Disabilities

by
Christina Modh
1,
Ann-Marie Öhrvall
2 and
Lisbeth Nilsson
3,*
1
Habilitation Resource Center, Region Stockholm, 113 61 Stockholm, Sweden
2
Department of Neurobiology, Care Sciences and Society, Karolinska Institutet, 171 77 Stockholm, Sweden
3
Department of Health Sciences, Lund University, P.O. Box 117, 221 00 Lund, Sweden
*
Author to whom correspondence should be addressed.
Disabilities 2023, 3(4), 477-492; https://doi.org/10.3390/disabilities3040030
Submission received: 2 August 2023 / Revised: 12 October 2023 / Accepted: 16 October 2023 / Published: 18 October 2023
Browse Figure
Versions Notes

Abstract

:
The aim of the study was to explore inter-rater reliability and rater experiences of applying the universal instrument Assessment of Learning Process (ALP) with persons with profound intellectual and multiple disabilities. The study used a mixed-methods design. Inter-rater reliability was tested by eight clinical raters who assessed 23 video clips of people learning to use a powered AKKA mobility platform with a line-follower system. The raters were four occupational therapists, two special education teachers, and two speech-language therapists. The qualitative Think-aloud method explored three of the raters experiences of applying the ALP instrument. The inter-rater reliability test of the universal ALP instrument showed a moderate linear weighted kappa value (0.45). All eight raters’ estimated the degree of difficulty or confidence for each assessment. Content analysis of Think-aloud data elucidated three of the raters experiences of observing performance characteristics and interpreting tool-use understanding. The reported inter-rater reliability of the universal ALP instrument can be considered acceptable for an observational study involving persons with profound intellectual and multiple disabilities. Thereby, the results suggest that the ALP can be applied to this population. This is important as assessing a person’s phase in the learning process guides the choice of ALP-facilitating strategies supporting progress in tool-use learning, which in turn may expand a person’s activity repertoire and participation.

Graphical Abstract

1. Introduction

Children and adults who have profound intellectual and multiple disabilities (PIMD) are dependent on pervasive support from others to be active and are often viewed as lacking agency and being marginalized in society [1]. They form a heterogeneous group in which each individual has a unique combination of physical, cognitive, visual, auditory, and speech/language impairments and a variety of medical diagnoses caused by congenital factors, trauma, or illness [2]. Each individual’s unique and complex disabilities typically involve profound limitations of communication and a need for support and specially adapted equipment to do activities. The multiplicity of their impairments negatively reinforces the total impact of the disability, implying that the accumulated effects of the impairments are more limiting in learning and performing activities than the effect of a single impairment. Their complex combinations of disabilities and reliance on pervasive support profoundly limit their opportunities to give voice to their needs, desires, and abilities, to explore and develop preferences for activities, and to participate in the world on their own terms of abilities [1,3,4]. The limited capabilities for independence and autonomy make it hard for people with profound intellectual and multiple disabilities to achieve meaningful participation [5]. By developing reciprocal relationships through dialogical interaction and learning from and with persons with profound intellectual and multiple disabilities, we may enable their development, identity formation, and the gaining of some agency in preferred activities [1].
Enabling activity and participation is empowering by reducing occupational marginalization and increasing occupational justice, as involvement in situations providing opportunities to be active can cultivate and reinforce self-determination and the development of preferences and desires [6,7,8]. To enable participation, we need experience, knowledge, and a variety of tools to provide opportunities for active engagement in situations the individual indicates a preference to participate in [5,9]. To be able to assess abilities and tailor interventions for each individual, we need instruments developed for children and adults with profound intellectual disabilities, but few assessments are standardized and developed to specifically meet their needs [10]. We also need to learn how to promote activities that instill meaning for the individual with profound disabilities, to use technologies supporting the development of potential capacity, and to adapt activities and environments [3,4,9]. An activity with the power to promote meaningful participation and active engagement is self-produced mobility in smart wheelchairs [11,12]. There are few descriptions of smart devices wherein electronic technologies are combined in a mobility platform, allowing the individual to stay seated on their own device while exploring powered mobility [13,14]. An interview study explored the benefits of using the AKKA mobility platform [13], and a case study explored one participant’s outcome of using a prototype of another power wheelchair trainer platform [14].

1.1. Using the AKKA for Self-Produced Power Mobility

The AKKA mobility platform (the AKKA) allows persons with profound intellectual disabilities to safely experience and explore self-produced mobility indoors [13]. This activity may enable their development of preferences, understanding of causal relationships, a sense of self, agency, and control, which opens up opportunities for choice-making and tool-use learning [4,15]. The AKKA is a powered 4-wheeled platform with low height used either for free driving or with a line-follower system, activated by a joystick or micro-switch [13,16]. The child or adult sits on their own device, which is strapped onto the top of the platform. When used with a line-follower system, a camera connected to the electronics, mounted underneath the platform, makes the platform follow a line of black tape on the floor. Thereby, the system enables safe, self-produced, powered mobility for people unable to control steering. One or more switches can be used to activate the system. The line shape can be laid out as a round loop, a straight path with turn-loops at each end, or as crossroads. Speed is set by an assisting person. Based on the number and set-up of micro-switches, the person can navigate in the environment along the line, forward or backward, in loops or at crossroads and learn to make decisions on where to stop. To our knowledge, very few clinical or educational contexts offer this safe and simplified powered mobility intervention. No original studies of the outcome of children or adults with disabilities using the AKKA were found.

1.2. Assessing the Learning Process

The universal Assessment of Learning Process (ALP) [17] is a refinement of the ALP tool, version 2.0, developed in the field of powered mobility intervention for children and adults with profound intellectual disabilities [15,17,18,19]. The ALP instrument is used to assess phases in the learning process based on observing characteristics in occupational performance that indicate the individual’s tool-use understanding, independent of which task the person performs. The development of the original instrument was based on observations of 126 individuals of different ages with different degrees of ability performing joystick-use in a powered wheelchair. In the analysis, characteristics emerged in tool-use performance, indicating each of the eight phases of learning [18]. The refinement was driven by the need to have one ALP tool applicable to any kind of tool-use learning. The original structure and text remain the same in the universal ALP instrument and ALP-facilitating strategies, and refinement only involved omitting and replacing text and indicators associated with powered mobility use. Where the text was changed, the behavioral indicators were replaced with similar but abstract concepts applicable for assessing and facilitating any tool-use learning process. The English original of the universal ALP tool was translated into Swedish for implementation in this study [17].
The ALP tool is process-based and includes the ALP instrument and the ALP-facilitating strategies [20,21]. The instrument and the facilitating strategies are used in a continuous circular process with the purpose of assessing and supporting tool-use learning. The ALP instrument illustrates the eight phases of the learning process with performance indicators in five observational categories: attention; activity and movement; understanding of tool use; expressions and emotions; interaction and communication [17]. Moreover, the eight phases, from Novice (phase 1) to Expert (phase 8), are divided into three stages of exploration: explore function (phases 1–3), sequencing (phases 4–5), and performance (phases 6–8). The ALP-facilitating strategies provide guidance for a facilitating approach and specific strategies for each stage and phase in the learning process. The focus is to understand a person’s actual phase of learning to enable matching tool-use understanding with a challenge that is just right to facilitate learning in a motivating situation tailored for the person [20,22]. The assessment with the ALP instrument is carried out in a tool-use situation arranged to fit each individual’s specific needs, desires, and existing abilities. Then the person is invited to take part in the tool-use situation with the specific tools, and dialogue is used to encourage exploration and initiative to act and interact. Each individual is allowed to explore using their abilities in their own way and at their own pace.
Studies of the inter-rater reliability of the original eight-phase instrument [18] and the ALP instrument, version 2.0 [15], both showed a very good degree of agreement with a linear weighted kappa value of 0.85 [23,24]. Professionals and researchers, in various powered mobility contexts internationally, have applied the ALP tool, version 2.0 [25,26,27,28,29,30], recommended it in guidelines [31], and also tested the ALP instruments’ inter-rater reliability [32]. Even though the structure and organization of the universal ALP instrument are identical with version 2.0, some concepts related to powered-mobility use were omitted or replaced by corresponding universal behavior indicators, which requires new tests of the instrument’s inter-rater reliability.

1.3. AKKA Intervention and Applying the Universal ALP Tool

At a county-wide Habilitation Resource Centre in Sweden, persons with profound intellectual and multiple disabilities are offered intervention in the AKKA with a line-follower system activated with one micro-switch. Activation of the micro-switch controls driving: on/go by doing a press; drive by keeping on pressing; and off/stop by releasing pressure. The choice of micro-switch is matched to individual abilities, and a switch can be big or small; mounted on a tray or a universal mounting system and pressed by the hand; or placed in the palm with a strap or mounted on a neoprene glove and pressed by the fingers against the palm, a hard surface, such as a tray, or onto a part of the other hand or arm.
Used with one micro-switch, the AKKA with a line-follower system can be used from a very simple to an advanced level of tool use. A simple level of tool use is achieved when the person understands that a press on the switch activates the system and sets the platform in motion. Following a line with turn-loops makes it possible to get back to the original spot by only forward driving. More advanced tool use is achieved when driving, and navigation is controlled in relation to engagement in other activity opportunities along the line in the setting. Negotiation and choice of direction at crossroads can be enabled in agreement with a supporting staff concealing lines leading in undesired directions.
Goals for the intervention are to enable participation and learning through the empowering experience of exploring self-produced power mobility in a varied physical and social indoor environment. Documentation, evaluation, and planning of the intervention involve journal notes and video recordings. Each individual is at the beginning of intervention and, whenever necessary, provided with the most suitable micro-switch for their needs with regard to type, placement, and functionality [33]. Individuals who explore how to use the AKKA go through a tool-use learning process [20]. Knowing their actual phase and stage in the process is important to be able to provide tailored facilitating strategies supporting their individual development of tool-use understanding.
Increased experience and knowledge of options to promote meaningful participation may help professionals overcome barriers to engaging people with profound intellectual and multiple disabilities in using assistive technologies such as the AKKA [34]. This is the first study involving an assessment of the learning process in persons with PIMD practicing the use of the AKKA. Therefore, it is important to establish if the universal ALP instrument also has inter-rater reliability and can be implemented to support professionals and staff in assessing an individual learner’s actual phase of tool-use understanding in the context of the AKKA intervention.
The aim was to test the inter-rater reliability of the universal ALP instrument when applied to persons with profound intellectual disabilities practicing self-produced powered mobility using one micro-switch to activate an AKKA mobility platform with a line-follower system and to explore clinical raters’ experiences of using the instrument.

2. Materials and Methods

This study used a mixed-methods design to explore the applicability of the universal instrument Assessment of Learning Process (ALP) with persons with profound intellectual and multiple disabilities taking part in AKKA intervention. A mixed-methods design was found appropriate as the ALP instrument is process-based and very different in nature compared to conventional task-based assessments. It is an assessment for learning, as the assessed phase of learning guides the choice of strategies facilitating progress in tool-use learning from the individual’s actual phase of tool-use understanding. As the population under study is considered challenging to do research with, it was also important to explore raters experiences of doing the assessment.
The quantitative method was used to test inter-rater rater reliability. Clinical raters applied the ALP instrument with video clips illustrating the eight phases in the learning process, selected from existing clinical documentation of the AKKA intervention. Psychometric analysis was used to compare the degree of agreement between clinical raters’ assessments of the observed phase in the learning process and a reference standard.
The qualitative method was used to explore raters’ experiences of applying the instrument. Latent content analysis of Think-aloud data captured qualitative aspects of raters’ experiences of observing performance and assessing phases of learning in video clips showing persons with profound intellectual disabilities using the AKKA.

2.1. Ethical Considerations

Relatives or legal guardians of persons participating in video clips from the AKKA intervention gave signed informed consent to use existing clinical video-graphic documentation in this study. The study of persons with profound intellectual disabilities practicing self-produced powered mobility was approved by the Regional Ethical Review Board in Umeå, Sweden, on four occasions (Ref no: 94–229, §311; Ref no: 97–101, §144; Ref no: 98–335, §422; Ref no: 03–030, §163). The clinical raters who applied the universal ALP instrument received information about the study orally and in writing and signed informed consent before inclusion in the study.

2.2. Participants

Eight clinical raters were recruited from a county-wide Habilitation Resource Center in Sweden. Inclusion criteria were: occupational therapist, special education teacher, or speech-language pathologist with experience working with persons with profound intellectual disabilities and experience using micro-switch interventions with this population.

2.3. Material and Procedure

2.3.1. Video-Graphic Documentation of the AKKA Intervention

The Habilitation Resource Center used video for clinical documentation of people who took part in the AKKA intervention. The intervention is part of the unit’s service for persons with profound intellectual and multiple disabilities, and video recording is part of the ordinary documentation of this intervention. The persons performing AKKA-use in the video-graphic documentation had unique combinations of disabilities that impacted their communication and performance of activities, as described in the introduction. Each person used one individually adapted micro-switch to activate the AKKA, and depending on their unique abilities, the type, size, and placement of the switch varied, which influenced how they performed AKKA use.
Video clips were extracted from this existing source of videographic data. Selection focused on finding three clips illustrating performance characteristics for each of the eight phases in the learning process. Three clips were motivated by having a similar number of clips for each phase of learning and by providing clips showing how individuals performance could vary but still have the characteristics indicating each of the eight phases in the instrument. The indicators are short abstract descriptions of characteristics of behaviors that can be recognized despite of age and combination of disabilities [17]. The selection of video clips required knowledge and experience of the population, of applying the AKKA intervention, of the individuals in the clinical documentation, and of applying the ALP instrument. The first author (C.M.) and third author (L.N.) planned the process of extracting relevant video clips for the study. C.M. had 23 years of experience with the AKKA intervention, personal knowledge of the population, and long experience using the ALP instrument. L.N. is one of the developers and an expert on the ALP instrument, with extensive experience in applying the instrument to the population. C.M. carried out the purposive sampling by assessing numerous video sequences with the universal ALP instrument to find and extract clips that solely represented each of the eight phases in the learning process and provided an as clear as possible view of activation of the micro-switch. The sampling resulted in 23 video clips, with lengths spanning from 27 s to 5 min and 21 s. In each video clip, one person with profound intellectual and multiple disabilities used an individually adapted micro-switch to activate an AKKA. The clips were supplemented with rater instructions and an assessment form.

2.3.2. Development and Application of Expert Reference Standard Assessment

C.M. and L.N., with extensive experience applying the ALP instrument, used a multi-step process to develop an expert reference standard for the assessment of the selected video clips. First C.M. made an independent assessment of the sampled 23 video clips, following the written instructions for raters used in previous studies of the instrument’s inter-rater reliability [23,24]. Then L.N. made an independent assessment of the video clips, following the same instructions. The subsequent comparison of their assessments showed some divergence in degree of agreement for a few of the 23 video clips. This was solved when C.M. and L.N. reviewed the videos together; they shared, compared, and reasoned on minor details in observed characteristics of performance which indicated phases of learning and finally reached full consensus on the reference standard for assessment of all video clips. The degree of agreement between the clinical rater’s assessments was examined by comparisons with the reference standard.

2.3.3. ALP Instructional Workshop

Prior to the raters’ assessment of the video clips, L.N. gave a 4.5-h workshop on how to apply the ALP instrument. A theoretical part elaborated on the tool-use learning process and its structure and concepts. A practical interactive part involved viewing video clips of other tool-use situations (self-feeding; using micro-switches to activate toys or communication devices). First, each rater independently used the ALP instrument to assess a video clip, and then they elaborated on their choices of stage of exploration and phase of tool-use understanding. The group discussions focused on details in tool-use performance and how observations could be aligned to the instrument’s structure and concepts, indicating which was a person’s phase of tool-use understanding.

2.4. Data Collection

One to two weeks after the workshop, the eight clinical raters individually completed their assessments of the 23 video clips. All raters received the video clips on a memory stick, the instruction and assessment form, and the full-length and short versions of the universal ALP instrument [17]. In the form, they noted how many times they watched each video clip before assessing the learning phase, how difficult it was for them to determine the actual phase, and how confident they were that other raters made an equivalent assessment. After rating all video clips, raters made free text commentaries on their experiences using the ALP instrument.

Think-Aloud Method

Think-aloud captures the concurrent verbalization of thoughts while performing a task [35,36]. Data was collected by audio recording from three of the eight raters, one from each profession. A rater should think aloud while assessing the phases of learning in four of the 23 video clips. The first author (C.M.), at individual sessions, audio-recorded each rater’s reasoning when they assessed the actual phase in video clips 6–9. Before recording, the rater became acquainted with the situation by assessing video clips 1–5. C.M. had minimal interaction with the raters while recording, so as not to disturb their expression of thoughts regarding the assessment of learning. If a rater was silent for too long (about 45 s) C.M. said “What are you thinking now”. The recorded audio data was transcribed verbatim.

2.5. Data Analysis

Analyze-it (Analyze-it Software, version 5.40.2, Standard Edition, Ltd., Leeds, UK) was used for the statistical analysis of inter-rater reliability, comparing the eight raters’ assessments of the ALP phase in the 23 videos with the reference standard developed by experts in applying the ALP instrument. Non-parametric statistics were used because the assessment of phases 1–8 with the ALP instrument provides categorical ordinal data. A linear weighted kappa coefficient was chosen to calculate the degree of agreement between raters’ assessments of phases of learning and the expert reference standard, as it provides a correction for the mean distance between ratings expected by chance [37] (p. 402). The kappa value was interpreted based on Landis and Koch [38], where <0 is described as poor, 0.00–0.20 light, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and 0.80–1.00 almost perfect. Analyze-it was used to make the inter-rater reliability results comparable with the results of previous studies of the original instrument [23] and the ALP instrument version 2.0 [24] using the same software and coefficient. The calculation of the linear weighted kappa coefficient was also motivated by the fact that a rater disagreement over one phase of learning has less weight than two or more phases. This means that although the assessment is not exact (one phase lower or higher), it still provides approximate guidance on which ALP-facilitating strategy should be chosen to support further tool use learning.
Descriptive statistics were used to create a diagram showing the contingency between the eight raters’ total number of assessments and the reference standard. The contingency of frequencies and relationships among assessments was calculated to illustrate total agreement and degree of disagreement, as disagreement in one ALP phase has less weight than that in two phases.
The transcribed Think-aloud data was analyzed using latent content analysis. The first author (C.M.) and second author (A.-M.Ö.) separately and iteratively familiarized themselves with the data and extracted meaning units relevant for the raters’ experiences of using the universal ALP instrument [39]. They individually extracted meaning units, and a comparison showed high consistency in selection. Together, they condensed units into codes. Then C.M. independently sorted the codes into categories and a theme. All three authors, in collaboration, made a re-analysis, leading to the refinement of categories and a theme.

3. Results

The mixed method provided a deeper understanding of the applicability of the universal Assessment of Learning Process. Inter-rater reliability was tested using the kappa statistic, and rater experiences were explored by the Think-aloud methodology. Eight clinical raters used the instrument to observe and assess tool-use performance in video clips of persons with profound intellectual and multiple disabilities practicing to use one micro-switch to activate an AKKA mobility platform with a line-follower system. The eight clinical raters were all female. Their professions, experiences with the population, and interventions before the study started are outlined in Table 1.
The raters assessed 23 purposively sampled video clips, illustrating all eight phases of the learning process. The video clips showed 13 people, aged from 15 to 54 years, who used one micro-switch to activate an AKKA mobility platform with a line-follower system. Micro-switch type and placement varied depending on individual needs, abilities, and combinations of disabilities. Data, including the number and length of each video clip, the type and placement of the micro-switch, the reference standard assessment of the ALP phase, and raters’ experiences regarding the difficulty of assessing the phase and low confidence that their assessment was in agreement with other raters, were juxtaposed in Table 2.

3.1. Inter-Rater Reliability of the Universal ALP Instrument

The test of the ALP instrument’s inter-rater reliability, comparing the clinical rater assessments with the expert reference standard assessment, showed a moderate degree of agreement with a linear weighted kappa value of 0.45 (Table 3). A minor variation in the weighted kappa value was observed when the raters’ degree of agreement with the reference standard was compared between groups of professions. The assessments made by the three occupational therapists who were experienced in applying the AKKA intervention and had knowledge of the ALP instrument gave the highest degree of agreement with a weighted kappa value of 0.50 (moderate). A lower degree of agreement (fair) was found for the two speech-language pathologists (0.41) and the two special education teachers (0.40), with a slightly wider range at the 95% confidence interval.
The eight raters altogether did 184 assessments of the 23 video clips, and 58 of these were in agreement with the reference standard assessment, giving a total agreement of 32%. Furthermore, the disagreement was one phase in 56 of the assessments (30%), two phases in 42 (23%), and three or more phases in 28 (15%) of the assessments. From the 126 assessments showing disagreements, 84 were underestimations and 42 were overestimations. The eight raters’ assessments of ALP phase were compared to the reference standard assessment. The distribution of frequencies for all raters’ assessments (184) for each of the eight ALP phases (1–8) was related to the reference standard assessments. The contingency diagram clearly shows that total agreement was more frequent in the earlier ALP phases (1–3) and that the raters had a strong tendency to assess a lower ALP phase compared to the reference standard (Table 4).

3.2. Rater Experiences of Assessing Phases in the Learning Process

3.2.1. Experiences Conveyed in the Assessment Form

All eight raters, for each video clip, estimated difficulty and confidence in doing the assessment (Table 2) and the number of viewings. On average, a video clip was viewed twice before assessing phase of learning. Finally, they summarized their total experience applying the ALP instrument. Many raters emphasized the need for better knowledge of the person to be able to assess his or her expressions for communication and interaction. They also expressed the need to be able to discuss their interpretations with others.
“Exciting to see all the videos, but you get tired because I feel very insecure about the instrument. Misses the chance to talk to others. Gets tired, I’m on my own, and am not accustomed to using the instrument. Difficult to assess interaction and communication when you do not know the person.”
There was good agreement between the eight raters’ experienced difficulty in assessing phases of learning and their experienced confidence that another rater judged the video clip equally (Table 2). No obvious patterns of relationships were recognized among factors experienced as influencing the assessment.

3.2.2. Think-Aloud Data from Three Raters

Think-aloud data was collected with three of the eight raters, one from each profession, all with extensive experience of the population and using micro-switches in intervention. The analysis of their reasoning while doing the assessment elucidated the complexity inherent in assessing phase of learning in persons with profound intellectual and multiple disabilities. The presentation of themes and categories is illustrated with citations of the raters’ reasoning, coded as R1, R2, and R3.

Interpreting and Understanding Tool-Use

The overarching theme of interpreting tool-use understanding involved the raters’ interpretation of each individual’s personal performance of AKKA-use and their interpretation of the concepts and structure of the ALP instrument describing the learning process. The raters repeatedly compared concepts and indicators describing the eight phases of the learning process with their personal observations of tool use in the video clips. They described how a person activated the micro-switch to set the AKKA in motion, reasoned about the person’s tool-use performance, and their tool-use understanding in the situation. The raters expressed varying degrees of difficulty in accurately assessing a person’s phase in the learning process.
“Here is a person that is both observant of the environment and interacts a lot with those around. Yes, this is tricky, we’ll see here, between phase 5 and 6. She has trouble with her body, so it is difficult for her to control the movement but you see when she wants to and that she tries. I think she can use the tool as it is meant to but often does so without motor precision. Yes, I would say phase 6.”
(R3)

Personal Performance of AKKA-Use

The raters reasoned on a person’s performance while exploring how to activate the individually adapted micro-switch to set the AKKA in motion. The raters focused on interpreting a person’s body control, intentional action, interaction and communication, and micro-switch use. The raters found it hard to explicate if a person’s repertoire of body movements was intentional, involuntary, or performed at random. They reasoned about their observations of personal performance and various movement patterns used to activate the micro-switch. They expressed how movement patterns could indicate intentional activation or deactivation of the micro-switch by pressing, holding on to, or releasing it. Patterns could vary from pressing and keeping pressing the micro-switch for a long while, just to drive, to doing short press-release sequences on the micro-switch repeatedly, using patterns hard to interpret as intentional or random.
“Tricky with those who do a lot of moving, voluntary or involuntary. Just as tricky as those who have a very slow movement pattern. How do you assess one long activation of the micro-switch in one minute?”
(R2)
When the raters made the interpretation that the person had body control and made intentional movements, they also interpreted the person’s different body movements and facial expressions as non-verbal communication and a way to convey something. One rater tried to interpret and understand if the person’s actions were indicating intention, and she felt insecure if she interpreted correctly.
“I think that this is a rather inexperienced person, but then if it is, what it depends on, because she sits there wiggling and fixing, and that I don’t know exactly what it stands for. If it is that she does not understand or that she wants something I don’t understand.”
(R3)
The raters observations of interaction in the video clips involved both interaction with the physical and social environment. Sometimes they found it difficult to interpret if acts were conscious or unaware when the person directed their attention towards the surroundings; if it was a sign of forgetting about the micro-switch and its function; or if the person was well aware that they could stop to look, do non-verbal communication, or make intentional movements towards something nearby in the environment.
“This person is focused on the surrounding environment oh, which you can interpret as both inattentive to the micro-switch or he has come so far that you do not have to focus on the micro-switch when he looks at other things around him.”
(R2)
The raters expressed uncertainty when they could not see how the person used the micro-switch, as when it was placed in the palm of the hand. They also expressed a need for longer video sequences to get more information to assess the person’s activation of the micro-switch when they pressed, kept pressing, and released it.
“This one was difficult; I think this one is difficult because she has the micro-switch in her hand you cannot really see what is happening.”
(R3)

ALP-Instrument Learning Process

The raters expressed how they compared their interpretation of the personal performance of AKKA use in the video clips to how they interpreted the description of the learning process in the ALP instrument. The ALP instrument is structured into phases and stages of the learning process. Each phase has performance indicators divided into five categories of observation, and there are many concepts to be interpreted. The raters expressed difficulties in keeping track of all parts of the instrument, which led to low confidence in their assessment. They experienced the assessment as a translation of personal tool-use performance into the instrument’s eight phases of learning. This translation required interpretation and understanding of the instrument’s structure and the concepts identifying and indicating each of the eight phases in the learning process.
“My experience of my own estimation was that it was difficult to know if I was judging “right”. There were many different fields of information to be weighted into the assessments and it was hard to keep track of.”
(R1)
Some concepts and indicators in the ALP instrument were described by the raters as more difficult to understand, such as the meaning of the text: openness to multi-level interactions.
“I think you see that she interacts with the environment and interacts with what is around her. And she presses the micro-switch adequately so I would like to choose phase 7 interacts on several levels. I choose phase 8. I was wondering if it would be a little lower but I interpret it that way.”
(R1)

4. Discussion

The interpretation of these mixed methods’ findings is that the process-based instrument universal Assessment of Learning Process is applicable for assessing tool-use understanding in persons with profound intellectual and multiple disabilities who learn to use the AKKA mobility platform with a line-follower system. The inter-rater reliability test gave a moderate degree of agreement between the eight raters and the expert reference standard assessment. The raters’ experienced the assessment of an individual’s phase of learning as multifaceted. Assessment involved observation of personal performance of micro-switch activation and AKKA use and relating these observations to indicators of the eight phases in the learning process in the ALP instrument.

4.1. Inter-Rater Reliability of the ALP Instrument

The moderate linear weighted kappa value (0.45) was considered relatively good because the assessment involved interpretations of multifaceted personal performance, variations in micro-switch type and placement, and indicators of phases in the learning process. Maes et al. [40] (p. 5) denote that the reported inter-rater reliability in observational studies of persons with profound intellectual and multiple disabilities is often lower compared to studies involving other groups. The raters 184 assessments were distributed across all eight phases of the learning process, thereby demonstrating the ALP instrument’s capacity to differentiate between phases of tool-use understanding in this population. The raters highest degree of agreement with the reference standard was found in the first three phases of the learning process (explore function). This suggests that the ALP instrument can sufficiently capture subtlety in personal performance, especially in the earliest phases of learning [40]. It was noticed that the raters had a strong tendency to underestimate the ALP phase compared to the reference standard assessment.
The two previous studies that tested the inter-rater reliability of the original instrument, “growing consciousness of joystick use” [23], and the ALP instrument, version 2.0 [24], both showed a very good linear weighted kappa value of 0.85. In previous studies, all participants used a powered wheelchair with a joystick center-mounted in a tray, meaning the device and how it was activated were constant.
In the current study, all participants had profound intellectual and multiple disabilities. Due to their impairments, each person used individual modes and patterns of performance to activate one individually adapted micro-switch, which resulted in huge variability in type, placement, and activation of the device. Similar circumstances influencing comparisons of outcomes were also found in other studies engaging people with profound intellectual and multiple disabilities in micro-switch-based activities [41,42]. This variability may have influenced the eight raters’ experiences of difficulty and confidence in assessing the phase of tool-use learning [22,34]. Moreover, the raters possibly had varied experiences and knowledge of what could be accounted for as advanced use of the AKKA line-follower system.

4.2. Raters’ Experiences of Using the ALP Instrument

All eight raters estimated their experiences of difficulty and confidence for each assessment with the ALP instrument (184). The Think-aloud data from three of the raters expanded their understanding of what influenced their assessment. The content analysis illuminated the multifaceted interpreting tool-use understanding, where the personal performance of AKKA-use was related to the ALP instrument and the learning process.
The three raters reasoned about how and whether cognitive problems, difficulties in voluntary body control, limited or no speech, or visual or hearing impairments affected the person’s tool-use performance. The raters reflected on whether the persons’ body movements, facial expressions, glances, and sounds were non-verbal ways of expressing themselves and/or involuntary movements due to their complex disabilities. They reflected on whether tool use was intentional or not; whether a person had stopped to take a break; or whether the person had progressed to the phase in the learning process where he or she could simultaneously navigate in relation to the environment and focus on other events in the surrounding environment. They also described interpreting the person’s communication and interaction with the environment in a video clip as a challenging part of the assessment.
Persons with profound intellectual and multiple disabilities have limited or no ability to communicate verbally. It is a complex endeavor to understand how they perceive the world and to support activity on their own terms of engagement [43,44,45]. Interpreting a person’s tool use and interaction with the environment is about observing, recognizing, and understanding the person’s performance in relation to the environment [22,46]. It is important to have knowledge of the person’s mode of using a tool and to stay open to variation in intentional use. Jonasson [13] described a person who used a micro-switch to activate the AKKA. The person could engage in “non-verbal actions” by moving independently in the environment between ’movement activity places’ and ’passive places of stillness’ that were meaningful when repeated. A “passive still” may be intentional rather than indicating a lack of awareness of the micro-switch, which may be easy to assume without knowing the person and the surrounding environment. We who support their activity have to rely on observing personal performance, and our interpretations have to be open and versatile depending on the situation [44,47,48] (p. 56). It requires an increased effort from the social environment to interpret personal performance and then respond and provide a challenge that is right for the specific situation [22,43].

4.3. Implications for Application of the ALP Instrument

Using the ALP instrument to assess a person’s phase in the learning process guides the choice of facilitating strategies that match actual tool-use understanding. A good match can provide a challenge that is just right—not too low or too high [20,49]. It is essential that we who support tool-use learning recognize and learn about each individual’s abilities, strengths, and potential for participation and social inclusion [5]. This knowledge is paramount to maintaining our efforts to engage each individual in an adapted or assisted activity of their own preference [5,8]. It is crucial to search for situations and activities that provide opportunities, motivating the person to take their own initiative and make their own choices, as this can increase alertness and facilitate active behaviors and actions, increasing their repertoire of activities [4,50,51].
Assessing tool-use understanding in people with profound intellectual and multiple disabilities is multifaceted, and numerous interpretations have to be made in the moment [4]. The ALP instrument has a process-based nature, meaning assessment is based on observing patterns of characteristic indicators in the personal performance of tool use. The assessment of learning phase and the use of facilitating strategies are applied in a continuous, circular process to support learning. In this ongoing process, the impact of assessing one phase lower or higher has less weight than a disagreement of two or more phases, which in our study means that 62% of the assessments could be useful to guide the choice of facilitating strategies for learning. Each individual has a unique way to perform tool use, and performance may vary depending on factors such as age, abilities, situation, motives, and choices. This makes process-based assessment more challenging compared to task-based assessments [20]. Therefore, an instructional course could support becoming a confident user of the ALP instrument. Issues with interpreting the instrument have been solved by a written guide for the ALP tool, explaining its application, structure, and defining concepts [17].
Video data can be helpful to increase confidence as it allows viewing and reviewing frame-by-frame. Individual testing of video assessment could develop confidence in recognizing indicators of performance and linking them to phases in the ALP instrument. Video also allows group elaborations and comparisons of videos from different occasions [52]. In clinical practice, confidence could be increased by continuously having collaborative interpretation and peer assessment of videos with colleagues of the same or of other professions and accompanying persons as relatives or personal assistants.

Study Benefits and Limitations

Mixed methods provided the opportunity to collect data in multiple ways, which was considered important to strengthen the credibility of the study [45]. The ALP instrument was developed based on data collected with children and adults with profound intellectual disabilities, which is a benefit as tests and instruments developed for this population are lacking [10,40]. The decision to use a weighted kappa coefficient was based on the comparison of categorical ordered data, i.e., ordinal data (phases of learning 1–8), and the observed proportion of agreement between two raters (in our study raters and the expert reference standard), with adjustment for the proportion of agreements that would be expected simply by chance. Intraclass correlation (ICC) should be used if the data is numerical. Kirkwood and Sterne [53] (p. 439) stated that with categorical ordered variables, the value for weighted kappa will be very close to the value of ICC, especially if the outcome of the test is moderate. We therefore think that our decision to use weighted kappa for our ordinal data is the correct way of calculating the degree of agreement between the raters in our study.
The raters’ assessments were distributed across the eight phases of the learning process, suggesting they could interpret the ALP instrument. However, it is noteworthy that the inter-rater reliability test showed that the raters’ degree of agreement was higher in the earlier phases of the learning process. This is a benefit, as there are few assessments that reliably differentiate behavior in the lower scores [20,40]. However, the raters showed a strong tendency to underestimate phase compared to the reference standard. This is confirmed by other research showing that conceptions and expectations of an individual’s abilities may be low [5] and also there is a tendency to underestimate learning in persons with profound intellectual and multiple disabilities who take part in intervention studies [40].
The Think-aloud data, in combination with the inter-rater reliability and the free commentaries in the assessment form, provided insights into raters experiences of difficulties in assessing a person’s phase of tool-use understanding. Their reasoning added evidence that the structure and indicators in the ALP instrument were perceived. The raters’ experiences also indicated a need to strengthen confidence in applying the ALP instrument. In this study, the raters took part in a 4.5-h workshop (theory/practice, 50/50). The practice time was, even though deemed acceptable, possibly too short and may have influenced the raters’ confidence both in interpreting the performance of AKKA-use and indicators in the ALP instrument. During the study, the authors tried to consider their own pre-understanding from long experience with people with profound intellectual disabilities, which may be considered both an advantage and a bias for the study. It was a benefit that the two authors involved in the sampling of video clips and developing a reference standard for assessment had long experience and knowledge of both the population and the application of the ALP instrument.

5. Conclusions

This mixed-methods study suggests that the process-based instrument universal Assessment of Learning Process (ALP) can be applied to persons with profound intellectual disabilities who perform self-produced mobility using one micro-switch to activate an AKKA mobility platform with a line-follower system. The moderate kappa value may be acceptable, as the ALP instrument is used to assess learning, and studies of this population often show lower inter-rater reliability. The assessments were distributed across all phases of the learning process, which indicated the raters could use the instrument to differentiate the eight phases of learning. The result also indicated that experience working with the population, knowledge of the individual, and knowledge of advanced use of the AKKA may have influenced the assessment of an individual’s phase in the tool-use learning process.
The assessment with the ALP instrument guides the choice of ALP-facilitating strategies, and the choice of appropriate strategies supports the individual’s achievement of tool-use understanding in a continuous circular process. For people with profound intellectual and multiple disabilities, the opportunity to be active and participate is a human right. Supporting the learning of self-produced mobility using the AKKA provides conditions for being active and making choices based on one’s own potential, which may increase a sense of control and agency and expand the activity repertoire on their own terms.

Author Contributions

Conceptualization, methodology, software, validation and formal analysis, C.M., A.-M.Ö. and L.N.; investigation, resources, data curation, writing—original draft preparation, C.M.; writing—review and editing, visualization, C.M., A.-M.Ö. and L.N.; supervision, A.-M.Ö. and L.N.; project administration, C.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding. The APC was funded by Lund University.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, based on approvals by the Regional Ethical Review Board in Umeå, Sweden (Ref no: 94–229, §311; Ref no: 97–101, §144; Ref no: 98–335, §422; Ref no: 03–030, §163).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study, as described in Ethical Considerations in Materials and Methods.

Data Availability Statement

The data presented in this study are available on request from the first author. The videographic data is not publicly available due to ethics and confidentiality.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Levy, S.; Young, H. Arts, Disability and Crip Theory: Temporal Re-Imagining in Social Care for People with Profound and Multiple Learning Disabilities. Scand. J. Disabil. Res. 2020, 22, 68–79. [Google Scholar] [CrossRef]
  2. Nakken, H.; Vlaskamp, C. A Need for a Taxonomy for Profound Intellectual and Multiple Disabilities. J. Policy Pract. Intellect. Disabil. 2007, 4, 83–87. [Google Scholar] [CrossRef]
  3. Channon, A. Intellectual Disability and Activity Engagement: Exploring the Literature from an Occupational Perspective. J. Occup. Sci. 2014, 21, 443–458. [Google Scholar] [CrossRef]
  4. Skarsaune, S.N. Self-determination of people with profound intellectual and multiple disabilities. Dev. Med. Child Neurol. 2023, 65, 16–23. [Google Scholar] [CrossRef]
  5. Talman, L.; Wilder, J.; Stier, J.; Gustafsson, C. Staff members and managers’ views of the conditions for the participation of adults with profound intellectual and multiple disabilities. J. Appl. Res. Intellect. Disabil. 2019, 32, 143–151. [Google Scholar] [CrossRef]
  6. Townsend, E.; Wilcock, A.A. Occupational justice and Client-Centred Practice: A Dialogue in Progress. Can. J. Occup. Ther. 2004, 71, 75–87. [Google Scholar] [CrossRef] [PubMed]
  7. Nilsson, I.; Townsend, E. Occupational justice—Bridging theory and practice. Scan. J. Occup. Ther. 2010, 17, 57–63. [Google Scholar] [CrossRef]
  8. Johnson, K.R.; Bagatell, N. “No! You can’t have it”: Problematizing choice in institutionalized adults with intellectual disabilities. J. Intellect. Disabil. 2020, 24, 69–84. [Google Scholar] [CrossRef]
  9. Haines, D.; Wright, J.; Comerasamy, H. Occupational Therapy Empowering Support Workers to Change How They Support People with Profound Intellectual and Multiple Disabilities to Engage in Activity. J. Policy Pract. Intellect. Disabil. 2018, 15, 295–306. [Google Scholar] [CrossRef]
  10. Wessels, M.D.; van der Putten, A.A.J.; Paap, M.C.S. Inventory of Assessment Practices in People with Profound Intellectual and Multiple Disabilities in Three European Countries. J. Appl. Res. Intellect. Disabil. 2021, 34, 1521–1537. [Google Scholar] [CrossRef]
  11. Simpson, R.C. Smart wheelchairs: A literature review. J. Rehabil. Res. Dev. 2005, 42, 423–435. [Google Scholar] [CrossRef]
  12. Simpson, R.; LoPresti, E.; Hayashi, S.; Nourbakhsh, I.; Miller, D. The smart wheelchair component system. J. Rehabil. Res. Dev. 2004, 41, 429–442. [Google Scholar] [CrossRef]
  13. Jonasson, M. The AKKA-board—Performing mobility, disability and innovation. Disabil. Soc. 2014, 29, 477–490. [Google Scholar] [CrossRef]
  14. Kenyon, L.K.; Farris, J.P.; Brockway, K.; Hannum, N.; Proctor, K. Promoting self-exploration and function through an individualized power mobility training program. Pediatr. Phys. Ther. 2015, 27, 200–206. [Google Scholar] [CrossRef]
  15. Nilsson, L.; Durkin, J. Assessment of learning powered mobility use-Applying grounded theory to occupational performance. J. Rehabil. Res. Dev. 2014, 51, 963–974. [Google Scholar] [CrossRef]
  16. JCM Elektronik. About AKKA: The AKKA System. Available online: https://jcmelektronik.se/sida/om-akka (accessed on 1 August 2023).
  17. Nilsson, L. Driving to Learn, the ALP-Tool. Available online: https://www.lisbethnilsson.se/en/alp-tool/ (accessed on 1 August 2023).
  18. Nilsson, L.; Eklund, M.; Nyberg, P.; Thulesius, H. Driving to learn in a powered wheelchair: The process of learning joystick use in people with profound cognitive disabilities. Am. J. Occup. Ther. 2011, 65, 652–660. [Google Scholar] [CrossRef]
  19. Durkin, J. Discovering powered mobility skills with children: ‘responsive partners’ in learning. Int. J. Ther. Rehabil. 2009, 16, 331–341. [Google Scholar] [CrossRef]
  20. Nilsson, L.; Kenyon, L.K. Assessment and intervention for tool-use in learning powered mobility intervention: A focus on tyro learners. Disabilities 2022, 2, 304–316. [Google Scholar] [CrossRef]
  21. Field, D.A.; Livingstone, R.W. Power mobility skill progression for children and adolescents: A systematic re-view of measures and their clinical application. Dev. Med. Child. Neurol. 2018, 60, 997–1011. [Google Scholar] [CrossRef] [PubMed]
  22. Nilsson, L.; Durkin, J. Powered mobility intervention: Understanding the position of tool use learning as part of implementing the ALP tool. Disabil. Rehabil. Assist. Technol. 2017, 12, 730–739. [Google Scholar] [CrossRef] [PubMed]
  23. Nilsson, L.; Eklund, M.; Nyberg, P. Driving to Learn in a powered wheelchair: Inter-rater reliability of a tool for assessment of joystick-use. Aust. Occup. Ther. J. 2011, 58, 447–454. [Google Scholar] [CrossRef]
  24. Svensson, E.; Nilsson, L. Inter-rater reliability of the assessment of learning powered mobility use, version 2.0, when applied with children and adults engaged in Driving to Learn in a powered wheelchair. Aust. Occup. Ther. J. 2021, 68, 115–123. [Google Scholar] [CrossRef] [PubMed]
  25. Livingstone, R.W.; Field, D.A. Exploring change in young children’s power mobility skill following several months’ experience. Disabil. Rehabil. Assist. Technol. 2020, 18, 285–294. [Google Scholar] [CrossRef] [PubMed]
  26. Kenyon, L.K.; Farris, J.P.; Aldrich, N.J.; Usoro, J.; Rhodes, S. Changes in Electroencephalography Activity in Response to Power Mobility Training: A Pilot Project. Physiother. Can. 2020, 72, 260–270. [Google Scholar] [CrossRef]
  27. Fujita, H. Early Introduction of power mobility devices for children with Fukuyama Congenital Muscular Dystrophy and its psychological impact on caregivers: A case report. Pediatr. Rep. 2023, 15, 403–413. [Google Scholar] [CrossRef]
  28. Kenyon, L.K.; Farris, J.; Veety, L.; Zondervan, D.K. The IndieTrainer system: A clinical trial protocol exploring use of a powered wheelchair training intervention for children with cerebral palsy. Disabil. Rehabil. Assist. Technol. 2023; 1–11, online ahead of print. [Google Scholar] [CrossRef]
  29. Livingstone, R.; Paleg, G. Enhancing Function, Fun and Participation with Assistive Devices, Adaptive Positioning, and Augmented Mobility for Young Children with Infantile-Onset Spinal Muscular Atrophy: A Scoping Review and Illustrative Case Report. Disabilities 2021, 1, 1–22. [Google Scholar] [CrossRef]
  30. Rosenberg, L.; Maeir, A.; Gilboa, Y. Feasibility Study of a Therapeutic Mobility Summer Camp for Children with Severe Cerebral Palsy: Power Fun. Phys. Occup. Ther. Pediatr. 2020, 40, 395–409. [Google Scholar] [CrossRef]
  31. Feldner, H.; Plummer, T.; Hendry, A. A Guideline for Introducing Powered Mobility to Infants and Toddlers. Available online: https://www.permobil.com/en/clinical-research-1/research-library/a-guideline-for-introducing-powered-mobility-to-infants-and-toddlers (accessed on 1 August 2023).
  32. Gefen, N.; Rigbi, A.; Tamar Weiss, P.L. Reliability and validity of pediatric powered mobility outcome measures. Disabil. Rehabil. Assist. Technol. 2022, 17, 882–887. [Google Scholar] [CrossRef]
  33. Smith, R.O.; Scherer, M.J.; Cooper, R.; Bell, D.; Hobbs, D.A.; Pettersson, C.; Seymour, N.; Borg, J.; Johnson, M.J.; Lane, J.P.; et al. Assistive technology products: A position paper from the first global research, innovation, and education on assistive technology (GREAT) summit. Disabil. Rehabil. Assist. Technol. 2018, 13, 473–485. [Google Scholar] [CrossRef]
  34. Nijs, S.; Maes, B. Assistive technology for persons with profound intellectual disability: A European survey on attitudes and beliefs. Disabil. Rehabil. Assist. Technol. 2021, 16, 497–504. [Google Scholar] [CrossRef] [PubMed]
  35. Fonteyn, M.E.; Kuipers, B.; Grobe, S.J. A description of Think Aloud method and protocol analysis. Qual. Health Res. 1993, 3, 430–441. [Google Scholar] [CrossRef]
  36. Charters, E. The Use of Think-aloud Methods in Qualitative Research an Introduction to Think-aloud Methods. Brock Educ. J. 2003, 12, 68–82. [Google Scholar] [CrossRef]
  37. Vanbelle, S. A New Interpretation of the Weighted Kappa Coefficients. Psychometrika 2016, 81, 399–410. [Google Scholar] [CrossRef]
  38. Landis, J.R.; Koch, G.G. The Measurement of Observer Agreement for Categorical Data. Biometrics 1977, 33, 159–174. [Google Scholar] [CrossRef] [PubMed]
  39. Graneheim, U.H.; Lundman, B. Qualitative content analysis in nursing research: Concepts, procedures and measures to achieve trustworthiness. Nurse Educ. Today 2004, 24, 105–112. [Google Scholar] [CrossRef] [PubMed]
  40. Maes, B.; Nijs, S.; Vandesande, S.; Van Keer, I.; Arthur-Kelly, M.; Dind, J.; Goldbart, J.; Petitpierre, G.; Van der Putten, A. Looking back, looking forward: Methodological challenges and future directions in research on persons with profound intellectual and multiple disabilities. J. Appl. Res. Intellect. Disabil. 2021, 34, 250–262. [Google Scholar] [CrossRef] [PubMed]
  41. Lancioni, G.E.; O’Reilly, M.F.; Basili, G. Use of microswitches and speech output systems with people with severe/profound intellectual or multiple disabilities: A literature review. Res. Dev. Disabil. 2001, 22, 21–40. [Google Scholar] [CrossRef]
  42. Roche, L.; Sigafoos, J.; Lancioni, G.E.; O’Reilly, M.F.; Green, V.A. Microswitch Technology for Enabling Self-Determined Responding in Children with Profound and Multiple Disabilities: A Systematic Review. Augment. Altern. Commun. 2015, 31, 246–258. [Google Scholar] [CrossRef]
  43. Martin, A.-M.; Andrews, T.; Goldbart, J.; Landers, M. Reconciling communication repertoires: Navigating interactions involving persons with severe/profound intellectual disability, a classic grounded theory study. J. Intellect. Disabil. Res. 2022, 66, 332–352. [Google Scholar] [CrossRef]
  44. Hostyn, I.; Maes, B. Interaction with a person with profound intellectual and multiple disabilities: A case study in dialogue with an experienced staff member. J. Intellect. Dev. Disabil. 2013, 38, 189–204. [Google Scholar] [CrossRef] [PubMed]
  45. Selick, A.; Durbin, J.; Salonia, C.; Volpe, T.; Orr, E.; Hermans, H.; Zaretsky, L.; Malhas, M.; Lunsky, Y. The Nuts and Bolts of Health Care: Evaluating an initiative to build direct support professional capacity to support the health care of individuals with intellectual disabilities. J. Appl. Res. Intellect. Disabil. 2022, 35, 623–632. [Google Scholar] [CrossRef] [PubMed]
  46. Martin, A.-M.; O’Connor-Fenelon, M.; Lyons, R. Non-verbal communication between nurses and people with an intellectual disability: A review of the literature. J. Intellect. Disabil. 2010, 14, 303–314. [Google Scholar] [CrossRef] [PubMed]
  47. Nijs, S.; Vlaskamp, C.; Maes, B. Children with PIMD in interaction with peers with PIMD or siblings. J. Intellect. Disabil. Res. 2016, 60, 28–42. [Google Scholar] [CrossRef] [PubMed]
  48. Nilsson, L. Driving to Learn: The Process of Growing Consciousness of Tool Use: A Grounded Theory of De-Plateauing. Ph.D. Thesis, Faculty of Medicine, Lund University, Lund, Sweden, 9 March 2007. [Google Scholar] [CrossRef]
  49. Scherer, M. Assessing the benefits of using assistive technologies and other supports for thinking, remembering and learning. Disabil. Rehabil. 2005, 27, 731–739. [Google Scholar] [CrossRef]
  50. Munde, V.; Vlaskamp, C. Initiation of activities and alertness in individuals with profound intellectual and multiple disabilities. J. Intellect. Disabil. Res. 2015, 59, 284–292. [Google Scholar] [CrossRef] [PubMed]
  51. Talman, L.; Wilder, J.; Stier, J.; Gustafsson, C. Staff’s and managers’ conceptions of participation for adults with profound intellectual disabilities or profound intellectual and multiple disabilities. Scand. J. Disabil. Res. 2019, 21, 78–88. [Google Scholar] [CrossRef]
  52. Nilsson, L. Using video data in grounded theory research. In Grounded Theory: The Philosophy, Method and Work of Barney Glaser, 1st ed.; Martin, V.B., Gynnild, A., Eds.; Brown Walker: Irvine, CA, USA, 2012; pp. 103–116. [Google Scholar]
  53. Kirkwood, B.R.; Sterne, J.A.C. Essential Medical Statistics, 2nd ed.; Blackwell Science Ltd.: Hoboken, NJ, USA, 2003. [Google Scholar]
Table 1. Rater profession, clinical experience, and knowledge of the ALP instrument.
Table 1. Rater profession, clinical experience, and knowledge of the ALP instrument.
RaterProfessionYears of Experience with Population and Micro-SwitchesExperience of the AKKA InterventionKnowledge of the ALP Instrument
1Occupational Therapist3XX
2Occupational Therapist6XX
3Occupational Therapist22XX
4Occupational Therapist30
5Speech–Language Pathologist14
6Speech–Language Pathologist30
7Special Education Teacher12
8Special Education Teacher16
Table 2. Data on video clips, individual equipment and adaptations, reference standard ratings, and raters’ experiences of high difficulty or low confidence when assessing the video clip.
Table 2. Data on video clips, individual equipment and adaptations, reference standard ratings, and raters’ experiences of high difficulty or low confidence when assessing the video clip.
Video Clip NoLength
Min/S		Micro-Switch TypeMicro-Switch Diameter
Inch/cm		Micro-Switch PlacementReference Standard RatingN Rating High Difficulty *
(4 or 5)		N Rating Low Confidence **
(1 or 2)
15.21Spec switch1.4/3.5In hand palm733
21.20Big red5.0/12.7On tray211
31.58Pico button1.18/3.0In hand palm455
40.55Spec switch1.4/3.5On forearm532
51.56Big red5.0/12.7On tray345
6 *0.40Big red5.0/12.7Universal arm101
7 *1.26Spec switch1.4/3.5In hand palm325
8 *1.34Big red5.0/12.7On tray342
9 *1.27Spec switch1.4/3.5On forearm813
102.02Big red5.0/12.7On tray534
111.21Spec switch1.4/3.5In hand palm113
122.26Spec switch1.4/3.5In hand palm546
131.48Spec switch1.4/3.5In hand palm476
141.30Big red5.0/12.7Universal arm223
150.46Big red5.0/12.7On tray356
161.34Big red5.0/12.7On tray246
171.52Big red5.0/12.7On tray514
184.30Spec switch1.4/3.5On hand back745
192.37Big red5.0/12.7On tray545
201.05Big red5.0/12.7On tray632
211.02Pico button1.18/3.0In hand palm722
220.27Jellybean2.5/6.4On tray145
231.33Big red5.0/12.7Universal arm613
N = number of raters experiencing high difficulty or low confidence; * Difficulty to assess clip, rated from 1 (low) to 5 (high); ** Confidence that assessment is accurate, rated from 1 (low) to 5 (high).
Table 3. Inter-rater reliability comparing all raters and grouped by profession and experience.
Table 3. Inter-rater reliability comparing all raters and grouped by profession and experience.
Comparisons (N)KW95% CI
RS—All raters (8)0.450.37–0.53
RS—Occupational Therapists, experience with the AKKA platform (3)0.500.37–0.63
RS—Occupational Therapists (4)0.490.38–0.61
RS—Speech Language Pathologists (2)0.410.25–0.57
RS—Special Education Teacher (2)0.400.25–0.55
RS = Reference standard.
Table 4. Diagram showing the contingency between eight raters’ total number of assessments of ALP phase (184) and the reference standard.
Table 4. Diagram showing the contingency between eight raters’ total number of assessments of ALP phase (184) and the reference standard.
PhaseRaters’ Assessment of ALP Phase (1–8)Total *
12345678
Reference Standard
ALP-phase
(1–8)		117 **61 24
24153 11 24
3181055 2132
42552 12 17
5131011248 39
6 133161116
7 126474 24
8 113128
Total *253934281322194184
* Raters number of assessments, per phase and total (184); ALP = Assessment of Learning Process. ** The bold numbers in the diagonal represent total agreement of assessed ALP-phase when the eight raters’ assessment was compared to the reference standard assessment.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Modh, C.; Öhrvall, A.-M.; Nilsson, L. Assessing the Tool-Use Learning Process in Persons with Profound Intellectual and Multiple Disabilities. Disabilities 2023, 3, 477-492. https://doi.org/10.3390/disabilities3040030

AMA Style

Modh C, Öhrvall A-M, Nilsson L. Assessing the Tool-Use Learning Process in Persons with Profound Intellectual and Multiple Disabilities. Disabilities. 2023; 3(4):477-492. https://doi.org/10.3390/disabilities3040030

Chicago/Turabian Style

Modh, Christina, Ann-Marie Öhrvall, and Lisbeth Nilsson. 2023. "Assessing the Tool-Use Learning Process in Persons with Profound Intellectual and Multiple Disabilities" Disabilities 3, no. 4: 477-492. https://doi.org/10.3390/disabilities3040030

Article Metrics

Back to TopTop