Adults Do Not Appropriately Consider Mass Cues of Object Brightness and Pitch Sound to Judge Outcomes of Collision Events

Abstract

Adults judge darker objects to be heavier in weight than brighter objects, and objects which make lower pitch sounds as heavier in weight than objects making higher pitch sounds. It is unknown whether adults would make similar pairings if they saw these object properties in collision events. Two experiments examined adults’ judgements of computer-generated collision events based on object brightness and collision pitch sound. These experiments were designed as a precursor for an infant study, to validate the phenomenon. Results from the first experiment revealed that adults rated the bright ball likely event (where the bright ball displaced a stationary object a short distance after colliding with it) higher than the bright ball unlikely event. Conversely, adults rated the dark ball unlikely event (where the dark ball displaced a stationary object a short distance after colliding with it) higher than the dark ball likely event. Results from the second experiment demonstrated that adults judged the low pitch unlikely event (where the ball displaced a stationary object a short distance with a low pitch sound) higher than the low pitch likely event. Moreover, adults judged the high pitch likely event (where the ball displaced a stationary object a short distance with a high pitch sound) higher than the high pitch unlikely event. Results of these experiments suggest adults do not appropriately consider object brightness and pitch sound in collision events.

1. Introduction

In recent years, it has been documented that object brightness and pitch of sound an object emits cue mass in adults [1,2]. Adults judge brighter objects and higher pitch sounds to be lighter in weight, and darker objects and lower pitch sounds to be heavier in weight [1]). These associations are a special sort of cross-sensory correspondence [1,3]. Cross-sensory perception occurs when an event stimulates more than one sense [4,5]. In the case of the special sort of cross-sensory pairing of mass, and pitch and brightness, vision provides information about surface lightness and audition about the sound pitch that the visual object emits [2]. Why these pairings exist and what purpose they serve remains unknown [6,7]. Mondloch and Maurer [8] suggest these associations related to weight can be explained by the associations humans form in the natural world. For example, correspondences between object brightness and weight in humans exist in the natural world, because most materials in the world such as wood, soil, and sand become darker in colour and heavier when wet and might be enough to create the association [1]. Similarly, associations between pitch and weight may exist due to a physical basis because animals that produce a low pitch sound are usually bigger in size, thus heavier in weight [1].Other cross-sensory associations involve matchings between brightness and loudness demonstrated in adults [9,10] whereby adults match light grey patches of colour with louder sounds. Similarly, adults pair dark grey patches of colour with quieter sounds [9]. Furthermore, adults demonstrate associations between brightness and pitch [11,12]. Adults pair high-pitched sounds with brighter visual stimuli, and louder sounds with higher contrast visual stimuli [11,12]. There is accumulated evidence to suggest that adults make associations between smaller, sharper, brighter, spatially higher visual images with high-frequency sounds [13]. Similarly, adults make associations between high-pitch tones with sharp, thin, and speedily ascending visual stimuli [13,14,15,16,17,18,19].

Some of the associations between these cross-sensory pairs suggest a pairing based on similarity in weight and mass, but indirectly. For example, dark colours (more mass/heavy weight) are associated with low pitch sounds (more mass/heavy weight), and brighter colours (less mass/light weight) with higher pitch sounds [9,10,11,12]. Similarly, the associations between smaller (less mass/light weight), thin (less mass/light weight), brighter (less mass/light weight), with high-frequency sounds (less mass/light weight) suggest a pairing based on mass hence weight [13,14,16,19]. These associations also influence decision making in adults [13,20,21]. Adults prefer events, situations, and objects that match their cross-sensory associations [20]. This is demonstrated by their slow responses and less accurate estimates of visual stimuli when they are paired with a distractor auditory stimulus that do not match the visual stimuli in terms of cross-sensory associations [4]. For example, adults would be slow and less accurate when presented with a bright stimulus that is paired with a distractor auditory stimulus that is low in pitch. Similarly, adults would be slow in their responses to visual stimuli when the elevation of the visual stimulus is mismatched with pitch [21]. For example, individuals would respond slower to a visual stimulus that is high in space when it is accompanied by a low pitch sound or low in space when accompanied by a high pitch sound [21]. Similarly, participants find it harder to put the visual stimulus into a category in the context of its size (as either large or small) when the task-irrelevant sound on each set of trials are incongruent (e.g., when a large visual target was accompanied by a high-pitch sound), than when trials are congruent (e.g., large visual target accompanied by a low-pitch sound). Furthermore, adults’ preference for cross-sensory mappings has been discovered in speed discrimination tasks as well [13].

Uncertainty remains as to whether object brightness and sound pitch independently cue mass in collision events and as a result affect adults’ perception of dynamic events. The cross-sensory literature highlights the association between object brightness and pitch sound suggest a pairing based on similarity in weight or mass [9,10,11,12]. Moreover, adults consider object properties and masses in assessment of object momentums in dynamic events [22,23]. Vicovaro and Burigana [22] demonstrated material properties (namely mass and elasticity) of objects influenced adults’ judgments of collision event outcomes. For example, collisions between objects of low elasticity such as ‘plasticine’ were expected to produce less displacement than collisions between objects of high elasticity such as ‘wooden’ objects [22]. Similarly, Teixeira and Hecht [23] reported similar links between judgements of elasticity of an object and momentum patterns of that object. Other object properties and masses are also considered in adults’ assessment of object momentums in dynamic events [22]. However, little is known about adults’ perception of the mass cues of object brightness and pitch sound in collision events, and how these influence judgements of object momentums in dynamic events. For example, how far an object of a certain brightness or pitch sound associated with a certain mass can displace another fixed property object.

This paper aimed to investigate whether adults make brightness-mass and pitch sound- mass associations in computer-generated collision events. Specifically, we tested whether dark coloured objects and objects that elicited low pitch sounds would be expected to displace another object further than bright coloured objects and objects which elicited high pitch sounds. The present investigation considered the object properties brightness and pitch sound, designed as a precursor for an infant audience. Thus, the two experiments with adults herein served as a validation process prior to use with infants. The experiment followed a child friendly design by Kotovsky and Baillargeon [24], but a computer-generated approach similar to Hohenberger et al. [25]. In this context, adults were presented with two different collision outcomes for each object property; a shorter and longer travelled distance of stationary object after collision with an object. In a collision event between object A (agent) and object B (patient), we tested whether object A brightness (Experiment 1) or the pitch of sound it produced (Experiment 2) would affect adults’ judgement of the exerted force on object B. We hypothesised a priori collision event outcomes aligned with cued mass of object A (through brightness or sound pitch) would be rated higher (i.e., more likely) than collision event outcomes that did not. Adults rated events on a Likert scale based on how real-life they were from 1 (very unlikely) to 10 (very likely).

2. Experiment 1 Methods

2.1. Participants

Twenty-four adults recruited from Lancaster University between ages 19 years and 30 years (aged 24 ± 3 years) participated. Participants had normal or corrected-to-normal eyesight, no form of synaesthesia, and no colour blindness according to self-report. Of these 24 participants, 12 were female (aged 25 ± 3 years) and 12 were male (aged 23 ± 2 years).

2.2. Materials and Apparatus

Computer-generated collision events were created using Animate C.C (2016), Adobe Systems. Participants watched dynamic collision events on a 34 cm Macbook full screen and were seated 10 cm away from the screen. The backdrop consisted of an image of a wooden table (W = 21 cm, H = 5 cm), background of three houses (W = 14 cm, H = 6 cm), a ramp (W = 4 cm, H = 4 cm), a cube (W = 3 cm, H = 3 cm), a hand (W = 3 cm, H = 2 cm), a white/bright ball (H = 2 cm, W = 2 cm), and a black/dark ball (H = 2 cm, W = 2 cm).

In the events, adults saw the cube propelled by a ball to a position either before the midpoint (shorter distance) or at the endpoint (longer distance) of screen. In the events, the cube was either propelled by a ball of physical properties cuing greater mass (dark/black coloured ball) or lesser mass (bright/white coloured ball). Test events showed a dark/black ball or bright/white ball propel the grey cube to a shorter or longer distance. In the likely outcomes the bright/white ball propelled the cube to before the midpoint, and the dark/black ball propelled the cube to the endpoint of the screen. In the unlikely outcomes the bright/white ball propelled the cube to endpoint and the dark/black ball propelled the cube to before midpoint. The grey ball in Figure 1 remained stationary and was merely a prop.

Participants were shown test event scenes in which a hand was presented but the ball was hidden (for 1 s). Subsequently, the hand was hidden and then visible again holding the ball (for 1 s). The hand placed the ball on the ramp, pressed it down, and after 1 s, the hand was lifted. The ball rolled down the ramp (for 1 s) and propelled the cube in front of the first house to either the end of the first house or to the last house (for 1–2 s). These events continued 1 s after movement ended to allow participants time to perceive the event in its entirety. In total, events in which the cube propelled to the end of the first house lasted 6 s (240 frames, 48 frames/s), and events in which the cube propelled to the last house lasted 7 s (288 frames, 48 frames/s). The cube travelled 2 cm/s from the start of the first house to the end of the first house (shorter condition). The cube travelled 1 cm/s from the start of the first house to the middle of the third house (longer condition).

The auditory stimulus that was presented during collision was a natural sound of a billiard ball hitting a wooden cube. Audition C.C. (2016), Adobe Systems was used to amplify the sound. This stimulus was used for all test events for all experiments. The stimulus had a duration of 0.3 s, an acoustic amplitude of 50–58 dB (range) and an auditory frequency of 32–851 Hz (range). The impact sound (i.e., when the ball hit the cube) was 851 Hz and 58 dB.

2.3. Procedure

Adults were randomly assigned to one of four groups according to the order in which events were watched:

Group one: Dark ball (likely–unlikely)—Bright ball (unlikely–likely)

Group two: Bright ball (unlikely–likely)—Dark ball (likely–unlikely)

Group three: Dark ball (unlikely–likely)—Bright ball (likely–unlikely)

Group four: Bright ball (likely–unlikely)—Dark ball (unlikely–likely)

Participants saw each event once and watched events on a Macbook Air 34 cm screen with headphones on, and verbally assessed the collision events by rating them on a scale from 1 (very unlikely) to 10 (very likely) on how real life they were.

3. Experiment 1 Results

All data were analysed using Jamovi version 2.0 (The Jamovi Project). Data were tested for normal distribution by Shapiro–Wilk’s test and for homogeneity of variance using Levene’s test. Our data violated the normality and parametricity assumptions, thus we performed a Friedman’s Test to examine for differences in ratings between bright likely, bright unlikely, dark likely and dark unlikely collision outcomes. Subsequently, pairwise comparisons (Durbin-Conover) were performed to locate differences. We report alpha levels as exact p values, without dichotomous interpretation of ‘significant’ or ‘non-significant’ as advised by the American Statistical Association [26]. Effect sizes are reported using Kendall’s W for Friedman’s Tests and rank biserial correlations (r) for pairwise comparisons. Both measures of effect are interpreted using Cohen’s guidelines (0.1− < 0.03 small, 0.3− < 0.5 moderate, and ≥ large [27]. Figures were generated in GraphPad Prism (GraphPad Prism 8.4.3, GraphPad Software Inc, San Diego, CA, USA) and display grouped dot plots as recommended by Drummond and Vowler [28] and Weissgerber et al. [29]. Parametric data are summarised in text as mean ± SD and non-parametric data as median [interquartile range (IQR)]

Friedman’s test revealed a difference between ratings of the four collision outcomes based on object brightness, × 2(3, n = 24) = 30.3, p < 0.001, W = 0.42 (moderate effect). Median ratings were higher (p < 0.001, r = 0.94) for the bright likely (shorter distance) event (7 [2]) compared to bright unlikely (longer distance) event (2 [2]). Dark unlikely (shorter distance) event (6 [4]) were rated more likely than the dark likely (longer distance) event (4 [5]) (p = 0.071, r = 0.39) (Figure 2).

4. Experiment 1 Discussion of Findings

Results from study 1 suggest adults differ in ratings for likely and unlikely test events. Participants rated the bright ball likely test event higher than the bright ball unlikely test event (large effect). Interestingly, participants rated the dark ball unlikely test event higher than the dark ball likely test event. This partly supports our hypothesis that collision event outcomes aligned with the mass cue of brightness would be rated higher (i.e., more likely) than collision event outcomes that did not. However, our hypothesis was only supported for the bright ball test events. Adults displayed higher ratings for the dark ball unlikely test event compared to dark ball likely test event, but the magnitude of effect was only moderate for dark ball events, and large for bright ball events. We are therefore inclined to suggest that participants considered other cues to mass, such as size of object A, size of object B, pre-collision velocity, post-collision velocity, coefficient of restitution, and collision sound more than the mass cue of brightness. It appears the event whereby the cube travelled a shorter distance was perceived as the most likely, for reasons we cannot conclusively determine. However, this effect was stronger for bright ball events than dark ball events, suggesting some participants considered object A brightness, at least in some part. For that reason, we conclude adults sometimes use brightness to judge mass in collision events.

Future research could benefit from examining the mass cue of object brightness in collision events to determine under what circumstances object brightness influences decision making. In conclusion, results from this experiment indicate adults sometimes expect certain collision outcomes based on brightness of objects but consider other visual cues more. Expectation of collision event outcomes may not be limited to exclusively visual object cues, so Experiment 2 tested the effect of auditory object cues to mass on expectations of collision event outcomes.

5. Experiment 2 Methods

5.1. Participants

Twenty-four adults, recruited from Lancaster University, between ages 18 years and 64 years (aged 28 ± 11years) participated. Participants had normal or corrected-to-normal eyesight, no form of synaesthesia and no colour blindness by self-report. Of these 24 participants, 12 were female (aged 30 ± 8 years) and 12 were male (aged 27 ± 13 years).

5.2. Materials and Apparatus

Animations were the same as in Experiment 1, except that the balls were the same colour (grey; Figure 3). Additionally, a 0.3 s sound during the collision was pitched high or low for the test events. The low-pitch sound had an acoustic amplitude of 50–56 dB (range) and an auditory frequency of 29–633 Hz (range). The impact sound (i.e., when the ball hit the cube) was 633 Hz and 56 dB. The high-pitch sound had an acoustic amplitude of 50–60 dB (range) and an auditory frequency of 43–1011 Hz (range). The impact sound (i.e., when the ball hit the cube) was 1011 Hz and 60 dB. The amplitudes of low pitch and high pitch collision sounds were adjusted until they were perceived equal by a pilot sample (N = 5). The final dB used for the high and low pitch collision sounds were the average dB suggested by the pilot sample.

5.3. Procedure

Other than the sound manipulation, the procedure was the same as in Experiment 1. Participants were divided into following groups and viewed each event once shown on Figure 3 in the following sequence:

Group one: A-B-C-D

Group two: B-A-D-C

Group three: C-D-A-B

Group four: D-C-B-A

6. Experiment 2 Results

The same statistical analysis was adopted as for Experiment 1. Friedman’s test revealed a difference between ratings of the four collision outcomes based on object pitch sound, ×2(3, n = 24) = 42.2, p < 0.001, W = 0.58 (large effect). Medians indicated ratings were higher (p < 0.001, r = 0.91) for the high pitch likely (shorter distance) event (8 [2]) compared to high pitch unlikely (longer distance) event (2 [1]), with a large magnitude of effect. The median indicated ratings were higher (p < 0.001, r = 0.92) for the low pitch unlikely (shorter distance) event (8 [2]) compared to low pitch likely (longer distance) event (2 [2]), again with a large effect (Figure 4).

7. Experiment 2 Discussion of Findings

Results from study 2 suggest adults rated the short distance outcome as more likely for both high and low pitch events. These findings suggest adults failed to use pitch sounds as a mass cue in collision events. For that reason, we conclude adults might have been influenced by the identical visual cues (e.g., size, object velocity) and found the shorter distance outcomes as more plausible as a result of these copious visual cues.

We cannot fully elucidate the reason the shorter distance outcome was perceived as the most likely and this may be an area for future research. As visual object cues across collision test events in Experiment 2 were identical, and participants rated the shorter distance for both high and low pitch sounds higher compared to longer distance, this auditory information between events (i.e., low or high pitch collision sound) was not considered. Attendance to identical visual information resulted in identical ratings of likeliness and evident ignorance of sound pitch differences or effects of these differences. However, the collision sound pitch is a product of two objects since it was presented during collision. Consequently, it may not have been a valid cue to the launching object’s (object A’s) mass. Regardless, the lower pitch impact sound between two objects should give the impression the collision was between two heavy objects thus object B should have been displaced further. Similarly, a higher pitch impact sound during collision should give the impression the collision was between two light objects thus object B should have been displaced to a shorter distance.

Future research could examine sound pitch in collision events to determine circumstances in which it cues for mass. In conclusion, results from this experiment indicate adults do not consider sound pitch as a mass cue in collision events, and do not hold a certain expectation of collision outcomes based on this.

8. General Discussion

Findings from Experiment 1 highlight adults sometimes considered object brightness as a mass cue (i.e., expect a shorter displacement of a fixed object and a bright object), but considered other cues to mass (such as size of object A, size of object B, pre-collision velocity, post-collision velocity, coefficient of restitution, and collision sound) more than brightness of object A. This experiment included several visual cues which we kept constant (e.g., size of object A, size of object B, pre-collision velocity, post-collision velocity, coefficient of restitution, and collision sound) so these visual cues might have been considered more than object brightness. Previous studies have considered object properties such as size and material properties (i.e., polystyrene, wood, and iron) in collision events [22,23], but not brightness. In these experiments, material properties of balls varied whilst dimensions of balls were constant. Object brightness has been considered in the cross-sensory literature and reported to cue weight in adults when objects have been lifted or assessed verbally [1,2]. This object property has not been considered in the context of collision events until now. Findings of experiment suggest adults may consider object brightness, but weight other mass cues more strongly in computer-generated collision events.

Walker [1] demonstrated adults make brightness and weight associations when judging object weight. For example, bright objects are perceived to weigh less than dark objects. Based on Walker’s [1] findings concerning object brightness, adults in our experiment should have displayed a consideration for object brightness as a cue to mass in collision events. However, the methodology of this experiment and those of Walker [1] differ. The experiment presented herein required adults integrate knowledge of object brightness into context of collision events and judge outcomes based on object brightness. Participants in the Walker [1] investigation were required to judge weight of objects based on property of brightness without further integration. Judgement of collision events based on object brightness is thus complex because of conflicting properties that cue for mass in collision events. Furthermore, object brightness is not considered as strongly in collision events, because collision events involve two objects. For that reason, it involved adults’ assessment of both objects, object properties, and their behaviours in collision events.

Experiment 2 highlighted that adults failed to consider sound pitch as a mass cue in collision events and considered other cues to mass (possible size of object A, size of object B, pre-collision velocity, post-collision velocity, and coefficient of restitution) more than sound pitch, again rating the shorter displacement events as more likely. As results were similar across Experiment 1 and Experiment 2, it is worth noting near identical methods were employed in these experiments. Therefore, it is pertinent to consider why the employed methodology resulted in unexpected findings described above. Concerning cross-sensory literature focusing on sound pitch, Walker [1] reported adults made sound pitch and weight associations. For example, higher pitch sounds were perceived to weigh less than lower pitch sounds. Our results were not in line with Walker’s [1] findings, possibly as a result of divergent methodologies. Adults in the present study were required to integrate knowledge of pitch sound and mass cues into context of collision events and judge the outcome whereas in Walker’s [1] investigation, adults were required to judge the weight of pitch sounds based on sounds alone. In the present study, judgement of pitch sound was complex because the sound occurred during collision, thus may have been perceived as the product of both objects. Regardless, pitch sound, if considered, would still suffice to influence judgement of collision events. However, if pitch sound was not attended to, there were various visual object properties that participants may have considered instead. For example, identical sizes of object A and object B, identical pre- and post-collision velocities of both object A and B. Past research has demonstrated adults make size and pitch sound matchings [1]. According to Walker [1], adults pair large visual stimuli with low pitch sound and small visual stimuli with high pitch sound. In the present experiment, balls were a constant size, colour, and pre-event velocity and post-event velocity to control for visual mass cues. Collision events whereby the cube was displaced to the short distance was rated as most likely which suggests adults may not attend to auditory cues. As both events where the cube was displaced to the short distance was rated as most likely we are inclined to hypothesise participants considered other cues to mass, such as size of object A, size of object B, pre-collision velocity, post-collision velocity, and coefficient of restitution more than collision pitch sound in this specific context.

In the test events utilised in Experiments 1 and 2 of this paper, many mass cues ((i.e., size of object A, size of object B, pre-collision velocity, post-collision velocity, coefficient of restitution, and size and angle of ramp) were present. Vicovaro and Burigana (2016) observed that adults integrate Newtonian mechanics into perception of collision events, despite not necessarily being able to explain the phenomena. Thus, in the present study, these ubiquitous mass cues between trials may have been considered to a greater extent than the independent variables we manipulated in an attempt to cue mass. In this context, if we assume participants ignored pitch sound (Experiment 2) in favour of other mass cues, object A and B were uniform in all trials, as was pre-collision acceleration of object A, post-collision acceleration of object B, and post-collision deceleration of object B. Therefore, if we consider Newton’s second law (F = m·a), mass (m) of objects A and B were perceptually uniform, acceleration (a) of objects A and B were perceptually uniform, so force (F) would be perceptually uniform across trials. Therefore, the same likeliness rating could be expected in short displacement events, and the same likeliness rating could be expected in longer displacement events. This was evident in our findings from Experiment 2 whereby the shorter distance events were rated as 8 [2] and 8 [2], and the longer distance events were rated as 2 [1] and 2 [2], demonstrating short displacement events were uniformly considered more likely. The same can be argued in Experiment 1 as shorter displacement events were rated as 7 [2] (bright) whilst longer displacement events were rated as 2 [2] (bright). As in Experiment 2, short displacement events were uniformly considered more likely in Experiment 1. However, it could be argued participants partially considered brightness as a mass cue, as there was a difference between bright trials and dark trials. Yet, it still appears that the mass cues present during the shorter displacement trials outweighed the independent variable mass cue of brightness. Taken together, it appears participants in the present experiments considered the wholistic scene of computer-generated collision events more than specific variables (i.e., brightness and pitch sound) we manipulated. Secondly, it appears brightness was integrated more into perception of collision event outcomes than pitch sound. Our future work will seek to extend these findings with an infant audience to examine whether these phenomena manifest at an earlier age.