A Large Scale, App-Based Behaviour Change Experiment Persuading Sustainable Mobility Patterns: Methods, Results and Lessons Learnt

1

Figure 2   describes the different stages of the research. The reviewer noticed that the   first stage and the second stage have a longer time elapse than the period   between second and third stage. Is there any special reason?

Or is it more   due to the time to process the data in the first stage and select qualify   groups to continue toward the second and third stage?

Or is it because   it is better to have the pre- and post-intervention mobility data collected   during the same period?

Also, the   reviewer sees the intervention period is during winter, which may have   disadvantaged climate than other seasons (e.g. biking in winter is more   challenging than warm seasons).

Is it possible   to have some travel inventory to show seasonable changes in mobility behavior   as an additional control factor here?

As the reviewer correctly argues, the experiment was   designed in order to guarantee that pre-intervention (period A) and   post-intervention (period C) data are directly comparable, namely they take   place during the same months of the year (March and April, respectively 2016   and 2017). Once this condition was guaranteed, setting the specific period of   the year for the intervention (period B) was less critical a choice. We opted   for running the intervention well after the Summer holidays and to offer it   for a sufficiently long period of time to favour adoption of enduring less   car-dependent mobility patterns.

Use of public transport is minimally affected by seasonal   variations; instead, we acknowledged promoting bicycle use during colder   months might be more difficult. Due to this design choice, the GoEco! intervention was run in the   most adverse conditions and therefore its possible impacts might be   under-estimated (since bicycle use might have been higher in the periods A   and C Spring months).

Finally, since the assessment of the intervention is made   by comparing mobility data collected between period C and period A, which   refer to the same season, and the experiment also includes a control group, which   allows to account for any variations in external variables, including the   meteorological seasonal ones, we opted for not accounting for any seasonal   control factor and for simply comparing data collected during period A and   period C.

Line 228

Since the assessment of the   intervention is made by comparing mobility data collected between period C   and period A, the specific time of the year chosen for period B is a less   critical choice; it is set from October to January to avoid overlapping with   the Summer holidays, during which management of the intervention is expected   to be more complex and less effective.

2

Page 7 described the rules to   categorize users in Period A.

For the “active weeks”, do you treat weekend and weekdays the   same during counting? Or does the individual have to have at least one   weekend day and three weekdays out of the four “active days”.

Also, for pure curiosity, the   reviewer wonders how “active participates” are relate to the “prices/CHF   provided”, the rank as shown in Figure 1, and the type of prices an   individual chooses. For instance, an individual who get higher price may be   more motivated and an individual who wins the folding bicycle might have more   biking trips afterward (and should be excluded from the analysis?).

For the active weeks, we simply looked for four days with   at least one collected data, without distinctions between weekdays and   weekends, since for our analysis regarding hypotheses H1 and H2 we considered   average weekly mobility patterns.

Having a larger dataset, it would have definitely been   interesting to analyse working and non-working days separately.

We added a short reference to this aspect in the text.

We did not analyze if and how prizes actually influenced   the level of activity of a user within the experiment, nor if they stimulated   stronger adoption of specific transport modes. They were actually introduced   with the aim of keeping the interest of the users high towards app use, thus   guaranteeing a sufficiently large    amount of  data collected. Since   they were not attributed based on  the   use of any specific transport mode, but only based on  trip-validation in the app, we suppose they   had a limited influence in driving specific choices of the transport mode .   Also, the folding bicycle offered by the final random draw was only   attributed once all project activities had concluded, since the draw was   performed after the end of tracking period C. Therefore, by the very design   the attribution of that prize could not influence the experiment.

Line 266

Thus, at the end of period A, the full set of routes   collected by GoEco! Tracker for   each participant is processed, in order to decide whether they are allowed to   enter period B. Period A data is processed based on the following rules:

consider only days   with at least one validated route (→“active days”; no difference is made between working and non-working   days);

consider only weeks   with at least four “active days” (→“active weeks”);

if the participant   has at least three active weeks, at least fifty routes, and at least   validated 80% of the routes (→“active participant”), she can enter period B   and has to be randomly attributed to treatment or control group; otherwise,   she has to be excluded from the experiment.

Line 293

For vouchers, users are allowed to choose between Swiss   retailers, public transport tickets, or charity donations.  Since gathering correct mobility data is   essential for the quantitative assessment of our research hypotheses, such prizes   are only attributed if participants have validated all their routes and are only distributed at the end of the tracking period   when they are issued. Note that such prizes are introduced with the only aim   of maintaining the users' interest over time, namely to guarantee collection   of a sufficiently large and reliable amount of data. Since their attribution   is not related to any specific transport mode choice by the users, and only   depends on trip-validation in the app, they are not expected to produce any   influence on the users' mobility behaviour. Neither the potential folding   bicycle prize offered by the final random draw can influence the users'   mobility patterns and introduce a bias in the experiment, since the draw   takes place after the end of the three tracking periods.

3

The paper   briefly mentioned about the estimation of the Co2 emissions.

Based on the   reviewer’s personal experience, unless a device is plugged into a vehicle,   this amount will be an estimated number based on some emission models.

For models such   as MOVES in U.S., the vehicle type is an essential component for estimating   the total emissions. Therefore, does the survey contain a question about the   user’s vehicle type? Or does the app generate the costs based on “average”   amounts?

Also, when   using public transit that is also motorized, how does the app determine the   “reduction in emissions”?

We agree with the reviewer, CO₂ emissions   produced on every trip

depend on the specific vehicle model, on the driving   style, on the number of passengers, etc., and providing correct values of CO₂   emissions is not possible without specific measuring devices.

Therefore, the values provided by GoEco! are only meant as a rough estimate, which is expected to   support the comparison between the transport modes and to increase the user   awareness, favouring progress from the “pre-contemplation” to the   “contemplation” of behaviour change.

In this framework, we opted for adopting by default the   “Mobitool” emission factor values, which estimate the average CO₂ emission   per kilometre per transport mode in Switzerland, therefore accounting for the   specific mix of the Swiss electricity generation system and of the Swiss   fleet of passenger cars.

For cars, users were allowed to enter the average fuel   consumption value of their car (expressed in fuel liters per 100 kilometers),   otherwise GoEco! automatically   considered the Mobitool default   values for ICE cars, as they still make up the majority of cars on Swiss   roads.

To better clarify these aspects, we updated the related   text.

GoEco! did not   show reductions in emissions; instead, for every travelled route it always   showed the estimated of the produced emissions, depending on the specific   transport mode used.

Line 103

Soon after validation, the users are provided with   feedback about the individual routes they have travelled (kilometers and   travel time, as well as CO₂ emissions produced   and energy consumed, estimated based on the   Mobitool consumption and emissions factors available for Switzerland [36], that depend on the transport mode used and on the amount of   kilometers travelled, independently on the vehicle’s occupancy rate). To get   more realistic estimates of impact, users are also allowed to set the average   fuel consumption value of their car, expressed in fuel liters per 100   kilometers, which is then used by GoEco!   to customize the Mobitool estimates.  Once per week, they are also given a summary   of their mobility patterns and impacts (average weekly kilometers and   travelling time, percentage use of transport modes). After four weeks of app   use, they are also provided with information about their "baseline   mobility patterns", namely how they travel on average and the related impact on energy consumption and CO₂   emissions.

4

The paper   discusses that a large part of users is not qualify because they have not   validate their travels and activities.

The reviewer wonders if this can be improved in the future study   by (1) allow some grace periods (sometime, a person is just too busy to   respond in one day and may have more time during weekends),

and (2) provide some additional privacy option ( choose to stop   recording for one day or two due to some private business that they don’t   want to share, which may not be a bad idea since the absence of those data   could be another research topic).

Since the survey lasts for quite a long time, it may also be   helpful to distribute those random prices (weekly draw, monthly draw, end of   survey “anniversary” draw) and enable users to earn points toward prices   (like Starbucks and many other stores).

The approach suggested by the reviewer exactly coincides   with the GoEco! approach, that we   had not presented in details in the manuscript. In fact, users were not given   any deadline to check and validate their routes, being free to do it whenever   they preferred it.

Also, they were allowed to temporarily stop us recording   their data, by temporarily stopping the Moves mobility tracking app.

Finally, prizes were only attributed at the end of the   tracking period when they were issued (and prizes of the final random draw   where only attributed at the end of the whole project), and only to users who   had validated all their tracked routes. So, unfortunately, these strategies   were not sufficient to significantly reduce the number of drop-outs.

Instead, we opted for not attributing points, since we   preferred to overcome “one-size-fits-all” solutions, which in previous   research were criticized. A detailed explanation of why we opted for the   specific GoEco! design is provided   in another manuscript about the GoEco!   process, which has just been published

(Cellina,   F.; Bucher, D.; Veiga Simão, J.; Rudel, R.; Raubal, M.  Beyond Limitations of Current   Behaviour Change Apps for Sustainable Mobility:  Insights from a User-Centered Design and   Evaluation Process. Sustainability2019,11.    doi:10.3390/su11082281).

Line 96

[…] GoEco!   tracks all the travelled routes and, by means of   on-purpose developed algorithms [36],  detects the transport mode used, asking   users for a manual validation of the detected transport modes (either   confirmation or correction). All the travelled routes   are automatically collected by the app, with no manual activities requested   by the users, apart for the validation of the transport mode. If needed,   users can temporarily stop the Moves tracking tool, and they can provide the   validations at their own convenience, for example in the evening or in the   weekends, going back to all the routes travelled in the previous week.   Soon after validation, […]

Line 293

For vouchers, users are allowed to choose between Swiss   retailers, public transport tickets, or charity donations.  Since gathering correct mobility data is   essential for the quantitative assessment of our research hypotheses, such   prizes are only attributed if participants have validated all their routes and are only distributed at the end of the tracking period   when they are issued. Note that such prizes are introduced with the only aim   of maintaining the users' interest over time, namely to guarantee collection   of a sufficiently large and reliable amount of data. Since their attribution   is not related to any specific transport mode choice by the users, and only   depends on trip-validation in the app, they are not expected to produce any   influence on the users' mobility behaviour. Neither the potential folding   bicycle prize offered by the final random draw can influence the users'   mobility patterns and introduce a bias in the experiment, since the draw   takes place after the end of the three tracking periods.

5

One question related to analysis method: why choosing Wilcoxon   rank-sum test? The reviewer does not find any discussion on the selection of   methods.  

The data collected showed that the distribution of the   outcome measures (CO₂ emissions and energy consumption)  in the collected sample is skewed.   Therefore, also considering the limited size of the sample, we have preferred   to avoid parametric tests which assume Gaussianity of the tested variable and   apply more robust non-parametric tests. These explanations are provided at   lines 374 and 395; for convenience’s sake, we report them on the right. 

Anyway, we have also verified that using the traditional   t-test leads to the same conclusions.

Line 405

As the field data clearly showed that the distribution of   the variables of interest is not Gaussian, our hypotheses were tested using   non-parametric tests, i.e., the Wilcoxon signed-rank test was used to compare   the emissions and consummations of each individual in periods A and C   (dependent samples), whereas the Wilcoxon rank-sum test was used to compare   the improvements in the treatment and control groups (independent samples).

Line 426

As already mentioned, for   this purpose, considering that data are observably non Gaussian, we used the   non-parametric Wilcoxon signed-rank test.

1

The question that remains and that   I would like to see addressed in the paper is how are (could be) carpooling   initiatives taken into account in the approach?

The GoEco! app   is not capable of automatically detecting the number of users travelling in   the same car, therefore ridesharing or car-pooling routes cannot be   automatically accounted for by the app, as well as the related benefits in   terms of reduced energy consumption and CO₂ emissions.

Therefore, the feedback on one’s own mobility impact is   meant as a

reference, for comparison with the other transport modes.

We revised the text in order to introduce this concept.

Line 103

Soon after validation, the users are provided with   feedback about the individual routes they have travelled (kilometers and   travel time, as well as CO₂ emissions produced   and energy consumed, estimated based on the   Mobitool consumption and emissions factors available for Switzerland [36], that depend on the transport mode used and on the amount of   kilometers travelled, independently on the vehicle’s occupancy rate). To get   more realistic estimates of impact, users are also allowed to set the average   fuel consumption value of their car, expressed in fuel liters per 100   kilometers, which is then used by GoEco!   to customize the Mobitool estimates.  Once per week, they are also given a summary   of their mobility patterns and impacts (average weekly kilometers and   travelling time, percentage use of transport modes). After four weeks of app   use, they are also provided with information about their "baseline   mobility patterns", namely how they travel on average and the related impact on energy consumption and CO₂   emissions.

Line 682

Alternatively, partnerships could   be created with any of the existing carpooling or car-sharing companies or,   more generally, with companies offering Mobility-as-a-Service solutions [61],   requiring their customers to enable location tracking services. Such a   collaboration would not only simplify the location tracking, but also allow   integrating their services into the Gamification part, e.g. by rewarding the   use of carpooling. Note, however, that most of their customers are already   "converted" users of public transport or innovative mobility   services, which would still provide a biased sample.

Line 751

Partnering with them would allow to collect data less   affected by volunteer selection bias, for both the treatment and the control   group, better meeting the requirements for external validity of the   experiment, and might allow the direct integration of   strongly ICT-supported mobility concepts such as carpooling or car-sharing.

2

Line 92 and Line 557:

(Not) Citing a work as “currently under review” does not seem   the most appropriate procedure. I do not know the journal’s policy on this   matter. Perhaps that under review paper could be published in the arXiv   preprint. Obviously, in case it gets accepted while this paper is still under   revision, you should simply update it.

We thank the reviewer for noticing this. In the meanwhile,   our other paper was accepted, therefore at both lines we introduced the   correct reference ([33]).

(Cellina, F.; Bucher, D.; Veiga Simão, J.; Rudel, R.;   Raubal, M. Beyond Limitations of Current Behaviour Change Apps for   Sustainable Mobility: Insights from a User-Centered Design and Evaluation   Process. Sustainability 2019, 11, 2281.)

Line 91

Since the app is largely presented in another work [34], here we limit ourselves to introduce its key   features, following the overview provided in Table 1.

Line 589

The recommendations we collected through the final GoEco! questionnaire and interviews,   reported in details in another work [34], can   be summarized as follows:

3

Line 96:

Moves app is not properly referenced or presented, besides it is   shut down since July 2018.

We updated the text in order to better refer to the Moves   app. Since Moves has been discontinued, we opted for referring to a   conference paper of ours, where we briefly presented Moves.

Line 95

By exploiting basic activity tracking features provided by   the commercial app Moves [35] (discontinued since   July 2018), GoEco! …

4

Figures:

Figure 1 and Figure 2 are not mentioned in the text.

We thank the reviewer for noticing this. We added   references to both figures.

Line 91

Since the app is largely presented in another work [34], here we limit ourselves to introduce its key   features, following the overview provided in Table 1,   and to show some screenshots (Figure 1).

Line 214

Overall, as shown in Figure 2,   the experiment is designed around three mobility tracking periods:

5

Tables:

In my opinion, it would be more intuitive if Table 2 and Table 3   shown the “average difference between periods” as XC – XA. We would be   looking for negative values in CO2 and Energy consumption per km, meaning   there was a decrease between those periods. Table 4 would be revised   accordingly.

We thank the reviewer for this suggestion. We modified   values in Tables 2, 3 and 4 accordingly, and also modified the other parts of   the text where the  ”before/after”   comparison was mentioned (replacing it with “after/before”).

Line 195

Hypothesis 1 (H1) states that,  after treating individuals with the use of   the GoEco! app,  (i) average CO2 emissions per kilometer and   average energy consumption per kilometer are lower than before the   intervention, and (ii) the after/before   difference between these variables is higher than the after/before difference between the same variables, calculated in   the same period for a comparable group of individuals, that are not treated   with the GoEco! app.

Line 416

[Values in Table 2 are modified]

Line 430

In Zurich, instead, no statistically significant after/before difference is found for any of the   considered variables.

Line 433

[Values in Table 3 are modified]

Line 456

[Values in Table 4 are modified]

1

As Gamification   approaches are referred in the introduction, authors are advised to review an   additional reading, which focuses on Gamification approaches for different   transportation concepts [1].

We thank the reviewer for providing us with this very interesting   manuscript. We included it in the references of our manuscript.

Line 46

For an in-depth discussion, one can refer to [22- 25].

22. Shaheen, S.; Cohen, A.; Zohdy, I.; Kock, B. Smartphone   applications to influence travel choices: practices and policies.  Technical report, U.S. Department of   Transportation, Federal Highway Administration, 2016.

23. Anagnostopoulou, E.; Bothos, E.; Magoutas, B.;   Schrammel, J.; Mentzas, G. Persuasive technologies for sustainable urban   mobility.arXiv preprint arXiv:1604.059572016.

24. Sunio, V.; Schmöcker, J.D. Can we promote sustainable   travel behavior through mobile apps? Evaluation and review of evidence.   International Journal of Sustainable Transportation 2017,11, 553–566.

25. Vlahogianni,  E.I.;    Barmpounakis,  E.N.   Gamification  and  sustainable    mobility: Challenges  and   opportunities in a changing transportation landascape.  In Low Carbon Mobility for Future Cities:   Principles and Applications; Hussein, D., Ed.; UK: Institution of Engineering   and Technology, 2017; p. 277.

2

Authors are advised to share more details on the GoEco! App   (Section 2).

Does the app record data all day long or does the user choose to   start recording his/her data?

In Section 2 we on purpose avoided to enter into details   regarding the GoEco! app, both to   keep the focus of the manuscript on the experiment and since the app features   and components are extensively detailed into another work (Cellina, F.; Bucher, D.; Veiga Simão, J.; Rudel, R.; Raubal, M.   Beyond Limitations of Current Behaviour Change Apps for Sustainable Mobility:   Insights from a User-Centered Design and Evaluation Process. Sustainability   2019, 11, 2281).

Therefore, in this manuscript we added the relevant   information suggested by the reviewer, which was not properly addressed in   the other manuscript as well.

Line 96

[…] GoEco!   tracks all the travelled routes and, by means of   on-purpose developed algorithms [36],  detects   the transport mode used, asking users for a manual validation of the detected   transport modes (either confirmation or correction). All   the travelled routes are automatically collected by the app, with no manual   activities requested by the users, apart for the validation of the transport   mode. If needed, users can temporarily stop the Moves tracking tool, and they   can provide the validations at their own convenience, for example in the   evening or in the weekends, going back to all the routes travelled in the   previous week. Soon after validation, […]

Bucher,  D.;  Mangili,    F.;  Cellina,  F.;    Bonesana,  C.;  Jonietz,    D.;  Raubal,  M.      From  location  tracking to    personalized  eco-feedback:   A    framework  for  geographic    information  collection,  processing    and visualization to promote sustainable mobility behaviors. Travel   behaviour and society2019,14, 43–56.

3

What about user privacy?

We added explicit reference to the possibility for the   users to temporarily stop automatic mobility tracking, by temporarily   disabling the Moves app.

4

How does the   app detect the traveling mode? Is it the same classifier that is described in   [2] or is it based on manual entries from the users?

The app performs automatic detection of the transport   mode, based on the same algorithms presented at the ICT2016 conference. A   detailed description of such automatic tracking algorithms is however   presented in [36], therefore we added explicit reference to the manuscript.

Users are however requested to perform a validation of the   detected transport mode, which implies they either confirm or change the   detected mode.

5

How does the   app suggest alternative routes? Although a reference is provided by the   authors, they are advised to include a very short description to make it   easier for the reader to follow. 

We added a short description of how potential mobility   patterns are computed.

Line 118

For a detailed description of how   we perform this activity, please refer to [36].  Basically, the assessment of someone’s   potential is based on the comparison to optimal travel behavior, where the   optimisation criterion is, in this case, the amount of CO₂   emissions: for every travelled route, and specifically for every starting and   final point, we search for all    available itineraries and transport mode alternatives and select the   one with the lowest CO₂ emissions. If the selected alternative   does not provide a relevant improvement compared to the original route, no   suggestion is made.

6

Authors are   advised to clarify how users are motivated. The last part of 2.4 (Even though   prizes for the participants … only have temporary effects.) and 2.6 seem to   describe two different cases.

Indeed, in both Sections 2.4 and 2.6 we were referring to   the same set of tangible prizes. We added a cross-reference, to better   clarify this aspect.

Line 254

Even though prizes for the participants remaining active   until the end of the field experiment are envisioned (see   Section 2.6), the campaign does not explicitly focus on them, in order   to address intrinsic motivational factors as much as possible.

7

In Section 3.2   authors could consider omitting the part regarding the non-parametric   Wilcoxon signed-rank test as it has been repeated in the previous section.

We prefer recalling which test was used in this section   and why, therefore we slightly reformulated the text.

Line 416

As already mentioned, for   this purpose, considering that  data are observably non Gaussian, we   used the non-parametric Wilcoxon signed-rank test.

8

Finally, regarding the end of Section 4.2 and following the   previous comments:

Have the authors considered the popularity   of monetization mechanics?

Tangible and monetary rewards were in principle avoided in   GoEco!, since our goal was to   stimulate mobility behaviour change as a personal, intrinsic choice of app   users, instead of buying it in exchange for money or other tangible goods,   which has been found to only have temporary effects [1]. However, based on   the GoEco! experience, we thought   this approach could be reconsidered, and tangible/monetary rewards could be   initially used as a lure to attract “mainstream car drivers” – and they   should be gradually replaced to let intrinsic motivational factors prevail in   the process of change. Evidence of such a strategy, however, still needs to   be found on the field.

[1] Deci, E.L. Effects of externally mediated rewards on   intrinsic motivation. J. Personal. Soc. Psychol.

1971, 18, 105–115.

Line 164

Also, note that monetary or   tangible performance-based rewards are not included among the GoEco! motivational factors, since its   aim is mostly to stimulate mobility behaviour change as a personal, intrinsic   choice of app users, instead of buying it in exchange for money or other tangible   goods, which has been found to only have temporary effects [38].

[38] Deci, E.L. Effects of   externally mediated rewards on intrinsic motivation. Journal of personality   and Social Psychology1971,18, 105.

Line 667

In this framework, a strategy to raise the interest of   mainstream citizens might be to explicitly include tangible and/or monetary rewards, namely to rely on extrinsic   motivational factors much more than we have done in the GoEco! experiment.

9

Did the users   answer that they would like to receive more push-notifications? What about   blocking spam push notifications?

The reviewer is correct, in showing some perplexities   about increasing notifications. In fact, users asked indeed for more push   notifications, provided however that they were more timely and customized   (such as for example notifications reminding about possible alternatives for   a given route, soon after that route was detected, or notifications noticing   that an ongoing challenge could be completed with limited extra effort).

We slightly reworded the related sentence, to make this   clearer and avoid misunderstandings.

Line 592

-            increase the   frequency of push-notifications, provided that   they are made more user-specific and   personal; […]

1

How we can capture that individuals decide to change in which   stage(s) (based on the table 1)?

According to the transtheoretical model of behaviour   change we referred to, the process of behaviour change necessarily goes   across a number of stages. Individuals start “contemplating” change, then   “prepare” for change, and then take “action” and maintain it over time.   Methods were developed to automatically identify when individuals start   contemplating change, though they were not sufficiently advanced to be used   in GoEco. We added a reference to   such methods

Line 125

Once they get accustomed to this piece of information,   individuals start contemplating change. Even though   ideas and methods for automatically determining the time at which this   happens (from passively recorded mobility data) exist [38], they were not   advanced enough to be used in GoEco.   Therefore, GoEco! assumes that   people start contemplating change after they have been supplied with such a   mobility feedback. 

[38] Jonietz, D.; Bucher, D.   Continuous trajectory pattern mining for mobility behaviour change   detection.  LBS 2018: 14th   International Conference on Location Based Services. Springer, 2018, pp.   211–230.

2

How the change patterns are different between car users and   public transportation users?

Since the final sample of active participants we could   consider in our analyses dramatically reduced with respect to the initial   sample, the size of each subsample of project participants was too small   to  perform reliable analyses aimed at   investigating the effects of the intervention on different  categories of users, obtained based on   their initial mobility patterns and on their environmental attitude.

In the “Conclusions” section we added a specific reference   to these future analyses.

Line 745

Particularly, future works might   be aimed at quantitatively investigating if and how the potential behaviour   change is affected by initial individual mobility patterns and environmental   attitudes.

3

How the environmental attitudes (table 5) affect the switching   behaviour?

4

These questions may require further modeling in future.