Educational Technology Procurement at Canadian Colleges and Universities: An Environmental Scan

Round 1

Reviewer 1 Report

Educational Technology Procurement at Canadian Colleges and Universities: An Environmental Scan

Review

This study developed a questionnaire and conducted interviews to explore how important standards are, and to identify the associated issues with implementing centralized procurement at universities and colleges in Canada. The participants answered to questions related to the current standards adopted by the institutions, the role of standards within the institutions’ procurement process, and their perceptions of other aspects of educational technology related to the adoption future standards.

The topic is interesting: the article contains questionnaire-based analysis and interviews.

In the following are some suggestions for improving the manuscript:

-          References must be enriched.

-          Figures (bars or charts) accompanying the very big tables with responses from questionnaires would be more helpful to the readers.

-          The standards found from this study are not clearly presented: mainly, which are the discovered standards which are mandatory for the procurement of educational software?

-        A section Conclusions must be added, to summarize the results from questionnaires, to focus on their significance and applicability.

Author Response

Thank you for the feedback and comments on our manuscript. The following revisions have been made. The introduction was expanded by adding some information regarding the cost benefits of the various procurement processes (a suggestion from another reviewer), as well as adding some references.

Two figures were added to the results section. Figure one replaced what was previously Table 6, as we felt this information was better portrayed with a graph. Figure two was added to accompany what is now Table 9 to help readers better understand this information as was suggested.

There was a comment that it was unclear which standards were discovered to be mandatory for the procurement of educational software. Our study showed that most institutions vet the EdTech they produce for various criteria such as privacy/security, Privacy & Security, Accessibility and Care of Data Practices etc. However, the procurement process unfortunately does not often vet the EdTech against specific standards related to these areas. This may be because of a lack of a widely accepted standard. This study highlights the need for standards that address these areas and can be adopted by institutions. This is now clarified in the last paragraph of the discussions section. A conclusions section was also added and reviews the findings and importance of this study.

We hope that the revisions made adequately address the comments provided. We look forward to hearing from you.

Thank you,

Hannah Ali

Reviewer 2 Report

The study surveyed individuals involved in EdTech procurement at universities and colleges across Canada about the standards and procurement processes used. The analysis suggests ways to move towards a procurement process that better incorporates standards and addresses identified challenges, thus improving the efficiency and efficacy of EdTech procurement in post-secondary institutions. It provides an overview of how centralised versus decentralised and top-down versus bottom-up approaches to EdTech procurement currently work in Canadian post-secondary institutions.

The study utilises a mixed methods approach incorporating both a questionnaire and interviews to gather rich data from multiple Canadian post-secondary institutions; the research includes perspectives from various units and roles involved in EdTech procurement, providing a well-rounded understanding from numerous viewpoints. The analysis helps fill a significant gap in understanding this topic within the Canadian context.

Weaknesses and limitations of the article:

Unfortunately, the authors use a small sample size. Only 10 Canadian post-secondary institutions participated in the questionnaire and 2 in the interviews. A larger and more representative sample could strengthen the findings.

The article notes that students were the least involved in procurement processes among stakeholders. Getting more student input could provide an essential missing perspective. 

While the study makes practical suggestions for improving procurement processes, it doesn't provide a roadmap to prioritise the most critical or feasible changes to make first.

The analysis discusses the benefits of greater standardisation but does not consider the potential costs or tradeoffs institutions may face in implementing it.

Recommendations to improve the research:

1. Collect concrete procurement policy documents, rubrics, reports, etc., from participating institutions as artefacts to substantiate and compare against self-reported data. It could reveal inconsistencies between policies and actual practices.

2. Focus specifically on developing a prioritised roadmap for incremental changes in institutions to facilitate adopting and centralising standards to produce more actionable recommendations for transparent procurement processes.

3. Present cost-benefit considerations to model potential financial and non-financial costs institutions may incur from transitioning their procurement processes to be more standards-aligned and centralised. Compare it to expected benefits to understand tradeoffs.

Practical recommendations for improving transparency, efficiency, and standardisation would provide readers with valuable insights for decision-making at their institutions.

Anyway, the research questions and goals outlined in the introduction are relevant and clear, and they aid in understanding and framing the overall research. The article has strong potential and would interest readers of journals focused on educational technology and technology procurement practices. It addresses essential gaps and challenges.

There are no clear concerns regarding ethical conduct:

The paper states the University of Toronto Research Ethics Board approved the study. It signals appropriate oversight and review were conducted to protect human subjects.

There is no obvious indication of a failure to obtain informed consent or deception. Recruitment communications framed the purpose, incentives were provided, and questionnaire/interview participation was voluntary.

Confidentiality of responses appears to have been protected.

Author Response

Thank you for the revisions and feedback provided. We have made changes to the manuscript to address the comments. One of the recommendations was to create a roadmap to highlight the changes that institutions can make to improve the procurement processes. This was added to the discussions section. As noted by the reviewer, students were the least involved in the procurement process. Their involvement may have provided useful information and an important perspective. It is recommended in the roadmap that the students are involved in the process.

It was suggested that we provide information on the cost or tradeoffs institutions may experience with implementing greater standardization. Unfortunately, our research did not provide enough information to speak on this. However, we recognize the importance of this information and added some information on cost to the introduction section when the advantages and disadvantages of a centralized and decentralized approach were outlined. We also suggested in the discussions section that future research further assesses this. The introduction section was further revised with more references added.

Lastly, it was recommended that we “collect concrete procurement policy documents, rubrics, reports, etc., from participating institutions”. Although we understand how this would have added to the value of our research, we decided not to contact participating institutions as our study period had ended. Participants were informed, that they would be participating in the questionnaire, and only be contacted for interviews if they chose to participate in that. This aligns with our ethics application.

We hope all amendments made to the manuscript are sufficient and we look forward to hearing from you!

Thank you,

Hannah Ali

Round 2

Reviewer 1 Report

Following previous suggestions, the manuscript can be recommended for publication.

Reviewer 2 Report

Congratulations on your effort for improvement!

The revised paper meets the publishing criteria.