Online Remote Proctoring Software in the Neoliberal Institution: Measurement, Accountability, and Testing Culture

Cristyne Hébert

University of Regina

As COVID-19 spread in early 2020, a lockdown was implemented across Canadian provinces and territories, resulting in the shuttering of physical post-secondary campuses. Universities quickly pivoted to remote learning, and faculty members adjusted their instructional and assessment approaches to align with virtual environments. To aid with this process, a number of institutions acquired licenses to remote online proctoring services. Remote online proctoring functions by mimicking in-person proctoring, “(a) verify[ing] test taker identity, (b) observ[ing] test taker behavior to minimize cheating, and (c) secur[ing] test content” (Langenfeld, 2020, p. 24). The goal of online remote proctoring is to reduce instances of academic dishonesty through surveillance.

Several concerns have been raised about remote online proctoring, including the invasiveness of the software (Chin, 2020; Hubler, 2020), lack of transparency regarding data collection and management (Haq et al., 2015; Morrison & Heilweil, 2020), and the potential for algorithmic bias (Hu, 2020). Demonstrating their opposition to the software, students across the country also petitioned to halt its use (Sandlin, 2020; Walsh, 2020). I share these apprehensions while also taking issue with assessment practices that create space for more corporate involvement in higher education (Hébert, 2015, 2017). Though it has been almost 10 years since I have assigned a quiz, test, or final exam in any of the classes I teach, I recognize that the needs of my colleagues, especially those in STEM subject areas, might differ from my own. Though I have never used remote online proctoring, I set out, in writing this paper, to better understand why one might.

In this article, I argue that the adoption of online remote proctoring represents post-secondary institutions’ commitments to audit culture and continued reliance on testing culture as grounding for assessment practices. I begin with cheating and online remote proctoring, focusing specifically on a lack of research that demonstrates the efficacy of this mode of assessment, coupled with the impact of shifts in the testing environment, and increased student anxiety. In the second section, I offer an overview of some of the reasons provided in adopting online remote proctoring, linking them to neoliberalism and audit culture. And in the last section, I turn to contemporary research on assessment practices in higher education, advocating for an institutional shift to an assessment culture that prioritizes students and their learning.

Online Proctoring, Cheating, and Student Mental Health

Cheating, typically considered one element of academic dishonesty, involves using unauthorized materials or engaging in unauthorized practices as a means of gaining an advantage, resulting, ideally, in enhanced performance on a test (King et al., 2009; Ranger et al., 2020). In face-to-face environments, a number of measures have been employed to curb cheating during testing. Verifying students’ identities against university- or government-issued identification cards is intended to address identity-based cheating, or cheating that occurs when someone other than the registered student completes course work. For access-based cheating¾accessing unsanctioned materials, such as notes or other versions of the same test or exam, and the internet, and engaging in unapproved opportunities for collaboration¾facilitators closely monitor students while they complete tests, distribute different versions of the test, and limit testing time (Owens, 2015).

In online environments, measures to address access-based cheating have included algorithmic test banks, time limits and windows for completion, and preventing test-takers from returning to completed test pages (Holden et al., 2021), and for identity-based cheating, academic integrity or honour codes, which require students confirm that they will uphold the university’s academic integrity policy (Baron & Crooks, 2005; Tatum et al., 2018). Though honour codes depend upon adherence to ethical conduct that students inclined to cheat may simply disregard, some research has suggested that cheating can be minimized when codes are effectively written, regularly communicated to students, and enforced (Gurung et al., 2012; McCabe et al., 2001). Concerned that these measures do not go far enough to limit cheating online, online remote proctoring was created, which mirrors some elements of the face-to-face testing environment by validating student IDs, establishing time frames for tests, and providing proctors who either monitor students live or later review video recorded footage of students’ test taking.

Two factors quite obviously complicate online remote proctoring, both linked to a decrease in control over the testing environment. First, as students take tests from a location of their choosing, proctors both have little command over materials accessed and are unable to view student actions beyond what is visible in the camera screen. These concerns are addressed by adding measures of surveillance, including, most commonly, preventing students from switching between windows during test taking, and closely monitoring their eye, mouth, and hand movements. Second, unlike in face-to-face testing where the proctor is responsible for creating and maintaining the rather stable testing environment, here, responsibility shifts to the student to produce “minimally viable test conditions” (Michel, 2020, p. 29). To complete a test using Proctortrack, for example, students must be in a well-lit and private area, far from potential sources of noise. Students are instructed to remain in “the middle of the webcam’s view,” ensure that their face is visible during the assessment, and “sit upright in an area where other people won’t talk and cannot pass behind [them]” (Verificient Technologies, 2021). Similarly, standard test rules for the platform Examity include “alone in room,” “clear desk and area,” “no phones or headphones,” “no leaving seat” “no talking” and “the proctor must be able to see you for the duration of the test” (Examity, 2019). And for ProctorU, the list of what is prohibited during the exam includes “talking aloud,” “being out of camera view,” meaning that students’ “face, chin to forehead, needs to be in the camera view at all times,” having anyone speak directly to the test taker, and any “additional noises,” as well as “looking off-screen” (ProctorU, 2020). These policies appear rather incompatible with an unpredictable home environment, where one might need to tend to a pet or knock at the door, and particularly punitive during a global pandemic when schooling and childcare may not always be available. They also are likely to advantage privileged students, who may be more readily able to replicate viable test conditions (e.g. having access to a quiet and solitary place from which to complete exams). And, much like other forms of technology, can reinforce racial bias, with some users reporting that the facial recognition component of online remote proctoring has had difficulties identifying users with darker skin (Swauger, 2020a, 2020b). In many respects, these concerns undercut the argument offered by some university administrators that online remote proctoring helps ensure that remote exams are fair and equitable (Selwyn et al., 2021).

Online remote proctoring is still in its infancy, but to date, research around its effectiveness has been mixed. Some research has shown little difference between non-proctored and proctored online exam scores (Hylton et al., 2016), proctored in person and proctored remote exam scores (Lewis, 2020; Stack, 2015), and in person proctored and online non-proctored exam scores (Hollister & Berenson, 2009). Recent research focusing exclusively on online environments has demonstrated a decrease in test scores when online exams are not proctored compared to those that are remotely proctored, which might suggest to some that cheating is occurring in non-proctored testing environments (Goedl & Malla, 2020; Reisenwitz, 2020). It is also important to note that this research relies almost exclusively on the comparison of scores between tests that were completed through online remote proctoring and those that relied upon another mode. One might raise concerns about this methodology, specifically, the feasibility of isolating cheating as the cause of differences in scores between modes of delivery, coupled with the potential impact of vastly different testing environments on student performance. Fask et al. (2014) identified a number of issues with this approach, speaking specifically of comparing online and in-person proctored exams:

The attempt to detect student cheating on online versus proctored exams has been handicapped by the confounding relationships…between student performance and possible differences in the two environments with respect to the level of distractions, student comfort, technical problems related to the use of computer technology and the opportunity to have the content of exam questions clarified. (p. 111)

Other research has called attention to the potential impact of stress on online remote proctored test scores. In a study comparing proctored and non-proctored online exam scores, Alessio et al. (2017) found that students who took a proctored exam that was on lockdown (with students unable to access certain resources on the internet) scored higher than students who completed proctored exams with a video monitor, raising questions about the impact of video surveillance on test taking. Karim et al. (2014) shared concerns about video surveillance during testing, acknowledging that while it “may be marginally effective at decreasing cheating. …effects may be small and the technology may be viewed as more invasive and thus raise feelings of pressure and tension” (p. 566). Similarly, Woldeab and Brothen’s (2019) research demonstrated that anxiety had an effect on online proctored test scores, with students with diagnosed anxiety scoring lower on exams overall, but even lower if these exams were proctored remotely online. Wuthisatian (2020) also found that students scored lower on online proctored than in-person proctored exams, citing not only test anxiety, but also lack of support and technological issues.

The potential for technological issues to arise and interfere with the completion of a test has also been reported as a source of stress around computer-based testing (Medina & Castleberry, 2016). Research has indicated that the most commonly reported challenges for students, specifically those who have used and were unsatisfied with online remote proctoring, were technical issues and a resultant loss of testing time (Okada et al., 2019). Students have also identified that meeting physical test conditions (Milone et al., 2017) and more general lack of familiarity with and control during online remote proctoring (Michel, 2020) were causes of concern. Chen et al.’s (2021) student survey of online learning environments highlighted a general sense of student dis-ease around online proctoring. As they noted, “Our students express enormous concern regarding online proctoring technologies….The themes of these student responses include stress and anxiety, privacy and trust, effectiveness, and financial burdens” (Sect. 4, para. 4). Given the nascence of online proctoring, it is possible that student anxiety levels will decrease as they gain familiarity with online remote proctoring platforms (J. W. Lee, 2020). But as Eaton and Turner (2020) emphasized, there is,

an urgent need to rigorously explore what relationship, if any, exists between e-proctoring services and students’ mental health. The proliferation of e-proctoring services has escalated rapidly on a worldwide scale during the COVID-19 crisis, with little empirical evidence about what impact such services, and in particular, the phenomenon of remote surveillance under testing conditions, might be having on students’ emotional or physical well-being. (p. 38)

In the next section, I shift to common justifications provided for the adoption of online proctoring software, focusing on institutional integrity and the reliability and validity of assessments.

Audit Culture, Institutional Integrity, and Psychometric Assessments

Online proctoring is typically offered as a response to two central concerns around assessment. The first is maintaining the integrity of both the institution and the degrees it grants, securing public confidence in the quality of education universities provide (Aaron & Roche, 2013; Milliron & Sandoe, 2007). Following this line of reasoning, online remote proctoring instills confidence in a university’s ability to safeguard against academic dishonesty in the maintenance of academic integrity.

At a practical level, Canadian universities are required to adhere to certain guidelines about academic integrity in order to become and remain degree granting institutions. Though no pan-Canadian policy on academic integrity exists, in 2007, the Council of Ministers of Education established a Canadian Degree Qualifications Framework (DQF), stipulating that Bachelor’s degrees can be granted to students who have engaged in “behaviour consistent with academic integrity” (Council of Ministers of Education, 2007). A number of higher education quality assurance boards (Universities Canada, 2021) have also been established throughout the country, either by individual provinces or external agencies, which audit higher education programs for their alignment with quality assurance standards. A few of these councils have adapted the DQF (e.g. Maritime Provinces Higher Education Commission, n.d.; Ontario Universities Council on Quality Assurance, 2021; Saskatchewan Higher Education Quality Assurance, 2014), while others have developed their own frameworks, which include an element of academic integrity (Campus Alberta Quality Council, 2005). Much like face-to-face proctoring, then, online remote proctoring serves as a demonstrable measure taken by universities to prevent cheating and thus enable them to abide by the academic integrity component of quality assurance frameworks.

Audits of Canadian public universities began in the 1990s when Ontario’s auditor general took an interest in “determin[ing] whether funds [in post-secondary institutions] were being appropriately used and whether the province was receiving value for expenditure” (Baker & Miosi, 2010, p. 33). Coupled with cuts in public finding and the privatization of many facets of the post-secondary system, governments looked to eliminate so-called redundancies and streamline services, all in the name of cost-effectiveness, accountability, and transparency (Davidson-Harden & Majhanovich, 2006). Taken together, public cuts and auditing represent “‘roll back’ (i.e. deregulation) and ‘roll out’ (i.e. reregulation) processes that structure behavior utilizing varying forms of surveillance, regulation, and competition” (Aikens & Hargis, 2019, p. 24). Under the “academic capitalism” (Schulze-Cleven & Olson, 2017) of neoliberalism, university education becomes a commodity to be bought and sold within an imposed scarcity model; universities compete both for the business of students, now consumers in a free marketplace, and sponsorship of corporations, now funding bodies used to address a capital shortages. Within such systems, reputation is important, including a university’s placement in national and international university ranking systems (Lynch, 2015; OECD, 2007). Beyond ensuring institutional integrity then, considering the commodification of higher education, online remote proctoring might be perceived as a defense against the devaluing of particular university degrees in the marketplace, with cheating potentially defacing the university “brand.”

The second concern online remote proctoring is said to address is safeguarding the reliability and validity of student test scores (Draaijer et al., 2017; Rios & Liu, 2017; Weiner & Hurtz, 2017). In psychometrics, reliability centers on test design; a reliable test is one that produces scores that are “precise and free from measurement error” (Mead, 2019). Reliable tests are also consistent over time (i.e. a student should be able to receive the same score if administered at two different points in time) and across contexts (e.g. regardless of whether it is administered online or in person) (Arnold, 2012). Relatedly, validity refers to the “capacity of a test to measure what it is purported to measure” (Truijens et al., 2019, para. 1). This typically includes construct validity, or the test’s ability to measure a student’s performance in that area as well as content validity, the test’s alignment with the subject it is intended to assess, face validity, the test appearing to measure what it purports to measure, and convergent and predictive validity, being able to build on and speak to past and future test results respectively (Darling-Hammond et al., 2013; Soh, 2016). Applied to online remote proctored tests, scores are said to be unreliable if too much variation exists between tests completed in proctored and non-proctored environments and invalid if students cheat, insofar as what is demonstrated or “measured” is not understanding and/or mastery of course content, at least not according to how the test was designed.

The language of reliability and validity, however, is largely misapplied to online remote proctoring employed in post-secondary educational contexts to support classroom-based assessment. In face-to-face environments many faculty do not adhere to psychometric principles, instead designing tests by “choosing questions from a publisher’s text bank with very little, if any, background in test theory and design” (Maxwell & Gleason, 2019, p. 216); this might then, for some, raise questions about the reliability and validity of post-secondary assessments as a whole. But more importantly, devoid of a standardized measurement system, cross-contextual analyses of student test scores make little sense when students are not only assessed differently, but when variability exists between instruction and classroom environments (Markus & Borsboom, 2013; Moss, 2003). Under audit culture, a focus on measurement also often signals a desire to quantify student learning, with test scores taken up as a means for both comparing students, faculty, courses, and programs, and holding them accountable (Williamson & Piattoeva, 2019). “Assessment and evaluation,” Saunders and Blanco Ramirez (2017) explained,

increasingly involve technologies that commensurate all teaching without regard to content into a single metric, which take the curricular form of course evaluations, student exam scores, ratings and other static quantitative expressions of the necessarily creative and dynamic educational processes. (p. 397)

At the present time at least, faculty members’ academic freedom guarantees incommensurability across contexts, as faculty have the right to use their professional judgment in creating, administering, and grading subject-specific assessments within their classes (Canadian Association of University Teachers, 2015).

Psychometric approaches to assessment that value measurement are a far cry from so-called best practices in assessment in higher education (Pereira et al., 2016). In the next section, I discuss testing and assessment culture and offer a number of alternative forms of assessment to tests that might be utilized in STEM classes.

From Testing to Assessment Culture in Higher Education

Within a metrics-based culture, assessment is often reduced to testing, treated as a “discrete activity” divorced from teaching and learning. Under testing culture, content is delivered by faculty; learning is measured through high-stakes tests and exams completed in a solitary manner; and assessments are constructed and scored exclusively by professors, who may also view assessment as a form of accountability (Birenbaum, 2016; Moss, 2003). In contrast, assessment culture is student-centered, aimed at the development of metacognition around learning processes (Veenman et al., 2006). Learning is conceptualized as an ongoing process, a “constructive, cumulative, self-regulated, goal-directed, situated, collaborative and individually different process of meaning construction and knowledge building” (Hoidn, 2017, p. 3). Concordantly, the role of faculty shifts under assessment culture from administrator to guide or facilitator. Under this model, faculty offer multiple opportunities for self- and peer-assessment (Adachi et al., 2018; Ashenafi, 2017), while engaging in formative assessment and offering feedback (Hamp-Lyons, 2007).

Though definitions vary, formative assessment essentially involves producing targeted information about student performance in order to direct both learning and instruction (Yorke, 2003). Informal formative assessments might include observations and conversations with students, and more formal approaches, any type of scaffolding toward a summative task, such as commenting on a draft or allowing students to revise and resubmit an assignment based on feedback provided. Feedback is also considered an integral part of the formative process, but only when used appropriately. Within a testing culture, feedback might be employed exclusively to justify a grade awarded, an instructor-driven practice rather than a dialogue between faculty and students (Nicol, 2010). In contrast, under assessment culture, feedback is carried forward and used to inform student learning in the context of the course (often referred to as “feedforward”) (Reimann et al., 2019). Feedback provides students with rich, “high-quality” and timely information (Nicol & MacFarlane-Dick, 2006) that can be used to inform next steps. Nicol and Milligan (2006), writing specifically about feedback-rich online environments, outline a number of strategies that include providing opportunities for students to ask questions, gain clarity on what is being asked of them (e.g. using exemplars), and complete practice assessments that align with summative tasks.

This shift from testing to assessment culture, in many respects, requires not only a massive overhaul of assessment practices in higher education, but also a decreased reliance on traditional assessment tasks, like summative tests and exams, that tend to necessitate proctoring. In their place, faculty might turn to more authentic assessment tasks that require knowledge application in a new context (e.g. case studies) or link assessment to the “real-world” outside of the classroom (e.g. a semester-long multi-disciplinary project) (Frey et al., 2012; Gikandi et al., 2011). Authentic assessments are sometimes discussed alongside performance assessments (Barrett & Moore, 2011; Cummings et al., 2008), which ask students to demonstrate their understanding by performing a task (e.g. video recording the creation of a small machine for a robotics class) and project-based or problem-based learning wherein students complete a rather extensive project or solve a complicated problem (e.g. students solve the university’s garbage disposal problem in an environmental sciences course) (Barrett & Moore, 2011; J. S. Lee et al., 2014).

Examples of authentic assessment, currently in use in a wide range of STEM subjects, abound. In a statistics course, students locate examples of “real data” in the world, analyze it, and present findings to the class (Onquegbuzie & Leech, 2003). Chemistry students create concept maps based on their laboratory experience (Kaya, 2008); astronomy students construct a virtual solar system (Barab et al., 2000); and physics students produce physical representations of their learning (Boud, 2009). In any subject area, final exams might be replaced by oral exams or video responses where students demonstrate understanding by responding to questions¾a specific prompt that asks them to explain their thinking around a concept, or a more open-ended query, requiring students to articulate what they have learned throughout the course¾either individually or in conversation with another student. Faculty who may be a bit reluctant to move away from traditional forms of assessment, or who might need some time to make the requisite shift to assessment culture, can consider collaborative exams, where students work with others to respond to questions (either initially or as part of a two-step process, in the latter instance, first completing the exam individually and later with peers) (Efu, 2019), or exam wrappers, reflective questions that require students think through the relationship between exam preparation and performance based on feedback, to foster metacognition and support self-assessment (Lovett, 2013). Here, exams are framed as learning tools rather than strictly as means for demonstrating mastery. All of these assessments methods can quite easily be supported in remote/virtual environments with the assistance of video conferencing and collaborative digital platforms.

More broadly, this approach to assessment represents a rather extensive re-imagining of instruction in higher education classrooms. As assessment is integrally linked to and informed by instruction, the traditional sage on the stage approach to content delivery is, in many ways, antithetical to assessment culture, as it upholds traditional power structures while disempowering students. When student learning becomes the focus of the course, this shift can have a fundamental impact on how content is conceptualized from the perspective of faculty. Wright (2011), drawing on Weimer’s (2002) work, explained that

the need to “cover” the content of the course has lead…to a neglect of ensuring that the course objectives are being met. It has also led to erroneously equating a good course with a rigorous course, rather than a course in which students learn. (p. 93)

Embracing student-centered classrooms might necessitate re-thinking how class time is spent, leaning into active learning (Matsushita, 2018) and flipped classroom models of instruction (Reidsema et al., 2017), where students come to class to work through ideas and concepts, engage in analysis and problem solving, and have small-group discussions with peers, rather than sitting in dark lecture halls being “filled” with knowledge. In online environments, this approach might involve synchronous sessions and/or asynchronous contexts where faculty and students are provided space to do the work of learning together.

In shifting to a student-centered assessment culture that prioritizes learning, online remote proctoring software becomes largely unnecessary, both practically and conceptually. Cheating is largely confined to testing-based culture, where assessment takes place outside of the learning process and individual students are limited in the materials they can have access to during knowledge-retention tasks. When no discrete testing period exists, access-based cheating has limited application. Similarly, identity-based cheating becomes increasingly difficult in classrooms where faculty and tutorial leaders have developed relationships with students, and where assessment is a largely personal process. For example, as a faculty member, I get to know my students at the beginning of the course when they introduce themselves on Flipgrid and again through weekly synchronous discussions. Students also lead live class discussions of the readings and complete a short argumentative “paper” via video. Given such opportunities, it would be difficult for students to pose as someone else as they complete assessment tasks. Conceptually, the very language used to describe the functioning of online remote proctoring software becomes inappropriate. When assessment is student-centered, students need not be monitored as they complete assessments, but rather, supported. The Latin verb assidere, from which assessment derives its meaning, means to sit beside, suggestive of a collaborative rather than authoritative approach.

Final Thoughts and Steps Forward

At face value, online remote proctoring might appear to be a rather innocuous solution to a practical problem exacerbated by a rapid shift to remote learning during a global pandemic: How do faculty ensure that students are not cheating while taking remote tests/exams? Though research around the usefulness of online remote proctoring is largely inconclusive, relying on comparisons between drastically different testing conditions, the costs of online remote proctoring as a widespread experiment, of sorts, are being offloaded to students, both figuratively and, in some instances, literally. As has been argued in this paper, some students have reported that online remote proctoring has negatively impacted their mental health, owing to surveillance measures and the requirement to create and maintain an adequate testing environment that remains largely beyond students’ control. Financially, at a number of institutions, students are being charged approximately $30 to write each exam facilitated through one of these remote proctoring companies (Athabasca University, 2021; University of Alberta, 2021). At an institutional level, universities would do well to take pause, making data-informed decisions centered on the efficacy and effects of online remote proctoring.

In many ways, online remote proctoring has been offered as a solution to a problem that we have created, mainly, a continued reliance on poorly constructed assessments that tell us little about what students understand and that are far removed from the teaching and learning process. As I have aimed to demonstrate in this article, research around best practices for assessment in higher education has pivoted sharply away from the types of tests and exams that online remote proctoring might support. At an institutional level, rather than investing in simple yet costly solutions to address cheating, universities might finance centers for teaching and learning, led by pedagogical experts. At the University of Michigan-Dearborn, for example, after making a collective decision not to use online remote proctoring, the administration invested instead in instructional designers to aid faculty in the creation of authentic, or “people-centered,” assessments for their remote classes (Silverman et al., 2021). Though faculty are highly valued subject-area specialists, by and large, most receive little to no training around assessment and instruction (Blouin & Moss, 2015). Many need support to cultivate assessment literacy, deepening their understanding of assessment practices and the principles that underlie them (Deluca & Klinger, 2010). Devoid of any guidance on how to create a learner-centered assessment environment, well-intentioned faculty may continue to design assessments that lean into testing culture.

With cuts to higher education funding in provinces like Alberta, coupled with the implementation of performance-based funding models in Ontario, Alberta, and soon Manitoba, and uncertainty about the long-term impacts of COVID-19 on higher education, it is likely that faculty will face increasing pressure in the coming years to do more with less. With growing class sizes, myriad demands on faculty time, and a rise in reliance on contingent faculty, multiple-choice and short answer question exams that are invigilated through online remote proctoring may, for many, be the most efficient way to attain information about what students know. But at what and whose expense? Continued dependence on tools and technologies that support neoliberalism and the facilitation of audit culture within post-secondary institutions can result in a sort of complacency with reform. Shore (2008) argued that audit culture is particularly insidious in the way that it shapes subjectivities, acknowledging the “subtle and seductive manner in which managerial concepts and terminologies have become integrated into the everyday language of academia” (p. 283). One means of explicit resistance is re-affirming our commitment to universities as spaces for thoughtful scholarship, the exchange of ideas, and pursuit of knowledge, while also being careful not to romanticize the halcyon days of higher education that have traditionally valorized physically and emotionally taxing forms of testing as gatekeeping practices and rites of passage. Viewing students as learners, and not consumers, at the center of this process, might be a good place to start.

References

Aaron, L. S., & Roche, C. M. (2013). Stemming the tide of academic dishonesty in higher education: It takes a village. Journal of Educational Technology Systems, 42(2), 161–196. https://doi.org/10.2190/et.42.2.h

Adachi, C., Hong-Meng Tai, J., & Dawson, P. (2018). Academics’ perceptions of the benefits and challenges of self and peer assessment in higher education. Assessment and Evaluation in Higher Education, 43(2), 294–306. https://doi.org/10.1080/02602938.2017.1339775

Aikens, K., & Hargis, K. (2019). Policy conflicts on the move: a ‘mobilities’ case study of neoliberal postsecondary policy. Journal of Education Policy, 34(1), 22–43. https://doi.org/10.1080/02680939.2017.1405075

Alessio, H. M., Malay, N., Maurer, K., Bailer, A. J., & Rubin, B. (2017). Examining the effect of proctoring on online test scores. Online Learning Journal, 21(1). https://doi.org/10.24059/olj.v21i1.885

Arnold, S. D. (2012). Assessing student learning online: Overcoming reliability issues. IADIS International Conference on Cognition and Exploratory Learning in Digital Age, CELDA 2012, Spain, 189–196.

Ashenafi, M. M. (2017). Peer-assessment in higher education—twenty-first century practices, challenges and the way forward. Assessment and Evaluation in Higher Education, 42(2), 226–251. https://doi.org/10.1080/02602938.2015.1100711

Athabasca University. (2021). Examination services. http://registrar.athabascau.ca/exams/proctoru_fees.php

Baker, D. N., & Miosi, T. (2010). The quality assurance of degree education in Canada. Research in Comparative and International Education, 5(1), 32–57. https://doi.org/10.2304/rcie.2010.5.1.32

Barab, S., Squire, K., Hay, K., & Yamagata-Lynch, L. (2000). Virtual solar system project: Learning through a technology-rich, inquiry-based, participatory learning environment. Journal of Science Education and Technology, 97(1–4), 131–141. https://doi.org/10.1023/A

Baron, J., & Crooks, S. M. (2005). Academic integrity in web based distance education. TechTrends, 49(2), 40–45. https://doi.org/10.1007/bf02773970

Barrett, T., & Moore, S. (2011). New approaches to problem-based learning: Revitalising your practice in higher education. Routledge.

Birenbaum, M. (2016). Assessment culture versus testing culture: The impact on assessment for learning. In D. Laveault & L. Allal (Eds.), Assessment for learning: Meeting the challenge of implementation (pp. 275–292). Springer. https://doi.org/10.1007/978-3-319-39211-0_16

Blouin, D. D., & Moss, A. R. (2015). Graduate student teacher training: Still relevant (and missing?) 20 years later. Teaching Sociology, 43(2), 126–136. https://doi.org/10.1177/0092055X14565516

Boud, D. (2009). How can practice reshape assessment. In G. Joughin (Ed.), Assessment, learning and judgement in higher education (pp. 29–43). https://doi.org/10.1007/978-1-4020-8905-3

Campus Alberta Quality Council. (2005). Organizational standards. https://caqc.alberta.ca/review-process/standards/organizational-standards/

Canadian Association of University Teachers. (2015). Academic freedom in the assignment of student grades. https://www.caut.ca/about-us/caut-policy/lists/caut-policy-statements/academic-freedom-in-the-assignment-of-student-grades

Chen, C., Landa, S., Padilla, A., & Yur-Austin, J. (2021). Learners’ experience and needs in online environments: Adopting agility in teaching. Journal of Research in Innovative Teaching & Learning, 14(1), 18-31. https://doi.org/10.1108/jrit-11-2020-0073

Chin, M. (2020). Exam anxiety: How remte test-proctoring is creeping students out. The Verge. https://www.theverge.com/2020/4/29/21232777/examity-remote-test-proctoring-online-class-education

Council of Ministers of Education. (2007). Ministerial statement on quality assurance of degree education in Canada (pp. 1–14). http://www.cmec.ca/Publications/Lists/Publications/Attachments/95/QA-Statement-2007.en.pdf

Cummings, R., Maddux, C. D., & Richmond, A. (2008). Curriculum-embedded performance assessment in higher education: Maximum efficiency and minimum disruption. Assessment and Evaluation in Higher Education, 33(6), 599–605. https://doi.org/10.1080/02602930701773067

Darling-Hammond, L., Newton, S. P., & Wei, R. C. (2013). Developing and assessing beginning teacher effectiveness: The potential of performance assessments. Stanford Center for Opportunity Policy in Education. https://edpolicy.stanford.edu/sites/default/files/publications/developing-and-assessing-beginning-teacher-effectiveness-potential-performance-assessments.pdf

Davidson-Harden, A., & Majhanovich, S. (2006). Privatisation in education in Canada: A survey of trends. In J. Zajda (Ed.), Decentralisation and privatisation in education: The role of the state (pp. 31–56). Springer. https://doi.org/10.1007/978-1-4020-3358-2

Deluca, C., & Klinger, D. A. (2010). Assessment literacy development: Identifying gaps in teacher Candidates’ learning. Assessment in Education: Principles, Policy and Practice, 17(4), 419–438. https://doi.org/10.1080/0969594X.2010.516643

Draaijer, S., Jefferies, A., & Somers, G. (2017). Online proctoring for remote examination: A state of play in higher education in the EU. International Conference on Technology Enhanced Assessment, 96–108. https://link.springer.com/chapter/10.1007/978-3-319-97807-9_8

Eaton, S. E., & Turner, K. L. (2020). Exploring academic integrity and mental health during COVID-19: Rapid review. Journal of Contemporary Education, Theory & Research, 4(2), 35–41.

Efu, S. I. (2019). Exams as learning tools: A Comparison of traditional and collaborative assessment in higher education. College Teaching, 67(1), 73–83. https://doi.org/10.1080/87567555.2018.1531282

Examity. (2019). Live proctoring test-taker guide. https://www.examity.com/wp-content/uploads/2019/11/Canvas_Student_Quick_Guide.pdf

Fask, A., Englander, F., & Wang, Z. (2014). Do online exams facilitate cheating? An experiment designed to separate possible cheating from the effect of the online test taking environment. Journal of Academic Ethics, 12(2), 101–112. https://doi.org/10.1007/s10805-014-9207-1

Frey, B. B., Schmitt, V. L., & Allen, J. P. (2012). Defining authentic classroom assessment. Practical Assessment, Research and Evaluation, 17(2), 1–18.

Gikandi, J. W., Morrow, D., & Davis, N. E. (2011). Online formative assessment in higher education: A review of the literature. Computers and Education, 57(4), 2333–2351. https://doi.org/10.1016/j.compedu.2011.06.004

Goedl, P. A., & Malla, G. B. (2020). A study of grade equivalency between proctored and unproctored exams in distance education. American Journal of Distance Education, 34(4), 280–289. https://doi.org/10.1080/08923647.2020.1796376

Gurung, R. A. R., Wilhelm, T. M., & Filz, T. (2012). Optimizing honor codes for online exam administration. Ethics and Behavior, 22(2), 158–162. https://doi.org/10.1080/10508422.2011.641836

Hamp-Lyons, L. (2007). The impact of testing practices on teaching. In J. Cummins & C. Davison (Eds.), International handbook of English language teaching (pp. 487–504). Springer. https://doi.org/10.1007/978-0-387-46301-8_35

Haq, A., Jamal, A., Butt, U., Majeed, A., & Ozkaya, A. (2015). Understanding privacy concerns in online courses: A case study of proctortrack. Communications in Computer and Information Science, 534, 137–150. https://doi.org/10.1007/978-3-319-23276-8_12

Hébert, C. (2015). (Student) teaching inside the box: Stories of teaching and learning in and against the edTPA [Unpublished doctoral dissertation]. York University.

Hébert, C. (2017). What do we really know about the edTPA? Research, PACT, and packaging a local teacher performance assessment for national use. The Educational Forum, 81(1), 68–82. https://doi.org/10.1080/00131725.2016.1242680

Hoidn, S. (2017). Student-centered learning environments in higher education classrooms. Palgrave Macmillan.

Holden, O., Norris, M., & Kuhlmeier, V., (2020). Academic integrity in online testing: A research review. Frontiers in Education, 6, 1-13. https://doi.org/10.3389/feduc.2021.639814

Hollister, K. K., & Berenson, M. L. (2009). Proctored versus unproctored online exams: Studying the impact of exam environment on student performance. Decision Sciences Journal of Innovative Education, 7(1), 271–294. https://doi.org/10.1111/j.1540-4609.2008.00220.x

Hu, J. (2020). Online test proctoring claims to prevent cheating. But at what cost. Slate. https://slate.com/technology/2020/10/online-proctoring-proctoru-proctorio-cheating-research.html

Hubler, S. (2020, May 10). Keeping online testing honest? Or an Orwellian overreach? The New York Times. https://www.nytimes.com/2020/05/10/us/online-testing-cheating-universities-coronavirus.html

Hylton, K., Levy, Y., & Dringus, L. P. (2016). Utilizing webcam-based proctoring to deter misconduct in online exams. Computers and Education, 92–93, 53–63. https://doi.org/10.1016/j.compedu.2015.10.002

Karim, M. N., Kaminsky, S. E., & Behrend, T. S. (2014). Cheating, reactions, and performance in remotely proctored testing: An exploratory experimental study. Journal of Business and Psychology, 29(4), 555–572. https://doi.org/10.1007/s10869-014-9343-z

Kaya, O. N. (2008). A student-centred approach: Assessing the changes in Prospective Science Teachers’ conceptual understanding by concept mapping in a general chemistry laboratory. Research in Science Education, 38(1), 91–110. https://doi.org/10.1007/s11165-007-9048-7

King, C. G., Guyette, R. W., & Piotrowski, C. (2009). Online exams and cheating: An empirical analysis of business students’ views. Journal of Educators Online, 6(1), 1–11. https://doi.org/10.9743/JEO.2009.1.5

Langenfeld, T. (2020). Internet-based proctored assessment: Security and fairness issues. Educational Measurement: Issues and Practice, 39(3), 24–27. https://doi.org/10.1111/emip.12359

Lee, J. S., Blackwell, S., Drake, J., & Moran, K. A. (2014). Taking a leap of faith: Redefining teaching and learning in higher education through project-based learning. Interdisciplinary Journal of Problem-Based Learning, 8(2), 3–13. https://doi.org/10.7771/1541-5015.1426

Lee, J. W. (2020). Impact of proctoring environments on student performance: Online vs offline proctored exams. Journal of Asian Finance, Economics and Business, 7(8), 653–660. https://doi.org/10.13106/JAFEB.2020.VOL7.NO8.653

Lewis, S. E. (2020). Chemistry assessments through the sudden implementation of online instruction. Journal of Chemical Education, 97(9), 3418–3422. https://doi.org/10.1021/acs.jchemed.0c00697

Lovett, M. (2013). Make exams worth more than the grade. In M. Kaplan, N. Silver, D. LaVaque-Manty, & D. Mizlish (Eds.), Using reflection and metacognition to improve student learning (pp. 18–52). Stylus.

Lynch, K. (2015). Control by numbers: New managerialism and ranking in higher education. Critical Studies in Education, 56(2), 190–207. https://doi.org/10.1080/17508487.2014.949811

Maritime Provinces Higher Education Commission. (n.d.). Maritime degree level qualifications framework. Retrieved March 2, 2021, from http://www.mphec.ca/resources/Maritime_Degree_Level_Qualifications_Framework.pdf

Markus, K., & Borsboom, D. (2013). Frontiers of test validity theory: Measurement, causation and meaning. Taylor and Francis Group.

Matsushita, K. (2018). An invitation to deep active learning. In K. Matusushita (Ed.), Deep active learning: Toward greater depth in university education (pp. 15–33). Springer.

Maxwell, M., & Gleason, J. (2019). Item efficiency: An item response theory parameter with applications for improving the reliability of mathematics assessment. International Journal of Mathematical Education in Science and Technology, 50(2), 216–243. https://doi.org/10.1080/0020739X.2018.1492038

McCabe, D. L., Treviño, L. K., & Butterfield, K. D. (2001). Cheating in academic institutions: A decade of research. Ethics and Behavior, 11(3), 219–232. https://doi.org/10.1207/S15327019EB1103_2

Mead, A. (2019). Psychometric reliability: Definition, estimation, and application. In Wiley Stats Ref: Statistics Reference Online. https://doi.org/10.1002/9781118445112.stat06409.pub2

Medina, M. S., & Castleberry, A. N. (2016). Proctoring strategies for computer-based and paperbased tests. American Journal of Health-System Pharmacy, 73(5), 274–277. https://doi.org/10.2146/ajhp150678

Michel, R. S. (2020). Remotely proctored K-12 high stakes standardized testing during COVID-19: Will it last? Educational Measurement: Issues and Practice, 39(3), 28–30. https://doi.org/10.1111/emip.12364

Milliron, V., & Sandoe, K. (2007). The net generation cheating challenge. Innovate, 4(2), 97. http://www.innovateonline.info/index.php?view=article&id=499

Milone, A. S., Cortese, A. M., Balestrieri, R. L., & Pittenger, A. L. (2017). The impact of proctored online exams on the educational experience. Currents in Pharmacy Teaching and Learning, 9(1), 108–114. https://doi.org/10.1016/j.cptl.2016.08.037

Morrison, S., & Heilweil, R. (2020). How teachers are sacrificing student privacy to stop cheating. Vox. https://www.vox.com/recode/22175021/school-cheating-student-privacy-remote-learning

Moss, P. A. (2003). Reconceptualizing validity for classroom assessment. Educational Measurement: Issues and Practice, 22(4), 13–25. https://doi.org/10.1111/j.1745-3992.2003.tb00140.x

Nicol, D. (2010). From monologue to dialogue: Improving written feedback processes in mass higher education. Assessment and Evaluation in Higher Education, 35(5), 501–517. https://doi.org/10.1080/02602931003786559

Nicol, D., & MacFarlane-Dick, D. (2006). Formative assessment and selfregulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090

Nicol, D., & Milligan, C. (2006). Rethinking technology-supported assessment practices in relation to the seven principles of good feedback practice. In C. Bryan & K. Clegg (Eds.), Innovative Assessment in Higher Education (pp. 64–77). Routledge.

OECD. (2007). How do rankings impact on higher education. Programme on Institutional Management in Higher Education. https://www.oecd.org/education/imhe/39802910.pdf

Okada, A., Whitelock, D., Holmes, W., & Edwards, C. (2019). e-Authentication for online assessment: A mixed-method study. British Journal of Educational Technology, 50(2), 861–875. https://doi.org/10.1111/bjet.12608

Onquegbuzie, A., & Leech, N. (2003). Assessment in statistics courses: More than a tool for evaluation. Assessment and Evaluation in Higher Education, 28(2), 115–127.

Ontario Universities Council on Quality Assurance. (2021). Quality Assurance Framework. https://oucqa.ca/wp-content/uploads/2021/10/Quality-Assurance-Framework-Oct-2021-1.pdf

Owens, H. S. (2015). Cheating within online assessments: A comparison of cheating behaviours in proctored and unproctored environments. [Unpublished doctoral dissertation]. Mississippi State University.

Pereira, D., Flores, M. A., & Niklasson, L. (2016). Assessment revisited: A review of research in Assessment and Evaluation in Higher Education. Assessment and Evaluation in Higher Education, 41(7), 1008–1032. https://doi.org/10.1080/02602938.2015.1055233

ProctorU. (2020). What am I allowed and not allowed to do during my exam. https://support.proctoru.com/hc/en-us/articles/360043127892-What-am-I-allowed-and-not-allowed-to-do-during-my-exam-

Ranger, J., Schmidt, N., & Wolgast, A. (2020). The detection of cheating on e-exams in higher education—The performance of several old and some new indicators. Frontiers in Psychology, 11(October), 1–16. https://doi.org/10.3389/fpsyg.2020.568825

Reidsema, C., Hadgraft, R., & Kavanagh, L. (2017). Introduction to the flipped classroom. In C. Reidsema, L. Kavanagh, R. Hadgraft, & N. Smith (Eds.), The flipped classroom (pp. 3–14). Springer.

Reimann, N., Sadler, I., & Sambell, K. (2019). What’s in a word? Practices associated with ‘feedforward’ in higher education. Assessment and Evaluation in Higher Education, 44(8), 1279–1290. https://doi.org/10.1080/02602938.2019.1600655

Reisenwitz, T. H. (2020). Examining the necessity of proctoring online exams. Journal of Higher Education Theory and Practice, 20(1), 118–125. https://doi.org/10.33423/jhetp.v20i1.2782

Rios, J. A., & Liu, O. L. (2017). Online proctored versus unproctored low-stakes internet test administration: Is there differential test-taking behavior and performance? American Journal of Distance Education, 31(4), 226–241. https://doi.org/10.1080/08923647.2017.1258628

Sandlin, C. (2020). Call on the University of Regina to remove ProctorTrack. https://www.change.org/p/university-of-regina-administration-call-on-the-university-of-regina-to-remove-proctortrack

Saskatchewan Higher Education Quality Assurance. (2014). Saskatchewan Higher Education Quality Assurance Board Quality Assurance Review Process. https://publications.saskatchewan.ca/api/v1/products/92827/formats/109877/download

Saunders, D. B., & Blanco Ramírez, G. (2017). Against ‘teaching excellence’: Ideology, commodification, and enabling the neoliberalization of postsecondary education. Teaching in Higher Education, 22(4), 396–407. https://doi.org/10.1080/13562517.2017.1301913

Schulze-Cleven, T., & Olson, J. R. (2017). Worlds of higher education transformed: Toward varieties of academic capitalism. Higher Education, 73(6), 813–831. https://doi.org/10.1007/s10734-017-0123-3

Selwyn, N., O’Neill, C., Smith, G., Andrejevic, M., & Gu, X. (2021). A necessary evil? The rise of online exam proctoring in Australian universities. Media International Australia, 1–16. https://doi.org/10.1177/1329878X211005862

Shore, C. (2008). Audit culture and illiberal governance: Universities and the politics of accountability. Anthropological Theory, 8(3), 278–298.

Silverman, S., Caines, A., Casey, C., Garcia de Hurtado, B., Riviere, J., Sintjago, A., & Vecchiola, C. (2021). What happens when you close the door on remote proctoring? Moving toward authentic assessments with a people-centered approach. To Improve the Academy, 39(3), 115–132. https://doi.org/10.3998/tia.17063888.0039.308

Soh, K. (2016). Understanding test and exam results statistically: An essential guide for teachers and school leaders. Springer. https://doi.org/10.1007/978-981-10-1581-6_14

Stack, S. (2015). The impact of exam environments on student test scores in online courses. Journal of Criminal Justice Education, 26(3), 273–282. https://doi.org/10.1080/10511253.2015.1012173

Swauger, S. (2020a). Our bodies encoded: Algorithmic test proctoring in higher education. Hybrid Pedagogy. https://hybridpedagogy.org/our-bodies-encoded-algorithmic-test-proctoring-in-higher-education/

Swauger, S. (2020b). Software that monitors students during tests perpetuates inequality and violates their privacy. MIT Technology Review. https://www.technologyreview.com/2020/08/07/1006132/software-algorithms-proctoring-online-tests-ai-ethics/

Tatum, H. E., Schwartz, B. M., Hageman, M. C., & Koretke, S. L. (2018). College students’ perceptions of and responses to academic dishonesty: An investigation of type of honor code, institution size, and student–faculty ratio. Ethics and Behavior, 28(4), 302–315. https://doi.org/10.1080/10508422.2017.1331132

Truijens, F., Cornelus, S., Desmet, M., De Smet, M., & Meganck, R. (2019). Validity beyond measurement: Why psychometric validity is insufficient for valid psychotherapy research. Frontiers in Psychology, 10(532).

Universities Canada. (2021). Provincial quality assurance systems. https://www.univcan.ca/universities/quality-assurance/provincial-quality-assurance-systems/

University of Alberta. (2021). Remote proctor. https://www.ualberta.ca/information-services-and-technology/services/learning-assessment-centre/remote-proctor.html

Veenman, M. V. J., Van Hout-Wolters, B. H. A. M., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and Learning, 1(1), 3–14. https://doi.org/10.1007/s11409-006-6893-0

Verificient Technologies. (2021). Let’s get started. https://ca-testing.verificient.com/614e76646a472c210d255003/tests/test/db38a9834c1845aab0d7c686141b6178/best_practices/form/

Walsh, D. (2020). UBC must ban Proctorio, university-wide. https://www.change.org/p/the-university-of-british-columbia-ubc-must-dissociate-from-proctorio-at-the-highest-level

Weimer, M. (2002). Learner-centered teaching: Five key changes to practice. Jossey-Bass Inc.

Weiner, J. A., & Hurtz, G. M. (2017). A comparative study of online remote proctored vs. onsite proctored. Journal of Applied Testing Technology, 18(1), 13–20.

Williamson, B., & Piattoeva, N. (2019). Objectivity as standardization in data-scientific education policy, technology and governance. Learning, Media and Technology, 44(1), 64–76. https://doi.org/10.1080/17439884.2018.1556215

Woldeab, D., & Brothen, T. (2019). 21st century assessment: Online proctoring, test anxiety, and student performance. International Journal of E-Learning & Distance Education, 34(1), 1–11.

Wright, G. B. (2011). Student-centered learning in higher education. International Journal of Teaching and Learning in Higher Education, 23(3), 92–97. https://doi.org/10.4324/9781315559605

Wuthisatian, R. (2020). Student exam performance in different proctored environments: Evidence from an online economics course. International Review of Economics Education, 35, 100196. https://doi.org/10.1016/j.iree.2020.100196

Yorke, M. (2003). Formative assessment in higher education: Moves towards theory and the enhancement of pedagogic practice. Higher Education, 45(4), 477–501. https://doi.org/10.1023/A:1023967026413