The COVID-19 pandemic of 2020 changed a lot all over the world. This was no different for instructors that typically taught their students in class. For Jason Hanlon, this was the first time as an instructor at CCAPS teaching CMGT 4545: Materials and Structures II. While fortunate that the course was blended, so a good portion of the work could be offered online, there was the question of how to administer the usual in-person midterm and final exams remotely.
Inspired to Mentor Jason Hanlon is a structural engineer and Business Unit Leader at Braun Intertec in their Department of Building and Structure Sciences, specializing in assessment, investigation, and testing of structures. Jason is licensed in 12 states and a designated Model Law Structural Engineer (MLSE) through NCEES. His over 23 years of experience range through the design of single-family structures, full-structural design and detailing of multistory steel-framed office buildings, site investigation of storm-damaged structures, and review and assessments of historic buildings.
“It is nice to be able to communicate and pass on some of my experience and knowledge to those [students] coming into the construction field.”
His path to teaching at CCAPS through the Construction Management (CMGT) program was driven through his love of mentoring new engineers and people seeking to work in various capacities in the construction industry, something he even does in Braun Intertec. Although the aspects of teaching in a more academic setting was quite new, it is something he has taken in great stride.
What should have been a typical first semester teaching for Jason changed around March 2020. As soon as the State of Minnesota announced a lock-down, and the University of Minnesota followed suit, Jason like many other teachers had to create new accommodations for remote teaching. While in-class sessions could be accomplished with Zoom, and many of the assignments were turned in through the Canvas course site, how would the in-person Midterm that was meant to be administered about a week away work online?
Jason talked with the CCAPS-CMGT faculty director, Peter Hilger, then got in touch with myself about bringing his exams online. The challenges for creating the exams online included these needs:
- A proctored exam to prevent cheating through student collaboration or using unauthorized methods of using notes (i.e., mobile phones, tablets, or other computers) However, students are allowed to use their own written notes and textbook during the exam.
- Variations in questions giving students a unique set of problems to solve
- Since the problems involved math and many steps to reach an answer, granting the students the ability to show their work and the steps taken since the evaluation is not necessarily based on the final answer but the process the student uses to get to the answer.
Proctorio: The Double-edged Sword
Proctorio is the remote proctoring software that is offered centrally by the University of Minnesota. Rather than using the methods of other remote proctoring services that have students sit in front of a camera watched by an individual, Proctorio uses a mix of recording the student and their screen during the exam and uses AI to evaluate instances of potential cheating. The software also contains settings preventing potential cheating methods such as using multiple screens, opening other browser windows, printing the quiz, etc. The software grants the instructor a risk score and highlights instances of potential cheating in the recording. This software was chosen by the University of Minnesota since it was considered the least invasive option in a remote teaching setting.
When the need for some form of proctoring was brought up, Proctorio seemed to fit rather well. Even so, we had to try to anticipate any problems with the software. Students are required to use a specific browser (Google Chrome), a special plugin, and have a stable internet connection for the entirety of the exam. On top of that, early on in Proctorio’s life at the University of Minnesota there have been technical issues with its integration in Canvas. Some of these issues had fixes that for the most part had been ironed out by the time we were using it for Spring 2020, but we wanted to be prepared. To make sure that our staff could accommodate any problems that might arise with the student’s technology before they had to take a high stakes exam, a practice exam which tested whether students could access and take an exam with Proctorio was provided to students. This proved to be a good method to prepare them for what would be needed when they finally had to take the exam.
Math in a Digital World
While online quizzes in Canvas are great for creating randomized pools of questions, making the second requirement easy to accommodate, the matter of having students show their work was not.
Quizzes in Canvas mainly have question types that need a narrow selection of options for the computer to evaluate including multiple choice, true/false, matching and simple fill-in-the-blanks using strings or floating point numbers. These types of questions rely on assessing a final answer which is not always conducive for instructors wanting to access the student’s process. The only option types of questions that would allow for more flexibility include essay type and file upload. For a placeholder for the answers, we decided on the essay type question so there was something for the instructor to evaluate and compare to the student’s worksheet.
When discussing paths, we discussed using the file upload method to allow for an image the student can take of their worksheet and upload during the exam. While this method could work, Jason brought up the concern that it would be unfair to have students try to figure out how to take a picture to upload while the quiz timer was counting down. Also taking pictures in front of the webcam while Proctorio was running could potentially make the AI software trigger a cheating risk event.
What was finally decided was a method suggested by former instructional designer, Mark Kayser. He previously set up proctored exams for the online Physics courses for CCAPS that have a Canvas assignment where students could upload their images tied to the quiz instance that hosts the exam questions. This assignment would only be open for the limited amount of time that the exam was offered meaning students would have enough time to take images of their worksheets and upload them. Jason mentioned that he could see when students uploaded their worksheets and get a sense they were getting to him in a timely manner.
The First Exams
While the first exams went well, there were a few hiccups. “That [midterm] was a short-fuse to pull and get that together. There were some things we learned after that. There was some feedback where we found there are students that rely on their technology for their note-taking.” The purpose of limiting technology for “using unauthorized methods of using notes”, was not to prevent students from using notes, since they were allowed to use their own notes and even the textbook, but to limit students looking out on the internet for answers.
One instance was when a student tried to connect a second screen to their computer while taking the exam and then they were not able to finish the exam because Proctorio locked them out. This was a function that we knew existed in Proctorio and the student on the landing screen should have been informed about this but the student tried to trick the system and plug the screen in after the exam started. One claim that was rather unusual: a student in the class had their tablet connected to their UMN account and while they were trying to take the exam, Proctorio “froze” them out of their use of the tablet. Jason stated “That one I could understand being very frustrated. You wouldn’t think that if you are going to be just opening up your notes on a tablet that as soon as it was logging in to a system it froze everything.” This was an event that made the particular student believe that Proctorio had unauthorized access to their personal data. The student complained about the use of Proctorio to the instructor on the grounds of privacy. This issue was resolved with offering an alternate proctoring option using Zoom.
Another problem was with the limited files that could be uploaded into a Canvas assignment. At the time Canvas would only allow a single file upload for an assignment. This would be more difficult with images. I had suggested having students ZIP their images together and upload that as a solution. That turned out to be too convoluted of a process to upload image files. It not only took too many steps for the students to create and upload the ZIP file, but also took a lot of time on Jason’s part to Un-ZIP their image files and keep them organized when evaluating, annotating feedback, and sending it back to the student.
Changes for the Now and the Future
This year, many courses are still being taught remotely. This is no different for CMGT 4545. Jason is still using the same exams but with some tweaks. Since it was realized that students were using their own technology for notes, the Proctorio restrictions on opening other browser tabs was allowed since Proctorio could record what resources they were trying to use during the exam. Also, since the first time these exams were administered, Canvas was updated to allow students to upload more than one file per assignment. This helped simplify the instructions since students didn’t have to upload their worksheet images in a zipped file. Furthermore, Jason found that Google Drive made updates where you can take pictures of documents and the software recognizes it as a document, saving it as a PDF. Students have been able to utilize this feature to upload PDFs that are easier to annotate in the Canvas SpeedGrader.
“This particular class moving forward is more of a hybrid-style, so there is going to be an in-person component regardless and therefore the need to continue to use Proctorio for this class should not be necessary. It’s been a little rough these past two semesters trying to administer all the exams on-line and labs virtually.”
While this technology proved to be effective for remote exams and teaching in an online environment, Jason is looking forward to being able to teach his course and administer them in person. The process of having to grade and annotate images then upload them online takes Jason two times longer than having to write directly on exam sheets that students have submitted in person. Jason commented, “If I were to do this in class. Students would get a 3-4 page packet and an extra worksheet they could have. I am looking at the same format for every single one of them. They get this and know what area of space to work with. But you then do it the Proctorio way, there are so many formats and it is all over the place.”
I asked Jason for any final pieces of advice on Proctorio and he had this to say: “I think it could work pretty well for essay type questions where you are not going to be expecting a wide range of formatting of answers... If you know what the deliverable format is going to be and it is consistent throughout, that will be easier from a grading and reviewing standpoint. The only other thing I would suggest is to understand the technology restrictions settings Proctorio has to offer, what each restriction means and how that is imposed on students. Once you have that fully defined, clearly communicate that information to the students prior to administering the exam.”
If you have any questions about adding Proctorio, quizzes or if you need assistance with making additions or changes, we welcome CCAPS faculty to reach out to us for assistance through the CCAPS-ATD Learning Management Assistance form.