06/12/2023

Brave New World?

Written by Kane Murdoch

(Editor's Note: During UC San Diego’s Virtual Symposium on “The Threat & Opportunities of Artificial Intelligence and Contract Cheating: Charting a Teaching & Learning Path Forward”, Kane Murdoch gave a talk on Detecting Contract Cheating. This blog post is a follow-up to that talk.)

________________________

Amidst the uproar and shock of ChatGPT and similar Generative Artificial Intelligence (GenAI) tools, it's easy to forget that while there is certainly a brave new world ahead of us, it is not yet here in some ways. GenAI has not so much supplanted all of the existing forms of academic misconduct, as supercharged them. In fact, GenAI makes contract cheating services even cheaper for both providers and students. That is, contract cheating remains a very serious problem for higher education.

So, we're left with not a singular problem, but multiple simultaneous problems. Truth be told, this is one of those times where short-term academic remedies won't be as useful as we would like. As I see it, rather than twisting our underwear in the absence of a better approach, we should be breaking up our challenges into short-to-medium and long term, and planning the steps we can take now, and later, as the case may be.

To begin, then, what core short-to-medium term problems does contract cheating, and now GenAI, pose for assessment and academic integrity? Our primary problem has existed for a long time now - we assess products (e.g., essays) and not processes (e.g., the thinking behind the essay or the writing that produced it). Why is this a problem? We simply do not know the provenance of a disturbingly large proportion of those products. Did the student actually complete the assessment or are they merely handing in a product that someone (or something) produced for them?

When processes are observed, they are usually observed by the course instructor, but this has limitations. Take oral exams, for example. Although some faculty have found some successes in this approach, holding an oral exam for every student may not be tenable in a modular subject-based degree structure. If academics and staff work together, though, we can do something as a team. There are relatively cheap and very effective steps that we can take to gain some visibility of the process students use to create products, thereby giving us a method for assessing the integrity of that process. I'm going to provide two very powerful examples of scalable techniques to identify where students are not learning, ones that my team and I use every day, to assess whether students are doing their "own work", as the slogan goes.

Analysing Document Metadata

As you may know, Word documents, excel sheets, and pdfs contain some information about when the document was created, by whom, and any user who may have edited the document. So, when we think about assessable products, such as essays and the like, we should reasonably be able to view patterns of authorship. Like most of us, students follow their own patterns of behaviour, typically using a relatively small number of devices to create their work. Viewing the provenance of documents across a whole set of submissions becomes significantly easier when you have the whole submission history. Turnitin offers a product that can do this (no, I have no relationship with Turnitin), but you can also do this yourself by downloading all of a student's assignments and looking at the info tab of the files. Everybody has to start somewhere, and that's where I started.

Along with the user who authored ("created") a document, and the user who "last modified" a document, you can also find what software was used, what language that software had as a default, and other pieces of information to build a picture of who was responsible for producing these assessment items.

However, if we want to be more confident of the integrity of the process, additional information would be helpful. This brings us to the second scalable technique.

Interrogating LMS (aka VLE) Logs

Your LMS (or VLE depending on where you are) is a particular kind of website. And like most websites, it logs who is on it, when, and what they're doing, and from where they are doing it. By analysing these logs, universities can gain insight into how students learn. This is basically learning analytics. However, I’m suggesting we can also use these logs for "non-learning analytics"; that is, as a source for information that may indicate an enrolled student has not been learning and is not engaging in the process through assessment.

As anyone who has observed contract cheating up close may know, the 1-1 model of students having an essay written for them by a writer is only one form of cheating. Unfortunately, organised contract cheating has scaled up to involve teams of people carrying out work for many students. Many. And it's vastly more profitable to serve many students in the same subject. So large subjects, such as electives, often become riddled with contract cheating workers acting as students. But this can be visible to those of us in educational institutions. When students are connected through IP addresses to many other students, when they login to the LMS directly from multiple countries in a semester, for example, all of this is visible. There are a range of behaviours which can be searched and found in logs, if only we start looking.

Summary

The goal here is not to catch students cheating, but to better support them. The business model of contract cheating threatens students as well as institutions, and we can only break their business model when we start breaking their profitability. Sadly, as I mentioned above, the patterns clearly visible in data become all the more apparent as the course of study progresses. It is, in truth, quite easy to detect students cheating across the course of a degree. The most important final juncture, to my view, is that a student who has not demonstrated their learning should not be able to graduate with a degree from our institutions. Aside from the obvious reputational risks we run when we graduate a student who is not who we say they are (knowledgeable, skilled, etc) there are further risks that we run. Firstly, the risk that other students see cheating as a perfectly valid approach to gaining a qualification. This is the stage we are at now. Thousands of students have graduated, and are currently completing their studies, without nearly the learning we expect them to have. Cheating is extremely widespread, and largely unchecked. Secondly, and most importantly, we run the far greater risk that our society will stop seeing the value in what we do. They will stop seeing the value in education. If that happens, nothing will save us.

Thank you for being a member of ICAI. Not a member of ICAI yet? Check out the benefits of membership and consider joining us by visiting our membership page. Be part of something great!