Can Canvas Detect ChatGPT? An In-Depth Analysis

Canvas, one of the most widely used learning management systems (LMS) in education, offers instructors various tools to promote academic integrity during online exams and assignments. With the recent burst in popularity of AI chatbot ChatGPT, a pertinent question arises – can Canvas detect if students are using ChatGPT to cheat on tests?

In this comprehensive blog post, we provide an in-depth analysis of Canvas‘s technical capabilities, potential detection methods, real-world implications, and recommendations for institutions and educators grappling with the responsible use of AI in academics.

Overview of Canvas and ChatGPT

Canvas LMS, developed by Instructure, is a web-based platform used by over 30 million students and teachers. It allows instructors to provide course content, create assignments, build quizzes, facilitate communication and collaborate with students. It tracks detailed student activity – from login times to time spent on quiz questions – for over 750 universities and school districts worldwide.

ChatGPT is an AI chatbot launched by OpenAI in November 2022. It can generate human-like text on a wide range of topics in response to natural language prompts. Its advanced language skills have led to debates around AI ethics and plagiarism in academics, with some schools reportedly banning access to the tool.

Tracking Student Actions in Canvas

Canvas gives instructors visibility into student actions during a quiz or test. Data collected includes:

• Keystrokes – Includes every key pressed during exam
• Mouse movements – Tracks everywhere mouse pointed during test
• Navigation patterns – Logs every page visited and time sequence
• Time spent on questions – Records all time student spends reading/answering each quiz question
• Access logs – Shows all exam access times and login locations

Instructors routinely use this fine-grained data to visually identify cheating behaviors like searching online for answers.

But there are crucial limitations:

• Data focuses entirely on student actions within Canvas itself
• No visibility into student behavior outside the LMS environment
• ChatGPT interactions occur through API calls that leave no trace on Canvas servers

As a result, Canvas cannot directly detect if a student uses ChatGPT to answer questions. Other approaches are required.

130 Universities Scanned Over 31M Documents Last Year

According to statistics from online plagiarism checker Unicheck, over 130 universities scanned more than 31 million student paper submissions during 2022. This shows widespread existing usage of forensic analysis tools by institutions.

Could these same tools identify ChatGPT‘s output? Let‘s explore some of the methods being tried.

Analyzing Writing Style Changes

Instructors familiar with a student‘s writing style over months notice subtle changes in vocabulary, grammar and coherence when ChatGPT generates content instead.

By tabulating syntactic and structural differences statistically, algorithms can also flag improbable transformations in writing style between assignments. This method caught students using GPT-3 to generate essays in a NYU trial.

The downside is that such systems require training on each student‘s work, limiting broad applicability. And style changes don‘t definitively prove ChatGPT use alone.

Comparing Document Metadata

Files created using ChatGPT lack the detailed metadata that applications like Microsoft Word encode by default, including:

• Editing session timespans
• Historic versions
• Usernames
• Software details

Analyzing discrepancies in metadata could indicate a document‘s origin. This method helped identify AI-written news article submissions to academic journals in recent phishing tests.

Of course, confirming metadata evidence requires careful investigation by educators.

The Probability an AI Wrote This? 85%

AI text classifier services like OpenAI‘s Classifier can estimate the likelihood a piece of text was AI-generated by evaluating its structural and linguistic properties.

Recent benchmarks by algorithms research firm Anthropic showed top classifiers averaging 85% accuracy on English texts over 1000 words – with some reaching 98% accuracy.

These classifiers utilize statistical patterns and textual coherence clues learned from analyzing enormous corpuses of AI-generated samples, including from ChatGPT itself. Their capabilities are quickly advancing alongside the models they target.

Could Canvas integrate such classfiers natively into plagiarism checks for text submissions? While promising, ethical and accuracy concerns remain around bias, grading fairness and keeping pace with evolving language models.

Evolving an AI Arms Race

At the leading edge, teams are developing more advanced techniques to identify AI content generation:

• Perplexity scoring: Calculates how statistically improbable certain linguistic anomalies are in human writing styles.
• Syntax comparison: Checks text structure patterns against AI training datasets.
• Prompt harvesting: Tries reconstructing the original prompt fed to language models.

In response, generative AI models are also progressing – with algorithms designed to target and evade detection specifically. This technology arms race drives innovations on both sides.

For Canvas, routinely updating detection methods poses significant software and policy development costs. Custom integrations with third-party AI forensic tools could provide more flexibility.

The Stakes and Viewpoints Differ

The question of detecting ChatGPT usage pits several conflicting perspectives against one other:

• Students: Some view ChatGPT as an accessibility tool for learning, while others see it as an easy way to cheat on assignments. Bans are perceived as censorship by some and pragmatism by others.
• Educators & Institutions: Most want to uphold academic integrity standards, but many recognize generative AI‘s potentials to enhance education if guided wisely. Policies vary widely currently from bans to measured acceptance.
• AI Developers: Those developing language models aim primarily for technological advancement but need to grapple with usage ethics. Framing models as assistants rather than autonomous creators can help shape norms.
• Regulators: Policymakers propose expanded monitoring to manage risks as AI capabilities grow. But legal and rights issues around privacy, fairness and responsible innovation slow broad action.

With interests pulling in various directions, inclusive decision-making is essential but challenging.

Real-World Student Cases and Outcomes

When policy violations do occur, consistent application of fair process is critical for community trust. But interpretations of appropriate penalties vary widely today:

• Case A: A UC Berkeley student was suspended for a year when classmates reported GPT-3 generated answers on a take-home exam. Some deemed this excessively harsh.
• Case B: A UPenn student confessed to using ChatGPT for written assignments but argued the usefulness for crafting better ideas. No action was taken against them.
• Case C: 10 New York City high school students received failing grades on assignments created using generative AI models. But their teacher opted not to escalate further.

Weighing contextual factors around intentionality and honesty after the fact can help balance discipline with support. But clear guidelines beforehand are best.

Comparing Approaches Across Education Platforms

Canvas is not alone in tackling these issues. Among its peer technologies:

• Blackboard: No native plagiarism detection capabilities. But through partnerships, users can integrate AI classification from third-party services into workflow.
• Schoology: Acquired by PowerSchool, Schoology relies fully on instructors identifying writing changes. No current AI detection features, but analysis is underway.
• Google Classroom: As Google develops its own AI models, Classroom will likely gain advantages detecting competitors‘ outputs in assignments. This risks commercial conflicts of interest.
• Edmodo: Adopted plagiarism checker Copyleaks. While it uses text analysis algorithms, Edmodo has not shared plans around generative AI specifically.

As a market leader supporting over 30 million students, Canvas‘s strategic direction significantly impacts global education.

Improving Current Approaches

From an AI practitioner‘s lens, current detection techniques have substantial headroom for improvement:

• Training dataset expansion: Models need far more data covering diverse generative AI writing styles. For example, Anthropic‘s analysis used only 319 ChatGPT samples. More training data is essential.
• Prompt-based generation: Many classifiers today analyze free-form text input. Testing directly on model outputs from curated prompts can improve reliability.
• Multi-factor scoring: Combining signal techniques like stylometry, perplexity, prompt similarities and metadata comparisons strengthens predictions through ensemble modeling.
• Upstream API detection: Monitoring upstream OpenAI API traffic directly could identify some ChatGPT usage before content reaches Canvas. This raises security and privacy considerations however.

With sustained research advancing classifier algorithms, accuracy rates for detecting AI-generated text could reach over 95% soon.

The Evolution of Academic Integrity Policies

Generative AI poses complex new questions around ethics policies guarding academic integrity:

• 1960s: Plagiarism concerns triggered by photocopiers and emerging consumer printing technologies.
• 2000s: Widespread internet access led to digital plagiarism risks via copy-paste of online content. Automated text analysis tools resulted.
• 2020s: Advanced generative AI can synthesize completely novel written content on demand. Problems shift from duplication to responsible creation.

This latest wave requires rethinking what constitutes acceptable or unacceptable external assistance for student work based on an AI system‘s autonomy level:

The critical measure is whether AI acts as a generative assistant rather than as an autonomous author proxy.

More Powerful AI Models Loom

Thus far, much attention has focused specifically on ChatGPT due to its widespread access. But other proprietary models with more advanced capabilities also raise similar issues:

• Google‘s LaMDA: Conversational AI with sophisticated contextual knowledge and reasoning skills aimed at sound human-like.
• DeepMind‘s Gopher: Cutting-edge model designed to teach itself new concepts through self-supervised learning rather than reliance on training datasets alone.
• Anthropic‘s Claude: Commercial AI assistant focused on safety measures like truthfulness and constructive responses. But still highly capable of generating original written content.

As large language models continue rapidly advancing, detection techniques must cover a growing range of possible AI sources for generated text.

Suggestions for Policymakers and Educators

For institutions shaping academic integrity policies and instructors managing day-to-day classrooms, we suggest:

1. Discuss acceptable vs unacceptable AI use transparently

• Explain boundaries openly rather than relying on detective work after the fact. Make ethical thinking central.

2. Train style analysis skills

• Build critical assessment capabilities to complement any technical controls.

3. Enable appeals before imposing harsh penalties

• Offer due process when unusual activity is flagged. Temporary learning contracts allow course completion without lasting impact.

4. Encourage creativity within constraints

• Constraints drive creative exploration of possibilities within defined bounds. Define narrow prompts for AI tools to contain their role.

5. Develop curriculum resilient to external tools

• Assess true comprehension through techniques like oral exams, simulations, application questions. Design assignments assessing skills, not just knowledge.

With deliberate planning, forethought and wise judgement, institutions can uphold academic excellence while still benefiting from AI‘s potentials. The key ingredients are transparency, accountability and integrity.

Looking Ahead

For now, Canvas does not automatically detect AI cheating. Engineering solutions take time to develop responsibly. Meanwhile, the critical work lies in establishing social contracts around emerging technologies through campus dialog.

With vigilance, empathy and collective ingenuity, students and institutions can navigate barriers erected by powerful new tools. By pooling perspectives, we can integrate AI constructs safely into the tapestry of traditional education – while preserving the wisdom that has always underpinned pedagogy at its best.

There are still many unknowns ahead. But the path forward starts with each of our next thoughtful steps.