2. Applications of Generative AI in Education

3. Limitations and Risks of Generative AI in Education

Generative AI provides opportunities such as personalised learning and content creation, but it also poses risks including misinformation, bias, and privacy concerns, which require responsible and critical use (2-5).

Reliability and Accuracy

AI often produces errors or misleading outputs because it cannot reliably distinguish truth from falsehood. In some cases, it fabricates information, known as hallucinations, including false references that appear credible but do not exist. Performance can also decline over time through a process called algorithmic drift. At present, this occurs when accuracy and reliability decrease because the system continually repeats and recycles the same information, reinforcing patterns and errors rather than adapting to new or changing data (4,5).

Bias and Fairness

AI systems are trained on vast datasets that may include stereotypes, cultural biases, or discriminatory patterns, which are often reproduced and even amplified in their outputs. There is a lack of transparency in how these systems generate responses, often described as the black box problem, which limits accountability and a clear insight into decision-making processes. Generally, AI presents information in a confident and authoritative manner, which can encourage users to accept flawed or misleading content without applying sufficient critical evaluation (2,4).

Academic Integrity

By producing work with AI that does not reflect their own understanding, students risk committing plagiarism or academic dishonesty and graduating without genuine expertise or professional competence. In fact, such excessive reliance contradicts university pedagogical principles, as it weakens critical thinking, problem-solving, and independent learning skills (6,7).

Data Privacy and Compliance

AI systems often process sensitive personal or research data, raising risks of breaches, surveillance, or misuse. Involving third party vendors, which are external organisations that store, manage, or analyse data on behalf of an institution, introduces additional risks of interception, unauthorised access, or exposure outside the institution’s direct control. Many AI systems are trained on copyrighted material without permission, which raises questions about intellectual property, unclear ownership of outputs, and potential legal disputes. Furthermore, accessibility and equality requirements must be addressed to ensure full compliance with General Data Protection Regulation (GDPR) and Data Protection Act (DPA) obligations (3,5).

Ethical and Social Concerns

AI outputs may conflict with institutional values or misalign with ethical standards. In particular, overuse of AI, where automated systems are relied upon more than human judgment, can restrict academic freedom and discourage independent perspectives. This dependence may also reduce creativity and undermine professional expertise by replacing original thought and specialist knowledge with formulaic outputs. Staff–student relationships may also decline if AI is seen as replacing human roles, reducing trust, mentorship, and personal interaction. As a result, students may feel less supported or valued, while staff may experience a loss of professional identity and authority, ultimately weakening the sense of academic community (3,8).

Business and Reputational Risk

Failure to implement responsible AI innovation may compromise a university’s reputation and diminish its competitive position, while inadequate governance or misuse can erode institutional trust and credibility. Universities that adopt inconsistent or poorly defined AI policies risk exposure to privacy breaches, intellectual property disputes, and heightened public scrutiny (9).

4. Designing AI Resilient Assessments

Assessments most vulnerable to AI are formulaic, decontextualised, and focused on reproducing surface level knowledge. These are precisely the tasks that AI is designed to perform. The aim is not to create assessments that AI cannot complete, since this threshold continually shifts, but to emphasise human judgement, lived experience, contextual understanding, ethical reasoning, and creativity (10,11,12).

AI resilient assessment design combines selective invigilation, live components, contextualised and process-based tasks, collaboration, ethical judgment, experiential learning, and guided AI use to promote integrity, critical thinking, and authentic engagement (6–8).

Invigilated and On-Campus Components

• Incorporate supervised or invigilated elements such as in-person exams, vivas, or practical demonstrations to confirm authorship and understanding.
• Use theOn-Campus Sign-Off Method or short oral defences to verify student ownership of submitted work.

Contextual and Localised Assessment Tasks

• Design assignments that draw on module-specific readings, seminars, or case studies.
• Require reference to local data, examples, or institutional contexts that AI systems are less able to replicate accurately.

Interlinked and Developmental Assessment

• Create portfolios or linked assessments across a module or programme to assess coherence and sustained learning.
• Ask students to provide annotated drafts, research notes, or reflective commentaries that demonstrate process and progression.

Higher-Order Thinking and Critical Engagement

• Emphasise skills such as analysis, synthesis, and evaluation in both task design and marking rubrics.
• Use criteria that reward original argumentation, integration of theory, and engagement with course materials, reducing the grade potential of generic AI outputs.

Authentic and Scenario-Based Tasks

• Frame assignments around realistic case studies, simulations, or applied problems that include specific constraints (defined time limits, limited data sets, local examples, or scenario-based conditions).
• Ask students to adapt content for different audiences or purposes to demonstrate audience awareness and critical reflection.
• Include opportunities for personal insight or experience, connected to academic evidence and theoretical frameworks.

Incorporation of AI within Assessment

• Allow or require guided AI use under a traffic light system of “Amber” or “Green” conditions, where students critically evaluate AI outputs or demonstrate effective prompting.
• Tasks may include comparing AI and human writing, improving AI-generated drafts, or reflecting on the ethical implications of AI use.

Programme-Level and Institutional Approaches

• Embed AI resilience across entire programmes through a balanced mix of supervised, process-based, and long-form assessments.
• Review assessment structures regularly with educational developers to ensure alignment with emerging AI capabilities and institutional policies.
• Provide staff development through workshops and communities of practice on authentic assessment design and AI literacy.
• Engage students in discussions and learning activities that clarify acceptable AI use, fostering transparency and shared responsibility for academic integrity

5. Pen & Paper Tests

Some universities are turning back to using handwritten, invigilated assessments to ensure authenticity and prevent student reliance on AI. For example:

The University of Liverpool emphasises that assessments must “never replace original thought, independent research, and the production of original work,” recommending traditional, invigilated, handwritten formats as part of a wider strategy to safeguard authenticity and uphold academic integrity (14).

Newcastle University recommends using assessment formats that “are less vulnerable to AI misuse, such as invigilated, in-person pen and paper exams, to preserve academic integrity in a world where generative AI is widely accessible” (15).

7. The Dos and Don’ts of using AI at university

This infographic outlines key dos and don’ts for using AI responsibly in academic work. Presenting the information visually makes it easier to understand, compare, and remember essential points on transparency, critical thinking, and ethical AI use.