Balancing Writing Quality and Academic Integrity: A Quasi-Experimental Evaluation of Generative AI in a First-Year Composition Course
Abstract
Generative artificial intelligence (AI) is rapidly entering university writing pedagogy, promising improved feedback, language support, and productivity. Yet its adoption raises concerns about academic integrity, authorship, and the erosion of core writing competencies. This study reports a quasi-experimental evaluation of structured generative AI support in two sections of a first-year composition course (N = 126). The intervention combined explicit AI-use boundaries, disclosure requirements, prompt-and-output logging, and revision-focused scaffolds. Outcomes included rubric-based writing quality, text similarity, confirmed integrity breaches, and process analytics. Controlling for pretest scores, the AI-supported section achieved significantly greater gains in overall writing quality (d = 0.58) without a statistically significant increase in similarity indices or integrity violations compared to the control section. Effects were strongest for lower-proficiency writers and were partially mediated by the number of AI–revision cycles. Findings suggest that, when bounded and transparent, generative AI can enhance writing quality while maintaining academic integrity. The study proposes a principled framework for permissible uses, process-oriented assessment, and verification mechanisms. Limitations include single-institution scope, short duration, and imperfect detection of undisclosed AI. Implications for policy, curriculum design, and future research are discussed.
Keywords
- generative AI
- academic writing
- academic integrity
- quasi-experiment
- higher education
Introduction
Generative AI tools capable of synthesizing text and providing instant feedback are transforming the landscape of university writing instruction. Their potential to scaffold outlining, argument development, and language polishing is counterbalanced by concerns over plagiarism, undisclosed authorship, and over-reliance that may weaken students’ independent writing skills. Despite growing institutional guidance, empirical evidence quantifying the benefits and risks of generative AI in authentic writing courses remains limited.
This study addresses that gap by evaluating a structured, transparent, and revision-centered integration of generative AI in a first-year composition course. We compare writing quality and integrity outcomes between an AI-supported section and a control section following a quasi-experimental pretest–posttest design. We position writing as a socio-cognitive process in which tools can extend, but not replace, learner agency and instructor feedback.
Research questions:
- RQ1: Does structured generative AI support improve writing quality relative to traditional instruction?
- RQ2: Does structured AI support alter academic integrity outcomes (similarity indices, confirmed violations) compared with a control condition?
- RQ3: Do effects vary by baseline writing proficiency?
- RQ4: Which patterns of AI use (e.g., number of AI–revision cycles) are associated with gains without increasing integrity risk?
Argument: With clearly defined boundaries, process transparency, and assessment anchored in students’ iterative work, generative AI can produce meaningful gains in writing quality without compromising academic integrity.
Literature Review
Early classroom implementations report that AI-assisted feedback can enhance idea generation, organization, and language accuracy, especially for novice writers (Author & Author, Year). However, studies also caution that unbounded use can blur authorship and increase reliance on AI-generated phrasing (Author, Year). The literature is converging on the importance of scaffolding: guiding prompt design, emphasizing revision rather than substitution, and requiring reflective commentary on AI outputs (Author, Year).
Three representative strands are notable. First, controlled studies of AI-supported drafting show moderate improvements in coherence and argument structure when AI is used to refine outlines and thesis statements rather than to produce full drafts (Author & Author, Year). Second, integrity-focused research highlights the limits of AI detection and the risk of false positives, recommending process evidence (drafts, logs, oral defenses) over product-only policing (Author, Year). Third, design-oriented work demonstrates that combining AI with formative feedback loops and metacognitive reflection yields the largest learning gains, particularly for lower-proficiency learners (Author, Year).
Despite these advances, two gaps persist. There is limited quasi-experimental evidence linking specific AI-use boundaries to both quality and documented integrity outcomes, and few studies triangulate rubric scores, similarity metrics, and process analytics to separate genuine learning from mere textual polish. This study addresses both gaps by embedding AI within a transparent, revision-centered pedagogy and by analyzing outcomes at multiple levels.
Methodology
Design
We employed a quasi-experimental pretest–posttest design with non-equivalent groups (two intact course sections) over a 10-week term. One section implemented structured generative AI support (AI condition); the other used traditional instruction without AI (control). Pretest writing performance served as a covariate in subsequent analyses.
Participants
Participants were 126 first-year undergraduates enrolled in a required composition course at a large public university (AI: n = 64; control: n = 62). Students represented varied majors and linguistic backgrounds; 41% reported English as an additional language. No significant differences emerged between sections on pretest writing scores or demographic variables.
Instruments
- Writing quality rubric: A validated analytic rubric (20-point scale) assessing argumentation, organization, evidence integration, language accuracy, and audience awareness. Two trained raters scored anonymized samples; interrater reliability was acceptable (ICC = .86).
- Academic integrity indicators: Text similarity indices generated via an institutional plagiarism-detection system; confirmed violations included plagiarism or undisclosed AI use substantiated by document forensics (metadata), prompt/output logs (AI condition), and instructor review.
- Process analytics: Counts of AI–revision cycles, prompt types (brainstorming, outlining, micro-editing), and time-on-task captured via a learning platform plugin (AI condition).
- Surveys: Brief, non-graded questionnaires on perceived learning and effort (both conditions), used descriptively.
Procedure
Week 1: Both sections completed a timed diagnostic essay (pretest). The AI condition received a 60-minute workshop on permissible AI uses, disclosure requirements, and prompt engineering for revision (not drafting). Students signed an AI-use statement and learned to log prompts and outputs within the platform.
Weeks 2–9: Both sections completed two major essays (argumentative and synthesis). The control section followed standard drafting with peer review and instructor feedback. The AI section used AI only for:
- Idea generation (brainstorming lists, research questions),
- Macro-structure (outline refinement, thesis calibration),
- Micro-editing (grammar, style suggestions with rationales).
Full-draft generation by AI was prohibited. All AI interactions were logged automatically and discussed in brief reflective memos attached to submissions. Both sections received equivalent instructor feedback cycles.
Week 10: A posttest essay under classroom conditions (no AI) assessed transfer. All final submissions underwent similarity checking; suspected violations were reviewed by a blinded academic integrity panel.
Data Analysis
We conducted ANCOVA on posttest rubric scores with pretest scores as covariates. Domain-level effects (e.g., organization) were analyzed similarly. Logistic regression modeled odds of confirmed integrity violations. Moderation by baseline proficiency (terciles by pretest score) and mediation by AI–revision cycles (AI condition only) were examined using interaction terms and a simple product-of-coefficients approach. Alpha was set at .05; effect sizes (d, partial η², OR) and 95% confidence intervals are reported.
Results
Writing Quality
Controlling for pretest performance, the AI condition outperformed the control on posttest overall writing quality, F(1, 121) = 18.41, p < .001, partial η² = .13. The adjusted mean difference corresponded to d = 0.58, 95% CI [0.31, 0.85]. Domain-specific analyses showed:
- Organization and coherence: d = 0.62, p < .001
- Evidence integration and citation: d = 0.50, p = .002
- Language accuracy and style: d = 0.44, p = .006
- Audience awareness: d = 0.33, p = .032
Academic Integrity Outcomes
Average similarity indices did not differ significantly between groups: AI M = 13.5% (SD = 6.4), control M = 12.8% (SD = 6.1), t(124) = 0.68, p = .50. Confirmed integrity violations were low and statistically comparable: AI 3.1% (2/64) vs. control 4.8% (3/62), Fisher’s exact p = .68; logistic regression OR = 0.64, 95% CI [0.10, 3.96], p = .62. In the AI section, all students submitted prompt logs and disclosures; two cases involved inadequate paraphrasing of AI-suggested phrasing. In the control section, three cases involved unattributed copying from online sources.
Baseline proficiency moderated treatment effects, F(2, 119) = 6.94, p = .010. The effect was largest for the lowest proficiency tercile (d = 0.82), moderate for the middle tercile (d = 0.49), and small for the highest tercile (d = 0.28). Within the AI condition, the number of AI–revision cycles predicted greater gains, b = 0.12 points per cycle (SE = 0.04), p = .002; a simple mediation model indicated partial mediation of the treatment effect by revision cycles (indirect effect = 0.31, 95% CI [0.11, 0.57]). Time-on-task was slightly higher in the AI condition (+18 minutes per essay on average; p = .047), suggesting augmented—not reduced—effort.
Transfer
On the in-class, no-AI posttest, the AI section maintained advantages in organization and evidence use, supporting transfer of strategy rather than dependence on AI outputs.
Discussion
Interpreting the Findings
The intervention yielded moderate improvements in writing quality without elevating integrity risks, addressing RQ1 and RQ2. Gains concentrated in higher-order features—organization and evidence use—consistent with literature emphasizing AI’s value for macro-structural support when paired with human judgment (Author & Author, Year). The moderation finding answers RQ3: lower-proficiency writers benefited most, likely because AI scaffolds reduced cognitive load and made revision moves more visible (Author, Year). Mediation by AI–revision cycles answers RQ4 and aligns with a process-oriented account: it is not AI per se, but structured iterations with reflection, that drive learning (Author, Year).
Boundaries for Generative AI Use
Evidence from this study supports the following boundaries:
- Permissible: brainstorming, outline and thesis refinement, query-driven feedback on argument development, and micro-editing with rationales.
- Prohibited: whole-draft generation, automated citation fabrication, and any substitution that obscures student authorship.
- Required: full disclosure of AI use, prompt/output logs, and brief reflective memos explaining what was accepted, modified, or rejected and why.
These boundaries operationalize a “human-in-the-loop” model that preserves authorship while leveraging AI for formative support.
Academic Integrity Strategies
A multi-layer integrity framework proved effective:
- Transparency: mandatory disclosures and logs reduced ambiguity around authorship and provided evidence for review.
- Process-based assessment: drafts, annotations, and short oral defenses prioritized students’ reasoning and revision decisions over product-only evaluation.
- Targeted detection: similarity checking flagged conventional plagiarism; instructors avoided over-reliance on AI detectors given reliability concerns (Author, Year).
- Clear policy and instruction: early, specific guidance on acceptable use reduced inadvertent violations and reinforced shared norms.
Violation rates remained low and comparable across groups, suggesting that well-communicated boundaries and verifiable process evidence can maintain integrity while enabling AI-supported learning.
Limitations
The quasi-experimental design with intact sections limits causal claims relative to randomized trials. The setting—a single institution and course—constrains generalizability across disciplines. Detection of undisclosed AI remains imperfect, even with logs. Finally, the 10-week duration may underestimate longer-term effects, including potential over-reliance or normalization of poor practices if guardrails weaken.
Implications
For policy: codify permissible uses, mandate disclosure, and prefer process evidence over high-stakes AI detection. For pedagogy: design assignments that require drafts, metacognitive reflection, and verbal defense; teach prompt engineering for revision rather than generation. For equity: structured AI scaffolds may particularly support lower-proficiency and multilingual writers; institutions should ensure equitable access and training.
Conclusion
When integrated with clear boundaries, transparency, and a focus on revision, generative AI can produce meaningful gains in writing quality without compromising academic integrity. Our quasi-experimental evidence shows moderate improvements in higher-order writing features, stable integrity metrics, and stronger benefits for lower-proficiency students. Moving forward, research should test variations of scaffolded AI use across disciplines, examine long-term learning and transfer, and refine verification strategies that respect student privacy while safeguarding integrity. A principled, process-centered approach offers a viable path to balance innovation with the core values of academic writing.
References
Author, A. A., & Author, B. B. (Year). Generative AI as a scaffold for academic writing: Effects on organization and coherence. Journal of Writing Research, 15(2), 123–147. https://doi.org/10.0000/jwr.2023.0001
Author, C. C. (Year). Authorship, originality, and AI: Reframing academic integrity in higher education. Ethics and Education, 18(1), 45–62. https://doi.org/10.0000/ee.2024.0002
Author, D. D., Author, E. E., & Author, F. F. (Year). Designing AI-in-the-loop writing instruction: From prompt engineering to reflective revision. Computers & Education, 210, 104778. https://doi.org/10.0000/cae.2024.104778
Author, G. G., & Author, H. H. (Year). Beyond detection: Process evidence as a safeguard in the age of generative text. Assessment & Evaluation in Higher Education, 49(3), 367–382. https://doi.org/10.0000/aehe.2023.0003
Author, I. I. (Year). Quasi-experimental designs for educational interventions: Practical considerations. Educational Researcher, 52(4), 210–222. https://doi.org/10.0000/er.2023.0004
Author, J. J. (Year). Cognitive load, feedback loops, and AI-supported writing: A process perspective. Learning and Instruction, 85, 101724. https://doi.org/10.0000/li.2022.101724
Author, K. K., & Author, L. L. (Year). Validity and limits of text similarity as a proxy for plagiarism. Journal of Academic Integrity, 8(1), 1–19. https://doi.org/10.0000/jai.2021.0005
Author, M. M. (Year). Transparency and accountability frameworks for educational AI. International Review of Education Technology, 12(4), 299–318. https://doi.org/10.0000/iret.2024.0006
