Testing Your Assignments with AI

Author: G. Alex Ambrose

Do you know how well AI can complete your assignments? This post explores practical steps for educators to test AI tools on real assignments to uncover their strengths and weaknesses. Many educators may have already experimented with testing assignments using generative AI, while others might not have had the chance yet or might feel uncertain or even apprehensive about using AI. Regardless of your experience level, it's crucial to recognize that AI tools are advancing rapidly, with new models emerging every few months. These tools offer great potential for improving assignments and understanding student use of AI.

Steps to Test AI on Your Assignments

To evaluate AI’s performance with your course assessments, follow these steps:

1. Select an Assignment: Take an existing paper or project assignment from your course.
2. Test with AI: Use AI to complete the assignment and evaluate its performance to understand its current capabilities and limitations.
3. Compare Platforms: Repeat the process on a second large language model (LLM) platform to compare output quality and user experience. See below for starting points of different tools.

Which Tools? Understanding the Leading AI Platforms

Different AI platforms can produce vastly different results for the same task. It's beneficial to focus on the major companies behind these generative AI tools:

• OpenAI (partnered with Microsoft): Using GPT-4 for premium users and GPT-3.5 for free users.
• Google: Employing Gemini.
• Meta (formerly Facebook): Utilizing the open-source LLaMA.
• Anthropic (heavily funded by Amazon): Featuring Claude.

We've provided login links and account requirements to access these free versions. For example, Microsoft Copilot offers limited GPT-4 access without a premium subscription, and using your non-personal Notre Dame Gmail can grant access to Google's Gemini.

*Note: The table has five columns. If you don’t see all five, please scroll to the right. You can also view the table in Google Sheets.

Current, Biggest, and Free GenAI Chatbots  (as of August 2024)
Company	OpenAI (Microsoft)		Google	Meta (Facebook)	Anthropic (Amazon)
LLM	GPT-4o, 3.5	GPT-4	Gemini 1.5 Flash	Llama 3.1	Claude 3.5 Sonnet
Chatbot Name	ChatGPT 3.5 GPT-4o	Copilot (Formerly Bing Chat)	Gemini (Formerly Bard & Google Assistant)	MetaAI	Claude
Login Link	https://chatgpt.com/	https://copilot.microsoft.com/	https://gemini.google.com/	https://www.meta.ai/	https://claude.ai/chats
Email/Account Needed	ChatGPT 3.5 = No Login GPT-4o = Any email	Free limited version;  Microsoft Outlook or Hotmail	(Non-ND) Google account	Facebook, Instagram, or WhatsApp account	Any email
Live Internet	ChatGPT 3.5 = No GPT-4o = Yes	Yes	Yes	No	No
App	ChatGPT	Microsoft Copilot AI	Gemini	Direct Message in WhatsApp, Instagram, or Facebook Messenger	No
Text to Image	ChatGPT-4o: DALL-E	DALL-E	Imagen 3 (no people)	Emu	No

Practical Applications and Examples

In practice, it's useful to see how AI performs across different disciplines. Here are two examples:

Humanities Example: Professor Ted Beatty, who teaches history and global affairs, experimented with AI in his undergraduate World Economic History course. He uploaded a research paper assignment into Claude and carefully reviewed the AI's response. This exercise led him to realize that he wanted to place greater emphasis on the assignment's purpose and grading, specifically on how well the paper engaged in a conversation between the two articles within the context of the class discussions. Additionally, he recognized the critical importance of including a formative checkpoint with a preliminary assignment to better guide students through this process.

STEM Example: In our physics department, Professor Kevin Lannon conducted an insightful experiment using GPT-4-turbo. He uploaded a picture of an introductory physics problem and prompted the AI with, "Can you answer this?" The AI responded with a detailed, step-by-step solution, showcasing a significant improvement over GPT-3.5. This experiment highlighted the potential for students to use AI to circumvent the learning process, which underscored the necessity of integrating additional formative quizzes. These quizzes ensure that students develop independent problem-solving skills rather than relying too heavily on AI assistance.

Evaluating AI’s Performance

Here are some key questions to ask when evaluating AI performance:

• Comprehension: Did the AI understand the assignment prompt correctly?
• Critical Thinking: Does the AI demonstrate critical thinking and analysis, or is the response superficial?
• Creativity: Did the AI provide unique insights or solutions, or was the output generic?
• Accuracy: Are the facts and information provided by the AI accurate and reliable?
• Relevance: Did the AI generate content that is relevant to the assignment topic?
• Depth: Does the AI’s response show a deep understanding of the subject matter?
• Tone: Is the tone of the AI’s output appropriate for the assignment?
• Technical Proficiency: Does the AI handle technical terms and concepts accurately?
• Formatting: Is the AI’s output formatted correctly according to the assignment guidelines?
• Citation: Did the AI properly cite sources and references where necessary?
• Limitations: What limitations or weaknesses are evident in the AI’s response?
• Future Use: How can the insights from the AI’s response be used to improve future assignments and teaching methods?

Conclusion

Testing AI on your assignments provides valuable insights into its capabilities and limitations. By evaluating AI tools, you can improve your assignment design and maintain academic integrity. Remember, AI is constantly evolving. Keep experimenting and adapting to stay ahead.

Next Steps

• Try out the suggested chatbots.
• Apply the evaluation criteria to one of your course assignments
• Share your findings with colleagues.
• Get support from the Kaneb Center for Teaching Excellence

Related Resources:

• Watch the video companion to this article
• Access the slide deck companion