Hey, Testers!
AI in Testing is the latest buzzword and one of the most hyped topics in the tech world today. Understanding the potential benefits and being mindful of the risks associated with using AI, particularly Generative AI, in testing is crucial. In this compilation, you’ll find a curated list of materials, references, and resources on AI in Testing. It’s important to note that this is distinct from Testing in AI, so don’t get the two confused. Let’s dive in!
Foundations
Generative AI
LLMs
- Tiktokenizer – Calculates the number of tokens for a given prompt.
- LLM Visualization
- decanting the web for the finest text data at scale
- HuggingFaceFW/fineweb-edu · Datasets at Hugging Face
- All Large Language Models Directory – All LLMs
Prompting
Prompting Hubs
Playgrounds
- Create Prompt | Google AI Studio – Prompting Studio
- Vertex AI – Prompt Optimization & Refinement
- AICamp – Mix LLMs & prompting under one camp
- LM Studio – Discover, download, and run local LLMs
- GitHub – x1xhlol/system-prompts-and-models-of-ai-tools
Gen AI in Testing
Basics
- 30 Days of AI in Testing | Ministry of Testing
- AI in Testing Roadmap: A Learning Pathway
- Prompting Checklist for Testers – Rahul’s Testing Titbits
- WAIT (Workshop on AI in Testing) – Satisfice, Inc.
- What Skills Are Required for an AI-Assisted Testing Team?
- Incorporating AI in Testing at Work
- Prompt Engineering Guide
- Prompt Engineering for Testers: Unleashing the Power of LLMs
- Software Testing with Generative AI by Mark Winteringham
- Overview of prompting strategies | Generative AI | Google Cloud
- OpenAI Academy
Critical thinking
- Appendix: ChatGPT Sucks at Being a Testing Expert
- A Super-Quick Guide to Evaluating “AI” Claims
- The fine print of using AI tools in testing – AI Disclaimer
- It can’t be that stupid, you must be prompting wrong
- A Reply to “Running a crowd-sourced experiment on using LLMs for testing” – DevelopSense
- The LLMentalist Effect
- Shining a Light on “Shadow Prompting”
- James Bach Shares His Opinion on #SoftwareTesting and #ChatGPT
- Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”
- Who answers it better – ChatGPT vs StackOverflow
- The 70% problem: Hard truths about AI-assisted coding
Follow these
Experts
Tools
AI in Testing Tools
- AI Testing Tools – Compilation
- TestCraft – AI-Powered Testing Assistant
- Joe Bot AI | Test Guild – Automation Testing Tools Community
- TestHatch – Test Generator Tool
- Code Tests for Busy Devs | CodiumAI
- Kusho AI – Transform your API specs into exhaustive tests using AI
- The multimodal synthetic data platform for developers — Gretel.ai
AI in Testing Tools – II
- Use AI-powered prompt writing tools | Generative AI | Google Cloud
- GitHub – PromptPro – App designed to help you enhance your prompt outputs
- ChatGPT – PromptGPT – Convert task into organized Markdown-based prompt.
- ChatHub – All-in-one chatbot client – Compare responses of multiple LLMs.
- Future Tools – Find The Exact AI Tool For Your Needs
Update Log:
- 28 July 2024: First version released. This resource directory will continue to grow with new updates and resources coming up over time.
- Sept 2024: Added tools
- Oct 2024: Added section “How AI & LLMs Work”
- 06 Dec 2024: Added link to “All LLM Directory” under “How AI & LLMs” Work Section
- 07 Dec 2024: Added gpt-tokenizer playground to “How AI & LLMs” Work. Also added the Future Tools platform for AI in Testing Tools – III section.
- 29 March 2025: Refactored design, added sections like “Foundations”, “Advanced”, etc.
Enjoyed this post? Here’s what you can do next:
- 📢 Share this post | Twitter | LinkedIn | WhatsApp
- 📚 Checkout my free testing checklists
- 📖 Buy a copy of Ultimate Productivity Toolkit
- 🎓 Book a call with me on Topmate
Thank you for reading! 😊