Prompting Checklist for Testers

I have been learning AI in Testing through the 30 Days of AI in Testing Challenge by the Ministry of Testing. As part of this challenge, I get to engage with fellow participants. This checklist is the result of one such collaborative effort of me and one of my fellow participants and awesome tester, Joyz Ng. We collaborated on our learnings from the study on prompt engineering, prompting techniques as well as practical experience with AI LLM bots and came up with this checklist to help our fellow testers understand the art of crafting better prompts to help them with testing.

Check this and share your feedback with us in the comments section or via LinkedIn messages. Happy Prompting!

For good examples of testing prompts that use this checklist, you can refer to the AI Prompt Repository for Testers – Rahul’s Testing Titbits

General

• Use simple and complete sentences
• Define the task / instruction well
• Be specific (precision is important) on what you want to recieve
• Provide all core prompt elements – Context, Instruction, Input / References, Output Indicator
• Minimize ambiguity (avoid vague terms). Ex: Give me 10 data sets vs Give me some data sets.
• Set the desired temperature level – Creative, Precise, Balanced
• Mention tone / target audience
  • Examples:
  • Professional
  • Personal
  • Funny
  • Creative
• Mention formatting guidelines
  • Examples:
  • Bullet Pointers
  • “Double Quotes”
  • CSV
  • Tabular
• Avoid sharing sensitive / intellectual information on public LLMs

Context

• Ask it to act like a [persona]:
  • Examples:
  • Tester
  • Test Lead
  • Regular Customer
  • Developer
  • Automation Engineer, etc.
• Specify background information. Ex: Sites or people to refer to (e.g., works of Context Driven Community)
• Explain & Define your domain, application type, and system under test
• Via Images
• Via System Description
• Via Product Model
• Via Technical Description

Input & Output

• Specify What’s Expected from the LLM?
  • Examples:
  • Positive Cases Only or Diverse Cases?
  • Functional Coverage or Data Coverage?
  • Test Data or Program to Generate Test Data?
• Give Examples of Desired Format and Style Expected
  • Examples:
  • Code Snippet
  • Email Template
  • Report Structure
• Define Data Requirements
  • Examples:
  • Username – String (20 characters)
  • Age – Integer (>18 characters)
• Reference Existing Content or Link for Inspiration
• Examples:
• Coding Guideline
• Testing Mnemonics
• Writing Sample
• Ask for “Step by Step” explanation for reasoning related tasks – Chain of Thought Prompting Hack
• Give examples of desired output format – Few Shot Prompting Hack
• Ask it to double check the correctness of the answers – Time to Think Illusion Hack

LLM Models

• Provide feedback to LLM & Refine prompts (RHLF)
• Don’t stop at first response. Ask follow up questions. – Experiment, Explore, Evaluate
• Compare Results with Other AI Models’ Responses
• • Examples:
  • Gemini
  • Copilot
  • ChatGPT
  • Claude

Update Log:

• First Draft Created by Rahul Parwal & Joyz Ng on 13th March 2024
• Published on 14th March 2024
• v1.1 Updated on 19th March 2024. Updated the checklist based on first-level feedback from Maneesh.
• v2.0 Updated on 3rd August 2024. Updated based on more experiments and discussions among the creators.