AI model selection for testers

A model to guide software testers in selecting the right LLM based on task complexity and output quality needs.

AI models come in many forms. Some are free. Some are fast. Others provide a depth of insight that testers find essential. When you are beginning to explore AI tools, the choices can feel overwhelming.

AI can speed up tasks, generate insights, and offer creative solutions. Yet, it is not magic. Testers must choose wisely. They need to know which model fits each task best.

This framework divides tasks into two factors:

• Reasoning depth: How much thought does the task require?
• Output quality: How refined must the final result be?

The two key factors

1. Reasoning depth
   Some tasks need minimal thought. Others demand deep analysis.
   • Writing basic test data needs little thought.
   • Deciphering a tricky bug takes careful reasoning.
2. Output quality
   Some results can be rough. Others must be polished.
   • An idea dump does not need perfection.
   • A risk analysis report must be sharp and clear.

These two dimensions guide your model selection.

The four quadrants explained

1. When tasks are simple (low reasoning, low quality)

These are routine tasks. AI can handle them quickly. You save time for more complex work.

Examples:

• Generating test data.
• Drafting basic test cases.

Tools to use:

• Llama 2.
• Claude (free).
• Mistral.

Tips:

• Let AI do the groundwork.
• Use the output as a starting point.
• Always check the basics.

2. When the task needs ideas but not perfection (high reasoning, low quality)

Here, you want an AI to think along with you. You need a brainstorming partner, not a finished product.

Examples:

• Exploring test ideas.
• Debugging an unusual issue.
• Analyzing a vague bug report.

Tools to use:

• Claude (free).
• ChatGPT (free).

Tips:

• Ask it to “think step by step.”
• Use the ideas as a foundation.
• Tweak and refine later.

3. When precision is key but logic is simple (low reasoning, high quality)

For tasks that require neat, structured output, quality matters more than deep thought.

Examples:

• Writing automation code.
• Creating detailed checklists.
• Crafting test charters.

Tools to use:

• GPT-4 Turbo (Pro).
• Claude Pro.
• Gemini Advanced.
• Cohere or Anthropic (paid).

Tips:

• Provide clear formatting instructions.
• Review and polish the output.
• Use it as a solid draft.

4. For the toughest tasks (high reasoning, high quality)

This zone is for complex, critical tasks. These require both insight and impeccable presentation.

Examples:

• Conducting risk analysis.
• Performing root cause investigations.
• Designing comprehensive test strategies.

Tools to use:

• GPT-4 Turbo (advanced models).
• Perplexity Pro.
• Claude Opus.
• Gemini Advanced.

Tips:

• Supply ample context.
• Ask for detailed explanations.
• Build your prompts in layers.

Mixing and matching models

Not every task fits one category perfectly. Sometimes, you need to switch between models.

Example workflow:

• Start with a general model to generate ideas.
• Use a thinking model to refine your hypothesis.
• Finish with a precision model to document your findings.

Benefits:

• Saves time and resources.
• Matches the tool to the task.
• Keeps quality high without overcomplicating simple tasks.

Summary

Ask yourself: Do you rely on a single model for every task? Could varying your approach lead to better outcomes?

Consider mapping your next set of testing tasks onto this framework. Choose different tools for different needs and compare the results.

Ultimately, this framework isn’t about finding the best AI model overall. It’s about matching the tool to the task. Fast models handle routine work. More sophisticated ones support critical decisions. And in every case, the choice empowers you to work smarter, not harder.

Let the right model carry the heavy load while you focus on the art of testing. Experiment boldly, compare openly, and learn continuously. The harmony between tool and task is where efficiency meets innovation.

Also, check out the AI usage for testers: Quadrants model – Rahul’s Testing Titbits that was published last week.