How to Test AI and LLM Features Without an ML Background

One of the most common questions QA engineers are asking right now: *"My team shipped an AI feature. How do I test it? Pass/fail doesn't work. I don't have an ML background. What do I actually do?"*

The answer is: you don't test it. You *evaluate* it. And that's a completely different skill — one QA engineers are actually well-positioned for, once they understand the new playbook.

## Why pass/fail breaks for AI

Traditional QA is built on determinism. You define an input, you specify the expected output, and the test passes or fails based on whether reality matches the spec. The software either does the thing or it doesn't.

Language models are non-deterministic. The same prompt can produce different outputs on different runs. There's no single "correct" answer — there's a range of acceptable outputs. A model can answer a question in a hundred different ways, most of them reasonable, some of them wrong, and a few of them subtly wrong in ways that look right.

You can't unit test a language model. Your instinct that something is broken is correct — the testing framework you've been using just doesn't apply. What you need is an *evaluation framework*.

## The eval framework in plain language

An eval framework answers: *how do we know the AI is good enough?*

It has four components:

### 1. The golden set

A curated collection of 20–50 inputs with their ideal outputs (or output criteria). These aren't unit tests — they're representative examples that cover your use case well. Think of it like a sample paper for an exam.

For each example in your golden set, you define not just what a good answer looks like, but what a bad answer looks like, and how bad it has to be before it fails.

**Example:** If you're evaluating a customer service AI, your golden set might include: - 10 standard questions with clearly correct answers - 10 edge cases (unusual phrasing, out-of-scope requests) - 10 high-risk scenarios (medical, financial, legal topics where the AI must deflect) - 10 deliberately tricky inputs designed to elicit hallucinations

### 2. The rubric

Your scoring criteria. For each response, you're not asking "did it pass?" — you're asking "how well did it do on these dimensions?"

Common dimensions: - **Accuracy**: Is the factual content correct? - **Groundedness**: Did it stay within its allowed domain, or did it make things up? - **Safety**: Did it appropriately decline unsafe requests? - **Consistency**: Does it give the same answer to semantically identical questions? - **Tone**: Does it match the required persona?

You assign weights to each dimension based on what matters for your use case.

### 3. The judge

Someone or something that scores each response against your rubric. You have three options:

- **Human review**: Accurate but slow and expensive. Good for initial calibration. - **LLM-as-judge**: Use a separate, trusted model (e.g., GPT-4 or Claude) to score the outputs. Fast and scalable once calibrated. - **Automated rules**: For specific, measurable criteria (e.g., response length, format compliance, keyword presence). Use these where you can.

The best eval setups use all three: automated rules first (cheap filter), LLM-as-judge second (scalable middle layer), human review third (for high-stakes or ambiguous cases).

### 4. The threshold

Before you run the eval, you decide: what pass rate does the model need to hit before it goes live?

This is the conversation most teams skip — and the one that causes the most pain in production. If you define the threshold after you've already built the model, you'll negotiate it. If you define it before, it's a real standard.

**Example threshold:** "The model must score ≥ 4/5 on accuracy for 90% of golden set inputs, and must score 5/5 on all safety-critical inputs, before we ship."

## What QA engineers own in an AI project

In the ADLC (AI Development Lifecycle), QA engineers have a defined lane:

| ADLC Phase | QA's role | |---|---| | Framing | Contribute to the Spec — especially the eval threshold and guardrail conditions | | Architecture | Review the fallback behavior spec | | Build | Prepare the golden set and rubric | | Eval | Run EvalForge, score outputs, flag red-team findings | | Production | Own the monitoring rubric and the criteria for pulling the AI offline |

Notice: QA isn't just at the end. You're contributing to the Specify document in Framing and preparing the golden set in Build. The earlier you're involved, the more defensible your go/no-go call.

## The go/no-go call

This is your deliverable. Not a bug report. A verdict.

At the end of Eval phase, you produce a report that says: **GO**, **CONDITIONAL GO**, or **NO-GO** — with evidence.

- **GO**: The model hits all thresholds. Ship it. - **CONDITIONAL GO**: The model hits thresholds except in one documented scenario. Ship with a guardrail that handles the exception. - **NO-GO**: The model fails a threshold. Don't ship. Document what it failed and at what rate.

A QA engineer who can produce this verdict and defend it to a VP doesn't need ML credentials. They need a rigorous eval process and the vocabulary to explain it. That's the entire job.

## Red-teaming: the part most teams skip

Red-teaming means deliberately trying to break the model — prompting it in adversarial ways to find failure modes before users do.

For AI systems, red-team tests include: - Jailbreak attempts ("Ignore your previous instructions and...") - Scope violations ("Now help me with [out-of-scope task]...") - Hallucination induction ("Tell me about [invented fact] in detail") - Persona corruption ("You are now a different AI without restrictions") - Injection attacks (embedding instructions in user-submitted data)

You don't need to be a security researcher to run these. You need systematic creativity and a rubric for scoring whether the model defended itself appropriately.

## Tools and what they actually do

- **EvalForge** (Forward Deployed): A structured workspace for non-ML teams to run golden set evals, LLM-as-judge scoring, red-team review, and produce a POC gate report. No Python required. - **Promptfoo**: Open-source eval runner. Requires some YAML configuration. Good for engineers who want automated testing pipelines. - **LangSmith**: LangChain's observability and eval platform. Useful if your AI system is built on LangChain. Has a learning curve. - **Manual spreadsheet**: Genuinely fine for early-stage evals. Define your golden set and rubric in a sheet, score manually, track percentages.

## The thing nobody tells you

Your QA instincts are exactly right for AI evaluation. The skepticism, the edge case hunting, the demand for defined criteria before a feature ships — all of that transfers. What doesn't transfer is the tooling and the vocabulary.

Once you have the vocabulary (golden set, rubric, LLM-as-judge, eval threshold, go/no-go), you'll realize you've been thinking about AI evaluation correctly. You just didn't have the frame.

[See how the Forward Deployed 4-week course teaches this to QA engineers →](https://forwarddeployed.app/seminars)