{"id":6667,"date":"2026-08-07T14:29:51","date_gmt":"2026-08-07T14:29:51","guid":{"rendered":"https:\/\/qyrus.com\/qapi\/?p=6667"},"modified":"2026-08-07T14:31:58","modified_gmt":"2026-08-07T14:31:58","slug":"automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing","status":"publish","type":"post","link":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/","title":{"rendered":"Automating AI Quality: How to Build a CI\/CD Pipeline for LLM Testing\u00a0"},"content":{"rendered":"\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

A prompt change ships on Thursday. The engineer tested it on five\u00a0examples\u00a0and it looked better. Two Mondays later, there are 60 support tickets about an answer pattern \u2014 the exact category the prompt change was supposed to\u00a0fix, but\u00a0now failing on a different edge case nobody tested.\u00a0<\/p>

This is the problem automated LLM evaluation is built to solve. As the\u00a0Galtea\u00a0engineering blog on CI\/CD quality gates\u00a0defines it: automated LLM evaluation is a CI\/CD pipeline where every change to a prompt, model version, or retrieval configuration triggers an eval run against a versioned golden dataset.\u00a0<\/p>

Most software teams already know how to do this for code. Commit, test, gate, deploy. The same logic applies to LLM applications.\u00a0It’s\u00a0just harder to implement because LLM outputs are probabilistic, not deterministic \u2014 and most teams\u00a0haven’t\u00a0set up the infrastructure yet.\u00a0<\/p>

This is a practical guide to building that pipeline. Step by step, with the specific design decisions that\u00a0determine\u00a0whether it\u00a0actually works\u00a0in production.\u00a0<\/p>

Why “Vibe Checks” Don’t Scale<\/b>\u00a0<\/h2>

The dominant evaluation method for LLM applications in production is still eyeballing outputs. Someone on the team\u00a0reads through\u00a0a sample of responses and decides whether quality is acceptable.\u00a0<\/p>

A\u00a02024 Stack Overflow developer survey\u00a0found that 76% of developers using LLMs in production relied primarily on manual review to assess output quality. The same survey found that only 23% had implemented any form of automated quality monitoring for their AI features. These are production systems serving real users, evaluated by gut feel.\u00a0<\/p>

This works when your query volume is low enough to read manually, your team has time to check before every deployment, and quality is either clearly good or clearly broken \u2014 not subtly degrading. None of those conditions hold at production scale.\u00a0<\/p>

Braintrust’s 2026 guide to AI evaluation in CI\/CD\u00a0makes the case directly: the best LLM applications\u00a0aren’t\u00a0built through endless manual testing sessions.\u00a0They’re\u00a0built through systematic, automated evaluation that runs with every code change. Teams are moving beyond one-off evaluations to continuous validation that runs automatically with every deployment.\u00a0<\/p>

The shift requires three things: a dataset that\u00a0represents\u00a0what real quality looks like, metrics that measure it automatically, and a pipeline that runs both on every change.\u00a0Here’s\u00a0how to build each one.\u00a0<\/p>

Step 1: Define What You’re Testing<\/b>\u00a0<\/h4>

Before writing any pipeline code, you need to answer one question: what does “good” mean for your specific LLM application?\u00a0<\/p>

For a RAG-based customer support bot, “good”\u00a0probably includes:\u00a0<\/p>

\u2022\u00a0<\/b>Answers that accurately reflect the company’s documented policies (faithfulness)\u00a0<\/p>

\u2022\u00a0<\/b>Responses that address what the customer\u00a0actually asked\u00a0(relevancy)\u00a0<\/p>

\u2022\u00a0<\/b>Answers that\u00a0don’t\u00a0add details the documentation\u00a0doesn’t\u00a0support (groundedness)\u00a0<\/p>

\u2022\u00a0<\/b>Responses that are consistent in tone and format (consistency)\u00a0<\/p>

For an internal document summarization tool, the metrics shift. Faithfulness and\u00a0groundedness\u00a0still matter, but exact citation accuracy might matter more than answer relevancy. For a coding assistant, you care about syntax correctness, whether generated code compiles, and whether it follows your existing patterns.\u00a0<\/p>

The mistake most teams\u00a0make:\u00a0applying a generic evaluation template before deciding what matters for their specific case.\u00a0<\/p>

The\u00a0ContextQA\u00a02026 guide to LLM testing tools\u00a0is specific about calibration: measure judge agreement against 20 to 30 human-labeled\u00a0examples specific to your task before using it as a CI gate. A judge achieving less than 80% agreement with human evaluators on your specific task type is not reliable enough for automated quality gates.\u00a0<\/p>

This step takes longer than it should. Budget a full sprint for it. The clarity you get from defining “good” in measurable terms before writing any pipeline code pays back across every\u00a0subsequent\u00a0evaluation run.\u00a0<\/p>

Step 2: Build Your Golden Dataset\u00a0From\u00a0Real Queries<\/b>\u00a0<\/h4>

The golden dataset is the most important piece of your evaluation pipeline and the\u00a0most commonly done\u00a0wrong.\u00a0<\/p>

A golden dataset is a curated set of inputs (queries), expected outputs or reference answers, and \u2014 for RAG systems \u2014 the relevant source documents. Every automated evaluation run measures your system’s outputs against this dataset.\u00a0<\/p>

The problem: most teams build golden datasets from hypothetical questions written before launch. The engineering team gathers in a room, writes down the questions they think users will ask, and calls it done.\u00a0<\/p>

A\u00a02025 paper by Shankar et al. on LLM evaluation validity\u00a0found that evaluation datasets built before product launch diverged significantly from real production query distributions within 90 days. The queries users\u00a0actually ask\u00a0look nothing like the questions developers\u00a0anticipate. That divergence means your pre-launch eval set will pass on queries it was designed to catch while missing the ones that\u00a0actually cause\u00a0problems.\u00a0<\/p>

Building the Dataset in Practice<\/b>\u00a0<\/p>

Start with real production traffic, even if\u00a0you’re\u00a0pre-launch.<\/b>\u00a0If you have a staging environment or beta users, instrument it to log queries. Even 100 real queries\u00a0is\u00a0more valuable than 500 hypothetical ones.\u00a0<\/p>

If\u00a0you’re\u00a0truly pre-launch,<\/b>\u00a0use a synthetic generator as a starting point. The\u00a0RAGAS framework\u00a0includes a synthetic dataset generator that creates question-answer pairs from your document corpus. Treat the generated dataset as a scaffold that domain experts then review and refine.\u00a0<\/p>

Seed with known failure modes.<\/b>\u00a0When an edge case breaks production, add it to the golden dataset\u00a0immediately. The dataset should grow to include every failure mode\u00a0you’ve\u00a0discovered.\u00a0<\/p>

Version it like code.<\/b>\u00a0The\u00a0Galtea\u00a0evaluation guide\u00a0is direct on this point: the golden dataset drifts over time as product scope changes, as failure modes are discovered, and as the team adds coverage for new query types. Dataset management is a first-class engineering problem. Treat dataset versions the same way you treat code versions \u2014 with a changelog, a review process, and clear ownership.\u00a0<\/p>

For most teams starting out, 50 to 100 high-quality, diverse examples\u00a0is\u00a0enough to get meaningful signal. You\u00a0don’t\u00a0need 10,000 examples to start.\u00a0<\/p>

Step 3: Choose Your Evaluation Metrics<\/b>\u00a0<\/h4>

With a golden dataset in hand, you need to decide what to measure.\u00a0<\/p>

For RAG Applications<\/b>\u00a0<\/p>

Faithfulness.<\/b>\u00a0Does the generated answer reflect what was in the retrieved context? The\u00a0original RAGAS paper (Es et al., 2024)\u00a0defines faithfulness as the fraction of claims in the answer that can be inferred from the retrieved context. This is the primary metric for catching hallucinations in grounded generation.\u00a0<\/p>

Context recall.<\/b>\u00a0Of all the information needed to correctly answer the question, how much appeared in the retrieved chunks? This is a retriever-level metric, not a generator-level one. Low recall means the retriever is missing relevant content before the model ever sees the query.\u00a0<\/p>

Context precision.<\/b>\u00a0Of the chunks that were retrieved, what fraction were\u00a0actually relevant? Low precision means the model is working with noise, which increases hallucination risk even when the right content is somewhere in the context.\u00a0<\/p>

Answer relevancy.<\/b>\u00a0Does the response address the actual question? High faithfulness plus low relevancy means the model answered a different question accurately.\u00a0<\/p>

Groundedness.<\/b>\u00a0A stricter version of faithfulness \u2014 every claim in the answer must be traceable to a specific retrieved passage. This is the right metric for high-stakes domains like healthcare or legal, where the standard for evidence is higher.\u00a0<\/p>

For LLM Applications Without RAG<\/b>\u00a0<\/p>

G-Eval \/ LLM-as-judge.<\/b>\u00a0Use a separate language model to score outputs on custom criteria. A\u00a02023 paper by Liu et al.\u00a0validated this approach, showing that LLM-based evaluators correlate well with human judgments on open-ended generation tasks \u2014 significantly better than string-match or n-gram metrics.\u00a0<\/p>

Task-specific deterministic metrics.<\/b>\u00a0For coding assistants: does the output compile and pass unit tests? For summarization: do named entities and key facts appear? These are your most reliable signals because\u00a0they’re\u00a0deterministic, not probabilistic.\u00a0<\/p>

A Note on LLM-as-Judge Reliability<\/b>\u00a0<\/p>

LLM-as-judge is flexible but has a known failure mode. As the\u00a0Galtea\u00a0evaluation guide\u00a0explains: an LLM judge that scores faithfulness is itself a model \u2014 it can produce false negatives. An eval run that produces a 0.82 faithfulness score is an estimate with error bars, not a deterministic pass\/fail.\u00a0<\/p>

A\u00a02024 meta-evaluation study by Zhu et al.\u00a0found that LLM judges show systematic biases toward longer, more verbose responses, responses that match their own training style, and responses that appear authoritative regardless of accuracy. Calibrate your judge against human labels before trusting it as a deployment gate.\u00a0<\/p>

Step 4: Set Quality Thresholds<\/b>\u00a0<\/h4>

A threshold is the\u00a0minimum\u00a0acceptable score below which a build fails. Setting them is uncomfortable because it forces you to be specific about what “acceptable” means.\u00a0<\/p>

Start realistic, not aspirational. If your current faithfulness score is 0.84, set the gate at 0.78. This catches meaningful drops while allowing normal variance. Raise the threshold as your system improves.\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"Three-tier\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Tiered thresholds work better than single thresholds in practice.<\/b>\u00a0<\/p>

\u2022\u00a0Hard block:<\/b>\u00a0Build fails completely. Reserve this for catastrophic failures \u2014 hallucination rate above 20%, or faithfulness below 0.60 on a medical or legal system.\u00a0<\/p>

\u2022\u00a0Soft warning:<\/b>\u00a0Build passes but the team is alerted. Use this for scores in a warning range \u2014 not failing, but worth reviewing.<\/p>

\u2022\u00a0<\/b>Trend alert:<\/b>\u00a0Scores are within acceptable range this run but have declined for three consecutive runs.\u00a0<\/p>

The trend alert is the one most\u00a0teams\u00a0skip and most teams later regret. A\u00a02026 analysis by\u00a0ContextQA\u00a0found that a 5% weekly quality decline reveals itself as a trend over six weeks \u2014 early warning before it becomes a user-visible degradation event. A point-in-time threshold\u00a0wouldn’t\u00a0catch it. A trend alert would.\u00a0<\/p>

Step 5: Wire It\u00a0Into\u00a0Your Pipeline<\/b>\u00a0<\/h4>

The pipeline structure is straightforward once steps 1\u20134 are done:\u00a0<\/p>

PR opened \/ commit pushed\u00a0<\/p>

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u2193\u00a0<\/p>

Run eval suite against golden dataset\u00a0<\/p>

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u2193\u00a0<\/p>

Score outputs (faithfulness, relevancy,\u00a0groundedness)\u00a0<\/p>

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u2193\u00a0<\/p>

Compare scores against thresholds\u00a0<\/p>

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u2193\u00a0<\/p>

Hard block? \u2192 Fail build, surface results\u00a0<\/p>

Soft warning? \u2192 Pass build, notify team\u00a0<\/p>

All clear? \u2192 Continue to deployment\u00a0<\/p>

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u2193\u00a0<\/p>

Post-deployment: continuous monitoring against production traffic\u00a0<\/p>

GitHub Actions:<\/b>\u00a0Add the eval run as a workflow step. The eval suite produces a result artifact \u2014 metric scores and a pass\/fail determination. The workflow branches based on that artifact.\u00a0<\/p>

Jenkins \/ GitLab CI:<\/b>\u00a0Same structure. The key integration point is publishing the eval results as a build artifact and using exit codes to signal\u00a0pass\/fail to\u00a0the pipeline orchestrator.\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"LLM\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

One critical design constraint the\u00a0Galtea\u00a0guide\u00a0flags clearly: the pipeline needs to track trends rather than point scores, catch regressions in aggregate rather than single-case failures, and route borderline cases to human review rather than treating them as hard blocks. An eval run that scores 0.76 faithfulness on one query out of 200 is not the same failure as a run where 60% of queries score below 0.76.\u00a0Aggregate,\u00a0don’t\u00a0fail on individual query results.\u00a0<\/p>

Step 6: The Two Failure Modes Nobody Prepares For<\/b>\u00a0<\/h4>

Prompt Drift<\/b>\u00a0<\/p>

A prompt change looks like an improvement on the tested examples. It ships. Two weeks later,\u00a0it’s\u00a0failing on a query category that\u00a0wasn’t\u00a0in the test set.\u00a0<\/p>

The\u00a0Agenta CI\/CD guide for LLM prompts\u00a0identifies\u00a0this pattern explicitly: in most organizations, a prompt change means editing a string in the codebase and deploying the whole application. Or worse, someone pastes a new prompt into a config file and pushes straight to main. Changes are deployed by gut feel.\u00a0<\/p>

The fix is treating prompt versions with the same versioning discipline as code. Every prompt change creates a versioned artifact. The eval suite runs against the new prompt version on the full golden dataset before it reaches production. Regression against the\u00a0previous\u00a0version’s scores catches cases where the new prompt improved some query categories while degrading others.\u00a0<\/p>

Silent Model Updates<\/b>\u00a0<\/p>

Model providers update their underlying models without always announcing it. A model that scored 0.89\u00a0groundedness\u00a0in March might score 0.81 in June \u2014 not because your code changed, but because the model serving that endpoint changed.\u00a0<\/p>

A\u00a02024 study by Chen et al.\u00a0tracked the\u00a0behavior\u00a0of GPT-3.5 and GPT-4 across six months and found statistically significant shifts in output distributions across multiple tasks \u2014 with no corresponding announcements from the provider. Teams relying on pre-deployment evaluation alone would have no visibility into these shifts.\u00a0<\/p>

The only way to catch this is continuous production monitoring: running your eval suite against a sample of real production traffic on a schedule, not just on deployments. This turns your evaluation pipeline from a deployment gate into an ongoing quality monitor.\u00a0<\/p>

Step 7: When the Pipeline Catches Something<\/b>\u00a0<\/h4>

A failing eval run is not a crisis.\u00a0It’s\u00a0the system working. The workflow when a build fails:\u00a0<\/p>

Look at which queries failed, not just the aggregate score.<\/b>\u00a0A drop in average faithfulness could mean a few specific query types are now failing badly.\u00a0Identifying\u00a0the pattern is faster than debugging the overall score.\u00a0<\/p>

Check what changed.<\/b>\u00a0Was there a prompt change, a model version change, a retrieval configuration change, or an API schema change upstream? The\u00a0ContextQA\u00a02026 guide\u00a0is clear on this: RAG applications have two independent failure modes. Testing only the combined pipeline tells you whether\u00a0the end result\u00a0is good. Testing each layer separately tells you which one to fix.\u00a0<\/p>

Compare side by side with the\u00a0previous\u00a0run.<\/b>\u00a0Which query types improved, which degraded? A prompt change that improves 80% of query types while degrading 20% is different from one that improves 20% and degrades 80%.\u00a0<\/p>

Don’t\u00a0lower the threshold to make the build pass.<\/b>\u00a0This is the failure mode that gradually erodes the value of the entire evaluation pipeline. Thresholds exist to catch real problems. If the build is failing, fix the quality issue.\u00a0<\/p>

What\u00a0qAPI\u00a0Adds to This Pipeline<\/b>\u00a0<\/h2>

Building all of this from scratch \u2014 dataset management, metric scoring, CI integration, reporting, production monitoring \u2014 takes significant engineering time. Most teams that build it themselves spend more time\u00a0maintaining\u00a0the evaluation infrastructure than using the results.\u00a0<\/p>

qAPI’s\u00a0LLM Evaluator replaces that homegrown stack. You connect your model endpoint, import your golden\u00a0dataset\u00a0or let\u00a0qAPI\u00a0generate a starter set from your production queries, set your thresholds, and wire into your CI\/CD pipeline.\u00a0<\/p>

What’s\u00a0different from rolling your own:\u00a0<\/p>

The API testing layer is included.<\/b>\u00a0When a retrieval API changes its response structure and causes a quality drop, that shows up in\u00a0qAPI\u00a0as an API test failure alongside the LLM eval failure \u2014 in the same report. You\u00a0don’t\u00a0need to separately instrument your API layer.\u00a0<\/p>

Production monitoring\u00a0doesn’t\u00a0require a separate tool.<\/b>\u00a0The same platform that runs your CI quality gates also\u00a0monitors\u00a0production traffic on a schedule. Trend alerts surface when scores are declining across consecutive runs.\u00a0<\/p>

Team-shareable reporting.<\/b>\u00a0The eval results\u00a0aren’t\u00a0pytest\u00a0output that only engineers can parse.\u00a0They’re\u00a0a shareable report where QA leads see the validation trace, product managers see the quality summary, and engineering leads see the metric trends \u2014 all from the same run.\u00a0<\/p>

The Practical Roadmap<\/b>\u00a0<\/h2>

The teams that successfully implement LLM CI\/CD do it in stages:\u00a0<\/p>

Week 1\u20132:<\/b>\u00a0Define your critical prompt paths and build a starter golden dataset. Even 30\u201350 high-quality examples\u00a0is\u00a0enough to start. This is the slowest week because it requires human judgment.\u00a0<\/p>

Week 3:<\/b>\u00a0Set up your evaluation metrics and calibrate them against human judgment on your specific task type. Verify 80%+ judge agreement before treating it as a gate.\u00a0<\/p>

Week 4:<\/b>\u00a0Wire the eval run into your CI pipeline with soft-warning thresholds only.\u00a0Don’t\u00a0block deployments yet \u2014 just collect data.\u00a0<\/p>

Week 5\u20136:<\/b>\u00a0Analyze\u00a0the first few weeks of eval runs. Adjust thresholds based on observed score distributions.\u00a0<\/p>

Week 7:<\/b>\u00a0Promote critical-path metrics to hard-block status. You now have a functioning LLM quality gate.\u00a0<\/p>

Ongoing:<\/b>\u00a0Grow the golden dataset. Add every real-world failure mode as it surfaces. Raise thresholds as quality improves. Add production monitoring once the deployment gate is stable.\u00a0<\/p>

The teams that fail at this try to build the whole thing in week one. Starting small and hardening over time is the approach that\u00a0actually ships.\u00a0<\/p>

The Bottom Line<\/b>\u00a0<\/h2>

An LLM CI\/CD pipeline\u00a0isn’t\u00a0fundamentally different from any other software quality pipeline. The principles are identical: define expected\u00a0behavior, measure it automatically, gate deployments on meeting the standard,\u00a0monitor\u00a0continuously in production.\u00a0<\/p>

The only difference is that LLM outputs are probabilistic, not deterministic. That changes the measurement\u00a0methodology\u00a0but not the engineering discipline.\u00a0<\/p>

If a prompt change can break production silently, it deserves the same pipeline scrutiny as a code change that breaks a unit test.\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

A prompt change ships on Thursday. The engineer tested it on five\u00a0examples\u00a0and it looked better. Two Mondays later, there are 60 support tickets about an answer pattern \u2014 the exact category the prompt change was supposed to\u00a0fix, but\u00a0now failing on a different edge case nobody tested.\u00a0 This is the problem automated LLM evaluation is built…<\/p>\n","protected":false},"author":9,"featured_media":6678,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","inline_featured_image":false,"footnotes":""},"categories":[17,10],"tags":[],"class_list":["post-6667","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-resources"],"acf":[],"yoast_head":"\nAutomating AI Quality: How to Build a CI\/CD Pipeline for LLM Testing\u00a0 - qAPI<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Automating AI Quality: How to Build a CI\/CD Pipeline for LLM Testing\u00a0 - qAPI\" \/>\n<meta property=\"og:description\" content=\"A prompt change ships on Thursday. The engineer tested it on five\u00a0examples\u00a0and it looked better. Two Mondays later, there are 60 support tickets about an answer pattern \u2014 the exact category the prompt change was supposed to\u00a0fix, but\u00a0now failing on a different edge case nobody tested.\u00a0 This is the problem automated LLM evaluation is built...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/\" \/>\n<meta property=\"og:site_name\" content=\"qAPI\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/profile.php?id=61571758838201\" \/>\n<meta property=\"article:published_time\" content=\"2026-08-07T14:29:51+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-08-07T14:31:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2026\/08\/Three-tier-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"4455\" \/>\n\t<meta property=\"og:image:height\" content=\"2271\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"R Varun\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@testwithqapi\" \/>\n<meta name=\"twitter:site\" content=\"@testwithqapi\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"R Varun\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"13 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/\"},\"author\":{\"name\":\"R Varun\",\"@id\":\"https:\/\/qyrus.com\/qapi\/#\/schema\/person\/33d511c123d8cd9b9e9dc5ee9e0e5c90\"},\"headline\":\"Automating AI Quality: How to Build a CI\/CD Pipeline for LLM Testing\u00a0\",\"datePublished\":\"2026-08-07T14:29:51+00:00\",\"dateModified\":\"2026-08-07T14:31:58+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/\"},\"wordCount\":2677,\"publisher\":{\"@id\":\"https:\/\/qyrus.com\/qapi\/#organization\"},\"image\":{\"@id\":\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2026\/08\/thumbnail-1.png\",\"articleSection\":[\"Blog\",\"Resources\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/\",\"url\":\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/\",\"name\":\"Automating AI Quality: How to Build a CI\/CD Pipeline for LLM Testing\u00a0 - qAPI\",\"isPartOf\":{\"@id\":\"https:\/\/qyrus.com\/qapi\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2026\/08\/thumbnail-1.png\",\"datePublished\":\"2026-08-07T14:29:51+00:00\",\"dateModified\":\"2026-08-07T14:31:58+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#primaryimage\",\"url\":\"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2026\/08\/thumbnail-1.png\",\"contentUrl\":\"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2026\/08\/thumbnail-1.png\",\"width\":1280,\"height\":720,\"caption\":\"thumbnail\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/qyrus.com\/qapi\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Automating AI Quality: How to Build a CI\/CD Pipeline for LLM Testing\u00a0\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/qyrus.com\/qapi\/#website\",\"url\":\"https:\/\/qyrus.com\/qapi\/\",\"name\":\"qAPI\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/qyrus.com\/qapi\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/qyrus.com\/qapi\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/qyrus.com\/qapi\/#organization\",\"name\":\"qAPI\",\"url\":\"https:\/\/qyrus.com\/qapi\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/qyrus.com\/qapi\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2025\/02\/qAPI-Youtube-DP-98-x-98.png\",\"contentUrl\":\"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2025\/02\/qAPI-Youtube-DP-98-x-98.png\",\"width\":409,\"height\":409,\"caption\":\"qAPI\"},\"image\":{\"@id\":\"https:\/\/qyrus.com\/qapi\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/profile.php?id=61571758838201\",\"https:\/\/x.com\/testwithqapi\",\"https:\/\/www.linkedin.com\/company\/testwithqapi\/?viewAsMember=true\",\"https:\/\/www.instagram.com\/testwithqapi\/\",\"https:\/\/www.youtube.com\/@testwithqapi\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/qyrus.com\/qapi\/#\/schema\/person\/33d511c123d8cd9b9e9dc5ee9e0e5c90\",\"name\":\"R Varun\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/qyrus.com\/qapi\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/62344175a96575918f882055650fdf8d3c6c18886a2248ce250f7cd05e3ca866?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/62344175a96575918f882055650fdf8d3c6c18886a2248ce250f7cd05e3ca866?s=96&d=mm&r=g\",\"caption\":\"R Varun\"},\"url\":\"https:\/\/qyrus.com\/qapi\/author\/rvarunqyrus-com\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Automating AI Quality: How to Build a CI\/CD Pipeline for LLM Testing\u00a0 - qAPI","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/","og_locale":"en_US","og_type":"article","og_title":"Automating AI Quality: How to Build a CI\/CD Pipeline for LLM Testing\u00a0 - qAPI","og_description":"A prompt change ships on Thursday. The engineer tested it on five\u00a0examples\u00a0and it looked better. Two Mondays later, there are 60 support tickets about an answer pattern \u2014 the exact category the prompt change was supposed to\u00a0fix, but\u00a0now failing on a different edge case nobody tested.\u00a0 This is the problem automated LLM evaluation is built...","og_url":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/","og_site_name":"qAPI","article_publisher":"https:\/\/www.facebook.com\/profile.php?id=61571758838201","article_published_time":"2026-08-07T14:29:51+00:00","article_modified_time":"2026-08-07T14:31:58+00:00","og_image":[{"width":4455,"height":2271,"url":"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2026\/08\/Three-tier-1.png","type":"image\/png"}],"author":"R Varun","twitter_card":"summary_large_image","twitter_creator":"@testwithqapi","twitter_site":"@testwithqapi","twitter_misc":{"Written by":"R Varun","Est. reading time":"13 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#article","isPartOf":{"@id":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/"},"author":{"name":"R Varun","@id":"https:\/\/qyrus.com\/qapi\/#\/schema\/person\/33d511c123d8cd9b9e9dc5ee9e0e5c90"},"headline":"Automating AI Quality: How to Build a CI\/CD Pipeline for LLM Testing\u00a0","datePublished":"2026-08-07T14:29:51+00:00","dateModified":"2026-08-07T14:31:58+00:00","mainEntityOfPage":{"@id":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/"},"wordCount":2677,"publisher":{"@id":"https:\/\/qyrus.com\/qapi\/#organization"},"image":{"@id":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#primaryimage"},"thumbnailUrl":"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2026\/08\/thumbnail-1.png","articleSection":["Blog","Resources"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/","url":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/","name":"Automating AI Quality: How to Build a CI\/CD Pipeline for LLM Testing\u00a0 - qAPI","isPartOf":{"@id":"https:\/\/qyrus.com\/qapi\/#website"},"primaryImageOfPage":{"@id":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#primaryimage"},"image":{"@id":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#primaryimage"},"thumbnailUrl":"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2026\/08\/thumbnail-1.png","datePublished":"2026-08-07T14:29:51+00:00","dateModified":"2026-08-07T14:31:58+00:00","breadcrumb":{"@id":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#primaryimage","url":"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2026\/08\/thumbnail-1.png","contentUrl":"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2026\/08\/thumbnail-1.png","width":1280,"height":720,"caption":"thumbnail"},{"@type":"BreadcrumbList","@id":"https:\/\/qyrus.com\/qapi\/automating-ai-quality-how-to-build-a-ci-cd-pipeline-for-llm-testing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/qyrus.com\/qapi\/"},{"@type":"ListItem","position":2,"name":"Automating AI Quality: How to Build a CI\/CD Pipeline for LLM Testing\u00a0"}]},{"@type":"WebSite","@id":"https:\/\/qyrus.com\/qapi\/#website","url":"https:\/\/qyrus.com\/qapi\/","name":"qAPI","description":"","publisher":{"@id":"https:\/\/qyrus.com\/qapi\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/qyrus.com\/qapi\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/qyrus.com\/qapi\/#organization","name":"qAPI","url":"https:\/\/qyrus.com\/qapi\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/qyrus.com\/qapi\/#\/schema\/logo\/image\/","url":"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2025\/02\/qAPI-Youtube-DP-98-x-98.png","contentUrl":"https:\/\/qyrus.com\/qapi\/wp-content\/uploads\/2025\/02\/qAPI-Youtube-DP-98-x-98.png","width":409,"height":409,"caption":"qAPI"},"image":{"@id":"https:\/\/qyrus.com\/qapi\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/profile.php?id=61571758838201","https:\/\/x.com\/testwithqapi","https:\/\/www.linkedin.com\/company\/testwithqapi\/?viewAsMember=true","https:\/\/www.instagram.com\/testwithqapi\/","https:\/\/www.youtube.com\/@testwithqapi"]},{"@type":"Person","@id":"https:\/\/qyrus.com\/qapi\/#\/schema\/person\/33d511c123d8cd9b9e9dc5ee9e0e5c90","name":"R Varun","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/qyrus.com\/qapi\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/62344175a96575918f882055650fdf8d3c6c18886a2248ce250f7cd05e3ca866?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/62344175a96575918f882055650fdf8d3c6c18886a2248ce250f7cd05e3ca866?s=96&d=mm&r=g","caption":"R Varun"},"url":"https:\/\/qyrus.com\/qapi\/author\/rvarunqyrus-com\/"}]}},"_links":{"self":[{"href":"https:\/\/qyrus.com\/qapi\/wp-json\/wp\/v2\/posts\/6667","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qyrus.com\/qapi\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qyrus.com\/qapi\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qyrus.com\/qapi\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/qyrus.com\/qapi\/wp-json\/wp\/v2\/comments?post=6667"}],"version-history":[{"count":7,"href":"https:\/\/qyrus.com\/qapi\/wp-json\/wp\/v2\/posts\/6667\/revisions"}],"predecessor-version":[{"id":6684,"href":"https:\/\/qyrus.com\/qapi\/wp-json\/wp\/v2\/posts\/6667\/revisions\/6684"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/qyrus.com\/qapi\/wp-json\/wp\/v2\/media\/6678"}],"wp:attachment":[{"href":"https:\/\/qyrus.com\/qapi\/wp-json\/wp\/v2\/media?parent=6667"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qyrus.com\/qapi\/wp-json\/wp\/v2\/categories?post=6667"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qyrus.com\/qapi\/wp-json\/wp\/v2\/tags?post=6667"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}