06 — Azure DevOps CI/CD & Quality Gates

JD: "Integrate test suites into Azure DevOps CI/CD pipelines as automated quality gates." Know the vocabulary and be able to sketch a YAML pipeline from memory.

---

1. Azure DevOps vocabulary

Term	In plain words
Pipeline	The whole automated build/test/deploy definition (azure-pipelines.yml).
Stage	A big phase (Build, Test, Deploy). Stages run in order, can gate each other.
Job	A unit that runs on one agent. Jobs in a stage can run in parallel.
Step / Task	A single command or prebuilt action inside a job.
Agent	The machine (Microsoft-hosted or self-hosted) that runs a job.
Trigger	What starts the pipeline (push, PR, schedule).
Environment	A deploy target with approvals & checks (the manual/automated gate).
Artifact	Files passed between stages (build output, test reports).
Variable / Variable group / Key Vault	Config & secrets.

Quality gate = a stage that must pass before the next runs. If tests fail (or coverage/pass-rate is below threshold), the pipeline stops and deployment is blocked.

---

2. A pipeline sketch (be able to write this)

trigger:
  branches: { include: [main] }
pr:
  branches: { include: [main] }        # run on PRs = the gate

variables:
  - group: qa-secrets                  # API_URL, tokens from a variable group

stages:
- stage: Test
  jobs:
  - job: PytestGate
    pool: { vmImage: 'ubuntu-latest' }
    steps:
    - task: UsePythonVersion@0
      inputs: { versionSpec: '3.12' }
    - script: pip install -r requirements.txt
      displayName: Install deps
    - script: |
        pytest -m "smoke" -n auto \
          --junitxml=results.xml \
          --cov=app --cov-report=xml
      displayName: Run tests
    - task: PublishTestResults@2         # surfaces pass/fail in the UI
      condition: always()
      inputs:
        testResultsFiles: 'results.xml'
        testResultsFormat: 'JUnit'
    - task: PublishCodeCoverageResults@2
      inputs: { summaryFileLocation: 'coverage.xml' }

- stage: Deploy
  dependsOn: Test
  condition: succeeded()                 # <-- the gate: only deploy if Test passed
  jobs:
  - deployment: DeployApp
    environment: 'production'            # environment can require manual approval
    strategy:
      runOnce:
        deploy:
          steps:
            - script: ./deploy.sh

Key gate mechanics to name: - condition: succeeded() / dependsOn chains stages. - PublishTestResults@2 makes failures visible and blocks by default. - Environment approvals & checks = human sign-off or automated checks (e.g. a Langfuse eval score) before deploy. - Branch policies on main: require the PR pipeline to pass before merge.

---

3. Playwright & UI tests in the pipeline

- script: |
    pip install pytest-playwright
    playwright install --with-deps chromium
    pytest tests/e2e --tracing=retain-on-failure --junitxml=e2e.xml
  displayName: E2E
- task: PublishPipelineArtifact@1        # keep traces/videos for failure triage
  condition: failed()
  inputs: { targetPath: 'test-results', artifact: 'playwright-traces' }

Headless by default; publish traces/videos as artifacts so you can debug CI failures with the trace viewer (file 02).

---

4. Parallelism & speed

• pytest -n auto (xdist) inside a job.
• Matrix / multiple jobs to shard suites or run across Python versions/browsers:

  strategy:
    matrix:
      chromium: { BROWSER: chromium }
      firefox:  { BROWSER: firefox }
  

• Cache pip deps to speed builds (Cache@2).

---

5. Where AI-agent quality gates fit (the differentiator)

Beyond unit/API/UI tests, add gates specific to this platform:

1. Contract/schema gate — Pydantic/Schemathesis on the agent API (file 03).
2. Eval gate — run the LLM eval harness over the golden set; fail the build if the aggregate score drops below threshold or regresses vs the baseline (files 07/10).
3. Langfuse assertion gate — assert traces have expected spans/tool-calls/scores (file 09).
4. Model-version check — if the deployed model version changed, force the full eval + require sign-off (file 10).
5. Audit-integrity gate — verify audit events emitted with correlation IDs (file 12).

- script: python -m eval.run --dataset golden.jsonl --min-score 0.85 --baseline baseline.json
  displayName: LLM eval gate      # exits non-zero if score < 0.85 or regressed

Interview line: "I treat evaluation as a first-class quality gate: the pipeline runs the eval harness over a frozen golden set and blocks deployment if the aggregate score drops below threshold or regresses against the recorded baseline — the same way a failing unit test blocks a merge."

---

6. Secrets, artifacts, notifications

• Secrets from Azure Key Vault or secret variables — never in YAML/logs (isSecret).
• Publish JUnit + coverage + Playwright traces as artifacts.
• Notify on failure (Teams/email); dashboards/widgets for pass-rate trends.

---

Rapid-fire recall

• Stage → job → step; jobs parallel, stages sequential.
• Gate = dependsOn + condition: succeeded(); environment approvals for deploy.
• PublishTestResults@2 surfaces & blocks on failures.
• Publish Playwright traces as artifacts for CI triage.
• Add an eval gate: block deploy if golden-set score drops/regresses.
• Secrets via Key Vault / variable groups, never inline.