Skip to content

PR Quality Gate Setup

This guide walks you through setting up and using the PR Quality Gate system for automated pull request scoring.

Overview

The PR Quality Gate calculates a comprehensive quality score (0-100) for each pull request based on:

  • Tests (40%): Test pass rate from pytest
  • Coverage (30%): Code coverage delta vs baseline
  • Static Analysis (30%): Ruff linting + Pyright type checking

PRs must score ≥75 to pass the quality gate.

Local Testing

Run scoring locally before pushing:

cd review_eval

# Run tests and generate reports
uv run pytest --junitxml=junit.xml --cov --cov-report=xml

# Run static analysis
uv run ruff check . --output-format json > ruff-results.json
uv run pyright --outputjson > pyright-results.json

# Calculate score
uv run python -m review_eval score \
  --config ../.github/reviewer-gate.yaml \
  --junit junit.xml \
  --coverage coverage.xml \
  --baseline-coverage 85.0 \
  --static-analysis ruff-results.json,pyright-results.json \
  --threshold 75

GitHub Actions Setup

The workflow runs automatically on:

  • PR opened
  • PR synchronized (new commits)
  • PR ready for review

Required Secrets

Configure these in your repository settings under Settings → Secrets:

  • OPENROUTER_API_KEY - For future AI review integration
  • QDRANT_URL - For future AI review integration
  • QDRANT_API_KEY - For future AI review integration

Note

Secrets are optional if not using AI review features.

Workflow File

The workflow is already configured in .github/workflows/pr-gate.yml. It:

  1. Runs tests with coverage
  2. Runs static analysis (ruff + pyright)
  3. Calculates PR score
  4. Comments score breakdown on PR
  5. Fails workflow if score < threshold

Configuration

Edit .github/reviewer-gate.yaml to customize scoring:

Adjusting Thresholds

Strict (recommended for production):

scoring:
  threshold: 85
  weights:
    tests: 0.50      # Higher test weight
    coverage: 0.25
    static_analysis: 0.25

Lenient (recommended for legacy projects):

scoring:
  threshold: 65
  weights:
    tests: 0.40
    coverage: 0.10   # Lower coverage weight
    static_analysis: 0.50

Tolerance Thresholds

Small PRs can mathematically decrease coverage even with good tests. Use tolerance to prevent false positives:

scoring:
  tolerance:
    coverage_delta: 0.1  # Ignore changes < 0.1%

Troubleshooting

Score Calculation Failed

Symptom: "Unable to calculate score" in PR comment

Fixes:

  1. Check workflow logs for errors
  2. Verify all input files exist (junit.xml, coverage.xml)
  3. Ensure pyproject.toml has correct dependencies

Coverage Delta Incorrect

Symptom: Large unexpected coverage changes

Fixes:

  1. Ensure baseline is fetched correctly
  2. Check if coverage.xml includes all files
  3. Adjust tolerance threshold in config

Static Analysis Errors

Symptom: Too many errors failing PRs

Fixes:

  1. Run uv run ruff check --fix . locally
  2. Add per-file ignores in pyproject.toml
  3. Adjust normalization curve in config

Workflow Not Triggering

Symptom: PR gate doesn't run on PRs

Fixes:

  1. Verify .github/workflows/pr-gate.yml exists
  2. Check PR is not in draft mode
  3. Ensure GitHub Actions are enabled

Example PR Comment

When the workflow completes, it posts a comment like this:

## ✅ PR Quality Score: 87.5/100

**Status**: PASS
**Threshold**: 75

### 📊 Breakdown

| Metric | Score | Weight | Details |
|--------|-------|--------|---------|
| ✓ Tests | 95.0/100 | 40% | 38/40 passed |
| ✓ Coverage | 82.0/100 | 30% | +2.5% |
| ✓ Static Analysis | 85.0/100 | 30% | 3 errors |

---
*Generated by Open Reviewer PR Gate*

Testing

Run the test suite to verify the scoring system:

cd review_eval

# Run all tests
uv run pytest tests/test_scoring_engine.py tests/test_collectors.py -v

# Run with coverage
uv run pytest --cov=review_eval --cov-report=term-missing

Test Coverage:

  • 8 scoring engine tests
  • 12 collector tests
  • 20/20 passing ✓

Next Steps