Gates and regression mode

The gate decides whether a run passes. Set it with fail_on on GitHub or QUALITYGATE_FAIL_ON on GitLab and Bitbucket. QualityGate computes the decision on its own servers from the uploaded findings, so it cannot be changed from inside the CI job.

Absolute limits

The default, critical=0,serious=0, fails the run when the page has any critical or serious finding. Raise the numbers to tolerate some:

with:
  fail_on: critical=0,serious=5

Both critical and serious must be given. Moderate and minor findings are always reported and never fail a run.

Absolute limits suit new projects and sites that are already clean. On a site with existing issues, every pull request fails until all of them are fixed, which usually leads teams to switch the check off. Regression mode solves that.

Regression mode

fail_on: regression fails a run only for issues the pull request introduced. Existing issues are still reported; they just no longer decide the result.

with:
  fail_on: regression                 # no new critical or serious issues
  # fail_on: regression,critical=1    # allow one new critical issue
  # fail_on: regression,serious=3     # allow three new serious issues

The comment leads with what changed:

New in this pull request 3 · Existing 47 · Fixed in this pull request 5

Fixed issues are shown as progress and never fail a run.

Recording a baseline

Regression mode compares a pull request with the baseline: the last result of the same page and bundle on the branch the pull request merges into. A baseline is recorded by a run on that branch itself, so add a push trigger for your default branch next to pull_request:

name: QualityGate

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write

jobs:
  quality:
    runs-on: ubuntu-latest
    steps:
      - run: npx -y playwright@1.61.1 install --with-deps chromium
      - uses: qualitygate/scan-action@v1
        with:
          url: ${{ github.event_name == 'push' && 'https://www.example.com/' || format('https://pr-{0}.preview.example.com/', github.event.pull_request.number) }}
          fail_on: regression

On pushes to main the example scans production; on pull requests it scans the preview. Push runs post no comment and count as a normal run. On GitLab and Bitbucket, the default-branch pipeline plays the same role.

How issues are matched

Findings are matched by a fingerprint of the rule and the element, computed from the page path rather than the full URL. A different preview hostname, a new wrapper div, or a renumbered sibling element is not treated as a new issue. The same page path with the same bundle is compared across hosts, so /pricing on a preview is compared with /pricing on production.

Before the first baseline

Until a baseline exists for a page and bundle, regression runs pass and the report says that no comparison was possible, rather than claiming there were no new issues. Merge one pull request, or push to the default branch, to record it.

Choosing a starting point

Situation	Suggested gate
New project, or a site with no serious issues	critical=0,serious=0 (default)
Existing site with a backlog of issues	regression
Trying QualityGate without blocking anyone	Any gate plus continue-on-error: true
Agency tracking a client’s site	regression on pull requests, and a scheduled scan of production