Regression and Continuous Integration

A flow changes all the time. Scripts are fixed, tool versions are upgraded, libraries are recharacterised, and settings are tuned for one block that then affect every other block. Each change might improve results, or might quietly make them worse somewhere the author did not look. The only reliable way to know is to run the flow on a known set of designs and compare the results with known good values.

This lesson describes such regressions: what goes into a regression suite, how results are compared with a golden reference, how a continuous integration (CI) system runs the comparison automatically, what should block a change and what should only be reported, and how to deal with runs that fail for reasons unrelated to the change being tested.

Reading time: about 9 minutes.

8.1 What a flow regression is

A regression suite is a fixed set of test designs, each with a fixed configuration, that is run through the flow to check that a change has not broken anything. The term comes from software testing, where a regression is a feature that used to work and no longer does.

A good suite is small enough to run regularly and varied enough to exercise every part of the flow. Typically it contains a few tiny designs that run in minutes, a few medium blocks that run in hours, and one or two designs chosen for a particular feature – several clocks, multiple power domains described in UPF, hard macros, or an unusually dense floorplan. Each test design is chosen because it would reveal a particular kind of breakage. A suite of ten copies of the same kind of block finds the same bugs ten times.

8.2 Nightly regressions

The full suite is usually run overnight, when the compute farm (lesson 6) is less busy. A nightly run checks out the latest flow, libraries and tool settings, runs every test design to completion, extracts QoR with the script of lesson 7, and compares the results with the golden reference. By the morning, a summary shows each design as passed, improved, degraded or failed, with a link to its run directory.

The nightly run has one property that makes it valuable beyond any single result: it runs every night, with the same designs. A degradation therefore appears on the first night after the change that caused it, and the list of changes merged that day is short. This is the trend view of lesson 7 used as a diagnostic tool.

8.3 Comparing against a golden reference

The golden reference for a test design is a stored set of results that the team has accepted as correct – its QoR record, and sometimes key files such as the final netlist. A regression passes for that design if the new results agree with the golden ones within agreed tolerances.

Exact equality is rarely the right test for physical design results, for the reason given in lesson 7: placement and routing respond to small input changes, so metrics move a little from run to run. The comparison therefore uses a tolerance per metric and a direction – for slack, higher is better; for area, power and DRC count, lower is better.

# compare.py -- classify each metric against the golden record
# metric: (tolerance type, tolerance, +1 if higher is better else -1)
TOL = {"wns":   ("abs", 0.010, +1),
       "tns":   ("abs", 0.050, +1),
       "area":  ("rel", 0.005, -1),
       "power": ("rel", 0.010, -1),
       "drc":   ("abs", 0,     -1)}

def compare(new, gold):
    result = {}
    for key, (kind, tol, sense) in TOL.items():
        n, g = new.get(key), gold.get(key)
        if n is None:
            result[key] = "MISSING"
            continue
        diff = n - g
        if kind == "rel" and g:
            diff = diff / abs(g)
        gain = sense * diff          # positive means better than golden
        if gain < -tol:
            result[key] = "WORSE"
        elif gain > tol:
            result[key] = "BETTER"
        else:
            result[key] = "SAME"
    return result

When a change improves results deliberately, the golden reference must be updated, or every later run would be compared with an outdated standard. This re-baselining should be a reviewed change, recorded with its reason, just like a code change. A golden reference that anyone can overwrite to make a failure disappear is not a reference.

8.4 Continuous integration

Continuous integration is the practice of testing every change automatically before, or as soon as, it is merged. A CI server watches the flow repository. When an engineer proposes a change, the server runs a pipeline – an ordered set of jobs – and reports the result on the change itself. Jenkins and GitLab CI are two widely used systems. In Jenkins, the pipeline is usually described in a Jenkinsfile stored in the repository; in GitLab CI, in a file named .gitlab-ci.yml. Both send jobs to worker machines (agents in Jenkins, runners in GitLab), which in a chip flow are machines able to submit to the farm.

A minimal GitLab CI description of the smoke pipeline might look like the following. The file names under flow/ are the project's own scripts from lessons 3, 4 and 7.

stages: [lint, smoke, compare]

lint_scripts:
  stage: lint
  script:
    - python3 -m py_compile flow/*.py

smoke_gcd:
  stage: smoke
  tags: [eda-farm]                 # run on a runner that can reach the farm
  script:
    - make -C designs/gcd all
    - python3 flow/collect_qor.py designs/gcd/runs/latest > qor.json
  artifacts:
    when: always                   # keep logs even when the job fails
    paths: [qor.json, designs/gcd/runs/latest/]

compare_gcd:
  stage: compare
  script:
    - python3 flow/compare.py qor.json golden/gcd.json

Each script line is a shell command, and a non-zero exit status fails the job – the same exit-status contract as in lessons 3 and 4. A nightly run is usually the same pipeline started by a schedule (GitLab's scheduled pipelines, or a timed trigger in Jenkins) with the full suite selected.

8.5 What to gate on

A gate is a check whose failure blocks a change. Choosing gates is a balance. Too few, and broken changes reach the project. Too many, and engineers learn to override the gate, after which it protects nothing.

8.6 Flaky runs

A flaky run is one that fails, or changes its results, without any change to its inputs. In a chip flow the usual causes are infrastructure: no licence seat was available, an execution host failed, a disk filled up, or a run-time limit was hit on a busy night. Some tools are also not fully deterministic when run with many threads, so QoR might vary slightly between identical runs.

Flakiness is dangerous because it teaches people to ignore failures. Let us consider a gate that fails one night in five for licence reasons. Within a week, the team reruns every failure without reading it, and a real failure is rerun and missed as well. Some practices limit this:

• Classify failures. The wrapper and the log parser should distinguish infrastructure errors (licence checkout failure, a job killed by the scheduler for memory) from flow or design errors, and report them differently.
• Retry only infrastructure failures, automatically and a limited number of times. A flow error is never retried, because a retry that passes would hide it.
• Measure the noise. Running the same unchanged design several times shows how much each metric varies naturally. Tolerances should be set just above that variation, not guessed.
• Fix the cause. A recurring licence failure means the scheduler does not know about licences (lesson 6). A recurring timeout means the limit or the request is wrong.

8.7 Interview questions

What is a golden reference in a flow regression? A stored, accepted set of results for each test design – QoR metrics and sometimes key files – against which new runs are compared within tolerances.

Why compare with tolerances instead of exact values? Placement and routing results move slightly with small input changes and sometimes with thread scheduling. Exact comparison would fail on noise; tolerances fail only on real changes.

When should the golden reference be updated? When a deliberate change improves or intentionally alters results, through a reviewed change recorded with its reason – never simply to make a failing run pass.

How do you handle a flaky regression? Classify the failure as infrastructure or flow, retry only infrastructure failures automatically, measure natural variation to set tolerances, and fix the underlying cause such as missing licence scheduling.