QoR Metrics and Reporting

A flow that runs is only half a flow. The other half is knowing what each run produced: did timing close, how large is the design, how much power does it draw, are the routes clean. These numbers are called the quality of results, or QoR, and they are scattered across dozens of reports written by different tools in different formats.

This lesson lists the core QoR metrics, shows how to extract them from report text with a short Python script, and then turns the extracted numbers into tables that compare runs side by side and trends that follow one block over time. The comparison against a reference in lesson 8 is built directly on this.

Reading time: about 8 minutes.

7.1 The core metrics

Every project adds its own metrics, but a small set appears in almost every QoR table.

The sign convention deserves a note. Some tools report WNS as the worst slack even when it is positive; others clip it at zero. TNS is zero for a design with no violations. It is better to record exactly what the tool printed and to note the convention in the table header, than to guess.

7.2 Where the numbers live

Let us consider the end of a routing and STA run in OpenROAD, which uses OpenSTA for timing. A few of its report commands are report_wns, report_tns, report_design_area and report_power, and the router prints a violation count. An excerpt of such a log looks like the following – exact wording varies between versions, which is the subject of section 7.4.

wns -0.042
tns -0.315
Design area 15234 u^2 62% utilization.
Group                  Internal  Switching    Leakage      Total
                          Power      Power      Power      Power (Watts)
----------------------------------------------------------------
Sequential             1.12e-03   2.10e-04   3.05e-08   1.33e-03  38.9%
Combinational          9.40e-04   1.15e-03   5.12e-08   2.09e-03  61.1%
----------------------------------------------------------------
Total                  2.06e-03   1.36e-03   8.17e-08   3.42e-03 100.0%
...
Number of violations = 0.

A person reads these lines easily. A regression of a hundred blocks cannot be read by a person, so a script must do it.

7.3 Extracting metrics with regular expressions

A regular expression describes a pattern of text. For QoR extraction, each metric gets one pattern that finds its line and captures the number in parentheses. The script below parses the excerpt above.

#!/usr/bin/env python3
"""qor.py LOG... -- extract QoR metrics from tool logs into JSON."""
import json, re, sys
from pathlib import Path

NUM = r"(-?\d+(?:\.\d+)?(?:[eE][-+]?\d+)?)"        # 12, -0.042, 3.42e-03
PATTERNS = {
    "wns":   (rf"^wns\s+{NUM}",                     float),
    "tns":   (rf"^tns\s+{NUM}",                     float),
    "area":  (rf"^Design area\s+{NUM}\s+u\^2",      float),
    "util":  (rf"{NUM}%\s+utilization",             float),
    "power": (rf"^Total\s+(?:\S+\s+){{3}}{NUM}",     float),
    "drc":   (r"Number of violations\s*=\s*(\d+)",  int),
}

def parse(text):
    qor = {}
    for key, (pat, conv) in PATTERNS.items():
        hits = re.findall(pat, text, re.MULTILINE)
        qor[key] = conv(hits[-1]) if hits else None   # last value wins
    return qor

if __name__ == "__main__":
    for log in sys.argv[1:]:
        q = parse(Path(log).read_text())
        missing = [k for k, v in q.items() if v is None]
        print(json.dumps({"log": log, **q, "missing": missing}))

A few choices in this script are deliberate.

• Anchors. ^ with re.MULTILINE matches only at the start of a line, so ^tns does not match a word that merely contains "tns".
• The power pattern skips three columns (internal, switching, leakage) with (?:\S+\s+){3} and captures the fourth, the total. In the Python f-string the braces are doubled, {{3}}, so that they reach the regular expression as {3}.
• The last match wins. A router prints its violation count after every iteration. Only the final one is the result, so the script takes hits[-1].
• Missing is not zero. If a pattern finds nothing, the value is None and the metric is listed as missing. Reporting a missing DRC count as 0 would turn a crashed run into a clean one.

Applied to the excerpt, the script prints wns −0.042, tns −0.315, area 15234, util 62, power 0.00342 (watts) and drc 0, with nothing missing.

7.4 Making extraction robust

Report parsing breaks in predictable ways, and each has a known remedy.

• Format changes between tool versions. A new release might reword a line or add a column. This is one reason, from lesson 1, why tool upgrades are qualified: the QoR script is part of what is qualified. A missing-metric list, as above, turns a silent change into a visible one.
• Units. Power might be printed in watts or milliwatts, time in nanoseconds or picoseconds, depending upon the library and tool settings. The parser should record the unit, or convert to one unit, never mix them.
• Several values of one metric. Timing reports for several corners or path groups contain several WNS values. The pattern must capture the context (corner name, group) as well as the number.
• Machine-readable output. Where a tool can write metrics directly in a structured form, it is better to use it. OpenROAD, for example, accepts a -metrics option that writes a JSON file of metrics. Regular expressions then become a fallback rather than the main path.

7.5 QoR tables: comparing runs

Once each run produces a small JSON record, comparing runs is a matter of placing records side by side. The most useful table puts runs in rows and metrics in columns, with the reference run first.

Two habits make such tables trustworthy. First, every row links to its run directory and its run_info.json (lesson 5), so that anyone can check which versions produced it. Second, only runs that differ in one known way are compared; a table that mixes a utilisation change with a library change explains nothing.

7.6 Trend tracking

A trend follows one block through many runs over time – typically the nightly regression of lesson 8. The value of a trend is that it shows when something changed, which usually identifies what changed.

Every metric moves slightly from run to run, because placement and routing algorithms are sensitive to small input changes. A trend should therefore be judged against a tolerance, not against exact equality – for example, WNS may move by a few picoseconds, but a drop of tens of picoseconds or a change of sign is flagged. Choosing those tolerances is part of the methodology, and lesson 8 uses them to decide when a regression passes.

7.7 Interview questions

What is the difference between WNS and TNS? WNS is the slack of the single worst violating path; TNS is the sum of negative slacks over all violating endpoints. A design can have a small WNS but a large TNS if many endpoints fail by a little.

Why should a QoR parser report a missing metric as missing rather than zero? A crashed or truncated run produces no report line. Reading that as 0 would show, for example, zero DRC violations or zero TNS for a run that never finished.

Why take the last match when parsing a router log? Routers print intermediate counts after each iteration; only the final value is the result of the run.

Why compare runs that differ in only one way? So that a change in QoR can be attributed to that one change. Mixing changes makes the table impossible to interpret.