Running Jobs on a Compute Farm

A single place-and-route run on a large block can use a dozen cores and well over a hundred gigabytes of memory for many hours, and a project might run hundreds of such jobs every night. No engineer's workstation can do this. The work is sent instead to a compute farm – a pool of shared machines managed by a batch scheduler – and the flow has to know how to ask for machines politely and correctly.

This lesson describes how a scheduler works, shows job submission with two widely used schedulers, IBM Spectrum LSF and SLURM, and explains why licences, not machines, are often the real limit. It ends with running timing corners in parallel, the case where a farm pays off most directly.

Reading time: about 8 minutes.

6.1 How a batch scheduler works

Let us consider what happens when an engineer submits a job. The engineer, or the Makefile from lesson 4, runs a submission command on a login or submission host. The command does not run the tool; it hands a job description to the scheduler. The job description states what to run and what it needs: number of cores, amount of memory, expected run time, and perhaps a licence.

The scheduler holds the job in a queue (LSF's term; SLURM calls the equivalent a partition) until an execution host with enough free resources is available, and until the job's priority is high enough compared with other waiting jobs. Most farms use a fair-share policy, so that one user or project submitting thousands of jobs cannot take the whole farm. When the job is dispatched, it runs on the execution host, writes its output to shared storage, and its exit status is recorded by the scheduler.

6.2 Submitting a job with LSF

In LSF, jobs are submitted with bsub. The options used most often in flows are shown below.

bsub [-J job_name] [-q queue] [-n cores]
     [-R "resource_requirement"] [-M mem_limit]
     [-W run_limit] [-o out_file] [-e err_file] [-K]
     command [arguments]

bsub -J place_aes -q normal -n 8 \
     -R "span[hosts=1] rusage[mem=32000]" -M 32000 \
     -W 6:00 -o logs/place.%J.log -K \
     python3 flow/run_stage.py place config/aes_core.json

Here -n 8 asks for eight job slots, span[hosts=1] keeps them on one machine (a multi-threaded EDA tool cannot use cores on another host), rusage[mem=...] reserves memory for scheduling, -M sets the memory limit beyond which the job may be killed, -W limits the run time to six hours, and %J in the output file name is replaced by the job ID. The unit of the memory values is set by the site's LSF configuration and is often megabytes, so it is better to confirm it with the farm administrators than to assume.

The option -K deserves attention. It makes bsub wait until the job finishes and return the job's exit status. Without it, bsub returns as soon as the job is queued, and a Makefile recipe built on it would "succeed" before the tool even started. Jobs are then followed with bjobs (status), bpeek (current output), bkill (cancel) and bhist (history).

6.3 Submitting a job with SLURM

SLURM jobs are usually written as a short shell script whose #SBATCH comment lines carry the options, then submitted with sbatch.

#!/bin/bash
#SBATCH --job-name=place_aes
#SBATCH --partition=normal
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=06:00:00
#SBATCH --output=logs/%x.%j.log

python3 flow/run_stage.py place config/aes_core.json

sbatch --wait place_aes.sh

--cpus-per-task requests cores on one node for a single multi-threaded process, which matches how EDA tools run. In the output name, %x is the job name and %j the job ID. --wait plays the role of LSF's -K: sbatch does not return until the job ends, and then returns its exit status. The companion commands are squeue (waiting and running jobs), scancel (cancel) and sacct (accounting, including peak memory of finished jobs).

6.4 Asking for the right amount

A resource request is a promise in both directions. Asking for too little gets the job killed or slowed; asking for too much makes it wait longer for a host and keeps resources idle that other jobs could have used.

• Cores: the request should match the tool's own thread setting. In OpenROAD this is set_thread_count; commercial tools have equivalent commands. A job that asks the farm for eight cores while the tool runs on one wastes seven. A tool told to use sixteen threads in an eight-core slot competes with other jobs on the host.
• Memory: the best estimate is the measured peak of a previous run of the same stage on the same block, plus a margin. The scheduler's accounting (bjobs -l or bhist -l in LSF, sacct in SLURM) reports it.
• Run time: a limit protects the farm from a hung job. It should be generous enough not to kill a normal run.

It is recommended that the flow, not the engineer, choose these numbers per stage and per block size, from the configuration file of lesson 3. Thus every run of a stage asks for the same resources, and the request can be tuned in one place.

6.5 Licences as a shared resource

Commercial EDA tools need a licence seat to run. A licence server – FlexNet Publisher is the most common licensing system in EDA – holds a fixed count of each feature, and every running tool checks a seat out and returns it on exit. Some tools need additional seats or separate features when running with many threads or in distributed mode.

Licences are expensive, so a project usually has far fewer seats than the farm has cores. In such a case, licences, not machines, decide how many jobs can run at once. If a job starts and no seat is free, the tool might wait for one or might exit with an error, depending upon how it is configured. Either behaviour wastes a slot on the farm.

The better approach is to tell the scheduler about licences, so that it does not dispatch a job until a seat is free. SLURM supports this directly with --licenses=name:count for licences the administrators have defined. LSF sites typically add licences as site-defined resources used in rusage[], or use IBM's License Scheduler add-on. The exact names are site configuration, so the flow should read them from configuration rather than hard-code them.

6.6 Running corners in parallel

Signoff STA (lesson 9) must check several corners and modes. Each corner is an independent run on the same routed database, which makes corners the natural case for parallel jobs.

Two common ways to launch the corners are a scheduler job array and a parallel Makefile. A SLURM array runs one script several times with a different index:

#SBATCH --array=0-3
CORNERS=(ss_0p72v_125c ss_0p72v_m40c ff_0p88v_m40c tt_0p80v_25c)
python3 flow/run_stage.py sta config/aes_core.json \
        --corner ${CORNERS[$SLURM_ARRAY_TASK_ID]}

LSF has the equivalent with -J "sta[1-4]" and the variable LSB_JOBINDEX. Alternatively, each corner is its own Makefile target (lesson 4) whose recipe uses bsub -K or sbatch --wait, and make -j 4 submits all four. The Makefile approach keeps the dependency logic in one place, which is why lesson 10 uses it.

6.7 Interview questions

Why use bsub -K or sbatch --wait inside a Makefile? Without them the submission command returns as soon as the job is queued, with a success status. make would then continue to the next stage before the current one has run, and would never see a failure.

Why should a job's core request match the tool's thread setting? Extra cores are reserved and left idle, while too few cores cause the tool's threads to compete with other jobs on the same host. Both waste farm capacity and make run times unpredictable.

How would you size a memory request? From the measured peak memory of an earlier run of the same stage on the same block, plus a margin, read from the scheduler's accounting data.

Why can licences limit throughput more than machines? A project typically has fewer seats of a tool than the farm has free slots, and each running job holds a seat. Once all seats are out, further jobs cannot do useful work however many hosts are free.