A note first: the final text and the illustrations are generated, of course. I have been using LLMs since the GPT-3 preview, so it feels odd to even make the comment. The point is that the thought to blog idea from New Blog Layout just works. The part I really love is the verification, which runs against a containerized GCS/OCS cluster.

Where a parallel job's tasks land is decided by the allocation_rule of the parallel environment. That is an admin setting, so a user who needs a different layout needs a different PE. It is how clusters end up with mpi_fillup, mpi_rr and mpi_single sitting side by side.

OCS 9.1.1 added qsub -par. It overrides the rule per job. One PE, any layout.

One parallel environment instead of one per layout

Setup

Three hosts, four slots each, one PE:

$ qconf -sp mpi
pe_name              mpi
allocation_rule      $fill_up
control_slaves       TRUE
job_is_first_task    TRUE

job_is_first_task TRUE means the job script counts as one of the tasks, so -pe mpi 6 is exactly six slots.

Overriding the rule

-par takes the same values as allocation_rule. The job reads its layout from $PE_HOSTFILE, and that file is what the output below shows.

The PE default is $fill_up. It fills one host and spills onto the next:

$ qsub -pe mpi 6 job.sh
ocs-worker2 4 all.q@ocs-worker2 UNDEFINED
ocs-worker1 2 all.q@ocs-worker1 UNDEFINED

Spread it instead:

$ qsub -pe mpi 6 -par $round_robin job.sh
ocs-master  2 all.q@ocs-master  UNDEFINED
ocs-worker2 2 all.q@ocs-worker2 UNDEFINED
ocs-worker1 2 all.q@ocs-worker1 UNDEFINED

One task per host:

$ qsub -pe mpi 3 -par 1 job.sh
ocs-worker1 1 all.q@ocs-worker1 UNDEFINED
ocs-worker2 1 all.q@ocs-worker2 UNDEFINED
ocs-master  1 all.q@ocs-master  UNDEFINED

Everything on one machine:

$ qsub -pe mpi 4 -par $pe_slots job.sh
ocs-worker1 4 all.q@ocs-worker1 UNDEFINED

The same PE placing tasks four different ways depending on the -par value

-par redistributes slots, it does not create them. -pe mpi 6 -par 1 on three hosts sits in qw forever. The request is legal, so there is no error at submit time. Turn on schedd_job_info and qstat -j will tell you why.

Different rules for master and slaves

The master task is the job script. The slaves are the ranks it starts. -scope switches which of them the following options apply to: global (the default), master, slave.

Master alone on its host, slaves packed four to a host:

$ qsub -pe mpi 9 -scope master -par 1 -scope slave -par 4 job.sh
$ qstat -g t
all.q@ocs-master   MASTER
all.q@ocs-worker1  SLAVE  SLAVE  SLAVE  SLAVE
all.q@ocs-worker2  SLAVE  SLAVE  SLAVE  SLAVE

The master task alone on one host, four slave tasks on each of the other hosts

The rules mix freely:

$ qsub -pe mpi 5 -scope master -par 1 -scope slave -par $fill_up job.sh
all.q@ocs-worker1  MASTER
all.q@ocs-worker2  SLAVE  SLAVE  SLAVE  SLAVE

Master host means wherever the master task landed. It has nothing to do with the qmaster host. -scope also scopes -l and -q, so the master and the workers can ask for different resources and different queues.

ign_sreq_on_mhost

-scope slave -l gpu=1 says every slave needs a GPU. Under $fill_up the master host carries slaves too, and they demand GPUs on the host you meant to keep as a launcher node.

Two GPUs per host, master pinned so the comparison is clean:

$ qsub -pe mpi 6 -par $fill_up -scope master -l h=ocs-master \
       -scope slave -l gpu=1 job.sh

The default is ign_sreq_on_mhost FALSE. Five slaves, five GPUs, two of them charged on the master host:

free GPUs:  ocs-master 0   ocs-worker1 0   ocs-worker2 1

With ign_sreq_on_mhost TRUE the two slaves on the master host consume nothing:

free GPUs:  ocs-master 2   ocs-worker1 1   ocs-worker2 0

GPU accounting on the master host with ign_sreq_on_mhost set to FALSE and to TRUE

Set it when the master forks its local workers as threads or subprocesses, so charging them again would count them twice. Pair it with master_forks_slaves TRUE, which multiplies the queue limits of the master by its slot count. Otherwise a master running three forked workers is still held to the memory limit of a single slot and gets killed.

One caveat, confirmed on 9.1.5. The manual says slave requests are ignored on the master host, but only the consumption is skipped. The capacity check still applies. This job pends at 2 GPUs per host and runs at 3, and it still reports all 3 as free while it runs:

$ qsub -pe mpi 4 -par $pe_slots -scope slave -l gpu=1 job.sh

So size the master host for the slave slots it may hold. The resource is a gate, not a bill.

Cheat sheet

qsub -pe <pe> <n> -par $fill_up          pack hosts one at a time
qsub -pe <pe> <n> -par $round_robin      one slot per host, cycling
qsub -pe <pe> <n> -par 1                 one task per host
qsub -pe <pe> <n> -par <k>               k tasks per host
qsub -pe <pe> <n> -par $pe_slots         everything on one host

qsub -pe <pe> <n> -scope master -par 1 \
                  -scope slave  -par 4   master alone, slaves packed 4/host

qsub -pe <pe> <n> -scope master -q io.q -l mem=1G \
                  -scope slave  -q gpu.q -l gpu=1   different queues/resources

PE settings that go with a scoped master task:

ign_sreq_on_mhost    TRUE    master host is not charged for local slave tasks
master_forks_slaves  TRUE    queue limits of the master scale with its slot count
daemon_forks_slaves  TRUE    same idea, applied to the slave hosts

-par and -scope work on qsub, qsh, qrsh, qlogin and qalter. A cluster now needs one PE per integration style, not one per layout.