Baseline Pattern: Single-Node Service + Workers

Your Compose stack, before

The baseline is a three-part Compose stack, sketched in Scheduler as Orchestrator: a coordinator that hands out work, workers that do it, and a one-shot check of the results. In Compose, each part is a service, and depends_on orders them.

The same stack in one allocation

One batch script asks the scheduler for one node. Inside that allocation it does what Compose did on your laptop: it makes a private job-scoped scratch folder, starts the coordinator, waits at the readiness gate, runs the workers, verifies the results, and cleans up.

The workflow source is the code that script and its helpers run. It is the complete, release-pinned package at workflows/baseline-slurm-apptainer in tag v0.1.0:

• slurm/baseline.sbatch, the launch script;
• bin/, the small Python programs the script starts: the coordinator, the worker, the readiness probe, the input validator, and the verifier;
• container/, the recipe and lock file for the image those programs run in;
• inputs/ and expected/, the work to do and the answers to check it against.

It serves two purposes. It is a demonstrator: it shows the pattern running end to end under a real scheduler, and the evidence badge on this page links to the test record of that run. It is also a template: you edit it to run your own application, using the list in “Adapt it”. This page shows only teaching excerpts, annotated fragments chosen for explanation. Read them in the annotated implementation reference, and open the package for the runnable code. The coordinator and worker process inert JSON tasks on purpose. They are placeholders that exercise the orchestration; they are not an application.

How the baseline works

The script moves through six steps. The table says what each step does and where it sits in baseline.sbatch (line numbers are from tag v0.1.0).

The readiness gate proves the coordinator can serve. By default it polls for up to 30 seconds and passes only on the exact expected answer. It does not prove that any result is right. That is verification, and it runs inside the job, as step 5 of the same script (baseline.sbatch lines 245–251), not as a separate command afterward. A running process is not yet a successful result.

#!/usr/bin/env bash
#SBATCH --job-name=bssw-baseline
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --time=00:10:00
#SBATCH --output=slurm-%j.out

set -Eeuo pipefail
# …
WORKER_COUNT=${WORKER_COUNT:-2}
TASK_COUNT=${TASK_COUNT:-4}
READINESS_TIMEOUT=${READINESS_TIMEOUT:-30}
CLEANUP_TIMEOUT=${CLEANUP_TIMEOUT:-10}
INPUT_PATH=${INPUT_PATH:-${WORKFLOW_ROOT}/inputs/tasks.json}
EXPECTED_PATH=${EXPECTED_PATH:-${WORKFLOW_ROOT}/expected/results.json}
IMAGE_PATH=${IMAGE_PATH:-${WORKFLOW_ROOT}/container/baseline.sif}
INPUT_SHA256_FILE=${INPUT_SHA256_FILE:-${WORKFLOW_ROOT}/inputs/tasks.json.sha256}
IMAGE_SHA256_FILE=${IMAGE_SHA256_FILE:-${IMAGE_PATH}.sha256}

step_exec "$APPTAINER_CMD" exec --cleanenv --bind "${RUNTIME_DIR}:/work" "$IMAGE_PATH" \
  python3 "$APP_ROOT/coordinator.py" --input /work/input.json \
  --results /work/results.json --endpoint-file /work/endpoint.json \
  > "$RUNTIME_DIR/logs/coordinator.log" 2>&1 &
COORDINATOR_PID=$!
CHILDREN+=("$COORDINATOR_PID")
emit_event "coordinator-start"

ENDPOINT_DEADLINE=$((SECONDS + READINESS_TIMEOUT))
while [[ ! -s "$RUNTIME_DIR/endpoint.json" ]]; do
  "$KILL_CMD" -0 "$COORDINATOR_PID" 2>/dev/null || \
    fail "BSSW-COORDINATOR-EXIT" 74 "the coordinator exited before publishing its endpoint"
  (( SECONDS < ENDPOINT_DEADLINE )) || \
    fail "BSSW-READY-TIMEOUT" 75 "the coordinator did not become semantically ready before the deadline"
  sleep 0.05
done
ENDPOINT=$(container_exec python3 -c \
  'import json; print(json.load(open("/work/endpoint.json", encoding="utf-8"))["endpoint"])') || \
  fail "BSSW-COORDINATOR-EXIT" 74 "the coordinator endpoint record is invalid"
container_exec python3 "$APP_ROOT/readiness.py" --endpoint "$ENDPOINT" \
  --task-count "$TASK_COUNT" --timeout "$READINESS_TIMEOUT" \
  > "$RUNTIME_DIR/logs/readiness.log" 2>&1 || \
  fail "BSSW-READY-TIMEOUT" 75 "the coordinator did not become semantically ready before the deadline"
"$KILL_CMD" -0 "$COORDINATOR_PID" 2>/dev/null || \
  fail "BSSW-COORDINATOR-EXIT" 74 "the coordinator exited immediately after readiness"
emit_event "coordinator-ready"

WORKER_PIDS=()
for ((index=1; index<=10#$WORKER_COUNT; index++)); do
  printf -v worker_id 'worker-%02d' "$index"
  step_exec "$APPTAINER_CMD" exec --cleanenv --bind "${RUNTIME_DIR}:/work" "$IMAGE_PATH" \
    python3 "$APP_ROOT/worker.py" --endpoint "$ENDPOINT" --worker-id "$worker_id" \
    --max-wait "$READINESS_TIMEOUT" > "$RUNTIME_DIR/logs/${worker_id}.log" 2>&1 &
  worker_pid=$!
  CHILDREN+=("$worker_pid")
  WORKER_PIDS+=("$worker_pid")
  emit_event "worker-start"
done
for worker_pid in "${WORKER_PIDS[@]}"; do
  if ! wait "$worker_pid"; then
    fail "BSSW-WORKER-EXIT" 76 "at least one exclusive worker step failed"
  fi
done
"$KILL_CMD" -0 "$COORDINATOR_PID" 2>/dev/null || \
  fail "BSSW-COORDINATOR-EXIT" 74 "the coordinator exited before result verification"
container_exec python3 "$APP_ROOT/verify.py" --input /work/input.json \
  --results /work/results.json --expected /work/expected.json --output /work/result.json \
  > "$RUNTIME_DIR/logs/verification.log" 2>&1 || \
  fail "BSSW-VERIFY-RESULT" 77 "result schema, count, uniqueness, or expected content verification failed"
[[ -s "$RESULT_PATH" ]] || \
  fail "BSSW-VERIFY-RESULT" 77 "the verifier did not atomically create the success result"
emit_event "verification-success"
printf 'BSSW workflow completed: result is in the job-scoped runtime directory.\n'

deadline = time.monotonic() + args.timeout
while time.monotonic() < deadline:
    try:
        with urllib.request.urlopen(f"{args.endpoint}/health") as response:
            health = json.load(response)
        if response.status == 200 and health == {
            "status": "ready",
            "schemaVersion": 1,
            "taskCount": args.task_count,
        }:
            raise SystemExit(0)   # ready
    except (OSError, ValueError, urllib.error.URLError):
        pass
    time.sleep(min(args.interval, deadline - time.monotonic()))
raise SystemExit(24)   # timed out

def health(self) -> dict:
    return {
        "status": "ready",
        "schemaVersion": 1,
        "taskCount": len(self.tasks),
    }

def claim(self) -> dict:
    # A worker asks for work; the coordinator hands out one task
    # or reports that all tasks are done (or that more may arrive).
    if self.pending:
        task_id = self.pending.popleft()
        return {"status": "task", "task": self.tasks[task_id]}
    status = "done" if len(self.results) == len(self.tasks) else "wait"
    return {"status": status}

# Load tasks, actual results, and the expected results.
tasks = load_tasks(args.input)
actual = json.load(open(args.results))["results"]
expected = json.load(open(args.expected))["results"]

seen = set()
for result in actual:
    if result["taskId"] in seen:
        raise ValueError("duplicate task id")
    if result["taskId"] not in {e["taskId"] for e in expected}:
        raise ValueError("unexpected task id")
    seen.add(result["taskId"])

# Every task must be covered exactly once.
if seen != {task["id"] for task in tasks}:
    raise ValueError("results do not cover each task exactly once")

# Atomically write the success marker.
success = {
    "schemaVersion": 1,
    "status": "success",
    "taskCount": len(tasks),
    "results": sorted(actual, key=lambda r: r["taskId"]),
}
atomic_json(args.output, success)

Files the workflow reads and writes

Every JSON file here is machine-parsed: it is written for programs to read. People write two of them (inputs/tasks.json, expected/results.json), and you open two to confirm success (result.json, events.jsonl); nothing else needs reading.

Warning: results.json and result.json differ by one letter and mean different things. The first is intermediate worker output. The second is the success marker.

The job writes its run files (input.json, expected.json, endpoint.json, results.json, result.json, events.jsonl, versions.txt, and a logs/ folder) into a private bssw-<job id> folder under the scratch root you choose.

Procedure

1. Get the code, check the prerequisites listed at the start of this module, then record your local values.
2. Get the container image and check the input.
3. Set the adapter values that differ at your center: account, partition, and scratch root.
4. Submit the job once, with the exact command on that page. Keep account names, private hosts, and sensitive paths out of logs.
5. Check that it worked.

Expected result

A successful run:

• exits zero and prints BSSW workflow completed: result is in the job-scoped runtime directory.;
• leaves result.json in the run folder with "status": "success", the only success marker;
• has an events.jsonl in which coordinator-ready comes before every worker-start, which is the written proof that no worker started early;
• reports that every task has exactly one result and that every invariant passed.

A zero exit alone is not success. Check result.json. The test record linked from this page’s evidence badge shows what a real run produced.

What varies by center

Check each of these with your center before you run or adapt the workflow:

• Account and partition. The #SBATCH header needs values your center accepts. Ask your center’s support, or read its job-script guide.
• Time, CPU, and memory limits. The defaults are small. Raise --cpus-per-task before you raise WORKER_COUNT.
• Apptainer. How the compute nodes provide it, and the module name. Run command -v apptainer in a job.
• Scratch root. Set BSSW_SCRATCH_ROOT to a private, job-scoped path your center approves. Do not rely on a fallback to /tmp on a shared node.
• sha256sum. It must be on the compute node’s path, or you override BSSW_SHA256_CMD.
• Exclusive single-task job steps. Some centers restrict them. A rejected srun step is the symptom.
• Loopback. The coordinator and workers talk over loopback on one node. Reaching the coordinator from outside the node needs access your center approves.

Failure modes

Every failure prints a stable ID that begins BSSW- and exits nonzero. Match the ID in your job output to its entry in Troubleshooting:

• Scheduler command or job context missing: BSSW-PREQ-SLURM
• Apptainer missing on the compute node: BSSW-PREQ-APPTAINER
• Unusable scratch root or bind: BSSW-PREQ-STORAGE
• Input checksum does not match: BSSW-INTEGRITY-INPUT
• Image checksum does not match: BSSW-INTEGRITY-IMAGE
• Coordinator did not become ready in time: BSSW-READY-TIMEOUT
• A worker step failed: BSSW-WORKER-EXIT
• Results were rejected: BSSW-VERIFY-RESULT

A readiness timeout stops the job at the gate: no worker starts and no result.json is written. That is the gate doing its job. Troubleshooting lists the remaining IDs.

Adapt it

Two checks guard a run, and they adapt differently:

• The readiness gate is part of the pattern. It replaces Compose depends_on: workers must not start until the service can really serve. Keep the gate and change only what “ready” means for your service.
• Result verification is part of your application. Compose has no equivalent. For your own application, expect to keep a verification step but not this one: replace bin/verify.py with your own definition of correct. Remove it only if a zero exit truly means success for you, which is rarely true. A process can exit cleanly after producing wrong, partial, or stale output, and nothing else in the pattern checks that.

To run your own application, expect to edit 12 files and to set up to 14 adapter variables. The files are slurm/baseline.sbatch (directive values and launch lines), inputs/tasks.json, inputs/tasks.json.sha256, expected/results.json (or delete it), bin/coordinator.py, bin/worker.py, bin/verify.py, bin/validate_input.py, bin/readiness.py, bin/common.py, container/Apptainer.def, and container/image-lock.json. These stay as they are: container/acquire.sh, diagnostics.json, and README.md. The 14 adapter variables are the ones in the package README’s adapter table: WORKER_COUNT, TASK_COUNT, READINESS_TIMEOUT, CLEANUP_TIMEOUT, IMAGE_PATH, IMAGE_SHA256_FILE, INPUT_PATH, INPUT_SHA256_FILE, EXPECTED_PATH, BSSW_SCRATCH_ROOT, BSSW_APPTAINER_CMD, BSSW_SRUN_CMD, BSSW_SLURM_VERSION_CMD, and BSSW_SHA256_CMD.

For a step-by-step translation of your own Compose file, use Compose-to-HPC translation.

Safety and scope limitations

The baseline covers one node, loopback communication, bounded inert inputs, and ephemeral allocation-scoped state. It does not establish multi-node networking, durable service operation, cross-center portability, performance scaling, or production support. GPUs, more than one node, and PBS-family schedulers are Designed, not run. The full list is at what hasn’t been tested yet.

Key points

• The workflow source is both a demonstrator (it shows the pattern running) and a template (you edit it for your own application).
• The readiness gate belongs to the pattern. Result verification belongs to your application, and it runs inside the job.
• Every JSON file here is read by a program. Only inputs/tasks.json and expected/results.json are also written by a person.
• Success means result.json says "status": "success". A zero exit alone is not success.
• To adapt it, expect to edit 12 files and set up to 14 adapter variables.