Production Campaign Workflows & Cluster Orchestration on Synthetic Injections (SLURM & HTCondor)

This guide covers how to set up, launch, and post-process multi-injection, multi-track Parameter Estimation (PE) campaign workflows on synthetic gravitational-wave injections in jaxpe across cluster scheduling systems—specifically SLURM GPU clusters (bin/slurm/) and HTCondor CPU pools (bin/condor/).

1. Overview of a Synthetic Injection Campaign Workflow Tree

A typical campaign workflow evaluates Bayesian parameter estimation algorithms across $N$ independent synthetic injections (e.g. 100 zero-noise or Gaussian-noise simulated BBH signals) across multiple PE method tracks (e.g., phenomd_hmc, phenomd_gpry, esigma_gpry).

Synthetic injection campaign workflows serve as the primary validation mechanism for parameter estimation frameworks: because the exact true parameters $\boldsymbol{\theta}_{\rm true}$ are known for synthetic signals, recovery accuracy, posterior calibration (P-P plots), and algorithm convergence rates can be systematically measured across sampler choices and waveform approximants.

The synthetic injection campaign workflow directory structure is organized as follows:

campaign_gpu/ (or campaign/)
├── injections/         # Synthetic injection JSON files (inj_0.json ... inj_99.json) + config.json
├── slurm/ (or condor/) # Workflow orchestration submit files (.slurm or .sub/.dag) and helper scripts
├── results/            # PE posterior outputs per variant (phenomd_hmc/, phenomd_gpry/, esigma_gpry/)
├── report/             # HTML report gallery, WebP thumbnails, and P-P plots
├── plan.md             # Campaign workflow plan & execution log
└── note.md             # Cluster/scheduler comparison notes

2. SLURM GPU Synthetic Campaign Workflow (`bin/slurm/`)

Step 0: Activate Environment

Activate the jaxpe conda environment containing JAX, CUDA, and optional waveform models (esigmapy, diffrax, numba, numbalsoda):

conda activate jaxpe

Step 1: Generate Synthetic Injections for the Workflow

Draw $N$ reproducible synthetic injections from the prior/target SNR distribution using a fixed seed (e.g., 20260807):

jaxpe generate-injections \
    --config examples/configs/campaign_10to80_aligned.json \
    --n-injections 100 \
    --network H1,L1 \
    --noise zero \
    --psd aligo \
    --seed 20260807 \
    --target-snr-range 10 20 \
    --target-snr-waveform esigma \
    --outdir campaign_gpu/injections

Note: Using --noise zero generates synthetic strain data containing purely the synthetic signal $h(t; \boldsymbol{\theta}_{\rm true})$, allowing deterministic likelihood benchmarking without random noise realizations.

Step 2: Generate SLURM Workflow Submit Scripts

Run bin/slurm/generate_campaign_slurm.py to build the workflow submit files:

python bin/slurm/generate_campaign_slurm.py \
    --config examples/configs/campaign_10to80_aligned.json \
    --injections-dir campaign_gpu/injections \
    --results-dir campaign_gpu/results \
    --slurm-dir campaign_gpu/slurm \
    --report-dir campaign_gpu/report \
    --n-injections 100 \
    --partition gpu \
    --max-concurrent 4

This creates the full workflow configuration:

phenomd_hmc.slurm, phenomd_gpry.slurm, esigma_gpry.slurm: SLURM Job Array templates (#SBATCH --array=0-99%4) for synthetic PE recovery.
postprocess_<variant>.slurm: Track-level post-processing script for individual variants.
postprocess.slurm: Combined comparative post-processing script across all tracks.
submit_<variant>.sh: Track-specific submitter queueing a PE job array and its dependent track post-processing (postprocess_<variant>.slurm).
submit_all.sh: Master orchestration script that submits all track job arrays, queues track-level post-processing as each track completes (afterok:JOB_<variant>), and triggers final combined comparative post-processing (afterok:POST_HMC:POST_GPRY:POST_ESIGMA).
submit_smoke.sh: Helper script to launch and verify the 3-job synthetic smoke test workflow.

Step 3: Launching Synthetic PE Workflows on SLURM

Run Synthetic Smoke Test Workflow:

bash campaign_gpu/slurm/submit_smoke.sh

Run Single Track Workflow (PE + Track-Level Postprocessing):
```
bash campaign_gpu/slurm/submit_esigma_gpry.sh
```
Run Full 300-Job Campaign Workflow (PE + Track Postprocessing + Final Combined Comparison):
```
bash campaign_gpu/slurm/submit_all.sh
```

3. HTCondor CPU Synthetic Campaign Workflow (`bin/condor/`)

For HTCondor clusters, generate the submit files and DAGs for the synthetic injection recovery workflow via bin/condor/generate_campaign_dag.py:

python bin/condor/generate_campaign_dag.py \
    --config examples/configs/campaign_10to80_aligned.json \
    --injections-dir campaign/injections \
    --results-dir campaign/results \
    --n-injections 100 \
    --condor-dir campaign/condor

Submit the workflow via HTCondor DAGMan:

condor_submit_dag campaign/condor/campaign.dag

4. Post-Processing & P-P Plot Diagnostics for Synthetic Injection Workflows

Synthetic injection campaign workflows enable rigorous statistical validation of parameter estimation algorithms:

P-P (Probability-Probability) Plots: Because the synthetic injected truths $\boldsymbol{\theta}_{\rm true}$ are exact, post-processing computes the percentile rank (credible level) of the true parameter within each recovered posterior. For a well-calibrated inference framework, these percentile ranks follow a Uniform $(0, 1)$ distribution, tested against Beta order statistics and Kolmogorov-Smirnov (KS) $p$-values.
Corner Plots & Gallery: Generates thinned corner plots comparing recovered posterior distributions against the synthetic injected truths (marked as truth lines/crosses).

Post-processing runs automatically as a dependent job in the workflow (postprocess.slurm or DAG POSTPROCESS node). It can also be run manually:

python bin/postprocess_campaign.py \
    --results-dir campaign_gpu/results \
    --outdir campaign_gpu/report

To view the generated report, open campaign_gpu/report/index.html in any browser.