Changes between Version 5 and Version 6 of RunningUCERF3Inversions

Timestamp:

Jan 7, 2014, 4:52:45 PM (10 years ago)

Author:

Kevin Milner

Comment:

--

RunningUCERF3Inversions

@@ -98 +98 @@
 The simplest way to run inversions on a cluster is to run a single inversion on each compute node, with a single submission script for each job. Sample job scripts are given below for both HPCC and Stampede. Paths and arguments would need to be modified for the user running the job.
 
-=== HPCC Example PBS SCript ===
+=== HPCC Example Single Node PBS SCript ===
 {{{
 #!/bin/bash
@@ -108 +108 @@
 exit $?
 }}}
+
+=== Stampede Example Single Node PBS SCript ===
+{{{
+#!/bin/bash
+
+#SBATCH -t 00:360:00
+#SBATCH -n 16
+#SBATCH -p normal
+
+/home1/00950/kevinm/java/default/bin/java -Djava.awt.headless=true -Xmx25000M -Xms25000M -cp /work/00950/kevinm/ucerf3/inversion/2013_05_03-ucerf3p3-production-first-five/OpenSHA_complete.jar:/work/00950/kevinm/ucerf3/inversion/parallelcolt-0.9.4.jar:/work/00950/kevinm/ucerf3/inversion/commons-cli-1.2.jar:/work/00950/kevinm/ucerf3/inversion/csparsej.jar scratch.UCERF3.inversion.CommandLineInversionRunner --completion-time 5h --sub-completion 1s --cool FAST_SA --nonneg LIMIT_ZERO_RATES --num-threads 5 --branch-prefix FM3_1_ABM_EllB_DsrTap_CharConst_M5Rate6.5_MMaxOff7.3_NoFix_SpatSeisU2_run0 --directory /work/00950/kevinm/ucerf3/inversion/2013_05_03-ucerf3p3-production-first-five --no-plots
+exit $
+}}}
+
+== Bundled Large MPI Jobs ==
+Some schedulers give preference to single large jobs over many small jobs. They may also have limits on the total number of jobs that can be submitted. You can get around this limitation by submitting a single MPI job that runs many inversions on many nodes. This has the added benefit of allowing you to run multiple inversions per node if enough processors/memory is available. UCERF3 production runs were run with this method on the Stampede supercomputer with 3 inversions per node.
+
+To use this method, your PBS script must call scratch.UCERF3.simulatedAnnealing.hpc.MPJInversionDistributor. You must also supply the "--exact-dispatch X" (where X is the number of threads per node). The total number of inversions must be less than X*NODES, so with 3 threads per node and 256 nodes, you can submit at most 768 inversions. You must also supply an xml file argument, which is described below. Additionally, you must download and install FastMPJ in your user account as this library is required.
+
+=== Stampede Batch PBS Script ===
+{{{
+#!/bin/bash
+
+#SBATCH -t 00:420:00
+#SBATCH -n 2048
+#SBATCH -p normal
+
+PBS_NODEFILE="/tmp/${USER}-hostfile-${SLURM_JOBID}"
+echo "creating PBS_NODEFILE: $PBS_NODEFILE"
+scontrol show hostnames $SLURM_NODELIST > $PBS_NODEFILE
+
+export FMPJ_HOME=/home1/00950/kevinm/FastMPJ
+export PATH=$PATH:$FMPJ_HOME/bin
+
+if [[ -e $PBS_NODEFILE ]]; then
+  #count the number of processors assigned by PBS
+  NP=`wc -l < $PBS_NODEFILE`
+  echo "Running on $NP processors: "`cat $PBS_NODEFILE`
+else
+  echo "This script must be submitted to PBS with 'qsub -l nodes=X'"
+  exit 1
+fi
+
+if [[ $NP -le 0 ]]; then
+  echo "invalid NP: $NP"
+  exit 1
+fi
+
+date
+echo "RUNNING FMPJ"
+fmpjrun -machinefile $PBS_NODEFILE -np $NP -dev niodev -Djava.library.path=$FMPJ_HOME/lib -Djava.awt.headless=true -Xmx25000M -Xms25000M -cp /work/00950/kevinm/ucerf3/inversion/2013_05_03-ucerf3p3-production-first-five/OpenSHA_complete.jar:/work/00950/kevinm/ucerf3/inversion/parallelcolt-0.9.4.jar:/work/00950/kevinm/ucerf3/inversion/commons-cli-1.2.jar:/work/00950/kevinm/ucerf3/inversion/csparsej.jar  -class scratch.UCERF3.simulatedAnnealing.hpc.MPJInversionDistributor --exact-dispatch 3 /work/00950/kevinm/ucerf3/inversion/2013_05_03-ucerf3p3-production-first-five/batch00.xml
+ret=$?
+
+date
+echo "DONE with process 0. EXIT CODE: $ret"
+
+exit $ret
+}}}
+
+=== XML Input File ===
+The XML input file simply supplies a list of arguments for each inversion. This is an example for 384 inversions. The "num" argument at the end is a sanity check which verifies the correct number of arguments.
+{{{
+<?xml version="1.0" encoding="UTF-8"?>
+
+<OpenSHA>
+  <InversionConfigurations num="384">
+    <InversionConfiguration index="0" args="--completion-time 5h --sub-completion 1s --cool FAST_SA --nonneg LIMIT_ZERO_RATES --num-threads 5 --branch-prefix FM3_1_ABM_Shaw09Mod_DsrUni_CharConst_M5Rate6.5_MMaxOff7.3_NoFix_SpatSeisU2_run0 --directory /work/00950/kevinm/ucerf3/inversion/2013_05_03-ucerf3p3-production-first-five --no-plots" num="15"/>
+    <InversionConfiguration index="1" args="--completion-time 5h --sub-completion 1s --cool FAST_SA --nonneg LIMIT_ZERO_RATES --num-threads 5 --branch-prefix FM3_1_ABM_Shaw09Mod_DsrUni_CharConst_M5Rate6.5_MMaxOff7.3_NoFix_SpatSeisU3_run0 --directory /work/00950/kevinm/ucerf3/inversion/2013_05_03-ucerf3p3-production-first-five --no-plots" num="15"/>
+    <InversionConfiguration index="2" args="--completion-time 5h --sub-completion 1s --cool FAST_SA --nonneg LIMIT_ZERO_RATES --num-threads 5 --branch-prefix FM3_1_ABM_Shaw09Mod_DsrUni_CharConst_M5Rate6.5_MMaxOff7.6_NoFix_SpatSeisU2_run0 --directory /work/00950/kevinm/ucerf3/inversion/2013_05_03-ucerf3p3-production-first-five --no-plots" num="15"/>
+    ...
+    <InversionConfiguration index="383" args="--completion-time 5h --sub-completion 1s --cool FAST_SA --nonneg LIMIT_ZERO_RATES --num-threads 5 --branch-prefix FM3_1_ZENGBB_Shaw09Mod_DsrTap_CharConst_M5Rate6.5_MMaxOff7.9_NoFix_SpatSeisU3_run0 --directory /work/00950/kevinm/ucerf3/inversion/2013_05_03-ucerf3p3-production-first-five --no-plots" num="15"/>
+  </InversionConfigurations>
+</OpenSHA>
+    
+}}}