Changes between Version 5 and Version 6 of UCERF3 Figure Reproducibility

Timestamp:

Oct 31, 2014, 2:22:50 PM (10 years ago)

Author:

Kevin Milner

Comment:

--

UCERF3 Figure Reproducibility

@@ -54 +54 @@
 = UCERF3-TD Report Supplements: =
 Go through contents of http://opensha.usc.edu/ftp/TimeDependentPreliminary/ with Kevin and Peter to figure out where everything ultimately came from.
+
+= Running "Compound Plots" =
+Many UCERF3 plots require calculations from the entire model, including all logic tree branch choices. The OpenSHA class scratch.UCERF3.analysis.CompoundFSSPlots can be used to generate plots across all of these branches. Practically speaking, high performance computing is needed to process the entire model in parallel. All "Compound" plots in the UCERF3 reports were computing using high performance computing resources and the class scratch.UCERF3.analysis.MPJDistributedCompoundFSSPlots which uses the FastMPJ MPI for Java library to handle intra-node communication. An example script for running MPJDistribtedCompoundFSSPlots is given below, which will recreate the Fault System Solution MFD plots. You will need a compound fault system solution file as input, for example: http://opensha.usc.edu/ftp/kmilner/ucerf3/2013_05_10-ucerf3p3-production-10runs/2013_05_10-ucerf3p3-production-10runs_COMPOUND_SOL.zip
+
+This script is used to run MPJDistribtedCompoundFSSPlots on the Stampede supercomputer at TACC:
+
+{{{
+login3.stampede(6)$ cat plot_gen.pbs
+#!/bin/bash
+
+# this is the wall clock time
+#SBATCH -t 00:200:00
+# this is the number of CPUs - number of nodes is this number divided by 16
+#SBATCH -n 1280
+# queue we are submitting to
+#SBATCH -p normal
+
+# this is the directory where we are running
+RUN_NAME="2013_05_10-ucerf3p3-production-10runs"
+COMPOUND_DIR="/work/00950/kevinm/ucerf3/inversion/compound_plots/${RUN_NAME}"
+# this is the path to the compound solution file
+INV_DIR="/work/00950/kevinm/ucerf3/inversion/${RUN_NAME}"
+COMPOUND_FILE="${INV_DIR}/${RUN_NAME}_COMPOUND_SOL.zip"
+# this is the set of plots that we want to generate
+PLOTS="--plot-mfds"
+# the number of threads per compute node
+THREADS="4"
+# the minimum number of jobs sent to a given node at a given time, typically this can be equal to the number of threads
+MIN_DISPATCH="4"
+# the amount of memory allocated to the JVM, should be as much as possible
+MEMORY="25G"
+
+# this gets the node list
+PBS_NODEFILE="/tmp/${USER}-hostfile-${SLURM_JOBID}"
+echo "creating PBS_NODEFILE: $PBS_NODEFILE"
+scontrol show hostnames $SLURM_NODELIST > $PBS_NODEFILE
+
+# paths to FastMPJ
+export FMPJ_HOME=/home1/00950/kevinm/FastMPJ
+export PATH=$PATH:$FMPJ_HOME/bin
+
+# this converts the node list to a format which FastMPJ can read
+if [[ -e $PBS_NODEFILE ]]; then
+  #count the number of processors assigned by PBS
+  NP=`wc -l < $PBS_NODEFILE`
+  echo "Running on $NP processors: "`cat $PBS_NODEFILE`
+else
+  echo "This script must be submitted to PBS with 'qsub -l nodes=X'"
+  exit 1
+fi
+
+# make sure there's at least one node
+if [[ $NP -le 0 ]]; then
+  echo "invalid NP: $NP"
+  exit 1
+fi
+
+date
+echo "RUNNING FMPJ"
+# this runs MPJDistributedCompoundFSSPlots in parallel with the previously selected arguments
+fmpjrun -machinefile $PBS_NODEFILE -np $NP -dev niodev -Djava.library.path=$FMPJ_HOME/lib -Djava.awt.headless=true -Xmx${MEMORY} -cp ${COMPOUND_DIR}/OpenSHA_complete.jar:${COMPOUND_DIR}/parallelcolt-0.9.4.jar:${COMPOUND_DIR}/commons-cli-1.2.jar:${COMPOUND_DIR}/csparsej.jar -class scratch.UCERF3.analysis.MPJDistributedCompoundFSSPlots --threads ${THREADS} --min-dispatch ${MIN_DISPATCH} $PLOTS ${COMPOUND_FILE} ${COMPOUND_DIR}
+ret=$?
+
+date
+echo "DONE with process 0. EXIT CODE: $ret"
+
+exit $ret
+}}}