= UCERF3-TI Report Figures, Tables, and Analyses: =

* http://opensha.usc.edu/ftp/kmilner/ucerf3/2013_05_10-ucerf3p3-production-10runs/ This directory contains outputs [#RunningCompoundPlots Compound Solution Plots].
 * 2013_05_10-ucerf3p3-production-10runs_COMPOUND_SOL_FM3_1_slip_rates_parent_sections.csv
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --plot-slip-rates option
 * slip_rate_plots/
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --plot-slip-rates option
 * fss_mfd_plots/
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --plot-mfds option
 * parent_sect_mfds/
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --plot-parent-mfds option
 * small_MFD_plots/
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --plot-mfds and --plot-parent-mfds options, contains smaller more publication friendly version of other plots
 * erf_mfd_plots/
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --plot-erf-mfds option
 * gridded_participation_plots/
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --plot-gridded-participations option
 * run_branch_averages/
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --build-mean option
* http://opensha.usc.edu/ftp/pmpowers/UCERF3.3/
 * BranchRatios
 * HazardCurves
* http://opensha.usc.edu/ftp/kmilner/ucerf3/2013_05_09-ucerf3p3-convergence-test
* http://opensha.usc.edu/ftp/kmilner/ucerf3/2013_05_09-ucerf3p3-convergence-test_COMPOUND_PLOTS/
* http://opensha.usc.edu/ftp/kmilner/ucerf3/2013_05_09-ucerf3p3-branch-wt-test/
* http://opensha.usc.edu/ftp/kmilner/ucerf3/2013_05_09-ucerf3p3-branch-wt-test_COMPOUND_PLOTS/
* http://opensha.usc.edu/ftp/kmilner/ucerf3/2013_05_09-ucerf3p3-subset-noMFD-nucl1/
* http://opensha.usc.edu/ftp/kmilner/ucerf3/2013_05_09-ucerf3p3-subset-noMFD-nucl1_COMPOUND_PLOTS/
* http://opensha.usc.edu/ftp/kmilner/ucerf3/2013_05_10-ucerf3p3-production-10runs-BRANCH-AVERAGED-MEAN-SOL/BRANCH-AVERAGED-MEAN-WARNING-BE-CAREFUL-MANY-PLOTS-USELESS/parent_sect_mfds/
* Fig31_HazardMapRatios (Peter)
* Fig33and35_NumRupsAboveWaterLevelTests (Kevin)

= UCERF3-TI Report Supplements =
Go through contents of http://pubs.usgs.gov/of/2013/1165/data/UCERF3_SupplementalFiles/UCERF3.3/index.html with Kevin and Peter to figure out where everything ultimately came from.

= UCERF3-TD Report Figures, Tables, and Analyses: =
As described in the README and get* files in the Figure, Table, and Analysis directories, which access contents at the following:

* http://opensha.usc.edu/ftp/kmilner/ucerf3/2013_05_10-ucerf3p3-production-10runs/ This directory contains outputs [#RunningCompoundPlots Compound Solution Plots].
 * small_MPD_faults/30yr_combined.pdf
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --plot-erf-prob-dists option. WARNING: Requires a lot of memory, 1 thread recommended
 * small_MPD_plots/30yr_combined.pdf
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --plot-erf-prob-dists option. WARNING: Requires a lot of memory, 1 thread recommended
 * gridded_time_dep_participation_prob_plots_30/*
 * erf_mag_prob_plots/30yr/2013_05_10-ucerf3p3-production-10runs_COMPOUND_SOL_30yr_MPD_ERF_LA_BOX_Region.txt
 * region_m6p7_branch_hists/30yr/
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --plot-erf-prob-dists option. WARNING: Requires a lot of memory, 1 thread recommended
 * gridded_participation_prob_plots.xml
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --plot-gridded-partic-probs option. WARNING: Requires a lot of memory, 1 thread recommended
 * 2013_05_10-ucerf3p3-production-10runs_COMPOUND_SOL_FM3_1_MEAN_BRANCH_AVG_SOL.zip
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --build-mean option
 * 2013_05_10-ucerf3p3-production-10runs_COMPOUND_SOL_FM3_2_MEAN_BRANCH_AVG_SOL.zip
  * Generated by [#RunningCompoundPlots Running Compound Solution Plots] with the --build-mean option
* http://opensha.usc.edu/ftp/kmilner/ucerf3/TimeDependent_AVE_RI_AVE_NORM_TIME_SINCE
 * main_fault_probabilities.xls
 * m6.7_30yr/*
 * m6.7_30yr/BranchSensitivityMaps/*
 * m6.7_30yr/BranchAveragedResults/*
 * m6.7_30yr/OtherSensitivityTests/*
* http://opensha.usc.edu/ftp/kmilner/ucerf3/TimeDependent_AVE_ALL/ *
* http://opensha.usc.edu/ftp/kmilner/ucerf3/ucerf3_time_dep_erf_plots/*
* SCEC VDO Fault Model plugin
* http://opensha.usc.edu/ftp/pmpowers/UC-TimeDep/

= UCERF3-TD Report Supplements: =
Go through contents of http://opensha.usc.edu/ftp/TimeDependentPreliminary/ with Kevin and Peter to figure out where everything ultimately came from.

= Running "Compound Plots" =
Many UCERF3 plots require calculations from the entire model, including all logic tree branch choices. The OpenSHA class scratch.UCERF3.analysis.CompoundFSSPlots can be used to generate plots across all of these branches. Practically speaking, high performance computing is needed to process the entire model in parallel. All "Compound" plots in the UCERF3 reports were computing using high performance computing resources and the class scratch.UCERF3.analysis.MPJDistributedCompoundFSSPlots which uses the FastMPJ MPI for Java library to handle intra-node communication. An example script for running MPJDistribtedCompoundFSSPlots is given below, which will recreate the Fault System Solution MFD plots. You will need a compound fault system solution file as input, for example: http://opensha.usc.edu/ftp/kmilner/ucerf3/2013_05_10-ucerf3p3-production-10runs/2013_05_10-ucerf3p3-production-10runs_COMPOUND_SOL.zip

This script is used to run MPJDistribtedCompoundFSSPlots on the Stampede supercomputer at TACC:

{{{
login3.stampede(6)$ cat plot_gen.pbs
#!/bin/bash

# this is the wall clock time
#SBATCH -t 00:200:00
# this is the number of CPUs - number of nodes is this number divided by 16
#SBATCH -n 1280
# queue we are submitting to
#SBATCH -p normal

# this is the directory where we are running. It just also contain all necessary OpenSHA jar files
RUN_NAME="2013_05_10-ucerf3p3-production-10runs"
COMPOUND_DIR="/work/00950/kevinm/ucerf3/inversion/compound_plots/${RUN_NAME}"
# this is the path to the compound solution file
INV_DIR="/work/00950/kevinm/ucerf3/inversion/${RUN_NAME}"
COMPOUND_FILE="${INV_DIR}/${RUN_NAME}_COMPOUND_SOL.zip"
# this is the set of plots that we want to generate
PLOTS="--plot-mfds"
# the number of threads per compute node
THREADS="4"
# the minimum number of jobs sent to a given node at a given time, typically this can be equal to the number of threads
MIN_DISPATCH="4"
# the amount of memory allocated to the JVM, should be as much as possible
MEMORY="25G"

# this gets the node list
PBS_NODEFILE="/tmp/${USER}-hostfile-${SLURM_JOBID}"
echo "creating PBS_NODEFILE: $PBS_NODEFILE"
scontrol show hostnames $SLURM_NODELIST > $PBS_NODEFILE

# paths to FastMPJ
export FMPJ_HOME=/home1/00950/kevinm/FastMPJ
export PATH=$PATH:$FMPJ_HOME/bin

# this converts the node list to a format which FastMPJ can read
if [[ -e $PBS_NODEFILE ]]; then
  #count the number of processors assigned by PBS
  NP=`wc -l < $PBS_NODEFILE`
  echo "Running on $NP processors: "`cat $PBS_NODEFILE`
else
  echo "This script must be submitted to PBS with 'qsub -l nodes=X'"
  exit 1
fi

# make sure there's at least one node
if [[ $NP -le 0 ]]; then
  echo "invalid NP: $NP"
  exit 1
fi

date
echo "RUNNING FMPJ"
# this runs MPJDistributedCompoundFSSPlots in parallel with the previously selected arguments
fmpjrun -machinefile $PBS_NODEFILE -np $NP -dev niodev -Djava.library.path=$FMPJ_HOME/lib -Djava.awt.headless=true -Xmx${MEMORY} -cp ${COMPOUND_DIR}/OpenSHA_complete.jar:${COMPOUND_DIR}/parallelcolt-0.9.4.jar:${COMPOUND_DIR}/commons-cli-1.2.jar:${COMPOUND_DIR}/csparsej.jar -class scratch.UCERF3.analysis.MPJDistributedCompoundFSSPlots --threads ${THREADS} --min-dispatch ${MIN_DISPATCH} $PLOTS ${COMPOUND_FILE} ${COMPOUND_DIR}
ret=$?

date
echo "DONE with process 0. EXIT CODE: $ret"

exit $ret
}}}

Map based plots will be output in an xml file format that can then be converted to a map. This is because the compute nodes which generate the map data often cannot communicate to the external OpenSHA server which does the plotting. The maps can be generated later by running this command, with the xml output file[s] as arguments:

{{{
java -Xmx8G -cp ${DIR}/OpenSHA_complete.jar:${DIR}/commons-cli-1.2.jar scratch.UCERF3.analysis.CompoundFSSPlots <xml-files>
}}}