OpenSpeedShop test utility: test-script.py
_______________________________________________________________________________
Quick start guide: (options described in main section)

cd openspeedshop-test-suite

./test-script.py --create-env --oss-module <your openspeedshop module>
./test-script.py --create-prof <module options>
./test-script.py --build-tests
./test-script.py --create-baseline
./test-script.py --run-tests
./test-script.py --compare tests

Note about -create-env and --create-prof commands
# ---- If your computer is listed in the example_config folder 
# ---- (currently just pleiedes) you can use the following instructions
# ---- and skip the first to test-script.py commands above
cp example_config/pleiedes_env.json env.json
cp example_config/pleiedes_profiles.json profiles.json
________________________________________________________________________________

The purpose of this utility is to test OpenSpeedShop on a number of sample 
programs and analyze the results. It supports three (3) major operations, 
building, runnning, and comparing.

    - Building (use --build-tests) calls one or more of the install scripts 
               located in the build_scripts directory.

    - Running the tests can be done with either of the two commands 
               (--create-baseline or --run-tests) the only difference between 
               these commands is the location of the database files. These 
               commands run all the oss tools on all the executable files in the 
               bin directory.  These should only be called after building the 
               database directory is as follows:
                   $install_dir/test_data/$oss_version/[baseline,results]/$date/

    - Compare (use --compare-tests) this will run openss -batch on each of the 
              database files in the results and baseline folders. Then it will 
              look at the corresponding results and check to see if they are 
              within acceptable variance (default 10.00%). results will be 
              summarized and printed to the screen.  By default the script will 
              look for the most recent date folder in the results and baseline 
              folders in the folder for the oss_version specified in environment 
              file. You can manually specify either of these directories with:
	         -b for baseline folder
       	         -r for results folder
	      These paths can be absolute or relative to the test_data dir
	         eg. -b oss_offline-2.2.2/baseline/2016-05-11_14:28:21.023070/
              or use:
     	         --baseline-version
  	      To use the most recent baseline dir for a version other than 
              what is specifed in the env file.
	         eg. --baseline-version oss_offline-2.2.2

NEW: Pleides users see README_PLEIDES
________________________________________________________________________________
SAMPLE WORKFLOW:
________________________________________________________________________________
1) Create test environment (Two steps)
________________________________________________________________________________

The test suite is configured via two (2) files, the env.json and profiles.json.
By default, the test suite looks for the files with these names in its install 
directory. You can manually specify other files using the -p option for profiles
and the -e option for environment flags.

_______________________
1) Step 1: Create the default environment file env.json with the command:
_______________________ 

./test-script.py --create-env --oss-module <your openspeedshop module>

If you wish to build the tests with the intel compiler, also add the following 
arguments:
    --ompt-root <path to ompt root>

This test-script.py command will create a default environment file env.json 
populated with some defaults. Please check that these are set to appropriate 
values before continuing.

The options are as follows:
    bin_dir             - Directory to store compiled tests in
    mpi_drivers         - List of commands you use to run mpi applications, 
                          include argments
                        	e.g. 'mpi_drivers':['mpirun -np 2']
    src_dir             - Directory containing the source for the tests
    test_data_dir       - Directory to store the test data
    input_dir           - Directory with input files for the tests that require 
                          them.
    oss_version         - The current version of openss. This needs to be 
                          manually specified until oss has a --version option
    acceptable_variance - Floating point value in percent. maximum factor by 
                          which baseline and results values can differ and still
                          be allowed to pass
    ompt_root           - Root directory of of ompt installation for libiomp, 
                          only used with intel compiler
    openss_module       - Location of the openss module file. this is required 
                          when using a job controller only.

_______________________
1) Step 2: Create the default profile file profiles.json with the command:
_______________________

Create the profiles.json file. this contains information about how to build the 
tests on your system, like the MPI implementation module to be used.

./test-script.py --create-prof <module options>

The module options specify which modules will be used to build the tests. these 
are separated into 2 categories; compilers (--cc-modules), and mpi 
implementations (--mpi-modules).

If no --cc-modules are specified, the script will assume that gcc is in your 
path and attempt to build with that.

MPI modules must be specified. here is an example showing how to configure with 
gcc, intel, openmpi, and intel-mpi


./test-script.py --create-prof --cc-modules gcc intelcc --mpi-modules openmpi mpi-intel

This file (default profiles.json) is a list of build configurations. each 
configuration in a json object with the following properties
    cc             - The compiler, options are intel, gnu, pgi
    mpi_imp        - The name of the mpi implementations, openmp, openmpi, 
                     mpt etc.  This needs to match name in the binary files
    cc_module      - The module for the compiler
    mpi_module     - The module for the mpi implementation
    mpi_cmake_flag - This variable tells cmake which type of mpi implementation 
                     you are using, options are, -DMPT_DIR, -DOPENMPIDIR, 
                     -DMICH_DIR (maybe others as well?)
    cmake_flag_var - The environment variable or path you wish the mpi_cmake_flag
                     to be set to, usually something like $MPI_ROOT or whatever 
                     is in your mpi module file.  If this starts with a $ it will
                     be treated as an environment variable, otherwise it will 
                     be treated as a path
    other_modules  - A list of additional modules you may need to use to compile


________________________________________________________________________________
2) Build the tests with the command
________________________________________________________________________________

./test-script.py --build-tests

This will call the build scripts in build_scripts_dir specifed by the compilers 
option in the env file.

If something fails here it likely that your env.json or profiles.json 
configuration files are incorrect.

________________________________________________________________________________
3) Create baselines for the various tests
________________________________________________________________________________

./test-script.py --create-baseline

This can take some time, results will be in test_data/baseline/$date by default

________________________________________________________________________________
4) Create run data for the various tests
________________________________________________________________________________

./test-script.py --run-tests

This can take some time, results will be in test_data/results/$date by default

________________________________________________________________________________
5) Compare the results to the baseline data
________________________________________________________________________________

./test-script.py --compare-tests

This tools looks in the most recent results and baseline folders by default
soon to add specific folder specification.

By default the tool checks that all values are within 5% of each other.
this number will be an argument soon

See ./test-script.py --help for more options

________________________________________________________________________________
Directory structure after building and running:
________________________________________________________________________________

openspeedshop-test-suite
    CMakeLists.txt
    cmake
	...
    bin
	...
    build_scripts
	run_build*.sh
    env.json
    src
	...
    test-script.py
    test_data
	baseline
  	    oss_version
		date
	  	    openss_dbs...
	results
	    oss_version
		date
		    openss_dbs...