Keyword Arguments

To make submission easier, this module defines a number of keyword arguments in the options.py file that can be used for all submission and Job() functions. These include things like ‘cores’ and ‘nodes’ and ‘mem’.

The following is a complete list of arguments that can be used in this version

Common: Used in every mode

Option	Description	Type	Default
depends	A job or list of jobs to depend on	list	None
clean_files	Auto clean script files when fetching outputs	bool	None
clean_outputs	Auto clean output files when fetching outputs	bool	None
cores	Number of cores to use for the job	int	1
modules	Modules to load with the module load command	list	None
syspaths	Paths to add to _sys.path for submitted functions	list	None
scriptpath	Folder to write cluster script files to, must be accessible to the compute nodes.	str	.
outpath	Folder to write cluster output files to, must be accessible to the compute nodes.	str	.
runpath	The working directory for the job	str	.
suffix	A suffix to append to job files (e.g. job.suffix.qsub)	str	cluster
outfile	File to write STDOUT to	str	None
errfile	File to write STDERR to	str	None

Func: Used for function calls

Option	Description	Type	Default
imports	Imports to be used in function calls (e.g. sys, os)	list	None

Cluster: Options that work in both slurm and torque

Option	Description	Type	Default
nodes	Number of nodes to request	int	1
features	A comma-separated list of node features to require	list	None
qos	A quality of service to require	str	None
time	Walltime in HH:MM:SS	str	12:00:00
mem	Memory to use in MB (e.g. 4000)	[‘int’, ‘str’]	4000
partition	The partition/queue to run in (e.g. local/batch)	str	None
account	Account to be charged	str	None
export	Comma separated list of environmental variables to export	str	None

Slurm: Used for slurm only

Option	Description	Type	Default
begin	Start after this much time	str	None

Synonyms

Synonym	Option
depend	depends
dependency	depends
dependencies	depends
stdout	outfile
stderr	errfile
queue	partition
memory	mem
cpus	cores
threads	cores
walltime	time
delete_files	clean_files
delete_outputs	clean_outputs
filedir	scriptpath
filepath	scriptpath
dir	runpath
path	runpath
paths	syspaths
syspath	syspaths
scriptdir	scriptpath
cleanfiles	clean_files
delfiles	clean_files
cleanouts	clean_outputs
delouts	clean_outputs
deloutputs	clean_outputs
cleanoutputs	clean_outputs

Note: Type is enforced, any provided argument must match that python type (automatic conversion is attempted), the default is just a recommendation and is not currently used. These arguments are passed like regular arguments to the submission and Job() functions, eg:

Job(nodes=1, cores=4, mem='20MB')

This will be interpretted correctly on any system. If torque or slurm are not available, any cluster arguments will be ignored. The module will attempt to honor the cores request, but if it exceeds the maximum number of cores on the local machine, then the request will be trimmed accordingly (i.e. a 50 core request will become 8 cores on an 8 core machine).

Adding your own keywords

There are many more options available for torque and slurm, to add your own, edit the options.py file, and look for CLUSTER_OPTS (or TORQUE/SLURM if your keyword option is only availble on one system). Add your option using the same format as is present in that file. The format is:

('name', {'slurm': '--option-str={}', 'torque': '--torque-option={}',
          'help': 'This is an option!', 'type': str, 'default': None})

You can also add list options, but they must include ‘sjoin’ and ‘tjoin’ keys to define how to merge the list for slurm and torque, or you must write custom option handling code in fyrd.options.options_to_string(). For an excellent example of both approaches included in a single option, see the ‘features’ keyword above.