Keyword Arguments¶
To make submission easier, this module defines a number of keyword arguments in the options.py file that can be used for all submission and Job() functions. These include things like ‘cores’ and ‘nodes’ and ‘mem’.
The following is a complete list of arguments that can be used in this version
Common: Used in every mode¶
Option | Description | Type | Default |
---|---|---|---|
depends | A job or list of jobs to depend on | list | None |
clean_files | Auto clean script files when fetching outputs | bool | None |
clean_outputs | Auto clean output files when fetching outputs | bool | None |
cores | Number of cores to use for the job | int | 1 |
modules | Modules to load with the module load command | list | None |
filepath | Folder to write cluster files to, must be accessible to the compute nodes. | str | . |
dir | The working directory for the job | str | path argument |
suffix | A suffix to append to job files (e.g. job.suffix.qsub) | str | cluster |
outfile | File to write STDOUT to | str | None |
errfile | File to write STDERR to | str | None |
Func: Used for function calls¶
Option | Description | Type | Default |
---|---|---|---|
imports | Imports to be used in function calls (e.g. sys, os) | list | None |
Local: Used only in local mode¶
Option | Description | Type | Default |
---|---|---|---|
threads | Number of threads to use on the local machine | int | 4 |
Cluster: Options that work in both slurm and torque¶
Option | Description | Type | Default |
---|---|---|---|
nodes | Number of nodes to request | int | 1 |
features | A comma-separated list of node features to require | list | None |
time | Walltime in HH:MM:SS | str | 12:00:00 |
mem | Memory to use in MB (e.g. 4000) | [‘int’, ‘str’] | 4000 |
partition | The partition/queue to run in (e.g. local/batch) | str | None |
account | Account to be charged | str | None |
export | Comma separated list of environmental variables to export | str | None |
Slurm: Used for slurm only¶
Option | Description | Type | Default |
---|---|---|---|
begin | Start after this much time | str | None |
Synonyms¶
Synonym | Option |
---|---|
depend | depends |
dependency | depends |
dependencies | depends |
stdout | outfile |
stderr | errfile |
queue | partition |
memory | mem |
cpus | cores |
walltime | time |
delete_files | clean_files |
delete_outputs | clean_outputs |
filedir | filepath |
runpath | dir |
path | filepath |
scriptpath | filepath |
scriptdir | filepath |
Note: Type is enforced, any provided argument must match that python type (automatic conversion is attempted), the default is just a recommendation and is not currently used. These arguments are passed like regular arguments to the submission and Job() functions, eg:
Job(nodes=1, cores=4, mem='20MB')
This will be interpretted correctly on any system. If torque or slurm are not available, any cluster arguments will be ignored. The module will attempt to honor the cores request, but if it exceeds the maximum number of cores on the local machine, then the request will be trimmed accordingly (i.e. a 50 core request will become 8 cores on an 8 core machine).
Adding your own keywords¶
There are many more options available for torque and slurm, to add your own, edit the options.py file, and look for CLUSTER_OPTS (or TORQUE/SLURM if your keyword option is only availble on one system). Add your option using the same format as is present in that file. The format is:
('name', {'slurm': '--option-str={}', 'torque': '--torque-option={}',
'help': 'This is an option!', 'type': str, 'default': None})
You can also add list options, but they must include ‘sjoin’ and ‘tjoin’ keys to
define how to merge the list for slurm and torque, or you must write custom
option handling code in fyrd.options.options_to_string()
. For an
excellent example of both approaches included in a single option, see the
‘features’ keyword above.