Console Scripts¶
This software is primarily intended to be a library, however some management tasks are just easier from the console. For that reason, fyrd has a frontend console script that makes tasks such as managing the local config and profiles trivial, it also has modes to inspect the queue easily, and to wait for jobs from the console, as well as to clean the working directory.
fyrd¶
This software has uses a subcommand system to separate modes, and has six modes:
- config — show and edit the contents of the config file
- profile - inspect and manage cluster profiles
- keywords - print a list of current keyword arguments with descriptions for each
- queue - show running jobs, makes filtering jobs very easy
- wait - wait for a list of jobs
- clean - clean all script and output files in the given directory
Several of the commands have aliases (conf and prof being the two main ones)
Examples¶
fyrd prof list
fyrd prof add large cores:92 mem:200GB partition:high_mem time:00:06:00
fyrd queue # Shows all of your current jobs
fyrd queue -a # Shows all users jobs
fyrd queue -p long -u bob dylan # Show all jobs owned by bob and dylan in the long queue
fyrd wait 19872 19876
fyrd wait -u john
fyrd clean
All Options¶
Manage fyrd config, profiles, and queue.
Author | Michael D Dacre <mike.dacre@gmail.com> |
Organization | Stanford University |
License | MIT License, use as you wish |
Version | 0.6.2-beta.7 |
usage: fyrd [-h] [-v] {conf,prof,keywords,queue,wait,clean} ...
Positional Arguments¶
{conf,prof,keywords,queue,wait,clean} | |
Possible choices: conf, config, prof, profile, keywords, keys, options, queue, q, wait, clean |
Named Arguments¶
-v, –verbose | Show debug outputs Default: False |
Sub-commands:¶
conf (config)¶
This script allows display and management of the fyrd config file found here: /home/docs/.fyrd/config.txt.
fyrd conf [-h] {show,list,help,update,alter,init} ...
Positional Arguments¶
cmnd | Possible choices: show, list, help, update, alter, init |
Sub-commands:¶
By default shows the current config, which is based on the config file, but with all parsing modifications applied, meaning that extra options or sections are ignored.
To print the file as is, pass –file, or to limit to a single section, pass –section <section>
fyrd conf show [-h] [-s [...]] [-f]
-s, –sections | Possible choices: jobqueue, queue, jobs Limit results to a list of sections |
-f, –file | Print file contents only Default: False |
Show info on every config option
fyrd conf help [-h] [-s [...]]
-s, –sections | Possible choices: jobqueue, queue, jobs Limit results to a list of sections |
Update one option at a time.
Sections: {‘jobqueue’, ‘queue’, ‘jobs’}
fyrd conf update [-h] section option value
section | Possible choices: jobqueue, queue, jobs Section to update |
option | Possible choices: clean_files, sleep_len, file_block_time, suffix, scriptpath, outpath, jobno, auto_submit, clean_outputs, generic_python, qsub, queue_update, profile_file, res_time, max_jobs, queue_type, sbatch Option to update |
value | New value for option |
prof (profile)¶
Fyrd jobs use keyword arguments to run (for a complete list run this script with the keywords command). These keywords can be bundled into profiles, which are kept in /home/docs/.fyrd/profiles.txt. This file can be edited directly or manipulated here.
fyrd prof [-h]
{show,list,add,new,update,alter,edit,remove-option,del-option,delete,del}
...
Positional Arguments¶
cmnd | Possible choices: show, list, add, new, update, alter, edit, remove-option, del-option, delete, del |
Sub-commands:¶
Print current profiles
fyrd prof show [-h] [-n NAME [NAME ...]]
-n, –name | Limit to only these profiles |
Add a new profile
fyrd prof add [-h] name options [options ...]
name | Profile name |
options | Options to update |
Update an existing profile
fyrd prof update [-h] name options [options ...]
name | Profile name |
options | Options to update |
keywords (keys, options)¶
Print available keyword arguments.
fyrd keywords [-h] [-t | -s | -l]
Named Arguments¶
-t, –table | Print keywords as a table Default: False |
-s, –split-tables | |
Print keywords as multiple tables Default: False | |
-l, –list | Print a list of keywords only Default: False |
queue (q)¶
Check the local queue, similar to squeue or qstat but simpler, good for quickly checking the queue.
By default it searches only your own jobs, pass ‘–all-users’ or ‘–users <user> [<user2>...]’ to change that behavior.
To just list jobs with some basic info, run with no arguments.
fyrd queue [-h] [-u [...] | -a] [-p [...]] [-r | -q | -d | -b] [-l | -c]
queue filtering¶
-u, –users | Limit to these users |
-a, –all-users | |
Display jobs for all users Default: False | |
-p, –partitions | |
Limit to these partitions (queues) |
queue state filtering¶
-r, –running | Show only running jobs Default: False |
-q, –queued | Show only queued jobs Default: False |
-d, –done | Show only completed jobs Default: False |
-b, –bad | Show only completed jobs Default: False |
display options¶
-l, –list | Print job numbers only, works well with xargs Default: False |
-c, –count | Print job count only Default: False |
wait¶
Wait on a list of jobs, block until they complete.
fyrd wait [-h] [-u USERS] [jobs [jobs ...]]
Positional Arguments¶
jobs | Job list to wait for |
Named Arguments¶
-u, –users | A comma-separated list of users to wait for |
clean¶
Clean all intermediate files created by the cluster module.
If not directory is passed, the default if either scriptpath or outpath are set in the config is to clean files in those locations is to clean those directories. If they are not set, the default is the current directory.
By default, outputs are not cleaned, to clean them too, pass ‘-o’
- Caution:
The clean() function will delete EVERY file with extensions matching those these:
.<suffix>.err .<suffix>.out .<suffix>.sbatch & .fyrd.script for slurm mode .<suffix>.qsub for torque mode .<suffix> for local mode _func.<suffix>.py _func.<suffix>.py.pickle.in _func.<suffix>.py.pickle.out
fyrd clean [-h] [-o] [-s SUFFIX] [-q {torque,slurm,local}] [-n] [dir]
Positional Arguments¶
dir | Directory to clean (optional) |
Named Arguments¶
-o, –outputs | Clean output files too Default: False |
-s, –suffix | Suffix to use for cleaning Default: “cluster” |
-q, –qtype | Possible choices: torque, slurm, local Limit deletions to this qtype |
-n, –no-confirm | |
Do not confirm before deleting (for scripts) Default: True |
aliases¶
Several shell scripts are provided in bin/ to provide shortcuts to the fyrd subcommands:
- my-queue (or myq): fyrd queue
- clean-job-files: fyrd clean
- monitor-jobs: fyrd wait
- cluster-keywords: fyrd keywords