Console Scripts

This software is primarily intended to be a library, however some management tasks are just easier from the console. For that reason, fyrd has a frontend console script that makes tasks such as managing the local config and profiles trivial, it also has modes to inspect the queue easily, and to wait for jobs from the console, as well as to clean the working directory.

fyrd

This software has uses a subcommand system to separate modes, and has six modes:

  • config — show and edit the contents of the config file
  • profile - inspect and manage cluster profiles
  • keywords - print a list of current keyword arguments with descriptions for each
  • queue - show running jobs, makes filtering jobs very easy
  • wait - wait for a list of jobs
  • clean - clean all script and output files in the given directory

Several of the commands have aliases (conf and prof being the two main ones)

Examples

fyrd prof list
fyrd prof add large cores:92 mem:200GB partition:high_mem time:00:06:00
fyrd queue  # Shows all of your current jobs
fyrd queue -a # Shows all users jobs
fyrd queue -p long -u bob dylan # Show all jobs owned by bob and dylan in the long queue
fyrd wait 19872 19876
fyrd wait -u john
fyrd clean

All Options

Manage fyrd config, profiles, and queue.

Author Michael D Dacre <mike.dacre@gmail.com>
Organization Stanford University
License MIT License, use as you wish
Version 0.6.2-beta.7

usage: fyrd [-h] [-v] {conf,prof,keywords,queue,wait,clean} ...

Positional Arguments

{conf,prof,keywords,queue,wait,clean}
 Possible choices: conf, config, prof, profile, keywords, keys, options, queue, q, wait, clean

Named Arguments

-v, –verbose

Show debug outputs

Default: False

Sub-commands:

conf (config)

This script allows display and management of the fyrd config file found here: /home/docs/.fyrd/config.txt.

fyrd conf [-h] {show,list,help,update,alter,init} ...
Positional Arguments
cmnd Possible choices: show, list, help, update, alter, init
Sub-commands:
show (list)

By default shows the current config, which is based on the config file, but with all parsing modifications applied, meaning that extra options or sections are ignored.

To print the file as is, pass –file, or to limit to a single section, pass –section <section>

fyrd conf show [-h] [-s  [...]] [-f]
Named Arguments
-s, –sections

Possible choices: jobqueue, queue, jobs

Limit results to a list of sections

-f, –file

Print file contents only

Default: False

help

Show info on every config option

fyrd conf help [-h] [-s  [...]]
Named Arguments
-s, –sections

Possible choices: jobqueue, queue, jobs

Limit results to a list of sections

update (alter)

Update one option at a time.

Sections: {‘jobqueue’, ‘queue’, ‘jobs’}

fyrd conf update [-h] section option value
Positional Arguments
section

Possible choices: jobqueue, queue, jobs

Section to update

option

Possible choices: clean_files, sleep_len, file_block_time, suffix, scriptpath, outpath, jobno, auto_submit, clean_outputs, generic_python, qsub, queue_update, profile_file, res_time, max_jobs, queue_type, sbatch

Option to update

value New value for option
init

Overwrites the exiting config and starts from scratch. Asks for input from user for default options.

fyrd conf init [-h] [--defaults] [--yes]
Named Arguments
–defaults

Non-interactive, just use builtin defaults

Default: False

–yes

Do not as for confirmation

Default: True

prof (profile)

Fyrd jobs use keyword arguments to run (for a complete list run this script with the keywords command). These keywords can be bundled into profiles, which are kept in /home/docs/.fyrd/profiles.txt. This file can be edited directly or manipulated here.

fyrd prof [-h]
          {show,list,add,new,update,alter,edit,remove-option,del-option,delete,del}
          ...
Positional Arguments
cmnd Possible choices: show, list, add, new, update, alter, edit, remove-option, del-option, delete, del
Sub-commands:
show (list)

Print current profiles

fyrd prof show [-h] [-n NAME [NAME ...]]
Named Arguments
-n, –name Limit to only these profiles
add (new)

Add a new profile

fyrd prof add [-h] name options [options ...]
Positional Arguments
name Profile name
options Options to update
update (alter, edit)

Update an existing profile

fyrd prof update [-h] name options [options ...]
Positional Arguments
name Profile name
options Options to update
remove-option (del-option)

Remove a profile option

fyrd prof remove-option [-h] name options [options ...]
Positional Arguments
name Profile name
options Options to remove
delete (del)

Delete an existing profile

fyrd prof delete [-h] name
Positional Arguments
name Profile name
keywords (keys, options)

Print available keyword arguments.

fyrd keywords [-h] [-t | -s | -l]
Named Arguments
-t, –table

Print keywords as a table

Default: False

-s, –split-tables
 

Print keywords as multiple tables

Default: False

-l, –list

Print a list of keywords only

Default: False

queue (q)

Check the local queue, similar to squeue or qstat but simpler, good for quickly checking the queue.

By default it searches only your own jobs, pass ‘–all-users’ or ‘–users <user> [<user2>...]’ to change that behavior.

To just list jobs with some basic info, run with no arguments.

fyrd queue [-h] [-u  [...] | -a] [-p  [...]] [-r | -q | -d | -b] [-l | -c]
queue filtering
-u, –users Limit to these users
-a, –all-users
 

Display jobs for all users

Default: False

-p, –partitions
 Limit to these partitions (queues)
queue state filtering
-r, –running

Show only running jobs

Default: False

-q, –queued

Show only queued jobs

Default: False

-d, –done

Show only completed jobs

Default: False

-b, –bad

Show only completed jobs

Default: False

display options
-l, –list

Print job numbers only, works well with xargs

Default: False

-c, –count

Print job count only

Default: False

wait

Wait on a list of jobs, block until they complete.

fyrd wait [-h] [-u USERS] [jobs [jobs ...]]
Positional Arguments
jobs Job list to wait for
Named Arguments
-u, –users A comma-separated list of users to wait for
clean

Clean all intermediate files created by the cluster module.

If not directory is passed, the default if either scriptpath or outpath are set in the config is to clean files in those locations is to clean those directories. If they are not set, the default is the current directory.

By default, outputs are not cleaned, to clean them too, pass ‘-o’

Caution:

The clean() function will delete EVERY file with extensions matching those these:

.<suffix>.err
.<suffix>.out
.<suffix>.sbatch & .fyrd.script for slurm mode
.<suffix>.qsub for torque mode
.<suffix> for local mode
_func.<suffix>.py
_func.<suffix>.py.pickle.in
_func.<suffix>.py.pickle.out
fyrd clean [-h] [-o] [-s SUFFIX] [-q {torque,slurm,local}] [-n] [dir]
Positional Arguments
dir Directory to clean (optional)
Named Arguments
-o, –outputs

Clean output files too

Default: False

-s, –suffix

Suffix to use for cleaning

Default: “cluster”

-q, –qtype

Possible choices: torque, slurm, local

Limit deletions to this qtype

-n, –no-confirm
 

Do not confirm before deleting (for scripts)

Default: True

aliases

Several shell scripts are provided in bin/ to provide shortcuts to the fyrd subcommands:

  • my-queue (or myq): fyrd queue
  • clean-job-files: fyrd clean
  • monitor-jobs: fyrd wait
  • cluster-keywords: fyrd keywords