depflow module
Installation
Run
pip install depflow
Defining a process
Instantiate the flow with flow = depflow.Depflow()
. Define steps in the
process as nullary functions decorated with @flow.depends(dep1, dep2, ...)
.
Dependencies can either be other steps or checks such as depflow.file(path)
and depflow.file_hash(path)
. Steps are run as they are defined.
Creating new checks
Depflow accounts for two different types of dependency checks.
Cached checks
Use @depflow.check
to decorate a function that returns a value representing
the state of some resource, where a dependent step only needs to be updated if
the state of the resource changes.
Example:
@depflow.check def kernel_version(): return 'kernel-version', subprocess.check_output('uname -r')
or to account for command line arguments:
flow = depflow.Depflow() parser = argparse.ArgumentParser() ... args = parser.parse_args() @depflow.check def args_hash(*keys): cs = hashlib.md5() for k in keys: cs.update(str(getattr(args, k))) return keys, cs.hexdigest() @flow.depends(arg_hash('output_dir', 'author')) def step_a(): build_file('a.dat', args.output_dir, author=args.author) @flow.depends(arg_hash('output_dir')) def step_b(): build_file('b.dat', args.output_dir, author='system')
Cached state is stored and compared based on a unique id generated from the first return value, the step name and the ids of its other dependencies.
Uncached checks
Use @depflow.raw_check
to decorate a function that returns a boolean
indicating that the dependent step needs to be updated.
Example:
flow = depflow.Depflow() @flow.raw_check def server_down(uri): return uri, not is_server_up(uri) @flow.depends(server_down('http://server1/state')): def server1_up(): start_server('server1')
Functions
def check(
function)
Converts a function that yields a key
then value
into a Dependency.
The key is a unique id for the object being checked. It is composed from tuples and primitives.
The value represents the state of the object. It is composed from tuples and primitives. If the value changes compared to a previous invocation, the dependant method will run.
Arguments
function
is the function to decorate.
def file(
*pargs, **kwargs)
Check for changes in a single file by timestamp.
Arguments
path
The path of the file.
def file_hash(
*pargs, **kwargs)
Check for changes in a single file by hash.
Arguments
path
The path of the file.
def no_file(
*pargs, **kwargs)
Run the step if the specified file doesn't exist.
Arguments
path
is the path of the file.
def raw_check(
function)
Decorator that converts a function that yields a key
then a value
into
a dependency.
The key
is a unique id for the object being checked. It is composed
from tuples and primitives.
The value
is True if the dependent should be updated, False otherwise.
Arguments
function
is the function to decorate.
def tree(
*pargs, **kwargs)
Check for changes in a file tree by timestamp.
Arguments
path
is the path of the root of the tree.
(opt) depth
indicates how many levels to descend into the path. 0 is
unlimited, 1 is only the specified directory itself, 2 would include the
first children, etc.
(opt) ignore
is a list of file and directory patterns to ignore. Each
pattern is a compiled regex applied against the file path from the tree
root.
def tree_hash(
*pargs, **kwargs)
Check for changes in a file tree by hash.
Arguments
path
is the path of the root of the tree.
(opt) depth
indicates how many levels to descend into the path. 0 is
unlimited, 1 is only the specified directory itself, 2 would include the
first children, etc.
(opt) ignore
is a list of file and directory patterns to ignore. Each
pattern is a compiled regex applied against the file path from the tree
root.
Classes
class Depflow
Ancestors (in MRO)
- Depflow
- builtins.object
Static methods
def __init__(
self, name='depflow')
(opt) name
is the name of the flow. This is used in logging
and persisting the state of various checks are stored between runs in
a database at .{name}.sqlite3
. The database filename can also be
overridden with the environment variable DEPFLOW_CACHE
.
def depends(
self, *nodes)
Runs the wrapped function if any dependency node has changed.
Dependency nodes
can be either other functions wrapped with depends
or checks.
When the step is used as a dependency for another step the function
name is used to identify the state of this step. If you have multiple
steps with the same function name, you need to either specify
the optional parameter qualification
(as tuples and primitives)
or use a scope
.
def scope(
self, *qualification)
Returns a new scope with the specified qualification
. Scopes
support all the same methods as Depflow
but decorated steps
will automatically receive these qualifications.
Calling scope
on a scope results in a new scope with the combined
qualifications.