Top

depflow module

Installation

Run

pip install depflow

Defining a process

Instantiate the flow with flow = depflow.Depflow(). Define steps in the process as nullary functions decorated with @flow.depends(dep1, dep2, ...). Dependencies can either be other steps or checks such as depflow.file(path) and depflow.file_hash(path). Steps are run as they are defined.

Creating new checks

Depflow accounts for two different types of dependency checks.

Cached checks

Use @depflow.check to decorate a function that returns a value representing the state of some resource, where a dependent step only needs to be updated if the state of the resource changes.

Example:

@depflow.check
def kernel_version():
    return 'kernel-version', subprocess.check_output('uname -r')

or to account for command line arguments:

flow = depflow.Depflow()
parser = argparse.ArgumentParser()
...
args = parser.parse_args()

@depflow.check
def args_hash(*keys):
    cs = hashlib.md5()
    for k in keys:
        cs.update(str(getattr(args, k)))
    return keys, cs.hexdigest()

@flow.depends(arg_hash('output_dir', 'author'))
def step_a():
    build_file('a.dat', args.output_dir, author=args.author)

@flow.depends(arg_hash('output_dir'))
def step_b():
    build_file('b.dat', args.output_dir, author='system')

Cached state is stored and compared based on a unique id generated from the first return value, the step name and the ids of its other dependencies.

Uncached checks

Use @depflow.raw_check to decorate a function that returns a boolean indicating that the dependent step needs to be updated.

Example:

flow = depflow.Depflow()

@flow.raw_check
def server_down(uri):
    return uri, not is_server_up(uri)

@flow.depends(server_down('http://server1/state')):
def server1_up():
    start_server('server1')

Functions

def check(

function)

Converts a function that yields a key then value into a Dependency.

The key is a unique id for the object being checked. It is composed from tuples and primitives.

The value represents the state of the object. It is composed from tuples and primitives. If the value changes compared to a previous invocation, the dependant method will run.

Arguments

function is the function to decorate.

def file(

*pargs, **kwargs)

Check for changes in a single file by timestamp.

Arguments

path The path of the file.

def file_hash(

*pargs, **kwargs)

Check for changes in a single file by hash.

Arguments

path The path of the file.

def no_file(

*pargs, **kwargs)

Run the step if the specified file doesn't exist.

Arguments

path is the path of the file.

def raw_check(

function)

Decorator that converts a function that yields a key then a value into a dependency.

The key is a unique id for the object being checked. It is composed from tuples and primitives.

The value is True if the dependent should be updated, False otherwise.

Arguments

function is the function to decorate.

def tree(

*pargs, **kwargs)

Check for changes in a file tree by timestamp.

Arguments

path is the path of the root of the tree.

(opt) depth indicates how many levels to descend into the path. 0 is unlimited, 1 is only the specified directory itself, 2 would include the first children, etc.

(opt) ignore is a list of file and directory patterns to ignore. Each pattern is a compiled regex applied against the file path from the tree root.

def tree_hash(

*pargs, **kwargs)

Check for changes in a file tree by hash.

Arguments

path is the path of the root of the tree.

(opt) depth indicates how many levels to descend into the path. 0 is unlimited, 1 is only the specified directory itself, 2 would include the first children, etc.

(opt) ignore is a list of file and directory patterns to ignore. Each pattern is a compiled regex applied against the file path from the tree root.

Classes

class Depflow

Ancestors (in MRO)

Static methods

def __init__(

self, name='depflow')

(opt) name is the name of the flow. This is used in logging and persisting the state of various checks are stored between runs in a database at .{name}.sqlite3. The database filename can also be overridden with the environment variable DEPFLOW_CACHE.

def depends(

self, *nodes)

Runs the wrapped function if any dependency node has changed.

Dependency nodes can be either other functions wrapped with depends or checks.

When the step is used as a dependency for another step the function name is used to identify the state of this step. If you have multiple steps with the same function name, you need to either specify the optional parameter qualification (as tuples and primitives) or use a scope.

def scope(

self, *qualification)

Returns a new scope with the specified qualification. Scopes support all the same methods as Depflow but decorated steps will automatically receive these qualifications.

Calling scope on a scope results in a new scope with the combined qualifications.