Writing Ingress Plugins

Umbra uses the venerable Plugin Oriented Programing paradigm as it is realized in the pop framework. This means that all of the features of pop are available.

Making an ingress plugin is easy, just add the plugin to umbra/mods/ingress. The ingress plugin subsystem only takes a single function, run. This can be a function of any type, a generator, a coroutine, an async generator or a standard function. The type of function you choose to implement defines how the ingress system will run. If the ingress system needs to be running continuously then an async generator is optimal.

This is a simple example of using a subprocess to tail a file. This example shows how to use subprocess with asyncio in python to shell out to an event stream. There are better ways to do what we are doing here, but this is a good example of using subprocess to stream, as it is often a great way to ingest data.

import asyncio
import json

async def run(hub):
    '''
    Shell out salt-run and send in the events from the salt event bus
    '''
    proc = await asyncio.create_subprocess_shell(
            'salt-run state.event',
            stdout=asyncio.subprocess.PIPE)
    while True:
        line = await proc.stdout.readline()
        line = line.decode()
        comps = line.split(maxsplit=1)
        if len(comps) < 2:
            continue
        tag = comps[0].strip()
        data = json.loads(comps[1])
        group = {}
        for pipe in conf:
            group[pipe] = []
        for pipe in conf:
            for match in conf[pipe]:
                if fnmatch.fnmatch(tag, match):
                    data = json.loads(comps[1].strip())
                    event = {'tag': tag, 'data': data}
                    group[pipe].append(event)
        for pipe in group:
            yield {'pipe': pipe, 'data': group[pipe]}

This example shows how to easily use asyncio subprocess to await lines by line feedback and yield the formatted ingestion data. This approach can work with virtually any shell program that continuously emits data.