What is This?

Pluggable Output Processor is an engine to wrap any executabe and capture its output through a pluggable module to colorize it and/or (re)format.

Motivation

Pluggable Output Processor is my attempt to get rid of a bunch of various colorizers from my system (like colorgcc, colordiff, colorsvn, …) ~~and take everything under control~~ ;-). Some of them are written on Perl (and I’m not a fun of it :-) and after few hacks I’ve made to improve colorgcc, I realized that I don’t want to ~~waste my time~~ learning Perl.

Yes, I know there is a lot of stuff like this, but I have few problems w/ it:

I’m not a Perl programmer (and don’t want to be), but I feel a permanent desire to improve that programs in different ways.
Some of that stuff is actually abandoned – i.e. even if I can fix or improve smth, there is nobody I can send my patch to…
Some of that stuff is ~~too much~~ end-user oriented ~~so inflexible~~ – they can colorize almost everything via regular expressions and configuration files. The only problem I have w/ them: some things I’d like to colorize (or fix formatting) is impossible or damn hard to express via regexes … particularly because line-by-line processing implemented in that tools have no state… Yep, I understand that, it will be hard to code, and even harder to use such a tool for end users (i.e. ppl w/o programming skills) – it is why authors took the easy way: allow to user to write regexes in configs.

Features

pluggable architecture – easy (to Python programmers ;-) to extend
256 and 16M color terminal support ;-) Configuration files in addition to standard named colors may contain color definitions as rgb(r,g,b) or gray(n)
colorizers for make, cmake, gcc, mount and diff out of the box (more plugins to come ;-)
some modules are not just a stupid colorizers ;-) For example gcc can reformat text for better readability (really helps to understand template errors). Also cmake module can reduce amount of lines printed during test by collapsing test intro message and result into a single one.

Installation

Easy!

$ pip install outproc

$ tar -xzf outproc-X.Y.tar.gz
$ cd outproc-X.Y
$ sudo easy_install .

For Gentoo users there is a live ebuild in my repository. Also (for Gentoo users again ;-) eselect module from contrib/ will be installed by the ebuild. Users of other distros have to make symlinks to required modules manually:

# ln -s /usr/bin/outproc /usr/lib/outproc/bin/<module-name>

and then make sure /usr/lib/outproc/bin placed before /usr/bin (and anything else) in your user/system PATH environment. Personally I use the following piece of code in /etc/profile.d/set_outproc_path.sh, because setting PATH via /etc/env.d files will not give desired place inside a list of paths:

if [ -d /usr/lib/outproc/bin ]; then
    export PATH="/usr/lib/outproc/bin:$PATH"
fi

Available modules (plugins) can be found at <python-site-packages-dir>/outproc/pp. For example, to install the gcc module do the following:

$ ln -s /usr/bin/outproc /usr/lib/outproc/bin/gcc

Then you may edit /etc/outproc/gcc.conf to adjust color settings.

Usage Examples

CMake

CMake module can recognize (and colorize) various detection ~~SPAM~~ info during test phase. Also it will reduce a little lines count to be printed by one trick: if a previous line (remembered internally after print) is a subset of the current one, just move one line above and print over it…

Before
After

GNU make

This module can colorize error and service messages from make (like entering/leaving directory). Also it can recognize some “information” messages from cmake and/or gcc command line (when make VERBOSE=1 used to build) – i.e. it works for cmake-based projects (my favorite build utility nowadays).

Before
After

GNU gcc

This module is capable to colorize errors, warnings, C++ code snippets (best viewed w/ 16M color terminals ;-) and reformat/simplify some loooong error messages…

mount

Mount plugin can format columns into a table. Also it can colorize filesystems by types:

kernel – i.e. service filesystems like cgroup, mqueue, devpts & etc
real – i.e. a real devices was mounted
rebind – i.e. filesystems w/ bind option

Before
After

Configuration Details

Here is mine /etc/outproc/gcc.conf for example:

# Settings for GNU gcc output processor

# Message colors by severity level
error = rgb(5,1,1)
warning = rgb(4,2,0)
notice = normal

# Source file and position
location = gray(8)+itallic

# Code snippet colors
code = white
code-keyword = rgb(4,5,0)+bold
code-builtin-type = rgb(5,1,0)
code-modifier = rgb(4,3,0)
code-std-namespace = rgb(2,5,0)
code-boost-namespace = rgb(0,2,1)
code-data-member = normal
code-preprocessor = green
code-numeric-literal = rgb(0,2,4)
code-string-literal = magenta+itallic
code-comment = rgb(2,2,1)+itallic
code-cursor = rgb(2,0,0)

# Add a new line after source code snippet
# NOTE The `gcc` module will remove lines w/ error position indicator ('^')
# (error position will be indicated in a source line w/ a background color
# specified by `code-cursor` color). As a result error messages become
# slightly condensed ;-) -- So this setting allow you to control whether
# a new line will be added instead of line w/ error position indicator.
new-line-after-code = false

// TBD

How to Extend

To add a new module you have to name it after a tool you want to wrap (process output from) + .py extension and put that module into ${python-site-packages-dir}/outproc/pp/ dir. Then make a symlink from /usr/bin/outproc to /usr/lib/outproc/bin/<wrapped-executable> or use eselect in Gentoo. Being executed outproc will realize (from the symlink name) what binary to execute and capture output from. All captured lines will be piped through your-module.Processor.handle_line().

A minimal plugin code will looks like this:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import outproc

class Processor(outproc.Processor):

    def __init__(self, config, binary):
        super().__init__(config, binary)
        # If you want some parameters from `/etc/outproc/<your-module>.conf
        # use `self.config` to get them...


    def handle_line(self, line):
        '''
            Handle one line from a wrapped executable
        '''
        # TODO transfrom the line and return it back...
        return line

Actually a plugin may control few more things like:

what configuration file to use
is it want to handle an output for particular command (analyzing command line options) or the engine should just exec() a wrapped command
it also may want to handle a whole block of output text instead of line-by-line

More details are coming...

TODO

~~continue to improve C++ tokenizer~~ now it is capable to handle almost everything I wanted to implement. Just one thing remains: reformat too long template names into a smth shorter than terminal width…
~~unit tests for tokenizer~~
test files w/ to cause various error messages from gcc (+ unit test for colorizer somehow)
continue to improve cmake support (+ unit tests)
~~turn mount output into a human readable look-n-feel~~
colorize df depending on free space threshold
colorize diff (easy! :-) – Done for -u mode
~~eselect module to manage tools under control~~
~~ask module is it want to handle a current command or we can do execv instead~~
implement STDIN reader (pipe mode)
handle KeyboardInterrupt and hide Python crap
ctest module to colorize test results
~~handle~~ gcc -Q --help=target|optimizers|warnings Done!