# Aggregation

## The ubiquitous VWAP

Every CEP supplier loves an example of VWAP calculation: it's small, it's about that quintessential CEP activity: aggregation, and it sounds like something from the real world.

A quick sidebar: what is the VWAP? It's the Value-Weighted Average Price: the average price for the shares traded during some period of time, usually a day. If you take the price of every share traded during the day and calculate the average, you get the VWAP. What is the value-weighted part? The shares don't usually get sold one by one. They're sold in the variable-sized lots. If you think in the terms of lots and not individual shares, you have to weigh the trade prices (not to be confused with costs) for the lots proportional to the number of shares in them.

I've been using VWAP for trying out the different approaches to the aggregation. There are multiple ways to do it, from fully manual, to the aggregator infrastructure with manual computation of the aggregations, to the simple aggregation functions. The cutest version of VWAP so far is implemented as a user-defined aggregation function for the SimpleAggregator. Here is how it goes:

    # VWAP function definition
    my $myAggFunctions = {
    	myvwap => {
    		vars => { sum => 0, count => 0, size => 0, price => 0 },
    		step => '($%size, $%price) = @$%argiter; '
    			. 'if (defined $%size && defined $%price) '
    				. '{$%count += $%size; $%sum += $%size * $%price;}',
    		result => '($%count == 0? undef : $%sum / $%count)',
    	},
    };

    my $uTrades = Triceps::Unit->new("uTrades");

    # the input data
    my $rtTrade = Triceps::RowType->new(
    	id => "int32", # trade unique id
    	symbol => "string", # symbol traded
    	price => "float64",
    	size => "float64", # number of shares traded
    );

    my $ttWindow = Triceps::TableType->new($rtTrade)
    	->addSubIndex("byId",
    		Triceps::IndexType->newHashed(key => [ "id" ])
    	)
    	->addSubIndex("bySymbol",
    		Triceps::IndexType->newHashed(key => [ "symbol" ])
    		->addSubIndex("fifo", Triceps::IndexType->newFifo())
    	)
    ;

    # the aggregation result
    my $rtVwap;
    my $compText; # for debugging

    Triceps::SimpleAggregator::make(
    	tabType => $ttWindow,
    	name => "aggrVwap",
    	idxPath => [ "bySymbol", "fifo" ],
    	result => [
    		symbol => "string", "last", sub {$_[0]->get("symbol");},
    		id => "int32", "last", sub {$_[0]->get("id");},
    		volume => "float64", "sum", sub {$_[0]->get("size");},
    		vwap => "float64", "myvwap", sub { [$_[0]->get("size"), $_[0]->get("price")];},
    	],
    	functions => $myAggFunctions,
    	saveRowTypeTo => \$rtVwap,
    	saveComputeTo => \$compText,
    );

    $ttWindow->initialize();
    my $tWindow = $uTrades->makeTable($ttWindow, "tWindow");

    # label to print the result of aggregation
    my $lbPrint = $uTrades->makeLabel($rtVwap, "lbPrint",
    	undef, sub { # (label, rowop)
    		&send($_[1]->printP(), "\n");
    	});
    $tWindow->getAggregatorLabel("aggrVwap")->chain($lbPrint);

    while(&readLine) {
    	chomp;
    	my @data = split(/,/); # starts with a string opcode
    	$uTrades->makeArrayCall($tWindow->getInputLabel(), @data);
    	$uTrades->drainFrame(); # just in case, for completeness
    }

The aggregators get defined as parts of the table type.

    Triceps::SimpleAggregator::make()

is a kind of a template that adds an aggregator definition to the table type that is specified in the option tabType. An aggeragtor doesn't live in a vacuum, it always works as a part of the table type. As the table gets modified, the aggregator also re-computes its aggregation results. The fine distinction is that the aggregator is a part of the table type, and is common for all the tables of this type. But the table stores its aggregation state, and when an aggregator runs on a table, it uses and modifies that state.

The name of the aggregator is how you can find its result later in the table: each aggregator has an output label created for it, that can be found with

    $table->getAggregatorLabel()

. The option idxPath defines both the grouping of the rows for this aggregator and their order in the group. The index type at the path determines the order and its parent defines the groups. In this case the grouping happens by symbol, and the rows in the groups go in the FIFO order. This means that the aggregation function

    last

will be selecting the row that has been inserted last, in the FIFO order.

The option result defines both the row type of the result and the rules for its computation. Each field is defined there with four elements: name, type, aggregation function name, and the function reference to select the value to be aggregated from the row. Triceps provides a bunch of pre-defined aggregation functions like

    first

,

    last

,

    sum

,

    count

,

    avg

and so on. But VWAP is not one of them (well, maybe now it should be, but then this example would be less interesting). Not to worry, the user can add custom aggregation functions, and that's what this example does.

The option functions contains the definitions of such user-defined aggregation functions. Here it defines the function

    myvwap

. It defines the state variables that will be used to keep the intermediate values for a group, a step computation, and the result computation. Whenever the group changes, the aggregator will reset the state variables to the default values and iterate through the new contents of the group. It will perform the step computation for each row and collect the data in the intermediate variables. After the iteration it will perform the result computation and produce the final value.

The VWAP computation in a weird one, taking two fields as arguments. These two fields get packed into an array reference by

    sub { [$_[0]->get("size"), $_[0]->get("price")];}

and then the step computation unpacks and handles them. In the aggregator computations the syntax

    $%name

refers to the intermediate variables and also to a few pre-defined ones.

    $%argiter

is the value extracted from the current row during the iteration.

And that's pretty much it: send the rows to the table, the iterator state gets updated to match the table contents, computes the results and sends them. For example:

> OP_INSERT,11,abc,123,100 tWindow.aggrVwap OP_INSERT symbol="abc" id="11" volume="100" vwap="123" > OP_INSERT,12,abc,125,300 tWindow.aggrVwap OP_DELETE symbol="abc" id="11" volume="100" vwap="123" tWindow.aggrVwap OP_INSERT symbol="abc" id="12" volume="400" vwap="124.5" > OP_INSERT,13,def,200,100 tWindow.aggrVwap OP_INSERT symbol="def" id="13" volume="100" vwap="200" > OP_INSERT,14,fgh,1000,100 tWindow.aggrVwap OP_INSERT symbol="fgh" id="14" volume="100" vwap="1000" > OP_INSERT,15,abc,128,300 tWindow.aggrVwap OP_DELETE symbol="abc" id="12" volume="400" vwap="124.5" tWindow.aggrVwap OP_INSERT symbol="abc" id="15" volume="700" vwap="126" > OP_INSERT,16,fgh,1100,25 tWindow.aggrVwap OP_DELETE symbol="fgh" id="14" volume="100" vwap="1000" tWindow.aggrVwap OP_INSERT symbol="fgh" id="16" volume="125" vwap="1020" > OP_INSERT,17,def,202,100 tWindow.aggrVwap OP_DELETE symbol="def" id="13" volume="100" vwap="200" tWindow.aggrVwap OP_INSERT symbol="def" id="17" volume="200" vwap="201" > OP_INSERT,18,def,192,1000 tWindow.aggrVwap OP_DELETE symbol="def" id="17" volume="200" vwap="201" tWindow.aggrVwap OP_INSERT symbol="def" id="18" volume="1200" vwap="193.5"

When a group gets modified, the aggregator first sends a DELETE of the old contents, then an INSERT of the new contents. But when the first row gets inserted in a group, there is nothing to delete, and only INSERT is sent. And the opposite, when the last row is deleted from a group, only the DELETE is sent.

After this highlight, let's look at the aggregators from the bottom up.

aggregation
VWAP
SimpleAggregator

## Manual aggregation

The table exanmple in &xrsp; prints the aggregated information (the average price of two records). This can be fairly easily changed to put the information into the rows and send them on as labels. The function

    printAverage()

has morphed into

    computeAverage()

, while the rest of the example stayed the same and is omitted:

    our $rtAvgPrice = Triceps::RowType->new(
    	symbol => "string", # symbol traded
    	id => "int32", # last trade's id
    	price => "float64", # avg price of the last 2 trades
    );

    # place to send the average: could be a dummy label, but to keep the
    # code smaller also print the rows here, instead of in a separate label
    our $lbAverage = $uTrades->makeLabel($rtAvgPrice, "lbAverage",
    	undef, sub { # (label, rowop)
    		&send($_[1]->printP(), "\n");
    	});

    # Send the average price of the symbol in the last modified row
    sub computeAverage # (row)
    {
    	return unless defined $rLastMod;
    	my $rhFirst = $tWindow->findIdx($itSymbol, $rLastMod);
    	my $rhEnd = $rhFirst->nextGroupIdx($itLast2);
    	&send("Contents:\n");
    	my $avg = 0;
    	my ($sum, $count);
    	my $rhLast;
    	for (my $rhi = $rhFirst;
    			!$rhi->same($rhEnd); $rhi = $rhi->nextIdx($itLast2)) {
    		&send("  ", $rhi->getRow()->printP(), "\n");
    		$rhLast = $rhi;
    		$count++;
    		$sum += $rhi->getRow()->get("price");
    	}
    	if ($count) {
    		$avg = $sum/$count;
    		$uTrades->call($lbAverage->makeRowop(&Triceps::OP_INSERT,
    			$rtAvgPrice->makeRowHash(
    				symbol => $rhLast->getRow()->get("symbol"),
    				id => $rhLast->getRow()->get("id"),
    				price => $avg
    			)
    		));
    	}
    }

    while(&readLine) {
    	chomp;
    	my @data = split(/,/);
    	$uTrades->makeArrayCall($tWindow->getInputLabel(), @data);
    	&computeAverage();
    	undef $rLastMod; # clear for the next iteration
    	$uTrades->drainFrame(); # just in case, for completeness
    }

For the demonstration, the aggregated rows sent to

    $lbAverage

get printed. The rows being aggregated are printed during the iteration too, indented after Contents:. And here is a sample run's result, with the input records shown in bold: > OP_INSERT,1,AAA,10,10 Contents: id="1" symbol="AAA" price="10" size="10" lbAverage OP_INSERT symbol="AAA" id="1" price="10" > OP_INSERT,3,AAA,20,20 Contents: id="1" symbol="AAA" price="10" size="10" id="3" symbol="AAA" price="20" size="20" lbAverage OP_INSERT symbol="AAA" id="3" price="15" > OP_INSERT,5,AAA,30,30 Contents: id="3" symbol="AAA" price="20" size="20" id="5" symbol="AAA" price="30" size="30" lbAverage OP_INSERT symbol="AAA" id="5" price="25" > OP_DELETE,3 Contents: id="5" symbol="AAA" price="30" size="30" lbAverage OP_INSERT symbol="AAA" id="5" price="30" > OP_DELETE,5 Contents:

There are a couple of things to notice about it: it produces only the INSERT rowops, no DELETEs, and when the last record of the group is removed, that event produces nothing.

The first item is mildly problematic because the processing downstream from here might not be able to handle the updates properly without the DELETE rowops. It can be worked around fairly easily by connecting another table to store the aggregation results, with the same primary key as the aggregation key. That table would automatically transform the repeated INSERTs on the same key to a DELETE-INSERT sequence.

The second item is actually pretty bad because it means that the last record deleted gets stuck in the aggregation results. The Coral8 solution for this situation is to send a row with all non-key fields set to NULL, to reset them (interestingly, it's a relatively recent addition, that bug took Coral8 years to notice). But with the opcodes available, we can as well send a DELETE rowop with the key fields filled, the helper table will fill in the rest of the fields, and produce a clean DELETE.

All this can be done by the following changes. Add the table, remember its input label in

    $lbAvgPriceHelper

. It will be used to send the aggregated rows instead of

    $tAvgPrice

. Then still use

    $tAvgPrice

to print the records coming out, but now connect it after the helper table. And in

    computeAverage()

change the destination label and add the case for when the group becomes empty (

    $count == 0

). The rest of the example stays the same.

    our $rtAvgPrice = Triceps::RowType->new(
    	symbol => "string", # symbol traded
    	id => "int32", # last trade's id
    	price => "float64", # avg price of the last 2 trades
    );

    our $ttAvgPrice = Triceps::TableType->new($rtAvgPrice)
    	->addSubIndex("bySymbol",
    		Triceps::IndexType->newHashed(key => [ "symbol" ])
    	)
    ;
    $ttAvgPrice->initialize();
    our $tAvgPrice = $uTrades->makeTable($ttAvgPrice, "tAvgPrice");
    our $lbAvgPriceHelper = $tAvgPrice->getInputLabel();

    # place to send the average: could be a dummy label, but to keep the
    # code smaller also print the rows here, instead of in a separate label
    our $lbAverage = makePrintLabel("lbAverage", $tAvgPrice->getOutputLabel());

    # Send the average price of the symbol in the last modified row
    sub computeAverage2 # (row)
    {
    	return unless defined $rLastMod;
    	my $rhFirst = $tWindow->findIdx($itSymbol, $rLastMod);
    	my $rhEnd = $rhFirst->nextGroupIdx($itLast2);
    	&send("Contents:\n");
    	my $avg = 0;
    	my ($sum, $count);
    	my $rhLast;
    	for (my $rhi = $rhFirst;
    			!$rhi->same($rhEnd); $rhi = $rhi->nextIdx($itLast2)) {
    		&send("  ", $rhi->getRow()->printP(), "\n");
    		$rhLast = $rhi;
    		$count++;
    		$sum += $rhi->getRow()->get("price");
    	}
    	if ($count) {
    		$avg = $sum/$count;
    		$uTrades->makeHashCall($lbAvgPriceHelper, &Triceps::OP_INSERT,
    			symbol => $rhLast->getRow()->get("symbol"),
    			id => $rhLast->getRow()->get("id"),
    			price => $avg
    		);
    	} else {
    		$uTrades->makeHashCall($lbAvgPriceHelper, &Triceps::OP_DELETE,
    			symbol => $rLastMod->get("symbol"),
    		);
    	}
    }

The change is straightforward. The label

    $lbAverage

now reverts to just printing the rowops going through it, so it can be created with the template

    makePrintLabel()

described in &xrsp;.

Then the output for the same input becomes:

> OP_INSERT,1,AAA,10,10 Contents: id="1" symbol="AAA" price="10" size="10" tAvgPrice.out OP_INSERT symbol="AAA" id="1" price="10" > OP_INSERT,3,AAA,20,20 Contents: id="1" symbol="AAA" price="10" size="10" id="3" symbol="AAA" price="20" size="20" tAvgPrice.out OP_DELETE symbol="AAA" id="1" price="10" tAvgPrice.out OP_INSERT symbol="AAA" id="3" price="15" > OP_INSERT,5,AAA,30,30 Contents: id="3" symbol="AAA" price="20" size="20" id="5" symbol="AAA" price="30" size="30" tAvgPrice.out OP_DELETE symbol="AAA" id="3" price="15" tAvgPrice.out OP_INSERT symbol="AAA" id="5" price="25" > OP_DELETE,3 Contents: id="5" symbol="AAA" price="30" size="30" tAvgPrice.out OP_DELETE symbol="AAA" id="5" price="25" tAvgPrice.out OP_INSERT symbol="AAA" id="5" price="30" > OP_DELETE,5 Contents: tAvgPrice.out OP_DELETE symbol="AAA" id="5" price="30"

All fixed, the proper DELETEs are coming out. The last line shows the empty group contents in the table but the DELETE row is still coming out.

Why should we worry so much about the DELETEs? Because without them, relying on just INSERTs for updates, it's easy to create bugs. The last example still has an issue with handling the row replacement by INSERTs. Can you spot it from reading the code?

Here is run example that highlights the issue (as usual, the input lines are in bold):

> OP_INSERT,1,AAA,10,10 Contents: id="1" symbol="AAA" price="10" size="10" tAvgPrice.out OP_INSERT symbol="AAA" id="1" price="10" > OP_INSERT,3,AAA,20,20 Contents: id="1" symbol="AAA" price="10" size="10" id="3" symbol="AAA" price="20" size="20" tAvgPrice.out OP_DELETE symbol="AAA" id="1" price="10" tAvgPrice.out OP_INSERT symbol="AAA" id="3" price="15" > OP_INSERT,5,AAA,30,30 Contents: id="3" symbol="AAA" price="20" size="20" id="5" symbol="AAA" price="30" size="30" tAvgPrice.out OP_DELETE symbol="AAA" id="3" price="15" tAvgPrice.out OP_INSERT symbol="AAA" id="5" price="25" > OP_INSERT,5,BBB,30,30 Contents: id="5" symbol="BBB" price="30" size="30" tAvgPrice.out OP_INSERT symbol="BBB" id="5" price="30" > OP_INSERT,7,AAA,40,40 Contents: id="3" symbol="AAA" price="20" size="20" id="7" symbol="AAA" price="40" size="40" tAvgPrice.out OP_DELETE symbol="AAA" id="5" price="25" tAvgPrice.out OP_INSERT symbol="AAA" id="7" price="30"

The row with id=5 has been replaced to change the symbol from AAA to BBB. This act changes both the groups of AAA and of BBB, removing the row from the first one and inserting it into the second one. Yet only the output for BBB came out. The printout of the next row with id=7 and symbol=AAA shows that the row with id=5 has been indeed removed from the group AAA. It even corrects the result. But until that row came in, the average for the symbol AAA remained unchanged and incorrect.

There are multiple ways to fix this issue but first it had to be noticed. Which requires a lot of attention to detail. It's much better to avoid these bugs in the first place by sending the clean and nice input.

aggregation manual
Coral8
aggregation helper table
aggregation of DELETEs

## Introducing the proper aggregation

Since the manual aggregation is error-prone, Triceps can manage it for you and do it right. The only thing you need to do is do the actual iteration and computation. Here is the rewrite of the same example with a Triceps aggregator:

    my $uTrades = Triceps::Unit->new("uTrades");

    # the input data
    my $rtTrade = Triceps::RowType->new(
    	id => "int32", # trade unique id
    	symbol => "string", # symbol traded
    	price => "float64",
    	size => "float64", # number of shares traded
    );

    # the aggregation result
    my $rtAvgPrice = Triceps::RowType->new(
    	symbol => "string", # symbol traded
    	id => "int32", # last trade's id
    	price => "float64", # avg price of the last 2 trades
    );

    # aggregation handler: recalculate the average each time the easy way
    sub computeAverage1 # (table, context, aggop, opcode, rh, state, args...)
    {
    	my ($table, $context, $aggop, $opcode, $rh, $state, @args) = @_;

    	# don't send the NULL record after the group becomes empty
    	return if ($context->groupSize()==0
    		|| $opcode == &Triceps::OP_NOP);

    	my $sum = 0;
    	my $count = 0;
    	for (my $rhi = $context->begin(); !$rhi->isNull();
    			$rhi = $context->next($rhi)) {
    		$count++;
    		$sum += $rhi->getRow()->get("price");
    	}
    	my $rLast = $context->last()->getRow();
    	my $avg = $sum/$count;

    	my $res = $context->resultType()->makeRowHash(
    		symbol => $rLast->get("symbol"),
    		id => $rLast->get("id"),
    		price => $avg
    	);
    	$context->send($opcode, $res);
    }

    my $ttWindow = Triceps::TableType->new($rtTrade)
    	->addSubIndex("byId",
    		Triceps::IndexType->newHashed(key => [ "id" ])
    	)
    	->addSubIndex("bySymbol",
    		Triceps::IndexType->newHashed(key => [ "symbol" ])
    		->addSubIndex("last2",
    			Triceps::IndexType->newFifo(limit => 2)
    			->setAggregator(Triceps::AggregatorType->new(
    				$rtAvgPrice, "aggrAvgPrice", undef, \&computeAverage1)
    			)
    		)
    	)
    ;
    $ttWindow->initialize();
    my $tWindow = $uTrades->makeTable($ttWindow, "tWindow");

    # label to print the result of aggregation
    my $lbAverage = makePrintLabel("lbAverage",
    	$tWindow->getAggregatorLabel("aggrAvgPrice"));

    while(&readLine) {
    	chomp;
    	my @data = split(/,/); # starts with a string opcode
    	$uTrades->makeArrayCall($tWindow->getInputLabel(), @data);
    	$uTrades->drainFrame(); # just in case, for completeness
    }

What has changed in this code? The things got rearranged a bit.The aggregator is now defined as a part of the table type, so the aggregation result row type and its computation function had to be moved up.

The AggregatorType object holds the information about the aggregator. In the table type, the aggregator type gets attached to an index type with

    setAggregator()

. In this case, to the FIFO index type. The parent of that index type determines the aggregation groups, grouping happening by its combined key fields (that is, all the key fields of all the indexes in the path starting from the root). For aggregation the working or non-working method

    getKey()

doesn't matter, so any of the Hashed, Ordered and Sorted/SimpleOrdered index types can be used. The index type where the aggregator type is attached determines the order of the rows in the groups. If you use FIFO, the rows will be in the order of arrival. If you use Ordered or Sorted, the rows will be in the sort order. If you use Hashed, the rows will be in some random order, which is not particularly useful.

At present an index type may have no more than one aggregator type attached to it. There is no particular reason for that, other than that it was slightly easier to implement, and that I can't think yet of a real-word situation where multiple aggregators on the same index would be needed. If this situation will ever occur, this support can be added. However a table type may have multiple aggregator types in it, on different indexes. You can save a reference to an aggregator type in a variable and reuse it in the different table types too (though not multiple times in the same table, since that would cause a naming conflict).

The aggregator type is created with the arguments of

- result row type,
- aggregator name,
- group initialization Perl function (which may be

      undef

  , as in this example),
- group computation Perl function or source code snippet,
- the optional arguments for the functions.

Note that there is a difference in naming between the aggregator types and index types: an aggregator type knows its name, while an index type does not. An index type is given a name only in its hierarchy inside the table type, but it does not know its name.

When a table is created, it finds all the aggregator types in it, and creates an output label for each of them. The names of the aggregator types are used as suffixes to the table name. In this example the aggregator will have its output label named tWindow.aggrAvgPrice. This puts all the aggregator types in the table into the same namespace, so make sure to give them different names in the same table type. Also avoid the names in, out and pre because these are already taken by the table's own labels. The aggregator labels in the table can be found with

    $aggLabel = $table->getAggregatorLabel("aggName");

The aggregator types are theoretically multithreaded but the way the Perl threads work, the Perl code has to be recompiled from the source code in each thread. So for a table type with aggregators to be exportable to the other threads, the aggregators must have their logic specified as the Perl source code, not a compiled Perl function.

After the logic is moved into a managed aggregator, the main loop becomes simpler.

The computation function gets a lot more arguments than it used to. The most interesting and most basic ones are

    $context

,

    $opcode

, and

    $rh

. The rest are useful in the more complex cases only.

The aggregator type is exactly that: a type. It doesn't know, on which table or index, or even index type it will be used. And indeed, it might be used on multiple tables and index types. But to do the iteration on the rows, the computation function needs to get this information somehow. And it does, in the form of aggregator context. The manual aggregation used the last table output row to find, on which exact group to iterate. The managed aggregator gets the last modified row handle as the argument

    $rh

. But our simple aggregator doesn't even need to consult

    $rh

because the context takes care of finding the group too: it knows the exact group and exact index that needs to be aggregated (look at the index tree drawings in &xrsp; for the difference between an index type and an index).

The context provides its own

    begin()

and

    next()

methods. They are actually slightly more efficient than the usual table iteration methods because they take advantage of that exact known index. The most important part, they work differently.

    $rhi = $context->next($rhi);

returns a NULL row handle when it reaches the end of the group. Do not, I repeat, DO NOT use the

    $rhi->next()

in the aggregators, or you'll get some very wrong results.

The context also has a bit more of its own magic.

    $rh = $context->last();

returns the last row handle in the group. This comes very handy because in most of the cases you want the data from the last row to fill the fields that haven't been aggregated as such. This is like the SQL function

    LAST()

. Using the fields from the argument

    $rh

, unless they are the key fields for this group, is generally not a good idea because it adds an extra dependency on the order of modifications to the table. The

    FIRST()

or

    LAST()

(i.e. the context's

    begin()

or

    last()

) are much better and not any more expensive.

    $size = $context->groupSize();

returns the number of rows in the group. It's your value of

    COUNT(*)

in SQL terms, and if that's all you need, you don't need to iterate.

    $context->send($opcode, $row);

constructs a result rowop and sends it to the aggregator's output label. Remember, the aggregator type as such knows nothing about this label, so the path through the context is the only path. Note also that it takes a row and not a rowop, because a label is needed to construct the rowop in the first place.

    $rt = $context->resultType();

provides the result row type needed to construct the result row. There also are a couple of convenience methods that combine the row construction and sending, that can be used instead:

    $context->makeHashSend ($opcode, $fieldName => $fieldValue, ...);
    $context->makeArraySend($opcode, @fieldValues);

The final thing about the aggregator context: it works only inside the aggregator computation function. Once the function returns, all its methods start returning

    undef

. So there is no point in trying to save it for later in a global variable or such, don't do that.

As you can see,

    computeAverage()

has the same logic as before, only now it uses the aggregation context. And I've removed the debugging printout of the rows in the group.

The last unexplained piece is the opcode handling and that comparison to

    OP_NOP

. Basically, the table calls the aggregator computation every time something changes in its index. It describes the reason for the call in the argument

    $aggop

(aggregation operation). Depending on how clever an aggregator wants to be, it may do something useful on all of these occasions, or only on some of them. The simple aggregator that doesn't try any smart optimizations but just goes and iterates through the rows every time only needs to react in some of the cases. To make its life easier, Triceps pre-computes the opcode that should be used for the result and puts it into the argument

    $opcode

. So to ignore the non-interesting calls, the simple aggregator computation can just return if it sees the opcode

    OP_NOP

.

Why does it also check for the group size being 0? Again, Triceps provides flexibility in the aggregators. Among other things, it allows to implement the logic like Coral8, when on deletion of the last row in the group the aggregator would send a row with all non-key fields set to NULL (it can take the key fields from the argument

    $rh

). So for this specific purpose the computation function gets called with all rows deleted from the group, and

    $opcode

set to

    OP_INSERT

. And, by the way, a true Coral8-styled aggregator would ignore all the calls where the

    $opcode

is not

    OP_INSERT

. But the normal aggregators need to avoid doing this kind of crap, so they have to ignore the calls where

    $context->groupSize()==0

.

And here is an example of the output from that code (as usual, the input lines are in bold):

> OP_INSERT,1,AAA,10,10 tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="1" price="10" > OP_INSERT,3,AAA,20,20 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="1" price="10" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="3" price="15" > OP_INSERT,5,AAA,30,30 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="3" price="15" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="5" price="25" > OP_DELETE,3 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="5" price="25" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="5" price="30" > OP_DELETE,5 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="5" price="30"

As you can see, it's exactly the same as from the manual aggregation example with the helper table, minus the debugging printout of the group contents. However here it's done without the helper table: instead the aggregation function is called before and after each update.

This presents a memory vs CPU compromise: a helper table uses more memory but requires less CPU for the aggregation computations (presumably, the insertion of the row into the table is less computationally intensive than the iteration through the original records).

The managed aggregators can be made to work with a helper table too: just chain a helper table to the aggregator's label, and in the aggregator computation add

    return if ($opcode == &Triceps::OP_DELETE
    	&& $context->groupSize() != 1);

This would skip all the DELETEs except for the last one, before the group collapses.

There is also a way to optimize this logic right inside the aggregator: remember the last INSERT row sent, and on DELETE just resend the same row, as will be shown in &xrsp;. This remembered last state can also be used for the other interesting optimizations that will be shown in &xrsp;.

Which approach is better, depends on the particular case. If you need to store the results of aggregation in a table for the future look-ups anyway, then that table is no extra overhead. That's what the Aleri system does internally: since each element in its model keeps a primary-indexed table (materialized view) of the result, that table is used whenever possible to generate the DELETEs without involving any logic. Or the extra optimization inside the aggregator can seriously improve the performance on the large groups. Sometimes you may want both.

Now let's look at the run with the same input that went wrong with the manual aggregation:

> OP_INSERT,1,AAA,10,10 tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="1" price="10" > OP_INSERT,3,AAA,20,20 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="1" price="10" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="3" price="15" > OP_INSERT,5,AAA,30,30 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="3" price="15" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="5" price="25" > OP_INSERT,5,BBB,30,30 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="5" price="25" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="3" price="20" tWindow.aggrAvgPrice OP_INSERT symbol="BBB" id="5" price="30" > OP_INSERT,7,AAA,40,40 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="3" price="20" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="7" price="30"

Here it goes right. Triceps recognizes that the second INSERT with id=5 moves the row to another group. So it performs the aggregation logic for both groups. First for the group where the row gets removed, it updates the aggregator result with a DELETE and INSERT (note that id became 3, since it's now the last row left in that group). Then for the group where the row gets added, and since there was nothing in that group before, it generates only an INSERT.

The handling of the fatal errors (as in

    die()

) in the aggregator functions is an interesting subject. The errors propagate properly through the table, and the table operations confess with the Perl handler's error message. But since an error in the aggregator function means that things are going very, very wrong, after that the table becomes inoperative and will die on all the subsequent operations as well. You need to be very careful in writing these functions.
AggregatorType
index aggregation
aggregation context
AggregatorContext
aggregation iteration
aggregation last
aggregation first
SQL
aggregation count
aggregation opcode
Coral8
aggregation helper table
aggregation optimization
Aleri
error handling

## Tricks with aggregation on a sliding window

Now it all works as it should, but there is still some room for improvement, related to the way the sliding window limits are handled.

Let's look again at the sample aggregation output with row deletion, copied here for convenience:

> OP_INSERT,1,AAA,10,10 tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="1" price="10" > OP_INSERT,3,AAA,20,20 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="1" price="10" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="3" price="15" > OP_INSERT,5,AAA,30,30 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="3" price="15" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="5" price="25" > OP_DELETE,3 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="5" price="25" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="5" price="30" > OP_DELETE,5 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="5" price="30"

When the row with id=3 is deleted, the average price reverts to 30, which is the price of the trade with id=5, not the average of trades with id 1 and 5.

This is because the table is actually a sliding window, with the FIFO index having a limit of 2 rows

    my $ttWindow = Triceps::TableType->new($rtTrade)
    	->addSubIndex("byId",
    		Triceps::IndexType->newHashed(key => [ "id" ])
    	)
    	->addSubIndex("bySymbol",
    		Triceps::IndexType->newHashed(key => [ "symbol" ])
    		->addSubIndex("last2",
    			Triceps::IndexType->newFifo(limit => 2)
    			->setAggregator(Triceps::AggregatorType->new(
    				$rtAvgPrice, "aggrAvgPrice", undef, \&computeAverage1)
    			)
    		)
    	)
    ;

When the row with id=5 was inserted, it pushed out the row with id=1. Deleting the record with id=3 does not put that row with id=1 back. You can see the group contents in an even earlier printout with the manual aggregation, also copied here for convenience:

> OP_INSERT,1,AAA,10,10 Contents: id="1" symbol="AAA" price="10" size="10" lbAverage OP_INSERT symbol="AAA" id="1" price="10" > OP_INSERT,3,AAA,20,20 Contents: id="1" symbol="AAA" price="10" size="10" id="3" symbol="AAA" price="20" size="20" lbAverage OP_INSERT symbol="AAA" id="3" price="15" > OP_INSERT,5,AAA,30,30 Contents: id="3" symbol="AAA" price="20" size="20" id="5" symbol="AAA" price="30" size="30" lbAverage OP_INSERT symbol="AAA" id="5" price="25" > OP_DELETE,3 Contents: id="5" symbol="AAA" price="30" size="30" lbAverage OP_INSERT symbol="AAA" id="5" price="30" > OP_DELETE,5 Contents:

Like the toothpaste, once out of the tube, it's not easy to put back. But for this particular kind of toothpaste there is a trick: keep more rows in the group just in case but use only the last few for the actual aggregation. To allow an occasional deletion of a single row, we can keep 3 rows instead of 2.

So, change the table definition:

    ...
    			Triceps::IndexType->newFifo(limit => 3)
    ...

and modify the aggregator function to use only the last 2 rows from the group, even if more are available:

    sub computeAverage2 # (table, context, aggop, opcode, rh, state, args...)
    {
    	my ($table, $context, $aggop, $opcode, $rh, $state, @args) = @_;

    	# don't send the NULL record after the group becomes empty
    	return if ($context->groupSize()==0
    		|| $opcode == &Triceps::OP_NOP);

    	my $skip = $context->groupSize()-2;
    	my $sum = 0;
    	my $count = 0;
    	for (my $rhi = $context->begin(); !$rhi->isNull();
    			$rhi = $context->next($rhi)) {
    		if ($skip > 0) {
    			$skip--;
    			next;
    		}
    		$count++;
    		$sum += $rhi->getRow()->get("price");
    	}
    	my $rLast = $context->last()->getRow();
    	my $avg = $sum/$count;

    	my $res = $context->resultType()->makeRowHash(
    		symbol => $rLast->get("symbol"),
    		id => $rLast->get("id"),
    		price => $avg
    	);
    	$context->send($opcode, $res);
    }

The output from this version becomes:

> OP_INSERT,1,AAA,10,10 tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="1" price="10" > OP_INSERT,3,AAA,20,20 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="1" price="10" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="3" price="15" > OP_INSERT,5,AAA,30,30 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="3" price="15" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="5" price="25" > OP_DELETE,3 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="5" price="25" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="5" price="20" > OP_DELETE,5 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="5" price="20" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="1" price="10"

Now after

    OP_DELETE,3

the average price becomes 20, the average of 10 and 30, because the row with id=1 comes into play again. Can you repeat that in the SQLy languages?

This version stores one extra row and thus can handle only one deletion (until the deleted row's spot gets pushed out of the window naturally, then it can handle another). It can not handle the arbitrary modifications properly. If you insert another row with id=3 for the same symbol AAA, the new version will be placed again at the end of the window. If it was the last row anyway, that is fine. But if it was not the last, as in this example, that would be an incorrect order that will produce incorrect results.

But just change the table type definition to aggregate on a sorted index instead of FIFO and it becomes able to handle the updates while keeping the rows in the order of their ids:

    my $ttWindow = Triceps::TableType->new($rtTrade)
    	->addSubIndex("byId",
    		Triceps::IndexType->newHashed(key => [ "id" ])
    	)
    	->addSubIndex("bySymbol",
    		Triceps::IndexType->newHashed(key => [ "symbol" ])
    		->addSubIndex("orderById",
    			Triceps::IndexType->newOrdered(key => [ "id" ])
    			->setAggregator(Triceps::AggregatorType->new(
    				$rtAvgPrice, "aggrAvgPrice", undef, \&computeAverage3)
    			)
    		)
    		->addSubIndex("last3",
    			Triceps::IndexType->newFifo(limit => 3))
    	)
    ;

The FIFO index is still there, in parallel, but it doesn't determine the order of rows for aggregation any more. Here is a sample of this version's work:

> OP_INSERT,1,AAA,10,10 tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="1" price="10" > OP_INSERT,3,AAA,20,20 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="1" price="10" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="3" price="15" > OP_INSERT,5,AAA,30,30 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="3" price="15" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="5" price="25" > OP_DELETE,3 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="5" price="25" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="5" price="20" > OP_INSERT,3,AAA,20,20 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="5" price="20" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="5" price="25" > OP_INSERT,7,AAA,40,40 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="5" price="25" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="7" price="35"

When the row with id=3 gets deleted, the average reverts to the rows 1 and 5. When the row 3 gets inserted back, the average works on rows 3 and 5 again. Then when the row 7 is inserted, the aggregation moves up to the rows 5 and 7.

The row expiration is still controlled by the FIFO index. So after the row 3 is inserted back, the order of rows in the FIFO becomes

    1, 5, 3

Then when the row 7 is inserted, it advances to

    5, 3, 7

At this point, until the row 3 gets naturally popped out of the FIFO, it's best not to have other deletions nor updates, or the group contents may become incorrect.

The FIFO and Ordered index types work in parallel on the same group, and the Ordered index always keeps the right order:

    1, 3, 5
    3, 5, 7

At long as the records with the two highest ids are in the group at all, the Ordered index will keep them in the right position at the end.

In this case we could even make a bit of optimization: turn the sorting order around, and have the Ordered index arrange the rows in the descending order. Then instead of skipping the rows until the last two, just take the first two rows of the reverse order. They'll be iterated in the opposite direction but for the averaging it doesn't matter. And instead of the last row take the first row of the opposite order. This is a simple modification and is left as an exercise for the reader.

Thinking further, the sensitivity to the ordering comes largely from the FIFO index. If the replacement policy could be done directly on the Ordered index, it would become easier. Would be a good thing to add in the future. Also, if you keep all the day's trades anyway, you might not need to have a replacement policy at all: just pick the last 2 records for the aggregation. There is currently no way to iterate back from the end (another thing to add in the future) but the same trick with the opposite order would work.

For a new subject, this table type indexes by id twice: once as a primary index, another time as a nested one. Are both of them really necessary or would just the nested one be good enough? That depends on your input data. If you get the DELETEs like

    OP_DELETE,3

with all the other fields as NULL, then a separate primary index is definitely needed. But if the DELETEs come exactly as the same records that were inserted, only with a different opcode, like

    OP_DELETE,3,AAA,20,20

then the primary index can be skipped because the nested sorted index will be able to find the rows correctly and handle them. The bottom line is, the fully correct DELETE records are good.
index ordered
index FIFO
table remove row

## Optimized DELETEs

I've already mentioned that the DELETEs coming out of an aggregator do not have to be recalculated every time. Instead the rows can be remembered from the insert time, and simply re-sent with the new opcode. That allows to trade the CPU time for the extra memory. Of course, this works best when there are many rows per aggregation group, then more CPU time is saved on not iterating through them. How many is many? It depends on the particular cases. You'd have to measure. Anyway, here is how it's done:

    sub computeAverage4 # (table, context, aggop, opcode, rh, state, args...)
    {
    	my ($table, $context, $aggop, $opcode, $rh, $state, @args) = @_;

    	# don't send the NULL record after the group becomes empty
    	return if ($context->groupSize()==0
    		|| $opcode == &Triceps::OP_NOP);
    	if ($opcode == &Triceps::OP_DELETE) {
    		$context->send($opcode, $$state);
    		return;
    	}

    	my $sum = 0;
    	my $count = 0;
    	for (my $rhi = $context->begin(); !$rhi->isNull();
    			$rhi = $context->next($rhi)) {
    		$count++;
    		$sum += $rhi->getRow()->get("price");
    	}
    	my $rLast = $context->last()->getRow();
    	my $avg = $sum/$count;

    	my $res = $context->resultType()->makeRowHash(
    		symbol => $rLast->get("symbol"),
    		id => $rLast->get("id"),
    		price => $avg
    	);
    	${$state} = $res;
    	$context->send($opcode, $res);
    }

    sub initRememberLast #  (@args)
    {
    	my $refvar;
    	return \$refvar;
    }

    my $ttWindow = Triceps::TableType->new($rtTrade)
    	->addSubIndex("byId",
    		Triceps::IndexType->newHashed(key => [ "id" ])
    	)
    	->addSubIndex("bySymbol",
    		Triceps::IndexType->newHashed(key => [ "symbol" ])
    		->addSubIndex("last2",
    			Triceps::IndexType->newFifo(limit => 2)
    			->setAggregator(Triceps::AggregatorType->new(
    				$rtAvgPrice, "aggrAvgPrice", \&initRememberLast, \&computeAverage4)
    			)
    		)
    	)
    ;

The rest of the example stays the same, so it's not shown. Even in the part that is shown, very little has changed.

The aggregator type now has an initialization function. (This function is not of the same kind as for the sorted index!) This function gets called every time a new aggregation group gets created, before the first row is inserted into it. It initializes the aggregator group's Perl state by creating and returning the state value (the state is per aggregator type, so if there are two parallel index types, each with an aggregator, each aggregator will have its own group state).

The state is stored in the group as a single Perl variable. So it usually is a reference to a more complex object. In this case the value returned is a reference to a variable that would contain a Row reference. (Ironically, the simplest case looks a bit more confusing than if it were a reference to an array or hash). Returning a reference to a

    my

variable is a way to create a reference to an anonymous value: each time

    my

executes, it creates a new value. Which is then kept in a reference after the initialization function returns. The next time the function executes,

    my

would create another new value.

The computation function has that state passed as an argument and now makes use of it. It has two small additions. Before sending a new result row, that row gets remembered in the state reference. And then before doing any computation the function checks, whether the required opcode is DELETE, and if so then simply resends the last result with the new opcode. Remember, the rows are not copied but reference-counted, so this is fairly cheap.

The extra level of referencing is used because simply assigning to

    $state

would only change the local variable and not the value kept in the group.

However if you change the argument of the function directly, that would change the value kept in the group (similar to changing the loop variable in a foreach loop). So you can save a bit of overhead by eliminating the extra indirection. The modified version will be:

    sub computeAverage5 # (table, context, aggop, opcode, rh, state, args...)
    {
    	my ($table, $context, $aggop, $opcode, $rh, $state, @args) = @_;

    	# don't send the NULL record after the group becomes empty
    	return if ($context->groupSize()==0
    		|| $opcode == &Triceps::OP_NOP);
    	if ($opcode == &Triceps::OP_DELETE) {
    		$context->send($opcode, $state);
    		return;
    	}

    	my $sum = 0;
    	my $count = 0;
    	for (my $rhi = $context->begin(); !$rhi->isNull();
    			$rhi = $context->next($rhi)) {
    		$count++;
    		$sum += $rhi->getRow()->get("price");
    	}
    	my $rLast = $context->last()->getRow();
    	my $avg = $sum/$count;

    	my $res = $context->resultType()->makeRowHash(
    		symbol => $rLast->get("symbol"),
    		id => $rLast->get("id"),
    		price => $avg
    	);
    	$_[5] = $res;
    	$context->send($opcode, $res);
    }

    sub initRememberLast5 #  (@args)
    {
    	return undef;
    }

Even though the initialization function returns

    undef

, it still must be present. If it's not present, the state argument of the comparison function will contain a special hardcoded and unmodifiable

    undef

constant, and nothing could be remembered.

And here is an example of its work:

> OP_INSERT,1,AAA,10,10 tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="1" price="10" > OP_INSERT,2,BBB,100,100 tWindow.aggrAvgPrice OP_INSERT symbol="BBB" id="2" price="100" > OP_INSERT,3,AAA,20,20 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="1" price="10" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="3" price="15" > OP_INSERT,4,BBB,200,200 tWindow.aggrAvgPrice OP_DELETE symbol="BBB" id="2" price="100" tWindow.aggrAvgPrice OP_INSERT symbol="BBB" id="4" price="150" > OP_INSERT,5,AAA,30,30 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="3" price="15" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="5" price="25" > OP_DELETE,3 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="5" price="25" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="5" price="30" > OP_DELETE,5 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="5" price="30"

Since the rows are grouped by the symbol, the symbols AAA and BBB will have separate aggregation states.

aggregation optimization
aggregation initialization
aggregation state

## Additive aggregation

In some cases the aggregation values don't have to be calculated by going through all the rows from scratch every time. If you do a sum of a field, you can as well add the value of the field when a row is inserted and subtract when a row is deleted. Not surprisingly, this is called an additive aggregation.

The averaging can also be done as an additive aggregation: it amounts to a sum divided by a count. The sum can obviously be done additively. The count is potentially additive too, but even better, we have the shortcut of

    $context->groupSize()

. Well, at least for the same definition of count that has been used previously in the non-additive example. The SQL definition of count (and of average) includes only the non-NULL values, but in the next example we will go with the Perl approach where a NULL is taken to have the same meaning as 0. The proper SQL count could not use that shortcut but would still be additive.

Triceps provides a way to implement the additive aggregation too. It calls the aggregation computation function for each changed row, giving it an opportunity to react. The argument

    $aggop

indicates, what has happened. Here is the same example from &xrsp; rewritten in an additive way:

    sub computeAverage7 # (table, context, aggop, opcode, rh, state, args...)
    {
    	my ($table, $context, $aggop, $opcode, $rh, $state, @args) = @_;
    	my $rowchg;

    	if ($aggop == &Triceps::AO_BEFORE_MOD) {
    		$context->send($opcode, $state->{lastrow});
    		return;
    	} elsif ($aggop == &Triceps::AO_AFTER_DELETE) {
    		$rowchg = -1;
    	} elsif ($aggop == &Triceps::AO_AFTER_INSERT) {
    		$rowchg = 1;
    	} else { # AO_COLLAPSE, also has opcode OP_DELETE
    		return
    	}

    	$state->{price_sum} += $rowchg * $rh->getRow()->get("price");

    	return if ($context->groupSize()==0
    		|| $opcode == &Triceps::OP_NOP);

    	my $rLast = $context->last()->getRow();
    	my $count = $context->groupSize();
    	my $avg = $state->{price_sum}/$count;
    	my $res = $context->resultType()->makeRowHash(
    		symbol => $rLast->get("symbol"),
    		id => $rLast->get("id"),
    		price => $avg
    	);
    	$state->{lastrow} = $res;

    	$context->send($opcode, $res);
    }

    sub initAverage7 #  (@args)
    {
    	return { lastrow => undef, price_sum => 0 };
    }

The tricks of keeping an extra row from &xrsp; could not be used with the additive aggregation. An additive aggregation relies on Triceps to tell it, which rows are deleted and which inserted, so it can not do any extra skipping easily. The index for the aggregation has to be defined with the correct limits. If we want an average of the last 2 rows, we set the limit to 2:

    my $ttWindow = Triceps::TableType->new($rtTrade)
    	->addSubIndex("byId",
    		Triceps::IndexType->newHashed(key => [ "id" ])
    	)
    	->addSubIndex("bySymbol",
    		Triceps::IndexType->newHashed(key => [ "symbol" ])
    		->addSubIndex("last2",
    			Triceps::IndexType->newFifo(limit => 2)
    			->setAggregator(Triceps::AggregatorType->new(
    				$rtAvgPrice, "aggrAvgPrice", \&initAverage7, \&computeAverage7)
    			)
    		)
    	)
    ;

The aggregation state has grown: now it includes not only the last sent row but also the sum of the price, which is used for the aggregation, kept together in a hash. The last sent row doesn't really have to be kept, and I'll show another example without it, but for now let's look at how things are done when it is kept.

The argument

    $aggop

describes, why the computation is being called. Note that Triceps doesn't know if the aggregation is additive or not. It does the calls the same in every case. Just in the previous examples we weren't interested in this information and didn't look at it.

    $aggop

contains one of the constant values:

-     &Triceps::AO_BEFORE_MOD

  : the group is about to be modified, need to send a DELETE of the old aggregated row. The argument

      $opcode

  will always be

      OP_DELETE

  .

-     &Triceps::AO_AFTER_DELETE

  : the group has been modified by deleting a row from it. The argument

      $rh

  will refer to the row handle being deleted. The

      $opcode

  may be either

      OP_NOP

  or

      OP_INSERT

  . A single operation on a table may affect multiple rows: an insert may trigger the replacement policy in the indexes and cause one or more rows to be deleted. If there are multiple rows deleted or inserted in a group, the additive aggregator needs to know about all of them to keep its state correct but does not need (and even must not) send a new result until the last one of them has been processed. The call for the last modification will have the opcode of

      OP_INSERT

  . The preceding intermediate ones will have the opcode of

      OP_NOP

  . An important point, even though a row is being deleted from the group, the aggregator opcode is

      OP_INSERT

  , because it inserts the new aggregator state!

-     &Triceps::AO_AFTER_INSERT

  : the group has been modified by inserting a row into it. Same as for

      AO_AFTER_DELETE

  ,

      $rh

  will refer to the row handle being inserted, and

      $opcode

  will be

      OP_NOP

  or

      OP_INSERT

  .

-     &Triceps::AO_COLLAPSE

  : called after the last row is deleted from the group, just before the whole group is collapsed and deleted. This allows the aggregator to destroy its state properly. For most of the aggregators there is nothing special to be done. The only case when you want to do something is if your state causes some circular references. Perl doesn't free the circular references until the whole interpreter exits, and so you'd have to break the circle to let them be freed immediately. The aggregator should not produce any results on this call. The

      $opcode

  will be

      OP_NOP

  .

memory management

The computation reacts accordingly: for the before-modification it re-sends the old result with the new opcode, for the collapse it does nothing, and for after-modification it calculates the sign, whether the value from

    $rh

needs to be added or subtracted from the sum. I'm actually thinking, maybe this sign should be passed as a separate argument too, and then both the aggregation operation constants

    AO_AFTER_*

can be merged into one. We'll see, maybe it will be changed in the future.

Then the addition/subtraction is done and the state updated.

After that, if the row does not need to be sent (opcode is

    OP_NOP

or group size is 0), the function can as well return here without constructing the new row.

If the row needs to be produced, continue with the same logic as the non-additive aggregator, only without iteration through the group. The id field in the result is produced by essentially the SQL

    LAST()

operator.

    LAST()

and

    FIRST()

are not additive, they refer to the values in the last or first row in the group's order, and simply can not be calculated from looking at which rows are being inserted and deleted without knowing their order in the group. But they are fast as they are, and do not require iteration. The same goes for the row count (as long as we don't care about excluding NULLs, violating the SQL semantics). And for averaging there is the last step to do after the additive part is done: divide the sum by the count.

All these non-additive steps are done in this last section, then the result row is constructed, remembered and sent.

Not all the aggregation operations can be expressed in an additive way. It may even vary by the data. For

    MAX()

, the insertion of a row can be always done additively, just comparing the new value with the remembered maximum, and replacing it if the new value is greater. The deletion can also compare the deleted value with the remembered maximum. If the deleted value is less, then the maximum is unchanged. But if the deleted value is equal to the maximum,

    MAX()

has to iterate through all the values and find the new maximum.

There is also an issue with the floating point precision in the additive aggregation. It's not such a big issue if the rows are only added and never deleted from the group, but can get much worse with the deletion. Let me show it with a sample run of the additive code:

> OP_INSERT,1,AAA,1,10 tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="1" price="1" > OP_INSERT,2,AAA,1e20,20 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="1" price="1" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="2" price="5e+19" > OP_INSERT,3,AAA,2,10 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="2" price="5e+19" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="3" price="5e+19" > OP_INSERT,4,AAA,3,10 tWindow.aggrAvgPrice OP_DELETE symbol="AAA" id="3" price="5e+19" tWindow.aggrAvgPrice OP_INSERT symbol="AAA" id="4" price="1.5"

Why is the last result 1.5 while it had to be (2+3)/2 = 2.5? Because adding together 1e20 and 2 had pushed the 2 beyond the precision of floating-point number. 1e20+2 = 1e20. So when the row with 1e20 was deleted from the group and subtracted form the sum, that left 0. Which got then averaged with 3, producing 1.5.

Of course, with the real stock prices there won't be that much variation. But the subtler errors will still accumulate over time, and you have to expect them and plan accordingly.

Switching to a different subject, the additive aggregation contains enough information in its state to generate the result rows quickly without an iteration. This means that keeping the saved result row for DELETEs doesn't give a whole lot of advantage and adds at least a little memory overhead. We can change the code and avoid keeping it:

    sub computeAverage8 # (table, context, aggop, opcode, rh, state, args...)
    {
    	my ($table, $context, $aggop, $opcode, $rh, $state, @args) = @_;
    	my $rowchg;

    	if ($aggop == &Triceps::AO_COLLAPSE) {
    		return
    	} elsif ($aggop == &Triceps::AO_AFTER_DELETE) {
    		$state->{price_sum} -= $rh->getRow()->get("price");
    	} elsif ($aggop == &Triceps::AO_AFTER_INSERT) {
    		$state->{price_sum} += $rh->getRow()->get("price");
    	}
    	# on AO_BEFORE_MOD do nothing

    	return if ($context->groupSize()==0
    		|| $opcode == &Triceps::OP_NOP);

    	my $rLast = $context->last()->getRow();
    	my $count = $context->groupSize();

    	$context->makeHashSend($opcode,
    		symbol => $rLast->get("symbol"),
    		id => $rLast->get("id"),
    		price => $state->{price_sum}/$count,
    	);
    }

    sub initAverage8 #  (@args)
    {
    	return { price_sum => 0 };
    }

On

    AO_BEFORE_MOD

it doesn't do any change to the additive state but then produces the result row from that state as usual, using the supplied

    $opcode

value of

    OP_DELETE

. The other change in this example is that the sum gets directly added or subtracted in

    AO_AFTER_*

instead of computing the sign first. It's all pretty much self-explanatory.
aggregation optimization
aggregation additive
SQL
aggregation state
aggregation operation
SQL
aggregation floating point error

## Computation function arguments

Let's look up close at what calls are done to the aggregation computation function. Just make a computation that prints the call arguments:

    sub computeAverage9 # (table, context, aggop, opcode, rh, state, args...)
    {
    	my ($table, $context, $aggop, $opcode, $rh, $state, @args) = @_;

    	&send(&Triceps::aggOpString($aggop), " ", &Triceps::opcodeString($opcode), " ", $context->groupSize(), " ", (!$rh->isNull()? $rh->getRow()->printP(): "NULL"), "\n");
    }

It prints the aggregation operation, the result opcode, row count in the group, and the argument row (or NULL). The aggregation is done as before, on the same FIFO index with the size limit of 2.

To show the order of aggregator calls relative to the table label calls, I've added the labels that print the updates form the table:

    my $lbPre = makePrintLabel("lbPre", $tWindow->getPreLabel());
    my $lbOut = makePrintLabel("lbOut", $tWindow->getOutputLabel());

To make keeping track of the printout easier, I broke up the sequence into multiple fragments, with a description after each fragment:

> OP_INSERT,1,AAA,10,10 tWindow.pre OP_INSERT id="1" symbol="AAA" price="10" size="10" tWindow.out OP_INSERT id="1" symbol="AAA" price="10" size="10" AO_AFTER_INSERT OP_INSERT 1 id="1" symbol="AAA" price="10" size="10" > OP_INSERT,2,BBB,100,100 tWindow.pre OP_INSERT id="2" symbol="BBB" price="100" size="100" tWindow.out OP_INSERT id="2" symbol="BBB" price="100" size="100" AO_AFTER_INSERT OP_INSERT 1 id="2" symbol="BBB" price="100" size="100"

The INSERT of the first row in each group causes only one call. There is no previous value to delete, only a new one to insert. The call happens after the row has been inserted into the group.

> OP_INSERT,3,AAA,20,20 AO_BEFORE_MOD OP_DELETE 1 NULL tWindow.pre OP_INSERT id="3" symbol="AAA" price="20" size="20" tWindow.out OP_INSERT id="3" symbol="AAA" price="20" size="20" AO_AFTER_INSERT OP_INSERT 2 id="3" symbol="AAA" price="20" size="20"

Adding the second record in a group means that the aggregation result for this group is modified. So first the aggregator is called to delete the old result, then the new row gets inserted, and the aggregator is called the second time to produce its new result.

> OP_INSERT,5,AAA,30,30 AO_BEFORE_MOD OP_DELETE 2 NULL tWindow.pre OP_DELETE id="1" symbol="AAA" price="10" size="10" tWindow.out OP_DELETE id="1" symbol="AAA" price="10" size="10" tWindow.pre OP_INSERT id="5" symbol="AAA" price="30" size="30" tWindow.out OP_INSERT id="5" symbol="AAA" price="30" size="30" AO_AFTER_DELETE OP_NOP 2 id="1" symbol="AAA" price="10" size="10" AO_AFTER_INSERT OP_INSERT 2 id="5" symbol="AAA" price="30" size="30"

The insertion of the third row in a group triggers the replacement policy in the FIFO index. The replacement policy causes the row with id=1 to be deleted before the row with id=5 is inserted. For the aggregator result it's still a single delete-insert pair: First, before modification, the old aggregation result is deleted. Then the contents of the group gets modified with both the delete and insert. And then the aggregator gets told, what has been modified. The deletion of the row with id=1 is not the last step, so that call gets the opcode of

    OP_NOP

. Note that the group size with it is 2, not 1. That's because the aggregator gets notified only after all the modifications are already done. So the additive part of the computation must never read the group size or do any kind of iteration through the group, because that would often cause an incorrect result: it has no way to tell, what other modifications have been already done to the group. The last

    AO_AFTER_INSERT

gets the opcode of

    OP_INSERT

which tells the computation to send the new result of the aggregation. When the opcode is

    OP_INSERT

, reading the group size and the other group information becomes safe, because by this time all the modifications are guaranteed to be done, and the additive notifications have caught up with all the changes. > OP_INSERT,3,BBB,20,20 AO_BEFORE_MOD OP_DELETE 2 NULL AO_BEFORE_MOD OP_DELETE 1 NULL tWindow.pre OP_DELETE id="3" symbol="AAA" price="20" size="20" tWindow.out OP_DELETE id="3" symbol="AAA" price="20" size="20" tWindow.pre OP_INSERT id="3" symbol="BBB" price="20" size="20" tWindow.out OP_INSERT id="3" symbol="BBB" price="20" size="20" AO_AFTER_DELETE OP_INSERT 1 id="3" symbol="AAA" price="20" size="20" AO_AFTER_INSERT OP_INSERT 2 id="3" symbol="BBB" price="20" size="20"

This insert is of a dirty kind, the one that replaces the row using the replacement policy of the hashed primary index, without deleting its old state first. It also moves the row from one aggregation group to another. So the table logic calls

    AO_BEFORE_MOD

for each of the modified groups, then modifies the contents of the groups, then tells both groups about the modifications. In this case both calls with

    AO_AFTER_*

have the opcode of

    OP_INSERT

because each of them is the last and only change to a separate aggregation group. > OP_DELETE,5 AO_BEFORE_MOD OP_DELETE 1 NULL tWindow.pre OP_DELETE id="5" symbol="AAA" price="30" size="30" tWindow.out OP_DELETE id="5" symbol="AAA" price="30" size="30" AO_AFTER_DELETE OP_INSERT 0 id="5" symbol="AAA" price="30" size="30" AO_COLLAPSE OP_NOP 0 NULL

This operation removes the last row in a group. It starts as usual with deleting the old state. The next

    AO_AFTER_DELETE

with

    OP_INSERT

is intended for the Coral8-style aggregators that produce only the rows with the INSERT opcodes, never DELETEs, to let them insert the NULL (or zero) values in all the non-key fields. For the normal aggregators the work is all done after

    OP_DELETE

. That's why all the shown examples were checking for

    $context->groupSize() == 0

and returning if so. The group size will be zero in absolutely no other case than after the deletion of the last row. Finally

    AO_COLLAPSE

allows to clean up the aggregator's group state if it needs any cleaning. It has the opcode

    OP_NOP

because no rows need to be sent.

To recap, the high-level order of the table operation processing is:

1. Execute the replacement policies on all the indexes to find all the rows that need to be deleted first.
2. If any of the index policies forbid the modification, return 0.
3. Call all the aggregators with

       AO_BEFORE_MOD

   on all the affected rows.
4. Send these aggregator results.

5. For each affected row:

   1. Call the "pre" label (if it has any labels chained to it).
   2. Modify the row in the table.
   3. Call the "out" label.
6. Call all the aggregators with

       AO_AFTER_*

   , on all the affected rows.
7. Send these aggregator results.

aggregation arguments
table execution order

## Using multiple indexes

I've mentioned before that the floating numbers are tricky to handle. Even without additive aggregation the result depends on the rounding. Which in turn depends on the order in which the operations are done. Mostly. The modern Perl (at least as of 5.26.1) appears to be smart enough to preserve the extra precision in the computations, so the example below would work correctly anyway and not show the problem. But the older versions of Perl, and the &Cpp; code would show this issue, so I think it's still worth looking at. So let's look at a version of the aggregation code that might display this issue.

    sub computeAverage10 # (table, context, aggop, opcode, rh, state, args...)
    {
    	my ($table, $context, $aggop, $opcode, $rh, $state, @args) = @_;

    	# don't send the NULL record after the group becomes empty
    	return if ($context->groupSize()==0
    		|| $opcode != &Triceps::OP_INSERT);

    	my $sum = 0;
    	my $count = 0;
    	for (my $rhi = $context->begin(); !$rhi->isNull();
    			$rhi = $context->next($rhi)) {
    		$count++;
    		$sum += $rhi->getRow()->get("price");
    	}
    	my $rLast = $context->last()->getRow();
    	my $avg = $sum/$count;

    	my $res = $context->resultType()->makeRowHash(
    		symbol => $rLast->get("symbol"),
    		id => $rLast->get("id"),
    		price => $avg
    	);
    	$context->send($opcode, $res);
    }

    my $ttWindow = Triceps::TableType->new($rtTrade)
    	->addSubIndex("byId",
    		Triceps::IndexType->newHashed(key => [ "id" ])
    	)
    	->addSubIndex("bySymbol",
    		Triceps::IndexType->newHashed(key => [ "symbol" ])
    		->addSubIndex("last4",
    			Triceps::IndexType->newFifo(limit => 4)
    			->setAggregator(Triceps::AggregatorType->new(
    				$rtAvgPrice, "aggrAvgPrice", undef, \&computeAverage10)
    			)
    		)
    	)
    ;
    $ttWindow->initialize();
    my $tWindow = $uTrades->makeTable($ttWindow, "tWindow");

    # label to print the result of aggregation
    my $lbAverage = $uTrades->makeLabel($rtAvgPrice, "lbAverage",
    	undef, sub { # (label, rowop)
    		&sendf("%.17g\n", $_[1]->getRow()->get("price"));
    	});
    $tWindow->getAggregatorLabel("aggrAvgPrice")->chain($lbAverage);

The differences from the previously shown basic aggregation are:

- the FIFO limit has been increased to 4;
- the only result value printed by the

      $lbAverage

  handler is the price, and it's printed with a higher precision to make the difference visible;
- the aggregator computation only does the inserts, to reduce the clutter in the results and highlight the issue.

And here is an example of how the order of computation matters:

> OP_INSERT,1,AAA,1,10 1 > OP_INSERT,2,AAA,1,10 1 > OP_INSERT,3,AAA,1,10 1 > OP_INSERT,4,AAA,1e16,10 2500000000000001 > OP_INSERT,5,BBB,1e16,10 10000000000000000 > OP_INSERT,6,BBB,1,10 5000000000000000 > OP_INSERT,7,BBB,1,10 3333333333333333.5 > OP_INSERT,8,BBB,1,10 2500000000000000

Of course, the real prices won't vary so wildly. But the other values could. This example is specially stacked to demonstrate the point. The final results for AAA and BBB should be the same but aren't. Why? The precision of the 64-bit floating-point numbers is such that adding 1 to 1e16 makes this 1 fall beyond the precision, and the result is still 1e16. On the other hand, adding 3 to 1e16 makes at least a part of it stick. 1 still falls off but the other 2 of 3 sticks on. Next look at the data sets: if you add 1e16+1+1+1, that's adding 1e16+1 repeated three times, and the result is still the same unchanged 1e16. But if you add 1+1+1+1e16, that's adding 3+1e16, and now the result is different and more correct. When the averages get computed from these different values by dividing the sums by 4, the results are also different.

Overall the rule of thumb for adding the floating point numbers is this: add them up in the order from the smallest to the largest. (What if the numbers can be negative too? I don't know, that goes beyond my knowledge of floating point calculations. My guess is that you still arrange them in the ascending order, only by the absolute value.) So let's do it in the aggregator.

    our $idxByPrice;

    # aggregation handler: sum in proper order
    sub computeAverage11 # (table, context, aggop, opcode, rh, state, args...)
    {
    	my ($table, $context, $aggop, $opcode, $rh, $state, @args) = @_;
    	our $idxByPrice;

    	# don't send the NULL record after the group becomes empty
    	return if ($context->groupSize()==0
    		|| $opcode != &Triceps::OP_INSERT);

    	my $sum = 0;
    	my $count = 0;
    	my $end = $context->endIdx($idxByPrice);
    	for (my $rhi = $context->beginIdx($idxByPrice); !$rhi->same($end);
    			$rhi = $rhi->nextIdx($idxByPrice)) {
    		$count++;
    		$sum += $rhi->getRow()->get("price");
    	}
    	my $rLast = $context->last()->getRow();
    	my $avg = $sum/$count;

    	my $res = $context->resultType()->makeRowHash(
    		symbol => $rLast->get("symbol"),
    		id => $rLast->get("id"),
    		price => $avg
    	);
    	$context->send($opcode, $res);
    }

    my $ttWindow = Triceps::TableType->new($rtTrade)
    	->addSubIndex("byId",
    		Triceps::IndexType->newHashed(key => [ "id" ])
    	)
    	->addSubIndex("bySymbol",
    		Triceps::IndexType->newHashed(key => [ "symbol" ])
    		->addSubIndex("last4",
    			Triceps::IndexType->newFifo(limit => 4)
    			->setAggregator(Triceps::AggregatorType->new(
    				$rtAvgPrice, "aggrAvgPrice", undef, \&computeAverage11)
    			)
    		)
    		->addSubIndex("byPrice",
    			Triceps::IndexType->newOrdered(key => [ "price" ])
    			->addSubIndex("multi", Triceps::IndexType->newFifo())
    		)
    	)
    ;
    $ttWindow->initialize();
    my $tWindow = $uTrades->makeTable($ttWindow, "tWindow");

    $idxByPrice = $ttWindow->findIndexPath("bySymbol", "byPrice");

Here another index type is added, ordered by price. It has to be non-leaf, with a FIFO index type nested in it, to allow for multiple rows having the same price in them. That would work out more efficiently if the Ordered index could have a multimap mode, but that is not supported yet.

When the compute function does its iteration, it now goes by that index. The aggregator can't be simply moved to that new index type, because it still needs to get the last trade id in the order in which the rows are inserted into the group. Instead it has to work with two index types: the one on which the aggregator is defined, and the additional one. The calls for iteration on an additional index are different.

    $context->beginIdx()

is similar to

    $context->begin()

but the end condition and the next step are done differently. When

    $rhi->nextIdx()

reaches the end of the group, it returns not a NULL row handle but a handle value that has to be found in advance with

    $context->endIdx()

. Perhaps the consistency in this department can be improved in the future.

And finally, the reference to that additional index type has to make it somehow into the compute function. It can't be given as an argument because it's not known yet at the time when the aggregator is constructed (and no, reordering the index types won't help because the index types are copied when connected to their parents, and we need the exact index type that ends up in the assembled table type). So a global variable

    $idxByPrice

is used. The index type reference is found and placed there, and later when the compute function runs, it takes the reference from the global variable.

The printout from this version on the same input is:

> OP_INSERT,1,AAA,1,10 1 > OP_INSERT,2,AAA,1,10 1 > OP_INSERT,3,AAA,1,10 1 > OP_INSERT,4,AAA,1e16,10 2500000000000001 > OP_INSERT,5,BBB,1e16,10 10000000000000000 > OP_INSERT,6,BBB,1,10 5000000000000000 > OP_INSERT,7,BBB,1,10 3333333333333334 > OP_INSERT,8,BBB,1,10 2500000000000001

Now no matter what the order of the row arrival, the prices get added up in the same order from the smallest to the largest and produce the same correct (inasmuch the floating point precision allows) result.

Which index type is used to put the aggregator on, doesn't matter a whole lot. The computation can be turned around, with the ordered index used as the main one, and the last value from the FIFO index obtained with

    $context->lastIdx()

:

    our $idxByOrder;

    # aggregation handler: sum in proper order
    sub computeAverage12 # (table, context, aggop, opcode, rh, state, args...)
    {
    	my ($table, $context, $aggop, $opcode, $rh, $state, @args) = @_;
    	our $idxByOrder;

    	# don't send the NULL record after the group becomes empty
    	return if ($context->groupSize()==0
    		|| $opcode != &Triceps::OP_INSERT);

    	my $sum = 0;
    	my $count = 0;
    	for (my $rhi = $context->begin(); !$rhi->isNull();
    			$rhi = $context->next($rhi)) {
    		$count++;
    		$sum += $rhi->getRow()->get("price");
    	}
    	my $rLast = $context->lastIdx($idxByOrder)->getRow();
    	my $avg = $sum/$count;

    	my $res = $context->resultType()->makeRowHash(
    		symbol => $rLast->get("symbol"),
    		id => $rLast->get("id"),
    		price => $avg
    	);
    	$context->send($opcode, $res);
    }

    my $ttWindow = Triceps::TableType->new($rtTrade)
    	->addSubIndex("byId",
    		Triceps::IndexType->newHashed(key => [ "id" ])
    	)
    	->addSubIndex("bySymbol",
    		Triceps::IndexType->newHashed(key => [ "symbol" ])
    		->addSubIndex("last4",
    			Triceps::IndexType->newFifo(limit => 4)
    		)
    		->addSubIndex("byPrice",
    			Triceps::IndexType->newOrdered(key => [ "price" ])
    			->addSubIndex("multi", Triceps::IndexType->newFifo())
    			->setAggregator(Triceps::AggregatorType->new(
    				$rtAvgPrice, "aggrAvgPrice", undef, \&computeAverage12)
    			)
    		)
    	)
    ;
    $ttWindow->initialize();
    my $tWindow = $uTrades->makeTable($ttWindow, "tWindow");

    $idxByOrder = $ttWindow->findIndexPath("bySymbol", "last4");

The last important note: when aggregating with multiple indexes, always use the sibling index types forming the same group or their nested sub-indexes (since the actual order is defined by the first leaf sub-index anyway). But don't use the random unrelated index types. If you do, the context would return some unexpected values for those, and you may end up with endless loops.

aggregation multiple indexes
aggregation floating point error
index multimap
index ordered
index FIFO

## SimpleAggregator

Even though the writing the aggregation computation functions manually gives the flexibility, it's too much work for the simple cases. The SimpleAggregator template takes care of most of that work and allows you to specify the aggregation in a way similar to SQL. It has been already shown on the VWAP example, ans here is the trade aggregation example from &xrsp; rewritten with SimpleAggregator:

    my $ttWindow = Triceps::TableType->new($rtTrade)
    	->addSubIndex("byId",
    		Triceps::IndexType->newHashed(key => [ "id" ])
    	)
    	->addSubIndex("bySymbol",
    		Triceps::IndexType->newHashed(key => [ "symbol" ])
    		->addSubIndex("last2",
    			Triceps::IndexType->newFifo(limit => 2)
    		)
    	)
    ;

    # the aggregation result
    my $rtAvgPrice;
    my $compText; # for debugging

    Triceps::SimpleAggregator::make(
    	tabType => $ttWindow,
    	name => "aggrAvgPrice",
    	idxPath => [ "bySymbol", "last2" ],
    	result => [
    		symbol => "string", "last", sub {$_[0]->get("symbol");},
    		id => "int32", "last", sub {$_[0]->get("id");},
    		price => "float64", "avg", sub {$_[0]->get("price");},
    	],
    	saveRowTypeTo => \$rtAvgPrice,
    	saveComputeTo => \$compText,
    );

    $ttWindow->initialize();
    my $tWindow = $uTrades->makeTable($ttWindow, "tWindow");

    # label to print the result of aggregation
    my $lbAverage = makePrintLabel("lbAverage",
    	$tWindow->getAggregatorLabel("aggrAvgPrice"));

The main loop and the printing is the same as before. The result produced is also exactly the same as before.

But the aggregator is created with

    Triceps::SimpleAggregator::make()

. Its arguments are in the option format: the option name-value pairs, in any order.

    $tabType = Triceps::SimpleAggregator::make($optName => $optValue, ...);

It returns back the table type that it received as an option. But most of the time there is not a whole lot of use to that return value, and it gets simply ignored. Most of the options are actually mandatory. The aggregator type is connected to the table type with the options:

tabType
  Table type to put the aggregator on. It must be un-initialized yet.
idxPath
  A reference to an array of index names, forming the path to the index where the aggregator type will be set.
name
  The aggregator type name.

The result row type and computation is defined with the option result: each group of four values in that array defines one result field:

- The field name.
- The field type.
- The aggregation function name used to compute the field. There is no way to combine multiple aggregation functions or even an aggregation function and any arithmetics in a field computation. The workaround is to compute each function in a separate field, and then send the result rows to a computational label that would arithmetically combine these fields into one.
- A closure that extracts the aggregation function argument from the row (well, it can be any function reference, doesn't have to be an anonymous closure). That closure gets the row as the argument

      $_[0]

  and returns the extracted value to run the aggregation on.

The field name is by convention separated from its definition fields by

    =>

. Remember, it's just a convention, for Perl a

    =>

is just as good as a comma.

    SimpleAggregator::make()

automatically generates the result row type and aggregation function, creates an aggregator type from them, and sets it on the index type. The information about the aggregation result can be found by traversing through the index type tree, or by constructing a table and getting the row type from the aggregator result label. However it's often easier to save it during construction, and the option (this time an optional one!) saveRowTypeTo allows to do this. Give it a reference to a variable, and the row type will be placed into that variable.

Most of the time the things would just work. However if they don't and something dies in the aggregator, you will need the source code of the compute function to make sense of these errors. The option

    saveComputeTo

gives a variable to save that source code for future perusal and other entertainment. Here is the compute function that gets produced by the example above (it gets implicitly wrapped in a

    sub { ... }

, like any other source code argument):

      use strict;
      my ($table, $context, $aggop, $opcode, $rh, $state, @args) = @_;
      return if ($context->groupSize()==0 || $opcode == &Triceps::OP_NOP);
      my $v2_count = 0;
      my $v2_sum = 0;
      my $npos = 0;
      for (my $rhi = $context->begin(); !$rhi->isNull(); $rhi = $context->next($rhi)) {
        my $row = $rhi->getRow();
        # field price=avg
        my $a2 = $args[2]($row);
        { if (defined $a2) { $v2_sum += $a2; $v2_count++; }; }
        $npos++;
      }
      my $rowLast = $context->last()->getRow();
      my $l0 = $args[0]($rowLast);
      my $l1 = $args[1]($rowLast);
      $context->makeArraySend($opcode,
        ($l0), # symbol
        ($l1), # id
        (($v2_count == 0? undef : $v2_sum / $v2_count)), # price
      );

At the moment the compute function is quite straightforward and just does the aggregation from scratch every time. It doesn't support the additive aggregation nor the DELETE optimization. It's only smart enough to skip the iteration if all the result consists of only aggregation functions

    first

,

    last

and

    count_star

. It receives the closures for the argument extraction as arguments in

    @args

, SimpleAggregator arranges these arguments when it creates the aggregator.

The aggregation functions available at the moment are:

    first

  Value from the first row in the group.

    last

  Value from the last row in the group.

    count_star

  Number of rows in the group, like SQL

      COUNT(*)

  . Since there is no argument for this function, use

      undef

  instead of the argument closure.

    sum

  Sum of the values.

    max

  The maximal value.

    min

  The minimal value.

    avg

  The average of all the non-NULL values.

    avg_perl

  The average of all values, with NULL values treated in Perl fashion as zeroes. So, technically when the example above used

      avg

  , it works the same as the previous versions only for the non-NULL fields. To be really the same, it should have used

      avg_perl

  .

    nth_simple

  The Nth value from the start of the group. This is a tricky function because it needs two arguments: the value of N and the field selector. Multiple direct arguments will be supported in the future but right now it works through a workaround: the argument closure must return not just the extracted field but a reference to array with two values, the N and the field. For example,

      sub { [1, $_[0]->get("id")];}

  . The N is counted starting from 0, so the value of 1 will return the second record. This function works in a fairly simple-minded and inefficient way at the moment.

As usual in Triceps and Perl, the case of the aggregation function name matters. The names have to be used in lowercase as shown. There will be more functions to come, and you can even already add your own, as has been shown in &xrsp;.

The user-defined aggregation functions are defined with the option functions. Let's take another look at the code from the VWAP example:

    # VWAP function definition
    my $myAggFunctions = {
    	myvwap => {
    		vars => { sum => 0, count => 0, size => 0, price => 0 },
    		step => '($%size, $%price) = @$%argiter; '
    			. 'if (defined $%size && defined $%price) '
    				. '{$%count += $%size; $%sum += $%size * $%price;}',
    		result => '($%count == 0? undef : $%sum / $%count)',
    	},
    };

    ...

    Triceps::SimpleAggregator::make(
    	functions => $myAggFunctions,
    );

The definition of the functions is a reference to a hash, keyed by the function name. Each function definition in order is a hash of options, keyed by the option name. When the SimpleAggregator builds the common computation function, it assembles the code by tying together the code fragments from these options: Whenever the group changes, the aggregator will reset the function state variables to the default values and iterate through the new contents of the group. It will perform the step computation for each row and collect the data in the intermediate variables. After the iteration it will perform the result computation of all the functions and produce the final value.

The expected format of the values of these options varies with the option. The option result is mandatory, the rest can be skipped if not needed. The supported options are:

    argcount

  Integer. Defines the number of arguments of the function, which may currently be 0 or 1, with 1 being the default. If this option is 0, SimpleAggregator will check that the argument closure is

      undef

  . If the aggregation function needs more arguments than one, they have to be packed into an array or hash, and then its reference used as a single argument. The standard function

      nth_simple

  and the VWAP function provide the examples of how to do this.

    vars

  Reference to a hash. Defines the variables used to keep the context of this function during the iteration (the hash keys are the variable names) and their initial values (specified as the values in the hash).

    step

  String. The code fragment to compute a single step of iteration. It can refer to the variables defined in

      vars

  and to a few of the pre-defined values using the syntax

      $%name

  (which has been chosen because it's illegal in the normal Perl variable syntax). When SimpleAggregator generates the code, it creates the actual scope variables for everything defined in

      vars

  , then substitutes them for the

      $%

  syntax in the string and inserts the result into its group iteration code.

  If this option is not defined, SimpleAggregator assumes that this function doesn't need it. If no functions in the aggregation define the

      step

  , the iteration does not get included into the generated code altogether.

  The defined special values are:

  -     $%argiter

    - The function's argument extracted from the current row.

  -     $%niter

    - The number of the current row in the group, starting from 0.

  -     $%groupsize

    - The size of the group (

        $context->groupSize()

    ).

    result

  String. The code fragment to compute the result of the function. This option is mandatory. Works in the same way as

      step

  , only gets executed once per call of the computation function, and the defined special values are different:

  -     $%argfirst

    - The function's argument extracted from the first row.

  -     $%arglast

    - The function's argument extracted from the last row.

  -     $%groupsize

    - The size of the group (

        $context->groupSize()

    ).

I can think of many ways the SimpleAggregator can be improved, but for now they have been pushed into the future to keep it simple.

SQL
SimpleAggregator

## The guts of SimpleAggregator

The implementation of the SimpleAggregator has turned out to be surprisingly small. Not quite tiny but still small. I've liked it so much that I've even saved the original small version in the file

    xSimpleAggregator.t

. As more features will be added, the official version of the SimpleAggregator will grow (and already did) but that example file will stay small and simple.

It's a nice example of yet another kind of template that I want to present. I'm going to go through it, interlacing the code with the commentary.

    package MySimpleAggregator;
    use Carp;

    use strict;

    our $FUNCTIONS = {
    	first => {
    		result => '$%argfirst',
    	},
    	last => {
    		result => '$%arglast',
    	},
    	count_star => {
    		argcount => 0,
    		result => '$%groupsize',
    	},
    	count => {
    		vars => { count => 0 },
    		step => '$%count++ if (defined $%argiter);',
    		result => '$%count',
    	},
    	sum => {
    		vars => { sum => 0 },
    		step => '$%sum += $%argiter;',
    		result => '$%sum',
    	},
    	max => {
    		vars => { max => 'undef' },
    		step => '$%max = $%argiter if (!defined $%max || $%argiter > $%max);',
    		result => '$%max',
    	},
    	min => {
    		vars => { min => 'undef' },
    		step => '$%min = $%argiter if (!defined $%min || $%argiter < $%min);',
    		result => '$%min',
    	},
    	avg => {
    		vars => { sum => 0, count => 0 },
    		step => 'if (defined $%argiter) { $%sum += $%argiter; $%count++; }',
    		result => '($%count == 0? undef : $%sum / $%count)',
    	},
    	avg_perl => { # Perl-like treat the NULLs as 0s
    		vars => { sum => 0 },
    		step => '$%sum += $%argiter;',
    		result => '$%sum / $%groupsize',
    	},
    	nth_simple => { # inefficient, need proper multi-args for better efficiency
    		vars => { n => 'undef', tmp => 'undef', val => 'undef' },
    		step => '($%n, $%tmp) = @$%argiter; if ($%n == $%niter) { $%val = $%tmp; }',
    		result => '$%val',
    	},
    };

The package name of this saved simple version is MySimpleAggregator, to avoid confusion with the official SimpleAggregator class. First goes the definition of the aggregation functions. They are defined in exactly the same way as the vwap function has been shown before. They are fairly straightforward. You can use them as the starting point for adding your own.

    sub make # (optName => optValue, ...)
    {
    	my $opts = {}; # the parsed options
    	my $myname = "MySimpleAggregator::make";

    	&Triceps::Opt::parse("MySimpleAggregator", $opts, {
    			tabType => [ undef, sub { &Triceps::Opt::ck_mandatory(@_); &Triceps::Opt::ck_ref(@_, "Triceps::TableType") } ],
    			name => [ undef, \&Triceps::Opt::ck_mandatory ],
    			idxPath => [ undef, sub { &Triceps::Opt::ck_mandatory(@_); &Triceps::Opt::ck_ref(@_, "ARRAY", "") } ],
    			result => [ undef, sub { &Triceps::Opt::ck_mandatory(@_); &Triceps::Opt::ck_ref(@_, "ARRAY") } ],
    			saveRowTypeTo => [ undef, sub { &Triceps::Opt::ck_refscalar(@_) } ],
    			saveInitTo => [ undef, sub { &Triceps::Opt::ck_refscalar(@_) } ],
    			saveComputeTo => [ undef, sub { &Triceps::Opt::ck_refscalar(@_) } ],
    		}, @_);

The options get parsed. Since it's not a proper object constructor but a factory, it uses the hash

    $opts

instead of

    $self

to save the processed copy of the options. This early version doesn't have an option for the user-supplied aggregation function definitions.

    	# reset the saved source code
    	${$opts->{saveInitTo}} = undef if (defined($opts->{saveInitTo}));
    	${$opts->{saveComputeTo}} = undef if (defined($opts->{saveComputeTo}));
    	${$opts->{saveRowTypeTo}} = undef if (defined($opts->{saveRowTypeTo}));

The generated source code will not be placed into the

    save*

references until the table type gets initialized, so for the meantime they get filled with

    undef

s.

    	# find the index type, on which to build the aggregator
    	my $idx = $opts->{tabType}->findIndexPath(@{$opts->{idxPath}});
    	confess "$myname: the index type is already initialized, can not add an aggregator on it"
    		if ($idx->isInitialized());

Since the SimpleAggregator uses an existing table with existing index, it doesn't require the aggregation key: it just takes an index that forms the group, and whatever key that leads to this index becomes the aggregation key.

    	# check the result definition and build the result row type and code snippets for the computation
    	my $rtRes;
    	my $needIter = 0; # flag: some of the functions require iteration
    	my $needfirst = 0; # the result needs the first row of the group
    	my $needlast = 0; # the result needs the last row of the group
    	my $codeInit = ''; # code for function initialization
    	my $codeStep = ''; # code for iteration
    	my $codeResult = ''; # code to compute the intermediate values for the result
    	my $codeBuild = ''; # code to build the result row
    	my @compArgs; # the field functions are passed as args to the computation
    	{
    		my $grpstep = 4; # definition grouped by 4 items per result field
    		my @resopt = @{$opts->{result}};
    		my @rtdefRes; # field definition for the result
    		my $id = 0; # numeric id of the field

    		while ($#resopt >= 0) {
    			confess "$myname: the values in the result definition must go in groups of 4"
    				unless ($#resopt >= 3);
    			my $fld = shift @resopt;
    			my $type = shift @resopt;
    			my $func = shift @resopt;
    			my $funcarg = shift @resopt;

    			confess("$myname: the result field name must be a string, got a " . ref($fld) . " ")
    				unless (ref($fld) eq '');
    			confess("$myname: the result field type must be a string, got a " . ref($type) . " for field '$fld'")
    				unless (ref($type) eq '');
    			confess("$myname: the result field function must be a string, got a " . ref($func) . " for field '$fld'")
    				unless (ref($func) eq '');

This starts the loop that goes over the result fields and builds the code to create them. The code will be built in multiple snippets that will eventually be combined to produce the compute function. Since the arguments go in groups of 4, it becomes fairly easy to miss one element somewhere, and then everything gets real confusing. So the code attempts to check the types of the arguments, in hopes of catching these off-by-ones as early as possible. The variable

    $id

will be used to produce the unique prefixes for the function's variables.

    			my $funcDef = $FUNCTIONS->{$func}
    				or confess("$myname: function '" . $func . "' is unknown");

    			my $argCount = $funcDef->{argcount};
    			$argCount = 1 # 1 is the default value
    				unless defined($argCount);
    			confess("$myname: in field '$fld' function '$func' requires an argument computation that must be a Perl sub reference")
    				unless ($argCount == 0 || ref $funcarg eq 'CODE');
    			confess("$myname: in field '$fld' function '$func' requires no argument, use undef as a placeholder")
    				unless ($argCount != 0 || !defined $funcarg);

    			push(@rtdefRes, $fld, $type);

    			push(@compArgs, $funcarg)
    				if (defined $funcarg);

The function definition for a field gets pulled out by name, and the arguments of the field are checked for correctness. The types of the fields get collected for the row definition, and the aggregation argument computation closures (or, technically, functions) get also collected, to pass later as the arguments of the compute function.

    			# add to the code snippets

    			### initialization
    			my $vars = $funcDef->{vars};
    			if (defined $vars) {
    				foreach my $v (keys %$vars) {
    					# the variable names are given a unique prefix;
    					# the initialization values are constants, no substitutions
    					$codeInit .= "  my \$v${id}_${v} = " . $vars->{$v} . ";\n";
    				}
    			} else {
    				$vars = { }; # a dummy
    			}

The initialization fragment gets processed if defined. The unique names for variables are generated from the

    $id

and the variable name in the definition, so that there would be no interference between the result fields. And the initialization snippets are collected in

    $codeInit

. The initialization values are not enquoted because they are expected to be strings suitable for such use. That's why the undefined values in the function defnitions are not

    undef

but

    'undef'

. If you'd want to initialize a variable as a string

    "x"

, you'd use it as

    '"x"'

. For the numbers it doesn't really matter, the numbers just get converted to strings as needed, so the zeroes are simply

    0

s without quoting.

Another possibility would be to have the actual values as-is in the hash and then either put these values into the argument array passed to the computation function or use the closure trick from

    Triceps::Fields::makeTranslation()

described in &xrsp;.

    			### iteration
    			my $step = $funcDef->{step};
    			if (defined $step) {
    				$needIter = 1;
    				$codeStep .= "    # field $fld=$func\n";
    				if (defined $funcarg) {
    					# compute the function argument from the current row
    					$codeStep .= "    my \$a${id} = \$args[" . $#compArgs ."](\$row);\n";
    				}
    				# substitute the variables in $step
    				$step =~ s/\$\%(\w+)/&replaceStep($1, $func, $vars, $id, $argCount)/ge;
    				$codeStep .= "    { $step; }\n";
    			}

Then the iteration fragment gets processed. The logic remembers in

    $needIter

if any of the functions involved needs iteration. Before the iteration snippet gets collected, it has the

    $%

names substitutted, and placed into a block, just in case if it wants to define some local variables. An extra ; is added just in case, it doesn't hurt and helps if it was forgotten in the function definition.

    			### result building
    			my $result = $funcDef->{result};
    			confess "MySimpleAggregator: internal error in definition of aggregation function '$func', missing result computation"
    				unless (defined $result);
    			# substitute the variables in $result
    			if ($result =~ /\$\%argfirst/) {
    				$needfirst = 1;
    				$codeResult .= "  my \$f${id} = \$args[" . $#compArgs ."](\$rowFirst);\n";
    			}
    			if ($result =~ /\$\%arglast/) {
    				$needlast = 1;
    				$codeResult .= "  my \$l${id} = \$args[" . $#compArgs ."](\$rowLast);\n";
    			}
    			$result =~ s/\$\%(\w+)/&replaceResult($1, $func, $vars, $id, $argCount)/ge;
    			$codeBuild .= "    ($result), # $fld\n";

    			$id++;
    		}
    		$rtRes = Triceps::wrapfess
    			"$myname: invalid result row type definition:",
    			sub { Triceps::RowType->new(@rtdefRes); };
    	}
    	${$opts->{saveRowTypeTo}} = $rtRes if (defined($opts->{saveRowTypeTo}));

In the same way the result computation is created, and remembers if any function wanted the fields from the first or last row. And eventually after all the functions have been processed, the result row type is created. If it was asked to save, it gets saved.

    	# build the computation function
    	my $compText = "sub {\n";
    	$compText .= "  use strict;\n";
    	$compText .= "  my (\$table, \$context, \$aggop, \$opcode, \$rh, \$state, \@args) = \@_;\n";
    	$compText .= "  return if (\$context->groupSize()==0 || \$opcode == &Triceps::OP_NOP);\n";
    	$compText .= $codeInit;
    	if ($needIter) {
    		$compText .= "  my \$npos = 0;\n";
    		$compText .= "  for (my \$rhi = \$context->begin(); !\$rhi->isNull(); \$rhi = \$context->next(\$rhi)) {\n";
    		$compText .= "    my \$row = \$rhi->getRow();\n";
    		$compText .= $codeStep;
    		$compText .= "    \$npos++;\n";
    		$compText .= "  }\n";
    	}
    	if ($needfirst) {
    		$compText .= "  my \$rowFirst = \$context->begin()->getRow();\n";
    	}
    	if ($needlast) {
    		$compText .= "  my \$rowLast = \$context->last()->getRow();\n";
    	}
    	$compText .= $codeResult;
    	$compText .= "  \$context->makeArraySend(\$opcode,\n";
    	$compText .= $codeBuild;
    	$compText .= "  );\n";
    	$compText .= "}\n";

    	${$opts->{saveComputeTo}} = $compText if (defined($opts->{saveComputeTo}));

The compute function gets assembled from the collected fragments. The optional parts get included only if some of the functions needed them.

    	# compile the computation function
    	my $compFun = eval $compText
    		or confess "$myname: error in compilation of the aggregation computation:\n  $@function text:\n"
    			. Triceps::Code::numalign($compText, "  ") . "\n";

    	# build and add the aggregator
    	my $agg = Triceps::wrapfess
    		"$myname: internal error: failed to build an aggregator type:",
    		sub { Triceps::AggregatorType->new($rtRes, $opts->{name}, undef, $compFun, @compArgs); };

    	Triceps::wrapfess
    		"$myname: failed to set the aggregator in the index type:",
    		sub { $idx->setAggregator($agg); };

    	return $opts->{tabType};
    }

Then the compute function is compiled. In case if the compilation fails, the error message will include both the compilation error and the text of the auto-generated function. Otherwise there would be no way to know, what exactly went wrong. Well, since no user code is included into the auto-generated function, it should never fail. Except if there is some bad code in the aggregation function definitions. The compiled function and collected closures are then used to create the aggregator, which should also never fail.

The functions that translate the

    $%variable

names are built after the same pattern but have the different built-in variables:

    sub replaceStep # ($varname, $func, $vars, $id, $argCount)
    {
    	my ($varname, $func, $vars, $id, $argCount) = @_;

    	if ($varname eq 'argiter') {
    		confess "MySimpleAggregator: internal error in definition of aggregation function '$func', step computation refers to 'argiter' but the function declares no arguments"
    			unless ($argCount > 0);
    		return "\$a${id}";
    	} elsif ($varname eq 'niter') {
    		return "\$npos";
    	} elsif ($varname eq 'groupsize') {
    		return "\$context->groupSize()";
    	} elsif (exists $vars->{$varname}) {
    		return "\$v${id}_${varname}";
    	} else {
    		confess "MySimpleAggregator: internal error in definition of aggregation function '$func', step computation refers to an unknown variable '$varname'"
    	}
    }

    sub replaceResult # ($varname, $func, $vars, $id, $argCount)
    {
    	my ($varname, $func, $vars, $id, $argCount) = @_;

    	if ($varname eq 'argfirst') {
    		confess "MySimpleAggregator: internal error in definition of aggregation function '$func', result computation refers to '$varname' but the function declares no arguments"
    			unless ($argCount > 0);
    		return "\$f${id}";
    	} elsif ($varname eq 'arglast') {
    		confess "MySimpleAggregator: internal error in definition of aggregation function '$func', result computation refers to '$varname' but the function declares no arguments"
    			unless ($argCount > 0);
    		return "\$l${id}";
    	} elsif ($varname eq 'groupsize') {
    		return "\$context->groupSize()";
    	} elsif (exists $vars->{$varname}) {
    		return "\$v${id}_${varname}";
    	} else {
    		confess "MySimpleAggregator: internal error in definition of aggregation function '$func', result computation refers to an unknown variable '$varname'"
    	}
    }

They check for the references to the undefined variables and confess if any are found. That's it, the whole aggregator generation.

Now let's look back at the printout of a generated computation function that has been shown above.. The aggregation results were:

    	result => [
    		symbol => "string", "last", sub {$_[0]->get("symbol");},
    		id => "int32", "last", sub {$_[0]->get("id");},
    		price => "float64", "avg", sub {$_[0]->get("price");},
    	],

Which produced the function body:

      use strict;
      my ($table, $context, $aggop, $opcode, $rh, $state, @args) = @_;
      return if ($context->groupSize()==0 || $opcode == &Triceps::OP_NOP);
      my $v2_count = 0;
      my $v2_sum = 0;
      my $npos = 0;
      for (my $rhi = $context->begin(); !$rhi->isNull(); $rhi = $context->next($rhi)) {
        my $row = $rhi->getRow();
        # field price=avg
        my $a2 = $args[2]($row);
        { if (defined $a2) { $v2_sum += $a2; $v2_count++; }; }
        $npos++;
      }
      my $rowLast = $context->last()->getRow();
      my $l0 = $args[0]($rowLast);
      my $l1 = $args[1]($rowLast);
      $context->makeArraySend($opcode,
        ($l0), # symbol
        ($l1), # id
        (($v2_count == 0? undef : $v2_sum / $v2_count)), # price
      );

The fields get assigned the ids 0, 1 and 2.

    avg

for the

    price

field is the only function here that requires the iteration, and its variables are defined with the prefix

    $v2_

. In the loop the function argument closure is called from

    $args[2]

, and its result is stored in

    $a2

(again, 2 here is the id of this field). Then a copy of the step computation for

    avg

is copied in a block, with the variables substituted.

    $%argiter

becomes

    $a2

,

    $%sum

becomes

    $v2_sum

,

    $%count

becomes

    $v2_count

. Then the loop ends.

The functions make use of the last row, so

    $rowLast

is computed. The values for the

    $%arglast

fields 0 and 1 are calculated in

    $l0

and

    $l1

. Then the result row is created and sent from an array of substituted result snippets from all the fields. That's how it all works together.
SimpleAggregator
template