# The other templates and solutions

## The dreaded diamond

The diamond is a particular topology of the data flow, when the computation separates based on some condition and then merges again. Like in &xrsp;. It is also known as fork-join (the join here has nothing to do with the SQL join, it just means that the arrows merge to the same block).

The diamond topology.

This topology is a known source of two problems. The first problem is about the execution order. To make things easier to see, let's consider a simple example. Suppose the rows come into the block A with the schema:

    key => string,
    value => int32,

And come out of the blocks B and C into D with schema

    key => string,
    value => int32,
    negative => int32,

With the logic in the blocks being:

    A:
    	if value < 0 then B else C
    B:
    	negative = 1
    C:
    	negative = 0

Yes, this is a very dumb example that can usually be handled by a conditional expression in a single block. But that's to keep it small and simple. A real example would often include some SQL joins, with different joins done on condition.

Suppose A then gets the input, in CSV form:

INSERT,key1,10 DELETE,key1,10 INSERT,key1,20 DELETE,key1,20 INSERT,key1,-1

What arrives at D should be

INSERT,key1,10,0 DELETE,key1,10,0 INSERT,key1,20,0 DELETE,key1,20,0 INSERT,key1,-1,1

And with the first four rows this is not a problem: they follow the same path and are queued sequentially, so the order is preserved. But the last row follows a different path. And the last two rows logically represent a single update and would likely arrive closely together. The last row might happen to overtake the one before it, and D would see the incorrect result:

INSERT,key1,10,0 DELETE,key1,10,0 INSERT,key1,20,0 INSERT,key1,-1,1 DELETE,key1,20,0

If all these input rows arrive closely one after another, the last row might overtake even more of them and produce an even more disturbing result like

INSERT,key1,-1,1 INSERT,key1,10,0 DELETE,key1,10,0 INSERT,key1,20,0 DELETE,key1,20,0

Such misorderings may also happen between the rows with different keys. Those are usually less of a problem, because usually if D keeps a table, the rows with different keys may be updated in any order without losing the meaning. But in case if D keeps a FIFO index (say, for a window based on a row count), and the two keys fall into the same FIFO bucket, their misordering would also affect the logic.

The reasons for this can be subdivided further into two classes:

- asynchronous execution,
- incorrect scheduling in the synchronous execution.

If each block executes asynchronously in its own thread, there is no way to predict, in which order they will actually execute. If some data is sent to B and C at about the same time, it becomes a race between them. One of the paths might also be longer than the other, making one alternative always win the race. This kind of problems is fairly common for the Aleri system that is highly multithreaded. But this is the problem of absolutely any CEP engine if you split the execution by multiple threads or processes.

But the single-threaded execution is not necessarily a cure either. Then the order of execution is up to the scheduler. And if the scheduler gets all these rows close together, and then decides to process all the input of A, then all the input of B, of C and of D, then D will receive the rows in the order:

INSERT,key1,-1,1 INSERT,key1,10,0 DELETE,key1,10,0 INSERT,key1,20,0 DELETE,key1,20,0

Which is typical for, say, Coral8 if all the input rows arrive in a single bundle (see also the &xrsp;).

The multithreaded case in Triceps will be discussed separately in &xrsp;.

When the single-threaded scheduling is concerned, Triceps provides two answers.

First, the conditional logic can often be expressed procedurally:

    if ($a->get("value") < 0) {
    	D($rtD->makeRowHash($a->toHash(), negative => 1));
    } else {
    	D($rtD->makeRowHash($a->toHash(), negative => 0));
    }

The procedural if-else logic can easily handle not only the simple expressions but things like look-ups and modifications in the tables.

Second, if the logic is broken into the separate labels, the label call semantics provides the same ordering as well:

    $lbA = $unit->makeLabel($rtA, "A", undef, sub {
    	my $rop = $_[1];
    	my $op = $rop->getOpcode(); my $a = $rop->getRow();
    	if ($a->get("value") < 0) {
    		$unit->call($lbB->makeRowop($op, $a));
    	} else {
    		$unit->call($lbC->makeRowop($op, $a));
    	}
    });

    $lbB = $unit->makeLabel($rtA, "B", undef, sub {
    	my $rop = $_[1];
    	my $op = $rop->getOpcode(); my $a = $rop->getRow();
    	$unit->makeHashCall($lbD, $op, $a->toHash(), negative => 1);
    });

    $lbC = $unit->makeLabel($rtA, "C", undef, sub {
    	my $rop = $_[1];
    	my $op = $rop->getOpcode(); my $a = $rop->getRow();
    	$unit->makeHashCall($lbD, $op, $a->toHash(), negative => 0);
    });

When the label A calls the label B or C, which calls the label D, A does not get to see its next input row until the whole chain of calls to D and beyond completes. B and C may be replaced with the label chains of arbitrary complexity, including loops, without disturbing the logic.

The second problem with the diamond topology happens when the blocks B and C keep the state, and the input data gets updated by simply re-sending a record with the same key. This kind of updates is typical for the systems that do not have the concept of opcodes.

Consider a CCL example (approximate, since I can't test it) that gets the reports about borrowing and loaning securities, using the sign of the quantity to differentiate between borrows (-) and loans (+). It then sums up the borrows and loans separately:

    create schema s_A (
    	id integer,
    	symbol string,
    	quantity long
    );
    create input stream i_A schema s_A;

    create schema s_D (
    	symbol string,
    	borrowed boolean, // flag: loaned or borrowed
    	quantity long
    );
    // aggregated data
    create public window w_D schema s_D
    keep last per symbol, borrowed;

    // collection of borrows
    create public window w_B schema s_A keep last per id;
    // collection of loans
    create public window w_C schema s_A keep last per id;

    insert when quantity < 0
    	then w_B
    	else w_C
    select * from i_A;

    // borrows aggregation
    insert into w_D
    select
    	symbol,
    	true,
    	sum(quantity)
    group by symbol
    from w_B;

    // loans aggregation
    insert into w_D
    select
    	symbol,
    	false,
    	sum(quantity)
    group by symbol
    from w_C;

It works OK until a row with the same id gets updated to a different sign of quantity:

    1,AAA,100
    ....
    1,AAA,-100

If the quantity kept the same sign, the new row would simply replace the old one in w_B or w_C, and the aggregation result would be right again. But when the sign changes, the new row goes into a different direction than the previous one. Now it ends up with both w_B and w_C having rows with the same id: one old and one new!

In this case really the problem is at the fork part of the diamond, the merging part of it is just along for the ride, carrying the incorrect results.

This problem does not happen in the systems that have both inserts and deletes. Then the data sequence becomes

    INSERT,1,AAA,100
    ....
    DELETE,1,AAA,100
    INSERT,1,AAA,-100

The DELETE goes along the same branch as the first insert and undoes its effect, then the second INSERT goes into the other branch.

Since Triceps has both INSERT and DELETE opcodes, it's immune to this problem, as long as the input data has the correct DELETEs in it.

If you wonder, the CCL example can be fixed too but in a more round-about way, by adding a couple of statements before the insert-when statement:

    on w_A
    delete from w_B
    	where w_A.id = w_B.id;

    on w_A
    delete from w_C
    	where w_A.id = w_C.id;

This generates the matching DELETEs. Of course, if you want, you can use this way with Triceps too.

diamond
fork-join
execution order
Aleri
bundling
scheduling
CCL

## Collapsed updates

First, a note: the collapse described here has nothing to do with the collapsing of the aggregation groups. It's just the same word reused for a different purpose.

Sometimes the exact sequence of how a row at a particular key was updated does not matter, the only interesting part is the end result. Like the

    OUTPUT EVERY

statement in CCL or the pulsed subscription in Aleri. It doesn't have to be time-driven either: if the data comes in as batches, it makes sense to collapse the modifications from the whole batch into one, and send it at the end of the batch.

To do this in Triceps, I've made a template. Here is an example of its use with interspersed commentary:

    our $rtData = Triceps::RowType->new(
    	# mostly copied from the traffic aggregation example
    	local_ip => "string",
    	remote_ip => "string",
    	bytes => "int64",
    );

The meaning of the rows is not particularly important for this example. It just uses a pair of the IP addresses as the collapse key. The collapse absolutely needs a primary key, since it has to track and collapse multiple updates to the same row.

    my $unit = Triceps::Unit->new("unit");

    my $collapse = Triceps::Collapse->new(
    	unit => $unit,
    	name => "collapse",
    	data => [
    		name => "idata",
    		rowType => $rtData,
    		key => [ "local_ip", "remote_ip" ],
    	],
    );

Most of the options are self-explanatory. The dataset is defined with nested options to make the API extensible, to allow multiple datasets to be defined in the future. But at the moment only one is allowed. A dataset collapses the data at one label: an input label and an output label get defined for it, just as for the table. The data arrives at the input label, gets collapsed by the primary key, and then stays in the Collapse until the flush. When the Collapse gets flushed, the data is sent out of its output label. After the flush, the Collapse has no data in it, and starts collecting the updates again from scratch. The labels gets named by connecting the names of the Collapse element, of the dataset, and in or out. For this Collapse, the label names will be collapse.idata.in and collapse.idata.out.

Note that the dataset options are specified in a referenced array, not a hash! If you try to use a hash, it will fail. When specifying the dataset options, put the name first. It's used in the error messages about any issues in the dataset, and the code really expects the name to go first.

Like with the other shown templates, if something goes wrong, Collapse will confess.

    my $lbPrint = makePrintLabel("print", $collapse->getOutputLabel("idata"));

The print label gets connected to the Collapse's output label. The method to get the collapse's output label is very much like table's. Only it gets the dataset name as an argument.

    sub mainloop($$$) # ($unit, $datalabel, $collapse)
    {
    	my $unit = shift;
    	my $datalabel = shift;
    	my $collapse = shift;
    	while(&readLine) {
    		chomp;
    		my @data = split(/,/); # starts with a command, then string opcode
    		my $type = shift @data;
    		if ($type eq "data") {
    			my $rowop = $datalabel->makeRowopArray(@data);
    			$unit->call($rowop);
    			$unit->drainFrame(); # just in case, for completeness
    		} elsif ($type eq "flush") {
    			$collapse->flush();
    		}
    	}
    }

    &mainloop($unit, $collapse->getInputLabel($collapse->getDatasets()), $collapse);

There will be a second example, so I've placed the main loop into a function. It works in the same way as in the examples before: extracts the data from the CSV format and sends it to a label. The first column contains the command: data sends the data, and flush performs the flush from the Collapse. The flush marks the end of the batch. Here is an example of a run, with the input lines shown as usual in bold:

> data,OP_INSERT,1.2.3.4,5.6.7.8,100 > data,OP_INSERT,1.2.3.4,6.7.8.9,1000 > data,OP_DELETE,1.2.3.4,6.7.8.9,1000 > flush collapse.idata.out OP_INSERT local_ip="1.2.3.4" remote_ip="5.6.7.8" bytes="100"

The row for (1.2.3.4, 5.6.7.8) gets plainly inserted, and goes through on the flush. The row for (1.2.3.4, 6.7.8.9) gets first inserted and then deleted, so by the flush time it becomes a no-operation.

> data,OP_DELETE,1.2.3.4,5.6.7.8,100 > data,OP_INSERT,1.2.3.4,5.6.7.8,200 > data,OP_INSERT,1.2.3.4,6.7.8.9,2000 > flush collapse.idata.out OP_DELETE local_ip="1.2.3.4" remote_ip="5.6.7.8" bytes="100" collapse.idata.out OP_INSERT local_ip="1.2.3.4" remote_ip="5.6.7.8" bytes="200" collapse.idata.out OP_INSERT local_ip="1.2.3.4" remote_ip="6.7.8.9" bytes="2000"

The original row for (1.2.3.4, 5.6.7.8) gets modified, and the modification goes through. The new row for (1.2.3.4, 6.7.8.9) gets inserted now, and also goes through.

> data,OP_DELETE,1.2.3.4,6.7.8.9,2000 > data,OP_INSERT,1.2.3.4,6.7.8.9,3000 > data,OP_DELETE,1.2.3.4,6.7.8.9,3000 > data,OP_INSERT,1.2.3.4,6.7.8.9,4000 > data,OP_DELETE,1.2.3.4,6.7.8.9,4000 > flush collapse.idata.out OP_DELETE local_ip="1.2.3.4" remote_ip="6.7.8.9" bytes="2000"

The row for (1.2.3.4, 6.7.8.9) now gets modified twice, and after that deleted. After collapse it becomes the deletion of the original row, the one that was inserted before the previous flush.

The Collapse also allows to specify the row type and the input connection for a dataset in a different way:

    my $lbInput = $unit->makeDummyLabel($rtData, "lbInput");

    my $collapse = Triceps::Collapse->new(
    	name => "collapse",
    	data => [
    		name => "idata",
    		fromLabel => $lbInput,
    		key => [ "local_ip", "remote_ip" ],
    	],
    );

    &mainloop($unit, $lbInput, $collapse);

Normally

    $lbInput

would be not a dummy label but the output label of some element. The dataset option fromLabel tells that the dataset input will be coming from that label. So the Collapse can automatically both copy its row type for the dataset, and also chain the dataset's input label to that label. And also allowing to skip the option unit at the main level. It's a pure convenience, allowing to skip the manual steps. In the future a Collapse dataset should probably take a whole list of source labels and chain itself to all of them, but for now only one.

This example produces exactly the same output as the previous one, so there is no use in copying it again.

Another item that hasn't been shown yet, you can get the list of dataset names (well, currently only one name):

    @names = $collapse->getDatasets();

The Collapse implementation is reasonably small, and is another worthy example to show. It's a common template, with no code generation whatsoever, just a combination of ready components. As with SimpleAggregator, the current Collapse is quite simple and will grow more features over time, so I've copied the original simple version into

    t/xCollapse.t

to stay there unchanged.

The most notable thing about Collapse is that it took just about an hour to write the first version of it and another three or so hours to test it. Which is a lot less than the similar code in the Aleri or Coral8 code base took. The reason for this is that Triceps provides the fairly flexible base data structures that can be combined easily directly in a scripting language. There is no need to re-do a lot from scratch every time, just take something and add a little bit on top.

So here it is, with the interspersed commentary.

    sub new # ($class, $optName => $optValue, ...)
    {
    	my $class = shift;
    	my $self = {};

    	&Triceps::Opt::parse($class, $self, {
    		unit => [ undef, sub { &Triceps::Opt::ck_ref(@_, "Triceps::Unit") } ],
    		name => [ undef, \&Triceps::Opt::ck_mandatory ],
    		data => [ undef, sub { &Triceps::Opt::ck_mandatory(@_); &Triceps::Opt::ck_ref(@_, "ARRAY") } ],
    	}, @_);

    	# Keeps the names of the datasets in the order they have been defined
    	# (since the hash loses the order).
    	$self->{dsetnames} = [];

    	# parse the data element
    	my %data_unparsed = @{$self->{data}};
    	my $dataset = {};
    	&Triceps::Opt::parse("$class data set (" . ($data_unparsed{name} or 'UNKNOWN') . ")", $dataset, {
    		name => [ undef, \&Triceps::Opt::ck_mandatory ],
    		key => [ undef, sub { &Triceps::Opt::ck_mandatory(@_); &Triceps::Opt::ck_ref(@_, "ARRAY", "") } ],
    		rowType => [ undef, sub { &Triceps::Opt::ck_ref(@_, "Triceps::RowType"); } ],
    		fromLabel => [ undef, sub { &Triceps::Opt::ck_ref(@_, "Triceps::Label"); } ],
    	}, @{$self->{data}});

The options parsing goes as usual. The option data is parsed again for the options inside it, and those are placed into the hash

    %$dataset

.

    	# save the dataset for the future
    	push @{$self->{dsetnames}}, $dataset->{name};
    	$self->{datasets}{$dataset->{name}} = $dataset;
    	# check the options
    	&Triceps::Opt::handleUnitTypeLabel("Triceps::Collapse data set (". $dataset->{name} . ")",
    		"unit at the main level", \$self->{unit},
    		"rowType", \$dataset->{rowType},
    		"fromLabel", \$dataset->{fromLabel});
    	my $lbFrom = $dataset->{fromLabel};

If fromLabel is used, the row type and possibly unit are found from it by

    Triceps::Opt::handleUnitTypeLabel()

. Or if the unit was specified explicitly, it gets checked for consistency with the label's unit. See &xrsp; for more detail. The early version of Collapse in

    t/xCollapse.t

actually pre-dates

    Triceps::Opt::handleUnitTypeLabel()

, and there the similar functionality is done manually.

    	# create the tables
    	$dataset->{tt} = Triceps::TableType->new($dataset->{rowType})
    		->addSubIndex("primary",
    			Triceps::IndexType->newHashed(key => $dataset->{key})
    		);
    	Triceps::wrapfess
    		"$myname: Collapse table type creation error for dataset '" . $dataset->{name} . "':",
    		sub { $dataset->{tt}->initialize(); };

    	Triceps::wrapfess
    		"$myname: Collapse internal error: insert table creation for dataset '" . $dataset->{name} . "':",
    		sub { $dataset->{tbInsert} = $self->{unit}->makeTable($dataset->{tt}, $self->{name} . "." . $dataset->{name} . ".tbInsert"); };

    	Triceps::wrapfess
    		"$myname: Collapse internal error: delete table creation for dataset '" . $dataset->{name} . "':",
    		sub { $dataset->{tbDelete} = $self->{unit}->makeTable($dataset->{tt}, $self->{name} . "." . $dataset->{name} . ".tbDelete"); };

The state is kept in two tables. The reason for their existence is that after collapsing, the Collapse may send for each key one of:

- a single INSERT rowop, if the row was not there before and became inserted,
- a DELETE rowop if the row was there before and then became deleted,
- a DELETE followed by an INSERT if the row was there but then changed its value,
- or nothing if the row was not there before, and then was inserted and deleted, or if there was no change to the row.

Accordingly, this state is kept in two tables: one contains the DELETE part, another the INSERT part for each key, and either part may be empty (or both, if the row at that key has not been changed). After each flush both tables become empty, and then start collecting the modifications again.

    	# create the labels
    	Triceps::wrapfess
    		"$myname: Collapse internal error: input label creation for dataset '" . $dataset->{name} . "':",
    		sub { $dataset->{lbIn} = $self->{unit}->makeLabel($dataset->{rowType}, $self->{name} . "." . $dataset->{name} . ".in",
    			undef, \&_handleInput, $self, $dataset); };

    	Triceps::wrapfess
    		"$myname: Collapse internal error: output label creation for dataset '" . $dataset->{name} . "':",
    		sub { $dataset->{lbOut} = $self->{unit}->makeDummyLabel($dataset->{rowType}, $self->{name} . "." . $dataset->{name} . ".out"); };

The input and output labels get created. The input label has the function with the processing logic set as its handler. The output label is just a dummy. Note that the tables don't get connected anywhere, they are just used as storage, without any immediate reactions to their modifications.

    	# chain the input label, if any
    	if (defined $lbFrom) {
    		Triceps::wrapfess
    			"$myname: Collapse internal error: input label chaining for dataset '" . $dataset->{name} . "' to '" . $lbFrom->getName() . "' failed:",
    			sub { $lbFrom->chain($dataset->{lbIn}); };
    		delete $dataset->{fromLabel}; # no need to keep the reference any more, avoid a reference cycle
    	}

And if the fromLabel was used, the Collapse gets connected to it. After that there is no good reason to keep a separate reference to that label, especially considering that it creates a reference loop that would not be cleaned until the input label get cleaned by the unit. So it gets deleted early instead.

    	bless $self, $class;
    	return $self;
    }

The final blessing is boilerplate. The constructor creates the data structures but doesn't implement any logic. The logic goes next:

    # (protected)
    # handle one incoming row on a dataset's input label
    sub _handleInput # ($label, $rop, $self, $dataset)
    {
    	my $label = shift;
    	my $rop = shift;
    	my $self = shift;
    	my $dataset = shift;

    	if ($rop->isInsert()) {
    		# Simply add to the insert table: the effect is the same, independently of
    		# whether the row was previously deleted or not. This also handles correctly
    		# multiple inserts without a delete between them, even though this kind of
    		# input is not really expected.
    		$dataset->{tbInsert}->insert($rop->getRow());

The Collapse object knows nothing about the data that went through it before. After each flush it starts again from scratch. It expects that the stream of rows is self-consistent, and makes the conclusions about the previous data based on the new data it sees. An INSERT rowop may mean one of two things: either there was no previous record with this key, or there was a previous record with this key and then it got deleted. The Delete table can be used to differentiate between these situations: if there was a row that was then deleted, the Delete table would contain that row. But for the INSERT it doesn't matter: in either case it just inserts the new row into the Insert table. If there was no such row before, it would be the new INSERT. If there was such a row before, it would be an INSERT following a DELETE.

    	} elsif($rop->isDelete()) {
    		# If there was a row in the insert table, delete that row (undoing the previous insert).
    		# Otherwise it means that there was no previous insert seen in this round, so this must be a
    		# deletion of a row inserted in the previous round, so insert it into the delete table.
    		if (! $dataset->{tbInsert}->deleteRow($rop->getRow())) {
    			$dataset->{tbDelete}->insert($rop->getRow());
    		}
    	}
    }

The DELETE case is more interesting. If we see a DELETE rowop, this means that either there was an INSERT sent before the last flush and now that INSERT becomes undone, or that there was an INSERT after the flush, which also becomes undone. The actions for these cases are different: if the INSERT was before the flush, this row should go into the Delete table, and eventually propagate as a DELETE during the next flush. If the last INSERT was after the flush, then its row would be stored in the Insert table, and now we just need to delete that row and pretend that it has never been.

That's what the logic does: first it tries to remove from the Insert table. If succeeded, then it was an INSERT after the flush, that became undone now, and there is nothing more to do. If there was no row to delete, this means that the INSERT must have happened before the last flush, and we need to remember this row in the Delete table and pass it on in the next flush.

This logic is not resistant to the incorrect data sequences. If there ever are two DELETEs for the same key in a row (which should never happen in a correct sequence), the second DELETE will end up in the Delete table.

    # Unlatch and flush the collected data, then latch again.
    sub flush # ($self)
    {
    	my $self = shift;
    	my $unit = $self->{unit};
    	my $OP_INSERT = &Triceps::OP_INSERT;
    	my $OP_DELETE = &Triceps::OP_DELETE;
    	foreach my $dataset (values %{$self->{datasets}}) {
    		my $tbIns = $dataset->{tbInsert};
    		my $tbDel = $dataset->{tbDelete};
    		my $lbOut = $dataset->{lbOut};
    		my $next;
    		# send the deletes always before the inserts
    		for (my $rh = $tbDel->begin(); !$rh->isNull(); $rh = $next) {
    			$next = $rh->next(); # advance the irerator before removing
    			$tbDel->remove($rh);
    			$unit->call($lbOut->makeRowop($OP_DELETE, $rh->getRow()));
    		}
    		for (my $rh = $tbIns->begin(); !$rh->isNull(); $rh = $next) {
    			$next = $rh->next(); # advance the irerator before removing
    			$tbIns->remove($rh);
    			$unit->call($lbOut->makeRowop($OP_INSERT, $rh->getRow()));
    		}
    	}
    }

The flushing is fairly straightforward: first it sends on all the DELETEs, then all the INSERTs, clearing the tables along the way. At first I've though of matching the DELETEs and INSERTs together, sending them next to each other in case if both are available for some key. It's not that difficult to do. But then I've realized that it doesn't matter and just did it the simple way.

    # Get the input label of a dataset.
    # Confesses on error.
    sub getInputLabel($$) # ($self, $dsetname)
    {
    	my ($self, $dsetname) = @_;
    	confess "Unknown dataset '$dsetname'"
    		unless exists $self->{datasets}{$dsetname};
    	return $self->{datasets}{$dsetname}{lbIn};
    }

    # Get the output label of a dataset.
    # Confesses on error.
    sub getOutputLabel($$) # ($self, $dsetname)
    {
    	my ($self, $dsetname) = @_;
    	confess "Unknown dataset '$dsetname'"
    		unless exists $self->{datasets}{$dsetname};
    	return $self->{datasets}{$dsetname}{lbOut};
    }

    # Get the lists of datasets (currently only one).
    sub getDatasets($) # ($self)
    {
    	my $self = shift;
    	return @{$self->{dsetnames}};
    }

The getter functions are fairly simple. The only catch is that the code has to check for

    exists

before it reads the value of

    $self->{datasets}{$dsetname}{lbOut}

. Otherwise, if an incorrect

    $dsetname

is used, the reading would return an

    undef

but along the way would create an unpopulated

    $self->{datasets}{$dsetname}

. Which would then cause a crash when

    flush()

tries to iterate through it and finds the dataset options missing.

That's it, Collapse in a nutshell! Another way to do the collapse will be shown in &xrsp;. And one more piece to it is shown in &xrsp;.

Collapse
CCL
Aleri
batch

## Large deletes in small chunks

If you have worked with Coral8 and similar CEP systems, you should be familiar with the situation when you ask it to delete a million rows from the table and the model goes into self-contemplation for half an hour, not reacting to any requests. It starts responding again only when the deletes are finished. That's because the execution is single-threaded, and deleting a million rows takes time.

Triceps is succeptible to the same issue. So, how to avoid it? Even better, how to make the deletes work in the background, at a low priority, kicking in only when there is no other pending requests?

The solution is do do it in smaller chunks. Delete a few rows (say, a thousand or so) then check if there are any other requests. Keep processing these other request until the model becomes idle. Then continue with deleting the next chunk of rows.

Let's make a small example of it. First, let's make a table.

    our $uChunks = Triceps::Unit->new("uChunks");

    # data is just some dumb easily-generated filler
    our $rtData = Triceps::RowType->new(
    	s => "string",
    	i => "int32",
    );

    # the data is auto-generated by a sequence
    our $seq = 0;

    our $ttData = Triceps::TableType->new($rtData)
    	->addSubIndex("fifo", Triceps::IndexType->newFifo())
    ;
    $ttData->initialize();
    our $tData = $uChunks->makeTable($ttData, "tJoin1");
    makePrintLabel("lbPrintData", $tData->getOutputLabel());

The data in the table is completely silly, just something to put in there. Even the index is a simple FIFO, just something to keep the table together.

Next, the clearing logic.

    # notifications about the clearing
    our $rtNote = Triceps::RowType->new(
    	text => "string",
    );

    # rowops to run when the model is otherwise idle
    our $trayIdle = $uChunks->makeTray();

    our $lbReportNote = $uChunks->makeDummyLabel($rtNote, "lbReportNote"
    );
    makePrintLabel("lbPrintNote", $lbReportNote);

    # code that clears the table in small chunks
    our $lbClear = $uChunks->makeLabel($rtNote, "lbClear", undef, sub {
    	$tData->clear(2); # no more than 2 rows deleted per run
    	if ($tData->size() > 0) {
    		$trayIdle->push($_[0]->adopt($_[1]));
    	} else {
    		$uChunks->makeHashCall($lbReportNote, "OP_INSERT",
    			text => "done clearing",
    		);
    	}
    });

We want to get a notification when the clearing is done. This notification will be sent as a rowop with row type

    $rtNote

to the label

    $lbReportNote

. Which then just gets printed, so that we can see it. In a production system it would be sent back to the requestor.

The clearing is initiated by sending a row (of the same type

    $rtNote

) to the label

    $lbClear

. Which does the job and then sends the notification of completion. In the real world not the whole table would probably be erased but only the old data, from before a certain date, like was shown in the &xrsp;. Here for simplicity all the data get wiped out.

But the method

    clear()

stops after the number of deleted rows reaches the limit. Since it's real inconvenient to play with a million rows, we'll play with just a few rows. And so the chunk size limit is also set smaller, to just two rows instead of a thousand. When the limit is reached and there still are rows left in the table, the code pushes the command row into the idle tray for later rescheduling and returns. The adoption part is not strictly necessary, and this small example would work fine without it. But it's a safeguard for the more complicated programs that may have the labels chained, with our clearing label being just one link in a chain. If the incoming rowop gets rescheduled as is, the whole chain will get executed again. which might not be desirable. Re-adopting it to our label will cause only our label (okay, and everything chained from it) to be executed.

How would the rowops in the idle tray get executed? In the real world, the main loop logic would be like this pseudocode:

    while(1) {
    	if (idle tray is empty)
    		timeout = infinity;
    	else
    		timeout = 0;
    	poll(file descriptors, timeout);
    	if (poll timed out)
    		run the idle tray;
    	else
    		process the incoming data;
    }

The example from &xrsp; can be extended to work like this. But it's hugely inconvenient for a toy demonstration, getting the timing right would be a major pain. So instead let's just add the command idle to the main loop, to trigger the idle logic at will. The main loop of the example is:

    while(&readLine) {
    	chomp;
    	my @data = split(/,/); # starts with a command, then string opcode
    	my $type = shift @data;
    	if ($type eq "data") {
    		my $count = shift @data;
    		for (; $count > 0; $count--) {
    			++$seq;
    			$uChunks->makeHashCall($tData->getInputLabel(), "OP_INSERT",
    				s => ("data_" . $seq),
    				i => $seq,
    			);
    		}
    	} elsif ($type eq "dump") {
    		for (my $rhit = $tData->begin(); !$rhit->isNull(); $rhit = $rhit->next()) {
    			&send("dump: ", $rhit->getRow()->printP(), "\n");
    		}
    		for my $r ($trayIdle->toArray()) {
    			&send("when idle: ", $r->printP(), "\n");
    		}
    	} elsif ($type eq "clear") {
    		$uChunks->makeHashCall($lbClear, "OP_INSERT",
    			text => "clear",
    		);
    	} elsif ($type eq "idle") {
    		$uChunks->schedule($trayIdle);
    		$trayIdle->clear();
    	}
    	$uChunks->drainFrame(); # just in case, for completeness
    }

The data is put into the table by the main loop in a silly way: When we send the command like

    data,3

, the mail loop will insert 3 new rows into the table. The contents is generated with sequential numbers, so the rows can be told apart. As the table gets changed, the updates get printed by the label

    lbPrintData

.

The command dump dumps the contents of both the table and of the idle tray.

The command clear issues a clearing request by calling the label

    $lbClear

. The first chunk gets cleared right away but then the control returns back to the main loop. If not all the data were cleared, an idle rowop will be placed into the idle tray.

The command idle that simulates the input idleness will then pick up that rowop from the idle tray and reschedule it.

All the pieces have been put together, let's run the code. The commentary are interspersed, and as usual, the input lines are shown in bold:

> data,1 tJoin1.out OP_INSERT s="data_1" i="1" > clear tJoin1.out OP_DELETE s="data_1" i="1" lbReportNote OP_INSERT text="done clearing"

This is pretty much a dry run: put in one row (less than the chunk size), see it deleted on clearing. And see the completion reported afterwards.

> data,5 tJoin1.out OP_INSERT s="data_2" i="2" tJoin1.out OP_INSERT s="data_3" i="3" tJoin1.out OP_INSERT s="data_4" i="4" tJoin1.out OP_INSERT s="data_5" i="5" tJoin1.out OP_INSERT s="data_6" i="6"

Add more data, which will be enough for three chunks.

> clear tJoin1.out OP_DELETE s="data_2" i="2" tJoin1.out OP_DELETE s="data_3" i="3"

Now the clearing does one chunk and stops, waiting for the idle condition.

> dump dump: s="data_4" i="4" dump: s="data_5" i="5" dump: s="data_6" i="6" when idle: lbClear OP_INSERT text="clear"

See what's inside: the remaining 3 rows, and a row in the idle tray saying that the clearing is in progress.

> idle tJoin1.out OP_DELETE s="data_4" i="4" tJoin1.out OP_DELETE s="data_5" i="5"

The model goes idle once more, one more chunk of two rows gets deleted.

> data,1 tJoin1.out OP_INSERT s="data_7" i="7" > dump dump: s="data_6" i="6" dump: s="data_7" i="7" when idle: lbClear OP_INSERT text="clear"

What will happen if we add more data in between the chunks of clearing? Let's see, let's add one more row. It shows up in the table as usual.

> idle tJoin1.out OP_DELETE s="data_6" i="6" tJoin1.out OP_DELETE s="data_7" i="7" lbReportNote OP_INSERT text="done clearing" > dump > idle

On the next idle condition the clearing picks up whatever was in the table for the next chunk. Since there were only two rows left, it's the last chunk, and the clearing reports a successful completion. And a dump shows that there is nothing left in the table nor in the idle tray. The next idle condition does nothing, because the idle tray is empty.

The deletion could also be interrupted and cancelled, by removing the row from the idle tray. That would involve converting the tray to an array, finding and deleting the right rowop, and converting the array back into the tray. Overall it's fairly straightforward. The search in the array is linear but there should not be that many idle requests, so it should be quick enough.

The delete-by-chunks logic can be made into a template, just I'm not sure yet what is the best way to do it. It would have to have a lot of configurable parts.

On another subject, scheduling the things to be done on idle adds an element of unpredictability to the model. It's impossible to predict the exact timing of the incoming requests, and the idle work may get inserted between any of them. Presumably it's OK because the data being deleted should not be participating in any logic at this time any more. For repeatability in the unit tests, make the chunk size adjustable and adjust it to a size larger than the biggest amount of data used in the unit tests.

A similar logic can also be used in querying the data. But it's more difficult. For deletion the continuation is easy: just take the first row in the index, and it will be the place to continue (because the index is ordered correctly, and because the previous rows are getting deleted). For querying you would have to remember the next row handle and continue from it. Which is OK if it can not get deleted in the meantime. But if it can get deleted, you'll have to keep track of that too, and advance to the next row handle when this happens. And if you want to receive a full snapshot with the following subscription to all updates, you'd have to check whether the modified rows are before or after the marked handle, and pass them through if they are before it, letting the user see the updates to the data already received. And since the data is being sent to the user, filling up the output buffer and stopping would stop the whole model too, and not restart until the user reads the buffered data. So there has to be a flow control logic that would stop the query when output buffer fills up, return to the normal operation, and then reschedule the idle job for the query only when the output buffer drains down. I've kind of started on doing an example of the chunked query too, but then because of all these complications decided to leave it for later.

chunks
table remove row
label chaining
label adoption