# Queue API description

A Tarantool/Box instance can serve as a Queue Manager, along
with any other database work necessary.

A single properly configured Tarantool/Box space can store any
number of queues. Multiple spaces can be used as well - for
partitioning or logical separation of queues.

Queues support task priority. Priority value lays in the range
[0, 255], with default value being 127. A higher value means higher
priority, lower value - lower priority.

Each queue has one (currently) associated *fiber* taking care of
it. The fiber is started upon first access to the queue. The job
of the fiber is to monitor orphaned tasks, as well as prune and
clean the queue from obsolete tasks.

To configure a space supporting queues, use the following parameters:

```cfg
space = [
    {
        enabled = 1,
        index = [
            {
                type = "TREE",
                unique = 1,
                key_field = [
                    {
                        fieldno = 0,
                        type = "STR"
                    }
                ]
            },
            {
                type = "TREE",
                unique = 0,
                key_field = [
                    {
                        fieldno = 1,    # queue, aka tube
                        type = "STR"
                    },
                    {
                        fieldno = 2,    # status
                        type = "STR"
                    },
                    {
                        fieldno = 4,    # ipri
                        type = "STR"
                    },
                    {
                        fieldno = 5    # pri
                        type = "STR"
                    }
                ]
            },
            {
                type    = "TREE",
                unique  = 0,
                key_field = [
                    {
                        fieldno = 3,    # next_event
                        type = "NUM64"
                    }
                ]
            }
        ]
    }
]
```

It may also be desirable to tune server `readahead` configuration
variable if many consumers re-use the same socket for getting and
acknowledging tasks.

The recommended value can be calculated as:

```
  consumers-per-socket * (256 + largest task size)
```

For example, if the largest task size is 256 bytes, and average
number of consumers per socket is 10, the recommended `readahead`
value must be at least 51 200 bytes.

## Terminology

* *Consumer* - a process, taking and executing tasks
* *Producer* - a process adding new tasks

### Arguments of queue API functions

* `space` (number) space id. To avoid confusion and broken statistics,
  it's necessary to consistently use numbers to identify spaces,
* `tube` (string) - queue name,
* `delay` (number) - a delay between the moment a task is queued
  and is executed, in seconds
* `ttl` (number) - task time to live, in seconds. If `delay` is
  given along with `ttl`, the effective task time to live is
  increased by the amount of `delay`,
* `ttr` (number) - task time to run, the maximal time allotted
  to a consumer to execute a task, in seconds,
* `pri` (number) - task priority [0..255],
* `id` (string) - task id,
* `timeout` (number) - timeout in seconds for the Queue API function.

### Task states

* `ready` - a task is ready for execution,
* `delayed` - a task is awaiting task `delay` to expire, after
   which it will become `ready`,
* `taken` - a task is taken by a consumer and is being executed,
* `done` - a task is complete (but not deleted, since the consumer
   called `done` rather than `ack`),
* `buried` - a task is neither ready nor taken nor complete, it's
   excluded (perhaps temporarily) from the list of tasks for
   execution, but not deleted.

### The format of task tuple

Queue API functions, such as `put`, `take`, return a task.
The task consists of the following fields:

1. `id` (string) - task identifier
1. `tube` (string) - queue identifier
1. `status` (string) - task status
1. task data (all fields passed into `put`/`urgent` when
   the task was created)

## API

### Producer

#### queue.put(space, tube, delay, ttl, ttr, pri, ...)

Enqueue a task. Returns a tuple, representing the new task.
The list of fields with task data ('...')is optional.

#### queue.urgent(space, tube, delay, ttl, ttr, pri, ...)

Enqueue a task. The task will get the highest priority.
If `delay` is not zero, the function is equivalent to `put`.

### Consumer

#### queue.take(space, tube, timeout)

If there are tasks in the queue `ready` for execution,
take the highest-priority task.
Otherwise, wait for a `ready` task to appear in the queue, and, as
soon as it appears, mark it as `taken` and return to the consumer.
If there is a `timeout`, and the task doesn't appear until the
timeout expires, return 'nil'. If timeout is not given, wait
indefinitely until a task appears.

All the time while the consumer is working on a task, it must keep
the connection to the server open. If a connection disappears while
the consumer is still working on a task, the task is put back on the
`ready` list.

#### queue.ack(space, id)

Confirm completion of a task. Before marking a task as complete,
this function verifies that:

* the task is `taken` and
* the consumer that is confirming the task is the one which took it.

Consumer identity is established using a session identifier. In
other words, the task must be confirmed by the same connection
which took it. If verification fails, the function returns an
error.

On success, delete the task from the queue.

#### queue.release(space, id [, delay [, ttl ] ])

Return a task back to the queue: the task is not executed.
Additionally, a new time to live and re-execution delay can be
provided.
If `ttl` is not defined (or zero) the method won't prolong ttl.


#### queue.requeue(space, id)

Return a task to the queue, the task is not executed. Puts
the task at the end of the queue, so that it's executed only
after all existing tasks in the queue are executed.

#### queue.bury(space, id)

Mark a task as `buried`. This special status excludes
the task from the active list, until it's `dug up`.
This function is useful when several attempts to execute a task
lead to a failure. Buried tasks can be monitored by the queue
owner, and treated specially.

#### queue.done(space, id, ...)

Mark a task as complete (`done`), but don't delete it.
Replaces task data with the supplied fields ('...').

### Common functions (neither producer nor consumer).

#### queue.dig(space, id)

'Dig up' a buried task, after checking that the task
is buried. The task status is changed to `ready`.

#### queue.kick(space, tube [, count] )

'Dig up' `count` tasks in a queue. If `count` is not given,
digs up just one buried task.

#### queue.unbury(space, id)

An alias to `dig`.

#### queue.delete(space, id)

Delete a task from the queue (regardless of task state or status).

#### queue.meta(space, id)

Return taks metadata:

1. `id` (string) - task id
1. `tube` (string) - queue id
1. `status` (string) - task status
1. `event` (time64) - time of the next important event in task
   life time, for example, when `ttl` or `ttr` expires, in seconds
   since start of the UNIX epoch
1. `ipri` (string) - internal value of the task priority
1. `pri` (string) - task priority as set when the task was added
   to the queue
1. `cid` (number) - consumer id, of the consumer which took the
   task (only if the task is `taken`)
1. `created` (time64) - time when the task was created (seconds
   since start of the UNIX epoch).
1. `ttl` (time64) - task time to live
1. `ttr` (time64) - task time to run
1. `cbury` (count) - how many times the task was buried
1. `ctaken` (Ч) - how many times the task was taken
1. `now` (time64) - time recorded when the meta was called

#### queue.peek(space, id)

Return a task by task id. Returned tuple has the following
fields:

1. `id` (string) - task identifier
1. `tube` (string) - queue identifier
1. `status` (string) - task status
1. task data (all fields passed into `put`/`urgent` when
   the task was created).

#### queue.statistics()

Return queue module statistics accumulated since server start.
The statistics is broken down by queue id. Only queues on which
there was some activity are included in the output.

The format of the statistics is a sequence of rows, where each
odd row is the name of a statistical parameter, and the
next even row is the value.

