# Scheduler

Administration

The `scheduler` is the part of the ZMailer that manages message processing outbound from the MTA proper.

The fig.zadmsch.zmprocs repeats earlier picture showing central components of the system, and where the `scheduler` is in relation to them all.

## ZMailer's processes; Scheduler

    TODO!FIXME!

      Here we present longer examples out of the  scheduler.conf, and
      reasons why the default script is as it is

      - Intro
      - Principles of scheduling, "threads"
      - Clause selectors
        - "local/*"
        - "smtp/*.xyz"
        - "smtp/*"
        - "OTHER/*"
        - How to roll your own when needed
      - Something about the resource control ?
      - scheduler.auth file, and its purpose
      - MAILQv1/v2 interface, mailq-command
        - but the protocols are REFERENCE material
      - (manual-)expirer
      - Diagnostics reporting, forms files

      Appendix B contains full samples of scheduler.conf, and scheduler.auth

The major action of the `scheduler` is to periodically start up Transport Agents and tell them what to do. This is controlled by a table in a configuration file that is read by the `scheduler` when it starts.

The `scheduler` receives the messages from the `router` in two files: Original message file in `$`POSTOFFICE`/queue/`, and `router` written transport-specifications in `$`POSTOFFICE`/transport/`.

Because usually (UNIX) systems don't like of having very large amounts of files in directories, as lookups for them become intolerably slow, the `scheduler` has a subdirectory hashing mechanism. Both `queue/` and `transport/` can be split to subdirectories at one or two levels with names of `A` thru `Z`. See for the `-H` option of the `scheduler` below.

The idea with the sub-directory hashes is to split the workset into as even subsets as possible, usually busy systems are run with `-HH` to have 26*26 sub-sub-directories into which the sets gets divided.

## Principiles of scheduling: Threads

The `router` produces recipients address quads which consists of four components:

- Channel

- Host

- User

- Privilege

Of these, Channel, and Host parts are used by the `scheduler` to classify message recipients and to choose to what Transport Agents to use, and how to use them.

To control how messages are sent out, the `scheduler` uses catenation of: Channel/Host to group recipients. All recipients with same values there are groupped together in what is called thread.

Another related thing is so called thread group, which is the collection of all threads within same scheduler Selector Clause. See zadm-scheduler-scheduler.conf.

### Scheduler's Threads/Thread Groups

In normal case the scheduling is done by running single transport agent for the thread, which delivers messages one at the time. This means that a single destination with thousands of messages does not block the system significantly more than some other destination with a single message.

Also in normal cases, when messages in given thread are all delivered, or otherwise determined that nothing can be done, the transport agent can be switched to another thread within the same thread group.

## `Scheduler` Resource Control

FIXME! TO BE WRITTEN!

Note, there are three kinds of resource-pool limitation parameters which control when a given channel+host pair (thread) is NOT taken into processing:

`MaxTA`: (Set in */* clause)

- GLOBAL parameter limiting the number of transport-agent processes that the scheduler can have running at the same time.

  With this you can limit the number of TA processes running at the same time lower than maximum allowed by your OS setup.

  The scheduler detects the max number of FDs allowed for a process, and analyzing how many FDs each TA interface will need -- plus reserving 10 FDs for the itself, result is probed maxkids.

`MaxChannel`: (default: probed maxkids)

- Selector clause specific value limiting how many transport-agent processes can be running on which the channel part is the same. You may specify dis-similar values for these as well. For example you may use value '50' for all your 'smtp' channel entries, except that you want always to guarantee at least five more for your own domain deliveries, and thus have:

            smtp/*your.domain
                  maxchannel=55

  If the sum of all maxchannel values in different channels exceeds that of maxta, then maxta value will limit the amount of work done in extreme load situations.

`MaxRing`: (default: probed maxkids)

- This limits the number of parallel transport agents within each selector definition. This defined the size of the POOL of transport agent processes available for processing the threads matching the selector clause.

`MaxTHR`: (default: 1)

- This limits the number of parallel transport agents within each thread; that is, using higher value than default 1 will allow running more than one TA for the jobs at the thread.

  Do note that running more than one TA in parallel may also require lowering OVERFEED value. (E.g. having a queue of 30 messages will not benefit from more TAs, unless they all get something to process. Having OVERFEED per default at 150 will essentially feed whole queue to one TA, others are not getting any.)

`OverFeed`:

- This tells how many job specifiers to feed to the TA when the TA process state is STUFFING Because the scheduler is a bit sluggish to spin around to spot active TAs, it does make sense to feed more than one task to a TA, and then wait for the results.

## The `scheduler.conf` file

Any line starting with a `#` character is assumed to be a comment line, and is ignored, as are empty lines. All other lines must follow a rigid format.

The `scheduler` configuration file consists of a set of clauses. Each clause is selected by the pattern it starts with. The patterns for the clauses are matched, in sequence, with the channel/host string for each recipient address. When a clause pattern matches an address, the parameters set in the clause will be applied to the scheduler's processing of that address. If the clause specifies a command, the clause pattern matching sequence is terminated. Example of the clause can be seen in figure fig.adm.clause-example.

### Example of `scheduler.conf` clause

    local/*
            interval=10s
            expiry=3h
            # want 20 channel slots in case of blockage on one
            maxchannel=20
            # want 20 thread-ring slots
            maxring=20
            command="mailbox -8"

A clause consists of:

- A selection pattern (in shell style) that is matched against the channel/host string for an address.

- 0 or more variable assignments or keywords (described below).

If the selection pattern does not contain a '/', it is assumed to be a channel pattern and the host pattern is assumed to be the wildcard '*'.

The components of a clause are separated by whitespace. The pattern introducing a clause must start in the first column of a line, and the variable assignments or keywords inside a clause must not start in the first column of a line. This means a clause may be written both compactly all on one line, or spread out with an assignment or keyword per line.

If the clause is empty (i.e., consists only of a pattern), then the contents of the next non-empty clause will be used.

The typical configuration file will contain the following clauses:

- a clause matching all addresses (using the pattern */*) that sets up default values.

- a clause matching the local delivery channel (usually local).

- a clause matching the deferred delivery channel (usually hold).

- a clause matching the error reporting channel (usually error).

- clauses specific to the other channels known by the `router`, for example: smtp and uucp.

The actual names of these channels are completely controlled by the `router` configuration file.

Empty lines, and lines whose first non-whitespace character is #, are ignored.

Variable values may be unquoted words or values or double quoted strings. Intervals (delta time) are specified using a concatenation of numbers suffixed with 's', 'm', 'h', or 'd'; modifiers designating the number as a second, minute, hour, or day value. For example: 1h5m20s.

The known variables and keywords, and their typical values and semantics are:

`interval` (1m)

- specifies the primary retry interval, which determines how frequently a transport agent should be scheduled for an address. The value is a delta time specification. This value, and the retries value mentioned below, are combined to determine the interval between each retry attempt.

`idlemax`

- When a transport agent runs out of jobs, they are moved to idle pool, and if a transport agent spends more than idlemax time in there, it is terminated.

`expiry` (3d)

- specifies the maximum age of an address in the scheduler queue before a repeatedly deferred address is bounced with an expiration error. The actual report is produced when all addresses have been processed.

`retries` (1 1 2 3 5 8 13 21 34)

- specifies the retry interval policy of the scheduler for an address. The value must be a sequence of positive integers, these being multiples of the primary interval before a retry is scheduled. The scheduler starts by going through the sequence as an address is repeatedly deferred. When the end of the sequence is reached, the scheduler will jump into the sequence at a random spot and continue towards the end. This allows various retry strategies to be specified easily:

  brute force (or jackhammer)

- `retries=0`

constant primary interval

- `retries=1`

instant backoff

- `retries="1 50 50 50 50 50 50 50 50 50 50 50 50"`

slow increasing (fibonacci) sequence

- `retries="1 1 2 3 5 8 13 21 34"`

s-curve sequence

- `etries="1 1 2 3 5 10 20 25 28 29 30"`

exponential sequence

- `retries="1 2 4 8 16 32 64 128 256"`

`maxta` (0)

- FIXME: REVISE RESOURCE CHECK NOTES! if retrying an address would cause the number of simultaneously active transport agents to exceed the specified value, the retry is postponed. The check is repeated frequently so the address may be retried as soon as possible after the scheduled retry interval. If the value is 0, a value of 1000 is used instead. Keep in mind that all running transport agents will keep open two `pipe`(2) file-handles, (or one `socketpair`(2), if system has such bidirectional pipe entity,) and thus system-wide limits may force a lower maximum than 1000. On a system with a maximum of 256 open files, this would most likely succeed at 120.

`maxchannel` (0)

- if retrying an address would cause the number of simultaneously active transport agents processing mail for the same channel to exceed the specified value, the retry is postponed. The check is repeated frequently so the address may be retried as soon as possible after the scheduled retry interval. If the value is 0, a value of 1000 is used instead.

`maxring` (0)

- Recipients are groupped into threads, and similar threads are groupped into thread-rings, where the same transport agent can be switched over from one recipient to another. This defines how many transport agents can be running at any time at the ring.

`skew` (5)

- is the maximum number of retries before the retry time is aligned to a standard boundary (seconds since epoch, modulo primary interval). The lower this number (1 is lowest), the faster the alignment is done. The purpose of this alignment is to ensure that eventually a single transport agent invokation will be able to process destination addresses that arrived randomly at the scheduler.

`user` (root)

- is the user id of a transport agent processing the address. The value is either numeric (a uid) or an account name.

`group` (daemon)

- is the group id of a transport agent processing the address. The value is either numeric (a gid) or a group name.

`command` (smtp -srl ${LOGDIR}/smtp ${host}

- is the command line used to start a transport agent to process the address. The program pathname is specified relative to the `$`MAILBIN`/ta/` directory. The string ${channel} is replaced by the current matched channel, and ${host} is replaced by the current matched host, from the destination address, and ${`LOGDIR`} substitutes ZENV variable `LOGDIR` value there.

  It is strongly recommended that the `${host}` is not to be used on command definition, as it limits the recyclability of the idled transporter.

`bychannel`

- is a keyword (with no associated value which tells the `scheduler` that the transport agent specified in the command will only process destination addresses that match the first destination channel it encounters.

  This is automatically set when the string `${channel}` occurs in the command, but may also be specified manually. This is rarely used.

`queueonly`

- a clause with queueonly flag does not auto-start at the arrival of a message, instead it must be started by means of `smtpserver`(8) command ETRN thru an SMTP connection.

An example of full `scheduler.conf` file is in figure fig.adm.full-schedconf-sample.

### Example of full `scheduler.conf` file

    # Default values
    */*       interval=1m expiry=3d retries="1 1 2 3 5 8 13 21 34"
              maxring=0 maxta=0 skew=5 user=root group=daemon
    # Boilerplate parameters for local delivery and service channels
    local/*   interval=10s expiry=3h maxchannel=2 command=mailbox
    error     interval=5m maxchannel=10 command=errormail
    hold/*    interval=5m maxchannel=1 command=hold

    # Miscellaneous channels supported by router configuration
    smtp/*.toronto.edu
    smtp/*.utoronto.ca maxchannel=10 maxring=2
              command="smtp -srl ${LOGDIR}/smtp"
    smtp      maxchannel=10 maxring=5
              command="smtp -esrl ${LOGDIR}/smtp"

    uucp/*    maxchannel=5 command="sm -c ${channel} uucp"

The first clause (*/*) sets up default values for all addresses. There is no command specification, so clause matching will continue after address have picked up the parameters set here.

The third clause (error) has an implicit host wildcard of *, so it would match the same as specifying error/* would have.

The fifth clause (smtp/*.toronto.edu) has no further components so it selects the components of the following nonempty clause (the sixth).

Both the fifth and sixth clauses are specific to address destinations within the TORONTO.EDU and UTORONTO.CA organization (the two are parallel domains). At most 10 deliveries to the smtp channel may be concurrently active, and at most 2 for all possible hosts within TORONTO.EDU. If $host is mentioned in the command specification, the transport agent will only be told about the message control files that indicate SMTP delivery to a particular host. The actual host is picked at random from the current choices, to avoid systematic errors leading to a deadlock of any queue.

## `Scheduler`'s `Mailq`

FIXME! TO BE WRITTEN!

## `Scheduler`'s `scheduler.auth` control file

FIXME! TO BE WRITTEN!

## `manual-expirer`

FIXME! TO BE WRITTEN!

## `Scheduler`'s Diagnostics Reporting

FIXME! TO BE WRITTEN!

### `Scheduler`'s Diagnostics Reporting, Forms Files

FIXME! TO BE WRITTEN!