<sect1>Scheduler
<p>

The <em>scheduler</em> daemon manages the delivery processing of 
messages in ZMailer.

The <em>router</em> creates message control files in the
<tt>&dollar;POSTOFFICE/transport/</tt> directory. These refer to 
the original message files in the <tt>&dollar;POSTOFFICE/queue/</tt> 
directory.

The <em>scheduler</em> reads each message control file from
<tt>&dollar;POSTOFFICE/transport/</tt>, translates the contained 
message and destination information into internal data structures, 
and unlinks the message control file.

Based on scheduling, priority, and execution information read from a
configuration file, the <em>scheduler</em> arranges to execute 
Transport Agents relevant to the queued messages.

At the time scheduled for a particular transport agent invocation, the
<em>scheduler</em> will start a transport agent (or use one from idle-pool),
and tell it one by one which message control files to process.
When all the destination addresses in a message have been processed, the
<em>scheduler</em> performs error reporting tasks if any, and then deletes the
message control file in <tt>&dollar;POSTOFFICE/transport/</tt> and 
the original message file in <tt>&dollar;POSTOFFICE/queue/</tt>.

All message delivery is actually performed by Transport Agents, which
are declared in a configuration file for the <em>scheduler</em>.
Each transport agent is executed with the same current directory as the
<em>scheduler</em>. <em>scheduler-transporter</em> interaction protocol is
described later.

The standard output of each transport agent are destination address
delivery reports; either successful delivery, unsuccessful delivery, or
deferral of the address. Each report uses byte offsets in the message
control file to refer to the address. Reports may also include a comment
line which will be displayed in the reports of the <em>scheduler</em>.

Two types of reports are produced:
<enum>
<item>Error messages caused by unsuccessful delivery of a message
are appended to its message control file. Occasionally,
for example, when all addresses have been processed, the <em>scheduler</em>
generates an error message to the error return address of the message
(usually the original sender).  
<item>The <em>scheduler</em> binds itself to a well-known TCP/IP port
<tt>(MAILQ, TCP port 174)</tt> on startup.  Any connections to this port
are processed synchronously in the scheduler at points in the execution
where the state is internally consistent. The <em>scheduler</em>
simply dumps its internal state in a terse format to the TCP stream.
It is expected that the client program will reconstruct the data structures
sufficiently to give a user a good idea of what the scheduler thinks the
world looks like. The <em>mailq (1)</em> program serves this purpose.
</enum>

<bf>Usage</bf>

Invoking <em>scheduler</em> without any parameter will start it as a daemon.

<tscreen><verb>
scheduler [-divFHQSVW] [-f configfile] [-E newentsmax] [-Llogfile] [-lstatisticslog]
[-Ntranspmaxfno] [-P postoffice] [-R maxforkfreq] [-q rendezvous]
</verb></tscreen>

<bf>Parameters</bf>

<descrip>
<tag/<tt>-d</tt>/ run as a daemon, usually used after -v to log daemon activity 
in great detail.
<tag/<tt>-E newentsmax</tt>/ when globbing new tasks from the directory, pick 
only first <tt>newentsmax</tt> of them, and leave rest for a new scan run.
<tag/<tt>-f configfile</tt>/ overrides the default configuration file 
<tt>MAILSHARE/scheduler.cf</tt>.
<tag/<tt>-F</tt>/ Freeze -- don't actually run anything, just do queue scanning.
(For debug purposes..)
<tag/<tt>-H -HH</tt>/ Use multi-level hashing at the spool directories.
This will efficiently reduce the lengths of the scans at the directories
to find some arbitary file in them.
One <tt>H</tt> means <tt>single level hashing</tt>, two <tt>HH</tt> mean
<tt>dual level hashing</tt>. <tt>Hash</tt> is directory which name is
single upper case alphabet (A-Z).
<tag/<tt>-i</tt>/ run interactively, i.e., not as a daemon.
<tag/<tt>-L logfile</tt>/ overrides the default log file location 
<tt>LOGDIR/scheduler</tt>.
<tag/<tt>-l statisticslog</tt>/ starts the appending of delivery statistics 
information (ASCII form) into given file.  No default value.
<tag/<tt>-N transmaxfno</tt>/ sets how many filehandles are allocated for the 
<em>scheduler</em>'s started children (if the system has adjustable resources.)
<tag/<tt>-P postoffice</tt>/ specifies an alternate <tt>POSTOFFICE/</tt> directory.
<tag/<tt>-q rendezvous</tt>/ the rendezvous between machines without TCP/IP networking, 
<tag/<tt>-Q</tt>/ The `<tt>Q</tt>'-mode, don't output the old style data into the queue 
querier, only the new-style one.
<tag/<tt>-S</tt>/ Synchronous startup mode, scans all jobs at the directory before 
starting even the first transporter.
<tag/<tt>-v</tt>/ Verbose logging in interactive mode -- for debug uses.
<tag/<tt>-V</tt>/ Print version, and exit
<tag/<tt>-W</tt>/ Another option for debugging, delay the start of the verbose logging
until after all jobs have been scanned in, and it is time to start the transporters. 
<em>scheduler</em> and <em>mailq (1)</em> is done using a well-known named pipe. 
This option overrides the default location for this special file, either 
<tt>&dollar;RENDEZVOUS</tt> or <tt>/usr/tmp/.mailq.text</tt>.
</descrip>


<sect2>Configuration Language
<p>
The <em>scheduler</em> configuration file consists of a set of clauses. 

<sect3>Clause selection
<p>Each clause is selected by
the pattern it starts with. The patterns for the clauses are matched,
in sequence, with the <tt>channel/host</tt> string for each recipient
address. When a clause pattern matches an address, the parameters set in
the clause will be applied to the <em>scheduler</em>'s processing of 
that address. If the clause specifies a command, the clause pattern 
matching sequence is terminated. This is a clause:
<tscreen><verb>
local/*	interval=10s
	expiry=3h
	# want 20 channel slots in case of blockage on one
	maxchannel=20
	# want 20 thread-ring slots
	maxring=20
	command="mailbox -8"
</verb></tscreen>

A clause consists of:
<itemize>
<item>A selection pattern (in shell style) that is matched against the
<tt>channel/host</tt> string for an address.
<item>0 or more variable assignments or keywords (described below).
</itemize>
If the selection pattern does not contain a <tt>/</tt>, it is assumed to
be a channel pattern and the host pattern is assumed to be the wildcard
<tt>*</tt>.

<sect3>Clause components
<p>
The components of a clause are separated by whitespace. The pattern
introducing a clause must start in the first column of a line, and the
variable assignments or keywords inside a clause must not start in the
first column of a line. This means a clause may be written both compactly
all on one line, or spread out with an assignment or keyword per line.

If the clause is empty (i.e., consists only of a pattern), then the
contents of the next non-empty clause will be used.

The typical configuration file will contain the following clauses:
<itemize>
<item>a clause matching all addresses (using the pattern <tt>*/*</tt>)
that sets up default values.
<item>a clause matching the local delivery channel (usually <tt>local</tt>).
<item>a clause matching the deferred delivery channel (usually <tt>hold</tt>).
<item>a clause matching the error reporting channel (usually <tt>error</tt>).
<item>clauses specific to the other channels known by the <em>router</em>,
for example, <tt>smtp</tt> and <tt>uucp</tt>.
</itemize>
The actual names of these channels are completely controlled by the
<em>router</em> configuration file.

Empty lines, and lines whose first non-whitespace character is <tt>&num</tt>,
are ignored.

Variable values may be unquoted words or values or doublequoted strings.
Intervals (delta time) are specified using a concatenation of: numbers
suffixed with `s', `m', `h', or `d' modifiers designating the number as
a second, minute, hour, or day value. For example: <tt>1h5m20s</tt>.

<sect3>Variables and keywords
<p>
The known variables and keywords, and their typical values and semantics are:
<descrip>
<tag/<tt>interval (1m)</tt>/ specifies the primary retry interval, which 
determines how frequently a transport agent should be scheduled for an address. 
The value is a delta time specification. This value, and the <tt>retries</tt>
value mentioned below, are combined to determine the interval between
each retry attempt.

<tag/<tt>idlemax (3x interval)</tt>/ When a transport agent runs out of jobs, 
they are moved to <tt>idle pool</tt>, and if a TA spends more than 
<tt>idlemax</tt> time in there, it is terminated.

<tag/<tt>expiry (3d)</tt>/ specifies the maximum age of an address in the 
<em>scheduler</em> queue before a repeatedly deferred address is bounced 
with an expiration error. The actual report is produced when all addresses 
have been processed.

<tag/<tt>retries (1 1 2 3 5 8 13 21 34)</tt>/ specifies the retry interval 
policy of the <em>scheduler</em> for an address. The value must be a sequence 
of positive integers, these being multiples of the primary interval before a 
retry is scheduled. The <em>scheduler</em> starts by going through the 
sequence as an address is repeatedly deferred. When the end of the sequence 
is reached, the <em>scheduler</em> will jump into the sequence at a random 
spot and continue towards the end. This allows various retry strategies to be 
specified easily:
<itemize>
<item>brute force (or "jackhammer"):
<tscreen><verb>
retries=0
</verb></tscreen>
<item>constant primary interval:
<tscreen><verb>
retries=1
</verb></tscreen>
<item>instant backoff:
<tscreen><verb>
retries="1 50 50 50 50 50 50 50 50 50 50 50 50"
</verb></tscreen>
<item>slow increasing (fibonacci) sequence:
<tscreen><verb>
retries="1 1 2 3 5 8 13 21 34"
</verb></tscreen>
<item>s-curve sequence:
<tscreen><verb>
retries="1 1 2 3 5 10 20 25 28 29 30"
</verb></tscreen>
<item>exponential sequence:
<tscreen><verb>
retries="1 2 4 8 16 32 64 128 256"
</verb></tscreen>
<item>etc.
</itemize>

<tag/<tt>maxta (0)</tt>/ if retrying an address would cause the number of 
simultaneously active transport agents to exceed the specified value, 
the retry is postponed. The check is repeated frequently so the address 
may be retried as soon as possible after the scheduled retry interval. 
If the value is 0, a value of 1000 is used instead. Keep in mind that all 
running transport agents will keep open <bf>two</bf> <tt>pipe (2)</tt> 
file-handle, and thus system-wide limits may force lower maximum, than 1000.
On a system with max 256 open files, this would most likely succeed at 120.

<tag/<tt>maxchannel (0)</tt>/ 
if retrying an address would cause the number of simultaneously active
transport agents processing mail for the same channel to exceed
the specified value, the retry is postponed.
The check is repeated frequently so the address may be retried as soon
as possible after the scheduled retry interval. If the value is 0,
a value of 1000 is used instead.

<tag/<tt>maxring (0)</tt>/ Recipients are groupped into <tt>threads</tt>, 
and similar threads are groupped into <tt>thread-rings</tt>, where same 
transport agent can be switched over from one recipient to another. This 
defines how many transport agents can be running at any time at the ring.

<tag/<tt>skew (5)</tt>/ is the maximum number of retries before the retry 
time is aligned to a standard boundary (seconds since epoch, modulo primary 
interval). The lower this number (1 is lowest), the faster the alignment is 
done. The purpose of this alignment is to ensure that eventually a single 
transport agent invocation will be able to process destination addresses 
that arrived randomly to the <em>scheduler</em>.

<tag/<tt>user (root)</tt>/ is the user id of a transport agent processing 
the address. The value is either numeric (a uid) or an account name.

<tag/<tt>group (daemon)</tt>/ is the group id of a transport agent processing 
the address. The value is either numeric (a gid) or a group name.

<tag/<tt>command</tt>/ <tt>command (smtp -srl /var/log/smtp &dollar;host)</tt> 
is the command line used to start a transport agent to process the address.
The program pathname is specified relative to the <tt>&dollar;MAILBIN/ta/</tt>
directory. The string `<tt>&dollar;channel</tt>' is replaced by the current
matched channel, and `<tt>&dollar;host</tt>' replaced by the current matched 
host, from the destination address. It is strongly recommended that the 
`<tt>&dollar;host</tt>' is not to be used on a command definition, as it limits 
the usability of idled transporter.

<tag/<tt>bychannel</tt>/ is a keyword (with no associated value) that tells 
the <em>scheduler</em> that the transport agent specified in the command will 
only process destination addresses that match the first destination channel it 
encounters. This is automatically set when the string `<tt>&dollar;channel</tt>' 
occurs in the command, but may also be specified manually by this keyword. 
This is rarely used.

<tag/<tt>sm (8)</tt>/ to avoid the <em>scheduler</em> having to discover that 
the transport agent didn't consider a destination address the <em>scheduler</em>
thought it should have. The penalty for not using this keyword in such situations 
is that delivery of passed-over addresses will be staggered.

<tag/<tt>queueonly</tt>/ a clause with <tt>queueonly</tt> flag does not auto-start 
at the arrival of a message, instead it must be started by means of 
<em>smtpserver(8)</em> command <bf>ETRN</bf> through an SMTP connection.

</descrip>

For example, this is a complete configuration file:
<tscreen><verb>
# Default values
*/*	interval=1m expiry=3d retries="1 1 2 3 5 8 13 21 34"
	maxring=0 maxta=0 skew=5 user=root group=daemon
# Boilerplate parameters for local delivery and service channels
local/*	interval=10s expiry=3h maxchannel=2 command=mailbox
error	interval=5m maxchannel=10 command=errormail
hold/*	interval=5m maxchannel=1 command=hold
# Miscellaneous channels supported by router configuration
smtp/*.toronto.edu
smtp/*.utoronto.ca maxchannel=10 maxring=2
	command="smtp -srl /var/log/smtp"
smtp/*	maxchannel=10 maxring=5
	command="smtp -esrl /var/log/smtp"
uucp/*	maxchannel=5 command="sm -c $channel uucp"
</verb></tscreen>
The first clause (<tt>*/*</tt>) sets up default values for all
addresses. There is no command specification, so clause matching
will continue after address have picked up the parameters set here. 

The third clause (<tt>error</tt>) has an implicit host wildcard of `*', 
so it would match the same as specifying <tt>error/*</tt> would have.
 
The fifth clause (<tt>smtp/*.toronto.edu</tt>) has no further components 
so it selects the components of the following non-empty clause (the sixth).

Both the fifth and sixth clauses are specific to address destinations
within the <tt>TORONTO.EDU</tt> and <tt>UTORONTO.CA</tt> organization (the 
two are parallel domains). At most 10 deliveries to the <tt>smtp</tt> channel
may be concurrently active, and at most 2 for all possible hosts within
<tt>TORONTO.EDU</tt>. If `<tt>&dollar;host</tt>' is mentioned in the command 
specification, the transport agent will only be told about the message 
control files that indicate SMTP delivery to a particular host. The actual 
host is picked at random from the current choices, to avoid systematic
errors leading to a deadlock of any queue. 


<sect2>Resource Management
<p>

For resource management there are following configuration attributes: 

<descrip>
<tag/<tt>maxta=nnn</tt>/ Max number of transporter processes under 
the <em>scheduler</em>. 
<tag/<tt>maxchannel=nnn</tt>/ Max number of processes with this same 
`<tt>channel</tt>'. 
<tag/<tt>maxring=nnn</tt>/ Max number of processes with this set of 
threads. 
<tag/<tt>idlemax=time</tt>/ How long the non-active (idle) transporter 
processes are allowed to keep around. 
<tag/<tt>overfeed=nnn</tt>/ Max number of tasks to feed from the 
thread to the transporter agent when feeding jobs to it. The 
<em>scheduler</em> main-loop at the <tt>mux()</tt> is a bit sluggish, 
thus with this we can keep the transporters busy. 
<tag/<tt>ageorder</tt>/ Default is to randomize the order of tasks 
at the queue, when it is started, with this the order is that of 
the original spool-file MTIME. Oldest first. 
</descrip>


<sect2>mailq
<p>

Upon  accepting	a  TCP	connection on the MAILQ port (TCP
port 174), the scheduler dumps data to the TCP  stream  in
the  following  format  and immediately closes the connec-
tion:

The TCP stream syntax is:
<tscreen><verb>
version id\n
data in id-dependent format<close>
</verb></tscreen>

The first line (all bytes up to	an ASCII LF character,
octal 12) is used to identify the syntax of all bytes fol-
lowing the line terminator LF.  The first 8 characters  of
the first line are ``version'' as a check that this is
indeed a MAILQ port server that has been reached, the
remaining  bytes are the real data format identification.
The data is interpreted according to that format until the
terminating connection close.

Format identifiers should be registered with the author.
The only one currently defined is <tt>zmailer 1.0</tt>. For that
data format, the syntax of the data following the first LF
is:
<tscreen><verb>
Vertices:\n
(<key>:\t><msg-file>\t><n-addrs>; <off1>(,<offN>)*\t>[#<text>]\n)*
(Channels:\n
(<word>:\t>(><key>)+\n)+
Hosts:\n
(<word>:\t>(><key>)+\n)+)?
</verb></tscreen>
Where:
<descrip>
<tag/<verb>\n</verb>/ is an ASCII linefeed
<tag/<verb>\t</verb>/ is an ASCII tab
<tag/key/ is an unsigned decimal number
<tag/msg-file/ is a contiguous string (it is the message file name relative 
to a known directory)
<tag/n-addrs/ is an unsigned decimal number (number of addresses)
<tag/off1...offN/ are unsigned decimal numbers (address byte offsets)
<tag/text/ is a string not containing an ASCII linefeed (status message)
<tag/word/ is a contiguous string (a ``contiguous string'' is a sequence of 
printable non-space characters
</descrip>
For example, here is sample output from connecting to  the
MAILQ port:
<tscreen><verb>
version zmailer 1.0
Vertices:
311424:37141; 116
311680:64722; 151,331#128.100.8.4: Null read! (will retry)
312192:63471; 152#128.89.0.93: connect: Connection timed out (will retry)
Channels:
smtp:>311424>311680>312192
Hosts:
scg.toronto.edu:>311424
mv04.ecf.toronto.edu:>311680
relay1.cs.net:>312192
</verb></tscreen>

This is sufficient information to be able to reconstruct
the transport queues as seen by the scheduler process, and
to find more information than what is shown here by 
actually  looking  up  the  message	control	 and  data  files
referred to.

<sect2>Security Issues
<p>Text to be inserted here.














