Known/suspected bugs/Missing Features on current ZMailer sources

"zmailer newfqdnroute" runs wrong thing... (no thing, actually..)

contrib/zmailcheck  -script   vs.  mailq output ?
	Fix the first, or both.

FEATURE: smtpserver ratetracker subsystem:
	- UDP unicast talking of events to configured list of peers
	  (for server cluster state tracking) (or TCP mesh of sockets ?)
	- Track HELO/EHLO greeting text changes per client IP
	- Track MAIL FROM address changes per client IP
	- Track user authentication userid changes per client IP (?)
	- For authenticated users, track:
	  - Changes on HELO parameters
	  - MAIL FROM address changes
	  - Message counts
	  - RCPT counts
	  ... etc ...
	(2006-1-30: half-way thru, need: config stuff, number of FIXMEs,
	including peer authentication handshakes, actual message xmit to
	peers, and message processing at reception.)

VERIFY: router/daemonsub:
	the queue in router is trivial (run in any|age-order, unless bounced
	around due to e.g. router crashing, which puts the thing at the tail
	of the queue --> some messages go thru, but some keep crashing the
	system...)
VERIFY: router/daemonsub.c around line 1470 the routerchilds[]
	dirq[] allocation compared with  nrouters  setting..
	What happens, when   nrouters  is smaller, than the
	number of processes needed for ROUTERDIRS ?
	How will  NOTIFY  messages affect the job pickup ?

VERIFY:  smtpserver: 
	hunting down odd corner situations, where:
	- domain X isn't listed in 'localnames' file
	- domain X does not have MXes, but has A/AAAA matching our IP
	- domain X has MXes  with lowest being ours
	( The 'domain does not have MX data' case works ok,
	  the 'domain has MX data' case isn't fully verified! )

sleepycat db integration:
   - Support RPC client mode!
     - Observe new error modes to creep up everywhere!
     - Documentation speaks of 'CLIENT*' structure, which
       is  <rpc/clnt.h>

smtp-ta: TLS mode demands a bit too much of remote server's certificates..
	 How to relax it more ?

VERIFY:  router/rfc822hdrs.c line 720 onwards "NillHeaderSemantics"
	processing.  Can it break headers sometimes ?
	What about line 655 onwards ?  Earlier in that routine ?


Wanted done for 3.0:

  - router  aliases.cf  to have similar 'protocols' feature as
    domain routing has currently

  - WEB-based configuration/administration interface
	Experimental stuff for queue watching is in  utils/webtools/
	(2003-May)

  - scheduler to kill message/thread online without a need to
    run  "manual-expirer"

  - scheduler to be able to "reroute" given channel/host, e.g.
    push the message recipient(s) back to router
	manual-version exists (2003-May)

  - A work-over of the documentation



check: Is router/db.c::db() now consistent at  argv1  construction ??

check: scheduler.conf: command=  $channel  expansion ??
	2.99.52-pre3  has apparently some problems!

proto/newdbprocessor.in:
	Support  '-i -t ordered,$MAILVAR/db/realdata.dat -f ... '

- Everywhere:  Spell-check  (propably -> probably, among others)

- "Overview" -- revise at least the list subscription instructions;
		verify also other document files

- BUG(security sensitivity):
  - "zmailer newdb" script will run "newdb" database compilations
    without becoming same uid+gid as the source file is (and with
    matching umask) -- perhaps the current PERL script must be turned
    into e.g. C program ? 
- FEATURE:
  - Also, how about enabling non-privileged users to execute "zmailer newdb"
    to recompile their own databases even though they won't be able to
    recreate all dbases and their hook .zmsh scripts ?
  - Support 'compile only those needing it' thing in "zmailer newdb", e.g.
    when source file is newer, than destination...
  - Don't overwrite produced .zmsh files in "zmailer newdb", when result
    has not changed at all..

INSTALL:
	Needs further updates (2003-May)

router:

    FEATURE:
	A way to generate new arbitrary headers (for e.g. router
	integrated spamassassin..)

    FEATURE:
	router/rfc822.c:  sequencer()
	Consider providing some data/function/whatever telling to
	the configuration script how many "Received:" headers there
	are so that the message can be trapped for a loop-prevention
	in the script, and not hardwired in the sequencer() code.
	(Now has a global variable that gets the count of "Received:"
	 headers, but the loop-count exceeded trap is done in the
	 same C-code without calling the scripts...)

    FEATURE:
	List expansions should not relay DSN data thru as is, but rather
	handle NOTIFY=SUCCESS locally ("expanded"), and then continue
	with whatever fancy the thing may want..

	Error address pickup  ( router/rfc822.c: erraddress() )
	picks *bad* choices sometimes; e.g. it might consider
	'Sender:' or 'Errors-To:' headers as good picks for the
	address where to send the report to...   It _really_ should
	prefer eFrom address.

	RFC 822 special bug(?):  '%' is a special, and/or combination
	of   "From: foo%bar <fii@faa.fuu>"  formats causes the scanner
	to react mistakenly with 'Illegal-Object:'.

	The BIND subset might not handle the 'bigmx.zmailer.org'
	correctly -- or at all..


transports/smtp:

   NUISANCE:
	In EZMLM mode: if some recipient is bounced, ones following it
	seem to be getting rather spurious looking errors. (case hotmail.com
	giving occasional 'transaction failed').   Possibly semi-generic
	problems the EZMLM-rcpt-splitting is a subcase in larger code.
	Possibly also old code running...  Updated the system, and begun
	to watch for the phenomena...
	2003-02-18: Actually this is MS Exchange 2000 bug
	            in BDAT implementation.
        --> microsoft fixes their bugs ?
	--> optionally disable BDAT when addressing some targets ?

   VERIFY:
	That failed connection to site A won't lead to recipient
	diagnostic() calls before all possible MXes/addresses have been
	gone thru for connection attempts.

   BUGLET:
	Right now we make eminently sensible commentary diagnostics carrying
	individual recipient addresses, but how shall we do diagnostics when
	e.g. DATA/dot phase ends up on violation ?  Now we get the *last*
	recipient address before DATA, not them all..


   BUG:	If reply waiting in smtp_sync() gets SIGALRM, program crashes..
	(syncing for BDAT, for example.)  (not anymore ?)

   VERIFY: BDAT (and DATA) status processing in case the remote
	yields disk-full temporary error.

   FEATURE:
	To rewrite the MX processing in streamlined flashion:
	  In case two domains have common (sub-)set of MX servers, and
	  we have been feeding mail to domain A thru MX-server B, and
	  now we change to deliver domain C which too has MX-server B.
	  We are to reuse the already open channel to server B to feed
	  the message out even though the domain C might have had server
	  D with lower preferrence..
	  (Noting that all MX-hosts with same or larger privilege,
	   than ourselves are to be pre-removed from the list!)
	  sendmail does this by observing that target domains A and B
	  have same hosts in same preferrence order (*not* pref. value!)
	  and then considering them as one.  Some MTAs choose to use even
	  finger-grained parallellism -- if connection is active to server
	  address N, and it is MX for several domains, all those domains
	  are routed via it.
	Worthwhileness of this is *not* unambiguous.
	How many domains really are served by same system ?
	At some places a lot, but elsewere perhaps not...

   BUG?
	smtp TA to ETRN type destinations (usually not reachable) work
	poorly ever since non-blocking connection was taken into use.
	(cured at 2.99.52 ??)


- Somewhere ? (smtpserver, and perhaps smtp transport agent ?)
	Implement OUTBOUND message size limit mechanism.
	Possibly a multi-value system which works against
	IP/reversed domain information, and adjust maximum
	inbound message size at the smtpserver per that
	given server.

	A further complication is need to limit outbound
	email's sizes when the inbound size is limited so
	that our system will not send out anything to which
	error report size is larger than our inbound limit.

	Problem being, of course, that outbound channels
	need to interrogate that same configuration mechanism
	to do "Sorry, this is too large for sending" processing.

- All transport agents:
	- Unify the content MIME processing routines;  Construct them
	  to be a common sub-layer ?
	- Do not decode Q-P or Base-64 if the resulting material exceeds
	  RFC-821/RFC-822 line length limits (or whatever limits have
	  been set).
	- Track the memory leakage on long-lifetime TA agents
	  (at least  mailbox  appears to leak something! 2002-Dec)
	   (A fully generic leakage in mailbox  putmail() - check others)

	(router + TA library + TA programs):
	Router to scan message MIME structure, and to store that
	into dense concise format which doesn't need prerescan of
	the message *body* for every delivery attempt in case some
	"in-flight" data conversions are desired.
	(2.99.52patch2 prepared for it: _CF_MIMESTRUCT block defined
	 for the TA-specfile.)

	- transporters: When decoding MIME-QP the final line MAY be of
		".....=", which means there is no final NEWLINE.
	Fixed on SMTP, must fix on MAILBOX, and SM!  (4-Jan-95)
	(Prepared to do it, however only on SMTP it is fatal -- I think..)

- 'channel error' detection is partial at places;
	transporters: hold, errormail


- scheduler:
    FEATURE/BUG:
	When a transport-agent dies due to e.g. SIGSEGV, simulate
	a diagnostic message telling of this situation!

    BUG?
	Sometimes will generate spurious successfull-delivery-
	reports when the command is retried, and the tag selector
	is unable to start the real processor.
		(quite old entry -- 1997-1998 ?)

    FEATURE:
	Tool/mechanism for requeueing a message from the transport
	back to the router -- all message recipients that have not
	been processed yet.  (reroute/manual-reroute: 2002/3 - 2003/4)
	(SCHEDULER thingie missing!)

    FEATURE:
	Tool/mechanism for the scheduler to command expiration of all
	recipients in given thread.  (alike 'manual-expiry', except
	a bit smarter..)

    FEATURE:
	- Delivery messages plain-text part needs easier human
	  understandable format on the messages.

    FEATURE:
	Reports about delivery DELAY.

    FEATURE: ??
	Place the scheduler's PARAMmailqpath= data into ZENV variable,
	so that mailq tool can share it.  For use in systems with
	multiple ZMailer instances.

    BUG?
	Have two waiting threads (one message each):
	- start both of them with ETRN
	- both issue report of "retryat +nn"
	- both are rescheduled about immediately..
	- system spins (FAST) on those two...
		(Fixed ?  Haven't seen this for a while)

    FEATURE:
	- Another:  maxlife  -- maximum lifetime for a transport agent
	  for limiting the time a smtp-channel stays open for a long-
	  living connection...


- smtpserver

    BUG:
	Ugh ?  Are we producing invalid ORCPT data, or are we rejecting
	valid ?  Or is just  mail3.centrum.cz  producing invalid one ?
	They have ZMailers from 1999 ..  what the exact version might be.

WIirGqB0000r    MAIL From:<> BODY=8BITMIME SIZE=4679
WIirGqB0000w    250 2.0.0 Ok (sourcechannel 'error' accepted) Ok
WIirGqB0000#    000--- pipeline input exists 1268 bytes
WIirGqB0000r    RCPT To:<linux-kernel-owner+jan.slezak=40centrum.cz@vger.kernel.org> ORCPT=rfc822;linux-kernel-owner+jan.slezak=40centrum.cz@vger.kernel.org
WIirGqB0000w    501-4.5.4 Invalid ORCPT value 'ORCPT=rfc822;linux-kernel-owner+jan.slezak=40centrum.cz@vger.kernel.org'
WIirGqB0000w    501 4.5.4 ORCPT-param data error!
WIirGqB0000#    000--- pipeline input exists 1252 bytes
WIirGqB0000r    BDAT 4649 LAST
WIirGqB0000w    503 5.5.2 Waiting for RCPT command


    FEATURE(?): Verify that MAIL FROM MX data resolves to valid
	IP addresses, and does not contain loopback.

    BUG:
	Control somehow if lack of configuration and/or PolicyDB is
	a system security offence, or not..

    BUG?:
        Timeout control on network socket writes ???

  - VERIFY: behaviour on BDAT (and DATA) phases when the disk fills
  -	Report that if the router subprocess crashes, interactive
	processing goes into mixed state -- it reports 2XX, and
	yet claims to be in wrong state (MAIL FROM or RCPT TO ?
	I have forgotten..)

    FEATURE:
        Script-controlled policy-testing -- without having ROUTER around ?
	Suggestion: If some *users* need RBL testing, do it for them, not
	for *all* addresses.
	(Script to do test order, choices, and reactions.)

	Counter suggestion: Delay the lookup until the somebody uses the data
	Refutation:         Needs storing connecting IP address somewhere

    FEATURE:
	accept email to postmaster(s), never mind what else policy testers
	have determined...  ( ==> must always accept MAIL FROM, no matter
	what;  ==> MAIL FROM rejects to be given on RCPT TOs...)

    BUG:
	"-s" option processing is entirely wrong...

    FEATURE:
	The connection source IP reversal is not verified paranoidly, we
	may accept people who claim dishonest reversals..  (But then we
	can test against their IP address..)

    FEATURE:
	Have some flag passing mechanism from the initial policy db
	back to the smtpserver proper so that it can:
	- give more meaningfull explanations of rejection reason
	- run more verification subroutines on addresses per policy
	  driver control; namely  "this domain can be valid source only
				   when it is coming from that IP address"
	  type of checkup...


- Integrate transports/fuzzyalias/ into the configuration system

- Manuals leave a lot to desire.. Especially a good users/administrators
  manual is in need
	[July 1997: technical writers are hard at work on this one]
	[April 1998: Completing the work is at Matti's hands..]


- transports/sm/sm.c:
	- What environment variables are set, and how ? (+ man-page!)
	- Support ESMTP + DELIVERBY (RFC 2852)
	- what UID it runs programs like procmail with ?
	  What uid it should run them ?
	  I don't think the current (up to 2.99.49p*) way of running
	  procmail/cyrus is entirely safe.
	- I have gotten comments that they are suid/sgid root/root
	  at most installations, and indeed so procmail is at my
	  workstation too...
	- DSN handling flags ?
	- "localize" the destination address properly.  That is, strip
	  quotes from around a quoted destination address according to
	  the RFC822 syntax. -- Is this always correct ?


   VERIFY:
	pass the address meta information also in environment
	variables, and handle multirecipient cases as well!
	(see what 'mailbox' does.)


- post-install-check
	- check existence of mailq/tcp in /etc/services
	- check MAILVAR/MAILSHARE/POSTOFFICE protections
	- check "nobody" and "daemon" accounts

- systemwide .forward checker
	- iterate all users in the system
	- check peoples directory ownerships and permissions
	- check peoples .forward ownerships and permissions

- Need a program to verify that given configuration is ok, checks
  for things like:
	- "nobody" userid
	- scheduler's resource control working to the maximum
	  extend that system supports
	- mmap() correct operation  (GNU autoconf does this)
	- sprintf() return type is autoconfigured correctly

- Need a program to run thru various file permission checks:
	- $MAILVAR/db/ -dir, and files in it
	- $MAILVAR/lists/ -dir, and files in it
	- ~/.forward -files (and home directories)

- IPv6 stuff:
	- lib/selfaddrs.c: Don't properly do automatic IPv6 interface address
			   picking (IPv4 is ok)
	- libc/myhostname.c: don't do automatic IPv6
	- transports/hold/hold.c: incomplete NS/AAAA, no IPv6 PTR
	- transports/mailbox/mailbox.c: BIFF gethostbyname() ?

- whole chain (smtpserver/router/mailbox+hold):
	- Have a NEW pseudoheader (NRCPT) in addition to the ORCPT, which
	  we now (in not completely kosher manner) create, if we don't get
	  it originally.  That is, similar entry for traversing unchanged
	  thru the router to tell what address we got as input.
	  (Thru some pseudo-alias..)  -- HAVE.
	- Retain (in some TA-header object) the original "MAIL FROM:<..>"
	  address, and be able to store it into the MAILBOX channel.
	  (That is, don't let the routing process to alter this!)

- transports/mailbox/mailbox.c:
	- When doing delivery into a pipe, give  MAIL  variable to
	  tell what would be this user's mailbox ?



- Autoconfig problems:
	- System mailbox locking schemes are sometimes non-obvious..
	- For the last PERHAPS must back introduce the host-configuring
	  for describing host-system dependencies like mailbox-lock schemes

- Got mail with a suggestion: (via zmailer-list)

 I suggest you to improve the 'crossbar' function of 'router'.
 It might return a list of header rewrite functions as well
 as a single function like now. So different rewrite methods
 could be applied for sender and recipient addresses.
 The crossbar() C function contains some comment that refers
 to some similar thing, but I found it unimplemented.
         /*    
          * We expect to see something like
          * (rewrite (fc fh fu) (tc th tu)) or
          * ((address-rewrite header-rewrite) (fc fh fu) (tc th tu))
          * back from the crossbar function.
          */

  5-Dec-1999: Reading the code, actually it *is* implemented.
	      ... or perhaps not, *usage* of that double-rewriter
		  form isn't done, existing code has always expected
		  single rewriter address, not two elt list.

- When killing previous routers/scheduler/smtpserver, should wait
  the previous process group leader to die before writing over
  the  $POSTOFFICE/.pid.KIND -file.

- Sometimes incoming SMTP can be a hellish load, need to introduce
  a load-limited incoming SMTP acceptor (smtpserver/smtpserver.c)
  (Rayan wrote it, but never released it..)
	13-Dec-94:  It exists for Linux ....
	23-Dec-94:  Pulled bits from "top" -program.  Now it
		    exists also for Suns..
        01-May-04:  Other approaches have made this unnecessary.
                    System does lots more, but things are lot less
                    weightly..

  A discussion on the ZMailer -list revealed that the acceptance blocking
  is a BIG can of worms, and that even the  BSD-sendmail has had a long
  journey along the rocky path to "get it right"...  (5-Jan-95)

  Hmm.. Perhaps we could do a `single process doing multi-stream reception ?'
  it would require fairly large rewrite of the SMTP-server..

  Also: What to do when there are more incoming SMTP sessions than the
  process can have open file descriptors ?  One for the SMTP socket, one
  for each spool file in active use at the input phase, and one for log.
  (The stdio is used only for SMTP responses, and for spooling out the
   accumulated message.  Thus there SHOULD be enough resources for all
   uses -- except when the system runs out of FDs per any individual
   process..)
