Mail::Toaster FAQ

- What features does the Toaster support?
- What programs will be installed?
- What are the mail retrieval options?
- How do I manage mail options for a domain?
- Where do I find the maildomain script?
- How do I convert Multilog date stamps to human readable date stamps?
- How do I find statistics about my mail queue:
- How do I enable the sqwebmail calendar?
- When will you add support for my platform?
- How do I block connections from servers with no reverse DNS?
- How do I use SpamAssassin & ClamAV?
- Whats the best way to view/expire messages from the queue?
- Why are IMAP connections getting reset?
- How do I delete old messages in the Spam folders?
- Where is the UCSPI-TCP patch?
- How can I modify the Sqwebmail interface?
- How can I modify the Qmailadmin interface?
- What httpd.conf changes are needed?
- Can you explain the Anti-Spam options please?
- Is portupgrade -a safe?
- Why does SpamAssassin take so long to check messages?
- How do I set up per-user SpamAssassin preferences?
- What needs to be in /etc/rc.conf?
- Should I use RBLs?
- How do I get QmailScanner Statistics to work?

# What features does the Toaster support?

      SMTP Mail Server (SMTP-AUTH, TLS, tarpitting, RBL)
        http://www.tnpi.biz/internet/mail/toaster/patches/
      Virtual Domain Hosting w/delegated administration
      Virtual Domain Users
      Mailing List (Ezmlm)
      AutoResponder
      Web Based E-Mail (multiple options)
      Web Based Domain Administration
      Mail Filtering (rblsmtpd, SpamAssassin, Qmail-Scanner)
      SMTP roaming via SMTP-AUTH, POP-AUTH, & IMAP-AUTH
      POP3, POP3-SSL
      IMAP, IMAP-SSL

# What programs will be installed?

      Qmail 1.03   : Secure, fast, and reliable MTA
      MySQL (3,4)  : Database engine
      ucspi-tcp    : Inetd replacement
      daemontools  : Tools for managing daemons
      ezmlm-idx    : Mailing list management
      Apache (1,2) : Secure HTTP server
      Apache mods  : mod_php, mod_ssl, mod_perl
      OpenLDAP     : LDAP client
      vpopmail     : Virtual Domain module
      qmailadmin   : Vpopmail domain administration CGI
      autorespond  : Auto Responder & vacation
      gdbm         : Database routines
      ispell       : Spell checking for web mail
      qmailadmin   : Web Control Panel Interface
      sqwebmail    : Web based email
      squirrelmail : Web based email
      courier-IMAP : IMAP & POP3 servers
      maildrop     : procmail like mail deliver agent
      spamassassin : spam filtering program
      clamAV       : Virus scanning software
      qmailscanner : content scanner for qmail
      maillogs     : mail log processor
      rrdutil      : MRTG like graphs
      Mail::Toaster: Perl modules for mail systems
      MATT::Bundle : Perl modules for unix systems

# What are the mail retrieval options?

      pop3://mail.example.com
      imap://mail.example.com
      pop3s://mail.example.com
      imaps://mail.example.com

      http://mail.example.com/cgi-bin/sqwebmail
      https://mail.example.com/cgi-bin/sqwebmail

      http://mail.example.com/squirrelmail
      https://mail.example.com/squirrelmail

Mail::Toaster::CGI showcases the toasters functionality. See it live at: http://mail.cadillac.net/

# How do I manage mail options for a domain?

Point a web broswer at https://mail.example.com/cgi-bin/qmailadmin

# Where do I find the maildomain script?

maildomain perl script - Front end to nearly everything you can do with the vpopmail programs and a bunch of handy fixup utilities for qmail systems. http://localhost.tnpi.biz/internet/mail/toaster/maildomain/maildomain.pl

UPDATE: This is now part of Mail::Toaster and has been renamed mailadmin.

# How do I convert Multilog date stamps to human readable date stamps?

      tai64nlocal < logfile > qmaillog.tmp

# How do I find statistics about my mail queue:

      1. qmail queue | more

      2. qqtool -a list

      3. rrdutil

      4. isoqlog

      5. Manually feed qmail-send output through matchup and pipe
        the results through a processor (all one one line):

         /usr/local/qmailanalog/bin/matchup <
         /var/log/mail/send/current |
         /usr/local/qmailanalog/bin/z*

        where z* is one of the following:

        zddist, zdeferrals, zfailures, zoverall, zrecipients,
        zrhosts, zrxdelay, zsenders, zsuccesses, zsuids

# How do I enable the sqwebmail calendar?

echo ``local'' &gt; /usr/local/share/sqwebmail/calendarmode

# When will you add support for my platform?

The scripts are extensible in anticipation of using them on platforms other than FreeBSD where they were developed. My next ``supported'' OS will likely be Mac OS X and then probably Linux. Linux fans might want to check out Bill Shupp's toaster pages. Here's a couple URLs for you: http://www.shupp.org/toaster/ and http://qmailtoaster.clikka.com/.

# How do I block connections from servers with no reverse DNS?

Contents of ~vpopmail/etc/tcp.smtp (fix wrapped lines):

     127.:allow,RELAYCLIENT=""
     209.218.8.2:allow
     =:allow
     :allow,RBLSMTPD="Blocked - Reverse DNS queries for
       your IP fail. You cannot send me mail."
     #:allow,RBLSMTPD="-Blocked - Reverse DNS queries for
       your IP fail. You cannot send me mail."

     line 1. Obvious, allows localhost to relay.
     line 2. Allows traffic from the inter7 email list
     line 3. Matches any mail message with reverse dns.

line 4. Matches what's left (no reverse DNS). By setting the RBLSMTPD environment variable, we actually get to pass a message back to the mail server we're blocking telling them why. That gives them a chance to fix it before the messages bounces.

They'll get a message like this in their mail logs:

Mar 27 08:40:43 seattle qmail: 1048783243.397888 info msg 6469: bytes 258 from qp 13226 uid 0 Mar 27 08:40:43 seattle qmail: 1048783243.438981 starting delivery 533: msg 6469 to remote matt@simerson.net Mar 27 08:40:43 seattle qmail: 1048783243.979048 delivery 533: deferral: 207.89.154.94_does_not_like_recipient./ Remote_host_said:_451_Blocked_ Reverse_DNS_queries_for_your_IP_failed. You_cannot_send_me_mail. Giving_up_on_207.89.154.94./

line 5. If you want to be a little more aggressive about it, use the 5th line instead of the fourth. Notice the '-' character in there. That tells rblsmtpd to return a permanent error (ie, don't try again!)

Mar 27 08:42:40 seattle qmail: 1048783360.776812 info msg 6475: bytes 250 from qp 13464 uid 0 Mar 27 08:42:40 seattle qmail: 1048783360.805534 starting delivery 534: msg 6475 to remote matt@simerson.net Mar 27 08:42:41 seattle qmail: 1048783361.259737 delivery 534: failure: 207.89.154.94_does_not_like_recipient. Remote_host_said:_553_Blocked_-_Reverse_DNS_queries_for_your_IP_fail. You_cannot_send_me_mail. Giving_up_on_207.89.154.94./ Mar 27 08:42:41 seattle qmail: 1048783361.269637 bounce msg 6475 qp 13467 Mar 27 08:42:41 seattle qmail: 1048783361.270564 end msg 6475

Notice that in the second case, the message bounces immediately. It's your mail server, you have to decide what policy you think is best. Bouncing messages seems to get more attention, and gets it faster than deferring connections.

That's all there is to it. Of course, that assumes you are running rblsmtpd as part of your smtp invocation.

# How do I use SpamAssassin & ClamAV?

See the Filtering section at http://www.tnpi.biz/internet/mail/toaster/filter.shtml

# Whats the best way to view/expire messages from the queue?

Use qqtool: http://www.tnpi.biz/internet/mail/qqtool/

# Why are IMAP connections getting reset?

See this post to the mailing list for a possible explanation: http://matt.simerson.net/cgi-bin/ezmlm.cgi?mss:2006:pnonldigcpgkfpehlcgo

# How do I delete old messages in the Spam folders?

find /usr/local/vpopmail/domains/*/*/Maildir/.SPAM/cur -type f -mtime +30 -exec rm -f {} \;

deletes files older than 30d

no guarantee, that just came off from the top of my head :) haven't tested, but worth the try :D regards, Andryan.

UPDATE: Matt thinks is a partial solution, as it won't find files within numbered buckets which you'll find on any domains with a large number of users.

# Where is the UCSPI-TCP patch?

http://www.tnpi.biz/internet/mail/toaster/patches/tcpserver-mysql.shtml

The patch adds MySQL query support to tcpserver. The patch will allow tcpserver to query a Mysql server for each new connection and set RELAYCLIENT based on the presence of the remote IP in the table. This is a very useful and popular patch.

# How can I modify the Sqwebmail interface?

The html temples live in /usr/local/share/sqwebmail/html/

# How can I modify the Qmailadmin interface?

The html templates live in /usr/local/share/qmailadmin/html/

# What httpd.conf changes are needed?

See contrib/httpd.conf-2.0.diff in the Mail::Toaster distribution.

# Can you explain the Anti-Spam options please?

There are several anti-spam mechanisms in place. The first is RBLs.

  Layer 1: RBLs

  RBL's that deny SMTP connections to your server have no concept of email addresses. They only know whether the remote address is a blacklisted or not. So, if someone is blacklisted, you only have two choices:

         a) use the blacklist and deny connections
         b) use the blacklist and whitelist specific IP's
         c) don't use the blacklist

  I choose b. I strongly desire to block volumes of crap before my SMTP server has to deal with it. In the few cases where a remote server is misconfigured and gets blacklisted, I have no problems being one of many servers they can't send mail to until they get it fixed. This is a form of policing the Internet and making careless or just plain naughty mail server owners become good netizens.

  If a RBL is unjustly blocking too much for your organizations policies, you'll want to disable it.

  Layer #2: Patches to qmail-smtpd.

  We block invalid users (based on qmail-chk-usr patch, and have the badmailfrom, badrcptto, and other patches which you'll need to read up on to understand how to use. However, there isn't (as far as I know) any whitelisting. Even if there was, I wouldn't consider it beneficial.

  Layer #3: Qmail-scanner.

  If you've chosen to use it. It can filter based on external programs (ClamAV, SpamAssassin, etc) as well as things like attachment extensions, keywords, etc. It's very powerful, but has substantial resource requirements so use it with care.

  The aforementioned layers are at the system level and settings there apply to your entire mail server. If you must employ different policies for different users, then you must do one of the following:

        a) run multiple SMTP servers
           (on unique IPs) with different policies
        b) choose a "lowest common denominator" policy

  The final layer of protection is at the mailbox level. This is where the user gets to opt into using SpamAssassin, and routing suspected spam into a .Spam folder. If there is to be user selectable preferences, this is where they belong, and eventually I'll work in some Squirrelmail -> SpamAssassin integration so users can tweak their preferences.

  There are other mechanisms available for fighting spam but none that are easily implemented or that don't have other nasty side affects.

  Layer 4: Maildrop

  Maildrop is much like procmail and is used for the same reasons. After the message has passed through all the checks above, we can do some very sophisticated processing of the message. For example, the default mailfilter I wrote and include with the toaster does the following:

       1. process messages < 256k through SpamAssassin
       2. Process any user specific rules
       3. verify users mail quotas
       4. delete any message with a spam score > 12
       5. Deliver spam message to Spam folder
       6. bounce message if Mailbox is over quota

You have some very powerful tools included with the Mail::Toaster that will let you deal with each message in any fashion that you'd like. The only limits are your knowledge of the available tools and you are encouraged to learn them.

# Is portupgrade -a safe?

Simple answer: No.

Long answer. I never recommend portupgrade -a on a production server. There are a variety of reasons for this:

  Broken Binaries

  Read the portupgrade man page for details.

  Overwrites custom binaries

  The toaster setup installs some customized binaries such as the heavily modified qmail package. If you let portupgrade ``update'' qmail, then the special anti-spam features go away. Ouch.

  Does not honor make options

  You'll note that toaster-watcher.conf has lots of options for configuration how certain toaster components are installed. We'll take MySQL for example. You might want it installed with ssl or linuxthreads. The toaster setup script honors those settings, but after portupgrade ``updates'' your MySQL, those special features are gone.

Because of the aforementioned reasons, many folks end up breaking their mail systems after running portupgrade. The best advise is have a development server to test on first. If it doesn't break your dev server, then it'll probably work find on your production one. Be very careful updating any programs installed via toaster_setup.pl. If you do upgrade something and break your toaster, run through the setup again, and it should repair the damage.

# Why does SpamAssassin take so long to check messages?

First, you need to understand what a ``normal'' processing time is. On my server, I do DCC, Pyzor, Razor2, and the rest of the ``normal'' checks. If I have RBL checking on, then processing a message takes a couple seconds. If I disable RBL checks, messages processing drops to less than a second. This is on a pretty old server (Dual PIII 650) with 1GB of RAM.

If your processing time is extraordinarily different, That could be one of several reasons:

  DNS timeouts

  By default, SpamAssassin does RBL lookups against several blacklists. Make sure your DNS works well. If you are doing RBL checks via SMTP, then you may want to disable the RBL lookups in SpamAssassin. You can do that by adding ``skip_rbl_checks 1'' to /usr/local/etc/mail/spamassassin/local.cf.

  If a RBL that SA is trying to contact is down, it may also introduce an extended timeout. Disabling the RBL checks is a good way to diagnose this. I run all the RBL checks at the SMTP level so I can deny the messages and thus avoid having to process them. As such, it makes little sense to have SA also running RBL checks.

  Firewall blocking

  SpamAssassin can use DCC, Razor2, and Pyzor. Those modules require contacting network servers. If your firewall rules are blocking those connections, this will introduce a timeout that hangs SA for 5 seconds per check. So, if your message processing time takes 11 seconds and it should be taking about 1, then you are likely blocking a couple checks. I added the following rules to my IPFW firewall:

       # Allow DCC (6277) & Pyzor (24441)
       ${fwcmd} add allow udp from ${oip} to any 6277,24441
       ${fwcmd} add allow udp from any 6277,24441 to ${oip} 1024-65535

# How do I set up per-user SpamAssassin preferences?

I did it as follows:

  Create the MySQL database

  http://eu.spamassassin.org/full/2.6x/dist/sql/README

  I named my database ``spamassassin'' and the table ``userpref'' as shown in the SA docs. The table name isn't significant, just make sure you are consistent in how you configure SA and sasql's access to it.

  Create a MySQL user and password for access to the spamassassin database. I did so with a grant such as: GRANT ALL PRIVILEGES ON spamassassin.* TO spamassassin@'locahost' IDENTIFIED BY 'secretword';

  Install SASQL

  http://www.squirrelmail.org/plugin_view.php?id=167

  Install it as you would any other squirrelmail plugin. Edit sasql_conf.php and set the DSN.

  Install WebUserPrefs

  http://www.pipegrep.net/webuserprefs/

  I did it as follows:

       cd /usr/local/www/mail
       fetch http://...
       tar -xzf webuserprefs-0.5.tar.gz
       mv webuserprefs-0.5 webuserprefs
       cd webuserprefs
       vi config.php

  Edit the prefs_source to ``db'', authorization to ``squirrelmail'' and set the database info. Point your browser at http://mail.example.com/webuserprefs/ and volia. The catch is that you must be logged in via squirrelmail in order to use it. The alternate solution is to use IMAP or POP3 but then you have to recompile PHP with IMAP support.

  Install Pear-DB

  You'll find it in /usr/ports/databases/pear-DB.

  I had mod_php4 installed before and this silly port was convinced it needed the lang/php4 port installed. Since I had another need for the PHP cli, I honored it's insistence by deinstalling www/mod_php4.

  Modify spamd's flags. The best way is to modify /etc/rc.conf and add the following lines:

        spamd_enable="YES"
        spamd_flags="-a -d -v -q -x -r /var/run/spamd.pid"

# What needs to be in /etc/rc.conf?

Here's the relevant portions of mine:

      enable_quotas="YES"
      check_quotas="YES"
      clamav_clamd_enable="YES"
      spamd_enable="YES"
      spamd_flags="-a -d -v -q -x -r /var/run/spamd.pid"
      ntpdate="YES"
      xntpd_enable="YES"
      xntpd_flags="-p /var/run/ntpd.pid"
      sendmail_enable="NONE"
      snmpd_enable="YES"

# Should I use RBLs?

Grab a cut of coffee, it's SMTP lesson time.

The type of statistics you'll see in your logs regarding SMTP blocks will reflect several things of interest (that are related to your question). The first, and most obvious is how you are using the RBLs.

The most important setting in that regard is rbl_enable_soft_failure in your toaster-watcher.conf file. If you have soft failure enabled, then when you deny a mail agent access, you are doing so with a temporary failure, which amounts to saying ``try again later''. The idea behind that is that if it's a legit mail server, it gives them a chance to fix their problem and then try sending again.

I'm not in that camp. My email addresses have been plastered on web pages for years and years, on mailing lists, usenet forums, etc. Thus, my `address(es)` are included in many of the spammers databases so I'm going to be a target regardless. I want to stop them dead so I set rbl_enable_soft_failure to 0, which chooses a 553 error ``sorry go away, permanent error, you spammer pig'' style. This tells the remote server, don't even bother trying again, it won't work.

Obviously with the latter arrangement, you'll see a lot fewer (than if using the former) RBL blocks. Many of the spammers don't honor the 553 error and try again anyhow but that's quite OK. The majority of mail servers (themselves victims of naughty people) will get the 553 and bounce the message, no longer trying to contact me. All is well. This is the very same reason why it's better to use a SMTP 553 error instead of simply packet filtering their IP. Sure, they're blackholed, but they'll get a SMTP unreachable error and just keep trying. I want the spam traffic destined to my server to go away, not add to the cost of my system (via connectivity). When your server has thousands of mailboxes on it, these policy decisions make very real and measurable differences.

Doing a RBL lookup on a message at SMTP invocation time is cheap. Extremely cheap. It's a simple DNS query so it's very very very lightweight. I may even have it cached from the last time the spammer tried. It costs almost nothing. If I can deny a spam message there, before having to pass it to qmail-scanner (medium cost), through the virus scanner (medium to expensive), bayesian filtering (low cost), spamassassin network tests: blacklists, DCC, pyzor, razor (low - medium), and then maildrop filtering, then you it's worth it to me.

So, reason #1: It costs me less to use RBLs.

The next reason for RBL usage is more political. The spammers revenue (that they charge advertisers) is based on message deliveries. They maintain databases of email accounts that they can deliver to. They have developed very sophisticated ways of determining if a message got delivered (well beyond ``did the SMTP server accept it'') and further, if it landed in a real persons mailbox. These include delivery notifications, embedded HTML (the web server logs then reflect success), and other clever mechanisms.

Very few Mail User Agents are very clever about dealing with spam messages. Unfortunately, the ones that are (like Mail.app) are not the most popular. A lions share of users use that icky software from Redmond which is a spammers dream come true. Even if I tag and bag the spam to a special folder, they end user may peruse through it, opening the message to verify and thus letting the spammer know the message was delivered successfully. Yay, spam 'em again!

So, if I can block the message entirely (not just from my mailbox, but from every landing on my server) then the spammer will never get back any indication of success. This increases the likelihood of being removed from the naughty miscreants list and thus reduces the likelihood of getting spammed from him again in the future.

Sounds great in theory right? So how does it play out in real life? I have quite a few ``case studies'' we'll call them, ranging from my own personal mail servers to mail systems with tens of thousands of mailboxes on them. In general, the spam to ham ratio of messages on the internet is rising, but in every case my mail servers (with RBLs), the number of RBL blocks I see is gradually decreasing. This suggests that either RBLs are becoming less effective or some other unknown `factor(s)` are at work.

So, let's look at my mail server as a case study. Over time my message volume on my server has continued to rise, so we should expect the level of spam messages as well as ham messages to be steadily increasing (or the number of RBL blocks to increase). The effectiveness of my spam filters has been fairly constant (>98%) for the past six months yet the number of spam messages being delivered on my mail server has steadily dropped. What accounts for the difference?

Reason #2: Keeping spam messages way from end users

About a year ago I was watching my mail server logs and noticed that quite a few of my mail users (of whom I know personally) were getting extraordinarily high levels of spam being delivered to them. I then decreased my spam tolerance by blocking mail servers without DNS (which required whitelisting a fair number of mail servers), increasing my RBL usage. Immediately after that the number of RBL hits spiked and then began a gradual decline ever since. This reduction of traffic is due almost entirely to the RBL usage as I tested the reverse DNS blocking at various times and ultimately removed it.

Approximately six months later I did another round of decreased spam tolerance and enabling the spam filtering for these users, all of whom are not very sophisticated mail users. They don't know or care about mail servers, filtering, etc. They just want their email to work. Overnight the levels of spam in their inboxes dropped to almost nothing (a condition I'm accustomed to living with). This, of course made them all very happy. It also gave me a way to track the effectiveness of the RBLs, as I had a very accurate indicator of their effectiveness. RBLs only caught about 50% of the spam, but that's still a remarkably good achievement for something that costs so little to employ.

So, RBLs aren't a ``one stop shopping'' spam solution, but they are a very effective tool.

Reason #3: Spammers hate RBLs

There is good reason why RBL operators are targets of DoS attacks. These days running a RBL means making sure you and your upstream provider are prepared to deal with massive DoS attacks, sometimes lasting over a week. These DoS attacks can cost many thousands of dollars in excess bandwidth fees, lost revenues, and related costs. This is the reason for several of the RBL operators decisions to step out of the game, including osirusoft which was one of, if not the most effective free RBL. If RBLs weren't effective, they wouldn't be targets.

Unfortunately, the spammers have millions of advertising dollars at stake. When a RBL is as effective as Osirusoft, it measurably effects their bottom line, and thus becomes a target. If I can hurt the spammers where it counts (in their wallet), then in my book, that's A Very Good Thing.

Matt

# How do I get QmailScanner Statistics to work?

I did as follows:

      cd /usr/local/www/data
      mkdir qss; cd qss
      download tar archive from:
        http://sourceforge.net/projects/qss/
      tar -xzf qss-2.0.2.tar.gz
      vi config.php

Change the start date to the earliest date shown in your log file (head /var/spool/qmailscan/quarantine.log). Change the logFile variable to point to ``/var/spool/qmailscan/quarantine.log''. Then open up the permissions a bit on the quarantine so qmail-scanner stats can read it:

      chmod o+x /var/spool/qmailscan
      chmod o+r /var/spool/qmailscan/quarantine.log

Qmail-Scanner changed the logging date format at version 1.20 but qmailscanner stats doesn't have a fix for it yet. If you log files with dates in this format: 23-09-2003 then it'll work great. If you are using a newer version of qmail-scanner, you'll notice that your log entries are not showing up. I have both (before/after upgrade) so I fixed qmailscanner stats it by editing the index.php as follows (The if..else block is my addition):

      vi index.php

       if ( eregi("(^[0-9]+)", $val[0]) )
       {
          # Calc the date timestamp
          $date = explode("/",$val[0]);
          $dateT = $date[0];
          $date[0] = $date[1];
          $date[1] = $dateT;
          $date = strtotime(implode("/",$date));
       }
       else
       {
          $date = strtotime ($val[0]);
       };

Point your browser at http://mail.example.com/qss/ and voila! If you can't get it working, just look at mine and pretend: https://mail.cadillac.net/qss/. ;-)