From mea@nic.funet.fi Tue Jul 14 00:57:37 1998
Received: from nic.funet.fi ([IPv6:::ffff:128.214.248.6]:15122 "EHLO nic.funet.fi" ident: "TIMEDOUT") by mea.tmt.tele.fi with ESMTP id <92217-5231>; Tue, 14 Jul 1998 00:57:27 +0300
Received: by nic.funet.fi id <20068-14900>; Tue, 14 Jul 1998 00:57:30 +0300
Received: from cal026031.student.utwente.nl ([130.89.228.161]:20560 "EHLO cal026031.student.utwente.nl" ident: "root") by nic.funet.fi with ESMTP id <19679-9853>; Mon, 13 Jul 1998 19:21:15 +0300
Received: from cal026031.student.utwente.nl ([130.89.228.161]:19792 "HELO cal026031.student.utwente.nl" ident: "mark") by cal026031.student.utwente.nl with SMTP id <156466-168>; Mon, 13 Jul 1998 18:20:49 +0200
Date:	Mon, 13 Jul 1998 18:20:48 +0200 (MET DST)
From:	Mark Visser <mark@cal026031.student.utwente.nl>
Reply-To: Mark Visser <mark@cal026031.student.utwente.nl>
To:	mea@nic.funet.fi
Subject: Re: Zmailer policybuilder
In-Reply-To: <19980712233156Z1151-3796+1544@nic.funet.fi>
Message-ID: <Pine.LNX.3.96.980713173843.6394B-100000@cal026031.student.utwente.nl>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: mea@nic.funet.fi
Return-Path: <mea@nic.funet.fi>
X-Orcpt: rfc822;mea@mea.tmt.tele.fi
Status: RO

On Mon, 13 Jul 1998 mea@nic.funet.fi wrote:

Matti,

> > 
> > I have fixed it in policybuilder.sh, and also made some adjustments, so
> > you're able to run it periodically from cron. If policybuilder goes wrong
> > with taking files, it now leaves your current policy in state, as to not
> > open up your system more then it was.
> 
>   Yes, you are right.
> I have committed something very much alike this one into the CVS tree.
> ( I just didn't like of pipe:
> 	cat file1 | slice-and-dice | sort | uniq > file1
>  I renamed the output file.  Think (or try!) when the "file1" has
>  size of - say - 100kB ?  Does this pipe do "file1" truncation before,
>  or after it has been opened for reading ? )
> 

Just did a checkout from the cvs, and I think there's still some minor
cleanup possible (yes, i'm sometimes an annoying perfectionist ;-))

A new patch. Mainly adds some documentation.

--
--- policy-builder.sh.in.orig	Mon Jul 13 17:40:09 1998
+++ policy-builder.sh.in	Mon Jul 13 18:18:34 1998
@@ -53,31 +53,45 @@
 		mv  smtp-policy.spam.new smtp-policy.spam
 	else
 	    #
-	    # Following IS NOT SAFE, if either produces errors, those
-	    # go (usually) to the result file, and in the end the result
-	    # OVERWRITES the "running"  smtp-policy.spam  file.
+	    # The following retrieves new spamlist from several locations
+	    # and then combines them.
+	    #
+	    # It lists users, and domains that are known spam sources.
+	    #
+	    # You could also use MAPS rbl ("http://maps.vix.com/rbl/")
+	    # That one is more up to date normally, but by running
+	    # this policybuilder once in a while you should get most
+	    # spammers out.
+	    #
+	    # Note that we don't have an easy method to check if everything
+	    # went correct. We do a simple check, but beware that if that
+	    # check is OK, it OVERWRITES the "running" smtp-policy.spam file.
 	    #
 	    > smtp-policy.spam.new
 	    lynx -source http://www.webeasy.com:8080/spam/spam_download_table \
-		>> smtp-policy.spam.new
+		>> smtp-policy.spam.new.tmp
 	    lynx -source http://www.sprocket.com/Security/SpamDomains | \
-		awk '{print $1}' >> smtp-policy.spam.new
-	    cat smtp-policy.spam.new | sed 's/^@//g' | tr "[A-Z]" "[a-z]" | \
-		    sort | uniq > smtp-policy.spam.new2
+		awk '{print $1}' >> smtp-policy.spam.new.tmp
+	    cat smtp-policy.spam.new.tmp | sed 's/^@//g' | tr "[A-Z]" "[a-z]" | \
+		    sort | uniq > smtp-policy.spam.new
+	    # Cyberpromo is a notorious spammer. If that one is in the
+	    # constructed file, we hope to have a correct file.
+	    #
 	    if [ `grep -c cyberpromo smtp-policy.spam.new` -gt "0" ]; then
 		mv smtp-policy.spam smtp-policy.spam.old
-		mv smtp-policy.spam.new2 smtp-policy.spam
-		rm -f smtp-policy.spam.new
+		mv smtp-policy.spam.new smtp-policy.spam
+		rm -f smtp-policy.spam.new.tmp
 	    else
 	        echo "Hmm....something went wrong while updating the spam policy."
-		echo "Please try again."
+		echo "Please investigate and try again."
  		exit 1
 	    fi
 	fi
     fi
 fi
 
-# Fork off a subshell to do it all...
+# Ok...We now have everything we need to know, let's put it together
+# We fork off a subshell to do it all...
 (
   # The basic boilerplate
   cat smtp-policy.src
@@ -101,18 +115,20 @@
     awk '/^#/{next;} NF >= 1 {printf "%s relaytarget +\n",$1;}'
   fi
 
-  # smtp-policy.spam
-  # (Lists users, and domains that are known spam sources)
-  # (We use file from "http://www.webeasy.com:8080/spam/spam_download_table"
-  #  which is intended for QMAIL, and thus needs to be edited..)
+  # smtp-policy.spam & smtp-policy.spam.manual
+  # (These lists users, and domains that are known spam sources)
+  # In smtp-policy.spam.manual you may enter additional spammers,
   if [ -f smtp-policy.spam -o -f smtp-policy.spam.manual ] ; then
     ( if [ -f smtp-policy.spam ] ; then
 	cat smtp-policy.spam
       fi
+      # unfortunately the file smtp-policy.spam.manual can contain
+      # the same hosts as smtp-policy.spam. We only need them once, 
+      # so we have to filter the list again.
       if [ -f smtp-policy.spam.manual ] ; then
 	cat smtp-policy.spam.manual
       fi ) | tr "[A-Z]" "[a-z]" | sed 's/^@//g' | sort | uniq | \
-    awk '/^\[/{ # an address block to reject
+	    awk '/^\[/{ # an address block to reject
 	    printf "%s rejectnet +\n", $1;
 	    next;
 	}
--

Mark

-------------------------------------------------------------------------
Mark Visser                |   Student Civiele Technologie & Management
Calslaan 26 - 31           |   E-mail: mark@cal026031.student.utwente.nl
7522 MC Enschede           |   SNT-mail: mark@snt.utwente.nl
Telephone: 053-4895038     |
-------------------------------------------------------------------------
Warning: You can get rid of all the bugs by disabling them from the main menu.


