Thanks for trying out Insync's patched-to-hell-and-back INN 1.7.2.  The
following patches have been intergrated for your INN pleasure:

  * expire hash improvement patch
  * symlink fastrm patch
  * Jeff Garzik's perl-filter-body patch
  * Jeff Garzik's improved history cache patch
  * MMAP overview patch
  * Jeff Garzik's pre-commit cache patch
  * Joe Greco's timer patch
  * Joe Greco's active daemon
  * DBZ 3.3.2
  * innd HEAD command fix
  * nnrpd large overview fix
  * Joe Greco's newsfeeds path patch
  * Jeremy Nixon's Clean Feed patch
  * overchan.c patch to ignore control groups

The only patches you actually have to do anything about are MMAP overview,
timer, and active daemon.

You may wish to (and really should) setup Jeremy Nixon's Clean Feed.  It's
an absolutely brilliant spam filter based on Jeff Garzik's EMP filter. See
http://www.exit109.com/~jeremy/news/antispam.html for all the info.

If you want to use the MMAPed overview files, define OVERSCREAM somewhere.
Generally the easiest way to do this is to add -DOVERSCREAM to your CLFAGS.

If you want to use Joe Greco's timer patch, define TIMER somewhere.  Again,
the easiest way to do it is to add -DTIMER to your CFLAGS.

You need to edit the last few lines of config.dist (after copying it to
config.data) to configure some actived stuff.  If you have an old
config.data you want to use, you can, but be sure you copy the last few
lines of config.dist (section 18) onto the end of your config.data or
the compile of actived will fail.  Please note that the values in
config.data are tab-delimited and must remain such.

If you're enabling Perl support, please note that this version of INN
supports dynamic loading of Perl modules.  Please read the comment on the
PERL_LIB entry in config.dist for information on how to successfully
compile with Perl support.

I did not write any of these patches myself - I'm not that good.  :-)  I've
made a couple of changes here and there to either get them to agree with
each other, get them to compile on more platforms, or perhaps even add a
tiny little feature or two.  If you experience problems with these patches,
they're most likely my fault, so blame me, not the original authors.  Just
send the original authors your thanks for their fantastic patches.  :)

Special thanks go out to the people who have submitted bugfixes and
additional patches.  They are, in no special order, Brian Young
<byoung@oru.edu>, Keith McCallion <keith@iswest.net>, and Mikhail A. Sokolov
<mishania@demos.su>.  I'm sorry if I forgot to list anyone I should have.
Feel free to rake me over the coals if I forgot you.

Have fun!

-Andrew Smith <aos@insync.net>
 Insync Internet Services, Inc.
 Houston, Texas, USA

---

Most of these patches are small enough that they don't really have any
release notes.  The active daemon release notes/instructions are in the
actived directory.  The other release notes (from their original authors)
follow:

--- - MMAP overview patch (<johnpc@xs4all.nl>)

OVERSCREAM - to make the overview database screaming fast, and because
I scream in terror about the previous implementation.
See http://www.xs4all.nl/~johnpc/inn/ for more information on this patch.

--- - pre-commit cache patch (Jeff Garzik <jeff.garzik@spinne.com>)

This patch is a little bit more experimental than the previous
history-cache patch.  It adds a "pre-commit" cache, which locks a
message id in a hash table for 30 seconds.  All CHECK and IHAVE requests
for that message id during those 30 seconds will be refused, [hopefully]
saving bandwidth.  Cache store collisions simply overwrite the existing
cache entry.  The logic is:

IHAVE
        if ((msg id cached) AND (not timed out))
                Refuse article
        otherwise (check history, ...)
        otherwise
                Put msg id in pre-commit cache
                (go to 'get article' state, ...)

CHECK
        if ((msg id cached) AND (not timed out))
                Refuse article
        otherwise (check history, ...)
        otherwise
                Put msg id in pre-commit cache
                (Reply with 'sendme' code, ...)

TAKETHIS
        (msg id checks...)
        Put msg id in pre-commit cache
        (go to 'get article' state, ...)


Like the previous patch, it applies to a clean inn-1.5.1 source tree.
Note that this conflicts with apb's jumbo patch, because of his
NCwritereply updates.

The commented-out #define NCPRECOM_PEDANTIC will, if enabled, remove
each msg id from the pre-commit cache after it is rejected or accepted
by the system -- ie. after it's in the history cache.  The default is
just let the msg ids expire after 30 seconds.

The commented-out #define NCPRECOM_DEFER will, if enabled, return a
defer code to a streaming server upon a cache hit, instead of refusing
the article.  This is more proper, but could slow down a feed.  The
default is to refuse the msg id for the duration of its cache life.

Creds to MD, JA, CO'N, and BB for ideas.  Bug me about problems.

--- - timer patch (Joe Greco <jgreco@ns.sol.net>)

This code was inspired by a similar bit of code by Nils McCarthy at
Primenet Services for the Internet.  Like his code, this code is capable
of reporting the amount of idle time that your innd currently has.
However, that was a very frustrating bit of information, in my opinion,
since usually you know if your innd is having performance problems.

I have traditionally used process profiling to gather better statistics
on how much time was being spent in various parts of the system.  This
worked very well, and was very detailed... almost too much so.  It was
also a bit of a pain in the rear to set up, and could only provide long
term cumulative performance data since the profiling code available to
me dumps its statistics at process termination.

It was not a good way to do it.

So, the following tool came to existence late one evening when I really
wanted to know why a machine was saturated.

Like the idle time tool, it reports to news.notice every five minutes and
provides a summary of time spent in the select() call (essentially waiting
for something to happen, i.e. idle).  However, it is also capable of
reporting on other events, and the number of times they occurred.

A sample:

Oct  9 11:39:25 news innd: ME time 300097 idle 196062(9493) artwrite 65207(1009)
 artlink 11(339) hiswrite 6879(1009) hissync 2(16) sitesend 232(5320) artctrl 0(
17) artcncl 0(1) hishave 6145(8876)

Notes:  all times reported in milliseconds.  300097 = 300.097 seconds.  The
numbers in parenthesis are the number of calls made to this particular
timer during the five minute interval.

The following are the numbers I extract from this raw information.

Percent busy:        (300097 - 196062) / 300097 = 35%
Time per article write:          65207 /   1009 = 64 ms
Percent time spent writing art:  65207 / 300097 = 21%
Time per crosslink:                 11 /    339 = 0 ms
Percent time spent crosslink:       11 / 300097 = 0%
Time per history write:           6879 /   1009 = 7 ms
Percent time spent writing his:   6879 / 300097 = 2%
Time per history lookup:          6145 /   8876 = 1 ms
Percent time spent hist lookup:   6145 / 300097 = 2%

(The history sync, site send, art control and art cancel numbers do not
seem to be useful, but I have not verified why this is yet.)

In any case, the above numbers are all calculated such that you want them
to be as close to zero as possible.

One further rule of thumb metric I have noticed is that the number of
elapsed milliseconds ("time") is a very good measure of whether or not
there is latency in the system, the closer to 300000 it is, the less
latency there is.  Once you get past 301000, you have lots of latency.

Another site:

Oct  9 12:07:45 news innd: ME time 306964 idle 7(50) artwrite 178598(1152) artli
nk 79069(603) hiswrite 25385(1.7.18) hissync 70(20) sitesend 235(6782) artctrl 0(0
) artcncl 0(0) hishave 2357(1969) hisgrep 11(16)

This site is clearly being dominated by spool access times.  Between the sum
of the artwrite and artlink times, a vast majority of the 306 second sample
time is being used.  The site is clearly not using crosspost, but even if
it did, further steps would be required to correct the spool problems.
The history lookup and write rates are also pretty bad, but not dominant.

Another site:

Oct  4 22:29:49 news innd: ME time 312518 idle 15090(1399) artwrite 90455(496) a
rtlink 30811(227) hiswrite 4380(503) hissync 253(98) sitesend 70(1.7.15) artctrl 2
737(17) artcncl 0(0) hishave 158409(1734) hisgrep 7(10)

Oct  8 00:38:04 news innd: ME time 300654 idle 99(1005) artwrite 1.7.1706(1.7.13) ar
tlink 58071(544) hiswrite 9347(1416) hissync 9(28) sitesend 191(4603) artctrl 43
69(53) artcncl 101(2) hishave 50061(4061) hisgrep 3(12)

Oct  8 00:53:34 news innd: ME time 300044 idle 182(1288) artwrite 199605(1462) a
rtlink 16(493) hiswrite 10086(1547) hissync 15(28) sitesend 314(6324) artctrl 20
35(38) artcncl 2322(54) hishave 66894(4854) hisgrep 3(7)

The first and second ones had no crosspost, the third one did.  The news
lib (history, active, et al) was on the root disk for the first sample,
but on a faster disk for the second two.

Quite frankly the hishave times stank at first.  At 91 ms per lookup, that
means a maximum of about 11 per second were possible.  After adding a
separate disk, this went down to 13 ms per lookup.  This is still not
"great" in my book, but it is much better!

Also notice when crosspost was enabled, right away, the artwrite rate
jumped from 1.7.13 to 1462, but the general spool access time was still
pretty crummy.  This is still the limiting factor of this news system.

In any case, the tool itself will not tell you why your news system
is so bad.  However, it can provide some good, detailed clues as to
what your news system is spending time doing, and that is pretty
useful information if you take the time to understand what it means.
I do not have any wonder tools to help postprocess the information.

Future possible additions:

A latency timer, timing the minimum, average, and maximum times to make
a complete pass through the select()'ed hosts.  This would give clues as
to the amount of time it took for INND to print a banner or kick off an
NNRP session, and would be helpful in judging the throughput possible on
non-streaming connections.

An article-write timer to help identify particularly slow spool disks
by recording the amount of time on a per-article basis taken to write
that article.

In any case:  a special thanks to:

Nils McCarthy for an excellent idea and a good Web page that has some
useful hints.  http://www.nntp.primenet.com/INN/

Michael Beckmann for being an excellent guinea pig.

Michael Shields, Joe St Sauver, Dave Pascoe, and a few other folks for
beta testing.

Good luck!

--- - nnrpd large overview fix (Edward S. Marshall <emarshal@xnet.com>)

Well, this was posted to Usenet, but just for good measure, I'll send it
here. This basically fixes a problem where nnrpd crashes on overview file
lines that are longer than the allowed buffer (can happen if the overview
files get corrupted for some reason).

--- - innd HEAD command fix (Yoshiro YONEYA <yone@pi.ntts.co.jp>)

For Usenet RKT subscribers.....

....Additional non-critical defect in innd discovered.....

Discovered by: Yoshiro YONEYA <yone@pi.ntts.co.jp>
               as reported by the message (excerpted below.)

Defect in: the innd implementation of the "HEAD" command.
           innd/nc.c

Versions: 1.5.1corr and derivatives, including 1.7

Symptoms: In the very rare case of a connection sending
          the "HEAD" command to innd, the connection may hang.

          This is not a security vulnerability.

          The "HEAD" command in nnrpd (readers) is not affected.

Background: In the normal mechanism of feeding, the HEAD command
            is never used, as far as I am aware.  So the defect
            would only be triggered by receiving a connection
            from a machine in hosts.nntp which was using very odd
            or experimental software, or telnet for debugging
            purposes.

            The NNTP "HEAD" command is described at:
                http://www.mibsoftware.com/userkt/nntp/0013.htm

Recommendations: Since this is not a security threat, and is only
            triggered in rare combinations of software, you may
            patch at your leisure.

An "enhanced" version of the patch is attached.  It credits
the author, and also fixes a similar defect in the NCarticle() 
command.  (The NCarticle() command is normally permanently disabled,
but fixing the defect at the same time is appropriate.)

This message, including the patch, will also be available at:
   http://www.mibsoftware.com/userkt/inn/patches/nc971031.msg

This message has been brought to you as a result of your
quarterly subscription.  Thank you.

Forrest Cavalier, Mib Software, originator of INN 1.7

--- - newsfeeds path patch (Joe Grego <jgreco@sol.net>)

This patch adds a "P" flag to the third field "W" options of newsfeeds(5).

This allows one to specify a file feed for Path: lines, to be used with
Landon Curt Noll's inpaths.  I got tired of waiting for him to write it.
:-)

You will want to stick an entry like

# PATH! - collect Path headers from articles for inpaths survey
PATH!:*:Tf,WP:/var/log/news/path.log

in your newsfeeds file.

--- - Clean Feed (Jeremy Nixon <jeremy@exit109.com>)

The README for this patch isn't included because Clean Feed is really a
separate entity (although I strongly encourage you to set it up) and the
README is reasonably large.  The only part of Clean Feed distributed with
this patch cluster is the necessary patches to INN itself.  You'll need to
get the rest of the Clean Feed distribution from Jeremy's web page
(http://www.exit109.com/~jeremy/news/antispam.html).

