    swbis - Distributed Software Administration
    --------------------------------------------
              Version 0.950

    swbis is a GNU package
    The official home page is http://www.gnu.org/software/swbis

    Please send bug reports to <bug-swbis@gnu.org>
    
    The latest version is found at
    ftp://ftp.gnu.org/pub/gnu/swbis    
 
    The entry in the FSF/UNESCO Free Software Directory is
    http://directory.fsf.org/GNU/swbis.html

Introduction
------------
   'swbis' aims to implement the POSIX packaging standard with useful
    extensions.  Working utilities include swpackage, swverify, swcopy,
    swlist, and swinstall.  Package security is provided by embedded
    control files containing md5, sha1, and sha512 digests, and GPG
    signatures. swbis is back-compatible with free software source
    tarballs because package meta-data is contained in-band in the
    prescribed directory.  Since swbis supports nil control directory
    names, the package layout need not change except for the addition
    of this directory.  The swbis utilities are network transparent by
    direct use of SSH, and do not have to be installed on the target
    hosts. swbis run-time packages can be delivered, by swinstall, to
    any GNU, BSD, or Unix-like host with no new provisioning
    requirements for the remote host. Since run-time packages are POSIX
    tar archives, they may be unpacked using the system tar utility.
    The spec is available at
    http://www.opengroup.org/publications/catalog/c701.htm. 

    GNU swbis is a free and from-scratch GPL'ed implementation of
    IEEE 1387.2-1995 Std (Part 2: Software Administration) which
    describes a format and utilities for software packaging.  The
    GNU implementation has extensions for including GPG signatures,
    MD5 and SHA1 message digests of the files and archive payload
    section.  [The IEEE standard is superseded by ISO 15068-2,1999,
    and it is almost identical to the Open Group spec CAE C701 which
    is available on the Internet at no monetary cost].

    The swbis project provides features similar to other packaging systems
    such as utilities for package creation, installation, and verification.
    The implementation of these features are, however, unique in that
    swbis is 1) non-intrusive to current packaging practice, 2) completely
    orthogonal to the GNU toolset, 3) makes no new requirements on system
    provisioning, and 4) is completely network transparent.
    
    It achieves the first because a POSIX package is a tar archive, and,
    swbis is general and flexible enough so that a swbis package can look
    just like a free software source tarball.  When packaging a directory,
    such as a source directory, swbis provides a tool to generate a GPG
    signed tar archive using GNU tar, avoiding use of new tools.
    Secondly, it is orthogonal with existing tools for verification and
    file loading, namely gpg and tar which swbis uses directly.  Thirdly,
    swbis can install and copy packages to all existing GNU/Linux, BSD,
    and Unix hosts whether or not swbis is installed on that host.  Swbis
    achieves this by relying on the POSIX shell's capability to read
    commands and data on the same standard input stream.  Lastly, network
    transparency is achieved because the process interface is a Unix
    pipe, namely standard input, whose location can easily be generalized
    by a remote shell client program such as ssh.
    
    * ISO/Posix Utilities:

       swcopy - Distribution copying utility.
    
             * Supports multiple ssh-hops to remote target or source.       
             * Uses ssh, sh, dd, and choice of pax, tar, star, or gtar.
             * Does not have to be installed on the target host.
             * target or source host may be any POSIX.2 conforming host.
             * Supports remote-source to remote-target copies.
    
       swpackage -  Packages files according to a PSF file.
   					(beta as of 2007-09-01)
 
             * Complete implementation of Extended Definitions.
             * Stand-alone operation, no use of temporary files.
             * Output format is bit-for-bit identical to GNU tar.
             * Supports creation of GPG embedded signature, and sha1,
               md5sum digests of the package payload section.
            
       swverify -  Authenticates a signed package.
    
             * Authenticates packages signed by swpackage.
             * Verifies installed software using the GPG signed meta-data.

       swinstall  -  Install a posix package.  (alpha as of 2004-02-15)
       					       (beta as of 2007-09-01)
                 
             * 'swinstall' does not have to be installed on the target host.
             * The target or source host may be any POSIX.2 conforming host.
             * creates an installed software catalog where the GPG signature
               can be verified using 'gpg'
             * Uses ssh for remote connections.
             * Supports multiple ssh-hops to remote target or source.
             * Uses tar (or pax) to install the files.
             * Supports full error monitoring during control script execution
               and records exit status and installation states.

       swlist  -  List the catalog (alpha as of 2006-12-01)

             * Supports listing and selection via the '-c -' to stdout.
             * Supports listing files from a distribution.
             * Supports listing product entries in the installed catalog.
             * Supports asserting dependency specs against the installed catalog.
       
       swremove  -  Remove installed software
 
    * Non-ISO/Posix (Ad Hoc) Utilities:

        swign  -  Produces signed tar archives using GNU tar and
                  swpackage and gpg as supporting tools.  This utility is
                  useful for creating signed source tarballs.
            
        lxpsf  -  The first step translator for RPM packages.  It translates
                  an RPM into a tar archive containing a PSF file which
                  further directs translation to a Posix package by 'swpackage'.
    
        arf2arf   -  Internal Library utility to extract and verify the signed
                     and digested byte streams.
    

How to make some tarballs For the really impatient
--------------------------------------------------

    Here's an example of making a binary package using
    the recipe file ./PSF.bin.in.  This shows how to use swign
    to process replacement parameters %__revision.  (Generating
    and re-writing PSFs are outside the scope of swbis (at the moment)
    but a limited capability has been forced into swign).
    This assumes the swbis source directory is the current directory
    and is compiled.

      cat PSF.bin.in |
      swign -s - --show-psf --no-psf-detect --name-version=swbis-0.1 |
      swpackage -pv  # -p is preview mode

    Now, try with some options 

      cat PSF.bin.in |
      swign -s - --show-psf --no-psf-detect --name-version=swbis-0.1 |
      SWPACKAGEPASSPHRASE=Iforgot swpackage -s - \
		--files \
		--file-digests \
		--sign \
		--gpg-name="Test User" \
		--gpg-path=var/gnupg \
		--passphrase-fd=env \
		-pv \
		@-

    Now, just remove the '-pv' and pipe it into swverify,  This shows
    how *not* to treat your passphrase, normally SWPACKAGEPASSPHRASE
    would be unset and --passphrase-fd is set to "agent" to use
    the GPG agent or "tty" to read from the terminal.

      cat PSF.bin.in |
      swign -s - --show-psf --no-psf-detect --name-version=swbis-0.1 |
      SWPACKAGEPASSPHRASE=Iforgot swpackage -s - \
		--files \
		--archive-digests \
		--file-digests \
		--sign \
		--gpg-name="Test User" \
		--gpg-path=var/gnupg \
		--passphrase-fd=env \
		@- |
	GNUPGHOME=var/gnupg swverify -d @-

Documentation
-------------
    Info Document: type
          info swbis
              -or-
          info -f ./doc/info/swbis.info

    Manual pages:  sw(5), swcopy(8), swpackage(5), swpackage(8)
    swverify(8), swign(1) 
          For example:
	    nroff -man <doc/man/man8/swpackage.8 | less -r

KNOWN BUGS
----------
    SWINSTALL with a Linux kernel and OpenSSH sshd (Extremely Rare)
    -------------------------------------------------------------- 

    It seems that swinstall with certain packages when accessing
    a Linux kernel host via OpenSSH sshd (using ssh protocol v2) will fail.
    When the failure occurs the swinstall will give a message like:

    swinstall: Warning: 30 NUL blocks at start of input for task Catalog directory removal
    tar: Unexpected EOF in archive
    tar: Error is not recoverable: exiting now
    swinstall: SWI_TASK_ENDS on target host low20.lowland.com: load fileset: status=2
    swinstall: SW_SOC_LOCK_REMOVED on target host low20.lowland.com: status=0
    swinstall: SW_SESSION_ENDS on target host low20.lowland.com: status=1
    swinstall: error loading fileset
    swinstall: SWBIS_TARGET_ENDS for @localhost:/tmp/aabbxx: status=1

    A subsequent (immediate) retry may succeed, in fact the failure
    may be reproducable only 1 time out of 20 or 30 attempts but only for
    that particular package.  Another package may never exibit the bug.  It
    seems dependent on CPU and network speed, and I/O pattern, and CPU load.
    The probablility of failure increases if a multi-hop ssh target is used
    such as   swinstall [...] @ localhost:localhost:localhost:/tmp/xxtest
    which chains together three sshd servers.

        Work-Arounds
        ------------
        Try one of the following:
        1)  Run the exact command again, it will probably work.
        2) --burst-adjust=8000 --pump-delay=100000 # add a 100000 nanosec
                                                   # delay every 8000 bytes
        Investigators may try:
        3) --ssh-options=1                # Use ssh protocol version 1
        4) --remote-shell=rsh             # Use rsh
           
    It has *never* been seen on a BSD kernel, nor for local installs [that
    do not use ssh], nor when using rsh instead of ssh.  It can be exibited
    with a single host using localhost and and a ssh OpenSSH
    server 4.x - 5.x versions.

    This bug was first noticed in 2004 on a 2.4.x kernel, OpenSSH v3.9 may
    mask the occurance.

      [Update --- April 2008]
      -----------------------
      An investigation was launched into OpenSSH v4.7p1 and v5.0p1 using
      2.6.9-34.ELsmp (WhiteBox RHEL3, respin4).  The client was excluded
      as a cause.  The server's byte pumping machinery was instrumented
      and observed during the actual failure.  Every indication was
      the data was properly arriving at the terminal write() [at the
      unix pipe out of the server] but not arriving properly at the
      program.  It was observed that a time delay in the client
      (data producer) via a usleep(), or in the program (data consumer) by
      fronting the reader with 'dd ibs=2', or in the loop in serverloop.c
      via a usleep() would mask the bug.

      The write() nbyte parameter at terminal write in channels.c was
      observered in real-time, it ranged in size from ~1k to 100k but could
      not be correlated to the failure.  The legal limit for a write to a
      pipe was researched, *there is no such limit*.  Nevertheless, a simple
      patch to sshd was constructed to limit (i.e normalize) the size of this
      write to PIPE_BUF (along the lines of atomicio()). The results were
      positive, the failure disappeared while maintaining the full speed
      throughput.

      Conclusion:  The ability to fix this via a normalizing write() at
      this point in sshd implicates something other than OpenSSH.
      It occurs on 2.4.x thru 2.6.25 Linux kernels. It never occurs on
      a BSD kernel running the same version of openssh, bash, dd, tar, etc.
      It never appears when using rsh either.  All of this points to a bug in
      the Linux kernel, or Intel CPUs (AMDs have not been tested), probably a
      rare interaction with the varying and/or large writes to a pipe that
      occur in ssh protocol version 2 in OpenSSH.

    AWK
    ---
    The awk code in swsupplib/shell_lib/shell_lib.sh trips a bug in
    /usr/xpg4/bin/awk on Solaris hosts (OpenSolaris build 54) when using
    relops (applied to revision numbers) in selections. Replacement
    of /usr/xpg4/bin/awk with gawk, mawk, or awk (the awk-book awk
    from AT&T) fixes the problem. A work-around to allow all awks
    to work has not been devised.
    For example:
        # swlist --products as1\*,r\>1.0 
    lists nothing,
    but replacing awk properly lists the catalog entry:
        as10k1                  r=1.0.8         v=1mdk

    All distributed SW<utilities>
    -----------------------------   
    If you hit ctrl-c during a remote install, stranded shell processes
    and its sshd parent might be left running on the target host.
    To clean this up, do:
         swremove --cleansh @ USER@HOST    # i.e. Same USER and HOST

    SSH Oddities and Observations
    -----------------------------
    Problems were observed when running swinstall on a SunOS host
    (SunOS sparc-solaris1 5.9) using the Sun ssh client (Sun_SSH_1.0)
    The problem only affected installs to remote hosts and seemed to
    go away if an OpenSSH ssh client was used instead of Sun's.

How to install
--------------
    See the file INSTALL.
 
Features of the POSIX Standard
------------------------------
    
    - Uses existing data formats: tar and cpio.
    
    - Package metadata is contained in ascii text files.
    
    - The package generation spec file supports explicit
      overriding of file attributes, the ability to specify a
      different source filename from the name appearing in the
      package, and the ability to specify all files in a directory,
      among others.
   
    - Supports multiple products in a single distribution file.
    
    - Supports selection of products based on based on architecture
      and system identification attributes.
    
    - Allows run-time packages to have control directories and 
      a leading package directory.  This can be useful for ISVs who
      wish to support manual installation by unpacking with tar
      and execution of a custom setup script.
  
    - Supports a rich set of control scripts.
 
    - Explicitly allows for additional vendor attributes
      and additional vendor specific control files.

    - Specifies an interface for copying and installing to
      remote hosts.
    
Features of the swbis implementation
------------------------------------
            
    (All of these are implemented unless stated otherwise.)
    
    - Non-root users are able to invoke root installs on
      remote hosts via ssh authentication.
    
    - Network transparent copying and installation.
      (To swbis, local and remote installs are the same.)
    
    - No new target requirements.  Relies on a POSIX shell and
      a minimal set of POSIX utilities. (swbis is not required
      on the target remote host.)
    
    - Supports multiple host-to-host hops to a target host.
            
    - Network connection is made by direct use of ssh or rsh.
    
    - 'swpackage' writes an archive to STDOUT, and makes no temporary
      copy of the package.
    
    - 'swpackage' supports nil ("") control directories which
      allows a traditional source tar archive to be cast as a POSIX
      package with no change to existing file layout, and allows a
      run-time package to have a form that is directly installable
      using tar [similar to slackware packages].

    - Supports a md5sum, sha1sum and sha512sum  attributes for
      storage files and the package payload.
    
    - Supports cryptographic authentication (Package signing) by
      use of Vendor Defined attributes consisting of a control file 
      containing a signature of the catalog section.
    
    - Multiple signatures are supported as separate control files in
      the archive (all with the same name) to accommodate the concept of
      upstream and downstream signers.
    
    - All package signatures are transferred (by swinstall) into
      the installed software catalog where they can be verified
      directly by 'gpg' alone.  That is, the signature is valid
      throughout the package life-cycle, and can be verified by
      'gpg' at every point in the life-cycle.

    - swpackage mimics GNU tar formats bit-for-bit which forms a
      basis for internal testing; and, allows verification of the GPG
      signature and archive md5 digest (of the unpacked signed package)
      using GNU tar and gpg [under some circumstances and with
      constraints].
    
Development Language
--------------------
        swbis is written in ANSI C and C++.
        Some utilities are portable Posix shell and utilities.
        The swpackage utility is the only C++ program, all other
        programs should not use C++.  The parts of swpackage that are
        written in object-oriented (late dispatch of virtual functions)
        C++ are limited to the package compiler section.  Many of the
        C++ opjects are wrappers around C objects and do not employ
        polymorphism. Since swpackage is complete, no new C++ is
        needed for this project.

To make the autotools
--------------------
 aclocal && autoconf && autoheader && automake

To make under a different revision
----------------------------------
	# This modifies the ./VERSION file
	YOUR_VERSION=0.600.1 sh bin/reversion.sh $YOUR_VERSION
	./configure && make && make distclean
 
Internal Design of the sw<utilities> (except swpackage)
--------------------------------------------------------

,-------------------------------------
| # Target Script read from stdin     |
|   "(                                |
|   bash -s # < Task_Script_1         |
|   bash -s # < Task_Script_2         |
|   bash -s # < Task_Script_3         |
|   )"       ^                        |
|           /|\  ,-----<< Data        |
|            |   |    ,---<< Errors,  |
|            |   |    |      Protocol |
|           fd0 fd1  fd2              |
`------------o---o----o---------------
            /|\  |    |
             |   |   \|/
             |  \|/   |
        ,----o---o----o---.
        |   fd1  fd0  fd2 |
        |                 |
        |    (bash -s)    |           
        |                 |
        |   fd0  fd1  fd2 |
        `----o----o---o---'
             |    |   |
             |    |   |
        ,----o----o---o---.
        |   fd0  fd1  fd2 |
        |                 |
        |      (sshd)     |           
        `---------o-------'     TARGET HOST
                  |            -------------
                  |                NETWORK 
      ,-----------o----------.
      |   Public Network     |  1 or more "ssh hops"
      `-----------o----------'
                  |                NETWORK 
                  |            -----------------
        ,---------o-------.     MANAGEMENT HOST
        |       (ssh)     |           
        |                 |
        |   fd0  fd1  fd2 |
        `----o----o-------'
             |    |    |
            /|\   |   errors &
             |    |  protocol events
             |  data   |
          script  |   \|/
             &    |    | 
           data  \|/   `------------------------,
             |    |                             |
        ,----o----o----o------------.           |
        |  fdN   fdM   fdP (closed) |           |
        |                           |          \|/
        |     (Main Program)        |           |
        |  sw<utility> [Parent]     |       ,---o-------------------,  
        |                           |       |  fd E                 |
        |                           |       |                       |
        |                           |       |  (Logger Process)     |
        |                           |       | sw<utility> [child]   |
        |                           |       |                       |
        | 1: Send Main Target Script|       | Process SWI_<events>  |
	|                           |       |                       |
        |                           |       | Monitor Remote stderr |
        | 2: Send Task_Script_N     |       | Send Events to Parent |
        |    Monitor SWI_<events>   | event | \ /                   |
        |     (See swicol.c)        |  fd   |  |                    |
        |    Wait for END event <---o---<---o--'                    |
        |    Check exit status      |       |                       |
        |        messages.          |       |       Write to stderr |
        |                           |       |             \|/       |
        |                           |       | Write Log    |        |
        |                           |       |   \|/        |        |
        `--o----------------o-------'       `----o---------o--------'
           |                |                    |         |
       STDIN_FILENO      STDOUT_FILENO          LOG    STDERR_FILENO
    
End of README
