GNU shtool - The GNU Portable Shell Tool 1.4.0 (02-Jul-1999) shtool [ global_options ] command [ command_options ] [ command_args ] The GNU shtool program is a compilation of small but very stable and portable shell scripts into a single shell tool. All ingredients were in successful use over many years in various free software projects. The compiled shtool program is intended to be used inside the source tree of those free software packages. There it can take over various (usually non-portable) tasks related to the building and installation of such packages.

Context Background
For the configuration, build and installation environment of modern free software packages one nowadays uses GNU autoconf and friends (i.e. usually autoconf, automake and libtool). Autoconf covers the configuration, automake covers the generation of the build environment and libtool covers most of the build process itself. But at least when it comes to the installation step one usually have to use a few auxiliary scripts like mkdir.sh, install.sh, etc. And these things are usually all the time needed. The result is usually an etc/ subdirectory in the source tree where over time a lot shell scripts accumulate.

Maintainance Problem
The problem with those etc/ shell scripts starts when one has to maintain lots of free software packages as it's the case for the author of shtool. Then over time all etc/ directories diverge and with every day it gets more and more nasty to always keep them in sync. Especially when some scripts were locally adjusted because no centralized maintainance location exists, of course. For autoconf no such problem exists, because the resulting configure script is generated on-the-fly. The same applies to automake and the various Makefiles. Only for libtool one always has to grab the latest copy. But because it's just two files (ltmain.sh and ltconfig), keeping a source trees in sync is not too complicated (especially not when using libtoolize). But the etc/ shell script mess is nasty, especially because there is no master version on the net. Additionally everytime one starts a new project, one has to establish a new source tree. For a hacker it's immediately clear that autoconf and friends are part of the game. But which etc/ shell scripts are needed this time? And from which existing source tree to copy them from? Hmmm...

The Aesthetic Problem
When a free software package has a large source tree (say, more than 50 files and especially with one or more subdirectories) it's usually no problem to have an additional etc/ subdirectory with some scripts. They then totally go down. But for smaller packages, especially those living in a single source directory (a degenerated tree), some people like the author of shtool have aesthetic problems. Because it looks strange to them that 20% of the files in the source tree are just auxiliary scripts. Sure, the actual amount of script code even shtool cannot reduce, but shtool merges them together into a single file and this way they optically totally disappear from the source tree. This is a pure aesthetical issue, of course. But hey, hacking is a piece of art. And a source tree is a piece of art for hackers, too. Oh, and those who insist on a technical reason: it's also easier to upgrade a single file than multiple files ;)

Filling the gap
So, wouldn't it be nice to have a fourth package (beside autoconf, automake and libtool) which fills the gap, i.e. which provides the functionality of the old files in etc/, is maintained externally and at a centralized location and even consists of just a single (perhaps large) script one can overtake as a black box the same way one already does this for libtool? The author thought this would be actually very useful and the result is the current shtool package which at least successfully solved the above problems of the author.

The goals in detail
To better understand the intentions behind shtool, here are the original goals of the shtool script: This was achieved by compiling the resulting shtool script out of the ingredient source scripts. The advantage is that shtool is still easily maintainable, because one can test each script seperately. But the final functionality then resides in an all-in-one script which can be easily spread over multiple source trees. This was achieved by (re)implementing really all functionality which experience showed is important in source trees of typical free software packages. This was achieved by using only well-proven shell code which already survived practice in other projects over more than a few months. Especially this means that a lot of complicated emulations are done to avoid the use of unportable Unix programs (like fmt, tr, etc) or unportable features of well-known Unix programs (like shell functions, special sed features, etc. pp). That's why shtool's code sometimes looks crazy and like overkill. But it's often this way mainly for portability reasons. This was achieved by reimplementing too ugly functionality from scratch and cleaning up old shell script code plus writing this manual page. This was achieved by placing the shtool package under the GNU General Public License (GPL). This way the shtool package itself is well protected and will always be free software, but the resulting shtool script can be nevertheless used in all types of source trees. Notice here: given that one includes GNU shtool verbatim into an own source tree, one is justified in saying that it remains separate from the own package, and that this way one is simply just using shtool. So, in this situation, there is no requirement that the package itself is licensed under the GNU General Public License in order to take advantage of shtool. Keep this in mind ;) The following global options are available for shtool. Any commands are ignored when one of them is present on the shtool command line. Displays a short help page describing the usage of shtool and it's ingredient commands in a compact way. Displays the version number of shtool. The following commands are provided by shtool. They are all called via ``shtool command''. Any trailing command_options are specific to the particular command. echo(1) style print command providing special expansion constructs (terminal bold mode, environment details, date) and newline control. Pretty-prints the last modification time of a file or directory. Pretty-print a field-sperarated list as a table. Display a processing indication though a running propeller. mv(1) style command, but can rename/move multiple files at once and allows source files just to be deleted when contents doesn't change. Install a program, script or datafile in a portable way. mkdir(1) style command providing support for auto-parent-dir creation, directory permission control and smart skipping when directory already exists. ln(1) style command providing automatic calculation and usage of relative links when possible. Create a shadow source tree by the help of symbolic links. Fix file permissions inside a source tree by cleaning up the permission bits. Simple operating system and platform architecture guesser which determines a GNU platform-triple style identification string. Extended archive command which can even put existing archives into an archive. Separate linker options by library class. An additional C source file pre-processor for sharing cpp(1) code, internal variables and internal functions. Generate and maintain a version information file in either text, C or Perl format. Deal with shell path variables. In the following the available commands and their corresponding command_options are described in detail. This is an echo(1) style print command which provides special expansion constructs (terminal bold mode, environment details, date) and newline control. Per default string is written to stdout followed by a newline character (``\n''). When option ``-n'' is used this newline character is left out. The str can contain special ``%x'' constructs which which are expanded before the output is written when option ``-e'' is used. Currently the following constructs are recognized: ``%B'' for switching to terminal bold mode, ``%b'' for switching terminal mode back to normal display mode, ``%u'' for the current user name, ``%U'' for the current user id (numerical), ``%h'' for the current hostname, ``%d'' for the current domain name, ``%D'' for the current day of the month, ``%M'' for the current month (numerical), ``%m'' for the current month name and ``%Y'' for the current year. The trick of this command is that it provides a portable ``-n'' option and hides the gory details needed to find out the environment details. Examples: # shell script shtool echo -n -e "Enter your name [%B%u%b]: "; read name shtool echo -e "Your Email address might be %u@%h%d" shtool echo -e "The current date is %D-%m-%Y" This command pretty-prints the last modification time of a file or directory path. Option ``-n'' suppresses the output of a trailing newline character, option ``-z'' pads numeric day (and optionally month) with a leading zero, option ``-s'' shortens the months name to an abbreviation of three characters, option ``-d'' replaces the month name with the corresponding digits, option ``-f'' uses str as the field separator (default is a single space character) and option ``-o'' specified the order in which the fields are printed. The default for spec is ``dmy'' which means an output of ``<day> <month> <year>''. Any combination of the chars ``d'', ``m'' and ``y'' or allowed for spec. The trick of this command is that it provides a portable way to find out the date of a file or directory while still allowing one to specify the format of the date display. Examples: # shell script shtool mdate -n / shtool mdate -f '/' -z -d -o ymd foo.txt shtool mdate -f '-' -s foo.txt This pretty-prints a sep-sperarated list of strings as a table. Per default a colon-seperated list (sep=":") is pretty printed as a three-column (<cols>=3) table no longer than 79 chars (strip=79) is generated where each column is 15 characters wide (width=15). The trick of this command is that it avoids to use the unportable tr(1) and fmt(1) commands and instead is based entirely on sh(1), awk(1) and sed(1) functionality. Example: # shell script shtool table -F , -w 5 -c 4 "1,2,3,4,5,6,7,8,9,10,11,12" This command displays a processing indication though a running propeller. The option ``-p'' can be used to set a particular prefix str which is displayed in front of the propeller. The default is no prefix string, i.e. the propeller is at the left border of the terminal. This command is intended to be run at the end of a pipe (``|'') sequence where on stdin logging/processing informations found. For every line on stdin the propeller cycles one step clock-wise. The trick of this command is that it provides a portable and easy to use way to display such nice and psychologically important process indicators. Example: # shell script configure 2>&1 |\ tee logfile |\ shtool prop -p "Configuring sources" This is a mv(1) style command, but with two special features: First when option ``-e'' (`expand') is used and an asterisk occurs somewhere in src one can use ``%n'' (where n is 1,2,...) in dst-file. This is useful for renaming multiple files at once. Second, when option ``-p'' (for `preserve') is used and src-file and dst-file are byte-wise the same it just deletes src-file. The intention is that the permissions and time stamps on dst are't changed which is important when dst-file is used in conjunction with Makefiles. Option ``-v'' (verbose) can be used to enable the output of extra processing information. Option ``-t'' (trace) can be used to enable the output of the essential shell commands which are executed. The trick of this command is that it can rename multiple files at once and preserves the timestamps when the contents isn't changed. Examples: # shell script shtool move -v -e *.txt %1.asc # Makefile scanner.c: scanner.l lex scanner.l shtool move -t -p lex.yy.c scanner.c This command installs a program, script or datafile (dependent on mode) in a portable way while providing all important options of the BSD install(1) command. Per default file is moved to the target path, but with option ``-c'' file is copied. The target file is created with owner/group set to the current active uid/gid, but when this script is called as root (uid 0) the options ``-o'' and ``-g'' can be used to override this. Additionally program executables is stripped with strip(1) after installation when option ``-s'' is used. Option ``-C'' is like ``-c'', except if the destination file already exists and the files are the same, the source is just removed. Option ``-e'' can be used to set an implicit executable extension. Option ``-v'' (verbose) can be used to enable the output of extra processing information. Option ``-t'' (trace) can be used to enable the output of the essential shell commands which are executed. The trick of this command is that it provides the functionality of BSD install(1) in a portable emulated way. Example: # Makefile install: : shtool install -c -s -m 4755 foo $(bindir)/ shtool install -c -m 644 foo.man $(mandir)/man1/foo.1 This is a mkdir(1) style command providing support for auto-parent-dir creation (when option ``-p'' is used), directory permission control (with option ``-m mode'' where mode can be in any of the formats specified to the chmod(1) command) and smart skipping when dir already exists (triggered by the force option ``-f''). Option ``-t'' (trace) can be used to enable the output of the essential shell commands which are executed. The trick of this command is that it provides both a portable ``-p'' functionality and the ability to be smart when the directory already exists which is important for installation procedures. Example: # Makefile install: shtool mkdir -f -p -m 755 $(bindir) shtool mkdir -f -p -m 755 $(mandir)/man1 : This is a ln(1) style command which provides automatic calculation and usage of relative links when possible, i.e. usually when src-path and dst-path are not absolute paths or at least they share a common prefix except the root directory (``/''). When more than one src-path is specified, all of them are linked into dst-path. Options ``-f'' and ``-s'' are similar to ln(1), i.e. they force the creation of the link (even when it exists) and create a symbolic link instead of a hard-link. Option ``-t'' (trace) can be used to enable the output of the essential ``ln'' command which is executed. The trick of this command is that it tried hard to calculate the paths to get the maximum possible relative paths. Example: # shell script shtool mkln -s foo/bar baz/quux This command creates a shadow tree of src-dir under dst-dir by recreating the directory hierarchy of src-dir under dst-dir and by creating the files of src-dir by linking them into the corresponding directories under dst-dir via symbolic links. When src-dir can be reached via relative paths from dst-dir, relative symbolic links are used, too. Option ``-v'' (verbose) can be used to enable some displaying of processing information. Option ``-t'' (trace) can be used to display all commands which are executed in order to construct dst-dir. Option ``-a'' (all) can be used to really shadow all files and directories in src-dir. Per default CVS related files and directories, backup files, object files, etc. are not shadowed. The trick of this is that is provides such a high-level functionality with a single command and hides all gory details. Example: # shell script shtool mkshadow -v -a . /tmp/shadow This command fixes file permissions inside a source tree under path by cleaning up the permission bits. It determines the cleaned up permission from the already set bits. It's intended to be run before a tarball is rolled out of the source tree. Option ``-v'' can be used to display some processing information. Option ``-t'' (trace) can be used to enable the output of the essential shell commands which are executed. The trick is that this is more convinient that having to set the permissions manually or with a large file list. Example: # Makefile.in dist: shtool fixperm -v * This command is a simple operating system and platform architecture guesser which determines a so-called ``GNU platform-triple'' style identification string ``arch-hardware-ososversion''. For instance a FreeBSD 3.1 running on a Pentium II is identified as ``i686-pc-freebsd3.1''. When you need a more sophisticated platform guesser, use the GNU config.guess/config.sub scripts, please. # configure.in OS=`shtool guessos` This is a wrapper around the archive (``ar'') tool. It provides the ability to create archives out of existing archives, i.e. when one of file matches ``*.a'' the archive member files of file are used instead of file itself. When option ``-t'' (trace) is given arx shows the actually involved shell commands. Option ``-C'' can be used to set the ``ar'' command to cmd. The trick of this command is the automatic handling of archive members which is especially interesting when one wants to construct a (usually top-level) library archive out of pre-build sub-library archives (usually staying inside subdirs) in a large source tree. Example: # Makefile AR=ar RANLIB=ranlib : OBJS=foo.o bar.o LIBS=baz/libbaz.a quux/libquux.a : libfoo.a: $(OBJS) $(LIBS) shtool arx -C $(AR) rc libfoo.a $(OBJS) $(LIBS) $(RANLIB) libfoo.a This command separates the linker options ``-L'' and ``-l'' by library class. It's argument line can actually be an abitrary command line where those options are contained. slo parses these two options only and ignores the remaining contents. The result is a trivial shell script on stdout which defines six variables containing the ``-L'' and ``-l'' options sorted by class: ``SLO_DIRS_OBJ'' and ``SLO_LIBS_OBJ'' contains the ``-L'' and ``-l'' options of static libraries, ``SLO_DIRS_PIC'' and ``SLO_LIBS_PIC'' contains the ``-L'' and ``-l'' options of static libraries containing PIC ("Position Independent Code") and ``SLO_DIRS_DSO'' and ``SLO_LIBS_DSO'' contains the ``-L'' and ``-l'' options of shared libraries. The intent of this seperation is to provide a way between static and shared libraries which is important when one wants to link custom DSOs against libraries, because not all platforms all one to link these DSOs against shared libraries. So one first has to separate out the shared libraries and link the DSO only against the static libraries. One can use this command also to just sort the options. Example: # configure.in LINK_STD="$LDFLAGS $LIBS" eval `shtool slo $LINK_STD` LINK_DSO="$SLO_DIRS_OBJ $SLO_LIBS_OBJ $SLO_DIRS_PIC $SLO_LIBS_PIC" : This command is an additional ANSI C source file pre-processor for sharing cpp(1) code segments, internal variables and internal functions. The intention for this comes from writing libraries in ANSI C. Here a common shared internal header file is usually used for sharing information between the library source files. The operation is to parse special constructs in files, generate a few things out of these constructs and insert them at position mark in tfile by writing the output to ofile. Additionally the files are never touched or modified. Instead the constructs are removed later by the cpp(1) phase of the build process. The only prerequisite is that every file has a ``#include "ofile"'' at the top. This command provides the following features: First it avoids namespace pollution and reduces prototyping efforts for internal symbols by recognizing functions and variables which are defined with the storage class identifier ``cname''. For instance when cname is ``intern'', a function ``intern void *foobar(int quux)'' in one of the files is translated into both a ``#define foobar __foobar'' and a ``extern void *foobar(int quux);'' in ofile. Additionally a global ``#define cname /**/'' is also created in ofile to let the compiler silently ignore this additional storage class identifier. Second, the library source files usually want to share typedefs, #defines, etc. over the source file boundaries. To achieve this one can either place this stuff manually into tfile or use the second feature of scpp: All code in files encapsulated with ``#if dname ... #endif'' is automatically copied to ofile. Additionally a global ``#define dname 0'' is also created in ofile to let the compiler silently skip this parts (because it was already found in the header). Option ``-v'' can be used to enable some processing output. Option ``-p'' can be used to make the decision whether to overwrite ofile independent of the generated ``#line'' lines. This is useful for Makefiles when the real contents of ofile will not change, just line numbers. Example: # Makefile SRCS=foo_bar.c foo_quux.c foo_p.h: foo_p.h.in shtool scpp -o foo_p.h -t foo_p.h.in \ -M %%MARK%% -D cpp -C intern $(SRCS) /* foo_p.h.in */ #ifndef FOO_P_H #define FOO_P_H %%MARK%% #endif /* FOO_P_H */ /* foo_bar.c */ #include "foo_p.h" #if cpp #define OURS_INIT 4711 #endif intern int ours; static int myone = 0815; intern int bar(void) { ours += myone; } /* foo_quux.c */ #include "foo_p.h" int main(int argc, char *argv[]) { int i; ours = OURS_INIT for (i = 0; i < 10; i++) { bar(); printf("ours now %d\n", ours); } return 0; } This command generates and maintains a version information file file for program name name in either textual (lang="txt"), ANSI C (lang="c") or Perl (lang="perl") language. The version is always described with a triple <version,revision,patch/alpha/beta-level> and is represented by a string which always matches the regular expression ``[0-9]+\.[0-9]+[.abps][0-9]+''. When the option ``-s'' is given, the contents of file is overridden with the specified version. When option ``-i'' is used the version ingredients in file are updated by adjusting one knob of the version where knob can be one of the following: ``a'', ``b'', ``p'', ``P'' and ``s'' just increase the alpha/beta/patch/snap level by 1 (and set the seperator character between version and the level position). ``r'' increases the revision by 1 and sets the alpha/beta/patch/snap level to 0. ``v'' increases the version by 1 and sets both the revision and alpha/beta/patch/snap level to 0. When option ``-d'' is given the current version in file is displayed either in short (type="short") or long (type="long") texual format or in a format suitable for use with GNU libtool (type="libtool") or in a hexadecimal format (type="hex"). When no option is given at all, ``-i P'' is assumed. Example: # shell script shtool version -l c -n FooBar -p foobar -s 1.2b3 version.c # configure.in V=`shtool version -l c -d long version.c` echo "Configuring FooBar, Version $V" This command deals with shell $PATH variables. It can find a program executable in $PATH or path through one or more filenames (given by one or more str arguments). The result is the absolute filesystem path to the program displayed on stdout plus an exit code of 0 when it was really found. The option ``-s'' can be used to suppress the output which is useful to just test whether a program exists with the help of the return code. The option ``-m'' enables some magic where currently for the programs ``perl'' and ``cpp'' an advanced magic search is done. The option ``-r'' can be used to transform a forward path to a subdirectory into a reverse path. Option ``-d'' and ``-b'' just output the directory or base name of str. Examples: # shell script awk=`shtool path -p "${PATH}:." gawk nawk awk` perl=`shtool path -m perl5 perl` cpp=`shtool path -m cpp` revpath=`shtool path -r path/to/subdir` sh(1), cp(1), rm(1), mkdir(1), awk(1), sed(1). Some scripts contained in shtool were already written in 1994 by Ralf S. Engelschall for use inside some private source trees. Then they evolved into more elaborated versions over the years and were used in various free software projects like ePerl, WML, iSelect, gFONT, etc. They were complemented with other scripts from the author which he wrote in March 1998 for the ``Apache Autoconf-style Interface'' (APACI). In April 1999 the shtool package was created out of the accumulated master versions of the scripts. In June 1999 it finally entered the status of an official GNU program. Ralf S. Engelschall rse@engelschall.com www.engelschall.com