Article 12866 of alt.sources:
Path: nntpd.lkg.dec.com!crl.dec.com!pa.dec.com!news1.digital.com!decwrl!genmagic!sgigate.sgi.com!swrinde!tank.news.pipex.net!pipex!news.mathworks.com!solaris.cc.vt.edu!swiss.ans.net!prodigy.com!darkstar.prodigy.com!davidsen
From: davidsen@darkstar.prodigy.com (Bill Davidsen)
Newsgroups: alt.sources
Subject: NewsTools-1.3 (1/2)
Date: 29 Sep 1995 19:03:27 GMT
Organization: I need to put my ORGANIZATION here.
Lines: 959
Message-ID: <44hftv$lg2@usenetw1.news.prodigy.com>
NNTP-Posting-Host: 199.4.137.118

Submitted-by: davidsen@tmr.com
Archive-name: NewsTools-1.3/part01

#!/bin/sh
# This is NewsTools-1.3, a shell archive (produced by GNU sharutils 4.1)
# To extract the files from this archive, save it to some FILE, remove
# everything before the `!/bin/sh' line above, then type `sh FILE'.
#
# Made on 1995-09-29 15:01 EDT by <davidsen@tmr.com>.
# Source directory was `/home/davidsen/binsrc/newstools'.
#
# Existing files will *not* be overwritten unless `-c' is specified.
#
# This shar contains:
# length mode       name
# ------ ---------- ------------------------------------------
#   1582 -rw-r--r-- NewsTools-1.3/./README
#    574 -rw-r--r-- NewsTools-1.3/./TOC
#   1271 -rw-r--r-- NewsTools-1.3/ncc/README
#   5269 -rwxr--r-- NewsTools-1.3/ncc/ncc
#    859 -r--r--r-- NewsTools-1.3/artrate/README
#   1632 -r-xr--r-- NewsTools-1.3/artrate/artrate
#   1649 -r-xr--r-- NewsTools-1.3/artrate/artrate.pl
#    731 -r--r--r-- NewsTools-1.3/artrate/Fract_hr.pl
#    931 -r--r--r-- NewsTools-1.3/artrate/Log_to_date.pl
#   1108 -rw-r--r-- NewsTools-1.3/connrate/README
#   2262 -r-xr--r-- NewsTools-1.3/connrate/connrate
#    429 -rw-r--r-- NewsTools-1.3/nslg/inet.h
#    541 -r--r--r-- NewsTools-1.3/nslg/loadtest/Makefile
#  27686 -rw-r--r-- NewsTools-1.3/nslg/loadtest/nslg.c
#    167 -rw-r--r-- NewsTools-1.3/nslg/loadtest/inet.h
#    404 -rw-r--r-- NewsTools-1.3/nslg/lib/Makefile
#   1552 -r--r--r-- NewsTools-1.3/nslg/lib/readline.c
#    284 -rw-r--r-- NewsTools-1.3/nslg/lib/writen.c
#
touch -am 1231235999 $$.touch >/dev/null 2>&1
if test ! -f 1231235999 && test -f $$.touch; then
  shar_touch=touch
else
  shar_touch=:
  echo
  echo 'WARNING: not restoring timestamps.  Consider getting and'
  echo "installing GNU \`touch', distributed in GNU File Utilities..."
  echo
fi
rm -f 1231235999 $$.touch
#
# ============= NewsTools-1.3/./README ==============
if test ! -d 'NewsTools-1.3'; then
  echo 'x - creating directory NewsTools-1.3'
  mkdir 'NewsTools-1.3'
fi
if test -f 'NewsTools-1.3/./README' && test X"$1" != X"-c"; then
  echo 'x - skipping NewsTools-1.3/./README (file already exists)'
else
  echo 'x - extracting NewsTools-1.3/./README (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'NewsTools-1.3/./README' &&
Newstools is a suite of programs developed over the years to help manage
news systems. I have several dozen tools, but limited time to package
them, so I'm releasing this work in progress on a fairly regular basis
to let people get some use from what's documented.
X
Questions and comments to newstools@tmr.com.
X
Thanks: some of these tools were developed while I was working under
contract for Prodigy Services. They have encouraged me to make them
available and are providing a fast net connection for posting.
X
Release notes:
X
09/29/95 - 1.3
X
X  Fixed nslg to include the sizes required to compile, they were moved
from the header file to the C source in a later version which is still
not stable.
X
X  Added connrate to the suite. This generates two plot files, one of
outgoing article rate to readers, one of new connects per second. This
operates against the news.notice file.
X
X  Supplied some missing PERL includes for the PERL version of artrate.
X
X  A few README files add, too.
X
09/23/95
X  This release includes nslg, the NNTP synthetic load generator. Note
that because this is under development you will want to rename the file
nslg.c as loadtest.c. With that change the program should compile and
run.
X
BUG LIST
X
X  Several people have reported that ncc does not skip .overview files
of you have NOV and news spool sharing the same hierarchy. All my
systems use NOV in it's own directory for performance (even on the same
device), but there is code to handle this case. I put checking this on
the "to-do" list. If you don't have this problem or solve it, please let
me know.
SHAR_EOF
  $shar_touch -am 0929144895 'NewsTools-1.3/./README' &&
  chmod 0644 'NewsTools-1.3/./README' ||
  echo 'restore of NewsTools-1.3/./README failed'
  shar_count="`wc -c < 'NewsTools-1.3/./README'`"
  test 1582 -eq "$shar_count" ||
    echo "NewsTools-1.3/./README: original size 1582, current size $shar_count"
fi
# ============= NewsTools-1.3/./TOC ==============
if test -f 'NewsTools-1.3/./TOC' && test X"$1" != X"-c"; then
  echo 'x - skipping NewsTools-1.3/./TOC (file already exists)'
else
  echo 'x - extracting NewsTools-1.3/./TOC (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'NewsTools-1.3/./TOC' &&
NSLG - NNTP Synthetic Load Generator
X
Allows a server to be loader with pseudo users with variable reading
behaviours. 
X
ARTRATE - plot data for incoming article rate
X
Reads the server.log file and generates X-Y data suitable for plotting
with gnuplot.
X
CONNRATE - plot connects/sec and articles read/sec
X
Generates XY files suitable for plotting with gnuplot.
X
NCC - News Consistency Checker
X
Helps find files in your spool which are not in your history database,
locate directories left over from deleted groups, finds non-article
files (like core), and general checking.
SHAR_EOF
  $shar_touch -am 0929124695 'NewsTools-1.3/./TOC' &&
  chmod 0644 'NewsTools-1.3/./TOC' ||
  echo 'restore of NewsTools-1.3/./TOC failed'
  shar_count="`wc -c < 'NewsTools-1.3/./TOC'`"
  test 574 -eq "$shar_count" ||
    echo "NewsTools-1.3/./TOC: original size 574, current size $shar_count"
fi
# ============= NewsTools-1.3/ncc/README ==============
if test ! -d 'NewsTools-1.3/ncc'; then
  echo 'x - creating directory NewsTools-1.3/ncc'
  mkdir 'NewsTools-1.3/ncc'
fi
if test -f 'NewsTools-1.3/ncc/README' && test X"$1" != X"-c"; then
  echo 'x - skipping NewsTools-1.3/ncc/README (file already exists)'
else
  echo 'x - extracting NewsTools-1.3/ncc/README (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'NewsTools-1.3/ncc/README' &&
ncc - News Consistency Checker
X
This is an evolution of "findmissing.pl" from the inn1.4unoff2 release.
I needed a higher level of checking than I was getting, so I started
diddling.
X
The original program found articles which were not in the history file
and noted them.
X
CHANGES
X
This version does not get upset with numeric directory names like
"alt.2600" and notes names of directories which are not reflected in the
active file. There is crude option parsing, but more than "is the first
arg -v?" provides. In particular there is the ability to use -v once for
a light trace, or twice (-v -v) for more information than you are likely
to find useful.
X
I also allowed for reading in the path names of history, active, etc,
from a file in the current directory. I maintain news on a number of
machines, and they are not at all compatible in directory use. Thus I
got tired of changing the source... If the file ".ncc_config" exists it
is assumed to hold the paths of the history file, spool dir, and active
file, in that order.
X
WARNING
X
This is a "proof of concept," there is a lot of work being done on this
so new versions will be coming out regularly for a few weeks. I plan to
check everything I can think of, and then selectively disable some
checks to save time.
SHAR_EOF
  $shar_touch -am 0915161695 'NewsTools-1.3/ncc/README' &&
  chmod 0644 'NewsTools-1.3/ncc/README' ||
  echo 'restore of NewsTools-1.3/ncc/README failed'
  shar_count="`wc -c < 'NewsTools-1.3/ncc/README'`"
  test 1271 -eq "$shar_count" ||
    echo "NewsTools-1.3/ncc/README: original size 1271, current size $shar_count"
fi
# ============= NewsTools-1.3/ncc/ncc ==============
if test -f 'NewsTools-1.3/ncc/ncc' && test X"$1" != X"-c"; then
  echo 'x - skipping NewsTools-1.3/ncc/ncc (file already exists)'
else
  echo 'x - extracting NewsTools-1.3/ncc/ncc (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'NewsTools-1.3/ncc/ncc' &&
#!/usr/local/bin/perl
# ncc - news consistency checker
X
$VersionID =
X  "v$Revision: 1.14 $ rev. $Date: 1995/09/15 19:36:51 $ GMT ($State: Exp $)\n";
X
# findmissing
# scan news spool, and report articles that are not in the history file.
X
# Written by Harry Bochner, 3/95
# This program may be distributed freely, but please annotate any changes,
# and please leave these comments in place.
#
# Modified 9/95 by Bill Davidsen (davidsen@tmr.com)
# Slower but more checking of non-newsgroup directories, files
# other than articles, etc.
X
# Adjust the next two lines to the local configuration.
$history = '/usr/news/history';
$spool= '/usr/spool/news';
$active = "/usr/news/active";
X
# Give -v flag for verbose progress report.
X
$verbose = 0;
# "getopts.pl" doesn't really do what I want...
#   neither does this, but it's better
while (0 <= $#ARGV && @ARGV[0] =~ /^-/) {
X  $_ = shift(@ARGV);
X  /^-v/ && (++$verbose, next);
X  /^-V/ && die $VersionID;
X  /^--/ && last;
X  die "$0: unknown option ($_)";
}
X
# if a .ncc_config file, use that instead of standard names
if (-f ".ncc_config") {
X  open(CONFIG, ".ncc_config") || die "Can't open config file";
X  ($history, $spool, $active) = (<CONFIG>);
X  chop $history; chop $spool; chop $active;
X  print STDERR "$history $spool $active\n" if $verbose;
X  close(CONFIG);
}
X
# open the output log files
open(CORE, ">ncc:coredumps") || die "$0: CORE";
open(BOGUS, ">ncc:bogus") || die "$0: BOGUS";
open(MISSING, ">ncc:missing") || die "$0: MISSING";
X
chdir($spool) || die "$0: $spool: $!";
X
# scan the active file and build a list of groups and parts
# of groups. Any other directory is bogus.
X
sub scan_active {
X  local (@actparts);
X  open(ACT, "<${active}") || die "$0: active: $!";
X  print STDERR "scanning active\n" if $verbose;
X
X  while (<ACT>) {
X    @actparts = split;
X    # if the 4th field is =other.group skip this
X    next if @actparts[3] =~ /^=/;
X    $group = shift @actparts;
X    # Set this as a group
X    @is_group{$group} = 1;
X    print STDERR "G: $group\n" if $verbose > 1;
X    # Now define hierarchy below the group
X    while ($group =~ /\./) {
X      # drop the last level
X      $group =~ s/\.[^.]+$//;
X      if (@is_hier{$group}) { $group = ""; }
X      else {
X        @is_hier{$group} = 1;
X        print STDERR "H: $group\n" if $verbose > 1;
X      }
X    }
X  }
}
X
# scan the news spool recursively, looking for articles; build the internal
# bit string representation that will be checked by check_history.
X
sub scan {
X  local($group, $path, $depth) = @_;
X  local($fn, @dirs, $min, $max, @nums, $size, $str, $dir);
X
X  $0 = "scanning $group" if $depth <= 2;
X  print STDERR "scanning spool\n" if $depth == 0;
X
X  $dir = $path || ".";
X  unless (opendir(DIR, $dir)) {
X    warn("$0: $dir: $!");
X    return;
X  }
X  print STDERR "scanning $path\n" if $verbose > 1;
X
X  $min = $max = 0;
X  while ($fn = readdir(DIR)) {
X    if (-d "${dir}/${fn}") {
X      $fullpath = $fullname = "$dir/$fn";
X      $fullname =~ s@^\./@@;
X      $fullname =~ s^/^.^g;
X      next if $fn =~ /\./ || $fn eq "lost+found";
X      unless (@is_group{$fullname} || @is_hier{$fullname}) {
X        # this dir is not part of the groups
X        print BOGUS "$fullpath\n";
X        next;
X      }
X      push(@dirs, $fn);
X      next;
X    }
X    if (!($fn =~ /[1-9][0-9]*/) && $fn ne ".overview") {
X      # any filename other than numeric is bogus (ie. core)
X      printf(CORE "%s/%s\n", $dir, $fn);
X      next;
X    }
X
X    $min = $fn if $min == 0 || $min > $fn;
X    $max = $fn if $fn > $max;
X    push(@nums, $fn);
X  }
X  closedir(DIR);
X
X  if ($min) {
X    $min{$group} = $min;
X    $size = ($max-$min+7) / 8;
X    $str = "\0" x $size;
X    foreach (@nums) {
X      vec($str, $_-$min, 1) = 1;
X    }
X    $arts{$group} = $str;
X  }
X
X  $group .= "." if $group;
X  $path  .= "/" if $path;
X  $depth++;
X  foreach $fn (@dirs) {
X    &scan("$group$fn", "$path$fn", $depth);
X  }
}
X
# Scan the history file, and clear the bit flags for all articles found there.
X
sub check_history {
X  local($arts, @arts, $group, $num);
X
X  print STDERR "scanning history\n"
X    if $verbose;
X
X  open(IN, "<$history") || die "$0: $history: $!";
X  while (<IN>) {
X    @arts = split;
X    shift @arts; shift @arts;	# skip first two fields
X    foreach (@arts) {
X      ($group, $num) = split(m,/,);
X      next unless $num;		# sanity check
X      $arts++;
X      # clear that bit
X      vec($arts{$group}, $num-$min{$group}, 1) = 0
X	if defined $min{$group} && $num >= $min{$group};
X    }
X    print STDERR "$. lines, $arts articles so far\n"
X      if $verbose && $. % 50000 == 0;
X    $0 = "history line $." if $. % 50000 == 0;
X  }
X  close(IN);
X
X  print STDERR "done reading history, $arts articles\n" if $verbose;
}
X
&scan_active;
&scan("", "", 0);
&check_history;
X
# now check all the bitstrings, and if there are any flags still set,
# report that article.
X
print STDERR "checking for missing articles\n" if $verbose;
foreach $group (sort keys %arts) {
X  $min = $min{$group};
X  $str = unpack("b*", $arts{$group});
X  delete $arts{$group}; delete $min{$group};	# recover memory?
X  next unless $str =~ /1/;
X
X  $path = "";
X  for (split(/x*/, $str)) {
X    if ($_) {
X      unless ($path) {
X	$path = $group; $path =~ s,\.,/,g;
X	$0 = "findmissing $group";
X      }
X      print MISSING "$path/$min\n";
X    }
X    $min++;
X  }
}
SHAR_EOF
  $shar_touch -am 0923070295 'NewsTools-1.3/ncc/ncc' &&
  chmod 0744 'NewsTools-1.3/ncc/ncc' ||
  echo 'restore of NewsTools-1.3/ncc/ncc failed'
  shar_count="`wc -c < 'NewsTools-1.3/ncc/ncc'`"
  test 5269 -eq "$shar_count" ||
    echo "NewsTools-1.3/ncc/ncc: original size 5269, current size $shar_count"
fi
# ============= NewsTools-1.3/artrate/README ==============
if test ! -d 'NewsTools-1.3/artrate'; then
  echo 'x - creating directory NewsTools-1.3/artrate'
  mkdir 'NewsTools-1.3/artrate'
fi
if test -f 'NewsTools-1.3/artrate/README' && test X"$1" != X"-c"; then
  echo 'x - skipping NewsTools-1.3/artrate/README (file already exists)'
else
  echo 'x - extracting NewsTools-1.3/artrate/README (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'NewsTools-1.3/artrate/README' &&
The artrate program reads the INN "server.log" file and generates an
output file or files which reflect the rate at which incoming articles
were received.
X
The artrate.pl program was a test to see if doing this in PERL would be
faster. It wasn't, at least using [ngm]awk as the AWK, by about a factor
of 4-5.
X
Example:
X
$ cd LOGFILES
$ for n in 4 3 2 1; do
>   zcat OLD/server.log.$n
> done | artrate -
Start date Aug004
Start date Aug005
Start date Aug006
Start date Aug007
Start date Aug008
$
X
Output files generated will be named artrate_xy.MMM0NN where MMM is the
month name and NN is the day of the month. A future release will use
YYNNDD notition, but there's a bug in that one right now.
X
The PERL version works the same way, but slower.
X
Tested on: AIX, Linux, SCO ODT.
Reported working on: Solaris, OSF/1
X
README 1.1 1995/09/23 11:25:48 davidsen Exp
SHAR_EOF
  $shar_touch -am 0923072595 'NewsTools-1.3/artrate/README' &&
  chmod 0444 'NewsTools-1.3/artrate/README' ||
  echo 'restore of NewsTools-1.3/artrate/README failed'
  shar_count="`wc -c < 'NewsTools-1.3/artrate/README'`"
  test 859 -eq "$shar_count" ||
    echo "NewsTools-1.3/artrate/README: original size 859, current size $shar_count"
fi
# ============= NewsTools-1.3/artrate/artrate ==============
if test -f 'NewsTools-1.3/artrate/artrate' && test X"$1" != X"-c"; then
  echo 'x - skipping NewsTools-1.3/artrate/artrate (file already exists)'
else
  echo 'x - extracting NewsTools-1.3/artrate/artrate (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'NewsTools-1.3/artrate/artrate' &&
#!/bin/sh
# artrate 1.8 1995/09/29 18:21:43 davidsen Prod
X
if [ -z "$AWK" ]; then AWK="mawk"; fi
X
# Set input file(s)
if [ $# -lt 1 ]; then
X  filelist=server.log
else
X  filelist="$*"
fi
X
# analyze
${AWK} '
X
# fractional day function (time in hundredths of an hour)
function thh(ds)
{
X  split(ds, dt, /:/)
X  sec = 3600*dt[1] + 60*dt[2] + dt[3]
X  return int(sec/36)
}
# process log lines
{
X  # convert "MMM DD" date to MMM0DD" format
X  date = substr($0, 1 ,6)
X  gsub(/ /, "0", date)
# see if the start of a new day
X  if (date != olddate) {
X    # new date, close old file if open and create new filename
X    if (length(xyfile)) {
X      close(xyfile)
X    }
X    xyfile = "artrate_xy."date
X
X    # progress report
X    print "Start date "date > "/dev/tty"
X
X    # clear count and value used as flag
X    olddate = date
X    oldtime = 0
X  }
X  # get time as 100th of an hour
X  time = thh($3)
X  # if same 100th, increment count, else process
X  # change to new time slice
X  if (time == oldtime) count++
X  else {
X    # if time or count are non-zero, output a plot point
X    if (oldtime || count) {
X      printf("%.3f %.3f\n", oldtime/100.0, count/36.0) > xyfile
X
X      # times with no data are zero rate, fill in zeros
X      # between input points
X      lapse = time - oldtime - 1
X      if (lapse < 0) lapse += 2400
X      current = oldtime + lapse
X      while (++oldtime < current) printf("%.3f 0\n", oldtime/100.0) > xyfile
X    }
X    # clear the count and value used as flag
X    oldtime = time
X    count = 1
X  }
}
X
END {
X  # if left-over count, output last point
X  if (count) printf("%.3f %.3f\n", oldtime/100.0, count/36.0) > xyfile
}' $filelist
SHAR_EOF
  $shar_touch -am 0929142195 'NewsTools-1.3/artrate/artrate' &&
  chmod 0544 'NewsTools-1.3/artrate/artrate' ||
  echo 'restore of NewsTools-1.3/artrate/artrate failed'
  shar_count="`wc -c < 'NewsTools-1.3/artrate/artrate'`"
  test 1632 -eq "$shar_count" ||
    echo "NewsTools-1.3/artrate/artrate: original size 1632, current size $shar_count"
fi
# ============= NewsTools-1.3/artrate/artrate.pl ==============
if test -f 'NewsTools-1.3/artrate/artrate.pl' && test X"$1" != X"-c"; then
  echo 'x - skipping NewsTools-1.3/artrate/artrate.pl (file already exists)'
else
  echo 'x - extracting NewsTools-1.3/artrate/artrate.pl (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'NewsTools-1.3/artrate/artrate.pl' &&
#!/usr/local/bin/perl
# artrate.pl 1.4 1995/09/29 18:33:12 davidsen Exp
X
require "Log_to_date.pl";
require "Fract_hr.pl";
X
while (<>) {
X  ($mon, $day, $timestr, $text) = split(/[ \t]+/, $_, 4);
X  $time = &Fract_hr100($timestr);
X  #
X  # Look for date rollover
X  if ($day ne $OLDday || $mon ne $OLDmon) {
X    # if valid time or count, save data point
X    if ($oldtime || $count) {
X      printf XYFILE "%.3f %.3f\n", $oldtime/100.0, $count/36.0;
X    }
X    # if the file has been defined, close it
X    close(XYFILE) if $xyfile ne "";
X
X    # convert the date to YYMMDD format, use in plot filename
X    $DS = &Log_to_date($mon,$day+0);
X    $xyfile = "artrate_xy.$DS";
X    # open the file and generate a progress report
X    open(XYFILE, ">$xyfile") || die "Can't create $xyfile";
X    print "Start date $DS\n";
X
X    # reset counts and values used as flags
X    $OLDmon = $mon; $OLDday = $day;
X    $oldtime = 0;
X    $count = 0;
X  }
X  #
X  # See if still same time, bump count of so
X  if ($time eq $oldtime) { ++$count; }
X  else {
X    # new slice, output point of time or count non-zero
X    if ($oldtime || $count) {
X      printf XYFILE "%.3f %.3f\n", $oldtime/100.0, $count/36.0;
X
X      # output zero points for any missing time slices
X      $lapse = $time - $oldtime - 1;
X      $lapse += 2400 if $lapse < 0;
X      $current = $oldtime + $lapse;
X      while (++$oldtime < $current) {
X        printf XYFILE "%.3f 0\n", $oldtime/100.0;
X      }
X    }
X    
X    # reset count and values used for flags
X    $oldtime = $time;
X    $count = 1;
X  }
}
X
# EOF processing, output last point if any
if ($count) {
X  printf XYFILE "%.3f %.3f\n", $oldtime/100.0, $count/36.0;
}
SHAR_EOF
  $shar_touch -am 0929143395 'NewsTools-1.3/artrate/artrate.pl' &&
  chmod 0544 'NewsTools-1.3/artrate/artrate.pl' ||
  echo 'restore of NewsTools-1.3/artrate/artrate.pl failed'
  shar_count="`wc -c < 'NewsTools-1.3/artrate/artrate.pl'`"
  test 1649 -eq "$shar_count" ||
    echo "NewsTools-1.3/artrate/artrate.pl: original size 1649, current size $shar_count"
fi
# ============= NewsTools-1.3/artrate/Fract_hr.pl ==============
if test -f 'NewsTools-1.3/artrate/Fract_hr.pl' && test X"$1" != X"-c"; then
  echo 'x - skipping NewsTools-1.3/artrate/Fract_hr.pl (file already exists)'
else
  echo 'x - extracting NewsTools-1.3/artrate/Fract_hr.pl (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'NewsTools-1.3/artrate/Fract_hr.pl' &&
# Fract_hr
# Fract_hr.pl 1.2 1995/09/29 18:43:51 davidsen Exp
#
#   Convert date string in HH:MM:SS to HH.FFF format
#
X
# convert to float hours to three digits (1000ths)
sub Fract_hr1k
{
X  local ($DS, $hr, $min, $sec, $val);
X  #
X  # parse inputs
X  $DS = @_[0];
X  ($hr, $min, $sec) = split(/:/, $DS);
X  #
X  # Convert and return as a string
X  $val = $hr + $min/60.0 + $sec/3600.0;
X  return sprintf("%.3f", $val);
}
X
# return time as 100ths of an hour
#   (ie. 00:30:00 is 0050)
sub Fract_hr100
{
X  local ($DS, $hr, $min, $sec, $val);
X  #
X  # parse inputs
X  $DS = @_[0];
X  ($hr, $min, $sec) = split(/:/, $DS);
X  #
X  # Convert and return as a string
X  $val = $hr + $min/60.0 + $sec/3600.0;
X  return sprintf("%.0f", $val*100);
}
TRUE;
SHAR_EOF
  $shar_touch -am 0929144395 'NewsTools-1.3/artrate/Fract_hr.pl' &&
  chmod 0444 'NewsTools-1.3/artrate/Fract_hr.pl' ||
  echo 'restore of NewsTools-1.3/artrate/Fract_hr.pl failed'
  shar_count="`wc -c < 'NewsTools-1.3/artrate/Fract_hr.pl'`"
  test 731 -eq "$shar_count" ||
    echo "NewsTools-1.3/artrate/Fract_hr.pl: original size 731, current size $shar_count"
fi
# ============= NewsTools-1.3/artrate/Log_to_date.pl ==============
if test -f 'NewsTools-1.3/artrate/Log_to_date.pl' && test X"$1" != X"-c"; then
  echo 'x - skipping NewsTools-1.3/artrate/Log_to_date.pl (file already exists)'
else
  echo 'x - extracting NewsTools-1.3/artrate/Log_to_date.pl (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'NewsTools-1.3/artrate/Log_to_date.pl' &&
# Log_to_date
# Log_to_date.pl 1.2 1995/09/29 18:44:21 davidsen Exp
#
#   Convert w netnews log file date in Mon DD format to
#   a YYMMDD format
#
X
# Convert INN log dates in "MMM DD" format yo "YYMMDD"
#   (assumes this is done within a few months of log time)
#
sub Log_to_date
{
X  local (@mon, $nmon);
X  local ($sec, $min, $hr, $day, $moy, $yr, $dow, $doy, $dst);
X  local ($Tmon, $Tday);
X
X  ($Tmon, $Tday) = @_;
X  @mon = ("Jan","Feb","Mar","Apr","May","Jun",
X    "Jul","Aug","Sep","Oct","Nov","Dec");
X  #
X  # find the numeric month (1 based)
X  for ($nmon = 0; $nmon < 12; ++$nmon) { last if $Tmon eq @mon[$nmon]; }
X  ++$nmon;
X  #
X  # Test for end of year rollover, log data in Dec, analysis next year
X  ($sec, $min, $hr, $day, $moy, $yr, $dow, $doy, $dst) = (localtime);
X  if ( $nmon == 12 && $mon < 12 ) { --$yr; }
X  #
X  # generate and return a numeric date string
X  return sprintf("%02d%02d%02d", $yr, $nmon, $Tday);
}
TRUE;
SHAR_EOF
  $shar_touch -am 0929144495 'NewsTools-1.3/artrate/Log_to_date.pl' &&
  chmod 0444 'NewsTools-1.3/artrate/Log_to_date.pl' ||
  echo 'restore of NewsTools-1.3/artrate/Log_to_date.pl failed'
  shar_count="`wc -c < 'NewsTools-1.3/artrate/Log_to_date.pl'`"
  test 931 -eq "$shar_count" ||
    echo "NewsTools-1.3/artrate/Log_to_date.pl: original size 931, current size $shar_count"
fi
# ============= NewsTools-1.3/connrate/README ==============
if test ! -d 'NewsTools-1.3/connrate'; then
  echo 'x - creating directory NewsTools-1.3/connrate'
  mkdir 'NewsTools-1.3/connrate'
fi
if test -f 'NewsTools-1.3/connrate/README' && test X"$1" != X"-c"; then
  echo 'x - skipping NewsTools-1.3/connrate/README (file already exists)'
else
  echo 'x - extracting NewsTools-1.3/connrate/README (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'NewsTools-1.3/connrate/README' &&
This is connrate, a script used to process your news.notice file and
generate XY plot file suitable for running with gnuplot.
X
The "mawk" interpreter is MUCH faster than most vendor versions of AWK,
and outperforms gawk on most machines, as well. Usually 50-200% faster
than the PERL scripts when available.
X
Looking at today's data
=======================
X
$ cd YOUR_LOG_DIR
$ connrate
starting date Sep020
$
X
If you roll your logs after midnight, part of today's data is on
yesterday's compressed log. To get all of today's data:
X
$ cd YOUR_LOG_DIR
$ zcat OLD/news.notice.1 | connrate - news.notice
start date Sep019
start date Sep020
$
X
The data for yesterday will be incomplete.
X
Generating plots of several days
================================
X
$ cd YOUR_LOG_DIR
$ for n in 4 3 2 1; do zcat OLD/news.notice.$n; done |
> connrate -
X
This concatenates your logs for several days, and handles the case where
you roll the logs over at a time other than exactly midnight. Note that
if you do have an offset log rollover (most sites do) the plot data in
the first and last days plotfiles will be incomplete.
SHAR_EOF
  $shar_touch -am 0929130095 'NewsTools-1.3/connrate/README' &&
  chmod 0644 'NewsTools-1.3/connrate/README' ||
  echo 'restore of NewsTools-1.3/connrate/README failed'
  shar_count="`wc -c < 'NewsTools-1.3/connrate/README'`"
  test 1108 -eq "$shar_count" ||
    echo "NewsTools-1.3/connrate/README: original size 1108, current size $shar_count"
fi
# ============= NewsTools-1.3/connrate/connrate ==============
if test -f 'NewsTools-1.3/connrate/connrate' && test X"$1" != X"-c"; then
  echo 'x - skipping NewsTools-1.3/connrate/connrate (file already exists)'
else
  echo 'x - extracting NewsTools-1.3/connrate/connrate (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'NewsTools-1.3/connrate/connrate' &&
#!/bin/sh
# connrate 1.6 1995/09/29 17:51:19 davidsen Prod
#	connrate - generate connect rate logging
X
# Set input file(s)
if [ $# -lt 1 ]; then
X  filelist="news.notice"
else
X  filelist="$*"
fi
X
# analyze
mawk '
X
# fractional day function (time in hundredths of an hour)
function thh(ds)
{
X  split(ds, dt, /:/)
X  sec = 3600*dt[1] + 60*dt[2] + dt[3]
X  return int(sec/36)
}
X
# get process if from field
function getpid(field)
{
X  field = sub(/^.*\[/, "", field)
X  field = sub(/\]/, "", field)
X  return field
}
X
# process the connect rates
/connect/ {
X  # convert the "MMM DD" date to a string, "MMM0DD"
X  date = substr($0, 1 ,6)
X  gsub(/ /, "0", date)
X
X  # if this is a new date since the last output, start
X  # a new output file with the date name
X  if (date != olddate) {
X    # if old files exist, close them
X    if (length(xyfile)) {
X      close(xyfile)
X      close(xxyfile)
X    }
X
X    # progress report
X    print "Start date "date > "/dev/tty"
X
X    # generate new filenames, files created on write
X    xyfile = "connrate_xy."date
X    xxyfile = "readrate_xxy."date
X
X    # clear counters and flags
X    olddate = date
X    oldtime = 0
X  }
X
X  # set time in hundredths of an hour
X  time = thh($3)
X
X  # if still in the same 100th, add to the count
X  # else transition to a new 100th
X  if (time == oldtime) count++
X  else {
X    # except for startup (time == count == 0) output
X    # the data to the plotfile
X    if (oldtime || count) printf("%.3f %.2f\n", oldtime/100.0, count/36.0) > xyfile
X
X    # reset the counters and flag
X    oldtime = time
X    count = 1
X  }
}
X
# look for article vs e.t. info
/exit articles/ {
X  # save the article count ref by PID
X  procid = getpid($5)
X  arts[procid] = $9
}
X
# this line contains the end time, match it up
# with the number of arts
/times user/ {
X  # skip if we dont have a file defined yet
X  if (length(xxyfile) == 0) next
X
X  # see if article data for this process
X  if (arts[procid]) {
X    # generate the start and end time, plot will be
X    # start, end, data format
X    procid = getpid($5)
X    time_end = thh($3)
X    time_srt = time_end - ($13/36)
X
X    # output the data
X    printf("%.3f %.6f %.3f\n", time_srt/100.0, $13/3600.0, arts[procid]/$13) > xxyfile
X
X    # release the memory
X    free arts[procid]
X  }
}' $filelist
SHAR_EOF
  $shar_touch -am 0929135295 'NewsTools-1.3/connrate/connrate' &&
  chmod 0544 'NewsTools-1.3/connrate/connrate' ||
  echo 'restore of NewsTools-1.3/connrate/connrate failed'
  shar_count="`wc -c < 'NewsTools-1.3/connrate/connrate'`"
  test 2262 -eq "$shar_count" ||
    echo "NewsTools-1.3/connrate/connrate: original size 2262, current size $shar_count"
fi
# ============= NewsTools-1.3/nslg/inet.h ==============
if test ! -d 'NewsTools-1.3/nslg'; then
  echo 'x - creating directory NewsTools-1.3/nslg'
  mkdir 'NewsTools-1.3/nslg'
fi
if test -f 'NewsTools-1.3/nslg/inet.h' && test X"$1" != X"-c"; then
  echo 'x - skipping NewsTools-1.3/nslg/inet.h (file already exists)'
else
  echo 'x - extracting NewsTools-1.3/nslg/inet.h (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'NewsTools-1.3/nslg/inet.h' &&
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#if defined(__linux__)
X  /* for some reason Linux lack select.h and defines in sys/time */
X  #include <sys/time.h>
#else
X  #include <sys/select.h>
#endif
#include <netinet/in.h>
#include <arpa/inet.h>
X
char *pname;
X
#define MAXLINE 512
X
int  readline(int fd,char *ptr,int maxlen);
int  writen(int fd,char *ptr,int nbytes);
SHAR_EOF
  $shar_touch -am 0828161995 'NewsTools-1.3/nslg/inet.h' &&
  chmod 0644 'NewsTools-1.3/nslg/inet.h' ||
  echo 'restore of NewsTools-1.3/nslg/inet.h failed'
  shar_count="`wc -c < 'NewsTools-1.3/nslg/inet.h'`"
  test 429 -eq "$shar_count" ||
    echo "NewsTools-1.3/nslg/inet.h: original size 429, current size $shar_count"
fi
# ============= NewsTools-1.3/nslg/loadtest/Makefile ==============
if test ! -d 'NewsTools-1.3/nslg/loadtest'; then
  echo 'x - creating directory NewsTools-1.3/nslg/loadtest'
  mkdir 'NewsTools-1.3/nslg/loadtest'
fi
if test -f 'NewsTools-1.3/nslg/loadtest/Makefile' && test X"$1" != X"-c"; then
  echo 'x - skipping NewsTools-1.3/nslg/loadtest/Makefile (file already exists)'
else
  echo 'x - extracting NewsTools-1.3/nslg/loadtest/Makefile (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'NewsTools-1.3/nslg/loadtest/Makefile' &&
# Makefile 1.2 1995/09/23 10:56:02 davidsen Stable
X
LIBS		= -llineio
CFLAGS		= -O2
LFLAGS		= -L../lib -s
OBJS		= loadtest.o
X
X.c.o:
X	$(CC) -c $(CFLAGS) $?
X
loadtest:	$(OBJS)
X	$(CC) -o loadtest $(LFLAGS) loadtest.o $(LIBS)
X
link:
X	$(CC) -o loadtest $(LFLAGS) loadtest.o $(LIBS)
#
#	o/s dependent or optimized rules
#
# Linux - test for Slackware 2.3 w/ 1.2.13 kernel
linux:
X	$(MAKE) loadtest CFLAGS="-O2 -fno-strength-reduce"
# AIX - tested for AIX 4.1.2
aix:
X	$(MAKE) loadtest
#
#	overhead stuff
#
clean:
X	rm -f loadtest *.o
tidy:
X	rm -f *.o
SHAR_EOF
  $shar_touch -am 0923074595 'NewsTools-1.3/nslg/loadtest/Makefile' &&
  chmod 0444 'NewsTools-1.3/nslg/loadtest/Makefile' ||
  echo 'restore of NewsTools-1.3/nslg/loadtest/Makefile failed'
  shar_count="`wc -c < 'NewsTools-1.3/nslg/loadtest/Makefile'`"
  test 541 -eq "$shar_count" ||
    echo "NewsTools-1.3/nslg/loadtest/Makefile: original size 541, current size $shar_count"
fi
: || echo 'restore of NewsTools-1.3/nslg/loadtest/nslg.c failed'
echo 'End of part 1, continue with part 2'
exit 0