[comp.binaries.ibm.pc.d] Getting files from Simtel20 via the BITNET mail servers

w8sdz@WSMR-SIMTEL20.ARMY.MIL (Keith Petersen) (04/16/89)

------------------------------------------------------------------------

                Accessing the SIMTEL20 archives from BITNET

                                                Updated 30 March 1989
------------------------------------------------------------------------

     This document describes a method for users of systems connected to
BITNET to obtain files from selected archives kept at the MILNET node
WSMR-SIMTEL20.ARMY.MIL.  The information applies specifically to the
file servers installed at NDSUVM1 and RPIECS (formerly RPICICGE).  (A
similar service is provided to EARN by a set of servers collectively
known as "TRICKLE"; those servers accept similar, but not identical,
commands.)

------------------------------------------------------------------------
Background
------------------------------------------------------------------------

     The US Army maintains a huge collection of public domain (and
"shareware") software and information on WSMR-SIMTEL20.ARMY.MIL, a
DECsystem-20 machine running the Tops-20 operating system at White Sands
Missle Range, New Mexico.  The collection covers a spectrum of
interests, including files of interest to CP/M and MSDOS users.

     The collection is "open to the public"; anyone may obtain copies
of the files using the Internet file transfer protocol, FTP.  The bad
news is that FTP is not a protocol available over BITNET.  BITNET users
can not directly obtain files from the SIMTEL20 collection.  The good
news is that there are several file servers located  throughout BITNET
that will accept requests for SIMTEL20 files and perform the appropriate
file transfer on the requestor's behalf.  However, please understand
that...

     The BITNET servers that provide access to the SIMTEL20 archives
     have no affiliation with the US Army nor with White Sands Missle
     Range.  Also, the BITNET servers are made available in the true
     spirit of volunteerism (both of the institutions where they are
     installed and of the individuals that support them) without any
     outside sponsorship for the service.
Also...
     Due to the large number of files available, neither the archive
     archive maintainers at SIMTEL20 nor the server maintainers in
     BITNET can possibly attempt to validate the proper operation of
     the various programs.  When a program bug is reported to an
     archive maintainer, immediate action is taken to either correct
     the error or remove the offending program from the archives.
     Still, users must understand that archive programs are offered
     AS-IS, and the archive maintainers and server maintainers
     specifically disclaim any liability should these programs
     malfunction or cause damage, incidental or otherwise.  When
     testing ANY software, be certain that all information stored on
     disk is backed-up before you start so that you can recover if files
     are damaged or erased.  This is particularly true if you have a
     hard disk, in which case malfunctions can be spectacularly
     disasterous.

     The BITNET servers provide access to the following subset of the
software archives residing at SIMTEL20:

CPM     Software and information for CP/M system users.  Contributions
        are gathered from a variety of sources, including the members
        of the Info-CPM electronic mail discussion group.  This archive
        is updated very frequently.

MSDOS   Software and information for PC-DOS and MSDOS system users.
        Contributions are gathered from a variety of sources, including
        the members of the Info-IBMPC electronic mail discussion group.
        This archive is updated very frequently.

PC-BLUE Software and information for PC-DOS and MSDOS system users.
        The archive contains the files distributed by the PC-Blue Users
        group.  New files are added as they become available.

SIGM    Software and information for CP/M system users.  The archive
        contains the files distributed by the SIG/M Users group.  New
        files are added as they become available.

MISC    Software and information for miscellaneous systems (mostly
        large systems like IBM/370 and DEC VAX).  Contributions are
        gathered from a variety of sources.



------------------------------------------------------------------------
SIMTEL20 path names, file names and file types
------------------------------------------------------------------------

     The Tops-20 operating system supports a hierarchical file system
structure not unlike that found on Unix, Vax/VMS, and even MSDOS
systems.  At SIMTEL20, the software collection is divided into
individual archives by category, each with its own file system
directory.  The archives are subdivided by topic into sub-directories.
The following example is a typical path name for a SIMTEL20 file:

          PD:<MSDOS.STARTER>UUDECODE.BAS

Here, PD is the name of the disk where the archives reside.  (Well,
actually it is an alias for a group of disks PD1, PD2, and so on.)
MSDOS is the name for the archive; STARTER is a sub-directory containing
generally useful programs and information.  UUDECODE.BAS is the name for
one such file in the STARTER sub-directory.

     File names of files in the SIMTEL20 archives generally conform to
the conventions of the target system (e.g. CP/M and MSDOS).  From the
example above, UUDECODE.BAS is a uudecode program written in BASIC.
(MSDOS.STARTER also contains UUDECODE.PAS and UUDECODE.C, versions of
the same program written in Pascal and C, respectively.)  The model of
"name.extension" should be familiar to most.  Extensions of DOC, HEX,
INF and ASM are associated with ASCII text files; COM and EXE, with
binary executables.  However, in an effort to reduce the online storage
required by the files, and to organize software into packages, most of
the files at SIMTEL20 have been through some flavor of data compaction
and/or library utility.  The file extensions used for such beasts may
be less familiar to some:

ARC   a collection of related files compacted and collected together
      into a single package, and called an ARChive.  An un-archive
      utility is needed to extract individual files from the package.

ARK   exactly the same as ARC.  ARK is used in preference to ARC in
      the CP/M archives.

LBR   a collection of related files compacted and collected together
      into a single package, and called a LiBRary.  An un-library
      utility is needed to extract individual files from the package.

xQx   a file that has been compacted using a Huffman encoding method
      known as sQueezing.  The extension is derived from that of the
      original file with the letter Q substituted in the middle.  (An
      ASM file that was squeezed would be stored as AQM.)  An
      un-squeeze utility is needed to recover the original file data.

xZx   the same as xQx except that an LZW-variant method known as
      crunching has been used.  An un-crunch utility is needed to
      recover the original file data.


Most of the software for MSDOS systems are stored in the ARC format.
All four formats are used in the software for CP/M systems.  (ARK and
ARC represent the same thing, but ARK is the more commonly used name.)
Only a  few "first-time-user" type files (like UUDECODE.BAS) are stored
in their raw form.  The section below titled "Getting Started" gives
some guidance about handling them.

------------------------------------------------------------------------
Using the BITNET Servers
------------------------------------------------------------------------

     In the United States, there are two BITNET servers that provide
access to the SIMTEL20 archives:

          LISTSERV@NDSUVM1   North Dakota State University.
          LISTSERV@RPIECS    Rensselaer Polytechnic Institute.

--------Note--------Note--------Note--------Note--------Note-----------
     In Europe, there are many EARN servers.  However, the information
provided here is specifically for the BITNET servers.  The EARN servers
have a similar user interface and may accept the same set of commands,
but information about using them is beyond the scope of this document.
The locations of the EARN servers and the principle contact person for
each are:

          TRICKLE@TREARN    ("Turgut Kalfaoglu"   <TURGUT@TREARN>)
          TRICKLE@IMIPOLI   ("Marco Gandolfi"     <MARCO@IMIPOLI>)
          TRICKLE@BANUFS11  ("Michel Daulie"      <DAULIE@BANUFS11>)
          TRICKLE@AWIWUW11  ("Gustaf Neumann"     <NEUMANN@AWIWUW11>)
          TRICKLE@DB0FUB11  ("Wolfram Fassbender" <EARNIE@DB0FUB11>)
          TRICKLE@EB0UB011  ("Oriol Robert"       <ZCCBORR@EB0UB011>)
--------Note--------Note--------Note--------Note--------Note-----------


     Requests may be sent to a server as RFC822-style mail.  The
commands to the server must appear in the body of the message, not the
Subject: line.  The server uses the From: header to determine how to
address the files to be returned.  The From: header must therefore
specify a valid, reachable network address from the server's point of
view.  Mail received from outside BITNET, particularly from UUCP, often
have unusable return addresses.

     Requests may also be sent as interactive BITNET messages if your
system supports such a facility.  On an IBM system, this service is
provided by the TELL command, as in

          TELL LISTSERV AT nodename  servercommand

     The server does enforce some limits on how much can be requested
by whom and from where.  Requests from EARN are not accepted; they must
be delivered to the nearest TRICKLE server in EARN.  For others, the
server restricts how many files and how many bytes of data a user may
request per day.  It also restricts how many files and how many bytes
a host system may request per day.  The limits are changed on occasion, they are
but they are in the neighborhood of

     3 files/user/day            10 files/host/day
   100 Kbytes/user/day          300 Kbytes/host/day

There are some files that are larger than the per-day limit for a user
(or host) would permit, so the server does allow the first request from
a user (or host) on any given day to exceed the byte limit.  Also, the
"host" in this context means what appears after the at-sign (@) in the
return address.  Mailed requests that pass through a gateway usually
appear to be from that gateway host, and so the server applies its host
limits accordingly.

--------Note--------Note--------Note--------Note--------Note-----------
     Although requests are sent to the LISTSERV address, the requests
are actually processed by userid TRICKLE.  Files sent back to you will
be from TRICKLE.  Do not let this mislead you, though:  Your requests
must go to LISTSERV, and not to TRICKLE at either NDSUVM1 or RPIECS.

     In EARN, LISTSERV is not used, and TRICKLE does accept requests
from users.  NOT IN BITNET.  Your requests must go to LISTSERV.
--------Note--------Note--------Note--------Note--------Note-----------


THE /PDDIR COMMAND

The /PDDIR command is used to list the names of files that match some
pattern.  The command has several forms.  They are:

        /PDDIR
        /PDDIR  PD:<directory>
        /PDDIR  PD:<directory.subdirectory>filename.ext  age

     The first form lists the names of all the archives known to the
server.  At present these are CPM, SIGM, PC-BLUE, MSDOS, and MISC.  The
second  form lists the names of all the subdirectories in a particular
archive.  (The directory name must be one of the known archives: CPM,
SIGM, etc.)  The third form lists the names of files in the archive that
match a particular pattern.  The age parameter limits how old a file in
the archive may be and still be considered.  If omitted, the default is
30, meaning 30 days old.  The directory name must be one of CPM, SIGM,
PC-BLUE, MSDOS, or MISC.  The subdirectory, filename, and ext may
include asterisks ('*') a "wild-card" characters.  The following are
examples.

        /PDDIR PD:<MSDOS>  --Lists subdirectories in the MSDOS archive.
        /PDDIR PD:<SIGM.*>*.*   --Lists files added in the last 30 days
        /PDDIR PD:<MISC.VAXVMS>*.* 9999  --Lists VAX/VMS related files.
        /PDDIR PD:<CPM.*>UUDECODE*.* 9999  --Lists uudecoders for CP/M.


THE /PDGET COMMAND

The /PDGET command is used to request a specific file.  No pattern-
matching is allowed.  The syntax for this command is as follows:

        /PDGET  format  simtel.filename  encoding

     The format specifies how the file is to be transmitted.  Allowed
values are NETDATA, PUNCH, and MAIL.

     NETDATA  -- suitable for transfer to BITNET hosts that can accept
                 files in IBM Netdata format.

     PUNCH    -- suitable for transfer to BITNET hosts that can accept
                 files but cannot decode the Netdata format.  Files
                 are sent as 80-byte card-images.

     MAIL     -- suitable for transfer to hosts that can accept only
                 mail or are accessible to BITNET only through gateways.
                 Large files sent via mail are split into several
                 smaller files that the recipient must reassemble.

If the format is omitted, NETDATA is assumed for BITNET hosts and MAIL
for all others.

     The encoding specifies any special translation for the file data:

     ASIS     -- suitable for hosts that can receive binary data.  The
                 file is sent exactly as it is stored on the server:
                 binary images of the file data.  ASIS may be used
                 only with format NETDATA.

     UUENCODE -- suitable for hosts that cannot receive binary data.
                 The file is sent uuencoded.

     TRANSLATE -- suitable for any host, but only when the file actually
                 represents readable text.  The file is translated to
                 EBCDIC.  (If you are on an ASCII machine, then your
                 system should automatically translate to ASCII when
                 the file arrives.)  TRANSLATE applied to a binary file
                 is treated as if UUENCODE were specified.

If no encoding is specified, then ASIS is assumed for NETDATA, and
UUENCODE for the others.

--------Note--------Note--------Note--------Note--------Note-----------
     In the actual archives at SIMTEL20 there are a few files stored in
the top-level directory.  (For example, PD:<MSDOS>FILES.IDX is a file
listing the names of all the files in the subdirecotries of the MSDOS
archive.)  The design of the BITNET server does not permit access to any
of these files.  However, since the files at the top-level directory
generally contain directory information, the need for them is superceded
by the /PDDIR command.
--------Note--------Note--------Note--------Note--------Note-----------

------------------------------------------------------------------------
Getting Started
------------------------------------------------------------------------

     Before all else, something you absolutely must have available is
a method for getting files from your host system to you micro computer.
It would be preferable if this method included support for transferring
binary files as well as normal text files.  If you do not already have
a way to communicate with your host and transfer files, consider getting
the appropriate Kermit implementations available from the KERMSRV file
server at CUVMA.

     Once that minor detail has been addressed, then you should consider
what additional utility programs you will need or that will be helpful.
Most files are in an archive format, so you will need a de-archive
utility or two.  You may also need a uudecode program, depending on your
ability to receive binary files on your host and your ability to
download binary files to you micro computer.

     This last point requires some explanation.  The server stores all
files from SIMTEL20 as-is in 128 byte sector image blocks.  They are
bit-for-bit identical to how they should appear on your micro computer.
The server makes no attempt to interpret the files; it simply sends them
on demand out through BITNET.  BITNET, though, is fundamentally an
EBCDIC network, and your micro computer is fundamentally an ASCII
machine.  This gives rise to two places along the path from server to
micro where the file data might be misinterpreted or corrupted.

     If your host system is ASCII-based (as are most non-IBM systems)
it will translate incoming BITNET files from EBCDIC to ASCII.  If your
host is EBCDIC-based, your communications software will translate files
you download from EBCDIC to ASCII.  But the files from the server do not
contain EBCDIC data.  You must either find a way to disable the
translations or encode the data in such a way that the original file
can be recovered.

     There are suggestions given later for specific host machines to
disable the translations.  For now assume data encoding is required.
You can ask the server to send files in encoded from.  If you request
encoding, the file is encoded using a technique know as uuencoding.
Uuencoded data is preserved through most of the EBCDIC/ASCII
translations the file might encounter.  So, all you need is a program
for you micro computer that decodes a uuencoded file.

     There are several decoders available from SIMTEL20.  The only
problem is how do you get the program to your micro computer.  Catch-22.
Well, you can ask the server to send ASCII text files in translated
form.  If you request translation, a file is first translated to EBCDIC
before it is sent.  This is not recommended as a standard option since
there may be some loss of information, but for getting started it may
be essential.

     If you need a program for CP/M to decode uuencoded files, send the
following command to the server:

        /PDGET PD:<CPM.STARTER-KIT>UUDECODE.HEX  TRANSLATE

The file contains the CP/M hex data for the program.  Download it.  Use
the CP/M commands LOAD and SAVE to create an executable program.  You
should end up with UUDECODE.COM, the desired program.

     If you need a program for MSDOS to decode uuencoded files, send the
following commands to the server:

        /PDGET PD:<MSDOS.STARTER>UUDECODE.xxx  TRANSLATE
        /PDGET PD:<MSDOS.STARTER>UUENCDEC.DOC  TRANSLATE

Replace "xxx" with either BAS, C, or PAS depending on which source
language you would prefer (BASIC, C, or Pascal, respectively).

     Next, you should consider requesting which ever of the following
files you feel appropriate for your micro computer system:

For PC-DOS and MSDOS machines:
     PD:<MSDOS.STARTER>ARCE31C.COM       Un-archive utility.
     PD:<MSDOS.STARTER>ARCE31C.DOC       ..and the documentation.
     PD:<MSDOS.STARTER>UUDECODE.EXE      Compiled uudecode utility

For CP/M machines:
     PD:<CPM.STARTER-KIT>DELBR11.COM     Un-library utility.
     PD:<CPM.STARTER-KIT>UNARC.COM-Z80   Un-archive utility, Z-80 only.
     PD:<CPM.STARTER-KIT>UNARCA.COM-8080 Un-archive utility.
     PD:<CPM.STARTER-KIT>UNARC.DOC       ..and the documentation.
     PD:<CPM.STARTER-KIT>UNCR-Z80.COM    Un-crunch utility, Z-80 only.
     PD:<CPM.STARTER-KIT>UNCR8080.COM    Un-crunch utility.
     PD:<CPM.STARTER-KIT>UNCR8080.DOC    ..and the documentation.
     PD:<CPM.STARTER-KIT>USQ120.COM      Un-squeeze utility.
     PD:<CPM.STARTER-KIT>USQ120.DOC      ..and the documentation.

There are many other useful utilities in these and other archive
directories.  Remember, though, if you need the server to UUENCODE the
files you request, you should explicitly ask for it.

     You may find two other files useful.  PD:<MSDOS.FILEDOCS>SIMIBM.ARC
and PD:<CPM.FILEDOCS>SIMCPM.ARC contain one-line descriptions for many
of the  other files in their respective archives.  Not all files are
described, but it does contain enough valuable information to help you
find other software.


IBM System Users.

If your host is an IBM system running either VM or MVS, you can avoid
the need for uuencoding.  Files received from BITNET will not be
translated, since the IBM is an EBCDIC machine.  Most down-load methods
support binary transfer, so you can defeat the translation that would
otherwise take place there.  For example, with CMS Kermit the command
SET FILE BINARY is all the is required before initiating a download.
If you are using a 3270 emulator and IND$FILE for file transfers, by
default no translation takes place.


VAX/VMS Users.

If your host is a DEC VAX system running VMS, with Jnet as your network
software, you can avoid the need for uuencoding.  You can tell the Jnet
software to bypass the usual EBCDIC/ASCII translation, but there are a
few additional steps needed before downloading a file.

    *  Receive the file with the Jnet command RECEIVE/BINARY.  The
       BINARY modifier suppresses the normal EBCDIC/ASCII translation.
       For the sake of discussion, assume that the file is now named
       SOFTWARE.FIL.  This file, as received, is almost correct; but
       there may be an error in how VMS interprets the records.

    *  Generate an FDL file for SOFTWARE.FIL using the command

          ANALYZE/RMS/FDL  SOFTWARE.FIL

    *  Edit the FDL file with the command

          EDIT/FDL  SOFTWARE

       Examine the CARRIAGE_CONTROL setting.  Change it to NONE.  Exit
       from the editor.

    *  Use the edited FDL to correct carriage control interpretation
       errors in the original SOFTWARE.FIL.

         CONVERT/FDL=SOFTWARE.FDL  SOFTWARE.FIL  FIXED_SOFTWARE.FIL

    *  Download the FIXED_SOFTWARE.FIL as a binary file using any
       reliable means.  (For VAX Kermit, use the SET FILE TYPE BINARY
       command before starting the download.)


------------------------------------------------------------------------
Common Problems
------------------------------------------------------------------------

Q. I downloaded this program to my micro, but when I run it, my machine
   hangs (or I get the message "Out of Memory" or ...).

A. Either the file became corrupted in transit (perhaps one of those
   nasty EBCDIC/ASCII translations), or the file was uuencoded and you
   have not decoded it.


Q. I downloaded an archive to my micro, but the de-archive utility
   would not process it.  I get messages like "File not an archive" or
   "Cannot extract member".

A. Same comments as above.


Q. I really, really need to get these special files that I absolutely
   must have, but the server limits how much I can request per day.  Is
   there any way I can get around these limits for this one special case

A. No.


Q. I am trying to get a file from the (top-level of the) MSDOS
   directory.  /PDDIR won't list it, /PDGET claims it can't find it,
   but I know it is there.

A. It may well be there at SIMTEL20.  However, the BITNET server is not
   capable of handling any request for a file from the top-level of an
   archive.  Generally, though, the files stored at the top level list
   the contents of the archive.  The /PDDIR command can be used to get
   a directory listing.


Q. I have been requesting this same file repeatedly.  Each time the
   server tells me my request has been "queued for processing," then a
   few days later it sends me a message that it has "abandoned" my
   request.  Other requests it has been handling just fine.

A. The server does maintain a large "cache" of recently requested files.
   Many requests are satisfied from this cache.  However, for all the
   rest the server must fetch it directly from SIMTEL20 using the
   Internet file transfer protocol, FTP.  "Directly" really is not
   all that direct since the path between server and SIMTEL20 includes
   many network segments and gateways.  To complete a transfer, an
   error-free connection must be maintained for the duration of the FTP
   transaction.  This is not always possible, whether it be from some
   dysfunction along the path or heavy network load.  The server will
   retry a failed FTP transaction, but if it continues to fail, the
   server eventually gives up.


Q. I keep sending requests to the server.  I never hear anything back.

A. The server responses in some way to everything it receives.  Your
   requests may not be arriving, possibly because you are miskeying the
   server's network address.  Perhaps you are sending your requests to
   TRICKLE rather than LISTSERV.  Your requests may be arriving, but
   with an unusable "From:" field in the mail header, so the response
   never gets back to you.


Q. Gee, this public-domain/shareware stuff is the greatest.  How do I
   go about adding my own contributions?

A. Remember, the archives are actually kept at SIMTEL20.  The servers
   only provide access to them.  Contributions must be sent to the
   people there.  Send an electronic mail message to:

     "Keith Petersen"  <W8SDZ@WSMR-SIMTEL20.ARMY.MIL>

   Be sure to tell him what it is you have and what it is for.  After
   he verifies he does not already have it, you and he can negotiate
   methods for submitting the software.


Q. Hey, I have FTP on my system.  How do I go about connecting to
   either RPIECS or NDSUVM1 and fetching the SIMTEL20 files?

A. Two points about the servers have been missed.  (1) The servers are
   there to provide access to the SIMTEL20 archives for people WITHOUT
   FTP capability.  Users on hosts that do support FTP have the
   privilege of connecting directly to WSMR-SIMTEL20.ARMY.MIL.  (2) The
   servers do not actually have a complete collection of the archives;
   only a varying set of recently requested files are stored locally.
   If you have FTP access to the Internet, connect to
   WSMR-SIMTEL20.ARMY.MIL and use anonymous login.


Q. Who do I contact with suggestions or unsolvable problems?

A. Depending on which server you normally use:

     "John Fisher"     <FISHER@RPIECS>
     "Marty Hoag"      <INFO@NDSUVM1>

   DO NOT send your comment or question about the server to the people
   at WSMR-SIMTEL20.ARMY.MIL.  However, if you wish to report program
   bug or something similar about a SIMTEL20 file, you may send it to

     "Keith Petersen" <W8SDZ@WSMR-SIMTEL20.ARMY.MIL>