w8sdz@tacom-emh1.army.mil (Keith Petersen) (08/06/89)
------------------------------------------------------------------------
Accessing the SIMTEL20 archives from BITNET
Updated 28 April 1989
------------------------------------------------------------------------
This document describes a method for users of systems connected to
BITNET to obtain files from selected archives kept at the MILNET node
WSMR-SIMTEL20.ARMY.MIL. The information applies specifically to the
file servers installed at NDSUVM1 and RPIECS (formerly RPICICGE). (A
similar service is provided to EARN by a set of servers collectively
known as "TRICKLE"; those servers accept similar, but not identical,
commands.)
------------------------------------------------------------------------
Background
------------------------------------------------------------------------
The US Army maintains a huge collection of public domain (and
"shareware") software and information on WSMR-SIMTEL20.ARMY.MIL, a
DECsystem-20 machine running the Tops-20 operating system at White Sands
Missle Range, New Mexico. The collection covers a spectrum of
interests, including files of interest to CP/M and MSDOS users.
The collection is "open to the public"; anyone may obtain copies
of the files using the Internet file transfer protocol, FTP. The bad
news is that FTP is not a protocol available over BITNET. BITNET users
can not directly obtain files from the SIMTEL20 collection. The good
news is that there are several file servers located throughout BITNET
that will accept requests for SIMTEL20 files and perform the appropriate
file transfer on the requestor's behalf. However, please understand
that...
The BITNET servers that provide access to the SIMTEL20 archives
have no affiliation with the US Army nor with White Sands Missle
Range. Also, the BITNET servers are made available in the true
spirit of volunteerism (both of the institutions where they are
installed and of the individuals that support them) without any
outside sponsorship for the service.
Also...
Due to the large number of files available, neither the archive
archive maintainers at SIMTEL20 nor the server maintainers in
BITNET can possibly attempt to validate the proper operation of
the various programs. When a program bug is reported to an
archive maintainer, immediate action is taken to either correct
the error or remove the offending program from the archives.
Still, users must understand that archive programs are offered
AS-IS, and the archive maintainers and server maintainers
specifically disclaim any liability should these programs
malfunction or cause damage, incidental or otherwise. When
testing ANY software, be certain that all information stored on
disk is backed-up before you start so that you can recover if files
are damaged or erased. This is particularly true if you have a
hard disk, in which case malfunctions can be spectacularly
disasterous.
The BITNET servers provide access to the following subset of the
software archives residing at SIMTEL20:
CPM Software and information for CP/M system users. Contributions
are gathered from a variety of sources, including the members
of the Info-CPM electronic mail discussion group. This archive
is updated very frequently.
MSDOS Software and information for PC-DOS and MSDOS system users.
Contributions are gathered from a variety of sources, including
the members of the Info-IBMPC electronic mail discussion group.
This archive is updated very frequently.
PC-BLUE Software and information for PC-DOS and MSDOS system users.
The archive contains the files distributed by the PC-Blue Users
group. New files are added as they become available.
SIGM Software and information for CP/M system users. The archive
contains the files distributed by the SIG/M Users group. New
files are added as they become available.
MISC Software and information for miscellaneous systems (mostly
large systems like IBM/370 and DEC VAX). Contributions are
gathered from a variety of sources.
------------------------------------------------------------------------
SIMTEL20 path names, file names and file types
------------------------------------------------------------------------
The Tops-20 operating system supports a hierarchical file system
structure not unlike that found on Unix, Vax/VMS, and even MSDOS
systems. At SIMTEL20, the software collection is divided into
individual archives by category, each with its own file system
directory. The archives are subdivided by topic into sub-directories.
The following example is a typical path name for a SIMTEL20 file:
PD:<MSDOS.STARTER>UUDECODE.BAS
Here, PD is the name of the disk where the archives reside. (Well,
actually it is an alias for a group of disks PD1, PD2, and so on.)
MSDOS is the name for the archive; STARTER is a sub-directory containing
generally useful programs and information. UUDECODE.BAS is the name for
one such file in the STARTER sub-directory.
File names of files in the SIMTEL20 archives generally conform to
the conventions of the target system (e.g. CP/M and MSDOS). From the
example above, UUDECODE.BAS is a uudecode program written in BASIC.
(MSDOS.STARTER also contains UUDECODE.PAS and UUDECODE.C, versions of
the same program written in Pascal and C, respectively.) The model of
"name.extension" should be familiar to most. Extensions of DOC, HEX,
INF and ASM are associated with ASCII text files; COM and EXE, with
binary executables. However, in an effort to reduce the online storage
required by the files, and to organize software into packages, most of
the files at SIMTEL20 have been through some flavor of data compaction
and/or library utility. The file extensions used for such beasts may
be less familiar to some:
ARC a collection of related files compacted and collected together
into a single package, and called an ARChive. An un-archive
utility is needed to extract individual files from the package.
ARK exactly the same as ARC. ARK is used in preference to ARC in
the CP/M archives.
LBR a collection of related files compacted and collected together
into a single package, and called a LiBRary. An un-library
utility is needed to extract individual files from the package.
xQx a file that has been compacted using a Huffman encoding method
known as sQueezing. The extension is derived from that of the
original file with the letter Q substituted in the middle. (An
ASM file that was squeezed would be stored as AQM.) An
un-squeeze utility is needed to recover the original file data.
xZx the same as xQx except that an LZW-variant method known as
crunching has been used. An un-crunch utility is needed to
recover the original file data.
Most of the software for MSDOS systems are stored in the ARC format.
All four formats are used in the software for CP/M systems. (ARK and
ARC represent the same thing, but ARK is the more commonly used name.)
Only a few "first-time-user" type files (like UUDECODE.BAS) are stored
in their raw form. The section below titled "Getting Started" gives
some guidance about handling them.
------------------------------------------------------------------------
Using the BITNET Servers
------------------------------------------------------------------------
In the United States, there are two BITNET servers that provide
access to the SIMTEL20 archives:
LISTSERV@NDSUVM1 North Dakota State University.
LISTSERV@RPIECS Rensselaer Polytechnic Institute.
--------Note--------Note--------Note--------Note--------Note-----------
In Europe, there are many EARN servers. However, the information
provided here is specifically for the BITNET servers. The EARN servers
have a similar user interface and may accept the same set of commands,
but information about using them is beyond the scope of this document.
The locations of the EARN servers and the principle contact person for
each are:
TRICKLE@TREARN ("Turgut Kalfaoglu" <TURGUT@TREARN>)
TRICKLE@IMIPOLI ("Marco Gandolfi" <MARCO@IMIPOLI>)
TRICKLE@BANUFS11 ("Michel Daulie" <DAULIE@BANUFS11>)
TRICKLE@AWIWUW11 ("Gustaf Neumann" <NEUMANN@AWIWUW11>)
TRICKLE@DB0FUB11 ("Wolfram Fassbender" <EARNIE@DB0FUB11>)
TRICKLE@EB0UB011 ("Oriol Robert" <ZCCBORR@EB0UB011>)
--------Note--------Note--------Note--------Note--------Note-----------
Requests may be sent to a server as RFC822-style mail. The
commands to the server must appear in the body of the message, not the
Subject: line. The server uses the From: header to determine how to
address the files to be returned. The From: header must therefore
specify a valid, reachable network address from the server's point of
view. Mail received from outside BITNET, particularly from UUCP, often
have unusable return addresses.
Requests may also be sent as interactive BITNET messages if your
system supports such a facility. On an IBM system, this service is
provided by the TELL command, as in
TELL LISTSERV AT nodename servercommand
The server does enforce some limits on how much can be requested
by whom and from where. Requests from EARN are not accepted; they must
be delivered to the nearest TRICKLE server in EARN. For others, the
server restricts how many files and how many bytes of data a user may
request per day. It also restricts how many files and how many bytes
a host system may request per day. The limits are changed on occasion, they are
but they are in the neighborhood of
3 files/user/day 10 files/host/day
100 Kbytes/user/day 300 Kbytes/host/day
There are some files that are larger than the per-day limit for a user
(or host) would permit, so the server does allow the first request from
a user (or host) on any given day to exceed the byte limit. Also, the
"host" in this context means what appears after the at-sign (@) in the
return address. Mailed requests that pass through a gateway usually
appear to be from that gateway host, and so the server applies its host
limits accordingly.
--------Note--------Note--------Note--------Note--------Note-----------
Although requests are sent to the LISTSERV address, the requests
are actually processed by userid TRICKLE. Files sent back to you will
be from TRICKLE. Do not let this mislead you, though: Your requests
must go to LISTSERV, and not to TRICKLE at either NDSUVM1 or RPIECS.
In EARN, LISTSERV is not used, and TRICKLE does accept requests
from users. NOT IN BITNET. Your requests must go to LISTSERV.
--------Note--------Note--------Note--------Note--------Note-----------
THE /PDDIR COMMAND
The /PDDIR command is used to list the names of files that match some
pattern. The command has several forms. They are:
/PDDIR
/PDDIR PD:<directory>
/PDDIR PD:<directory.subdirectory>filename.ext age
The first form lists the names of all the archives known to the
server. At present these are CPM, SIGM, PC-BLUE, MSDOS, and MISC. The
second form lists the names of all the subdirectories in a particular
archive. (The directory name must be one of the known archives: CPM,
SIGM, etc.) The third form lists the names of files in the archive that
match a particular pattern. The age parameter limits how old a file in
the archive may be and still be considered. If omitted, the default is
30, meaning 30 days old. The directory name must be one of CPM, SIGM,
PC-BLUE, MSDOS, or MISC. The subdirectory, filename, and ext may
include asterisks ('*') a "wild-card" characters. The following are
examples.
/PDDIR PD:<MSDOS> --Lists subdirectories in the MSDOS archive.
/PDDIR PD:<SIGM.*>*.* --Lists files added in the last 30 days
/PDDIR PD:<MISC.VAXVMS>*.* 9999 --Lists VAX/VMS related files.
/PDDIR PD:<CPM.*>UUDECODE*.* 9999 --Lists uudecoders for CP/M.
THE /PDGET COMMAND
The /PDGET command is used to request a specific file. No pattern-
matching is allowed. The syntax for this command is as follows:
/PDGET format simtel.filename encoding
The format specifies how the file is to be transmitted. Allowed
values are NETDATA, PUNCH, and MAIL.
NETDATA -- suitable for transfer to BITNET hosts that can accept
files in IBM Netdata format.
PUNCH -- suitable for transfer to BITNET hosts that can accept
files but cannot decode the Netdata format. Files
are sent as 80-byte card-images.
MAIL -- suitable for transfer to hosts that can accept only
mail or are accessible to BITNET only through gateways.
Large files sent via mail are split into several
smaller files that the recipient must reassemble.
If the format is omitted, NETDATA is assumed for BITNET hosts and MAIL
for all others.
The encoding specifies any special translation for the file data:
ASIS -- suitable for hosts that can receive binary data. The
file is sent exactly as it is stored on the server:
binary images of the file data. ASIS may be used
only with format NETDATA.
UUENCODE -- suitable for hosts that cannot receive binary data.
The file is sent uuencoded.
TRANSLATE -- suitable for any host, but only when the file actually
represents readable text. The file is translated to
EBCDIC. (If you are on an ASCII machine, then your
system should automatically translate to ASCII when
the file arrives.) TRANSLATE applied to a binary file
is treated as if UUENCODE were specified.
If no encoding is specified, then ASIS is assumed for NETDATA, and
UUENCODE for the others.
--------Note--------Note--------Note--------Note--------Note-----------
In the actual archives at SIMTEL20 there are a few files stored in
the top-level directory. (For example, PD:<MSDOS>FILES.IDX is a file
listing the names of all the files in the subdirecotries of the MSDOS
archive.) The design of the BITNET server does not permit access to any
of these files. However, since the files at the top-level directory
generally contain directory information, the need for them is superceded
by the /PDDIR command.
--------Note--------Note--------Note--------Note--------Note-----------
------------------------------------------------------------------------
Getting Started
------------------------------------------------------------------------
Before all else, something you absolutely must have available is
a method for getting files from your host system to you micro computer.
It would be preferable if this method included support for transferring
binary files as well as normal text files. If you do not already have
a way to communicate with your host and transfer files, consider getting
the appropriate Kermit implementations available from the KERMSRV file
server at CUVMA.
Once that minor detail has been addressed, then you should consider
what additional utility programs you will need or that will be helpful.
Most files are in an archive format, so you will need a de-archive
utility or two. You may also need a uudecode program, depending on your
ability to receive binary files on your host and your ability to
download binary files to you micro computer.
This last point requires some explanation. The server stores all
files from SIMTEL20 as-is in 128 byte sector image blocks. They are
bit-for-bit identical to how they should appear on your micro computer.
The server makes no attempt to interpret the files; it simply sends them
on demand out through BITNET. BITNET, though, is fundamentally an
EBCDIC network, and your micro computer is fundamentally an ASCII
machine. This gives rise to two places along the path from server to
micro where the file data might be misinterpreted or corrupted.
If your host system is ASCII-based (as are most non-IBM systems)
it will translate incoming BITNET files from EBCDIC to ASCII. If your
host is EBCDIC-based, your communications software will translate files
you download from EBCDIC to ASCII. But the files from the server do not
contain EBCDIC data. You must either find a way to disable the
translations or encode the data in such a way that the original file
can be recovered.
There are suggestions given later for specific host machines to
disable the translations. For now assume data encoding is required.
You can ask the server to send files in encoded from. If you request
encoding, the file is encoded using a technique know as uuencoding.
Uuencoded data is preserved through most of the EBCDIC/ASCII
translations the file might encounter. So, all you need is a program
for you micro computer that decodes a uuencoded file.
There are several decoders available from SIMTEL20. The only
problem is how do you get the program to your micro computer. Catch-22.
Well, you can ask the server to send ASCII text files in translated
form. If you request translation, a file is first translated to EBCDIC
before it is sent. This is not recommended as a standard option since
there may be some loss of information, but for getting started it may
be essential.
If you need a program for CP/M to decode uuencoded files, send the
following command to the server:
/PDGET PD:<CPM.STARTER-KIT>UUDECODE.HEX TRANSLATE
The file contains the CP/M hex data for the program. Download it. Use
the CP/M commands LOAD and SAVE to create an executable program. You
should end up with UUDECODE.COM, the desired program.
If you need a program for MSDOS to decode uuencoded files, send the
following commands to the server:
/PDGET PD:<MSDOS.STARTER>UUDECODE.xxx TRANSLATE
/PDGET PD:<MSDOS.STARTER>UUENCDEC.DOC TRANSLATE
Replace "xxx" with either BAS, C, or PAS depending on which source
language you would prefer (BASIC, C, or Pascal, respectively).
Next, you should consider requesting which ever of the following
files you feel appropriate for your micro computer system:
For PC-DOS and MSDOS machines:
PD:<MSDOS.STARTER>ARCE40C.COM Un-archive utility.
PD:<MSDOS.STARTER>ARCE40C.DOC ..and the documentation.
PD:<MSDOS.STARTER>UUDECODE.EXE Compiled uudecode utility
For CP/M machines:
PD:<CPM.STARTER-KIT>DELBR11.COM Un-library utility.
PD:<CPM.STARTER-KIT>UNARC.COM-Z80 Un-archive utility, Z-80 only.
PD:<CPM.STARTER-KIT>UNARCA.COM-8080 Un-archive utility.
PD:<CPM.STARTER-KIT>UNARC.DOC ..and the documentation.
PD:<CPM.STARTER-KIT>UNCR-Z80.COM Un-crunch utility, Z-80 only.
PD:<CPM.STARTER-KIT>UNCR8080.COM Un-crunch utility.
PD:<CPM.STARTER-KIT>UNCR8080.DOC ..and the documentation.
PD:<CPM.STARTER-KIT>USQ120.COM Un-squeeze utility.
PD:<CPM.STARTER-KIT>USQ120.DOC ..and the documentation.
There are many other useful utilities in these and other archive
directories. Remember, though, if you need the server to UUENCODE the
files you request, you should explicitly ask for it. Also, some of the
programs listed above may be replaced by newer versions. (For example,
ARCE40C.COM replaced the earlier ARCE31C.COM.) If you have trouble with
the server claiming "file not found", use the /PDDIR command to list the
the appropriate directory.
You may find two other files useful. PD:<MSDOS.FILEDOCS>SIMIBM.ARC
and PD:<CPM.FILEDOCS>SIMCPM.ARC contain one-line descriptions for many
of the other files in their respective archives. Not all files are
described, but it does contain enough valuable information to help you
find other software.
IBM System Users.
If your host is an IBM system running either VM or MVS, you can avoid
the need for uuencoding. Files received from BITNET will not be
translated, since the IBM is an EBCDIC machine. Most down-load methods
support binary transfer, so you can defeat the translation that would
otherwise take place there. For example, with CMS Kermit the command
SET FILE BINARY is all the is required before initiating a download.
If you are using a 3270 emulator and IND$FILE for file transfers, by
default no translation takes place.
VAX/VMS Users.
If your host is a DEC VAX system running VMS, with Jnet as your network
software, you can avoid the need for uuencoding. You can tell the Jnet
software to bypass the usual EBCDIC/ASCII translation, but there are a
few additional steps needed before downloading a file.
* Receive the file with the Jnet command RECEIVE/BINARY. The
BINARY modifier suppresses the normal EBCDIC/ASCII translation.
For the sake of discussion, assume that the file is now named
SOFTWARE.FIL. This file, as received, is almost correct; but
there may be an error in how VMS interprets the records.
* Generate an FDL file for SOFTWARE.FIL using the command
ANALYZE/RMS/FDL SOFTWARE.FIL
* Edit the FDL file with the command
EDIT/FDL SOFTWARE
Examine the CARRIAGE_CONTROL setting. Change it to NONE. Exit
from the editor.
* Use the edited FDL to correct carriage control interpretation
errors in the original SOFTWARE.FIL.
CONVERT/FDL=SOFTWARE.FDL SOFTWARE.FIL FIXED_SOFTWARE.FIL
* Download the FIXED_SOFTWARE.FIL as a binary file using any
reliable means. (For VAX Kermit, use the SET FILE TYPE BINARY
command before starting the download.)
------------------------------------------------------------------------
Common Problems
------------------------------------------------------------------------
Q. I downloaded this program to my micro, but when I run it, my machine
hangs (or I get the message "Out of Memory" or ...).
A. Either the file became corrupted in transit (perhaps one of those
nasty EBCDIC/ASCII translations), or the file was uuencoded and you
have not decoded it.
Q. I downloaded an archive to my micro, but the de-archive utility
would not process it. I get messages like "File not an archive" or
"Cannot extract member".
A. Same comments as above.
Q. I really, really need to get these special files that I absolutely
must have, but the server limits how much I can request per day. Is
there any way I can get around these limits for this one special case
A. No.
Q. I am trying to get a file from the (top-level of the) MSDOS
directory. /PDDIR won't list it, /PDGET claims it can't find it,
but I know it is there.
A. It may well be there at SIMTEL20. However, the BITNET server is not
capable of handling any request for a file from the top-level of an
archive. Generally, though, the files stored at the top level list
the contents of the archive. The /PDDIR command can be used to get
a directory listing.
Q. I have been requesting this same file repeatedly. Each time the
server tells me my request has been "queued for processing," then a
few days later it sends me a message that it has "abandoned" my
request. Other requests it has been handling just fine.
A. The server does maintain a large "cache" of recently requested files.
Many requests are satisfied from this cache. However, for all the
rest the server must fetch it directly from SIMTEL20 using the
Internet file transfer protocol, FTP. "Directly" really is not
all that direct since the path between server and SIMTEL20 includes
many network segments and gateways. To complete a transfer, an
error-free connection must be maintained for the duration of the FTP
transaction. This is not always possible, whether it be from some
dysfunction along the path or heavy network load. The server will
retry a failed FTP transaction, but if it continues to fail, the
server eventually gives up.
Q. I keep sending requests to the server. I never hear anything back.
A. The server responses in some way to everything it receives. Your
requests may not be arriving, possibly because you are miskeying the
server's network address. Perhaps you are sending your requests to
TRICKLE rather than LISTSERV. Your requests may be arriving, but
with an unusable "From:" field in the mail header, so the response
never gets back to you.
Q. Gee, this public-domain/shareware stuff is the greatest. How do I
go about adding my own contributions?
A. Remember, the archives are actually kept at SIMTEL20. The servers
only provide access to them. Contributions must be sent to the
people there. Send an electronic mail message to:
"Keith Petersen" <W8SDZ@WSMR-SIMTEL20.ARMY.MIL>
Be sure to tell him what it is you have and what it is for. After
he verifies he does not already have it, you and he can negotiate
methods for submitting the software.
Q. Hey, I have FTP on my system. How do I go about connecting to
either RPIECS or NDSUVM1 and fetching the SIMTEL20 files?
A. Two points about the servers have been missed. (1) The servers are
there to provide access to the SIMTEL20 archives for people WITHOUT
FTP capability. Users on hosts that do support FTP have the
privilege of connecting directly to WSMR-SIMTEL20.ARMY.MIL. (2) The
servers do not actually have a complete collection of the archives;
only a varying set of recently requested files are stored locally.
If you have FTP access to the Internet, connect to
WSMR-SIMTEL20.ARMY.MIL and use anonymous login.
Q. Who do I contact with suggestions or unsolvable problems?
A. Depending on which server you normally use:
"John Fisher" <FISHER@RPIECS>
"Marty Hoag" <INFO@NDSUVM1>
DO NOT send your comment or question about the server to the people
at WSMR-SIMTEL20.ARMY.MIL. However, if you wish to report program
bug or something similar about a SIMTEL20 file, you may send it to
"Keith Petersen" <W8SDZ@WSMR-SIMTEL20.ARMY.MIL>
Keith Petersen