jimr@maths.su.oz.au (Jim Richardson) (08/01/90)
Here is the final version of the Open Letter. Many thanks again to everyone
who has provided ideas and improvements.
A call for signatures will appear as a separate article shortly.
--
Jim Richardson
Department of Pure Mathematics, University of Sydney, NSW 2006, Australia
Internet: jimr@maths.su.oz.au ACSNET: jimr@maths.su.oz FAX: +61 2 692 4534
--
<start of letter>
OPEN LETTER TO HP
HEWLETT-PACKARD, APOLLO CUSTOMERS, AND THE INTERNET
Background: The Internet and Usenet
-----------------------------------
The "Internet" is a very large computer network using the TCP/IP protocols and
extending over much of the world. Among the services it provides are
electronic mail, file transfer via the FTP protocol, and "network news", a
conferencing system somewhat akin to the internal HP notes groups.
Network news is divided up into more than a thousand "newsgroups", each
covering a different discussion area. The computers within and beyond the
Internet which carry network news, the data links between those computers, and
the community of people who read the news, are collectively known as "USENET".
In mid-1990, the number of machines receiving Usenet articles was estimated at
over 26,000 and the total number of people who read some articles at
1,109,000 [reference: "USENET READERSHIP SUMMARY REPORT FOR JUN 90", Brian
Reid (reid@decwrl.DEC.COM), article <1990Jul2.154231.28843@wrl.dec.com> in Use-
net newsgroup news.lists, 2 July 1990].
Two of the newsgroups carry discussions among users of Apollo and HP com-
puters: these groups are called comp.sys.apollo and comp.sys.hp. It is
estimated that in June 1990, these groups respectively had 27,000 and 23,000
readers worldwide (with an unknown amount of overlap); in that month, there
were 166 articles totaling 250 kilobytes in comp.sys.apollo, and 215 articles
totaling 323 kilobytes in comp.sys.hp [reference: "USENET Readership report
for Jun 90", Brian Reid, article <1990Jul2.154323.29469@wrl.dec.com> in group
news.lists, 2 July 1990].
Many but by no means all of the machines connected to the Internet are in
educational or research institutions. Numerous large and small commercial and
industrial customers of HP also have Internet access.
Recent discussions in comp.sys.apollo
-------------------------------------
In June and July 1990, a discussion took place in comp.sys.apollo on safe
methods for distributing information about security bugs to system admin-
istrators. This led on, first, to comments on the difficulty many Apollo
sites have experienced in obtaining copies of patch tapes from HP, and thence
to wide-ranging criticisms of other aspects of HP's services to its cus-
tomers. Many Apollo system managers and users who had become increasingly
frustrated with HP's unresponsiveness began to realize that their problem or
their site or their national HP office was not an isolated case: Apollo
customers all over the world felt they were encountering similar difficulties.
The tone of the discussions was by no means all negative. Many people say:
I love my Apollo, BUT ...
Aspects of Apollos and Domain/OS that received particular praise included:
token ring; the intuitive, object-oriented, "automagically" networked file
system; ACLs (access control lists -- though these seem to be a matter of
taste); the Display Manager; DDE (Domain Distributed Debugger); dynamic swap
space instead of a permanently reserved partition; ability to run both BSD and
SysV Unix simultaneously.
There were also many favorable comments on superhuman *unofficial* efforts to
help customers by many individual HP staff, including those who are already
willing to post news articles on Usenet.
The big BUT: customer service problems
--------------------------------------
Unfortunately, many Apollo users have formed the impression that Apollo sup-
port has become the poor relation within HP. While customers with HP hardware
seem not unhappy with service (at least, not vocally so), large numbers of
Apollo customers are very dissatisfied. We feel that the problems are not the
fault of the hard-working Apollo support staff such as do exist: the cause
instead lies in insufficient *numbers* of such staff, and inadequate resource
provision to enable them to carry out their functions, combined with corporate
over-caution which hinders experiments with new approaches.
It might be argued, as far as educational customers are concerned, that a
lesser standard of service is appropriate, given the discount levels such
customers receive and the low levels of support contract they generally
choose. But this would be to ignore the fact that many of us feel we are not
even receiving the modest level of support for which we have contracted.
Moreover, at least some of us are reasonably sophisticated system admin-
istrators, able to deal with most manual-reference questions ourselves, only
referring *really* knotty questions to HP for advice, and sometimes able to
provide solutions that HP has not discovered itself. Remember too that while
educational institutions may not have enormous buying power themselves, the
students who use their machines will be the next generation of computer pur-
chasers in industry and commerce.
Here are some of the service problems which have been discussed among readers
of comp.sys.apollo:
* security issues, such as the open initial protections on Domain/OS direc-
tories and dangerous utilities, daemons' need for unprotected directories,
absence of restrictions normally present in certain Unix system control tools,
and downright bugs affecting security (details are deliberately omitted
here!); the most serious worries are the absence of any kind of "security
alert bulletin" by which HP could rapidly notify Apollo sites of security bugs
and fixes as they are found, and -- at least until very recently -- apparent
lack of any attempt whatever to provide such notification;
* although every Domain/OS manual solicits APRs (Apollo Product Reports) in
the introductory "Problems, Questions and Suggestions" section, and although
the on-line manual page for the mkapr utility gives an Internet electronic
mail address for APR submission and states "Customer Services will acknowledge
all product reports received", the reality is that APRs are not an effective
way of reporting problems: the email addresses often bounce; when email is
successful, and when APRs are submitted by more traditional means, acknow-
ledgements are often not received (especially for APRs from outside the US);
substantive responses to APRs never appear, or turn up after months or years;
even then the responses often fail to solve the original problem;
* unsuitability of telephone support for more technical questions (e.g., bug
reports involving tracebacks): while telephone support can be excellent for
simple questions and for new or naive users, an electronic mail service with
fast turn-around would be preferred by many experienced programmers and system
managers, could provide as effective a shield for back-room HP support staff
as the telephone service, and would more effectively handle time-zone differ-
ence problems for customers outside the US;
* long delays in delivery of software and hardware, e.g., NFS for SR10.2.p and
third-party sourced products like tape drives and Mathematica -- perhaps made
worse for non-US customers by poor communications between HP local and head
offices;
* perceived failure of procedures for timely automatic delivery of software
upgrades to customers with maintenance contracts;
* difficulties in finding out what bugs are known and what patches are avail-
able;
* difficulty and delays in obtaining patch tapes, even in cases where it was
apparent that a particular patch might be relevant to an operating system bug
which was causing problems at a particular site -- as noted above, such delays
are very dangerous in the case of security-relevant patches;
* "closed" policy on HP modifications to publicly available software: for ex-
ample, HP has changed the FTP daemon ftpd to handle Apollo filetypes, but has
not released the source changes, so that they cannot be incorporated into an
enhanced version of ftpd independently developed by Sam Shen at Berkeley which
permits anonymous ftp without the need for the chroot system call (missing
from Domain/OS).
A problem in the development rather than service area which has attracted much
comment is HP's reluctance to incorporate such well-known tools as perl and
GNU Emacs into its own operating system releases.
In the next three sections, we present three requests to HP the granting of
which we feel would go a long way towards solving the problems described
above.
Resource allocation within HP
-----------------------------
REQUEST 1: Hewlett-Packard should urgently take whatever policy decisions
and actions are necessary to ensure that the Apollo Systems Division has
resources available to it for support operations which are proportional to
those provided on the HP side. Personnel levels and organization in the
Apollo Division support sector need to be reviewed and improved.
It is clear from the complaints of Apollo users discussed above, contrasted
with the apparent relative happiness of HP/UX customers, that either Apollo
Division support resources available per customer are smaller than those for
HP/UX, or the resources in the Apollo Division are not being deployed effect-
ively enough to satisfy customers' perceptions of their need for support.
We will not presume to advise HP on the managerial details involved in
implementing Request 1. The brevity and simplicity of this section should be
taken as emphasizing our belief in the importance of this request and its
clear justification on grounds of equity.
Use of the Internet
-------------------
The Internet already allows users to support each other technically -- not to
mention in terms of morale. Although this certainly means great savings to HP,
it happens *in spite of* HP, not *in co-operation with* HP.
We propose that HP take steps to provide better services to its customers on
the Internet by using the Internet in an *organized and official* way. We
believe that this will not only benefit users, but will increase efficiency
and feedback and reduce duplication for HP as well.
REQUEST 2: We would like HP to set up an INTERNET LIAISON UNIT, with
sufficient staff, resources and authority to carry out the following oper-
ations:
a) Organize and oversee a new, effective system whereby APRs (and their
HP/UX counterpart) can be submitted by electronic mail, acknowledged
by return email, and then answered by email within a reasonable time
-- say two months. If a longer time is required, a progress report
should be sent say monthly.
b) Arrange for a mail gateway between the Internet and internal HP mail,
or publicize any such gateway already in existence, so that cus-
tomers on the Internet can conveniently communicate with their local
service people as an optional alternative to telephone service.
(The gateway could have a filter or alias mechanism so that other
internal HP staff would not be bothered with mail from outside if
they did not want it.)
c) Monitor the comp.sys.apollo and comp.sys.hp Usenet newsgroups, and
where appropriate arrange for responses to be provided from relevant
experts within HP. It would also be very worthwhile to organize and
maintain periodic news postings containing answers to frequently
asked questions.
A further task for the unit would be to set up a public archive accessible
from the Internet. We feel that this is important -- and perhaps contro-
versial -- enough to be stated as a separate request with a detailed
explanation.
A public archive
----------------
REQUEST 3: HP should establish a public archive on a new or existing
company machine connected to the Internet, to make customer support
materials available via FTP. The archive should be operated by the
Internet Liaison Unit, and should include at least the following:
a) an index of the latest version numbers of all supported software,
and which operating system versions they work under;
b) a regularly updated index of known bugs, e.g., a list of APRs, per-
haps similar to an on-line version of the "HP-UX Software
Release/Status Bulletin" series, with workarounds if available;
c) release notes for all current and beta versions of all supported
software (note that this would cover some bug reports; it would also
encourage customers to obtain upgraded versions);
d) a complete set of all current patches, say in compressed wbak or
"tar A" format, with release notes (see caveats about security and
major patches below);
e) source of HP modifications to generally available programs such as
ftpd and sendmail: this would allow us to keep those programs up to
date, enhance them, and send them back to HP (a good start in this
direction is /domain_examples/tcp/gated).
Note that the archive would be PUBLIC, so available to all Internet users
instead of being restricted to service contract holders. There are rival
precedents for this. SUN maintains a public archive of patches for FTP from
the Internet host uunet.uu.net. Apple has an extensive archive of development
materials, including system software, sample code, technical notes and docu-
mentation, for FTP from apple.com, and apparently there are plans to expand
this service. HP's own recent offering of a supported X11R4 server for HP/UX
via FTP from hpcvaaz.hp.com is a useful first step.
A public archive, available to all members of the Internet, is probably nec-
essary because of prohibition by the National Science Foundation and other
funding bodies on use of the Internet for commercial gain.
Such a service to all owners of HP equipment would probably not reduce the
number who take out service contracts appreciably: a contract would still be
needed to obtain software upgrades, and this is probably the greatest incent-
ive for a contract at most sites. (There would be no expectation that HP
would continue to support obsolete versions of software through patches or bug
lists in the archive.) Furthermore, the existence of a public archive would
demonstrate HP's commitment to its customers and to high standards, and would
represent a major enhancement to the attractiveness of HP products.
Our request for a public archive is not a novel one: see the column "The In-
side Track: On HP-UX patches" by Dave Taylor in The HP Chronicle of May 1990
for a persuasive argument in favor of such an archive from an HP/UX point of
view.
Two caveats about patches
-------------------------
Patches which address security problems should be included in the archive if
this can be done without causing security problems in itself. System managers
of machines connected to the Internet must be particularly conscious of secu-
rity questions, and have great interest in receiving security-related patches
as rapidly as possible. However, security-patch release notes should never
include any details of the problems which they aim to correct: such details
can themselves lead to breaches of security at unpatched sites. The release
notes should simply state that the patch in question is security-related and
urgent. A brief Usenet news item should announce the addition of each new
security patch to the archive.
Also note that some patches such as /sau*/domain_os may be so substantial as
to amount to de facto upgrades, and might need to be excluded from a public
archive. Excluded patches should be distributed by traditional means, but
much more effectively and rapidly than at present.
Benefits of using the Internet
------------------------------
Of course, the existence of support facilities on the Internet would not
reduce HP's traditional obligations to its service-contract customers, espec-
ially those without Internet connections (although it could be possible for
such customers to gain access by other means, for example, dial-up UUCP --
however unsatisfactory this method would be considered by Internet users).
But we believe that rapid and efficient dissemination of information via the
Internet would *save* HP money by cutting out duplicated effort.
For example, many of the more routine questions at present addressed to the
telephone service hotlines would be avoided if systems administrators had
on-line FTP and/or Usenet access to answers to frequently asked questions and
information such as usage tips and bug workarounds. This would free HP tele-
phone service and other support resources which could be applied to more rapid
or deeper investigation of unusual problems and subtler bugs.
Distribution of patches by FTP should be much cheaper and more efficient than
copying and physically distributing patch tapes.
HP would enhance its competitiveness by providing public services on the
Internet as other companies do already. Apart from the FTP archives mentioned
above, note these examples: IBM employees frequently post articles answering
queries in Usenet newsgroups such as comp.unix.aix and comp.sys.ibm.pc.rt, in
many cases without even a disclaimer; Wolfram Research accepts bug reports and
provides technical support for Mathematica via electronic mail; Adobe Systems
solicits input via electronic mail from users and developers on desired capa-
bilities in new printer driver software [reference: "new PostScript Printer
Driver for Macintosh from Adobe Systems", article <4394@adobe.UUCP> in news-
groups comp.lang.postscript et al, 24 July 1990].
The existence of a large community of HP machine users, programmers, and
systems administrators that HP could communicate with would foster a "global
village" style of interaction. This interaction would benefit both parties by
the dissemination of valuable first-hand information between the actual users
and HP engineers and management. This information could be used to improve
planning for the needs of the user community and the priorities of HP, and
would also promote better customer-vendor relationships. The result would be
increased sales and more widespread acceptance of HP machines.
More efficient support from HP for academic and research customers through
network connections would be a step in the direction of the microeconomic
reform and the closer ties between industry and academia for which government
and business are calling world-wide.
We believe that our Requests 2 and 3 can be satisfied by HP in accordance with
commercialism guidelines applying on the major publicly funded networks. If
it transpires that network use by HP of the kind we are requesting requires
justification under the guidelines, then we will be happy to collaborate with
HP in preparing a case for submission to the relevant network authorities.
Disclaimer
----------
This document was written collectively, and while all signatories support its
aims and general thrust, not everyone is necessarily in complete agreement on
the details of all points. The views expressed are those of individuals, and
do not in general represent official policy of the institutions or companies
of which the signatories are members. (This should not be taken as a license
to discount those views, however: in the long run the individual views of com-
puter users and system managers tend to affect or even determine institutional
computing policy and purchasing decisions.)
Conclusion
----------
We applaud the naming of HP's latest major computer line as the "HP *Apollo*
9000 Series 400", and note with approval that the sales literature gives
Domain/OS equal weight to HP/UX, and the Apollo DN10000 as much prominence as
the HP Model 635. As Apollo users we are pleased to see this concrete illus-
tration of HP's commitment to the continuation and flourishing of a strong
Apollo Systems Division.
The slogan for the new HP Apollo machines describes them as the "first work-
stations to combine the innovation of Apollo with the quality and reliability
of HP". We believe that at present your company's customer services combine
the conservatism of HP with the organizational haphazardness of Apollo; if
this perception continues, then all your other efforts may be in vain.
We hope that Hewlett-Packard will accept this critique in the same positive
spirit with which we have prepared it, and will act quickly to fulfill our
three requests. Individual replies are not expected: indeed, we will know
that we have been heard when we see a response from HP as a news article in
comp.sys.apollo on the Usenet.
1 August 1990
<end of letter>wwm@pmsmam.uucp (Bill Meahan) (08/01/90)
BRAVO! A well-worded request for HP to join the networked society. *****BUT******* BOO! HISS! All of us non-academic, non DoD, non "research" folk who have no connection to the Internet are left out in the cold. Unless HP is convinced to ALSO provide UUCP access for "us" the full benefit to the ENTIRE HP community will never come close to being realized. Come on, folks! Be INclsuive, not EXclusive. -- Bill Meahan WA8TZG uunet!mailrus!umich!pmsmam!wwm I don't speak for Ford - the PR department does that! Any attempt at wit is liable to offend _somebody_!