[comp.sys.hp] Netpower: OPEN LETTER to HP, Draft #2

jimr@maths.su.oz.au (Jim Richardson) (07/19/90)

Here is the second draft of the open letter to HP.  Many thanks to everyone
who has emailed and/or posted comments and suggestions.  Again, I have not
given due credit ... but then everybody's signatures will be on the final
letter! :-)

Passages marked !! are comments by me, not part of the open letter itself.
There are substantial additions and alterations, including the change in
viewpoint from that of all HP customers to that of Apollo customers only,
which was explained in <1990Jul14.085541.18550@metro.ucc.su.OZ.AU> ("Netpower:
Apollo/HP contrast").  I've made comments on a few of the other changes.

The letter is something like six A4 pages long now, so further additions are
probably not desirable.  I'd be very grateful to receive any corrections, or
notes of glaring omissions, for incorporation into the final version.

<Start of draft>

                     OPEN LETTER TO HP: DRAFT #2
                                   
          HEWLETT-PACKARD, APOLLO CUSTOMERS, AND THE INTERNET


Background: The Internet and Usenet
-----------------------------------

The "Internet" is a very large computer network using the TCP/IP protocols
and extending over much of the world.  Among the services it provides are
electronic mail, file transfer via the FTP protocol, and "network news", a
conferencing system somewhat akin to the internal HP notes groups.

Network news is divided up into approaching a thousand "newsgroups", each
covering a different discussion area.  The computers within and beyond the
Internet which carry network news, the data links between those computers,
and the community of people who read the news, are collectively known as
"USENET".  In mid-1990, the number of machines receiving Usenet articles was
estimated at over 26,000 and the total number of people who read some articles
at 1,109,000.  [ Reference: USENET READERSHIP SUMMARY REPORT FOR JUN 90, Brian
Reid (reid@decwrl.DEC.COM), article <1990Jul2.154231.28843@wrl.dec.com> in
Usenet newsgroup news.lists, 2 July 1990 ]

Two of the newsgroups carry discussions among users of Apollo and HP
computers: these groups are called comp.sys.apollo and comp.sys.hp.  It is
estimated that in June 1990, these groups respectively had 27,000 and 23,000
readers worldwide (with an unknown amount of overlap); in that month, there
were 166 articles totaling 250 kilobytes in comp.sys.apollo, and 215 articles
totaling 323 kilobytes in comp.sys.hp.  [ Reference: USENET Readership report
for Jun 90, Brian Reid, article <1990Jul2.154323.29469@wrl.dec.com> in
news.lists, 2 July 1990 ] 

Many but by no means all of the machines connected to the Internet are in
educational or research institutions.  Numerous large and small commercial
and industrial customers of HP also have Internet access.

!!  Maybe we need a catchy phrase to excite the HP PR department here ...
!!  let's see ... how about "The sun never sets on the Internet"? ...
!!  er, oops, sorry ... better forget that idea :-)


Recent discussions in comp.sys.apollo
-------------------------------------

In June and July 1990, a discussion took place in comp.sys.apollo on
safe methods for distributing information about security bugs to system
administrators.  This led on, first, to comments on the difficulty many Apollo
sites have experienced in obtaining copies of patch tapes from HP, and thence
to wide-ranging criticisms of other aspects of HP's services to its customers.
Many Apollo system managers and users who had become increasingly frustrated
with HP's unresponsiveness began to realize that their problem or their site
or their national HP office was not an isolated case: Apollo customers all
over the world felt they were encountering similar difficulties.

The tone of the discussions was by no means all negative.  Many people say:

			I love my Apollo, BUT ...

Aspects of Apollos and Domain/OS that received particular praise included:
token ring; the intuitive, object-oriented, "automagically" networked file
system; ACLs (access control lists -- though these seem to be a matter of
taste); the Display Manager; DDE (Domain Distributed Debugger); dynamic swap
space instead of a permanently reserved partition; ability to run both BSD and
SysV Unix simultaneously.

!!  "Good compilers" has been omitted because a lot of people said they weren't:
!!  e.g., there is no full ANSI C compiler yet.

There were also many favorable comments on superhuman *unofficial* efforts to
help customers by many individual HP staff, including those who are already
willing to post news articles on Usenet.


The big BUT: customer service problems
--------------------------------------

Unfortunately, many Apollo users have formed the impression that Apollo support
has become the poor relation within HP.  While customers with HP hardware seem
not unhappy with service (at least, not *vocally* so), large numbers of Apollo
customers are very dissatisfied.  We feel that the problems are not the fault
of the hard-working Apollo support staff who exist: the cause instead lies in
insufficient *numbers* of such staff, and inadequate resource provision to
enable them to carry out their functions, combined with corporate over-caution
which hinders experiments with new approaches.

It might be argued, as far as educational customers are concerned, that a lesser
standard of service is appropriate, given the discount levels such customers
receive and the low levels of support contract they generally choose.  But
this would be to ignore the fact that many of us feel we are not even receiving
the modest level of support for which we have contracted.  Moreover, at least
some of us are reasonably sophisticated system administrators, able to deal
with most manual-reference questions ourselves, only referring *really* knotty
questions to HP for advice, and sometimes able to provide solutions that HP has
not discovered itself.  Remember too that while educational institutions may
not have enormous buying power themselves, the students who use their machines
will be the next generation of computer purchasers in industry and commerce.

Here are some of the service problems which have been discussed among readers
of comp.sys.apollo:

* security issues, such as the open initial protections on Domain/OS directories
and dangerous utilities, daemons' need for unprotected directories, absence of
restrictions normally present in certain Unix system control tools, and
downright bugs affecting security (details are deliberately omitted here!);
the most serious worry is the absence of any kind of "security alert bulletin"
by which HP could rapidly notify Apollo sites of security bugs and fixes as
they are found;

* although every Domain/OS manual solicits APRs (Apollo Product Reports) in
the introductory "Problems, Questions and Suggestions" section, and although
the on-line manual page for the mkapr utility gives an Internet electronic mail
address for APR submission and states "Customer Services will acknowledge all
product reports received", the reality is that APRs are not an effective way
of reporting problems: the email addresses often bounce; when email is
successful, and when APRs are submitted by more traditional means,
acknowledgements are often not received (especially for APRs from outside the
US); substantive responses to APRs never appear or turn up after months or
years; even then the responses often fail to solve the original problem;

* unsuitability of telephone support for more technical questions (e.g., bug
reports involving tracebacks): while telephone support can be excellent for
simple questions and for new or naive users, an electronic mail service with
fast turn-around would be preferred by many experienced programmers and system
managers, could provide as effective a shield for back-room HP support staff
as the telephone service, and would more effectively handle time-zone
difference problems for customers outside the US;

* counter-productive rigidity in telephone support procedures which require
that even very technical conversations be limited to customer representatives
nominated for direct contact, sometimes resulting in *two* intermediaries
between the user with the problem and the HP expert with the answer;

!!  Any comments on how widespread this is?

* long delays in delivery of software and hardware, e.g., NFS for SR10.2.p
and third-party sourced products like tape drives and Mathematica -- perhaps
made worse for non-US customers by poor communications between HP local and
head offices;

* perceived failure of procedures for timely automatic delivery of software
upgrades to customers with maintenance contracts;

!!  Any comments?  Certainly the mechanisms don't seem to work automatically
!!  here in Sydney.

* difficulty and delays in obtaining patch tapes, even in cases where it was
apparent that a particular patch might be relevant to an operating system
bug which was causing problems at a particular site -- as noted above, such
delays are very dangerous in the case of security-relevant patches;

* "closed" policy on HP modifications to publicly available software: for
example, HP has changed the FTP daemon ftpd to handle Apollo filetypes,
but has not released the source changes, so that they cannot be incorporated
into an enhanced version of ftpd independently developed by Sam Shen at
Berkeley which permits anonymous ftp without the need for the chroot system
call (missing from Domain/OS).

A problem in the development rather than service area which has attracted much
comment is HP's reluctance to incorporate such well-known tools as perl and
GNU Emacs into its own operating system releases.

In the next three sections, we present three requests to HP the granting of
which we feel would go a long way towards solving the problems described above.


Resource allocation within HP
-----------------------------

  REQUEST 1: Hewlett-Packard should urgently take whatever policy decisions
  and  actions are necessary to ensure that the Apollo Systems Division has
  resources available to it for support operations which  are  proportional
  to  those  provided on the HP side.  Personnel levels and organization in
  the Apollo Division support sector need to be reviewed and improved.

It is clear from the complaints of Apollo users discussed above, contrasted
with the apparent relative happiness of HP/UX customers, that either Apollo
Division support resources available per customer are smaller than those for
HP/UX, or the resources in the Apollo Division are not being deployed
effectively enough to satisfy customers' perceptions of their need for support.

!!  I have talked throughout of "HP/UX customers", but is that the correct
!!  opposite?  What is MPE?  Do they talk about that in comp.sys.hp?

We will not presume to advise HP on the managerial details involved in
implementing Request 1.  The brevity and simplicity of this section should
be taken as emphasizing our belief in the importance of this request and its
clear justification on grounds of equity.


Use of the Internet
-------------------

The Internet already allows users to support each other technically -- not to
mention in terms of morale.  Although this certainly means great savings to HP,
it happens *in spite of* HP, not *in co-operation with* HP.

We propose that HP take steps to provide better services to its customers on
the Internet by using the Internet in an *organized and official* way.  We
believe that this will not only benefit users, but will increase efficiency
and feedback and reduce duplication for HP as well.

  REQUEST 2: We would like HP to set up an INTERNET LIAISON UNIT, with suf-
  ficient  staff,  resources  and  authority  to  carry  out  the following
  operations:

   a) Organize and oversee a new, effective system whereby APRs (and their
      HP/UX counterpart) can be submitted by electronic mail, acknowledged
      by return email, and then answered by email within a reasonable time
      -- say two months.  If a longer time is required, a progress report
      should be sent say monthly.

   b) Arrange for a mail gateway between the Internet and internal HP mail,
      or publicize it if one already exists, so that customers on the Internet
      can conveniently communicate with their local service people as an
      optional alternative to telephone service.  (The gateway could have a
      filter or alias mechanism so that other internal HP staff would not be
      bothered with mail from outside if they did not want it.)

   c) Monitor the comp.sys.apollo and comp.sys.hp Usenet newsgroups, and
      where appropriate arrange for responses to be provided from relevant
      experts within HP.  

A further task for the unit would be to set up a public archive accessible from
the Internet.  We feel that this is important -- and perhaps controversial --
enough to be stated as a separate request with a detailed explanation.


A public archive
----------------

  REQUEST  3:  HP  should  establish  a public archive on a new or existing
  company machine connected to  the  Internet,  to  make  customer  support
  materials  available  via  FTP.   The  archive  should be operated by the
  Internet Liaison Unit, and should include at least the following:

   a) an index of the latest version numbers of all supported software, and
      which operating system versions they work under;

   b) a regularly updated index of known bugs, e.g., a list of APRs,
      perhaps similar to an on-line version of the "HP-UX Software
      Release/Status Bulletin" series, with workarounds if available; 

   c) release notes for all current and beta versions of all supported
      software (note that this would cover some bug reports; it would
      also encourage customers to obtain upgraded versions); 

   d) a complete set of all current patches, say in compressed wbak
      format, with release notes (see caveats about security and major
      patches below);

   e) source of HP modifications to generally available programs such as
      ftpd and sendmail: this would allow us to keep those programs up
      to date, enhance them, and send them back to HP (a good start in
      this direction is /domain_examples/tcp/gated).

!!  I have removed the item in Draft #1 which read
!!	"perhaps, new product announcements -- preferably technical details
!!	not sales material."
!!  to make it quite clear-cut that we are *not* asking HP to break Internet
!!  commercialism rules.

Note that the archive would be PUBLIC, so available to all Internet users
instead of being restricted to service contract holders.  There is a rival
precedent for this in the public archive of patches SUN maintains for FTP
from the Internet host uunet.uu.net.

A public archive, available to all members of the Internet, is probably
necessary because of prohibition by the NSF and other funding bodies on
use of the Internet for commercial gain.

Such a service to all owners of HP equipment would probably not reduce the
number who take out service contracts appreciably: a contract would still
be needed to obtain software upgrades, and this is probably the greatest
incentive for a contract at most sites.  (There would be no expectation that
HP would continue to support obsolete versions of software through patches or
bug lists in the archive.)  Furthermore, the existence of a public archive
would demonstrate HP's commitment to its customers and to high standards, and
would represent a major enhancement to the attractiveness of HP products.

Our request for a public archive is not a novel one: see the column "The
Inside Track: On HP-UX patches" by Dave Taylor in The HP Chronicle of May
1990 for a persuasive argument in favor of such an archive from an HP/UX
point of view.

!!  The HP Chronicle is an independent non-HP publication of PCI from Austin,
!!  Texas -- (800) 888-5093 or +1 512 250 5518.

!!  I gather that much of Europe still does not have real Internet connections:
!!  can we say anything about alternatives to FTP access there?


Two caveats about patches
-------------------------

Patches which address security problems should be included in the archive
if this can be done without causing security problems in itself.  System
managers of machines connected to the Internet must be particularly conscious
of security questions, and have great interest in receiving security-related
patches as rapidly as possible.  However, security-patch release notes should
never include any details of the problems which they aim to correct: such
details can themselves lead to breaches of security at unpatched sites.
The release notes should simply state that the patch in question is
security-related and urgent.  A brief Usenet news item should announce
the addition of each new security patch to the archive.

Also note that some patches such as /sau*/domain_os may be so substantial as
to amount to de facto upgrades, and might need to be excluded from a public
archive.  Excluded patches should be distributed by traditional means, but
much more effectively and rapidly than at present.

!!  A VAX/VMS past is showing here.  I had not thought of the need for the
!!  last exclusion before, because I'd been subconsciously thinking of patches
!!  in terms of the way DEC distributes them using the VMS system PATCH utility
!!  (a sort of binary analogue of Larry Wall's Unix source patch tool).  Thus
!!  most VMS "patches" are *no use at all* if you haven't already got a proper
!!  copy of the executable image to start from.  Pity there isn't something
!!  similar for Domain/OS or Unix ... it would be a tall order, I know.


Benefits of using the Internet
------------------------------

Of course, the existence of support facilities on the Internet would not
reduce HP's traditional obligations to its service-contract customers,
especially those without Internet connections (although it could be possible
for those geographically close to the archive to access it by say dial-up
UUCP -- however unsatisfactory this method would be considered by Internet
users).  But we believe that rapid and efficient dissemination of information
via the Internet would *save* HP money by cutting out duplicated effort.

For example, many of the more routine questions at present addressed to the
telephone service hotlines would be avoided if systems administrators had
on-line FTP and/or Usenet access to answers to frequently asked questions and
information such as usage tips and bug workarounds.  This would free HP
telephone service and other support resources which could be applied to more
rapid or deeper investigation of unusual problems and subtler bugs.  

Distribution of patches by FTP should be much more efficient than copying and
physically distributing patch tapes.

The existence of a large community of HP machine users, programmers, and
systems administrators that HP could communicate with would foster a "global
village" style of interaction.  This interaction would benefit both parties by
the dissemination of valuable first-hand information between the actual users
and HP engineers and management.  This information could be used to improve
planning for the needs of the user community and the priorities of HP, and
would also promote better customer-vendor relationships.  The result would be
increased sales and more widespread acceptance of HP machines.

More efficient support from HP for academic and research customers through
network connections would be a step in the direction of the microeconomic reform
and the closer ties between industry and academia for which government and
business are calling world-wide.

!!  More like a nanoeconomic reform? :-)  Let me know if the buzzwords are
!!  passe' or unknown outside Australia.

We believe that our Requests 2 and 3 can be satisfied by HP in accordance with
commercialism guidelines applying on the major publicly funded networks.  If
it transpires that network use by HP of the kind we are requesting requires
justification under the guidelines, then we will be happy to collaborate with
HP in preparing a case for submission to the relevant network authorities.


Disclaimer
----------

This document was written collectively, and while all signatories support its
aims and general thrust, not everyone is necessarily in complete agreement on
the details of all points.  The views expressed are those of individuals, and
do not in general represent official policy of the institutions or companies
of which the signatories are members.  (This should not be taken as a license
to discount those views, however: in the long run the individual views of
computer users and system managers tend to affect or even determine
institutional computing policy and purchasing decisions.)


Conclusion
----------

We applaud the naming of HP's latest major computer line as the "HP *Apollo*
9000 Series 400", and note with approval that the sales literature gives
Domain/OS equal weight to HP/UX, and the Apollo DN10000 as much prominence
as the HP Model 635.  As Apollo users we are pleased to see this concrete
illustration of HP's commitment to the continuation and flourishing of a
strong Apollo Division.  

The slogan for the new HP Apollo machines describes them as the "first
workstations to combine the innovation of Apollo with the quality and
reliability of HP".  We believe that if your company's customer services
continue to combine the excessive conservatism of HP with the organizational
haphazardness of Apollo, then all your other efforts will be in vain.

We hope that Hewlett-Packard will accept this critique in the same positive
spirit with which we have prepared it, and will act quickly to fulfill our
three requests.  Individual replies are not expected: indeed, we will know
that we have been heard when we see a response from HP as a news article in
comp.sys.apollo on the Usenet.


Signatories
-----------

<End of draft>

!!  Please keep preparing your signatures -- maximum 10 lines x 78 columns.
!!  Do not send them yet!
!!
!!  You might also like to find out the name, position and address or FAX of
!!  one or two important HP managers for sending the letter to in printed form.
!!  As we've heard, many HP staff are already reading the newsgroups in some
!!  form, but to some extent we're preaching to the converted there :-)  We
!!  need to be able to reach top HP people who *don't* know about the Internet
!!  yet.  Please *do not post* any names or addresses of HP staff to the net:
!!  just have them ready.  A news item calling for signatures and suggesting
!!  methods of getting the open letter to HP will follow posting of the final
!!  version.
!!
!!  I would be very grateful to have a couple of volunteers, at sites with good
!!  mail connections in North America and Europe respectively, to help collect
!!  and collate signatures when the time comes.  Please mail me.
!!
!!  Following a sensible suggestion I have changed from British spelling
!!  to American, since the target audience mostly uses the latter.

--
Jim Richardson
Department of Pure Mathematics, University of Sydney, NSW 2006, Australia
Internet: jimr@maths.su.oz.au  ACSNET: jimr@maths.su.oz  FAX: +61 2 692 4534

wwm@pmsmam.uucp (Bill Meahan) (07/19/90)

Just one short comment:

What about those of us who are COMMERCIAL users of HP/UX?

Setting up the mechanisms proposed on the Internet are a terrific idea! BUT,
there are a large number of us out here in the commercial world who use HP/UX
who would like the same type of service!!  The ability to do anonymous UUCP
(preferably into a local phone number - the local HP sales office??) to obtain
archive items, plus a UUCP mail gateway for e-mail contact (again, via the
local HP sales office?) would be GREATLY appreciated.

Yes, I know about SupportLine and the BBS (I've used it as we _are_ on
service contract here) but my opinion of what I've seen so far is pretty
low.  The idea of CompuServe has an air about it that resembles the air
about the waste containers in a fish cannery after a long, hot, muggy
weekend.  Whoever thought that one up ought to to be forced to endure
a month of alt.flame.

As a peon engineer in a rather large company, with no personal authority
whatsoever (how's that for a disclaimer!) I offer my wholehearted support
for the concept of electronic interaction with HP/APOLLO.  Just don't
leave me (and my many equivalents elsewhere) out of the grand scheme!
-- 
Bill Meahan  WA8TZG		uunet!mailrus!umich!pmsmam!wwm
I don't speak for Ford - the PR department does that!

Any attempt at wit is liable to offend _somebody_!