jimr@maths.su.oz.au (Jim Richardson) (07/19/90)
Here is the second draft of the open letter to HP. Many thanks to everyone who has emailed and/or posted comments and suggestions. Again, I have not given due credit ... but then everybody's signatures will be on the final letter! :-) Passages marked !! are comments by me, not part of the open letter itself. There are substantial additions and alterations, including the change in viewpoint from that of all HP customers to that of Apollo customers only, which was explained in <1990Jul14.085541.18550@metro.ucc.su.OZ.AU> ("Netpower: Apollo/HP contrast"). I've made comments on a few of the other changes. The letter is something like six A4 pages long now, so further additions are probably not desirable. I'd be very grateful to receive any corrections, or notes of glaring omissions, for incorporation into the final version. <Start of draft> OPEN LETTER TO HP: DRAFT #2 HEWLETT-PACKARD, APOLLO CUSTOMERS, AND THE INTERNET Background: The Internet and Usenet ----------------------------------- The "Internet" is a very large computer network using the TCP/IP protocols and extending over much of the world. Among the services it provides are electronic mail, file transfer via the FTP protocol, and "network news", a conferencing system somewhat akin to the internal HP notes groups. Network news is divided up into approaching a thousand "newsgroups", each covering a different discussion area. The computers within and beyond the Internet which carry network news, the data links between those computers, and the community of people who read the news, are collectively known as "USENET". In mid-1990, the number of machines receiving Usenet articles was estimated at over 26,000 and the total number of people who read some articles at 1,109,000. [ Reference: USENET READERSHIP SUMMARY REPORT FOR JUN 90, Brian Reid (reid@decwrl.DEC.COM), article <1990Jul2.154231.28843@wrl.dec.com> in Usenet newsgroup news.lists, 2 July 1990 ] Two of the newsgroups carry discussions among users of Apollo and HP computers: these groups are called comp.sys.apollo and comp.sys.hp. It is estimated that in June 1990, these groups respectively had 27,000 and 23,000 readers worldwide (with an unknown amount of overlap); in that month, there were 166 articles totaling 250 kilobytes in comp.sys.apollo, and 215 articles totaling 323 kilobytes in comp.sys.hp. [ Reference: USENET Readership report for Jun 90, Brian Reid, article <1990Jul2.154323.29469@wrl.dec.com> in news.lists, 2 July 1990 ] Many but by no means all of the machines connected to the Internet are in educational or research institutions. Numerous large and small commercial and industrial customers of HP also have Internet access. !! Maybe we need a catchy phrase to excite the HP PR department here ... !! let's see ... how about "The sun never sets on the Internet"? ... !! er, oops, sorry ... better forget that idea :-) Recent discussions in comp.sys.apollo ------------------------------------- In June and July 1990, a discussion took place in comp.sys.apollo on safe methods for distributing information about security bugs to system administrators. This led on, first, to comments on the difficulty many Apollo sites have experienced in obtaining copies of patch tapes from HP, and thence to wide-ranging criticisms of other aspects of HP's services to its customers. Many Apollo system managers and users who had become increasingly frustrated with HP's unresponsiveness began to realize that their problem or their site or their national HP office was not an isolated case: Apollo customers all over the world felt they were encountering similar difficulties. The tone of the discussions was by no means all negative. Many people say: I love my Apollo, BUT ... Aspects of Apollos and Domain/OS that received particular praise included: token ring; the intuitive, object-oriented, "automagically" networked file system; ACLs (access control lists -- though these seem to be a matter of taste); the Display Manager; DDE (Domain Distributed Debugger); dynamic swap space instead of a permanently reserved partition; ability to run both BSD and SysV Unix simultaneously. !! "Good compilers" has been omitted because a lot of people said they weren't: !! e.g., there is no full ANSI C compiler yet. There were also many favorable comments on superhuman *unofficial* efforts to help customers by many individual HP staff, including those who are already willing to post news articles on Usenet. The big BUT: customer service problems -------------------------------------- Unfortunately, many Apollo users have formed the impression that Apollo support has become the poor relation within HP. While customers with HP hardware seem not unhappy with service (at least, not *vocally* so), large numbers of Apollo customers are very dissatisfied. We feel that the problems are not the fault of the hard-working Apollo support staff who exist: the cause instead lies in insufficient *numbers* of such staff, and inadequate resource provision to enable them to carry out their functions, combined with corporate over-caution which hinders experiments with new approaches. It might be argued, as far as educational customers are concerned, that a lesser standard of service is appropriate, given the discount levels such customers receive and the low levels of support contract they generally choose. But this would be to ignore the fact that many of us feel we are not even receiving the modest level of support for which we have contracted. Moreover, at least some of us are reasonably sophisticated system administrators, able to deal with most manual-reference questions ourselves, only referring *really* knotty questions to HP for advice, and sometimes able to provide solutions that HP has not discovered itself. Remember too that while educational institutions may not have enormous buying power themselves, the students who use their machines will be the next generation of computer purchasers in industry and commerce. Here are some of the service problems which have been discussed among readers of comp.sys.apollo: * security issues, such as the open initial protections on Domain/OS directories and dangerous utilities, daemons' need for unprotected directories, absence of restrictions normally present in certain Unix system control tools, and downright bugs affecting security (details are deliberately omitted here!); the most serious worry is the absence of any kind of "security alert bulletin" by which HP could rapidly notify Apollo sites of security bugs and fixes as they are found; * although every Domain/OS manual solicits APRs (Apollo Product Reports) in the introductory "Problems, Questions and Suggestions" section, and although the on-line manual page for the mkapr utility gives an Internet electronic mail address for APR submission and states "Customer Services will acknowledge all product reports received", the reality is that APRs are not an effective way of reporting problems: the email addresses often bounce; when email is successful, and when APRs are submitted by more traditional means, acknowledgements are often not received (especially for APRs from outside the US); substantive responses to APRs never appear or turn up after months or years; even then the responses often fail to solve the original problem; * unsuitability of telephone support for more technical questions (e.g., bug reports involving tracebacks): while telephone support can be excellent for simple questions and for new or naive users, an electronic mail service with fast turn-around would be preferred by many experienced programmers and system managers, could provide as effective a shield for back-room HP support staff as the telephone service, and would more effectively handle time-zone difference problems for customers outside the US; * counter-productive rigidity in telephone support procedures which require that even very technical conversations be limited to customer representatives nominated for direct contact, sometimes resulting in *two* intermediaries between the user with the problem and the HP expert with the answer; !! Any comments on how widespread this is? * long delays in delivery of software and hardware, e.g., NFS for SR10.2.p and third-party sourced products like tape drives and Mathematica -- perhaps made worse for non-US customers by poor communications between HP local and head offices; * perceived failure of procedures for timely automatic delivery of software upgrades to customers with maintenance contracts; !! Any comments? Certainly the mechanisms don't seem to work automatically !! here in Sydney. * difficulty and delays in obtaining patch tapes, even in cases where it was apparent that a particular patch might be relevant to an operating system bug which was causing problems at a particular site -- as noted above, such delays are very dangerous in the case of security-relevant patches; * "closed" policy on HP modifications to publicly available software: for example, HP has changed the FTP daemon ftpd to handle Apollo filetypes, but has not released the source changes, so that they cannot be incorporated into an enhanced version of ftpd independently developed by Sam Shen at Berkeley which permits anonymous ftp without the need for the chroot system call (missing from Domain/OS). A problem in the development rather than service area which has attracted much comment is HP's reluctance to incorporate such well-known tools as perl and GNU Emacs into its own operating system releases. In the next three sections, we present three requests to HP the granting of which we feel would go a long way towards solving the problems described above. Resource allocation within HP ----------------------------- REQUEST 1: Hewlett-Packard should urgently take whatever policy decisions and actions are necessary to ensure that the Apollo Systems Division has resources available to it for support operations which are proportional to those provided on the HP side. Personnel levels and organization in the Apollo Division support sector need to be reviewed and improved. It is clear from the complaints of Apollo users discussed above, contrasted with the apparent relative happiness of HP/UX customers, that either Apollo Division support resources available per customer are smaller than those for HP/UX, or the resources in the Apollo Division are not being deployed effectively enough to satisfy customers' perceptions of their need for support. !! I have talked throughout of "HP/UX customers", but is that the correct !! opposite? What is MPE? Do they talk about that in comp.sys.hp? We will not presume to advise HP on the managerial details involved in implementing Request 1. The brevity and simplicity of this section should be taken as emphasizing our belief in the importance of this request and its clear justification on grounds of equity. Use of the Internet ------------------- The Internet already allows users to support each other technically -- not to mention in terms of morale. Although this certainly means great savings to HP, it happens *in spite of* HP, not *in co-operation with* HP. We propose that HP take steps to provide better services to its customers on the Internet by using the Internet in an *organized and official* way. We believe that this will not only benefit users, but will increase efficiency and feedback and reduce duplication for HP as well. REQUEST 2: We would like HP to set up an INTERNET LIAISON UNIT, with suf- ficient staff, resources and authority to carry out the following operations: a) Organize and oversee a new, effective system whereby APRs (and their HP/UX counterpart) can be submitted by electronic mail, acknowledged by return email, and then answered by email within a reasonable time -- say two months. If a longer time is required, a progress report should be sent say monthly. b) Arrange for a mail gateway between the Internet and internal HP mail, or publicize it if one already exists, so that customers on the Internet can conveniently communicate with their local service people as an optional alternative to telephone service. (The gateway could have a filter or alias mechanism so that other internal HP staff would not be bothered with mail from outside if they did not want it.) c) Monitor the comp.sys.apollo and comp.sys.hp Usenet newsgroups, and where appropriate arrange for responses to be provided from relevant experts within HP. A further task for the unit would be to set up a public archive accessible from the Internet. We feel that this is important -- and perhaps controversial -- enough to be stated as a separate request with a detailed explanation. A public archive ---------------- REQUEST 3: HP should establish a public archive on a new or existing company machine connected to the Internet, to make customer support materials available via FTP. The archive should be operated by the Internet Liaison Unit, and should include at least the following: a) an index of the latest version numbers of all supported software, and which operating system versions they work under; b) a regularly updated index of known bugs, e.g., a list of APRs, perhaps similar to an on-line version of the "HP-UX Software Release/Status Bulletin" series, with workarounds if available; c) release notes for all current and beta versions of all supported software (note that this would cover some bug reports; it would also encourage customers to obtain upgraded versions); d) a complete set of all current patches, say in compressed wbak format, with release notes (see caveats about security and major patches below); e) source of HP modifications to generally available programs such as ftpd and sendmail: this would allow us to keep those programs up to date, enhance them, and send them back to HP (a good start in this direction is /domain_examples/tcp/gated). !! I have removed the item in Draft #1 which read !! "perhaps, new product announcements -- preferably technical details !! not sales material." !! to make it quite clear-cut that we are *not* asking HP to break Internet !! commercialism rules. Note that the archive would be PUBLIC, so available to all Internet users instead of being restricted to service contract holders. There is a rival precedent for this in the public archive of patches SUN maintains for FTP from the Internet host uunet.uu.net. A public archive, available to all members of the Internet, is probably necessary because of prohibition by the NSF and other funding bodies on use of the Internet for commercial gain. Such a service to all owners of HP equipment would probably not reduce the number who take out service contracts appreciably: a contract would still be needed to obtain software upgrades, and this is probably the greatest incentive for a contract at most sites. (There would be no expectation that HP would continue to support obsolete versions of software through patches or bug lists in the archive.) Furthermore, the existence of a public archive would demonstrate HP's commitment to its customers and to high standards, and would represent a major enhancement to the attractiveness of HP products. Our request for a public archive is not a novel one: see the column "The Inside Track: On HP-UX patches" by Dave Taylor in The HP Chronicle of May 1990 for a persuasive argument in favor of such an archive from an HP/UX point of view. !! The HP Chronicle is an independent non-HP publication of PCI from Austin, !! Texas -- (800) 888-5093 or +1 512 250 5518. !! I gather that much of Europe still does not have real Internet connections: !! can we say anything about alternatives to FTP access there? Two caveats about patches ------------------------- Patches which address security problems should be included in the archive if this can be done without causing security problems in itself. System managers of machines connected to the Internet must be particularly conscious of security questions, and have great interest in receiving security-related patches as rapidly as possible. However, security-patch release notes should never include any details of the problems which they aim to correct: such details can themselves lead to breaches of security at unpatched sites. The release notes should simply state that the patch in question is security-related and urgent. A brief Usenet news item should announce the addition of each new security patch to the archive. Also note that some patches such as /sau*/domain_os may be so substantial as to amount to de facto upgrades, and might need to be excluded from a public archive. Excluded patches should be distributed by traditional means, but much more effectively and rapidly than at present. !! A VAX/VMS past is showing here. I had not thought of the need for the !! last exclusion before, because I'd been subconsciously thinking of patches !! in terms of the way DEC distributes them using the VMS system PATCH utility !! (a sort of binary analogue of Larry Wall's Unix source patch tool). Thus !! most VMS "patches" are *no use at all* if you haven't already got a proper !! copy of the executable image to start from. Pity there isn't something !! similar for Domain/OS or Unix ... it would be a tall order, I know. Benefits of using the Internet ------------------------------ Of course, the existence of support facilities on the Internet would not reduce HP's traditional obligations to its service-contract customers, especially those without Internet connections (although it could be possible for those geographically close to the archive to access it by say dial-up UUCP -- however unsatisfactory this method would be considered by Internet users). But we believe that rapid and efficient dissemination of information via the Internet would *save* HP money by cutting out duplicated effort. For example, many of the more routine questions at present addressed to the telephone service hotlines would be avoided if systems administrators had on-line FTP and/or Usenet access to answers to frequently asked questions and information such as usage tips and bug workarounds. This would free HP telephone service and other support resources which could be applied to more rapid or deeper investigation of unusual problems and subtler bugs. Distribution of patches by FTP should be much more efficient than copying and physically distributing patch tapes. The existence of a large community of HP machine users, programmers, and systems administrators that HP could communicate with would foster a "global village" style of interaction. This interaction would benefit both parties by the dissemination of valuable first-hand information between the actual users and HP engineers and management. This information could be used to improve planning for the needs of the user community and the priorities of HP, and would also promote better customer-vendor relationships. The result would be increased sales and more widespread acceptance of HP machines. More efficient support from HP for academic and research customers through network connections would be a step in the direction of the microeconomic reform and the closer ties between industry and academia for which government and business are calling world-wide. !! More like a nanoeconomic reform? :-) Let me know if the buzzwords are !! passe' or unknown outside Australia. We believe that our Requests 2 and 3 can be satisfied by HP in accordance with commercialism guidelines applying on the major publicly funded networks. If it transpires that network use by HP of the kind we are requesting requires justification under the guidelines, then we will be happy to collaborate with HP in preparing a case for submission to the relevant network authorities. Disclaimer ---------- This document was written collectively, and while all signatories support its aims and general thrust, not everyone is necessarily in complete agreement on the details of all points. The views expressed are those of individuals, and do not in general represent official policy of the institutions or companies of which the signatories are members. (This should not be taken as a license to discount those views, however: in the long run the individual views of computer users and system managers tend to affect or even determine institutional computing policy and purchasing decisions.) Conclusion ---------- We applaud the naming of HP's latest major computer line as the "HP *Apollo* 9000 Series 400", and note with approval that the sales literature gives Domain/OS equal weight to HP/UX, and the Apollo DN10000 as much prominence as the HP Model 635. As Apollo users we are pleased to see this concrete illustration of HP's commitment to the continuation and flourishing of a strong Apollo Division. The slogan for the new HP Apollo machines describes them as the "first workstations to combine the innovation of Apollo with the quality and reliability of HP". We believe that if your company's customer services continue to combine the excessive conservatism of HP with the organizational haphazardness of Apollo, then all your other efforts will be in vain. We hope that Hewlett-Packard will accept this critique in the same positive spirit with which we have prepared it, and will act quickly to fulfill our three requests. Individual replies are not expected: indeed, we will know that we have been heard when we see a response from HP as a news article in comp.sys.apollo on the Usenet. Signatories ----------- <End of draft> !! Please keep preparing your signatures -- maximum 10 lines x 78 columns. !! Do not send them yet! !! !! You might also like to find out the name, position and address or FAX of !! one or two important HP managers for sending the letter to in printed form. !! As we've heard, many HP staff are already reading the newsgroups in some !! form, but to some extent we're preaching to the converted there :-) We !! need to be able to reach top HP people who *don't* know about the Internet !! yet. Please *do not post* any names or addresses of HP staff to the net: !! just have them ready. A news item calling for signatures and suggesting !! methods of getting the open letter to HP will follow posting of the final !! version. !! !! I would be very grateful to have a couple of volunteers, at sites with good !! mail connections in North America and Europe respectively, to help collect !! and collate signatures when the time comes. Please mail me. !! !! Following a sensible suggestion I have changed from British spelling !! to American, since the target audience mostly uses the latter. -- Jim Richardson Department of Pure Mathematics, University of Sydney, NSW 2006, Australia Internet: jimr@maths.su.oz.au ACSNET: jimr@maths.su.oz FAX: +61 2 692 4534
wwm@pmsmam.uucp (Bill Meahan) (07/19/90)
Just one short comment: What about those of us who are COMMERCIAL users of HP/UX? Setting up the mechanisms proposed on the Internet are a terrific idea! BUT, there are a large number of us out here in the commercial world who use HP/UX who would like the same type of service!! The ability to do anonymous UUCP (preferably into a local phone number - the local HP sales office??) to obtain archive items, plus a UUCP mail gateway for e-mail contact (again, via the local HP sales office?) would be GREATLY appreciated. Yes, I know about SupportLine and the BBS (I've used it as we _are_ on service contract here) but my opinion of what I've seen so far is pretty low. The idea of CompuServe has an air about it that resembles the air about the waste containers in a fish cannery after a long, hot, muggy weekend. Whoever thought that one up ought to to be forced to endure a month of alt.flame. As a peon engineer in a rather large company, with no personal authority whatsoever (how's that for a disclaimer!) I offer my wholehearted support for the concept of electronic interaction with HP/APOLLO. Just don't leave me (and my many equivalents elsewhere) out of the grand scheme! -- Bill Meahan WA8TZG uunet!mailrus!umich!pmsmam!wwm I don't speak for Ford - the PR department does that! Any attempt at wit is liable to offend _somebody_!