domo@tsa.co.uk (Dominic Dunlop) (07/05/90)
From: Dominic Dunlop <domo@tsa.co.uk> Report on ISO/IEC JTC1/SC22/WG15 (POSIX) Meeting of 11th - 15th June, 1990, Paris, France Dominic Dunlop -- domo@tsa.co.uk The Standard Answer Ltd. Introduction Working Group 15 of Subcommittee 22 of Joint Technical Committee 1 of the International Organization for Standardization and the International Electrotechnical Commission (ISO/IEC JTC1/SC22/WG15), or, more briefly, the ISO POSIX working group, met in Paris, France, from the 12th to the 15th of June. The meeting was hosted by AFNOR, (Association francaise de normalisation), the French national standards body, at its offices in La Defense, a high-rise business district a brisk train-ride away from the city centre, and was preceded on 11th June and the morning of the 12th by meetings of the rapporteur groups on conformance testing, internationalization and security. Attendance was good, with thirty "experts" (as the ISO Directives style us) representing nine countries, plus the European Community. I was present at the main meeting and at the internationalization rapporteur group as an observer with the brief of reporting back to you. This report is the fourth jointly commissioned by the European UNIX systems User Group (EUUG) and USENIX. As usual, it's a little imprecise in its references to ISO. Strictly, most of these should be to JTC1, or to some part of JTC1. Where precision is needed, I use it and give an explanation, but for the most part I refer simply to ISO, so as to avoid getting bogged down in unnecessary detail. If you have any comments, or need clarification or further information, please contact me at the mail address above. First, a summary of the most important aspects of the meeting: Summary o POSIX.1, the operating system interface standard, should be published as international standard 9945-1 Real Soon Now. As I write, the ballot on the document has yet to close, but it seems unlikely that any of the twenty countries voting will oppose acceptance. The meeting identified a number of trivial editorial changes to the current draft international standard, and these, taken together with continuing nit-picking comments from ISO's central secretariat, should result in a document which ISO will print. Its technical content will be identical to that of - 2 - ANSI/IEEE Std. 1003.1:1988, so implementors following the U.S. standard can be assured of ISO compliance when 9945-1 finally sees the light of day. o POSIX.2, the shell and tools standard, faces a bumpier ride before becoming international standard 9945-2. In an ISO ballot on acceptance of draft 9 of IEEE 1003.2 as a draft international standard, six countries voted against, with just five in favour. This is more of an embarrassment than a set-back: hindsight suggests that it was just too early to hold a ballot. o Showing its increasing concern and frustration at the lack of apparent progress within the IEEE on a (programming) language-independent version of the POSIX operating system interface standard, WG15 has refused to "register" the current, largely language- independent, draft of the 1003.4 realtime extensions standard on the grounds that it makes little sense to have language-independent extensions to a base standard which is specified in terms of the C language. Similarly, the group failed to register 1003.5 (Ada binding) and 1003.9 (FORTRAN-77 binding) because POSIX.1 lacks an explicit service definition to which they can bind. o The failure to register these documents causes a hiccup -- albeit a discreet one -- in the synchronization between IEEE and ISO POSIX standardization schedules. In the hope of avoiding such situations in the future, an informal "speak now, or forever hold your peace" mechanism has been put in place, allowing the international community to comment on the subject area, format, presentation and approach of IEEE documents at an early stage in their preparation. o Had it not been for this upset, the working group would have adopted a firm synchronization plan so as to ensure that the work of IEEE and of ISO is closely coordinated in the future. As it is, the "U.S. member body" -- ANSI -- has been asked to provide a revised plan for the working group's October meeting in Seattle. o The Open Software Foundation, UNIX International and X/Open are cooperating on a common approach to conformance testing, known as Phoenix. Governmental organizations with a strong interest in the field, such as the National Institute for Science and Technology (NIST) and the Commission of European Communities (CEC), seem broadly to welcome this move -- even if the - 3 - unaccustomed show of tripartite unity is, as one security rapporteur put it, "a bit spooky". o At an evening reception hosted by AFUU (Association francaise des utilizateurs de UNIX), the French UNIX users' group, Mike Lambert of X/Open said that "our hand is extended" to the international standardization community, with which his organization hopes to work in finding some more timely and responsive way of delivering formal consensus standards for open systems. The definition of this mechanism is left as an exercise for the reader -- or for the international standards community. Perhaps Mike has come to realize that standardizers too are prone to the Not Invented Here syndrome, and so must believe that they thought of something themselves before they can accept it. o Just to keep us all on our toes, ISO has updated its Directives, with the result that the procedure for turning a base document -- such as one of the IEEE drafts -- into an international standard is subtly changed. We now have to forget about Draft Proposals, and turn our minds instead to Working Drafts and Committee Drafts. Sigh... The rest of this report gives more detail most of these topics. 9945-1_--_Operating_System_Interface You may recall from my report of WG15's last meeting in October, 1989, that the group had in effect told ISO's central secretariat that, while the then-current draft of IS 9945-1 was not perfect, it was, in the group's opinion, good enough to publish, particularly since we'd undertake to fix things up on short order, allowing an improved version to be published within a year of the first edition. This proposal did not fly. Firstly, the central secretariat (now dynamically known as ITTF, the Information Technology Task Force), refused to publish a document that looked much more like an IEEE standard than an ISO standard; secondly, they deemed the changes needed to polish up the draft to be sufficiently great that it should go out to ballot again in order to provide a formal check that it was still acceptable to the group. This was duly done; the ballot closed on 29th June, and is expected to pass very comfortably. Nevertheless, the ballot gave people the opportunity to comment formally on the document again, with the result that a number of small bug-fixes and clarifications were - 4 - suggested, and were accepted for incorporation into the draft. We judge -- and we hope that ITTF agrees -- the changes are strictly editorial, and so will not merit yet another ballot. ITTF, which, in discussion with the IEEE, had slightly bent its drafting and presentation rules so as to come up with a compromise satisfactory to both parties, also suggested further changes, some in areas that we thought had already been addressed. This is a cause for concern: the prospect of the standard never being published because its layout and language do not meet some ill-defined guidelines does not appeal. Consequently, we are seeking "written harmonized editorial requirements" from the IEEE and ITTF. The ballot also produced a number of suggestions in the area of internationalization, such as how to handle (and indeed, how to refer to) wide, or multi-byte, characters. To have acted on these comments would have changed the technical content of the draft standard -- the equivalent of sliding down a snake in the game that leads to eventual publication. Happily, those who suggested the changes were content to leave these issues for the second edition of the standard, rather than further delay the first edition. The single exception that we could get away with was to replace Annex E's1 example international profile for the hypothetical (and extremely odd) land of Poz with a real example for the (only slightly odd) country of Denmark. Although this is a large change, it can be made because the annex (ISO-speak for appendix) is "non-normative"; that is, it is an explanatory example rather than a part of the official standard. Messaging, an issue which is currently exercising the minds of those concerned with the next edition of the 1003.1 standard[1], was also passed over by WG15: nobody had a strong preference for either the X/Open proposal or the UniForum proposal, so 1003.1 will have to handle the issue without guidance from from the ISO working group. Where all does this leave us? With no published international standard as yet, but with a very good prospect of having one this year. Until it arrives, keep using the bilious green book, IEEE std. 1003.1:19882, as its technical __________ 1. This material is not in the published IEEE std. 1003.1:1988. - 5 - content is identical to that of the eventual ISO standard. 9945-2_--_Shell_and_Tools Earlier in the year, there had been a ballot on moving forward draft proposal (DP) 9945-2, Shell and utility application interface for computer operating system environments, to become a draft international standard (DIS). Basically, while a DP is allowed -- even expected -- to differ considerably from the international standard which ultimately results, a DIS is expected to be in a format and to have contents which are very close to those of the ultimate standard3. That the ballot was six against to just five for (with nine countries not voting) indicates that the consensus is that 9945-2 has to go through quite a few more changes before it can be acceptable as an international standard. Many of these changes concern internationalization, as this topic affects POSIX.2 considerably more than POSIX.1. For example, the example Danish national profile to be incorporated into 9945-1 is just a part of the work that DS (Dansk Standardiseringraad) has done on the topic; the rest affects 9945-2. There is also an unresolved issue concerning the definition of collation sequences (sorting orders) for international character sets. Utilities such as sort clearly need to know about collation sequence, and so do the regular expression-handling utilities and functions defined by POSIX.2. The problem is that nobody in ISO seems to want to handle the matter. In particular, JTC1/SC2, which standardizes coded character sets, sees its job as assigning codes to characters, not as saying anything about how those codes should sort4. This is a reasonable attitude: collation is a surprisingly complex field, and to ____________________________________________________________ 2. You can buy a copy by calling IEEE customer service on +1 908 981 1393 (1 800 678 IEEE inside the U.S.) and giving them a credit card number. The cost is $37, plus $4 for overseas surface mail, plus another $15 for airmail. Alternatively, your national standards body may be able to sell you a copy. For example, BSI in the U.K. has them for sale at pounds 24. 3. DP 9945-2 is the last that we will see; under the new directives, DPs are no more; they are replaced by working drafts (WDs), which become committee drafts (CDs) before becoming DISs. This is not a big deal. - 6 - attempt to cover it in coded character set standards would break the ISO rule of one topic, one standard. This is all very well, but 9945-2 needs somebody to do the work, and WG15 may end up doing it itself if pleas for help from elsewhere in ISO fail. More work is on the way: 1003.2a, the User Portability Extension to POSIX.2, was registered for ballot as a Proposed Draft Amendment (PDAM) to DP 9945-2, giving the international community a chance to say what it thinks of the idea of standardizing interactive commands such as vi and language processors like cc -- or rather c89. Coordination The coordination arrangements which will make IEEE and ISO work on POSIX march forward in lock-step were almost complete before the small international rebellion on the matter of language independence made us stumble. (See below.) Because WG15 failed to register 1003.4, 1003.5 and 1003.9 at this meeting, the plan must be adjusted, although little slippage should result. We'll try to jump on board the revised plan at the next meeting. Internationalization In the one and a half days before the main meeting of WG15, its three rapporteur groups met. I attended the internationalization meeting, which had a hectic time of it: as the only rapporteur group directly concerned with the current content of 9945-1 and -2, we had a lot of comments to sift through prior to the main meeting. This we did, by the skin of our teeth. Our conclusions are largely reflected in the actions on the two drafts, reported above. Security The security rapporteur group is just getting off the ground. As with internationalization, activities scattered throughout JTC1 have to do with security. But in contrast to the current situation for internationalization, for security there is a (very new) subcommittee, SC27. Conceivably, SC27 could define some all-encompassing ISO security model5 which would not suit POSIX and the work of __________ 4. For oblique confirmation from "the father of ASCII", see [2]. - 7 - 1003.6, which is eventually to be integrated into the 9945 standards. The rapporteur group is doing all that it can to prevent this from happening. Happily, ISO is already aware of the issue, and has formed a special working group on security, to which WG15 will be sending a representative. Conformance_Testing The proceedings of the rapporteur group on conformance testing were rather overshadowed by the announcement of Project Phoenix. Quite from what ashes this has risen we did not learn; however, as it involves cooperation between the Open Software Foundation (OSF), UNIX International and X/Open, it is difficult to resist the temptation to speculate. But resist I shall... The goal of Phoenix is to develop a common conformance testing suite and methodology for the three organizations, or, to put it another way, to harmonize their activities in this area. Harmonization of standards for open systems is an important issue for WG15 in general. The issue affects the rapporteur group on conformance testing in particular because the European Community and NIST are giving a high priority to accrediting test houses which can check conformance to open systems standards. This means that they need to ensure that uniform test methods are being consistently applied. The rapporteur group will address this issue at its next meeting. In the mean time, WG15 has registered the current draft of the first part of 1003.3, which describes general test procedures, and we will review it before our next meeting -- despite the fact that there is as yet no well-defined route by which POSIX.3 can become an international standard. Language_Independence The issue of a making the POSIX standards independent of any particular computer language came to the fore at this meeting. What's all the fuss about? Well, ISO has firm -- and, in my view, sensible -- ideas about how to write standards which define the services available from information processing systems. Building on the doctrine that one standard should address just one topic, there ____________________________________________________________ 5. ISO likes models, and they're not without their uses. Work on a useful model for open systems is under way in several forums. - 8 - should be, says ISO, one document defining the service, and one or more documents describing ways of accessing that service. For example, a browse through the open systems interconnection standards reveals several groupings such as IS 8072, Transport Service Definition and IS 8073, Connection oriented transport protocol specification; and IS 8649, Service definition for the Association Control Service Element, and IS 8650, Protocol specification for the Association Control Service Element6. Similarly, in text communication, there is IS 9066-1, Reliable transfer -- model and service definition and IS 9066-2, Reliable transfer -- protocol specification, and in graphics, IS 7942, Graphical Kernel System functional description and IS 8651-1 through -3 giving GKS language bindings for FORTRAN, Pascal and Ada. (8651-4, giving bindings for C, is in the works.) POSIX, ISO has ordained, must eventually go along with this model, with the result that IS 9945-1, -2, and -3 (Operating system interface, shell and utilities, and administration respectively) will be concerned simply with defining services, and will not be bound to any particular programming language. The key word here is "eventually": because of the pressing need for an international standard for POSIX, a waiver has been granted, allowing the first version of the 9945-1 and -2 standards to be a combination of (purists might say "a collision between") a service definition and a C language binding to those services. The expectation is that a future revised new edition of each standard will be language-independent, and that bindings for the C language will be published as a separate standard at the same time7. So far, so good. But this is where the problems start. While this mandated future appears a long way off if one looks at the IEEE's work on POSIX.1, it seems very close __________ 6. Browsing through OSI standards may be a cure for insomnia. On the other hand, it may aggravate hypertension... 7. Under ISO's procedures, the bindings standards for POSIX will be allocated an ISO standard number just as soon as we register a base document for one of them. Until that happens, I shall have to refer to "future bindings standards", rather than having a convenient number to use as a handle. - 9 - when POSIX.4 (realtime extensions), POSIX.5 (Ada bindings) and POSIX.9 (FORTRAN-77 bindings) are considered. In the case of POSIX.4, language-independent extensions are being proposed for a standard which is not itself (yet) language- independent. And POSIX.5 and POSIX.9 define bindings to a set of services which is not explicitly defined, but rather is defined implicitly in terms of the binding to the C language given in POSIX.1. In the absence of a clear service definition, it is no surprise that, for good reasons which have to do with their respective languages, the Ada, C and FORTRAN versions of the operating system interface appear currently to be binding to slightly different sets of services. To some, the whole issue of language independence seems like an unnecessary and irksome imposition by ISO. In a recent posting to comp.std.unix[3], Doug Gwyn wrote: [Those of us who worked on 1003.1] did NOT set out to create a language-independent standard; C was specifically chosen for the obvious reason that it was the SOLE appropriate language for systems- level programming on UNIX, for a variety of reasons, including the fact that the UNIX kernel has a marked preference for being fed C data types. It is certainly true that, because, back in 1985, all the base documents for the IEEE POSIX work were written in terms of a C language interface, the working group made a good pragmatic decision to produce a standard centered on C. A different decision would have resulted in the standard which took longer to get out of the door, and which would not have been any more useful to its primary audience -- committed UNIX users concerned with the divergence among implementations of their chosen operating system. But the success of UNIX and of its offspring, POSIX, has greatly widened the audience for the standard. Whether we like it or not, ISO has revisited the original decision so as to ensure that the international standards for POSIX meet the needs of this new audience. As a result (to continue quoting from [3]): This "language binding" nonsense was foisted off on P1003 in an attempt to meet ISO guidelines. I think it must have been adopted by ISO as the result of Pascal types insisting that they never have to use any other language. Countering this, I would contend that, while the number of "Pascal types" is too small for their opinion to be of prime - 10 - concern, the number of FORTRAN types, COBOL types and perhaps even of Ada types is large enough that it would be at least polite to provide some well-defined means whereby these communities can create bindings which allow them to hook into POSIX services without having to learn a new programming language. In the future, the growing C++ community may decide to define the interface to POSIX services in an object-oriented manner; Steve Carter paid us a flying visit with news from the ANSI X3J16 C++ committee in order to open up channels of communication. Consider another topic which has come to the fore as POSIX has moved into the international arena: internationalization -- mechanisms which will allow non-English speakers to use POSIX-compliant systems without having to learn a new natural language. Like the current movement towards separating service definitions from bindings, this involves a considerable amount of work, yet does not appear to provide much that is of use to UNIX' original community of technical users. Accommodating the preferences (prejudices?) of ever greater numbers of people is, it seems to me, part of the price of success for the UNIX operating system. And it may well pay dividends. For example, internationalization work on regular expressions and collating has resulted in facilities which will be of use even to English speakers. Returning to the matter of the programming language used for bindings, it is true that AT&T-derived UNIX implementations prefer a diet of C data types. However, it certainly was an aim of 1003.1 to allow hosted POSIX implementations, which might well be riding on underlying operating systems with entirely different tastes. As a topical example, lightweight kernels such as Chorus and Mach live on messages, suggesting that their services could be bound to a data stream encoding8. I suspect that anybody who has tried to make ioctl() work across a network wishes that UNIX had anticipated their needs by following such a model from the start. But it didn't, and to redefine it in these terms would be a large piece of work which (thankfully) is a long way beyond the scope of WG15. __________ 8. More ISO-speak: broadly, if you have a protocol that lives above layer five (session) of the OSI stack, you'd better call it a data stream encoding. For example, the protocol for the X Window System is a data stream encoding by this definition. - 11 - There is no way in which all such requirements could have been anticipated, and accommodating the most important of them as the need arises inevitably causes pain. Both language independence and internationalization are unanticipated requirements which the international community wants retrofitted to POSIX on short order. And it's ANSI, as provider of the base documents to ISO, and the IEEE, as the body accredited by ANSI to produce the documents, that get beat on to do the real work, and to suffer the pain. In the view of WG15, the real work needed to make POSIX.1 a logical base for extensions such as POSIX.4, POSIX.5 and POSIX.9 is not being done fast enough. Trouble is, all standards are produced by volunteers -- often volunteers who have had to make a case to their managements that there's some percentage in their company being involved in standards work. There is clearly an eventual percentage in language independence for suppliers of POSIX-conformant systems if it encourages users of languages not traditionally found on UNIX systems to migrate to POSIX. But sadly, while not in any way criticizing the quality of the work done to date, there aren't enough IEEE volunteers interested in recasting POSIX.1 into language-independent form. Maybe, just maybe, if the international community is more interested than the U.S. in getting this work done, WG15 should encourage more people from outside the U.S. to participate actively and directly in the work of the IEEE. (Or, to put it another way, encourage more organizations outside the U.S. to put their hands more deeply into their pockets in order to pay for people to attend IEEE POSIX working group meetings.) The alternative is that WG15 does the work itself -- an alternative I'd rather not contemplate. For now, two action items on ANSI from WG15 sum up the situation: Pursue with vigour the production of a language- independent version of both 9945-1 and P1003.4 in conjunction with a C language binding for each in order that they are eligible as replacements for 9945-1:1990. Request the IEEE to expedite the completion of the language independent specification of 9945-1 that is precisely functionally equivalent to the existing 9945-1:1990 and provide a C language binding that is syntactically and semantically identical; and request that a detailed proposal status report on this issue including a - 12 - synchronization proposal be presented at the next meeting of WG15. Next_Meeting The next meeting of WG15 is in Seattle from 23rd to 26th October -- the week after the IEEE POSIX working group meeting in the same city (and the same week as the EUUG meeting in Nice, France9). Should be interesting! __________ 9. In two meetings, WG15 has managed to clash both with summer USENIX and with autumn EUUG. It almost looks as if we do it on purpose! But we don't, and will try to do better next year... - 13 - REFERENCES 1. June, 1990 Standards Update, Jeffrey S. Haemer, comp.std.unix Volume 20, Number 66, USENIX, 30 June 1990 2. Letter from R. W. Bremer, pp 34-35, Byte, volume 15, number 6, June 1990 3. Doug Gwyn, comp.std.unix Volume 20, Number 51, USENET, 27 June 1990 Volume-Number: Volume 20, Number 110
gwyn@BRL.MIL (VLD/VMB) (07/08/90)
From: Doug Gwyn (VLD/VMB) <gwyn@BRL.MIL> [ This was originally written as a letter to Dominic, but Doug agreed it would make a good comp.std.unix posting. -mod ] While I don't have any real problem with your use of quotations from my net posting, I do have a couple of comments on other things you said: The ballot also produced a number of suggestions in the area of internationalization, such as how to handle (and indeed, how to refer to) wide, or multi-byte, characters. For 1003.1, this is pretty straightforward. The C requirements on such character encodings are such that mbc strings can still be handled as uninterpreted NUL-terminated arrays of char. In the default "C" locale, a certain minimum set of characters must be represented, which permits the construction of portable filename strings. Even in the "C" locale, other characters are permitted, so for example a command-line argument containing "funny characters" can be used directly as an argument to open() etc. I know that there are various vendor approaches that make locales more visible to the operating system, but after all this is UNIX we're talking about, and one of the main lessons of UNIX is that the operating system can be designed to be happily oblivious to the uses to which people put the information that it manages according to simple rules. I first got involved in "internationalization" issues when I attended a BOF meeting at which the "expert" who was giving the presentation was explaining how complex the character set issues were, and when I said that I didn't see any inherent complexity was berated for my naivety. Years later, after studying the issues and conversing with the folks actively working in the field, I still maintain that simple solutions are possible. Unfortunately, vendors such as H-P started out with complicated schemes and have continue to think in those terms. This rubbed off on X3J11 when the multibyte character approach was adopted, which has the obvious problem that anyone programming for an international environment MUST change from traditional use of C strings to mbc arrays in his applications. The Japanese recognize this as an essential feature of their "long char" proposal, which X3J11 did NOT intend the mbc approach to be -- however, the fundamental need for library support using any such approach has now led to the Japanese requesting that such changes be made for the ISO C standard. I think the arguments I used for my alternative proposal to address these very concerns are being borne out, in spades. Returning to the matter of the programming language used for bindings, it is true that AT&T-derived UNIX implementations prefer a diet of C data types. However, it certainly was an aim of 1003.1 to allow hosted POSIX implementations, which might well be riding on underlying operating systems with entirely different tastes. To the contrary, we discussed this very matter in 1003.1 and decided that, while we did not wish to preclude layered implementations, we would not make any compromises to accommodate them. Very definitely our goal was to develop standards for genuine UNIX variants, not to provide a "Software Tools" style of Portable Operating System evironment. We used the same argument when we decided that NFS was simply going to have to be ruled non-compliant. UNIX applications rely on certain semantics of the file system that NFS did not properly support, and we decided that it would be a disservice to UNIX applications to remove the requirement that these useful semantics be preserved. Volume-Number: Volume 20, Number 115
domo@tsa.co.uk (Dominic Dunlop) (07/12/90)
From: Dominic Dunlop <domo@tsa.co.uk> In article <796@longway.TIC.COM> Dominic Dunlop <domo@tsa.co.uk> (that's me) writes of the forthcoming IS 9945-1 (POSIX operating system interface): > Its technical content will be identical to that of > ANSI/IEEE Std. 1003.1:1988, so implementors following > the U.S. standard can be assured of ISO compliance when > 9945-1 finally sees the light of day. I also say the same thing later: >Where all does this leave us? With no published >international standard as yet, but with a very good prospect >of having one this year. Until it arrives, keep using the >bilious green book, IEEE std. 1003.1:1988, as its technical >content is identical to that of the eventual ISO standard. A couple of people have pointed out to me that this ain't strictly so: a few small changes have crept in. If you say ``almost identical'', you're more correct -- if a little more worried. This year's revision of 1003.1 will bring it exactly into line with the eventual ISO standard. I have asked the respondents to make postings to this group summarizing the technical differences between the published ANSI/IEEE standard, and the ANSI/IEEE and ISO standards expected to be published later this year. (Thanks in advance.) -- Dominic Dunlop Volume-Number: Volume 20, Number 123