minshall@OPAL.BERKELEY.EDU (07/31/87)
(LONG MESSAGE - print and read at your leisure.) I have the pleasure of being the gongfermer (aka nightman) for 4.3 telnet and telnetd. One of my joys is replying to the recent discussion of same. Another of my joys is admitting that everything wrong with 4.3 telnet and telnetd is my fault. William R. King started out the discussion asserting that 4.3 telnet was sending \n when the user typed carriage return. Dave Borman, from Cray, (who, by the way, spent a LOT of time helping to improve 4.3 telnet/d over the 4.2 versions) suggested that this was unlikely to be the case. This is, indeed, unlikely to be the case. The "problem", as Steve Alexander (and various others in both the public discussion, and in private comments) has mentioned, is that the 4.3 telnetd (the server telnet), on receiving a TELNET <CR><LF>, sends in a \n towards the application program. Now, Mr. John Shriver (jas) has a bit of a flame. I, personally, don't agree with the emotion of his flame, but I'm willing to talk about the philosophy. If I seem to be a bit derisive, please bear with me (this is a very sore subject). Jas states "In user Telnet, you should send <CR><LF> when the user types the 'doit!' key." Now, I would very much like to see a reference in the RFC (854) to that effect; I won't, though. I've looked many times. Jas, of course, is not the only person who believes this to be the case. I don't know jas, but I know and trust Bob Braden at ISI, and he is as emphatic as jas is about this. Ken Pogran also agrees. So does Stephen Casner (and he quotes chapter and verse! Alas.). But, honest to god, it isn't in the spec. The spec talks a lot about what <CR><LF>, <LF>, <CR><NUL> will do to the printer head. The spec does say: "Even though it may be known in some situations (e.g., with remote echo and suppress go ahead options in effect) that characters are not being sent to an actual printer, nonetheless, for the sake of consistency, the protocol requires that a <NUL> be inserted following a <CR> not followed by a <LF> in the data stream." (And, yes, the 4.3 implementations do that.) So, I do not believe that <CR><LF> is NECESSARILY what should be sent when the user hits the "Enter", "Return", whatever, key. What I DO believe is that the group of people who believe <CR><LF> is the way to go are relying on, in many cases, their participation in "let's define and bring up TELNET" meetings, projects, etc., back when "everything" was starting. I really wish these people, all of whom have much more experience than I in this business, would realize this point. So, how does 4.3 telnet (the client side) work? If remote echoing is going on, and the user types '\r', we send <CR><NUL>. If local echoing is going on, and the user types '\r', we ECHO '\r\n', and send <CR><LF>. NOTE: I think that 4.3 telnet (and I know that 4.3 telnetd) is broken in binary mode, sending <CR><NUL> when it should send just <CR>. Null stuffing isn't valid in binary mode. That is how 4.3 telnet works. Now, we slip over to 4.3 telnetd (the server). Again, the 4.3 server conforms to the protocol (not out of ignorance, not out of independence, but out of thought and some consultation). When we receive <CR><NUL>, we send '\r' towards the application program. When we receive <LF>, we send '\n' towards the application program. What do we do when we receive <CR><LF>? Well, what should we do? Clearly the user has typed something. What they are sending is an "end of line function" sequence (cf the note from Merton Campbell Crockett to this list); they are signalling "newline" if you will (and we would). The Unix "newline" character is '\n' (NOT '\r'). If the 4.3 server receives a <CR><LF>, we send a '\n' (newline) towards the application program. This is a problem if the user on the client side has no way of sending <CR><NUL>. If the client side has no way of sending <CR><NUL>, then the application program is not going to see what it would have seen had a '\r' been sent its way (if it could have noticed a difference; most Unix applications would not notice the difference [but some do]). We could, it is true, have mapped the <CR><LF> to '\r'. That would also have been within the spec. It would violate, somewhat, the philosophy of Unix (that '\n', not '\r', is the newline character). If would also mean that anyone unfortunate enough to be connecting from a client unwilling to send a <LF> alone would face a problem getting a '\n' send towards their program. I hope I haven't offended anyone overly. I AM pissed off (and what's wrong with being pissed off?). I hope that we can move from "Berkeley is violating the protocol" (which, except in the case of binary mode mentioned above, we aren't), to a Unix discussion of "how best could it work interface to Unix" (since I believe that the discussion of <CR><LF> -> '\n' or '\r' is a Unix discussion). Or, to a general discussion of "when should a client send <CR><LF>, when should it send <CR><NUL>, when should it send <LF>?" (recently there was some talk about defining line and character mode; there may be some interaction here). In summary: In client mode, we sometimes send <CR><LF>, we sometimes send <CR><NUL>, and we sometimes send <LF> by itself. In server mode, we always pad a standalone <CR> with a <NUL>; in server mode, we map incoming <CR><NUL> to '\r', incoming <CR><LF> to '\n', and <LF> to '\n'. None of these actions violate the letter (or, to my mind, the spirit) of the RFC (or the Boland amendment, for that matter). Greg Minshall ps - Dan Hoey feels it is laughable that "if the <CR><LF> is split across whatever buffer boundary telnetd is using, the code turns it into <CR> instead of <LF>". This is not laughable, but IS embarassing. The intent of this was to continue to support brain-damaged 4.2 implementations, which (as has been noted time-after-time) send <CR> when the user types carriage return. pps - Those of you who are accepting of this missive may believe I come from any part of Berkeley you want; however, for the benefit of those who would use this letter to criticize the "Berkeley clique", I feel you should know that I have nothing whatsoever to do with the Computer Systems Research Group [home of Berkeley Unix].
MRC@PANDA.COM (Mark Crispin) (08/01/87)
Greg Minshall -
The last time I used a hardwired terminal connected to a Unix
system, I found that the RETURN key on the terminal's keyboard
would cause the Unix system to believe that it had received a
newline. While it is possible that the terminal had been modified
so that the RETURN key sent a linefeed, there was a separate LINE
FEED key. I forget whether or not I tried using the LINE FEED key
as an alternative newline.
What your Telnet implement has, in effect, done is eliminate
transparency. Most implementors of Telnet programs across the
network have been very careful to make sure that transparency is
preserved, since the ARPANET was traditionally a heterogenous
network. Now, it may be true that today's Internet is a mostly
homogenous Unix network, but there are still some non-Unix boxes
out there.
I cannot help but get the impression from the tone of your
message that you have no real justification for your implementation's
behavior other than a possible interpretation of admittedly ambiguous
standards. You haven't offered any operational explanation as to
why your implementation must behave in the way it does. Is there
something particularly special about 4.3 that demands this behavior
that other versions of Unix did not have?
Please don't get me wrong; I have been a frequent critic of
the explanations in the Telnet standards as they are presently
reflected in the RFC's. I have periodically pontificated about
various finer operational details that are glossed over in the
standards or are inadequately explained; during a recent TCP/IP
conference I had a presentation on these details (I'm also
supposed to write an article about this for Connexions...).
We're not out to criticize you or make your life difficult.
Your mistake is only one of many that have come up in various
Telnet implementations over the years. Please accept our
judgement not as a criticism of your basically fine work, but
rather as a source of information on what you must do to get
your implementation interoperating correctly with the rest of
the network. We all have a shared interest in maximizing
interoperability, and the folklore we have developed over the
years is based on long, hard experience and much compromise.
Thank you.
-- Mark --
-------Ata@RADC-MULTICS.ARPA ("John G. Ata") (08/03/87)
Greg,
I have to disagree with your TELNET sending a <CR><NL> if the
user types a <CR> with local echoing and sending a <CR><NUL> if remote
echoing takes place. While this may be useful in some UNIX
applications, who expect this convention, it certainly confuses some of
us non-UNIX hosts out there trying to talk to you. For example, we use
remote echoing to supress the local echo of a password. Thus, when a
user types her password, remote echoing takes place and the <CR> typed
at the end of the password gets sent as a <CR><NUL>. This gets
interpreted at our end as a <CR> which is NOT our end-of-line character.
Of course, this causes a hang which can be very frustrating. Please, do
not confuse local/remote echoing with some sort of private mode where
TELNET ASCII characters get interpreted differently. I realize that
this may be convenient for you, but it makes talking to different
machines on the network, much more difficult.
John G. AtaCASNER@VENERA.ISI.EDU (Stephen Casner) (08/05/87)
(Another LONG MESSAGE)
Greg Minshall -
You are correct, the Telnet spec does NOT make it clear whether the
client program should send CR-LF or CR-NUL when the user hits the
"return" key. The discussion is clear for characters flowing in the
opposite direction to the NVT printer, and we are left to infer a choice
for the NVT keyboard to server direction from statements about symmetry.
The following sentence is the one that I think causes trouble; you have
undoubtedly read it before, but I'll quote it here for discussion:
Therefore, the sequence "CR LF" must be treated as a single "new
line" character and used whenever their combined action is intended;
the sequence "CR NUL" must be used where a carriage return alone is
actually desired; and the CR character must be avoided in other
contexts.
What is intended when the user hits the "return" or "enter" key? Those
who believe the return key means the user is finished with the current
line and wants a new line might quote the first part of the sentence to
show that CR-LF "must be ... used". Those who believe that carriage
return alone is desired, because on many (most?) terminals the "return"
key generates a single CR character, might quote the second part of the
sentence to show that CR-NUL "must be used".
NEITHER group can "prove" its case from the spec, but this is not a
legal contest. As Mark Crispin says, "we all have a shared interest in
maximizing interoperability".
I believe the Telnet spec should say exactly which of these two choices
should be used, just to avoid the present confusion of interpretation.
In any case, by the "robustness principle", EITHER sequence (CR-LF or
CR-NUL) should be accepted by the Telnet server to mean that the
"return" / "enter" key was pushed because we know there are some systems
that send one and some systems that send the other.
So, for the moment at least, my complaint was and is not that the 4.3BSD
telnet client sends CR-NUL but only about the 4.3BSD telnetd maps CR-LF
to '\n' instead of '\r'.
You say that you could have mapped CR-LF to '\r', but that it would have
violated the philosophy of Unix that '\n' is the newline character. I
disagree, because the philosophy of Unix does not require that terminals
send '\n' to the tty driver; instead the tty driver receives '\r' from
the terminal and maps it to '\n' WHEN APPROPRIATE. Since telnetd does
not feed the application program directly, but rather feeds a pty, I
claim telnetd should map CR-LF to '\r' and let the pty driver map to
'\n' when appropriate. This is the same argument that Kevin Carosso
wrote. This change was also posted to comp.bugs.4bsd on 29-Jan-87 by
J. Robert Ward.
You gave a second reason that anyone unfortunate enough to be connecting
from a client unwilling to send a LF alone would face a problem getting
a '\n' sent towards their program. Again I disagree, because the pty
will map '\r' to '\n' for those programs that don't distinguish between
the two characters and expect only '\n'. To fully utilize a program
that uses cbreak or raw mode and does distinguish between '\r' and '\n',
the telnet client must be able to send two distinct codes: CR-LF or
CR-NUL when the "return" key is pushed, and LF alone when the "linefeed"
key is pushed. Some keyboards don't have "linefeed", but they probably
do have "control" and "J" keys. Is there any client that won't send LF
alone?
There are many clients that don't provide any mechanism to send CR-NUL
instead of CR-LF. They are NOT violating the spec either! Dave Borman,
Kevin Carosso and Dan Hoey suggest the Bridge box may be in error
because it doesn't or can't send CR-NUL. While it might be more robust
to have an option to send either CR-LF or CR-NUL, it's certainly not a
bug to send CR-LF. As you point out, many of the old hands in the
Telnet business, participants in the "let's define and bring up TELNET
meetings", believe CR-LF to be the correct choice over CR-NUL.
Bob Braden refreshed my memory that at the "TCP Bakeoffs" when the first
implementations of TCP/TELNET were being tested against each other, only
Charles Lynn's TENEX implementation sent CR-NUL and everyone else sent
CR-LF. It was subsequently agreed that CR-LF was the correct choice,
but obviously that choice wasn't clearly stated in the spec. BSD Unix
chooses CR-NUL and that causes trouble for some of the older server
implementations (cf John G. Ata's message). My personal choice would be
for BSD Unix to change to send CR-LF, but I recognize that this would
cause trouble when connecting to telnetd's that map CR-LF to '\n'.
Perhaps it would be wise to enhance the BSD telnet program so the user
can select whether CR-LF or CR-NUL will be sent. Our goal should be to
clarify this issue in the spec so that eventually all machines can
interoperate naturally without the need for such inconveniences.
You have stated your case well for the choices made in the 4.3 BSD
implementation. But given the additional information put forth by
several people since then, do you still believe telnetd should map CR-LF
to '\n' and not to '\r'? What will 4.4 BSD do?
-- Steve Casner
-------alexandr@uiucuxe.cso.uiuc.edu (08/06/87)
Sorry if this is too much UNIX for the rest of you...
Well, I think Greg has explained things very nicely. Since he feels
that this is a UNIX related issue, let me toss my two cents in.
1) Pty(4), like tty(4) has CRMOD. Therefore, it can already map
<CR> to <NL> for you. in.telnetd should take advantage of this fact.
2) Programs such as tip run in raw mode, expecting carriage returns,
not line-feeds. Therefore, it should always be easy to give
them one. Again, sending <CR> seems more reasonable.
Therefore, it seems to me that in.telnetd should always send a <CR>
to the pty unless the user explicitly sends an <LF>. This seems to
allow maximum compatibilty w/ existing software and other operating
systems.
-- Steve Alexander, Workstation Programmer,
National Center for Supercomputing Applications
stevea@lurch.ncsa.uiuc.edu
stevea%lurch@uxc.cso.uiuc.edu
stevea@uiucvmd.bitnet
...!{ihnp4,convex,pur-ee}!uiucdcs!uiucuxc!lurch!steveaabe@j.cc.purdue.edu (Vic Abell) (08/06/87)
Among the five telnet client and server applications available to me for for testing - 2.9BSD (4.1c/4.2 ancestry), 4.3BSD, ULTRIX 1.2, Wollongong VMS 3.0, and WISCNET/VM (IBM) - only one combination passes all my operability tests: 4.3BSD. (My tests did not including chaining - client to server to client to server, etc., etc.) 2.9BSD has all the 4.2 failings and more - particularly with respect to local character processing and CR forwarding. ULTRIX 1.2 telnet doesn't process local characters at all, but it does echo them. Its telnetd is reputed to have problems with echo mode changes. Both are 4.2 based. Wollongong telnet emits unnecessary NUL characters and doesn't seem to send anything but <CR><NUL> for end of line. WISCNET works well with everyone, but has only line-at-a-time mode, so character oriented applications (e. g., vi) don't work. Dave Borman and Greg Minshall should be complimented for having done well a difficult job of practical engineering in 4.3BSD. They managed to strike a reasonable compromise with the leaky and overloaded telnet protocol. It would be even better if 4.3BSD telnetd were changed to forward '\r' instead of '\n' to the application. That would represent another, good, practical compromise for assisting chaining and other character-oriented applications. Vic Abell Purdue University Computing Center
lamaster@pioneer.arpa (Hugh LaMaster) (08/07/87)
Stephen Casner writes: ********************** >You are correct, the Telnet spec does NOT make it clear whether the >client program should send CR-LF or CR-NUL when the user hits the >"return" key. The discussion is clear for characters flowing in the : >The following sentence is the one that I think causes trouble; you have >undoubtedly read it before, but I'll quote it here for discussion: Therefore, the sequence "CR LF" must be treated as a single "new line" character and used whenever their combined action is intended; the sequence "CR NUL" must be used where a carriage return alone is actually desired; and the CR character must be avoided in other contexts. >show that CR-LF "must be ... used". Those who believe that carriage >return alone is desired, because on many (most?) terminals the "return" >key generates a single CR character, might quote the second part of the >sentence to show that CR-NUL "must be used". I agree with ALMOST all of this. Now, I have just read the spec carefully again myself, and it seemed clear to me that CR-NUL was intended originally to be used on output for NVT printer overstrikes on the same line. It isn't clear what, if anything, it means on input. However, through long use (misuse?), CR-NUL should, it seems, ALSO be INTERPRETED as a newline on input. Does anyone know what a TAC sends when it gets a CR from a keyboard? >NEITHER group can "prove" its case from the spec, but this is not a >legal contest. As Mark Crispin says, "we all have a shared interest in >maximizing interoperability". EXACTLY. Since interoperability is the PURPOSE, implementors have an OBLIGATION to follow the robustness principle, even if it inconveniences Un*x raw mode I/O. >I believe the Telnet spec should say exactly which of these two choices >should be used, just to avoid the present confusion of interpretation. >In any case, by the "robustness principle", EITHER sequence (CR-LF or >CR-NUL) should be accepted by the Telnet server to mean that the >"return" / "enter" key was pushed because we know there are some systems >that send one and some systems that send the other. Yes, the spec should have defined ONE sequence exactly for what user telnet should send to mean "newline". However, the spec does state very clearly that CR-LF MUST (read it again) be INTERPRETED as a newline. Therefore, user telnet MUST SEND CR-LF unless it has negotiated or otherwise knows that something else is preferable. Versions of BSD telnet which don't follow this are not consistent with the standard, pure and simple. In my opinion (let us not get personal). >So, for the moment at least, my complaint was and is not that the 4.3BSD >telnet client sends CR-NUL but only about the 4.3BSD telnetd maps CR-LF >to '\n' instead of '\r'. I do not follow this. 4.3BSD telnetd MUST interpret CR-LF as a "newline" (which means that a user PROCESS must receive \n (or \n\0 depending on how you look at the \0) - whatever happens in between is the implementors problem) and MUST SEND CR-LF as a "newline" to a (non-Unix) host. Obviously, there is disagreement about this, but it seems very clear to me. Read the spec again. The convention used to handle the raw-mode case must be consistent with this. Some discussion seems to point to the need for programs in raw mode receiving a \r or \r\0. I am not sure I follow this, but if the cooked mode case is clear, then the raw mode case should make more sense. (discussion about the difference between terminals and what a process receives internally deleted - the discussion is correct: there is NO necessary connection between the characters that telnet NVT uses and what Unix uses internally, although 99% of the time they are the same). Hugh LaMaster, m/s 233-9, UUCP {seismo,topaz,lll-crg,ucbvax}! NASA Ames Research Center ames!pioneer!lamaster Moffett Field, CA 94035 ARPA lamaster@ames-pioneer.arpa Phone: (415)694-6117 ARPA lamaster@pioneer.arc.nasa.gov "IBM will have it soon" (Disclaimer: "All opinions solely the author's responsibility")
nowicki@SUN.COM (Bill Nowicki) (08/07/87)
My two cents worth on this subject: In SunOS 3.4 I changed the telnet daemon to send \r instead of \n to the pty when it gets <CR><LF>. However, this broke the local line edit mode. In SunOS 4.0 I originally put in a check to see if local editing was being done, and if so sent \n to the pty, otherwise \r. Steve Casner's message, however, convinced me that a slightly cleaner fix would be to just keep the pty in the mode that treats \r as \n when doing local editing. I don't understand the rationale behind the code that explicitly cleared it. The change is simple (your line numbers may vary). In the function dontoption(): 922,927c924,925 < case TELOPT_ECHO: < /* < * we should stop echoing, since the client side will be doing it, < * but keep mapping CR since CR-LF will be mapped to it. < */ < mode(0, ECHO); --- > case TELOPT_ECHO: /* we should stop echoing */ > mode(0, ECHO|CRMOD); This, in combination with the change from \n to \r for incoming <CR><LF>s seems to work for most telnets in both line and character modes. - WIN
jerry@oliveb.UUCP (Jerry F Aguirre) (08/07/87)
In article <555127536.0.CASNER@VENERA.ISI.EDU> CASNER@VENERA.ISI.EDU (Stephen Casner) writes: >You say that you could have mapped CR-LF to '\r', but that it would have >violated the philosophy of Unix that '\n' is the newline character. I >disagree, because the philosophy of Unix does not require that terminals >send '\n' to the tty driver; instead the tty driver receives '\r' from >the terminal and maps it to '\n' WHEN APPROPRIATE. Since telnetd does >not feed the application program directly, but rather feeds a pty, I >claim telnetd should map CR-LF to '\r' and let the pty driver map to >'\n' when appropriate. I agree. The telnet daemon is not talking to "Unix" but to the pty. For the pty driver linefeed is not necessarily the end of line character. The end of line character depends on the mode of the pty. I would add that the telnet daemon should map CR-LF to '\n' if the pty is in newline mode but should map it to '\r' if the mode is not newline. The user can then, either automatically (via getty), or explicitly (via stty), set the newline processing as appropriate for their terminal. This offers greater functionality with no loss that I can perceive. Jerry Aguirre
karels%okeeffe@UCBVAX.BERKELEY.EDU (Mike Karels) (08/12/87)
Hey, folks, this discussion on telnet has gone much too far, and far astray. Let me summarize the actual problem, which is quite minor, its effect and the proposed modification to mitigate its effect. First, let me remind people that a real problem exists ONLY in the following circumstance: when a user connects by telnet from a host of a certain type (including Bridge terminal servers) to a 4.3BSD host, and THEN runs a program such as telnet that runs in character mode that distinguishes between <CR> and <LF>. This includes trying to telnet (or tip) from the 4.3BSD host to some other host that distinguishes between <CR> and <LF>. The problem occurs if the originating host sends <CR><LF> when it receives a return from the terminal, and has no option to send <CR><NUL> instead. With such a combination of hosts and operations, an interoperability problem exists. In all other situations, things work correctly. All of the implementations involved are following the specification. There is a simple, straightforward patch to the 4.3BSD telnet server that mitigates this problem. While I agree with Greg Minshall's excellent summary of the 4.3BSD telnet implementation and rationale, we have both agreed that changing the telnet server in a pragmatic way will avoid this problem. An "official" Berkeley bug fix for 4.3 telnetd will be sent to the tcp-ip list soon and to the Usenet news group set up for official Berkeley fixes (comp.bugs.4bsd.ucb-fixes). I think that the lessons learned are that (1) protocol specifications that allow options for behavior may allow correct implementations that fail to interoperate (no big news here), and (2) the Telnet specification has some problems, especially in the area of character at a time mode versus line at a time mode (see Greg's excellent discussion), and it may need to specify different behavior in the server-to-client and client-to-server directions. Mike