[comp.unix.i386] SOLUTION to AIX Modem Wierdness

kevin@msa3b.UUCP (Kevin P. Kleinfelter) (07/21/89)

Recently I queried the net for assistance with AIX PS/2 not
handling modem control lines correctly.  Several people said that
they had the same problem, and they had found no solution.  In
documenting the problem for IBM, I found a work-around.

To summarize the problems:

Problem 1:  A 'fopen ("/dev/tty0", "r");' would block if CD were
  low (as expected), BUT IT WOULD NOT CONTINUE WHEN CD went high.

Problem2: If you get past problem 1, dropping CD would NOT cause
   the HANGUP trap to be executed until the NEXT attempt to open
   the tty (i.e. You still had a shell running after the  line
   was dropped).

Both problems are solvable!  The bug is in the "devices" program.
Running "devices" and selecting "dvam = 1" will create a minor
device number of 96.  The asy driver documents this minor device
number to mean "tty (not a printer) with modem control on
SERIAL_1".  However the correct number  should be 64.  The 96 is
the correct 64 with an additional, undefined bit set.  

DO NOT CREATE tty DEVICES WITH THE "devices" COMMAND!  DO IT BY
HAND!

The following is the text of a fax I just sent to IBM on our APAR.

################################################################
We have been experiencing trouble with the way AIX handles
modem control on the motherboard serial port.

To summarize:
   If a process attempts to open "/dev/tty0", the process
   will sleep forever if CD is not active BEFORE the open.
   The expected behavior is that the process would block
   until CD goes active, at which time it would continue.

   This causes a problem because we plan to have clients
   dial-in to a PS/2 model 80, running AIX.  The only way
   to prevent getty from sleeping forever on a port is to
   strap CD active, so that getty only sees CD as true.
   If CD is always true, then, if a client is disconnected
   due to line noise, the next client to dial-in on that
   phone line will get the previous client's process, and
   not a new login.



The following will reproduce APAR # IX04764:

Take a PS/2 Model 80/386 with 8192KB of memory and a 300MB
   ESDI disk.
Install AIX (Base operating system) using default minidisks.
Update  Base AIX.
Install DOS Merge
	C Language
	Graphics Support
	Application Development Toolkit
	Asynch. Terminal Emulation
	O/S Extensions
	Workstation Host Interface Prog.
Update all installed programs.

Attach one end of a straight, 25-pin ribbon cable to the 
motherboard serial port (configured as COM1). Attach the
other end to a breakout box (the connector labelled "To DTE").
Attach the other end of the breakout box to NOTHING.

Issue the following command:
	mknod /dev/tty0 c 5 64
(Note that 5 was chosen as the major device number because the
default "/etc/master" defines the major device number for the
"sa" driver to be 5.  64 was chosen as the minor device number
directly from the "asy" documentation in Tech Ref. vol 2, page
5-3 (manual #SC23-2033-0).

Strap DSR, CTS, and CD off (I strapped them to TD, since it
is low at this time).

Compile and run the following sample program (as root)::

   #include <stdio.h>
   void main (void)
   {
      fprintf (stderr, "program begins\n");
      open ("/dev/tty0", O_RDWR, 0);
      fprintf (stderr, "program ends\n");
   }


The PS/2 displays "program begins" and asserts RTS and DTR.

Strap DSR, CTS, and CD on (I strapped them to RTS, since it
is active at this time).  The program then displays
"program ends".
This is as expected.  Repeat the test (strap low, run program,
strap high).  The program continues to work as expected.

Now
   rm /dev/tty0

Run the devices command, add ttydev tty, with the following:
   tt	vt100
   dvam	  1
   bpc    8
   pt     none
   rts    19200
   ae     false
   nosb   1
   elevel 1,4
   ixp    false
   xscan  false
   aa     false
   pro    dc

The result includes the creation of "/dev/tty0" with a major
device number of 5, and a minor device number of 96.  The
96 is the expected 64, with an additional, undocumented bit
set.  

Strap DSR/CTS/CD low.
Run the test program -- it prints "program begins".
Strap DSR/CTS/CD high. The program does NOT proceed.
Interrupt the program via the Break key.
Leave the pins strapped high.  
Run the test program -- it runs to completion.

Strap the pins low again.
Run the test program -- it does not complete.
Interrupt via the Break key.


CONCLUSION 1: The devices command creates /dev/tty0 with a
   bad minor device number of 96, when 64 is desired.


##########################################################

"rm /dev/tty0".
"mknod /dev/tty0 c 5 64".
Run the test program -- it prints "program begins".
Strap the pins high -- the program does not complete.
Interrupt via the break key.
Leave the pins high.
Run the test program -- it DOES complete.

CONCLUSION 2: After the minor device number is set to 96,
   the device driver is still "broken" when the device number
   is set back to 64.


##########################################################

"rm /dev/tty0".
"mknod /dev/tty0 c 5 96".
Strap DSR/CTS/CD to RTS.
Run the test program -- it completes.
"rm /dev/tty0".
"mknod /dev/tty0 c 5 64".
Strap DSR/CTS/CD low.
Run the test program -- it prints "program begins".
Strap the pins high -- the program completes.


CONCLUSION 3: To clear the "broken" device driver, you must
   SUCCESSFULLY open /dev/tty0 with minor device of 96.



##########################################################

SUGGESTED ACTION:
   Fix "devices" command to create /dev/tty0 for com1 with
   a minor device number of 64.
-- 
Kevin Kleinfelter @ Management Science America, Inc (404) 239-2347
gatech!nanovx!msa3b!kevin