marwood@DMC-CRC.ARPA (G. J. Marwood) (01/02/88)
Thanks to everyone who replied to my message regarding character loss with MEX1.65 capture files, viz: >> The CP/M system that I use is an Apple ][+ and PCPI >> Applicard. While developing a new overlay and PCPI driver >> using interrupts, to avoid the usual character loss at high >> baud rates, I have come across a problem which seems to >> afflict the Apple/Applicard even without interrupts and at >> 1200 baud. At this baud rate, I experience no character loss >> on screen during normal operation. However, if I invoke a >> capture file, I do experience character loss at the place >> where the capture buffer is being written to disk (in this >> case it is the Applicard add-on Ramdisk). This loss occurs >> both on screen and in the captured file and, typically may >> be half a dozen or so characters. It seems that this bug has >> always been present in the MEX software that I have used, >> though I had not noticed it because I don't usually capture >> files as large as the capture buffer (16-17K). Another local >> user has done a similar test with MEX1.65B and a non-Apple >> CP/M machine and the loss does not occur. It does not, >> therefore, seem to be a basic problem with MEX. If any >> Apple/Applicard user has noticed this before and has any >> clues about solving the problem, I would be interested to >> know. This original message was obviously not totally clear, particularly to those not familiar with the Apple/Applicard. I should, therefore, make it clear that the Applicard is running as a Z80 co-processor in the Apple. The modem is connected into the Apple part of the system and a Driver in the Apple directly handles the interface with the modem (a Super Serial Card). In this Driver, which is entered following an Apple IRQ interrupt when a modem character arrives, there is a 4K buffer. This buffer is of a sufficient size (by a long way) that it should be unnecessary to use x-on/x-off protocol. The Ramdisk, which I use for the capture file, is the PCPI (2 in fact) Ramdisk which plugs directly onto the Applicard. It, therefore, has nothing to do with the Apple bus and is not controlled in the same way as drives which are on the Apple bus. I am aware that Apple interrupts are a problem when Apple 5 1/4 drives and standard drive controller are used, but they are not involved in this problem. The lost character problem, when the capture buffer is flushed to the disk file, occurs at 1200 baud, not just at the higher baud rates that I am really interested in. The following are some of the responses that I have received and some of my comments relating to them: > From: gatech!killer!bdav@hao.UCAR.EDU (Bob Davenport) > > Gordon, I have the same problem. But not with the same > hardware. The problem seems to be that an xoff is not sent > while the buffer is being written to disk. My OS(TurboDOS) > performs a read after write to verify all data being written. > With a large capture buffer, this takes about 1.5 seconds, long > enough to lose 10 to 15 characters. I have tried decreasing > the capture buffer size, but this does not seem to help. The > only method I have found that works, is to manually send an > xoff (^S) to stop the transmission, go to command mode, write > the buffer, re-open the buffer and then continue with the > capture. Not elegant, or problem solving, but it works. > > bob Davenport Bob - What system are you using ? Regarding the use of x-on/x-off, perhaps this is necessary with your system, though another MEX user local to me has done a test with a PMC 101 Micromate and hard disk, with STAT XON and STAT XOFF both OFF and has observed no character loss. I had thought that setting STAT QUEUE ON would ensure that there was no character loss during the disk write, but tests that I have done do not confirm this. I am not sure how often the modem status is being checked during the disk write operation. In the Driver which I have on the Apple side of the system, I do detect when the (4K) buffer is filling up and automatically send out a ^S. When MEX has reduced the backlog in the buffer a ^Q is automatically sent to restart the remote. However, this only seems to be necessary for my system at 9600 baud. At 1200 baud the buffer never seems to be filled. Despite this, I do lose 2 characters during the disk write. ____________________________________ > From: "Richard_F_Koehler.WBST128"@Xerox.COM > > Gordon, > I use an Apple II+ with a PCPI card also. I believe the reason > for the character loss is that Apple controls and writes to the > disk drive under software control rather than by using a > hardware disk controller. Since the timing is very critical, > all interrupts must be disabled while data is being written (or > read) to the disk drive. I also run MEX on my system but I have > not had occasion to exceed the capture buffer and I run at 1200 > baud or less, so I have not experienced the problem you > described. > > Rick Koehler Rick - As I have noted above the disk I am using for the capture file is the PCPI add-on Ramdisk. This has no connection with the Apple interrupts. Also, as noted, I am seeing this character loss at 1200 baud during the disk write. The original character loss problem, which started all this off for me was the loss that occurs at the begining of each line (this is nothing to do with capture buffers) at 1200 or 2400 baud and higher (depending on whether you are using a ][+ or ][e and which 80-col card). This loss is a result of the time taken by the 80-col card to scroll, which required re- writing the whole screen, whenever there is a CR and you are at the bottom of the screen. I think that it is also a result of the Apple/Applicard interface protocol, though I have not totally sorted this out yet. I am speculating that even with the modem port QUEUE ON, character loss occurs because the burst of characters which comes out of the queue once the disk write is complete is also too fast (faster than the normal 1200 baud character arrival rate) for the 80-col system to handle. In some way this may be linked to getting the characters into the capture file. This explanation may be nonsense, but it is the best that I can come up with at the moment. My lack of understanding of what the modem port queue is doing, and when it does it, is a limitation here. The above comments on character loss apply to MEX running with Apple interrupts off and, therefore, no buffering in the Apple Driver. ____________________________________ > From: "Richard_F_Koehler.WBST128"@Xerox.COM > > I suspect that even though you are writing to the RAMdisk, the > disk driver software still goes away completely from listening > to incoming characters for a while. You might want to check > there. > > Rick Rick - You are right, but I am not sure with STAT QUEUE ON, how often the modem is checked for new characters. In any case, in my arrangement the buffer in the Apple should cope with any wait while the disk is written. In fact my MEX overlay does not actually check the modem status (for new characters), it check for a difference in the 4K buffer input and output pointers. ____________________________________ > From: Michael Neary <MNeary.es@Xerox.COM> > > Gordon, > > I don't use MEX, but some general comments may help you: > > Your MEX has to be set so that it sends flow control characters > (usually X-ON/X-OFF) when writing to 'disk'. And the other end > of your conversation has to be willing to pause when it > receives flow control characters. > > If either of these is not true, then you will lose characters > when you go to disk, no matter what communications program you > are using. One possible exception is if your system can go to > 'disk' without disabling interrupts (rare). > > I would troubleshoot in this order: > 1) Is your MEX set to enable flow control? > 2) Are you using the same host as your friend is using? Does > that host support flow control? Is it the same flow > control as you have enabled on your MEX? > 3) Does your friend's RAM-Disk leave interrupts enabled in > RAM-Disk Writes? > > Cheers, > ~ Mike Mike - As noted above, I think that my comments regarding the buffer in the Apple answer your item 1. Item 2 - All the hosts involved support flow control. Item 3 - My friend is not using a RAM disk and he is not using interrupt driven MEX I/O. ____________________________________ > From MEDIN-T@SHARK.NOSC.MIL Wed Dec 30 12:27:09 1987 > > Well its not a bug its the design of the diskette driver. Apple > must have complete control of the cpu while they write the > diskette so they turn off interupts while they write the > diskette and vola! you lose characters coming down the line. I > suspect the ram disk tho quicker still locks out interupts. The > only solution we could come up with on kermit-65 was flow > control. So we stop flow before we write the diskette. If you > cant do flow control then i think you are stuck. Ted - As noted above the Apple disks are not involved. The PCPI Ramdisk has no influence on the Apple interrupts. ____________________________________ > From: Bridger Mitchell <bridger%rcc@rand-unix.ARPA> > > If mex is "smart", it should be polling the modem input status > between each sector-write call in the flushing routine, and > buffering any received characters for "catch-up" processing > when the buffer-flushing is completed. [Ron Fowler: does mex do > this?] But if not, it's likely that characters are arriving at > the modem port before the buffer-flushing operation completes. > Unless the arriving bytes are buffered before they reach mex, > the first ones will be overwritten by the later ones, creating > a gap in the input stream received by the mex software. Bridger - As you will have noted above, the arriving bytes are buffered before they reach MEX and this buffer does exercise automatic flow control. However, despite this, I am losing characters when the disk is written. > The real time required to flush a buffer is system-dependent, > probably dominated by the drive's step rate and the track > number of the capture file, and also affected by the CPU clock > rate. Another system with faster disks, a faster cpu, or even a > differently located file, could have no lossage. On your > system, if you capture to a ram disk, for example, there may be > no lossage. As noted, I am using a Ramdisk, operating directly in the Z80 environment. The CPU clock rate is 6 MHz at the moment (it can be increased to 10MHz). > > Many systems have 1-3 byte buffering in the UART chip that > receives the bytes from the modem; different systems use > different UARTS. More buffering can be obtained by writing an > interrupt-service routine for the modem ports; only a few > bioses have such support, but it may be possible to write the > mex overlay that way. > > --bridger I have implemented interrupts for both receive and transmit (on a 6551 ACIA) in the Apple modem-port Driver. In fact I know that the characters are coming from the modem into the 4K Apple buffer as I separately send them to the 80-col display directly from the Apple (the display characters do not pass through MEX). There is no character loss on the screen under this arrangement, only from the capture file. (In case there is some confusion here, there are two interrupt modes. In one, characters do not go via MEX to the screen in the other they do. The first scheme will work at 9600 baud, the second will not. I have included the latter to allow operation at lower baud rates and to retain the possibility of using the MEX terminal emulation modules). ___________________________________ > From: jimd%slacbbs.Berkeley.EDU@ucbvax.Berkeley.EDU > > I may be telling you something you already know, but MEX is > supposed to send an XOFF, save the disk buffer, then send an > XON. Possibly whatever system your calling either doesn't > respond to XON/XOFF or the modem may have a buffer that suspends > the result of the XOFF (so you lose stuff). I do not have an > Apple computer, but I use MEX (version 1.14). If you have the > source, check the area that saves the buffer, and make sure it > sends the XON/XOFF. Oh, and another thing I just thought of: MEX > is supposed to have a 'backup buffer' where stuff can go after > an XOFF is sent. Check this out too; it may be disabled or very > small on your version. BTW, is MEX1.65 the commercial version? > Or is the version number higher because of the port to Apple? > > Jim Dumser Jim - I think that I have covered most of your points above. I suspect that the 'back- up' buffer that you are referrring to is the modem port queue (defined by PQSIZE in MEXPAT). With no interrupts on (in the Apple) the character loss occurs with STAT QUEUE ON or OFF. MEX1.65 is also known as MEXPLUS of which it is the latest major revision. It has many features which do not exist in the PD MEX1.14. The features are described in the file MEX16SUM.WQ which is available by FTP from SIMTEL20 in PD1:<CPM.MEX>. There is also a file on GEnie called MEX16.AZN which provides a description. ____________________________________ Well, I have found this exchange of information quite useful and thought provoking. I am now beginning to draw some tentative conclusions. I think that I have been observing two different loss mechanisms here. With recent 1200 baud tests that I have done with the earlier MXH-AP55.ASM overlay and RINGBUF.A65 Apple Driver, I have found that I get character loss with and without STAT QUEUE ON and OFF (Note: this Driver will not run at 9600 baud, this is why I have been developing a new one!). I am not sure why the QUEUE is not preventing this loss, but I think it is to do with the Apple/Applicard interface and the time taken to write characters to the screen. With interrupts ON, this earlier version with its 256 byte buffer, does prevent character loss in the capture file. With my revised Driver I always lose 2 characters at the time the capture buffer is written to disk (the loss in the non-interrupt mode referred to above is always several more characters than this). I think, therefore, that I still have a bug to sort out in the Driver. (I should also point out that I cannot in the end use the modem port QUEUE as, at 9600 baud, the capture process gets so far behind that the queue is eventually overwritten). Thank you again for all of the responses. I still have some work to do! Gordon