ajc@bbn.com (Anthony J. Courtemanche) (01/03/90)
As a public service to whoever is doing any device driver works on Mips machines, I present the following information. ----------------------------------------------------------------------------- Lessons learned from porting a device driver to the MIPS M-2000 running Risc/OS 4.20. ----------------------------------------------------------------------------- I have successfully ported a device driver for a CMC ENP 100 ethernet node processor to the M-2000. My application calls for the device to transmit and receive ethernet packets over a private local area network which is used to support network distributed simulation. The device contains 256K of DRAM, an ethernet LANCE chip, and a Motorolla 68020 processor which controls interface to the device as well as housekeeping functions. Although software run by the 68020 can be purchased from CMC (and perhaps other sources) to run such things as TCP/IP, my application calls for a custom in-house program to run on the device. (We compile this downloaded program on a Masscomp.) The driver that I ported to the M-2000 had run on Sun/3's, Masscomps, and BBN Butterfly GP1000 multiprocessors running the Mach operating system. When I started my port, I had two machine specific questions that needed to be answered: 1) What is the best way to allocate Kernel buffer memory for the device to DMA (direct memory access) into? This memory must contiguous over several physical pages and must be unpaged (i.e. "locked down"), and uncached. 2) How can a user process map in this memory into its address space to that the user process can directly access this DMA buffer. I unfortunately did not get great responses from Mips, at least initially. The following are the solutions I ultimately used. 1) I allocated the DMA buffer statically in the device driver as a global array. Fortunately this statically allocated memory turned out to be in fact contiguous. Because this memory was in the kernel K0 segment, it was automatically non-pageable. By referencing it via the K1 segment, I could reference it uncached as well. Some people mentioned that there was a memory allocator that I could use in the kernel, but I was never able to get any details about what this allocator was. malloc(3) does not work in the kernel context. 2) I did not know it at the time, but there is a standard Unix way for mapping in device driver resources into an arbitrary process. If the device driver provides an xxxmmap() entry point to provide mappings between offsets into the device and physical pages to map these offsets into, an arbitrary process can map in device driver memory by opening the device and using mmap(2). I was able to successfully use this to map in the DMA buffer reserved by the device driver as well as the memory on the device itself. I placed my device in the A32 VMEbus address space at location 0x18000000. The device references the DMA buffer (in main memory on the Mips) via the 0x09 (extended 32-bit non-privileged) VME address modifier. There seems to be some contradiction in what address modifiers are legal in the Mips Technical Reference. Page 5-29 says that the Mips memory boards will respond to address modifier 0x0D (extended 32-bit privileged), but page 6-6 says that A32 devices must generate only 0x09 address modifiers, not 0x0D. I don't know whether 0x0D will work or not. It is clear that the Mips CPU cannot generate 0x0D address modifiers. The biggest obstacle in my device driver work was the fact that the -volatile switch on the compiler is buggy. Page 6-12 of the Technical Reference manual supplied by Mips suggests using this switch when compiling device drivers until you have declared all the necessary variables in the driver code as type volatile. This is to insure that the optimizing compiler does not reorder memory accesses to special device memory locations such as device control registers. When I compiled with the -volatile switch, my device driver could not reference the user structure correctly (u.). The offsets generated by the compiler were off by several words. I avoided using the -volatile switch by adding volatile declarations to the appropriate variables in the device driver source code itself. Mips has been notified (I think) about the problems with the -volatile compiler switch. Well, now my M-2000 and ENP-100 run happily together. I may plan in the future to try to measure the effect of the ENP-100 periodically accessing the DMA buffer in main memory to see how system performance is affected. The M-2000 has a private bus from cpu to main memory as well as the VME. I don't know when the private bus is used, but if the cpu always uses this bus to talk to main memory, than there should be zero additional bus contention and a probably very small amount of additional memory contention. --Anthony Courtemanche ajc@bbn.com