lenny@icus.islp.ny.us (Lenny Tropiano) (03/19/89)
In article <180@dms3b1.UUCP> dave@dms3b1.UUCP (Dave Hanna) writes: |>In article <633@icus.islp.ny.us> lenny@icus.islp.ny.us (*I*) write: ...[Talk about me trying to get the news up to patch level 17, recompiled with the dbm routines, and lastly using gcc]... |>With your experience and history, I have no doubt that you'll be successful, |>but I want to wish you good luck, and do let us know how it works out. Well thanks for the vote of confidence. I first want to thank all those who replied, a little too many to mention here. I must have gotten about 15 copies of the mdbm patches to use with expire (*sigh*), but thanks nevertheless. I'll enclose that patch that allows you to use the mdbm routines in place of the dbm routines, since the mdbm routines use a different extension on the indexed files. This way (using the patch), those files will be linked, and you'll have no problem. (Make sure you compile with the -DDBM and -DMDBM flags) Well here's a quick summary of my trials and tribulations of compiling this bugger. I'm finally up to patchlevel 17 of the news, and it is using the mdbm routines. This wasn't without some headaches along the way. I must have compiled the netnews software at least 5 times this past week (ugh). Patchlevel 17 did bring apon some bugs, but after accumulating a bunch of patches that were posted to the net shortly after the release of that, most, if not all, of the bad bugs were banished forever (or so I hope). I did compile the news with the gcc compiler (once) but gave up after it started core dumping all over the place. It wasn't something I really wanted to deal with at 3:00AM in the morning. I recompiled with the UNIX PC compiler, and installed it ... just in time for my days worth of news to be sent. A few people helpfully pointed out the problems with gcc and the dbm routines (where the core dump probably originated at). Brant Cheikes kindly gave me this information, but I also got it from a few other sources :-) |Some of the dbm routines return structs, rather than struct *'s. |Pcc and Gcc use incompatible struct-passing facilities. The upshot is |that if you compile with Gcc, you either must also compile your dbm |routines with Gcc, *or* you must use the -fpcc-struct-return option |(check that, I may not have gotten the name just right). | |Personally, I have a /usr/lib/libpdbm.a containing the pcc-compiled |dbm routines, and a libgdbm.a for the Gcc-compiled routines. Why? |Because I've heard that the pcc-struct-return option (new in 1.34, I |believe) is a bit buggy. |... |Brant Cheikes |University of Pennsylvania, Department of Computer and Information Science |brant@manta.pha.pa.us, brant@linc.cis.upenn.edu, bpa!manta!brant Gcc was a nice thought, but right now I don't have time to deal with possibly buggy code for the -fpcc-struct-return option. Maybe when it's fixed in 1.35, I'll deal with it again. Robert Granvin (rjg@sialis) pointed out that he is (and has been for a while) using the dbz.c set of routines that basically mimics the dbm routines store() and fetch(), which will satisfy the requirements for netnews at least. It's a short program so I'll include it here. All was necessary was to define -DDBM, and link it with dbz.o ... (instead of -ldbm or -lmdbm)... This seems a little easier, but who knows. A good idea that was pointed out to me, unfortunately 1 day too late, was a way to save the current history. I just removed the history.d/[0-9] files, and then did the /usr/lib/news/expire -R to [re]build the history files as DBM files (note this took over 1.5 hours on my machine with all the news I had). Kent Forschmiedt (kent@happym) suggested I do this: |o 2.11.14 does not keep a whole history file, only the ten pieces in | history.d, so cat them together and sort them: | | cat history.d/[0-9] | sort +1.6 -2 +1 >history | |o When everything has compiled, run expire -R to create the dbm files. | |o If you are using rn, you need to fix it, too. This is easy. Run | Configure as usual, do anything you need to do to it, and add | "#define DBM" to defs.h. News unbatches *so* much faster now. It used to take about 12-15 hours to unbatch 8000-10000 blocks of compressed news. Now it's done in about 2-4 hours. A big difference! I've noticed that expire's take longer, but that's because it has to recreate the history dbm files each time expire is run. This I don't care too much about, since it's chugging along at 3:00AM, and what do I care. For those who care, here's what my current history files look like for 15 days worth of history: 2480 -rw-r--r-- 1 news news 1269622 Mar 19 02:27 /usr/lib/news/history 7980 -rw-r--r-- 2 news news 4085760 Mar 19 02:27 /usr/lib/news/history.dat 9 -rw-r--r-- 2 news news 4352 Mar 19 02:26 /usr/lib/news/history.dir 9 -rw-r--r-- 2 news news 4352 Mar 19 02:26 /usr/lib/news/history.map 7980 -rw-r--r-- 2 news news 4085760 Mar 19 02:27 /usr/lib/news/history.pag Well if anything else is needed that I can be some assistance to, please let me know. -Lenny #! /bin/sh # This is a shell archive. Remove anything before this line, then unpack # it by saving it into a file and typing "sh file". To overwrite existing # files, type "sh file -c". You can also feed this as standard input via # unshar, or by typing "sh <file", e.g.. If this archive is complete, you # will see the following message at the end: # "End of shell archive." # Contents: mdbm.pat dbz.c # Wrapped by lenny@icus on Sun Mar 19 03:02:47 1989 PATH=/bin:/usr/bin:/usr/ucb ; export PATH if test -f mdbm.pat -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"mdbm.pat\" else echo shar: Extracting \"mdbm.pat\" \(1034 characters\) sed "s/^X//" >mdbm.pat <<'END_OF_mdbm.pat' X*** expire.c.old Mon Mar 13 17:11:14 1989 X--- expire.c Mon Mar 13 17:38:30 1989 X*************** X*** 841,846 **** X--- 851,864 ---- X (void) sprintf(tempname,"%s.dir", ARTFILE); X (void) strcpy(rindex(NARTFILE, '.'), ".dir"); X (void) rename(NARTFILE, tempname); X+ #ifdef MDBM X+ (void) sprintf(tempname, "%s.dat", ARTFILE); X+ (void) strcpy(rindex(NARTFILE, '.'), ".dat"); X+ (void) rename(NARTFILE, tempname); X+ (void) sprintf(tempname, "%s.map", ARTFILE); X+ (void) strcpy(rindex(NARTFILE, '.'), ".map"); X+ (void) rename(NARTFILE, tempname); X+ #endif /* MDBM */ X } X #endif X } X*************** X*** 1251,1256 **** X--- 1284,1297 ---- X (void) UNLINK(tempname); X (void) sprintf(tempname,"%s.dir", NARTFILE); X (void) UNLINK(tempname); X+ X+ #ifdef MDBM X+ (void) sprintf(tempname, "%s.dat", NARTFILE); X+ (void) UNLINK(tempname); X+ (void) sprintf(tempname, "%s.map", NARTFILE); X+ (void) UNLINK(tempname); X+ #endif /* MDBM */ X+ X #else /* !DBM */ X (void) UNLINK(ARTFILE); X #endif /* !DBM */ END_OF_mdbm.pat if test 1034 -ne `wc -c <mdbm.pat`; then echo shar: \"mdbm.pat\" unpacked with wrong size! fi # end of overwriting check fi if test -f dbz.c -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"dbz.c\" else echo shar: Extracting \"dbz.c\" \(5385 characters\) sed "s/^X//" >dbz.c <<'END_OF_dbz.c' X/* X Xdbz.c V1.0 X XCopyright 1988 Jon Zeeff (umix!b-tech!zeeff) XYou can use this code in any manner, as long as you leave my name on it Xand don't hold me responsible for any problems with it. X XThese routines replace dbm as used by the usenet news software X(it's not a full dbm replacement by any means). It's fast and Xsimple. X XBSD sites will notice some savings in disk space. Sys V sites without Xdbm will notice much faster operation. X XThis code relies on the fact that news stores a pointer to the history Xfile as the dbm data and that keys always end with > (meaning that a Xfull key should never match a partial key and that you don't need to Xkeep track of key size). It doesn't store another copy of the key Xlike dbm does so it saves disk space. All you can do is fetch() and Xstore() data. X XJust make news with the dbm option and link with dbz.o. X X*/ X X/* X Set this to the something several times larger than the maximum # of X lines in a history file. It should be a prime number. X*/ X X#define INDEX_SIZE 99991 X X#include <stdio.h> X#include <sys/types.h> X#include <sys/stat.h> X#include <fcntl.h> X#include <string.h> X Xlong lseek(); X Xtypedef struct { X char *dptr; X int dsize; X} datum; X Xstatic char buffer[1024]; /* used for fetch returns */ Xstatic int data_file; Xstatic FILE *index_file = NULL; Xstatic long data_ptr; X Xdbminit(name) Xchar *name; X{ X char string[1024]; X FILE * fopen(); X X if (index_file != NULL) X return - 1; /* init already called once */ X X data_file = open(name, O_RDWR); X X strcpy(string, name); X strcat(string, ".pag"); X index_file = fopen(string, "r+"); X X if (index_file == NULL) X return - 1; X X return 0; X X} X X Xdbmclose() X{ X if (index_file) { X fclose(index_file); X index_file = NULL; X close(data_file); X } X return 0; X} X X Xdatum Xfetch(key) Xdatum key; X{ X long index_ptr; X long data_ptr; X datum output; X char *strchr(); X long hash(); X long get_ptr(); X char *p; X X for (index_ptr = hash(key.dptr, key.dsize); (data_ptr = get_ptr(index_ptr)) != -1; ++index_ptr) { X /* we got a pointer into the history file, go see if it's the right one */ X X lseek(data_file, data_ptr, 0); X read(data_file, buffer, (unsigned)key.dsize); X p = buffer; /* convert string to lower case, as inews calls us that way */ X while (*p) { /* - ahby@bungia.mn.org */ X *p = tolower(*p); X p++; X } X X /* we can get away without the length info since we know that a X lhs portion of one key != any full key */ X X /* We use size - 1 since key has a null but hist file doesn't */ X X if (strncmp(key.dptr, buffer, key.dsize - 1) == 0) { X /* we found it */ X output.dptr = (char *) & data_ptr; X /* output.dptr = buffer; /* replaced per bug fix */ X /* output.dsize = strchr(buffer,'\n')-buffer + 1; not used by news */ X return output; X } X } X X /* we didn't find it */ X X output.dptr = NULL; X output.dsize = 0; X return output; X X} X X X/* add an entry to the database */ X Xstore(key, data) Xdatum key; Xdatum data; X{ X X return put_ptr(hash(key.dptr, key.dsize), *((long *)data.dptr)); X X} X X X/* get a data file pointer from the specified location in the index file */ X Xstatic long Xget_ptr(index_ptr) Xlong index_ptr; X{ X long data_ptr = 0; X int count; X X /* seek to where it should be */ X fseek(index_file, (long)(index_ptr * sizeof(long)), 0); X X /* read it */ X count = fread((char *) & data_ptr, sizeof(long), 1, index_file); X X if (count != 1 || data_ptr == 0) X return - 1; X X data_ptr -= sizeof(long); X X return data_ptr; X} X X X/* put a data file pointer into the specified location in the index file */ X/* move down further if slots are full */ X Xstatic Xput_ptr(index_ptr, data_ptr) Xlong index_ptr; Xlong data_ptr; X{ X long i = INDEX_SIZE; X X /* find an empty slot */ X while (i-- && get_ptr(index_ptr) != -1) X index_ptr = ++index_ptr % INDEX_SIZE; X X /* seek to spot */ X fseek(index_file, (long)(index_ptr * sizeof(long)), 0); X X /* write in data */ X data_ptr += sizeof(long); X fwrite((char *) & data_ptr, sizeof(long), 1, index_file); X X if (i > 0) X return 0; X return - 1; X X} X X X/* X A hash function X*/ X X/* some random # tables */ X Xstatic int tab1[16] = { X 1, 13, 17, 7, 26, 53, 32, 36, 39, 41, 45, 48, 6, 60, 61, 57}; X X Xstatic long tab2[64] = { X 2839722806L, X 2810802562L, X 3339723161L, X 1058549169L, X 1994454359L, X 3332972616L, X 2024961869L, X 1295774230L, X 1401270575L, X 550463474L, X 4149832130L, X 593802817L, X 19870556L, X 217796119L, X 2030513628L, X 2224253180L, X 4086624884L, X 2108878202L, X 4081713897L, X 1450431192L, X 2682324517L, X 3815655141L, X 3437116049L, X 1509200303L, X 3320765546L, X 592632684L, X 4115672507L, X 891714445L, X 2101308635L, X 1818339936L, X 1641458119L, X 2428086168L, X 955496606L, X 3732124452L, X 263468725L, X 4181304149L, X 1757671992L, X 4105130351L, X 415998664L, X 266182449L, X 4145924110L, X 3676915477L, X 3152392912L, X 3169100473L, X 3094692794L, X 449386310L, X 279014040L, X 1239368031L, X 4072577141L, X 2519571281L, X 1138855693L, X 2110586520L, X 2544733821L, X 621089071L, X 828396835L, X 901327585L, X 1528193535L, X 2448354394L, X 3531633039L, X 3272908074L, X 2376181107L, X 1827926286L, X 2717853871L, X 3969548575L}; X X Xstatic long Xhash(string, size) Xchar *string; Xint size; X{ X int value1 = 0; X long value2 = 0; X X register unsigned c; X X while (size--) { X c = *string++; X value2 += tab2[(value1 += tab1[c & 15]) & 63]; X value2 += tab2[(value1 += tab1[(c >> 4) & 15]) & 63]; X } X X if (value2 < 0) X value2 = -value2; X X return value2 % INDEX_SIZE; X X} X X END_OF_dbz.c if test 5385 -ne `wc -c <dbz.c`; then echo shar: \"dbz.c\" unpacked with wrong size! fi # end of overwriting check fi echo shar: End of shell archive. exit 0 -- Lenny Tropiano ICUS Software Systems [w] +1 (516) 582-5525 lenny@icus.islp.ny.us Telex; 154232428 ICUS [h] +1 (516) 968-8576 {talcott,decuac,boulder,hombre,pacbell,sbcs}!icus!lenny attmail!icus!lenny ICUS Software Systems -- PO Box 1; Islip Terrace, NY 11752