brad@looking.on.ca (Brad Templeton) (09/11/89)
It might be useful to list the ways in which articles can get in the wrong order. There are three classes of such problems: a) Inews goofs up on your own machine b) They are transmitted to you in the wrong order from a machine that got them in the right order c) The articles came by different routes resulting in the wrong order. And a special case: d) The articles only *look* in the wrong order. In fact, the first article was superseded (or cancel/repost). Here are some of my thoughts on different methods. If we can list these out, maybe we can make some news software that eliminates some or all of these. In particular, maybe we can make a batcher/unbatcher to do part of the work. A) The original comes in while expire's running. The followup comes in after expire runs, but before articles queued for later processing get processed. (Also applies to other things that spool articles for later processing) Solution: Empty the queue ASAP. If a new article comes in, empty anything in the queue first. B) This one's hard to figure. Normally uucp and other transport mechs send stuff in the order they got it, unless their sequence numbers wrap around. There might be problems caused by interrupted transmissions or people who move news at different priorities. Cross posting on a followup article that's not on the original can allow the followup to take a higher priority path, but this is rare. C) Again, for this to happen, we need something to go wrong. The faster of your feeds has to lose the original, so that the followup gets there before the slow route sends the original. Because normally originals and followups have the same newsgroups line, and they will both move on any channel that moves one. Problem C is the hardest to deal with. So my questions: What other methods are there? Which is the most *common* cause of articles in the wrong order? Can the most common causes be fixed? -- Brad Templeton, Looking Glass Software Ltd. -- Waterloo, Ontario 519/884-7473
karl@ddsw1.MCS.COM (Karl Denninger) (09/11/89)
In article <10478@looking.on.ca> brad@looking.on.ca (Brad Templeton) writes: >It might be useful to list the ways in which articles can get in the wrong >order. ..... >Here are some of my thoughts on different methods. If we can list these out, >maybe we can make some news software that eliminates some or all of these. >In particular, maybe we can make a batcher/unbatcher to do part of the work. > >A) > The original comes in while expire's running. The followup comes > in after expire runs, but before articles queued for later processing > get processed. (Also applies to other things that spool articles for > later processing) > > Solution: Empty the queue ASAP. If a new article comes in, > empty anything in the queue first. Better solution: Set "SPOOLNEWS" on your system. Then ALL articles coming in get spooled by definition. Local articles don't, but that isn't a problem since you can't follow up to something you haven't unpacked yet. This also doesn't require any modifications to the news 2.11b source or "C" News -- both can do this without trouble (it's how we've configured both here for ourselves and anyone I do news installations for). This also has the nice side effect that you never get into "uuxqt fights" which can happen with long-running uuxqt processes on some older uucp kits. >B) This one's hard to figure. Normally uucp and other transport mechs > send stuff in the order they got it, unless their sequence numbers > wrap around. There might be problems caused by interrupted > transmissions or people who move news at different priorities. > > Cross posting on a followup article that's not on the original can > allow the followup to take a higher priority path, but this is rare. This one is difficult. While uucp sends things in the order it gets the batches (most of the time), it doesn't always follow that uuxqt processes them in the same order. If I recall, uuxqt doesn't sort the list of work files before it starts -- which means you are at the mercy of the ordering that may 'fall out' in the directory scan. Usually this will be chronological -- but I wouldn't bet on it, especially if you often get files from more than one system at a time. This may be the major cause of mis-sorted articles at the present time. >C) Again, for this to happen, we need something to go wrong. The > faster of your feeds has to lose the original, so that the followup > gets there before the slow route sends the original. > > Because normally originals and followups have the same newsgroups > line, and they will both move on any channel that moves one. > >Problem C is the hardest to deal with. "C" is something I am not sure you can deal with. Sites do lose news occasionally; disks fail, power goes off, etc. When that happens you may very well get into this situation. If uuxqt is the source of the trouble, that may be the hardest to deal with, as most of us (myself included) don't have source access to our uucp packages! To deal with problem "B", one might try having the sending system order the files and pass on the ordering. This requires that the "rnews" command that is passed be changed to something like "rnews ddsw1xxxxxxx", where "xxxxxxx" is a hex-encoded time or a sequence number. Now the unbatcher can process in chronological order, and by site. This would require changes in the news software, although they should be relatively trivial. The site name has to be included, of course, else you risk two (or more) feeds scribbling on each other's batches. -- Karl Denninger (karl@ddsw1.MCS.COM, <well-connected>!ddsw1!karl) Public Access Data Line: [+1 312 566-8911], Voice: [+1 312 566-8910] Macro Computer Solutions, Inc. "Quality Solutions at a Fair Price"
syd@DSI.COM (Syd Weinstein) (09/11/89)
brad@looking.on.ca (Brad Templeton) writes: >It might be useful to list the ways in which articles can get in the wrong >order. >There are three classes of such problems: > b) They are transmitted to you in the wrong order from a machine > that got them in the right order >B) This one's hard to figure. Normally uucp and other transport mechs > send stuff in the order they got it, unless their sequence numbers > wrap around. There might be problems caused by interrupted > transmissions or people who move news at different priorities. > Cross posting on a followup article that's not on the original can > allow the followup to take a higher priority path, but this is rare. Actually this is the most common and hardest to fix. Many UUCP's decide what to do by sequentially reading the directory for jobs to process. Now entries in the directory are not assigned sequentially by id, but in the next available hole. As a site processes uucp jobs, the holes open up and batching fills them, thus later batches could easily go out earlier than their predicessors. This is especially true if a site picks up news less often that the sending site batches it. Since most people run binary copies of UUCP, fixing this one would be tough, the only cure is to have uucp process them in queue order not directory order and uucp doesn't do that unless you fix the source. -- ===================================================================== Sydney S. Weinstein, CDP, CCP Elm Coordinator Datacomp Systems, Inc. Voice: (215) 947-9900 syd@DSI.COM or {bpa,vu-vlsi}!dsinc!syd FAX: (215) 938-0235
henry@utzoo.uucp (Henry Spencer) (09/11/89)
In article <1989Sep11.033148.29604@ddsw1.MCS.COM> karl@ddsw1.MCS.COM (Karl Denninger) writes: >> The original comes in while expire's running. The followup comes >> in after expire runs, but before articles queued for later processing >> get processed. (Also applies to other things that spool articles for >> later processing) >> >> Solution: Empty the queue ASAP... > > Set "SPOOLNEWS" on your system... This is essentially a non-problem with C News. For one thing, all nonlocal articles go via input spooling; for another, incoming news can be processed during expire runs. (Incoming processing only *appends* to the history file, so it is not necessary to lock said file until expire hits EOF -- then, and only then, C News expire acquires a lock on the file and checks to see if any more has arrived. This reduces the lockup time to a few seconds.) -- V7 /bin/mail source: 554 lines.| Henry Spencer at U of Toronto Zoology 1989 X.400 specs: 2200+ pages. | uunet!attcan!utzoo!henry henry@zoo.toronto.edu