[news.software.b] Reading news: NNTP v.s. NFS for access to the database

fair@ucbarpa.Berkeley.EDU (Erik E. Fair) (07/04/88)

In the referenced article, bob@tut.cis.ohio-state.edu (Bob Sutterfield) writes:
	A secondary reason why I personally prefer to use NFS to
	read news is because it's actually faster than using the
	NNTP protocol.  I have run a "local" rn and a NNTP rrn
	side-by-side and rn comes up observably better in user
	responsiveness and feel.

Bob, can you quantify this? In particular, I'd like to know where
NNTP is taking the hit in performance. Context switching? I'd be
surprised if you found that NNTP had significantly higher overhead
than NFS for this application. Is it just that almost all the netnews
readers in existence were written (read "optimized") for having
filesystem access, rather than for network access (and NFS hides the
details better?)?

	anyone else got hard numbers?

	Erik E. Fair	ucbvax!fair	fair@ucbarpa.berkeley.edu

bob@kazoo.cis.ohio-state.edu (Bob Sutterfield) (08/02/88)

In article <4246@pasteur.Berkeley.Edu> fair@ucbarpa.Berkeley.EDU (Erik E. Fair) writes:
>In article <16342@tut.cis.ohio-state.edu> bob@tut.cis.ohio-state.edu (Bob Sutterfield) writes:
>>
>>A secondary reason why I personally prefer to use NFS to read news
>>is because it's actually faster than using the NNTP protocol.  I
>>have run a "local" rn and a NNTP rrn side-by-side and rn comes up
>>observably better in user responsiveness and feel.
>
>Bob, can you quantify this? In particular, I'd like to know where
>NNTP is taking the hit in performance.  Context switching? I'd be
>surprised if you found that NNTP had significantly higher overhead
>than NFS for this application.

Unfortunately, no, I can't quantify this, and I have no idea at all
where the hit is in the code - though see below for more specific
external observations.  I'd like to sit down with a profiler, but I
haven't had the chance (can you tell by the lag time on this response
that I've been away doing other things this summer?).

The biggest delay in rrn is when following a subject thread with ^N,
or when building an unread-articles subject list with =.  Kill files
seem to take forever, too.  This implies to me that you win with local
rn (or with rn reading from an NFS-mounted filesystem) because you
needn't ship the whole article across the wire just to peek at the
headers.

If my misunderstanding of rrn's guts is showing, I'll appreciate the
education I'm likely to receive :-)

>Is it just that almost all the netnews readers in existence were
>written (read "optimized") for having filesystem access, rather than
>for network access (and NFS hides the details better?)?

I think so.  The show-me-part-of-an-article functions are where local
filesystem access really shines.  If NNTP reader clients require
entire articles before they can peek at the header, then some smarter
clients are in order.  If local rn required that the entire article be
read via NFS before its headers could be scanned, you'd likely see the
performance balance tipped the other way.

If the header-scanning functions like subject threading and kill files
weren't part of rn, then I wouldn't notice the hit in rrn.  But then,
those niceties are a big reason I use rn.

Again, please correct my likely misunderstandings of reader clients -
but do it gently, please.
-=-
 Bob Sutterfield, Department of Computer and Information Science
 The Ohio State University; 2036 Neil Ave. Columbus OH USA 43210-1277
 bob@cis.ohio-state.edu or ...!{att,pyramid,killer}!cis.ohio-state.edu!bob

cmf@cisunx.UUCP (Carl M. Fongheiser) (08/04/88)

In article <19153@tut.cis.ohio-state.edu> bob@kazoo.cis.ohio-state.edu (Bob Sutterfield) writes:
>The biggest delay in rrn is when following a subject thread with ^N,
>or when building an unread-articles subject list with =.  Kill files
>seem to take forever, too.  This implies to me that you win with local
>rn (or with rn reading from an NFS-mounted filesystem) because you
>needn't ship the whole article across the wire just to peek at the
>headers.

Rrn *is* pretty stupid about such things.  If you're using one of the newer
ones, the '=' functionality isn't too bad, because then rrn will use the
XHDR extension.  ^N and friends could use the same trick, but they don't,
currently.  Kill files could be speeded up enormously if rrn would use
HEAD instead of ARTICLE to do the scanning.

Rrn's biggest problem (in my opinion) is that it always scribbles down
*everything* it got over the wire onto disk.  I don't see any reason why
rrn needs to write the whole article onto disk to do kill file processing.
For that matter, it never takes advantage of the hints the NNTP server provides.
The GROUP command always returns the range of articles in a newsgroup, but
rrn depends on using a local (and possibly inaccurate) copy of the active file.

In sum, the current rrn is an imperfect molding of a file-system based news
reader onto the NNTP model.  Rrn really *could* shine, but it'll take some
work.

				Carl Fongheiser
				University of Pittsburgh
				Computing & Information Systems
				...!pitt!cisunx!cmf
				cmf@unix.cis.pittsburgh.edu
				cmf@pittvms.BITNET

weemba@garnet.berkeley.edu (Obnoxious Math Grad Student) (08/04/88)

In article <11498@cisunx.UUCP>, cmf@cisunx (Carl M. Fongheiser) writes:
>Rrn's biggest problem (in my opinion) is that it always scribbles down
>*everything* it got over the wire onto disk.  I don't see any reason
>why rrn needs to write the whole article onto disk to do kill file
>processing.

Having written a comprehensive remote newsreader, the primary reason
for this is quite obvious to me: it's so much easier.  Doing a split
head and body fetch was quite tricky to get exactly right--among other
pitfalls, I had to suppress spurious(?) select(2) errors.

There's also the rare possibility of a /blah blah/a:j.  Well, maybe not
so rare--have you bothered to read news.admin recently?

Gnews doesn't support whole article searches, so this never occurs in
my case.  Also, Gnews does KILL processing on the fly, not all at once
at the beginning.

I'm in the process of rewriting Gnews internals to march down the
newsgroup in the background, caching up header and body information
while you're reading an article.  This is tricky enough in rn--to
do so asynchronously over NNTP is much more so.

Personally, I'd appreciate if NNTP would allow explicit buffering.  Ie,
not only "body ###", but say a "bodybuffer ### 30" to grab 30 lines;
the return message would tell how many lines remained to get.  The
all-or-nothing can be quite silly at times.  Among other applications,
this would permit fast implementation of KILLs based on a string within
the first so many lines.

>In sum, the current rrn is an imperfect molding of a file-system based
>news reader onto the NNTP model.  Rrn really *could* shine, but it'll
>take some work.

That's what I'm trying to make Gnews do.  Now that the first draft of
the manual is out of the way, I can dig into the internals again.

ucbvax!garnet!weemba	Matthew P Wiener/Brahms Gang/Berkeley CA 94720

rroot@edm.UUCP (Stephen Samuel) (08/05/88)

From article <19153@tut.cis.ohio-state.edu>, by bob@kazoo.cis.ohio-state.edu (Bob Sutterfield):
> In article <4246@pasteur.Berkeley.Edu> fair@ucbarpa.Berkeley.EDU (Erik E. Fair) writes:
>>In article <16342@tut.cis.ohio-state.edu> bob@tut.cis.ohio-state.edu (Bob Sutterfield) writes:
>>>A secondary reason why I personally prefer to use NFS to read news
>>>is because it's actually faster than using the NNTP protocol.  I

>>Bob, can you quantify this? In particular, I'd like to know where
>>NNTP is taking the hit in performance.  Context switching? I'd be
By my understanding, NNTP goes thru TCP/IP, while NFS is a different proto
alltoghether, which is optimized to transparent use of an FS over a net..
NFS seems to be a bit better at flinging the data fast (I remember comments
about people bragging about how their NFS implementation could swamp the
controller/software on X machine, and a solution Which ammounted to being
able to tell another implementation to "SLOW DOWN (damnit!)".

On the other hand, I've heard comments on how TCP/IP can be CPU intensive and
hardware support might help CPUs keep up with the abilities of faster 
network hardware.
-- 
-------------
 Stephen Samuel 
  {ihnp4,ubc-vision,vax135}!alberta!edm!steve
  or userzxcv@uofamts.bitnet

sow@eru.mt.luth.se (Sven-Ove Westberg) (08/05/88)

In article <11498@cisunx.UUCP> cmf@unix.cis.pittsburgh.edu (Carl M. Fongheiser) writes:
|In article <19153@tut.cis.ohio-state.edu> bob@kazoo.cis.ohio-state.edu (Bob Sutterfield) writes:
|>The biggest delay in rrn is when following a subject thread with ^N,
|>or when building an unread-articles subject list with =.  Kill files
|>seem to take forever, too.  This implies to me that you win with local
|>rn (or with rn reading from an NFS-mounted filesystem) because you
|>needn't ship the whole article across the wire just to peek at the
|>headers.
|
|Rrn *is* pretty stupid about such things.  If you're using one of the newer
|ones, the '=' functionality isn't too bad, because then rrn will use the
|XHDR extension.  ^N and friends could use the same trick, but they don't,
|currently.  Kill files could be speeded up enormously if rrn would use
|HEAD instead of ARTICLE to do the scanning.

This is not the NNTPs fault. I looked at it last spring and it
is not trival to fix since rn stores the "Header information" as
pointers to the diskfile, (sigh).

The = functionality is bad put this in your .rcinit and you will se.

-EXSUBJLINE="\
%(%s                                                  =\
^\\(.................................................\\)?%1) \
%(%t=^\\(........................\\)?%1:%t)"

All the problems is that rn relies a lot of diskfiles and pointers
in diskfiles.

Actually we miss ONE command in the NNTP protocol we can not abort
a transmission of an article so in some situations it will be slower.


Sven-Ove Westberg, CAD, University of Lulea, S-951 87 Lulea, Sweden.
Tel:     +46-920-91677  (work)                 +46-920-48390  (home)
UUCP:    {uunet,mcvax}!enea!cad.luth.se!sow
ARPA:    sow%cad.luth.se@ucbvax.berkeley.edu  (only dumb ARPA mailers)
Internet: sow@cad.luth.se
Bitnet:  sow%cad.luth.se@sekth.bitnet

aperez@cvbnet2.UUCP (Arturo Perez Ext.) (08/09/88)

From article <1167@luth.luth.se>, by sow@eru.mt.luth.se (Sven-Ove Westberg):
> In article <11498@cisunx.UUCP> cmf@unix.cis.pittsburgh.edu (Carl M. Fongheiser) writes:
> |In article <19153@tut.cis.ohio-state.edu> bob@kazoo.cis.ohio-state.edu (Bob Sutterfield) writes:


Actually I've found quite the contrary.  In my environment, using NNTP
is about 5 - oo (that's infinity) times fasting than using NFS.  Our
configuration here has the NNTP server machine on a different sub-net
than the one that I log onto.  When I mount the NEWS directory response
time is anywhere from 1 minute to 1/2 hour.  Don't know why but I would
guest that the NFS udp packets take too long on the round trip forcing
all kinds of retransmissions and other messy things.  Back before I 
got NNTP up, I would start a vn running as soon as I walked in and
attempt to read articles while waiting for compiles and so forth.
I never did manage to catch up.

With NNTP, I can read ALL my newsgroups within 1/2 hour.  I was really
disgusted (still am) with NFS once I found that out.

Arturo Perez
ComputerVision, a division of Prime
primerd!cvbnet!aperez
The difference between genius and idiocy is that genius has its limits.

bob@allosaur.cis.ohio-state.edu (Bob Sutterfield) (08/11/88)

In article <166@cvbnet2.UUCP> aperez@cvbnet2.UUCP (Arturo Perez Ext.) writes:
|...Our configuration here has the NNTP server machine on a different
|sub-net than the one that I log onto.  When I mount the NEWS
|directory response time is anywhere from 1 minute to 1/2 hour.  Don't
|know why but I would guest that the NFS udp packets take too long on
|the round trip forcing all kinds of retransmissions and other messy
|things...

Sounds like a severly misconfigured NFS installation.  It might have
been a severely overloaded gateway if you hadn't said that NNTP
performs fine.

|With NNTP, I can read ALL my newsgroups within 1/2 hour.  I was
|really disgusted (still am) with NFS once I found that out.

The problem and solution don't lie in the difference between NFS and
NNTP.  You should be "really disgusted" (but be nice anyway :-) with
whoever set up your systems, rather than with NFS.  You need to spend
some time looking at the retry, timeo, and retrans parameters in your
client /etc/fstab entries for mounting NFS filesystems.
-=-
 Bob Sutterfield, Department of Computer and Information Science
 The Ohio State University; 2036 Neil Ave. Columbus OH USA 43210-1277
 bob@cis.ohio-state.edu or ...!{att,pyramid,killer}!cis.ohio-state.edu!bob