mike@fionn.lbl.gov (Michael Helm) (07/01/89)
In article <239@baird.cs.strath.ac.uk> jim@cs.strath.ac.uk writes: >In article <240038@grlab.UUCP> scott@grlab.UUCP (Scott Blachowicz) writes: >> [he wants to do network backups] >The way to do that is not to use NFS for backups! Use rdump to backup >files to a remote tape drive or other backup device. Alas, HP-UX users are going to find this somewhat difficult.... >Alternatively, pick up one of the many versions of tar that ... >support ... driving a remote tape device. This is a good idea, but what you'd often like to have is a backup program that would also backup device special files &c. Michael Helm (mike@fionn.lbl.gov)
ashton@hpfcdj.HP.COM (Ashton Delahousaye) (07/05/89)
There are a variety of ways to remotely archive files. It depends
of course on your target machine, the type of archive you want to
create, etc. The following example should give you some ideas to create
your own remote archives.
to/from an HP from/to an Apollo machine:
tar cf - .|remsh remote_host dd obs=400b of=/dev/rct8
remsh remote_host dd ibs=400b if=/dev/rct8|tar xvBf -
Similarly, remote archives can be done using cpio.
This was used to backup an SGI machine to an HP
find . -print | cpio -ocv | rsh hpfcli "dd obs=400b | tcio -o /dev/rct/c0"
!!! Be sure to use tcio when talking to traditional HP drives !!!
I'm not familiar with rdump, and know nothing about why it isn't implemented.
There may be an easier way to do remote archives. These methods or
variations have worked for me, with a variety of manufacturer's machines.
Ashtonwunder@hp-ses.SDE.HP.COM (Walter Underwood) (07/06/89)
In article <1631@zen.co.uk> vic@zen.UUCP (Victor Gavin) writes: >HP don't support rdump! rdump and rrestore shoud be in 7.0. Check the dump man page when 7.0 shows up. In the meantime, you may be able to get a contrib copy. wunder
burzio@mmlai.UUCP (Tony Burzio) (07/07/89)
In article <920031@hp-ses.SDE.HP.COM>, wunder@hp-ses.SDE.HP.COM (Walter Underwood) writes: > rdump and rrestore shoud be in 7.0. Check the dump man page when 7.0 > shows up. Pretty neat. Now where's rfbackup? Sheesh, nobody's ever satisfied :-) ********************************************************************* Tony Burzio * Push till it hoits! Martin Marietta Labs * mmlai!burzio@uunet.uu.net * *********************************************************************
burzio@mmlai.UUCP (Tony Burzio) (07/11/89)
In article <1800007@hpfcdj.HP.COM>, ashton@hpfcdj.HP.COM (Ashton Delahousaye) writes: > There are a variety of ways to remotely archive files. It depends > of course on your target machine, the type of archive you want to > create, etc. The following example should give you some ideas to create > your own remote archives. (etc)... > I'm not familiar with rdump, and know nothing about why it isn't implemented. > There may be an easier way to do remote archives. These methods or > variations have worked for me, with a variety of manufacturer's machines. I've been trying various permutations of these remote backup commands to do backups from our 300 cluster to our 835s' mag tape. Tar, and the other 'ios don't seem to work because they don't handle multiple volumes in the tape drive. Anyone know a work around? Does anyone know where to get rdump? Is Version 7 (which will fix my problems) comming along, soon, or is HP still digesting Apollo? :-) ********************************************************************* Tony Burzio * Sigh, back to my 9144... Martin Marietta Labs * mmlai!burzio@uunet.uu.net * *********************************************************************
scott@grlab.UUCP (Scott Blachowicz) (07/15/89)
/ grlab:comp.sys.hp / vic@zen.co.uk (Victor Gavin) / 7:30 am Jul 1, 1989 / >In article <239@baird.cs.strath.ac.uk> jim@cs.strath.ac.uk writes: >>In article <240038@grlab.UUCP> scott@grlab.UUCP (Scott Blachowicz) writes: >>>Is it possible to disable the Super-user-maps-to-uid-65534(aka -2) behavior? >> >>then patch the kernel variable called "nobody". NFS servers map root NFS >>requests to this user-id. Normally it is set to -2 (explaining the >>behaviour you describe above). Setting it to 0 will grant clients super >>user access to the server. I think that involves using adb on our kernel... >This is only because HP runs an old version of NFS. In the newer versions, the >exports file describes how to map the root id when access is made to files on >that file system from specific machines. Maybe I'll wait for that...which version(s) of HP-UX? >>>I want a nice easy way to specify some remote directories when doing >>>system backups and such... >> >>The way to do that is not to use NFS for backups! Use rdump to backup >>files to a remote tape drive or other backup device. > >HP don't support rdump! Would/will it support REMOTE FILES to be backed up to a LOCAL TAPE DRIVE? What I'm trying to do is get files off of 2 systems onto 1 backup. (backup my home directory and some other people's plus some other stuff spread across 2 systems). Thanx, Scott Blachowicz scott@grlab.UUCP
jack@hpindda.HP.COM (Jack Repenning) (07/18/89)
>>then patch the kernel variable called "nobody". >I think that involves using adb on our kernel... Exactly. Jack
vic@zen.co.uk (Victor Gavin) (07/19/89)
In article <240042@grlab.UUCP> scott@grlab.UUCP (Scott Blachowicz) writes: >/ grlab:comp.sys.hp / vic@zen.co.uk (Victor Gavin) / 7:30 am Jul 1, 1989 / >>This is only because HP runs an old version of NFS. In the newer versions, the >>exports file describes how to map the root id when access is made to files on >>that file system from specific machines. >Maybe I'll wait for that...which version(s) of HP-UX? Only HP know when they'll support the newer versions. But don't hold your breath :-) >>HP don't support rdump! >Would/will it support REMOTE FILES to be backed up to a LOCAL TAPE DRIVE? What >I'm trying to do is get files off of 2 systems onto 1 backup. (backup my >home directory and some other people's plus some other stuff spread >across 2 systems). Rdump -- and correct me if I'm wrong -- doesn't allow you to store file systems for different (remote) machines onto the same tape If you have ``access'' to the remote files that you want to back up then you can use something like "find <nominiated file systems> -depth -print | cpio -oacxBC > /dev/rmt/0h" This is where the changing of the kernal's map of UID 0 to nobody becomes useful. To do this just adb -w /hp-ux nobody ?W 0 $q The problem is that now root on any machine can access all the files on the machine that uses this kernal. Of course this b*ggers up security no end. What the newer versions of NFS allow, is for you to specify when you export a file system, which machines can have root access to that file system eg On our Sun machine we have this line to allow our 840 to back up its / directory / -access=zen:zenvec,root=zen (-access specifies which machines can access the file system and -root says which machines have root capability on that file system) >Thanx, >Scott Blachowicz scott@grlab.UUCP vic -- Victor Gavin Zengrange Limited vic@zen.co.uk Greenfield Road ..!mcvax!ukc!zen.co.uk!vic Leeds England +44 532 489048 LS9 8DB
corrigan@net1.ucsd.edu (Mike Corrigan) (07/21/89)
In article <240042@grlab.UUCP> scott@grlab.UUCP (Scott Blachowicz) writes: > >>>>I want a nice easy way to specify some remote directories when doing >>>>system backups and such... >>> >>>The way to do that is not to use NFS for backups! Use rdump to backup >>>files to a remote tape drive or other backup device. >> >>HP don't support rdump! >Would/will it support REMOTE FILES to be backed up to a LOCAL TAPE DRIVE? What >I'm trying to do is get files off of 2 systems onto 1 backup. (backup my >home directory and some other people's plus some other stuff spread >across 2 systems). > Yes, rdump will dump multiple remote filesystems to the same local tape:-| From localhost: % remsh remotehost1 rdump 0uf localhost:/dev/rmt/B0hn /dev/rdsk/c0d0s0 ; remsh remotehost2 rdump 0uf localhost:/dev/rmt/B0hn /dev/rdsk/c1d0s0 (the B in the tape device name isn't necessary) dumps filesystems /dev/rdsk/c0d0s0 on remotehost1 and /dev/rdsk/c1d0s0 on remotehost2 to the same tape on localhost.
jim@cs.strath.ac.uk (Jim Reid) (07/25/89)
In article <1649@zen.co.uk> vic@zen.co.uk (Victor Gavin) writes: >Rdump -- and correct me if I'm wrong -- doesn't allow you to store file >systems for different (remote) machines onto the same tape You are correct. Rdump is meant to dump a local filesystem to a remote backup device. This is the sensible way to do things. [In any case, rdump/dump read the raw disk to backup files, so that's the *only* way it *can* work.] Doing it the other way round (backing up remote files to a local device) is silly - there's no way to read a raw filesystem when it lives on another machine's disk. Using alternatives like tar or cpio that read files in the usual manner via the block buffer cache cause other problems, outlined below. >If you have ``access'' to the remote files that you want to back up then you >can use something like > > "find <nominiated file systems> -depth -print | cpio -oacxBC > /dev/rmt/0h" This is not a clever idea. Recursive traverses of a remote file system with something like find or ls -R will stress the fileserver to a high degree. (Try it and watch the load average on the client and server go through the roof while the perfomance of both gets so bad that they're both unusable.) It is much better to do this sort of thing on the machine where the files physically live. This saves network bandwidth and servicing gazillions of remote file system protocol requests and acknowledgements that are typically done at interrupt time on both the client and server. As for using cpio for backups, the less said the better... Jim
jewett@hpl-opus.HP.COM (Bob Jewett) (07/28/89)
> > "find <nominiated file systems> -depth -print | cpio -oacxBC > /dev/rmt/0h" > > This is not a clever idea. Recursive traverses of a remote file system > with something like find or ls -R will stress the fileserver to a high > degree. (Try it and watch the load average on the client and server go > through the roof while the perfomance of both gets so bad that they're > both unusable.) I just did try it. The machine doing the find/cpio was an 835, the filesystem being traversed was NFS-mounted from a 350. The test (somewhat unscientific) was the startup time for "vi /etc/hosts" (1.2 megabytes) on the 350, which is also the fileserver for 15 other systems. It took only about twice as long to start (30 seconds) as on an unloaded 350 (~15 seconds). On what sort of configuration do you have a response-time problem? > It is much better to do this sort of thing on the machine where the > files physically live. This saves network bandwidth and servicing > gazillions of remote file system protocol requests and acknowledgements > that are typically done at interrupt time on both the client and server. We have found that the 835 can do a "du" on /nfs/server350 about twice as fast as the 350 can do a "du" on its own disks. Bob Jewett [ not an official statement of the Hewlett-Packard Company ]
jim@cs.strath.ac.uk (Jim Reid) (07/31/89)
In article <63300007@hpl-opus.HP.COM> jewett@hpl-opus.HP.COM (Bob Jewett) writes: >> > "find <nominiated file systems> -depth -print | cpio -oacxBC > /dev/rmt/0h" >> >> This is not a clever idea. Recursive traverses of a remote file system >> with something like find or ls -R will stress the fileserver to a high >> degree. (Try it and watch the load average on the client and server go >> through the roof while the perfomance of both gets so bad that they're >> both unusable.) > >I just did try it. The machine doing the find/cpio was an 835, the >filesystem being traversed was NFS-mounted from a 350. The test (somewhat >unscientific) was the startup time for "vi /etc/hosts" (1.2 megabytes) on the >350, which is also the fileserver for 15 other systems. It took only about >twice as long to start (30 seconds) as on an unloaded 350 (~15 seconds). > >On what sort of configuration do you have a response-time problem? Oh, on pretty much any NFS client/server combination. It doesn't matter who makes the clients and servers, let alone the h/ware configurations of particular boxes. High-bandwidth I/O controllers and fast processors can make things easier - but such machines can still be brought to their knees by doing a du of a large (100 MByte+) remote file system. >> It is much better to do this sort of thing on the machine where the >> files physically live. This saves network bandwidth and servicing >> gazillions of remote file system protocol requests and acknowledgements >> that are typically done at interrupt time on both the client and server. > >We have found that the 835 can do a "du" on /nfs/server350 about twice as >fast as the 350 can do a "du" on its own disks. This does not make sense. You must be seeing caching effects here. The server has to do the work of fetching the information of disk regardless of whether the request was local or remote. That work is a fixed overhead, no matter if the reason for the work was locally generated (by a system call) or from a remote client's NFS requests. Once a kernel is given that work to do, it generates disk requests. It then leaves the data in the buffer cache, the inode table and sundry other places where it can be picked up without going to disk for it again. This data can then get passed back to a local process by means of a system call return. For an NFS request, the kernel has to encode some of the response packet with XDR, compose an NFS reply packet and queue it for transmission by the ethernet device driver. This is much more work than simply returning from a system call. All other things being equal, a local system call should therefore be much quicker than an equivalent NFS request. (Bear in mind that the client has also to compose & transmit the NFS request packet beforehand.) The times you have seen suggest to me two things: [1] Your machines have a monstrous system call overhead. This is unlikely. [2] Your server was able to satisfy your client's NFS requests from locally-cached data structures without having to go out to disk to get the information. This is much more likely, so the difference can be explained by 'cache hits' on the server - e.g. stat-ing inodes already in the server's inode table - that save the server from making disk requests. You lose this cacheing effect when the fileserver is busy and is turning over inodes and disk blocks at a high rate because of other I/O. [Try running du on a bit of the filesystem that has more inodes than can fit in the kernel's inode table. Read a few big files too, so that the buffer cache is unlikely to contain anything useful like bits of the disk inode list.] When the client's request can't be satisfied from a cache, the server goes out to the disk and you take a performance hit. Now that's not the end of the story. While the server goes out to disk, the NFS server process that deals with that request blocks, waiting for the disk request to complete. It can't deal with another NFS request while it waits. If that is repeated for the other NFS nfsd processes, the server will be unable to deal with incoming client requests, though perhaps only for a few milliseconds. Those requests get lost and eventually are retransmitted by the clients, causing further overheads and delays. This is why it is generally better to do local I/O than use some network file system. For day-to-day usage, the performance impact can be lived with - most users tend not to do *that* much I/O. Recursive directory traverses and stat calls (typical of find and du) are a different story because of the enormous amounts of NFS traffic they cause. That NFS traffic will get processed at interrupt time, so if the NFS servers are busy for long periods of time, users will see response times drop because the CPU spends most/all its time processing NFS requests. Jim
jewett@hpl-opus.HP.COM (Bob Jewett) (08/02/89)
> >I just did try it. The machine doing the find/cpio was an 835, the > >filesystem being traversed was NFS-mounted from a 350. ... > >On what sort of configuration do you have a response-time problem? > > Oh, on pretty much any NFS client/server combination. It doesn't matter > who makes the clients and servers, ... > ... such machines can still be brought to their > knees by doing a du of a large (100 MByte+) remote file system. The filesystem I was traversing was ~300MBytes. > >We have found that the 835 can do a "du" on /nfs/server350 about twice as > >fast as the 350 can do a "du" on its own disks. > > This does not make sense. ... > You must be seeing caching effects here. Yes, it does not make sense, but it is what we observed. Again, the filesystem was large, so any cacheing should not have been effective. The actual value of "ninode" (the maximum # of in-core inodes) on the fileserver is 760. I just tried it again, and got nearly equal clock times in the two cases, but when the 835 is doing the du of the NFS-mounted system (300MBytes, 746 directories) it only uses about 16% of the 835 CPU, plus about 40% of the fileserver's CPU in nfs-demons. The local du by the fileserver (350) uses about 55% of the CPU. (In both cases the clock time was about 6 minutes.) > The times you have seen suggest to me two things: ... Another possibility is that looking at the data from the disk is a significant part of the job, in which case it is possible to win by doing the processing on a fast CPU. > This is why it is generally better to do local I/O than use some network > file system. Agreed. It's just that the penalty is not always as large as you seem to have experienced. > ... so if the NFS servers are > busy for long periods of time, users will see response times drop > because the CPU spends most/all its time processing NFS requests. We have seen unacceptable response time due to NFS in two cases: 1) A hard-mounted system goes down and blocks all the nfsd processes 2) A program dumps a megabyte core in an NFS directory, using up all the net bandwidth Bob Jewett [ not an official statement of the Hewlett-Packard Company ]
crd@otter.hpl.hp.com (Chris Dalton) (08/02/89)
[ stuff about impossibly fast du over NFS ] > Another possibility is that looking at the data from the disk is a > significant part of the job, in which case it is possible to win by doing the > processing on a fast CPU. >> This is why it is generally better to do local I/O than use some network >> file system. Hmm. What if ... using NFS made a difference to the amount read from the disk in any one request, compared with a local read? For example, if du via NFS caused 4K reads on the 300, and du on the 300 caused 1K reads... I'm only guessing, of course, but I've seen odd effects like this before. Chris