JAD@SWTEXAS.BITNET (08/05/87)
We are considering using volume shadowing on our VAXcluster RA81 drives.
Please comment on your experiences and general insights if you use
volume shadowing at your site.
Specific questions:
What hardware / software versions
How many volumes shadowed, types of drives, system, non-system
Performance changes (+ or -) actually seen
Volume backup procedures for system and non-system disks
Problems
Thanks,
Jim Dibble BITNET: JAD%UTA.MFENET@NMFECC.ARPA
University of Texas at Austin
Fusion Research Center RLM 11.230 Phone : (512) 471-6125
Austin, Texas 78712HIGHAMP@BYUVAX.BITNET (08/08/87)
At the Signetics Orem plant, we've been shadowing our cluster system disk. This last week, we stopped shadowing for a number of reasons. We were running six (yes, six) systems off of the same shadow set. The shadow set consisted of two (the limit currently _supported_ by DEC) RA81's. Why they limit you to two is something of a mystery, since their gurus at a recent DECUS regional meet were boasting about how great the performance was with three drives. Another limitation is that the "Cluster disk" cannot be on a shadow set. The main problem with volume shadowing is writes. The heads can read different parts of the disk nicely, but they have to synch when they write. We got around this by moving certain files (SYSUAF.DAT, ACCOUNTNG.DAT, JBCSYSQUE.DAT, etc.) to seperate disks. We also made sure that the page and swap files went elsewhere. DEC was unwilling to give their blessing to moving the operator log. Performance, particularly when considering the number of CPUs involved, was outstanding. We occasionally experienced some contention, and our FE complained about the number of collisions showing up in the error log, but the ease of maintenance more that made up for these infrequent problems. We stopped because of a probably unrelated hang up on our pair of HSC50s. They kept passing the buck to each other. I wasn't there at the time, but we have conjectured that the second shadow volume may have somehow faulted to an HSC other than the one that the shadow master was on. This doesn't work... DEC hardware folks complain that you cannot reliably have more than three systems on a single system disk. They may be right. Aside from the one crash which may or not have been related to shadowing, we saw reasonable success. After discussing the problem with the Colo Springs shadowing group, we are inclined to think that the problem was something of a fluke. They were quick to point out the the read optimization (sp?) performed by the shadowing software, combined with our relocation of files that take a lot of writes should have really made the disks fly. They also mentioned that there must be some reason why you can have sys0 thru sysf (minus one). I guess the only doubt is whether or not the hardware can handle it. I suppose what I am trying to say in my long winded way is: GO FOR IT! We had a great time. Just remember that DEC suggests no more than 40% writes to a database disks. I personally doubt that you will see a satisfactory upgrade in performance with this high a ratio of writes to reads. A couple of hints... We were running version 1.0, and have seen references to version 1.2, so maybe it runs even better now ;-) If you want to have fun, and have an HSC, you can do an HSC copy disk to disk, and create a temporary shadow set. It seems to use the same function as the shadowing sofware, and will show up as a shadow set during the copy. If you like faulting from one HSC to the other, don't do the shadow disks seperate from one another. You'll likely lose the shadow "slave". Just init the HSC. There are no error references supplied in vol. 4 of the the VMS doc set for shadowing problems. There were also none in our shadowing manual. It may be that this oversight has been corrected in recent revisions. It is VERY difficult to nail DEC down on shadowing. Their people obviously like it, but the disagreements between the folks at hardware and the good people at software were cause of some anxiety. There simply don't seem to be very many sites using shadowing. We stopped only because of political reasons. The ease of maintenance and security of a "backup" system disk were worth the pain. The only difference between a shadowed disk and a regular one is one quadword that contains the last shadow set creation date. If you mount the disk /OVERRIDE=SHADOW, this quadword is cleared, and you have a normal disk again. There is some lag at boot time, as the master disk is copied via HSC to the slave. Good luck, and have fun... Dave Higham System Programmer Signetics Corp. Orem, Utah
jdh@bsu-cs.UUCP (John Hiday) (08/08/87)
In article <8708080142.AA10031@jade.berkeley.edu> HIGHAMP@BYUVAX.BITNET writes: >At the Signetics Orem plant, we've been shadowing our cluster system disk. > ... > >We were running six (yes, six) systems off of the same shadow set. >... Another limitation is that the "Cluster disk" cannot be on a shadow >set. ^^^^^^^^^^^^^^ I assume that you mean a quorum disk here. You stated before that you have six systems in your cluster. A quorum disk is only needed for small (2 or 3 VAXen) clusters. DEC recommends that you do not use a quorum disk on larger clusters. >Performance, particularly when considering the number of CPUs involved, >was outstanding.... Same here. We've been running 3 785s and an 8650 off of a two member shadow set for about 6 months now and the difference is great. We've had everything bailed of the system disk for some time now (since we have 18,000 accounts and SYSUAF is > 30,000 blocks long) so we didn't have to make any special effort to move files which are written to a lot. Towards the end of school this year due to a severe lack of user disk space the bean counters wanted us to stop using shadowing and stop "wasting" a disk which could be put to "real" use. So we stopped shadowing for a week let the systems grunt along the week before finals. Well after that they stopped bitching about the waste of a disk. Not only was the difference noticeable to most users, SPM proved that without shadowing the system disk could only complete about half as many I/O requests per second (so it does increase performance almost x 2). >DEC hardware folks complain that you cannot reliably have more than three >systems on a single system disk. They may be right. Aside from the one >crash which may or not have been related to shadowing, we saw reasonable >success. The only shadowing related problems we have had are while booting. Since we shadowed the system disk, when booting the entire cluster at the same time there is always at least one, sometimes two of the 785s which get a device timeout when trying to bring up VMS. If you just try it again everything is OK. > >A couple of hints... > >If you like faulting from one HSC to the other, don't do the shadow disks >seperate from one another. You'll likely lose the shadow "slave". Just >init the HSC. No, no, no unless you want to sit back for about five minutes while the VAXes wait for the shadow set to stabilize. If the HSC hosting the shadow set dies it will take upwards of five minutes for it to failover to the other HSC and the dust to settle. Read the manual. This is only the way to go if you can stand to have your users hung up for a few minutes. Of course failing them over as normal, then adding the shadow copies back in later will also cause a severe performance hit during the ensuing shadow copy. Normally a shadow copy will take 20 minites or so when the system isn't too busy. However we have had the misfortune in the past of trying to add the second member while the system is very busy. Beleive me don't try it. We let it go for about 2 hours before finally killing the shadow copy. It really drug the system down during the copy too. >There is some lag at boot time, as the master disk is copied via HSC to >the slave. You may already be doing it, but the best solution to this is to delay mounting the second member of the set until the last thing. If you do it at the beginning of startup you've got the shadow copy pounding the disk contending with the startup process. -- == John Hiday UUCP: {ihnp4,seismo}!{iuvax,pur-ee}!bsu-cs!jdh == Ball State University / University Computing Services GEnie: JDHIDAY == Muncie, IN 47306