hayes@Apple.COM (Jim Hayes) (05/21/91)
Awhile back, I promised to tell you what we use/how we build our network.
This is pretty much it as for as the Product Development side goes.
I posted this originally in April '90. Now we've about doubled in size...
(This new version has been updated considerably. :-)
---
Apple's Engineering Network: (The network run by MIS is much
different, don't get them confused.)
We support three protocols (sometimes four, depending on who you
ask) on our internal Product Development network spanning 53 local
buildings and several foreign countries. They are AppleTalk,
TCP/IP, Apollo Domain and maybe DECnet.
We support AppleTalk Phase II and TCP/IP on all Ethernet and FDDI
networks within product development. We support Apollo Domain on
demand so it doesn't go everywhere. We may soon be supporting
DECnet in a limited fashion, but that is still up in the air with
a <5% probability.
We don't bridge anything.
There are two official Shiva FastPath boxes on our network that
handle DDP/IP translation for LocalTalk users. There are a few
renegade FastPaths on the network as well performing various
engineering tasks.
The network goes *everywhere* engineering goes, and appears at
every desktop in the form of a 10MB/s connection (Thin Ethernet,
UTP (pre 10baseT) Ethernet, or 10baseT.) and Localtalk. Depending
on where you sit, you may get RGB video at your desk too, or the
even the AppleTV cable system.
Stats: (as of 5/20/91)
515 LocalTalk networks (mostly laserprinters and backup
connections. If your coax breaks, just use
LocalTalk for a few hours until it's fixed.)
98 EtherTalk networks (mostly users, 10MB/sec)
100 IP Networks (the same wires as the EtherTalk net +
a couple transit networks)
225 AppleTalk Zones
326 Apple Internet Routers (most of the LocalTalk routing
is done here. )
31 IP/AppleTalk high-performance routers
5 IP only medium-performance routers
53 Buildings (not including foreign countries)
>1500 File Servers (Artifact of System 7.0 filesharing.)
105 QuickMail servers (w/Internet & Applelink gateways)
3811 QuickMail users
968 Laser printers (excluding shared spoolers)
1 Cray Supercomputer with a direct 100MB link to the
FDDI backbone. (XMP will be replaced with YMP-2E soon.
>1 Terabyte (1000 Gigabytes) of online network accessible
storage.
>5000 users (conservative guess. The software that provides
an accurate count isn't ready yet. :-)
23 Gigabytes routed per day is a typical load on some of our
high-performance routers.
Specs.:
Packets travel to stub networks mostly by fiber optic cable at 10
MB/sec. The backbone is a 100MB/sec FDDI Ring. (Soon to be split into
two separate FDDI rings with redundant routing between.)
Remote sites out of reach from our fiber-hub connect with 45Mb/s
T3, 1.5Mb/s T1 or 56Kb/s service. Transports may be encrypted
microwave (T3), dark copper or telco-supplied copper (T1), and
dark, switched, or telco copper for 56K. Various circumstances
may require that we mux several 56K links onto fiber as a transport
media to our facilities. The microwave system also supports a T1
channel and 2 broadcast quality composite video channels.
A packet arrives at its destination in under 5 milliseconds on
average, but it can be as fast as 1 ms. Packets from California
to Paris arrive in 112ms, packets to the East coast arrive in under
60ms. Packets to Japan arrive in 313ms.
The most distant end-to-end IP connection traverses 5 routers.
The most distant end-to-end AppleTalk connection traverses 9
AppleTalk routers.
Management:
The network is maintained by 12 people (3 technicians, 7 analysts
and a software engineer.) and 2 computers. Oh yeah-- there's the
boss in there somewhere. One analyst or technician is on-call
after-hours on a rotating basis.
The computers monitor network integrity and automatically alert
the appropriate people when failures occur via Motorola pagers.
They gather and analyze statistics by drawing pretty graphs and
perform trend analysis as well. This is accomplished by a Mac
II and a Sun 3 (Soon to be a SPARC!) running custom and commercial
software.
One person in each workgroup is appointed a network liaison and is
the first defense/resource used in solving user problems. The
liaisons (known as FONTs, or Federation of Network Technologists)
meet every so often to discuss networking issues and trade
troubleshooting hints.
If problems cannot be solved by the liaison, they come to the the
analysts. Depending on the skill level of the FONT, they may have
already restarted routers, used management tools or performed
physical layer troubleshooting.
Philosophy:
Each office is wired for: LocalTalk, 10Mb/sec. Ethernet and RS-232
(RS-232 comes from the telephone set.) Users use what works best
for them. Troubleshooting can occur down to the port in very short
order. With SNMP support it's even faster.
The network is not bridged anywhere. We believe strongly that a
routed network is a reliable network. Firewalls are quite necessary,
especially with the high number of third party systems sitting on the
network.
End-to-end connectivity: You can get to anywhere from anywhere on
the network using AppleTalk *OR* TCP/IP.
Security: No direct dial-in modems... Restricted access to/from
the national Internet. Periodic network sweeps for unauthorized
connections.
Purchasing: Invest in quality, high performance network components.
Avoid false economies in equipment, they are long term liabilities.
Test and Evaluate EVERYTHING before it goes on the network.
Stay on the bleeding edge of technology, and take calculated risks
when necessary.
Selected Hardware:
(This is a list of the major vendors we use, and does not constitute an
official endorsement for any particular vendor. Again, this is a partial
list of the major vendors)
Apple Computer Inc:
Macintosh II{x,cx,ci,fx)'s running the AppleTalk Internet Router
software.
Apple EtherTalk NB ethernet cards.
Cisco systems:
Various high-performance router models, mostly AGS and AGS+
boxes, usually 8 to 12 ethernet ports and 1 FDDI port on each.
Two 16 port terminal servers.
X.25 Protocol Converter for various test/production systems.
NetCentral Station monitoring software.
Cabletron:
Thick-Thin repeater, slowly being replaced by Twisted Pair stuff.
MMAC 10BaseT concentrators (one)
10Base T Transceivers
Fiber repeaters
Tons of TMS-3 Tranceivers
+Lots of RG58U coax & tranceiver cables.
Canoga Perkins:
8803 Fiber Repeaters, fantastic performance.
???? Fiber Mux for multiple V.35 to fiber connections
Digital Link:
CSU/DSU's for T1 connections.
FiberMux:
V.35 to Fiber for multiple V.35 connections
Microwave Networks, Inc.:
T-3/T-1/Video Encrypted microwave gear
Modunet:
"Phone-Net" like LocalTalk "drop boxes"
Network Systems:
EN641 routers. (less than 5) IP Only
8 ports per box + FDDI connections.
Retix:
Local and Remote SNMP bridges
Synoptics:
Model 2500 Concentrators in closets (twisted pair)
Model 2600 Twisted pair to thin-net converters.
3000 series chassis
Telebit:
TrailBlazer+, 1500, 1600, 2000, 2500 modems for
secured connections.
US Robotics:
Courier HST modems.
So, that's how we do it...
Jim Hayes, Apple Computer, Inc.
Inet: hayes@apple.com
UUCP: {amdcad|decwrl|ames}!apple!hayes
AppleLink: HAYESroy@phri.nyu.edu (Roy Smith) (05/27/91)
hayes@Apple.COM (Jim Hayes) writes:
-> Transports may be encrypted microwave (T3), dark copper or telco-supplied
-> copper (T1), and dark, switched, or telco copper for 56K.
What's "dark copper"?
--
Roy Smith, Public Health Research Institute
455 First Avenue, New York, NY 10016
roy@alanine.phri.nyu.edu -OR- {att,cmcl2,rutgers,hombre}!phri!roy
"Arcane? Did you say arcane? It wouldn't be Unix if it wasn't arcane!"