Microsoft chooses sonet for corporate network
The company?s new sonet-based Puget Sound network is being touted as the largest corporate atm network in the world. Are you surprised?
Grace F. Murphy Assistant Editor
When you are Micro-soft, you need an engine as powerful as any telecommunications carrier to power your corporate network. The foundation of Microsoft Corp.`s upgraded Puget Sound ring network in Redmond, WA, is a powerful Synchronous Optical Network (sonet) backbone built with 4-fiber bidirectional line-switched rings (blsrs).
In an effort to consolidate technologies, improve scalability, and keep up with bandwidth demands, Microsoft removed its centralized Fiber Distributed Data Interface (fddi) network and created a system that it considers "the largest corporate atm [Asynchronous Transfer Mode] network in the world, possibly riding over the largest private sonet backbone in the world."
Before the upgrade to an atm backbone in 1997, the Microsoft campus contained sonet equipment mostly geared toward DS-3 (44.736-Mbit/sec) and DS-1 (1.544-Mbit/sec) rates, according to Gregg Atkins, Microsoft Global Networking senior network engineer.
The transport backbone now contains 10 nodes of Alcatel SM sonet multiplexers in a blsr configuration. It also uses an Alcatel 1631 SX Wideband sonet Digital Cross Connect and an Alcatel 1633 SX Release 10 Broadband Digital Cross Connect (see figure).
The legacy network
During the five years before the upgrade, Microsoft used a centralized fddi network consisting of 2000 servers and 23 fddi rings connecting 30,000 PCs and 43 buildings on the Puget Sound campus. Microsoft used Fujitsu`s flm system in a unidirectional path-switched ring architecture. Digital Equipment Corp.`s Gigaswitch connected the fddi rings for corporate data.
US West, Denver, CO, installed the Alcatel crossconnects in the network more than three years ago and also maintains service for the network. Before being used in conjunction with the rings in today`s network, the crossconnects handled call center and voice center traffic, according to Fred Ellefson, Alcatel`s senior director of marketing and business development.
Although the legacy network would be considered state-of-the-art on many campus networks, Atkins says Microsoft began evaluating capacity limitations due to the architecture. "We had separate voice, video, and data networks. We`re converging toward the common infrastructure that supports improvements and efficiencies in the labor it takes to manage that work and allows us to be `out with the old, in with the new` and eliminate technologies that are no longer appropriate for us," he says.
Reasons for change
Much of Microsoft`s product development work takes place on and around the Puget Sound campus, and employees were finding that the network was slowing them down. The amount of data traveling on the network has increased over the past five years as the company has grown.
The company also found that its network was not scalable. A network capable of handling multimedia and multicast technologies was needed, Atkins says. The company chose a 4-fiber blsr approach because it has double the capacity of a 2-fiber blsr and more bandwidth availability. In addition, only one protection switch mode is available with a 2-fiber blsr--the ring switch. If all of the working traffic is on the protect ring, the entire ring is used up. The 4-fiber blsr has the same protect mode as well as a protect mode called span-switching.
"Because that span switch lets you use your protect-bandwidth more efficiently, you get much better availabil- ity by using a 4-fiber blsr," Alcatel`s Ellefson says. "In the Microsoft case, I think they needed both the capacity and the availability, and I think that`s what drove them to the 4-fiber blsr approach."
Atkins concurs. "The 4-fiber bidirectional ring is the most robust of all the ring architectures, both from ability to survive multiple failures as well as having high capacity. Those two reasons are why we chose that platform to continue the sonet architecture. Moving from DS-1 to DS-3 over to OC-3 [155-Mbit/sec] and OC-12 [622-Mbit/sec] rates suggested a higher-capacity ring. In fact, if OC-192 [10-Gbit/sec] technology were mature enough, we would`ve gone straight to that. But it wasn`t available in ring format at the time we were doing our evaluation," he says.
The upgraded network
Mike Rabe, account manager on US West`s Microsoft account team, says one option Microsoft originally looked at was a dark-fiber mesh network tying together atm switches. At the time, Alcatel had just introduced its 4-fiber, bidirectional ring.
"We were aware of the Alcatel products because we had already deployed Alcatel dcss [digital crossconnect systems] within the Microsoft network. We worked with Alcatel on how best to incorporate its 4-fiber bidirectional technology into the network design and came up with a three-ring configuration," Rabe says. One ring has four nodes, and two rings have three nodes each.
The three independent rings were installed and interlocked via Alcatel`s crossconnects and carry much of Microsoft`s backbone atm traffic throughout the campus.
Had Microsoft connected the rings using DS-3 or DS-1 lines, it would have been difficult for the company to set up a high-capacity data network, Ellefson says. "But OC-12 at 622 Mbits/sec gives some really large pipes that they can use to run atm, data traffic, and the various types of local area networks that they`ve got set up on their campus," he explains.
Lower-speed traffic rates such as DS-1 for private branch exchanges will drop to the 1631 crossconnects, Atkins says. Dropping an OC-12 out of the Fujitsu OC-48 nodes into the Alcatel crossconnects creates a mid-span meet. "The traditional grooming and filling function that those crossconnections perform, as well as the service restoration capabilities of the digital crossconnect, is why they`re there," he says.
Cisco`s Catalyst 5000 switches and its 7500 routers are at the edge of the network. The routers provide redundancy and increase port capacity, and the switches supply 10/100-Mbit/sec Ethernet to Microsoft desktops. The Cisco routers and switches will work with atm switches from fore Systems in the atm backbone.
"The combination of atm switching and sonet transport addresses scalability and manageability and reliability in the quantities that our development organization is requiring from us. Essentially, we`re acting as a carrier to our own development group," Atkins says.
Multiple runs of 144-strand singlemode fiber stretch within the Microsoft campus right-of-way. Meanwhile, 1- to 2-km runs of multimode fiber appear in lower quantities, such as 24 fiber-pairs. "We feel that when we go above OC-3 rates we`ll be using singlemode exclusively, except for very short runs. Between racks we can do multimode at 622 Mbits/sec, but we`ve got runs of multimode that basically were distance-limited based on the fddi specs," he says.
Installers worked nights and weekends to accomplish the 61-building upgrade goal of 4 buildings per week and 10 to 20 each month. The schedule was intended to minimize the impact on product development and release schedules. Most of the buildings are on the Puget Sound campus, with approximately one-third off-site. Connecting off-site buildings with the main campus is one of the reasons Microsoft went with sonet technology and US West, according to Gurpreet Pall, Microsoft lead program manager for information technology communications planning management.
"We have to connect in large quantities to these buildings off-site. The economy of scale suggested sonet technologies in the mid-1990s, and the convergence of atm and other data types has extended that philosophy for us," he says.
There are about 15,000 servers in the centralized, corporate data center and approximately 35,000 dedicated Ethernet/Fast Ethernet PC ports at the Redmond, WA, campus. Within the next six to twelve months, Microsoft plans similar network upgrades at several North American, European, Asian, and Pacific sites.
US West is working with Alcatel to provide equipment training and has scheduled an operations and engineering class at Microsoft. The carrier plans to continue working with Microsoft to find ways of interfacing atm traffic to the network.
Trend-setter
Because of size, scale, and technology, Microsoft`s corporate network leads networks of other companies and even carriers, according to US West`s Rabe.
"They are farther ahead of the carriers when it comes to deploying new technology in their network. Usually, when we`re working with Microsoft, we`re working with technology that we`re studying for use within our architecture but have not deployed. Microsoft is always pushing us to take a look at these new technologies. It`s ahead of the industry in using atm, sonet, atm over sonet, and other new technologies," Rabe says.
Alcatel`s biggest customer for the 1648 SM multiplexer is Sprint, followed by other carriers. The 4-fiber blsr capability is a favorite among long-distance carriers, and AT&T, Sprint, MCI, WorldCom, the seven regional Bell operating companies and others use the crossconnects, Ellefson says.
Although the 4-fiber blsrs and the Alcatel crossconnects are more often seen in carrier networks, Ellefson says Microsoft`s use of them in a corporate network could be the start of a trend. "We`ve got more and more corporate folks and big data customers that need bigger and bigger pipes. Microsoft is obviously at the leading edge and needed a very robust and a very large corporate network on its campus," he explains.
Microsoft`s Sunjeev Pandey, lead engineer for the Backbone Group, says it is difficult to predict when the network will need upgrading next. "It`s according to our own internal requirements, when we get to the point where the bandwidth on our internal network is not enough," he says.
Pandey says that Microsoft faces an unusual problem, in that as soon as it has installed the most up-to-date technology, it is ready for something new. "Once next-generation technologies are mature, we`re going to look and say, `Is that the right way to do this?` At that point, we look at what other competing technologies are on the horizon and then do a cost-benefit analysis," he says. "Unfortunately, the technology isn`t available to keep us ahead of the curve. It`s possibly unique to Microsoft." u