A "route once, switch many times" architecture makes efficient use of routing ports to put packets on wavelengths in an optically switched core.
Richard Norman
Hyperchip Inc.
To allow for the dramatic growth of bandwidth in wide area networks (WANs), today's SONET rings will have to be replaced by an all-optical mesh WAN core. While much attention has been given to the wavelength switches that will direct the 80% of traffic that passes through each node, the impact that this optical switching will have on the packet routers that feed this optically meshed core has been largely ignored.
Today's routers have port counts a few times greater than the number of fiber serviced in a bundle. This is presently sufficient since each fiber bundle can be treated as a single destination. But to efficiently use lambda switching, each incoming packet must be placed on a wavelength that can take it as close as possible to its final destination to eliminate intermediate routing. This works efficiently when each routing node supports a number of ports comparable to the number of wavelengths the node services.
Hardware vendor Cisco Systems already predicts that a large central office will need to handle 10 petabits of traffic by 2004. So how big a router will be needed? If the through-traffic is 80%, and that traffic is optically switched, the packet-routing demand is the remaining 2 petabits/sec per node. At OC-768 (40 Gbits/sec), this requires at least 50,000 ports.While such a routing node can be built as a network of smaller routers, the routing efficiencies gained from lambda switching are quickly lost to the penalty tax of intra-node connections between the routers, reducing user ports and increasing equipment costs. The solution is to build individual routers with port counts comparable to the total number of wavelengths being routed in each node, allowing every routing port to be used for revenue-generating traffic. The network planning challenge then lies in scaling an IP backbone from terabits to petabits-not from gigabits to terabits.
There is general agreement that WAN bandwidth is growing rapidly, but will this really continue to petabit levels, or will some bottleneck limit growth? Clearly fiber can scale; Nortel Networks will soon commercialize equipment that can pack 6.4 Tbits down each fiber, or roughly one petabit per fiber bundle, and Qwest Communications is laying conduit that can accommodate up to 100 bundles. Server farms can scale as easily as the WAN, and network unification will further simplify this through end-to-end multigigabit serial links. The challenge of the last mile is also being attacked on many fronts.
Will demand continue to grow? Telecommunications market researcher RHK forecasts Internet backbone traffic will increase by at least 300% per year until 2003, and UUNet, the world's largest Internet service provider, expects traffic on its regional trunks to hit the 1-10 petabit range over the next four to five years.
Two current trends are driving the need for a petabit scalable solution. First is the rollout of broadband multimedia services. With affordable T1 levels (1.544 Mbits/sec) of service to the home, broadband access to the Internet is becoming a reality. Access speeds will continue to accelerate, and 50-Mbit/sec DSL and Gigabit Ethernet over dark fiber to the enterprise are on the horizon. Even fiber to the home, with its multigigabit potential, is already undergoing trials in 26 U.S. cities. In turn, deployment of these higher bandwidth pipes will lead to the growth of multimedia applications over the Internet, resulting in a surge of traffic.
Cost is the second petabit driver. Optical transmission technologies such as DWDM, ultra-long-haul lasers, and optical add/drop multiplexers are lowering the cost of transporting bits across the world. This dramatic decline in transport costs is forcing network planners to reconsider basic assumptions about how to build networks.
Router vendors have made incremental changes over the years to try to get ahead of Internet demand, which is growing much faster than silicon technology improvements. These changes focused on combining routers in overlay networks with established cell- and frame-based switching technologies (frame relay, ATM, SONET), implementing application-specific integrated circuits (ASICs) for port-speed forwarding, and clustering routers to reach higher port counts. Routers today, however, increasingly end up in the core of massive service provider networks, and they are expected to meet demands for up to 1,000% annual performance boosts and offer quality-of-service (QoS) and carrier-class features.
Despite these changes, many carriers view current routers as lacking in scalability and reliability. The problem lies in the router architecture, which is an outgrowth of routing's original enterprise focus. The growth of optics has far outstripped the ability of silicon-based electronics to keep up, so routers cannot accommodate the orders of magnitude increases in capacity that optics provide.This inability to scale translates into a very short service life. In a recent study, RHK reports, "The life span of an Internet backbone router is less than one year, after which it is moved to the edge, again for less than a year. It is then decommissioned."
An IP network based on current routing technology thus becomes too complex and expensive to scale to petabit levels, and optical-packet routing networks are still deep in the lab. What is close at hand is a packet-flow-based IP network with carrier-grade routers feeding optical wavelength switches in the core (see Figure 1). This will provide the quality attributes of today's circuit-switched networks in a route-once architecture that makes efficient use of routing ports to put packets on wavelengths in an optically switched core.
To allow for dramatic bandwidth growth in the WAN, the network core is actually evolving into two cores: the IP core that terminates the transmission side and turns photons into packets with priorities and individual destinations, and an optically switched core that manages transmission bandwidth and provides the pipes to transport traffic (see Figure 2).
To take full advantage of optical switching, each incoming packet must be placed on a wavelength that can take it as close as possible to its final destination to eliminate intermediate routing. Once the packets are no longer re-routed at each node, each wavelength becomes a unique destination, and a node must route from any wavelength on any fiber to any other wavelength, rather than just from any bundle to any bundle.
This route-once, switch-many-times architecture works efficiently when the ports of each routing node are comparable to the total number of wavelengths the node services.
While a large routing node can still be built from a cluster of smaller routers, this is inefficient because the routers must be massively interconnected. For sufficiently large nodes, the routing efficiencies gained from lambda switching are lost to the penalty tax of intra-node connections between the routers.
Massive port-count routers eliminate this routing-port inefficiency by transferring the physical scalability requirements to the router's internal switch fabric, allowing all of the expensive routing ports to be used externally for revenue generation. This allows petabit-scalable routing nodes to combine the best features of both packet routers and optical switches.
Optics are great at the lambda level, but until the world is rich enough in fiber to provide a lambda for each user, routing technology will be needed to pack packets onto lambdas for the transport core to manage. But since the advances in optics are outpacing Moore's Law, the electronics-per-bit must be greatly reduced to allow electronic scalability to keep up with optics. To reach Cisco's 2004 predictions, bandwidth will have to grow 2,500-fold, and at the 1.5-to-1 price elasticity tracked by Bell Labs, this will require a 180-fold drop in the price per bit.
Even the movement to optical switching and route-once networks is insufficient, providing a mere five-fold savings on top of the eight-fold savings of four years of Moore's Law. Fortunately, routing software can bring about simplifications to the electronics as well.
One of the early promises for greater simplicity and reliability in a scalable IP network is the accelerating adoption of Multiprotocol Label Switching (MPLS) and the associated technologies that support traffic engineering, class of service, and virtual private networks (VPNs). MPLS-based VPNs can scale to thousands of VPNs with thousands of sites. By building IP-based VPNs, a network operator can eliminate the need to deploy multiple overlay networks and reduce its investment in traditional switches. In effect, MPLS provides a scalable connection-oriented path through IP.
From an electronics point of view, the most important feature of MPLS is that complex routing is done once for each flow, rather than once for each packet. Simple direct lookups are used on subsequent packets. With even the shortest micro-flows averaging 12 to 15 packets, this produces an additional four-fold savings-the amount required to bring the total electronic savings to the level needed for growth to petabit levels.
The result is routing platforms with non-blocking switch fabrics scalable to tens of thousands of high-speed ports. These high-performance switch fabrics, combined with flow-based MPLS traffic engineering, offer a highly scalable core-routing solution.
With hardware-based switching and packet-forwarding engines, a highly modular architecture and advanced engineering features, petabit routers can achieve increases in capacity without requiring continuous changes to the logical network topology or replacement of operational hardware. Moreover, this can be achieved with superior cost/performance for the router's price, footprint, and power consumption.
At this point the router truly evolves from an overgrown enterprise network box to a key part of the central office system. The net result is that service providers can grow their networks to the required capacity when needed in a future free from forklift upgrades.
Richard Norman founded Hyperchip (Montreal) in 1997 and is president and chief technology officer.