Ethernet over SONET delivers bandwidth efficiency to the IP world

Ethernet over SONET delivers bandwidth efficiency to the IP world

BOGDAN JAKOBIK and PIERRE-YVES PAU, Nortel Networks

Ethernet over SONET re-engineers the transport infrastructure into a packet-switched optical WAN and emerges as an attractive alternative to packet over SONET and ATM.

Imagine you`re the boss of a major corporation. In recent years, your company has invested millions of dollars in new technology to deliver a better service to your customers, and your investment is protected by the fact that you`re the only player in the market. Then one day you awake to a world that has changed. Not only are you no longer the only player in the market, but your customers want a service that is completely different from the one your equipment was designed to provide.

That`s exactly the situation the leaders of the world`s public telecommunications networks find themselves in today. They have awakened to a world where deregulation has removed the protected position of their investments in optical technology, and where the Internet has created a demand for a service alien to what that technology was designed to support. It`s a free-market Internet protocol (IP) world out there, and service providers are desperately trying to find their place in this new competitive and technological landscape.

These service providers have several options for solving this dilemma. They could, for instance, forklift a solution by building an IP network from scratch at great expense, but it wouldn`t match the reliability that existing Synchronous Optical Network (SONET) architectures provide. The clean-sheet IP network is still nowhere near carrier-grade. In fact, leveraging the existing base of SONET technology is the only sensible way out of this squeeze play. Of the different methods for using SONET-based architectures, Ethernet over SONET has emerged as a strong step toward the unification of voice circuits, Asynchronous Transfer Mode (ATM) cells, IP packets, and wavelengths over a single network.

Leveraging optical technology

While the telecommunications world in general may have changed, one thing remains the same: The business is still about bandwidth--and future predictions notwithstanding, bandwidth costs. That, in fact, is what led the architects of the public network to invest in SONET in the first place. The deployment of SONET and its international cousin, Synchronous Digital Hierarchy (SDH), was designed to improve bandwidth efficiency in the optical backbone for voice and time-division multiplexing (TDM) traffic. As for data, SONET got along very well with another new technology standard, ATM. Eventually, the sonet/atm enterprise grew to become an incredibly reliable and efficient network paradigm.

That is, until the arrival of the Internet. Now, you have a bursty, connectionless, and ubiquitous networking technology based on routers running throughout a public network designed for well-ordered little bits that run elegantly through circuit-switched connections. Somehow, this rowdy IP traffic, which is morphing into the biggest thing to hit telecommunications since, well, the beginning of telecommunications, has to be accommodated. That means tolerating costly bandwidth inefficiencies in the optical network. So now, our network leaders not only have to worry about the new free-market competitors putting price pressures on their business, they must deal with the escalating cost of doing business in an IP world.

However, there is a tremendous amount of business opportunity in the IP world that would offset the increasing cost of doing business--provided it is done right. Local area networks (LANs) are bursting at the seams. They are creating a shortage of leased-line capacity in the wide area networks (WANs), and the Internet is luring LAN users with promises of lower-cost virtual networks.

Leveraging the existing SONET infrastructure represents the common sense approach to addressing this market environment. There are basically three options: IP over ATM over SONET, IP over SONET, and Ethernet over SONET.

ATM was the first option public-network providers explored to meet the demands of IP traffic, since many already had investments in this technology. With IP/ATM over SONET, also known as multiprotocol-over-ATM (MPoA), the ATM switches aggregate traffic from the routers and interfaces with the transport system via electrical or optical interfaces (DS-1 to OC-12 and beyond).

But while cell-based ATM offers sophisticated protection schemes and bit-rate flexibility, it doesn`t mesh well with the packet-oriented IP traffic. Depending on the packet size, the ATM cell tax can be a big bandwidth drain, anywhere between 15% and 50% of the payload, which can become very costly over a long-haul network. Also, ATM port costs are expensive. Finally, ATM is not very scalable: Since every link to the router has to be engineered with private virtual circuits (PVCs), growth scenarios are exponentially complex, as well as costly. And the mapping of ATM`s packet-based traffic onto synchronous TDM pipes is inherently inefficient.

Then there is IP over SONET, also known as packet over SONET (PoS), which is the serial transmission of data over SONET frames through the use of point-to-point protocol. This data mapping of a router network is done at OC-3/STM-1 (155-Mbit/sec), OC-12/STM-4 (622-Mbit/sec), and OC-48/STM-16 (2.5-Gbit/sec) line rates.

And that`s part of the problem, at least in terms of bandwidth. While PoS has some advantages, it suffers a drawback that it shares with ATM: It is still based on a "leased-line" view of the transport facilities that requires the bandwidth to be predetermined. The traffic enters the optical network in the form of optical pipes at fixed granularity, an implementation inherited from the legacy TDM services. Because mapping of the packet flows onto optical bandwidth is not managed on the optical network element, network engineers are restricted to OC-3, OC-12, or OC-48 pipes when provisioning the links between routers. Further aggravating the problem is the necessity to over-provision bandwidth in a router network for an acceptable rate of performance under failure conditions (Layer 3 re-routing), in many cases by about 50%. All these technological limitations result in vast amounts of under-utilized optical capacity.

PoS has physical mesh connectivity with point-to-point connections. It is ideal for high-fill, low-mesh networks where there is a high mutual community of interest. The service provider doesn`t have any problems filling bandwidth where there is a high mutual community of interest (on the contrary, in fact); the provider`s problem is bandwidth efficiency with a distributed community of interest, and here PoS doesn`t offer much help.

Both the MPoA and PoS concepts can be traced back to leased-line services, which historically have had a poor track record for efficient bandwidth utilization. Many network studies have shown that the utilization rate of leased lines is more than 50% on average, with a fill factor often close to 20%. At the same time, relatively few fiber routes are connecting large communities of interest across the continent and will inevitably face exhaust.

Vendors from both the telecommunications and data worlds are proposing solutions. On the telecommunications side, multiservice optical network elements can be deployed at the network edge, integrating SONET, ATM, IP, as well as STM and optical bandwidth management into a lower-cost single platform. Data vendors, meanwhile, propose super routers, capable of routing at terabit speeds, deployed as traffic aggregation hubs to ensure maximum use of optical bandwidth. One problem with this latter solution is these super hubs could locally result in accelerated congestion of the optical transport plant and create single points of failure with the potential for a catastrophic network outage--the so-called Smoking Hole scenario. This possibility leads carriers to deploy fully redundant router infrastructures in the network points of presence (PoPs), further inflating the costs of providing IP service to ISPs, campuses, and enterprises.

Still, the main issue remains: How do you maximize IP traffic utilization of optical bandwidth across all transmission rates? One of the most promising approaches involves connecting IP routers via a Layer 2 switching cloud directly over the optical network (see Fig. 1). This approach is implemented by integrating Ethernet interfaces and capabilities directly onto optical network elements. Instead of multiple router and transport interfaces as you would have with PoS, there would be simply one type of interface: Ethernet.

Ethernet grows up

The pace of LAN technology evolution is nothing short of phenomenal. It is estimated that the churn rate of LAN-based equipment in 1999 will be about 60%. One reason is the deployment of IP-based virtual private network (VPN) services. These virtual networks operate in a connectionless environment where private networking is conducted with security and trust. The VPN is a promising alternative to traditional remote-access applications. In most large organizations, remote users dial in via modems directly to the company`s remote-access concentrator, which connects them to corporate data networks and systems. VPNs can be less expensive, since users can make local modem connections to the Internet, then establish a secure tunnel into the corporate network.

The VPN market is seeking carrier-grade solutions and is willing to pay for it. Infonetics Research estimates the VPN market was $250 million in 1997; the forecast for VPN is around $12 billion by 2001. In anticipation of the rise of VPNs, service providers are turning to Gigabit Ethernet.

Ethernet is the world`s most popular LAN technology, and Gigabit Ethernet--the development of which was driven by end-user demand--has taken off as a key component of the campus backbone. Gigabit Ethernet is being deployed on a wide scale to overcome the network bottlenecks created by growing numbers of users, richer content, and the centralization of servers.

This acceptance of Gigabit Ethernet is having an impact in the service-provider world as well. The Internet service provider (ISP) central office is starting to look more like a large campus backbone than a telephone company architecture. Using Gigabit Ethernet switches to handle server farms is evolving as a standard architecture throughout the industry. One by one, ISPs are announcing that they are standardizing on Gigabit Ethernet switches at the core.

Ethernet over SONET

Unlike PoS, Ethernet over SONET (EoS) is more intuitively in line with the evolution of connectionless network architecture. EoS, with inherently full-mesh connectivity, delivers connectionless networking ideal for distributed communities of interest, resulting in tremendous value in terms of optical bandwidth utilization for the service provider. Depending on the reach, bit rate, and bandwidth management requirements, Ethernet, Fast Ethernet, or Gigabit Ethernet can be mapped onto a SONET envelope or directly over a wavelength using a thin SONET overhead, ensuring the most effective scheme is used. For example, direct mapping over wavelengths can be more effective for short-haul, high-rate coarse-granularity traffic (at 2.5 or 10 Gbits/sec). This latter option is also called Ethernet over wavelength (EoW), and has the potential to deliver even higher bandwidth efficiency in dense wavelength-division multiplexing (DWDM) optical plants.

EoS implements a packet-switched optical network by combining the flexibility and resource optimization of a Layer 2 switched cloud interconnection with the capacity, bandwidth efficiency, and low protocol overhead of today`s optical networks. This capability in effect turns the whole optical backbone into a distributed learning bridge. When mapping Gigabit Ethernet on SONET, for instance, increments of STS-3c can be allocated to support different customers` bandwidth requirements--a much finer degree of granularity as compared to PoS. Labeling enables packets to be aggregated toward common destinations and avoid terminations at intermediate sites, which improves the end-to-end delay characteristic.

With PoS, traffic is point-to-point and split between multiple links. With EoS, traffic is carried on shared network resources. Optical bandwidth can be statistically shared among several IP streams. Combined with port consolidation (a single Gigabit Ethernet port can actually act like a channelized OC-24c interface), this bandwidth optimization feature can enable support for meshed configurations that otherwise would not be cost-effective.

EoS also enables carriers to provide differentiated services at the network edge by leveraging advanced SONET capabilities such as access to the protection bandwidth to create unprotected tributaries. By combining this feature with load-sharing configurations on the router side, any combination of protected versus unprotected bandwidth can be offered to route IP traffic according to the required grade of service, optimizing the revenue stream from the network infrastructure.

Economics of EoS

Let`s examine the comparative economics of the three IP solutions--ATM, PoS, and EoS--with data from an actual study of DS-3 traffic. The study took place in a region with more than 80 offices, including one major metropolitan area. Network costs were computed using representative industry pricing for Gigabit Ethernet interfaces. Finally, bandwidth utilization statistics were derived as the ratio of used versus provisioned STS time slots. Two types of configurations were considered: hub and mesh.

In the hub configuration, all traffic is routed via a central (tandem) node, which is logically a star topology (point-to-point links) mapped on a physical ring topology. The cost of network implementation is plotted as a function of the per-node traffic. Hubs are typical of metro and regional points-of-presence (MpoP and RpoP), where data traffic (mainly Internet) is aggregated and sent off to broadband PoPs.

The results show an approximate 25% cost savings primarily due to two things: port consolidation and the lower price of EoS and Ethernet router interfaces (see Fig. 2). Since this figure is typical for a point-to-multipoint implementation, cost savings would have been even more significant using a shared-ring implementation. Still, scalable bandwidth provisioning leads to optimized use of the optical network even in a hub configuration.

In many data networks, however, mesh or partial mesh configurations are required. Such configurations are not cost-effective when traffic is provisioned on point-to-point links such as leased lines. In this case, the shared ring architecture is recommended. Figure 3 illustrates a cost analysis based only on a comparison of EoS and leased lines, regardless of the networking technology used. The interconnected routers see a logical mesh configuration, supported via ring multiplexing capabilities, allowing optimal bandwidth provisioning and a significant reduction in bandwidth management complexity. Compared to the hub configuration, the cost savings are considerably greater.

Key to bandwidth utilization

A key aspect to the EoS cost comparison is bandwidth utilization over the WAN portion of the network. Here, SONET STS-3c pipes can be scaled up to STS-24c, more than enough to carry a 1-Gbit/sec traffic stream, while the unused portion of the WAN optical bandwidth can be used for other revenue-generating traffic (see Fig. 4).

We assumed network capacity varying from 8 to about 30 Gbits/sec, which is typical of PoP-to-PoP bandwidth requirements for networks of this size. The first scenario assumes leased OC-12 lines to join MpoPs to RpoPs and RpoPs to the mega-PoP. Because most links require substantially less than OC-12 capacity, the bandwidth utilization is significantly affected (approximately 50%). The second scenario takes advantage of the flexible bandwidth allocation of EoS, down to a minimum dedicated bandwidth of OC-3. This finer granularity resulted in much higher bandwidth utilization (approximately 80%).

The analysis was repeated for two additional cases. First, we grew the overall traffic from 21 to 30 Gbits/sec. Bandwidth utilization increases to 57% with dedicated OC-12 lines versus 85% with EoS, a difference of 28%. Then, we decreased overall traffic from 21 to 8 Gbits/sec, yielding utilization figures of 30% and 63%, respectively, for OC-12 lines and EoS. Note that these computed savings are based only on bandwidth scalability at the EoS interface; they do not take into account the potential savings due to bandwidth sharing within the EoS ring and are therefore on the conservative side.

Toward unified networks

The concept of a Layer 2 switched cloud is gaining momentum as a new IP transport paradigm. Carriers and equipment providers are striving to combine the capacity and robustness of the optical network infrastructure with the ubiquity and resource management flexibility of Ethernet interfaces. The solution is becoming commercially available in the form of Ethernet interfaces (from 10Base-T to Gigabit Ethernet, and to 10-Gigabit Ethernet in the future) integrated on optical network elements.

From the protocol perspective, the concept is currently implemented by mapping Ethernet packets on a SONET payload (EoS); however, products using direct mapping on wavelength via thin-SONET (EoW) are also anticipated, which will allow service providers to deploy the most cost-effective solution for a given application.

EoS is a good example of how the capabilities of optical network elements can be evolved to integrate TDM, ATM, and IP traffic over the same optical links, thus providing a major competitive edge for carriers and service providers. The capabilities of optical network elements will continue to evolve to deliver scalable multiservice platforms that can optimize bandwidth utilization and management at the STM and optical levels, offering Ethernet, sonet/sdh, and OC-48c/OC-192c optical interfaces on the same platform (see Fig. 5).

Some carriers already see that the technology can extend the life of their optical plant by several years, moving the telecommunications paradigm one step closer to unified networks carrying voice circuits, cells, and packets over a seamlessly managed optical infrastructure. q

Bogdan Jakobik is senior analyst of optical network planning and Pierre-Yves Pau is brand manager, high-capacity optical networks, at Nortel Networks (Brampton, ON, Canada)

Acknowledgements:

Fig. 5. In the future, technology advances promise the ability to handle multiple traffic types on the same platform in an optical network.