Switching to Ethernet over SONET in the metro

How to evaluate the costs and deployment issues of the different metro Ethernet architectures.

For years, carriers designed their networks around voice and circuit services to satisfy demand from their customers. As Internet usage among business and residential customers grows, however, so does packet-data traffic on metro networks.

Cable providers are experiencing similar trends, as wider usage of cable-modem technology for Internet access increases the packet-data traffic on their metro backbones. Further expansion by cable providers into business customer segments will drive the amount of packet-data traffic on these networks even higher.

In the Asian market-Japan and to a larger extent China and Korea-carriers are poised for rapid network expansion, both in terms of new build-outs and technology that supports new services on existing infrastructure. China is expected to install a cumulative 102 million fiber-km by 2005, according to market researcher KMI (Providence).

Packet-data services using Ethernet transport have vast potential for rapid growth in all of these markets. Since packet-data traffic is more efficiently transported using packet switching than circuit switching, the sheer economics of efficient data transport is forcing carriers to consider packet-switched Ethernet services on their metro networks.

As carriers begin to offer more data services, they can use their existing networks in a more efficient manner or build new infrastructure. With either option, carriers need to fully understand the tradeoffs that come with each of the technology choices.

Designed to solve some of the problems of older metro SONET/SDH solutions, resilient-packet-ring (RPR) technology optimizes the sharing of fiber-optic rings for packet-based data traffic. While SONET/

Despite its promise, RPR remains unproven. Running RPR over dark fiber means building a new network. And while fault management and performance monitoring are well understood in the SONET/SDH world, RPR is still making its way through the standards bodies. The technology supports TDM, according to RPR proponents, but that is only an assertion because no live networks implementing RPR carrying legacy voice and circuit traffic exist today. Therefore, carriers must test the performance and validity of RPR to evaluate how it handles packet-data traffic as well as legacy voice and circuit services.

Gigabit Ethernet over dark fiber
Ethernet is a standard protocol in LANs across the world, but there are significant shortcomings when it comes to deploying Gigabit Ethernet (GbE) natively across MANs, especially over dark fiber. First, while Ethernet ports are inexpensive when compared to other technologies, Ethernet in the MAN is a point-to-point technology-generally, new fiber needs to be lit every time a new node is added to the network. That can prove to be very expensive. Second, restoration time for native Ethernet implementations is on the order of 1-5 sec, which compares very poorly to the 50-msec restoration that SONET/SDH offers. While delivering on the promise of low cost, this solution has limited value in a true carrier-class environment that demands a resilient network with full protection and 99.999% availability.

Ethernet over SONET/SDH (EoS) allows carriers to offer Ethernet services over existing SONET/SDH rings without incurring the expense of lighting new fiber or undergoing the learning curve necessary to implement GbE over dark fiber or RPR. With EoS, packet-based data, legacy voice, and circuit services can coexist on the network.

Switched EoS enables each GbE port to switch traffic to multiple destinations on a metro ring at wire speeds. Carriers are no longer required to hard-wire Ethernet access ports to SONET/SDH circuits on the ring. Commercially available chips map Ethernet into SONET/SDH circuits using standards-based point-to-point protocol over high-level data link control for packet over SONET/SDH (PoS) or standards-based X.86 encapsulation for EoS. These chips enable full packet lookups and media access control or IP layer switching in the line cards in the SONET/SDH add/drop multiplexers (ADMs). By including commercially available switch fabrics in the design, packets arriving from the ring on any SONET/SDH circuit may be switched to any customer port on any line card.

Today, three architectures enable carriers to implement EoS:

• Legacy ADM solutions offering Ethernet multiplexing or mapping over SONET/SDH.
• Legacy ADM solutions offering partial Ethernet switching over SONET/SDH.
• Next-generation ADM solutions offering full Ethernet switching over SONET/SDH.

Multiplexed EoS
Most legacy ADM vendors offer multiplexed EoS solutions, which require each Ethernet customer port to reserve resources on the SONET/SDH ring. Each SONET/

SDH pipe is only capable of carrying traffic to and from a customer port that is assigned to that SONET/SDH pipe. This restriction severely limits the total number of Ethernet customer ports supported by a single OC-48/STM-16 (2.5-Gbit/sec) SONET/SDH ring, because a ring's resources must be reserved at a rate that is one-to-one mapped and equal to the number of Ethernet customer ports on the ring. Even if the customer port is not transmitting or receiving traffic, its resources on the ring are reserved and cannot be used to carry traffic to or from another Ethernet port.

For example, if each Fast Ethernet (FE) customer port has 1-Mbit/sec average traffic with a 50-Mbit/sec variance in traffic, multiplexed EoS forces the carrier to reserve 15 Mbits/sec on a ring for 97% ring availability. Assuming Gaussian distribution for traffic throughput at each customer port, for 97% service availability, the amount of bandwidth assigned on the ring per customer port is:
Mean + 2 x standard deviation 1 + 2 x SQRT(50) = 15 Mbits/sec.

The maximum number of FE customer ports supported on the ring is:
2.488 Gbits/sec/15 Mbits/sec = 165 FE customer ports.

In an eight-node network, that is an average of 20 customers per node.

Partially switched EoS
Some legacy ADM vendors are adding partial switching capabilities at the Ethernet line-card level. In this case, SONET/SDH ring resources are reserved on a per-Ethernet line-card basis, and traffic from any Ethernet customer port on that line card can be switched onto the reserved SONET/ SDH path on the ring.

Assuming 12 Ethernet ports each per line card, running an average of 1 Mbit/sec of traffic each with a variance of 50 Mbits/sec, the partial switching between the 12 ports allows this scenario to be modeled as a single customer with 12 Mbits/sec of average traffic and a traffic variance of 12 x 50 Mbits/sec. The total SONET/ SDH ring resources required to support 12 switched ports is:

12 x 1 Mbit/sec + 2 x SQRT(12 x 50) Mbits/sec = 61 Mbits/sec.

A SONET/SDH ring of 2,488 Mbits/sec can support:
2,488 Mbits/sec/61 Mbits/sec = 41 line cards.

Hence, the total number of customers that may be supported on a SONET/ SDH ring is:
41 x 12 customer ports = 492 customers.

Switched EoS
Next-generation ADMs offer fully switched EoS capabilities. Traffic from any Ethernet customer port on any line card in any ADM slot may be switched onto a given SONET/SDH path. Since ADM chassis have as many as eight to16 slots, with each slot supporting eight to 12 FE ports or two to four GbE ports, the statistical multiplexing gains are enormous.

Switching allows all customer ports to access the reserved SONET/SDH pipe. That is modeled as a single customer with a traffic mean of the number of customers per node x 1 and a bandwidth variance of the number of customers per node x 50.

Assuming eight nodes, to find the total number of customers per node an OC-48/STM-16 SONET/SDH ring can support, we need to solve the equation:

Per node:
(Number_of_customers x 1) + 2 x SQRT(number_of_customers x 50) = 2.488 Gbits/sec/eight nodes

Per node:
Number_of_customers + 2 x SQRT(number_of_customers x 50) = 311 Mbits/sec

Per node:
Number_of_customers = 142.

Hence, the total number of customers that a single OC-48/STM-16 SONET/SDH is capable of supporting in an eight-node network model is:
142 customers x eight nodes = 1,136 customer ports.

Destination-driven cost
Once the network is deployed, it is possible to add customers in the packet world without adding to the transport infrastructure. The infrastructure only needs an upgrade if the ring runs out of capacity. Unfortunately, with certain EoS architectures, more customers require upgrading of the transport infrastructure for connectivity, not capacity, reasons.

In a multiplexed EoS network implementation, a GbE port on a source node behaves like an extension of the SONET/SDH circuit that connects the source to a destination node on the ring. All traffic on that Ethernet port is multiplexed onto that one circuit. For each additional destination node, a separate circuit on the ring connects the source node to the new destination node. And for each new circuit on the ring, an Ethernet port must be reserved on the source node to transmit and receive traffic on that circuit. As the number of destination nodes in the network grows, the additional ports cause an exponential increase in the network's cost and complexity.

Using partially switched EoS solutions compounds the problem. Since each additional line card needs reserved SONET/SDH resources to each destination, port burn expense is replaced by SONET/SDH pipe burn when it comes to scaling the network. Switched EoS enables carriers to scale their networks linearly to the amount of packet-data traffic added to the network as opposed to incurring up-front port costs that increase exponentially with the number of destination nodes in the network (see Figure 1).

Mesh-based metro services
While a backhaul architecture is ideally suited for traffic that leaves the metro entirely-Internet access, for example-it is more complicated than a mesh architecture for certain metro services. These services by their very nature are mesh-based where the traffic pattern is intra-ring, because the source and destination exist within the metro. Transparent LAN service is one example where use of backhaul architecture creates multiple hops for what is a simple point-to-point service. MPLS virtual private networks within the metro face similar tradeoffs if a backhaul architecture is used. Switched EoS is the only architecture that allows carriers and cable providers to offer these services in a scalable and cost-efficient manner.

In a multiplexed EoS implementation, for a network of N nodes, the number of SONET/SDH pipes required for any-to-any mesh connectivity is Nx(N-1)/2. Each customer requires N-1 ADM Ethernet ports to reach N-1 destinations. If there are M customers on average at each node that each require connectivity to the other N-1 nodes in the network, the number of Ethernet ADM ports is Mx(N-1) ports. The number of SONET/SDH circuits required is MxNx(N-1)/2.

That leads to an explosive burn of SONET/SDH ring resources due to the inability of legacy ADMs to switch Ethernet packets onto any SONET/SDH circuit on the ring. This problem is solved to some degree by using a Layer 2 switch as the front-end to the legacy ADM. Using a Layer 2 switch front-end reduces the SONET/SDH ring resources by a factor of M. But it will not solve the destination-based port burn problem.

Implementing partially switched EoS does not fix the ring resource burn problem, either. The number of circuits required for full mesh connectivity is proportional to the number of line cards, not customers. If each line card supports 12 FE ports and there are M customers per node, then the number of SONET/SDH circuits reserved on the ring is (M/12)xNx(N-1)/2. Switched EoS solves the scalability problem without the need for a front-end Ethernet switch. The maximum number of SONET/SDH circuits required for mesh connectivity is limited to Nx(N -1)/2.

Once this full mesh is set up, mesh applications such as transparent LAN service, MPLS, and centralized or even distributed video-on-demand services can be offered across the metro in a simple cost-effective manner. The cost of a switched EoS network is lower than that of a multiplexed EoS solution by a factor of 10 (see Figure 2). That is primarily because a single SONET/SDH ring can support a large number of customers for mesh services using switched EoS, whereas multiple rings are needed to support the same number of customers in multiplexed and partially switched implementations of EoS. Carriers can derive this value regardless of the traffic pattern (backhauled or mesh).

The same level of simplicity and cost advantage cannot be derived by implementing legacy ADMs with Layer 2 switch front-ends. Without the ability to statistically multiplex across SONET/SDH ring resources, profitability and return on investment for metro services are pushed out beyond horizons acceptable in today's market.

P.G. Menon is vice president of marketing at Atoga Systems (Fremont, CA). He can be reached via the company's Website, www.atoga.com.