Multi-protocol congestion management in Ethernet networks: Bringing consistency of functionality to oversubscribed Ethernet
To minimize the degradation of network performance in cases of congestion and to ensure that critical traffic is transmitted, service providers should implement intelligent oversubscription.
By Mathew Steinberg, Ample Communications
In the enterprise network, customer- or client-facing links initially were deployed starting at 10 Mbits/sec. When 100-Mbit Ethernet emerged, the 100-Mbit interface became the network-facing interface with 10 Mbits remaining as the client interface. And when Gigabit Ethernet (GbE) emerged, 10/100 Ethernet became the standard client-facing interface and GbE became the network-facing interface. The same is occurring in the metro network today with the advent of 10 GbE. Triple-speed 10/100/1000 Ethernet is becoming the standard client-facing interface and 10 GbE the network-facing interface.
Service providers are using Ethernet in the metro as both a service interface and a network interface and now are offering 10- and 100-Mbit, 1-, and 10-GbE customer services. Whereas 1- and 10-Gbit ports were network-facing ports, they now can be customer facing or network facing. The interface (10-/100-Mbit and 1- or 10-Gbit) should not be confused with the actual bandwidth consumed; the interface allows a maximum bandwidth to be delivered to the customer. Thus, the actual bandwidth delivered might be far less than the actual interface is capable of delivering. For example, the service provider may offer the customer service over a 100-Mbit Ethernet connection, but the actual bandwidth offering and consumption might only be 10 Mbits.Customer links are typically under utilized
While consumed or delivered bandwidth may be much less than the interface bandwidth, Ethernet connections--especially client-facing connections--on average operate at a fraction of the available bandwidth anyway, thanks to the bursty nature of the data traffic. Typical enterprise utilization rates are shown in Table 1. With the exception of WAN connectivity of large data centers, historical evidence for service provider leased lines and data services also points to WAN connection utilization far less than 100%. For example, the Catalan R&D Network in Spain connects forty research institutions and might best mimic a large data center. In 2004, the Catalan R&D Network reported its peak Internet backbone connection utilization at 60% (http://pma.nlanr.net/Special/cesc1.html).
Table 1: Typical enterprise bandwidth utilization
Benefits of oversubscription
Oversubscription maximizes the number of customers served while minimizing the hardware cost. Just as highways are oversubscribed during rush hour, data networks today are oversubscribed at the network level. Even the PSTN is oversubscribed; for example, you may not get a dial tone on Mother's Day.
Because Ethernet connections are often underutilized, service providers now will implement oversubscription at the line card to lower per-port cost and increase connectivity. Oversubscription lowers cost by sharing common components, such as network processor unit (NPU) complexes, and optimizes their utilization (See Figure 2).
The diagram on the left of Figure 2 illustrates a system designed for 100% bandwidth utilization. For example, 12 x 1-GbE media access controllers (MACs) are used for each gigabit of the NPU's processing capability. When the links are not full--which is most of the time--only a fraction of the resources' capabilities are used, resulting in higher costs. The diagram on the right illustrates a system optimized for actual usage and takes advantage of the "bursty" nature of data traffic using oversubscription. Resources are shared and utilized efficiently. In cases where the link utilization exceeds the system processing bandwidth, traffic first is stored locally and then discarded in cases of extreme congestion when the network processor cannot process the data as fast as it is delivered to the system.
Figure 3 illustrates the system cost savings achieved when implementing 2:1 and 4:1 over-subscription. 2:1 oversubscription typically delivers better than 40% per-port cost reduction, and 4:1 oversubscription typically delivers better than 60% per-port cost reduction.
The need for intelligent oversubscription
However, oversubscription by itself is insufficient. When full system-side bandwidth is consumed, the tail drop method--where the last traffic into the system is the first traffic dropped--is insufficient for traffic management. What happens if the last traffic into the system is voice, it is sitting behind email and web traffic, and the voice traffic gets dropped? The voice quality will degrade significantly or the call may be dropped, which is simply unacceptable from a consumer perspective.
The requirements for programmed drop of traffic at 10/100/1000-Mbit and 10-GbE rates drive the need for systems and devices with enhanced capabilities across all physical interface types. The service provider will need to offer the same type of capabilities--quality of service (QoS), bandwidth guarantees, traffic shaping, etc.--regardless of port speed or whether the port faces the customer or the network.
To minimize the degradation of network performance in cases of congestion and to ensure that critical traffic is transmitted, intelligent oversubscription must be implemented. Intelligent oversubscription uses:
• Ingress frame classification to sort traffic into queues based on assigned class of service;
• Programmable methods, suitable to traffic type, to drop traffic as queues fill; and
• Rate metering to police the traffic as it leaves the NPU.
Since the customer interface can range from 10/100 Ethernet to 1 Gbit and 10 GbE, intelligent oversubscription must be implemented across these interfaces in a uniform fashion.
Classification: Metro Ethernet deployments require classification across multiple protocols, including Ethernet, IPv4, IPv6, and MPLS. First, control traffic must be identified and placed in a strict priority queue so that it may be transmitted to the NPU for processing. Next, classification and queuing must take place in the following order: Time-critical traffic such as voice-over-IP (VoIP) or IP video, then high-priority data traffic with service level agreements (SLAs), followed by best-effort data. Thus, any intelligently over-subscribed device must include a minimum of four priority queues: Strict priority for control traffic, high priority for time-critical traffic, medium priority for high-priority data traffic, and low priority for best-effort traffic.
Programmable traffic drop: Once traffic is queued, a combination of methods is used to drop the traffic to ensure that appropriate service levels are achieved for all traffic classes. Each drop scheme independently must be applied to each queue depending on the traffic type. Control-plane traffic must be preserved to optimize network performance; time-critical traffic, such as voice-over-IP (VoIP) and IP video, must be guaranteed low latency, which can be achieved by time stamping; and other traffic types must be serviced according to their SLAs. For other traffic types, a combination of Random Early Discard (RED) algorithms, such as Weighted RED (WRED), should be used to match the drop profile of a queue to the associated SLA. Next, Modified Deficit Round Robin (MDRR) determines how each of the queues is serviced and ensures that each queue is serviced fairly (as defined by the user). MDRR also ensures that high-priority traffic indeed gets high priority, and lower priority traffic always gets serviced and is not ignored.
Rate metering: Each port should have a traffic policer to ensure that minimum and/or maximum rate policing is observed. Rate metering establishes per-port rate guarantees and bandwidth control. The minimum rate ensures that no port is starved of bandwidth. The maximum rate limiter temporarily can prevent traffic from egressing on a port if the port has met its maximum rate limits.
Full rate and oversubscribed Ethernet MACs are available for 10/100/1000-Mbit and 10-GbE rates, offering intelligent oversubscription across the family of interfaces. Each device includes all the SerDes logic to connect from line optics to a network processor, as well as all the classification, queuing, and rate metering logic for intelligent oversubscription. The devices also feature chip packet buffers with dynamic memory assignment to improve efficiency while lowering system space, power, and cost.
Conclusion
In the metro, Ethernet is used as a service interface in increments of 10/100 Mbits/sec, 1 Gbit/sec, and 10 Gbits/sec. Service providers will offer services at intermediate rates with different services and SLAs. These services are best delivered with intelligent oversubscription, which lowers per-port cost, increases the number of customers served, and enables the enforcement of SLAs. There are products available today that enable service providers to offer the same services regardless of interface rate at 10/100-Mbit, 1-Gbit, or 10-GbE.
Mathew Steinberg is director of business development at Ample Communications (Fremont, CA), where he is responsible for responsible for licensing activities and partnerships. He may be reached via the company's Web site at www.amplecomm.com.