Serial backplane supports gigabit switch throughput

The demands of gigabit data networks threaten to overwhelm the architectures traditionally used for lan/wan switches. Serial architectures offer a solution.

Tom Palkert, Jack Basi, and Ken Prentiss Applied Micro Circuits Corp.

To support maximum bandwidth in today`s gigabit networks, a switched network is often the solution of choice. The need for continually higher switching is, however, creating an increasingly complex challenge for the switch designer. As the distinction between high-speed local-area and wide-area networks (lans/wans) blurs in the enterprise environment, there is also an increasing requirement to switch high-performance Synchronous Optical Network/Synchronous Digital Hierarchy/ Asynchronous Transfer Mode (sonet/ sdh/atm), Fibre Channel, and Gigabit Ethernet data transport streams across the enterprise. One of the major factors driving the need for higher-speed switching of multiple network protocols is the increasing acceptance and lower cost of gigabit optical networks and the use of wavelength-division multiplexing (wdm) to increase the bandwidth of installed fiber-optic connections.

As the complexity and performance requirements of lan/wan switches increase, it becomes difficult to design the switching functions using traditional techniques. Gigabit Ethernet and sonet/sdh/atm switch designers have implemented many varieties of switching mechanisms, but in general, they all use a parallel bus architecture.

When the bandwidth requirements for existing and future switches are calculated, it becomes obvious that there is a need to increase the throughput of the backplane without increasing the complexity of the system. With increasing bandwidth requirements and gigabit speeds, network switches have reached a level of complexity in which the parallel bus width is too wide to be cost-effective and reliable. An alternative serial backplane architecture described here provides a lower-power, more cost-effective solution for high-speed, high-bandwidth network switches. The serial backplane can be used to implement a number of switch backplane architectures.

For the purposes of this article, a high-performance network switch is defined as having a port bandwidth greater than 100 Mbits/sec. This includes Fast Ethernet, Gigabit Ethernet, atm, and Fibre Channel. The serial backplane design discussed here can be used by other network switching equipment, including hubs, bridges, and routers. However, this discussion specifically addresses the switch application.

How much bandwidth is needed?

A good switch design is easily scalable to higher port counts, such as 16 ¥ 16 to 32 ¥ 32, and to higher port bandwidths, such as 155 to 622 Mbits/sec. However, this scalability requires a significant increase in bandwidth. For atm, a 16 ¥ 16 switch requires bandwidth of 2.5 Gbits/sec for 155-Mbit/sec atm, and 10 Gbits/sec for 622-Mbit/sec atm. To scale this switch to 32 ¥ 32 would increase the bandwidth requirements to 5 Gbits/sec for 155-Mbit/sec atm, and to 20 Gbits/sec for 622-Mbit/sec atm. Similarly, a 16 ¥ 16 Fast Ethernet switch requires 1.6 Gbits/sec, while a corresponding 32 ¥ 32 switch requires 3.2 Gbits/sec. A Gigabit Ethernet switch would require the most bandwidth per port. A 16 ¥ 16 switch requires 20 Gbits/sec. Scaling this to 32 ¥ 32 increases the bandwidth requirement to 40 Gbits/sec.

The core of the gigabit network switch design is the switch fabric. The fabric`s architecture determines many of its critical performance parameters, such as maximum throughput, blocking level, scalability, and multicast efficiency.

The two major classes of switch fabric architectures are time- and space-division switching. As these terms indicate, the time-division architectures share common media (physical network of wires or fibers over which data is transported) or memory and divide the access into time slots; the space-division fabrics set up dedicated paths between input and output ports. In addition, switch fabrics can be designed using a centralized switch fabric or a distributed fabric. In the case of the centralized fabric, the backplane will bring all of the data to be switched to a central switch core that routes the data to the appropriate output port. With a distributed switch fabric, the routing decision is done by the input port, the output port, or a combination of the two.

The shared-media distributed switch fabric with a time-division bus is the classic design used in early switches that were based on mainframe computer architectures (see Fig. 1). This is typically implemented with a shared parallel bus. More state-of-the-art switch fabric architectures are shown in Fig. 2. They include the crossbar (2a); fully interconnected, or point-to-multipoint (2b); buffered banyan (2c); batcher banyan (2d), and ring (2e).

By using a nonblocking design, the crossbar switch architecture can support a bandwidth equivalent to the number of ports multiplied by the clock frequency of the crosspoint switch, assuming a serial crosspoint is used. This can greatly increase the throughput of the switch fabric, but it comes at the price of implementing a centralized out-of-band switch fabric control mechanism.

The fully interconnected or point-to-multipoint architecture is a distributed fabric using space-division multiplexing. It has the advantage of eliminating the need for a centralized control function and high throughput, but at the expense of additional wiring complexity. The architecture has a connection from each input to each output port. The number of backplane traces required to fully interconnect the port cards is equal to N2, where N is the number of port cards in the switch.

The banyan switches offer the advantage of an in-band routing capability and require a minimum of backplane routes, but are not fully nonblocking.

The ring architecture is a time-division distributed switch fabric that is implemented in a manner similar to a Fiber Distributed Data Interface or Fibre Channel arbitrated loop network. Each port card inserts and removes data from the ring as needed. The total bandwidth available to each port is equal to the bandwidth of the ring/N, where N is the number of port cards in the switch.

Parallel backplane architectures

A backplane in a network switch serves as the communications port between the port cards and the switch fabric. The traditional parallel implementation of a backplane design used in the shared-media fabric is shown in Fig. 1. In this architecture, the backplane media are shared among all of the port cards, and there must be some method of arbitrating for access to the bus. The bandwidth limitation of the architecture is about 3.2 Gbits/sec with a 64-bit parallel bus operating at a frequency of 50 MHz, which is adequate for a 16 ¥ 16 atm switch with 155-Mbit/sec ports. It will not, however, easily scale to higher port counts or higher bandwidth switches.

For example, a 32 ¥ 32 port atm switch supporting 155 Mbits/sec at each port would require a shared-media backplane bus capable of supporting 5 Gbits/sec. If we assume that the backplane clock is limited to 50 MHz, the bus width required is 100 bits (5 Gbits/sec/50 MHz). As the number of switch ports increases, the designer must either increase the operating frequency of the bus or the bus width. Either of these alternatives requires a redesign of the port card backplane interface.

In addition to the shared bandwidth limitation, parallel buses are also limited by impedance discontinuities, a high connector pin count, and a requirement for high-power bus drivers. Furthermore, hot plugging has been difficult to achieve as parallel backplanes have become more complex.

Serial backplane alternative

To overcome the limitations of the parallel backplane switch designs, it may be possible to employ serial links in place of the wide parallel buses. This reduces the pin count of the port-card connections and the routing complexity on the backplane.

Serial backplanes offer several advantages, including the following:

fewer backplane signals, which

reduces the size of the connectors required and the complexity of the backplane routing.

a single differential bus driver, which

eliminates the switching noise that is generated by large-pin-count, high-frequency parallel buses. This switching noise can cause electromagnetic interference failures, power supply noise, and crosstalk.

a hot plug, which allows port cards to

be replaced without affecting the operation of the switch fabric. When a port card is removed from a switch using a parallel shared bus, it will generate glitches on the bus due to the induced impedance change, resulting in the potential loss of data.

Like many good proposals, the use of serial backplanes comes with trade-offs. For example, it adds a small number of traces with "very good" impedance control versus a large number of parallel traces with "good" impedance control.

A serial backplane link operating above a frequency of 1 GHz requires tight impedance control of the interconnect. This means that high-frequency controlled-impedance connectors must be used, and the backplane trace impedance must be tightly controlled. Vertical-cavity surface-emitting laser arrays or CD lasers may also be used to eliminate the high-frequency connector and copper trace bandwidth limitations.

A second trade-off is the good clock skew for parallel links versus clock recovery for serial links. Because of the serial link`s high frequency, it is usually not feasible to transmit both the clock and the data. This requires the use of phase-locked loops (plls) at the transmit and receive interfaces. The transmit pll generates a serial clock using the parallel system clock as a reference, and the receive pll recovers the clock from the serial data stream.

A third trade-off is the non-return-to-zero data over parallel links versus encoded data over serial links. The serial link will probably be AC-coupled to facilitate hot-plug requirements. This requires the use of an encoding scheme to maintain a direct current balance in the data link. The encoding scheme also provides the user with a guarantee of transitions in the data stream to help the receive pll recover the serial clock from the data stream.

Recent technological advances in low-power bipolar circuit design have enabled high-speed serial links to become competitive with parallel links in terms of speed and power. As a result, it is now possible to implement the entire parallel-to-serial and serial-to-parallel interface, including the pll circuits, in one low-power integrated circuit. This means that a serial backplane link will dissipate less power and use less area than an equivalent parallel interface.

Easier serial implementation

The elements required for the infrastructure of a serial switching matrix are now available as standard products to simplify the backplane interconnection and switch fabric designs of a high-speed switch. The scalable backplane architecture described here is a low-cost, low-power, completely serial solution that eliminates the need for time-consuming development of proprietary backplane solutions.

As an example, a serial backplane architecture for a switch supporting a 32-Gbit/sec nonblocking fabric is shown in Fig. 3, using a crosspoint switch and high-speed transceivers to support 100/1000-Mbit/sec Ethernet, Fibre Channel, and atm ports. The architecture shown uses a 32 ¥ 32 nonblocking crosspoint switch capable of broadcast, multicast, or unicast capabilities. The example shown requires significantly less power than a comparable parallel backplane. To keep up with the high-bandwidth data stream, the serial backplane should be operated at 1 to 1.5 Gbits/sec, with matched throughput rates for all transceivers and the crosspoint switch.

Smaller differential crosspoint switches are also available from multiple industry sources, and larger switches can be created by using multiple devices. For example, a 64 ¥ 64 switch can be created using four 32 ¥ 32 devices. To complete the backplane interconnect solution, high-speed transceivers are available from a wide variety of sources.

Choice of serial backplane architectures

The point-to-multipoint and ring switch fabrics described previously can be implemented readily using serial links that directly replace their parallel counterparts (see Fig. 4).

The point-to-multipoint architecture requires multiple deserializers on each port card. However, it does not require a crosspoint switch, which eliminates the need for a central crosspoint switch control function. However, this architecture is not as easily scaled to higher port counts because the port cards must be redesigned for additional serializers each time the switch bandwidth or port count is increased.

The crosspoint architecture employs a nonblocking-type crosspoint switch in addition to serializer/deserializer functions. The number of ports supported by the switch can easily be increased simply by increasing the size of the crosspoint switch (e.g., from 16 ¥ 16 to 32 ¥ 32). The port bandwidth is also easily upgraded by increasing the serializer/deserializer and crosspoint operating frequency (e.g., from 622 Mbits/sec to 1.25 Gbits/sec to 2.4 Gbits/sec).

In summary, high-performance network switches require a high-bandwidth backplane. In these high-bandwidth applications, serial backplanes offer significant advantages over traditional parallel backplane architectures, including greater overall usable bandwidth, lower power, lower cost, and implementation of hot-swap capability. u

Tom Palkert is network products system engineering manager, Jack Basi is director of marketing for data communications and computer products, and Ken Prentiss is product marketing manager of sonet/atm products, for Applied Micro Circuits Corp. in San Diego, CA.