New directions in 10-Gbit/sec modules

March 21, 2003
By PETER BRADSHAW, BitBlitz Communications--Several factors have driven the creation of a variety of multi-source agreements. ICs have kept pace with the trend.
Figure 1. Comparison of typical module applications.

Several factors have driven the creation of a variety of multi-source agreements. ICs have kept pace with the trend.

Peter Bradshaw
BitBlitz Communications


Recently, the telecom-connection-oriented SONET and datacom-packet-oriented Ethernet specifications have converged. Several factors have pushed these specifications together. For example, their next bit transfer rates are almost identical, at approximately 10 Gbits/sec. The economic benefits of sharing infrastructure investments and the growth of voice over Internet Protocol (VoIP) within both the enterprise and the long-haul environments also have contributed to this trend.

Data rates of 10 Gbits/sec cause EMI problems, and the poor performance of standard PCB FR-4 materials also pushes the incorporation of the more difficult high-speed portions of the design into module-type components. This explains the proliferation of multi-source agreements (MSAs) to consolidate module specification and production, and of different ICs to operate within these modules and to drive them.

The need for new standard modules

The driving forces behind these MSA-defined modules are:


  • Lower system cost through increased manufacturing volume of standard parts and increased competition through standardization and interchangeability.
  • A common module format for a broad range of different applications, driving both copper cables and different types of optical fiber.
  • More rapid acceptance of newer technology through the establishment of multiple compatible sources.
  • Full compliance with as many standard specifications (IEEE 802.3, SONET, 10-Gigabit Fibre Channel, etc.) as possible.
  • Reduced size compared to the 200-pin and 300-pin MSA devices currently in use.
There are currently four main MSA groups developing 10-Gigabit Ethernet and OC-192 module specifications: XENPAK, XPAK, X2, and XFP. These are illustrated in Figure 1, summarized in Table 1, and described in more detail below. Each has a website with a downloadable spec.

These MSAs share certain specifications. For example, XAUI is the electrical interface, based on the IEEE 802.3ae specification in Clause 47 (http://www.ieee.org). It uses four differential lanes, each operating at 3.125 Gbits/sec with 8b/10b encoded data, and a control interface based on the Management Data (MDIO) standard in Clause 45, and is the basis for the host-to-module electrical interface of XENPAK, XPAK, and X2 modules.

The same IEEE specification defines several fiber-optic PMD standards, including the 850-nm (short reach, or SR), 1310-nm (long reach, LR) and 1550 nm serial (extended reach, ER) and 1310-nm four-channel WWDM (LX4) interfaces. The SC-duplex connector is a common fiber-optic standard. All the MSA groups use these standards.

Naturally, each of the 10-Gbit/sec MSAs has differentiating characteristics as well. These are described below.

XENPAK MSA details

In addition to the XAUI signals, the XENPAK design offers several control signals for module detection, reset, control, and status. It also accommodates 5.0-V and 3.3-V power supplies and an adaptive power supply (APS) that can be set to any voltage between 0.9 and 1.8 V on a per-module basis.

The specification includes conditions for module thermal characterization. An extensive series of MDIO registers allows the host to determine virtually every inherent characteristic and diagnostic status of the module.

The XENPAK specification is fairly mature, and several manufacturers are building modules.

XPAK MSA details

The electrical interface for XPAK is exactly the same as the XENPAK module, and uses the same connector and pinout, except that one pair of clock pins, removed from earlier versions of the XENPAK specification, have been restored. The standard additionally allows 10-Gigabit Fibre Channel rates (3.1875 Gbits/sec per lane) and SFI4-P2 rates, a flexibility no doubt enhanced by the provision of this clock, though they are referenced nowhere else in the document and nowhere in the current XENPAK specification.

Two height "profiles" are allowed, 9.8 mm and 22.25 mm. The second version accommodates a larger heat sink transfer area. Each has its own module holder, which adds 2 mm to the width over most of the space. There is also a mid-board mount version.

The published XPAK specification is still in an early stage. (More than a quarter of the current specification is devoted to such topics as who may belong and how they may be thrown out!).

X2 MSA details

The electrical interface is the same as the XENPAK module, using the same connector. However, there is mention of a reduced number of power pins and fewer MDIO port selection pins. Although the module is described as supporting other data rates than pure XAUI, including 10-Gigabit Fibre Channel and SONET OC-192, no mention is made of any external clock, which would facilitate such a capability.

The three height "personalities" accommodate a larger heat sink transfer area. All use the same rail system, which adds 1 mm to the width over most of the space occupied. The height of the bezel depends on the "personality" of the module and the EMI shielding. The descriptive material allows unlimited height variations, though interchangeability would seem to exclude that except in mid-board mount configurations.

The published X2 MSA specification is still relatively unfinished, though the latest version is filling up a bit. The new version contains some radical changes from the previous version.

XFP MSA details

The XFP electrical interface operates at the full 10-Gbit/sec serial speed, with a single differential pair in each direction, compatible with the IEEE 802.3ae, 10-Gigabit Fibre Channel, and SONET OC-192 specifications. A reference clock is required, at 1/64 of the data rate. Signal processing is expected to be limited to a CDR signal cleanup device in each direction. The control interface is an I2C interface standard, with a few additional control signals for detection, reset, power-down, etc. This interface specification is separately known as XFI, and is expected to become the basis for other similar electrical interface specifications.

The optical interface is not closely defined by the XFP specification, but allows for a wide range of options. It is probable that a standard connector will emerge, most likely the SC-Duplex. The standard also envisages an electrical external interface variation.

The top surface of the module is a heat transfer plate, and a "floating" heat sink, held in place by a retainer clip and effectively a part of the cage, contacts this surface to remove the module's heat. A separate specification for a mid-board mounting version is planned.

The allowed pitch permits 16 XFP modules to fit in a standard 19-inch rack. The module height is fixed, but the height taken on a board will depend on the heat sink used. A compatible double-width variation may be developed.

An extensive series of registers gives access to virtually every inherent characteristic and a thorough diagnostic status within the I²C register space. The system enables many modules to be serviced by a single I²C controller, with some minimal selection logic.

There are four classes of thermal dissipation. All modules must be capable of operating in a power-down mode at <1.5 W, where the host may determine the characteristics and decide whether to allow the module to come to full operation. The optical interface need not function in this mode. The specification includes conditions for thermal characterization.

The published XFP specification is very complete, including jitter, signal levels, eye diagrams, S parameters, impedance levels, the descriptive and diagnostic registers, test methodologies, thermal measurements, full mechanical dimensions, etc. Some XFP modules have recently become available.

Other projects

Other electrical specifications and projects are in the works. For example, the IEEE 10GBASE-CX4 Project has a website at http://www.ieee802.org/3/10GBCX4, and has an active study group developing a specification for a Twinax cable interface for XAUI-type signals over 15 m of InfiniBand Ibx4 cable. Several MSA specifications allow for "copper' variations, and the XAUI-based modules could easily add variations using the projected CX4 electrical specification in place of an LX4 optical interface.

The Optical Internetworking Forum (OIF) is currently exploring a set of serial 5- to 6- and 10- to 11-Gbit/sec specifications for short- and long-range PCB connections in the PLL group. There is also a project called HSBI, which is intended to provide a 6-Gbit/sec interface specification. It is probable that the OIF PLL specification will end up using this and/or the XFI specification as the baseline for their interface specification.

Future trends and developments among the MSAs

The XENPAK specification is fairly complete, and several vendors are beginning to introduce modules. We have examined several of these modules in our test lab for interoperation with our SERDES devices. However, there are signs of unhappiness among customers that the size and cost of these modules are still too great.

The XPAK and X2 MSAs have been hurt by the fact that they are two essentially overlapping groups. It would probably have been wise for the two groups to get together and amalgamate. Although the same ICs could be used in either module, the probably poor acceptance of a bifurcated pseudo-standard will limit the market.

The advantages of the XFP module in terms of size, and probably of cost as well, may well render the contest moot. From a system point of view, all configurations will convert some parallel data format, in one or more SERDES stages, to the ultimate serial data stream transmitted over the fiber or copper cable. Whether these SERDES devices are placed in the modules, on the host board, or incorporated into an ASIC, FPGA, or other switch fabric will depend on the relative economics and technical feasibility of the various configurations.

One "sleeper" advantage of the XFP module is that it pushes the fairly expensive 10-Gbit/sec SERDES out of the module, where it would be subjected to two mark-up cycles (the IC manufacturer's and the module manufacturer's) instead of one. Although the SERDES would have to be provided for unused module slots, this is not likely to offset the extra mark-ups, except in the most egregiously inefficient installed configurations. Further, the availability of multiple 10-Gbit/sec serial SERDES devices in a single package will soon reduce both the cost and power-dissipation penalties of the host-board location.

A number of high-speed, high-performance ICs are becoming available for use in these modules (Figure 2). In particular, SERDES parts now can interface the XGMII signals from MAC and framer or switch devices to the XAUI signals needed by the XENPAK, XPAK and X2 modules, and vice versa. The XAUI signals may be carried up to 1 m (39 inches) on low-cost FR-4 boards, and the transmitter pre-emphasis and receiver equalization can recover, retime, and realign the data even if the incoming eye is completely closed.

Within the modules, new retimer devices can perform similar data recovery, retiming, and realignment, and reset the timing domain, on XAUI signals within LX4 (or CX4) modules, ensuring error-free data transmission. For SR, LR, and ER XENPAK, XPAK and X2 modules, transceivers will convert XAUI signals to and from the 10-Gbits/sec serial signals needed for optical transmission. The transceivers can drive XFP modules to and from XAUI signals or, in combination with SERDES devices, to and from XGMII signals, providing a complete electrical solution in most cases.

Peter Bradshaw is applications manager at BitBlitz Communications. (Milpitas, CA) He has been involved in designing analog and mixed-signal ICs for over 40 years, using both bipolar and CMOS technology, at both large companies and several startups.
Figure 2. High-speed ICs within different modules.