The quick fix: Automating test-equipment deployment
Optical networks and DWDM magnify the importance of fully photonic network test access.
Net-HOPPER Systems Inc.
Pressure is building within the operations departments of network pro-viders. A growing customer demand for service level agreements (SLAs) amid other competitive pressures has resulted in zero tolerance for network downtime. Networks, however, are growing at a much faster rate than maintenance budgets. The result is often longer workdays for operations employees and increased network downtime.
On the technology side, the juggernaut toward optical networks and dense wavelength-division multiplexing (DWDM) has exacerbated some existing barriers to rapid and effective troubleshooting, making problems more difficult to trace through the network. Similarly, by offering sophisticated services such as virtual private networks (VPNs), voice over Internet Protocol, and other applications atop increasingly dynamic network architectures (switched virtual circuits, Multiprotocol Label Switching-MPLS, traffic engineering), network operations require specialized diagnostic tools well beyond those used just a few years ago (see Figure 1).
To keep networks up and running smoothly, network providers must look to recent innovations in network maintenance and management. Network test-access systems (NTASs) are diagnostic tool-management systems that accelerate the identification of network problems by enabling rapid remote deployment of test equipment. Remote test-equipment deployment involves the diagnostic capabilities of network equipment, the physical reconfiguration of network matrix switches, and coordinated control by centralized NTAS software.
At the highest level, an NTAS solution serves as network diagnostic "middleware" by automating the deployment of test equipment (analyzers, probes, testers) across widely deployed networks. Many network difficulties can be diagnosed and corrected quickly once the right tool is attached to the problem area. By deploying the right tool on the right network segment in seconds, NTAS can typically save hours or days of downtime. Also, by mechanically implementing all the cabling changes required for test-equipment movement, an NTAS eliminates the mistakes often associated with operators patching and unpatching cables as they insert and remove test equipment from the network. The results are improved diagnostics without introducing production network problems.
For example, it recently took 10 engineers at a large Internet service provider (ISP) approximately six-and-a-half hours to diagnose a critical network problem-65 man-hours of running around because of the heavy use of portable test equipment. According to the ISP's own estimate, an NTAS solution would have yielded 75% less downtime and required only a fraction of the manpower.
Network equipment always has some remotely accessible diagnostic functions. In many cases, these functions include traffic statistics and characteristics that can be queried and saved in a database for historical analysis. More critical to network diagnostics, however, is the ability of some network elements to provide test copies of traffic for complete, specific analysis by specialized, external tools. Often called mirroring or snooping, this access to network data enables the right external tool to aid tremendously in the examination of rather thorny network problems. With today's core network elements, especially optical-network elements, mirrored access is extremely rare; therefore, the physical-layer movement enabled by an NTAS's matrix switches is critical because it compensates for these limitations.
In general, matrix switching of copper circuits is an established technology used by the telecommunications industry for decades. Much more novel, however, are optical matrix switches, available for only a few years, with capabilities that have grown substantially during that time. Across the copper/fiber spectrum, matrix switches share a common purpose, which is to enable remote changes to the physical-layer interconnections of attached devices, eliminating manual on-site involvement. An NTAS makes use of matrix switches to rapidly alter the positioning of test equipment within a network. Test equipment and dozens, or hundreds, of network segments may be attached to the same matrix switches (see Figure 2). Once configured in this way, the test equipment can immediately be connected remotely to any segment using control software.
While the system described has the physical abilities for test-equipment movement, centralized control software is needed to coordinate all of the NTAS's efforts. Network engineers use NTAS software clients, which access the NTAS servers, to request that test equipment be placed within the network. These servers, in turn, implement and coordinate the test-equipment moving requests from the engineers. Typically, the NTAS servers are in constant communication with the matrix switches and network elements via simple network-management protocol (SNMP), transaction language 1 (TL/1), or telnet.
In the end, centralized NTAS software offers three-way protection. First, network engineers are relieved of the complexities surrounding network-element mirroring and matrix switch management, as well as the pain of physically lugging test equipment around a network. Second, both the production and test-access infrastructures are protected from unmoderated, uncoordinated provisioning commands that might otherwise come directly from network engineers. Finally, by attaining the quickest diagnosis possible, the production network's uptime is maximized.
The trend toward all-optical networks magnifies several existing difficulties of manual test-tool deployment and introduces new barriers. First, and most obvious, the higher network capacities rule out temporary network disconnections during diagnosis for all except the most catastrophic problems. For example, only a complete circuit failure can justify inserting a splitter or test device into an optical link because the insertion itself could affect thousands of customers.
The ability of optical networks to simultaneously transport a wide variety of services is becoming a boon to network efficiency. Unfortunately, this variety makes deployment of test equipment without an NTAS impractical; the range of services means that a variety of test tools need to be predeployed throughout the network. While single-service networks already find such preprovisioning unmanageable, all multiservice networks require dynamic, on-demand tester deployment.
With network providers' desire for ad hoc bandwidth scalability and multiservice transport efficiency, all-optical networks are increasingly deployed without a Synchronous Optical Network (SONET) core; instead routing, switching, or transparent technologies are utilized. Unfortunately, with all these alternatives, providers lose SONET's interoperable supervisory and management services. In the short-term, optical networks only offer basic signal-loss alarms without significant analysis capabilities. More specific problem detection is likely limited to the edges where the channel becomes electrical. Without highly informative telemetry from the network elements, mechanized movement of dedicated test equipment across the network is required for rapid fault isolation, in addition to diagnosis.
The use of DWDM renders test equipment physically incompatible or, more precisely, Layer 2 incompatible, with the network links. Historically, once an analyzer had physical access to a network segment, it could immediately be utilized for diagnosis of that segment. Now, with DWDM, physical access yields an overwhelming rainbow of noise hitting the tester. Only with additional optics can the desired lambda be isolated for analysis. Adding such lambda selection and isolation to NTAS is natural and efficient: Moving analyzers between wavelength channels is analogous to moving them between fiber circuits. Additionally, leveraging the optics across multiple fiber circuits and analyzers minimizes the cost and complexity of the required optics.
Network engineers should consider several factors when designing an NTAS solution, including the network's technologies and topology, the types of test equipment desired, and the required number of such testers. The most important of these factors is the network's topology because the movement of test equipment enabled by NTAS is limited to network segments within cabling range, typically within the same building or campus. Therefore, NTAS deployments should service locations that house significant amounts of network topology. At the extreme, for instance, NTAS should not be used in remote locations where only one fiber-optic channel exists-the tester could just as easily be deployed directly on that channel. Instead, the NTAS infrastructure is better located at a network hub where a number of sites' optical channels can share common test equipment.
Once the locations of network concentration are identified, the network technologies and the services provided need to be examined. Specifically, the types of test equipment required to monitor and diagnose these technologies and services must be considered. For example, if both ATM and packet over SONET (PoS) are utilized, it is important to have testing tools appropriate for both at that location. Similar considerations should be given to the network's services. The final key factor in NTAS planning is to determine how many network segments should share test equipment. If frequent problems are anticipated, then common test equipment for 30 to 100 segments may be appropriate; if fewer problems are expected, perhaps 1,000 or more segments can share testers.
With these considerations, an NTAS solution can be deployed with photonic matrix switches attached to the network equipment and segments as well as the chosen variety and quantity of monitoring and diagnostic equipment.
NTAS solutions composed of fully photonic switches controlled by centralized server software provide an environment where test equipment can be deployed precisely, safely and rapidly. The ability to remotely troubleshoot a variety of network technologies in today's exploding optical networks is critical to efficient network management.
Bert Lindgren is the chief technology officer of Net-HOPPER Systems Inc. (Norcross, GA). He can be reached at firstname.lastname@example.org.