Testing the testers: How new test systems are rolled out

Optical-network component makers must verify and characterize performance of each device, first during product development, again during manufacture, again during deployment, and then again during normal operations in regular maintenance. Testing and measurement become even more crucial when a problem arises that needs to be identified and solved immediately.

Noise, jitter, and signal degradation are just a few of the bugs that can wreak havoc with a network's bit-error rate. They must be detected, measured, and corrected. In the race for ever-higher transmission rates—and quicker revenue-generating rollouts—time-consuming testing tasks can make or break a company's plans. This chore becomes more daunting as higher speeds and more channels amplify and multiply irregularities in the signal streams. Testing becomes not only more crucial, but more difficult as the industry heads toward 40-Gbit/s standards.

Attention turns to test-and-measurement equipment providers as the pressure to produce high-speed results intensifies. Improvements in test equipment are needed to deploy more channels moving at faster speeds, but upgrading and testing the test equipment are major challenges for test-machine manufacturers.

In a new, three-story Tektronix building in Chelmsford, MA, teams of engineers, marketers, assemblers, and others are hard at work trying to satisfy the testing needs of today and the future. Their questions are the same as those of optical-component developers.

"How do we get an idea off the drawing board and into production rapidly?" is one of the questions Russ Morey, Tektronix manufacturing manager, attempts to answer each day. "How do we make certain what we dreamed about in the lab will really work in the field?"

The company's Chelmsford facility is focused on assembling and delivering innovations in testing and measurement. Here it produces optical-test-system (OTS) products that reliably measure transmit, receive, control, optics, O-E-O rates, and other characteristics. The Tektronix OTS product group uses modular-card architecture. With three chassis types available—a 14-slot rack-mounted system, an 8-slot bench-top system, and a portable 5-slot device—users can configure diverse combinations of transmit and receive test modules for 10-Gbit/s SONET/SDH and WDM testing.

TESTING TOP TO BOTTOM
The dynamics of the Chelmsford unit's responsibilities are mirrored in its physical layout. High-level engineers work on the top floor, designing new products to customers' increasing demands. Down on the ground floor are the assembly benches where OTS products are fabricated and packaged for delivery. Morey's office is in the middle, the second floor where every thing meets. Here, upstairs designs are tested and strategies devised for volume production downstairs.

Again, Tektronix faces the same problems in producing optical-test products that its manufacturing customers face—to make sure the plans of design engineers will really work and to translate these innovations into prototypes and then volume production. They must also determine what materials and parts are needed, and how to obtain them in required quantities and timeframes. Any interruption in any one of these paths can result in production delays.

"It's like there are four or five groups, all in disparate disciplines, going down different paths, and we all have to communicate and end up at the same place, at the same time," says manufacturing manager Morey. "My job is to see the bottlenecks coming and prevent them from arriving. And there are four key people I work with to do this—a test engineer, a new-product introduction person, a materials manager/planner, and a quality engineer."

Examining the roles, responsibilities, and relationships of these four people explains a lot about how a leading edge optical network manufacturer stays ahead of the competition.

TEST ENGINEER
The modules and circuit boards that are installed in the OTS product family of test devices produced at Chelmsford will be used to calibrate numerous dimensions of signal integrity. They must be modular, flexible, able to integrate with other measurement systems, and be upgradeable. Plus, they must be carefully calibrated and tested themselves.

The test engineer is involved from the start— hovering over the engineers, even in the idea stage, repeating the same question over and over again. "First in their minds, then in the writing, then when they're actually putting in the code or mapping out the circuit board, the test engineer keeps asking, 'How will we test this?'" says Morey. "We must make sure we can complete and calibrate it properly."

By creating the test capabilities as early as possible, the company is confident the new design can be accurately tested by the time it is ready to release a prototype. Software is written to help test and check the boards, even in the design stage.

"A great deal of the test engineer's time is spent in testing designs on how they make the boards," says Morey. "If there's a difficulty in a circuit, they find it, alter it, and make sure it is reliable and testable. This happens every day; we iterate and reiterate every day."

This painstaking preliminary work pays off in production. "We had a large run of a new product's prototype we will be releasing soon," says Morey. "We have the software to test using bulk techniques. If we'd done it the old way, on a bench with crude techniques, it would have taken much too long. Here, we eliminated a huge potential bottleneck by automating."

NEW PRODUCT INTRODUCTION
The challenges of high-speed production of new test devices prompted Tektronix to create a kind of ombudsman position for new-product introduction. "This is one manufacturing individual who is involved from womb to tomb on a new product," says Morey.

This "NPI" manager works with designers to get the parts and materials they need. She links with purchasing, and helps establish relationships with suppliers. Understanding an optics commodity supplier's capabilities and limitations can help maintain an uninterrupted flow of materials. Tektronix measures supplier performance, in quality, delivery, service, pricing, and other factors, to narrow the field. When a supplier meets the criteria, it can become a "preferred supplier"—a relationship that brings more opportunities to move its material into more Tektronix products.

"We want to make sure that our suppliers have the quality and volume to meet our requirements, and cull out the ones who can't sustain the demands," says Morey. "So when it's time to ramp up to meet our customer's product demands we won't be stopped."

MATERIALS MANAGER / PLANNER
Also involved in the supply chain is a manager focused on the overall production side. "The materials planner should be a senior person, someone who's been burned before, who can see the big picture," says Morey. This person, unlike the NPI person, is focused more on the business side of production, not just ordering new parts.

This planner consults with design engineers and with marketing people to determine client needs, cycle times, and delivery schedules. A great deal of planning goes into the new product's development and schedules must be adjusted as the designs change. The goal is first a pilot run, then an early run, then a volume ramp-up. The planner makes sure the whole process runs smoothly.

Testing is a major part of these planning responsibilities. When there are two to four boards all included in the same device, all being developed at the same time—but all being done by different engineers—it is difficult to keep everything on track with timely testing.

"In a positive way, the planner is challenging people, asking difficult questions," says Morey. "He even annoys me sometimes, always asking the 'what about this...' questions that I just want to go away."

But minute details overlooked in the rush of getting product out can come back to stop production later. He must track the changes and think of everything. "We want to work out any problems on the engineering bench, not on the production floor," says Morey.

QUALITY ENGINEER
The engineer responsible for overall quality lives in a world of constant testing, checking, and verifying. A wide variety of tests must be run on every product, from UL and CE certifications, to other safety, configuration, and performance tests. "Qualification is a lot more than just plugging it in and see if it works," says Morey. "It's a very complex process."

The test engineer coordinates the series of tests, making sure they are all conducted at the right step in the process. And it's becoming more complicated as the OTS product line expands, with more features added. OTS started as a rack mounted device, then it was compressed into a bench top size, and now, the latest model is a compact, five-slot portable chassis that only weighs about 40 lb. It had to be compatible across the board with the other OTS family's modules, software, test programs, and operational procedures. And it had to be compact, light, and rugged.

The connection to the customer closes the loop in the design-manufacturing process. The time from concept to deployment is becoming so compressed that the customer's needs are becoming a more important driver.

"Our technology is so leading edge, we respond to customer results to improve the products," says Morey. He cites one example of a customer who purchased and deployed an early version of OTS. "It didn't have certain characteristics they were looking for. We did a great deal of work with them—numerous iterations—and we came up with a nearly custom board for them, which was robust enough to handle demands. We also captured all our research data and used them in the next-generation design."

This bears out the Tektronix concept of a "lighthouse customer," —in which a key client's needs and responses are used to drive the development of new testing gear. Communication is crucial. Morey's primary job is to keep many different players involved in the process—including the customer. Says Morey, "The ideas is to get the whole supply chain integrated, working hand in hand."