Restoration, rebuilding continue in lower Manhattan

By MEGHAN FULLER

The telecommunications industry has always been prepared for natural disasters-earthquakes, hurricanes, floods-but how could anyone have prepared for the unnatural disaster that occurred Sept. 11, 2001?

"Nobody ever would have suspected that what happened in lower Manhattan could happen without a direct hit on a central office," admits Sam Greenholtz, senior analyst of optical networking at Communications Industry Researchers (CIR-Charlottesville, VA), "and yet, it totally wiped out lower Manhattan for a week."

Verizon, whose central office (CO) at 140 West St. was badly damaged, disrupting some 200,000 voice lines, is still in the process of restoring and rebuilding. And while its thorough disaster recovery plans enabled the carrier to quickly restore critical services, the events of that day have led many carriers to rethink not only their disaster recovery plans, but their basic network designs as well.

The CO, across the street from the World Trade Center site and adjacent to the collapsed 7 World Trade Center building, sustained major structural damage when steel I-beams penetrated its walls, causing equipment to dislodge, windows to shatter, and water mains to burst. Within the office itself, any equipment left standing was covered with dust and debris. Verizon estimates that 200,000 voice lines, 100,000 business lines, and 3.6 million data circuits were damaged or disrupted.

The outside plant faired even worse, as nearly all the fiber beneath the World Trade Center in the Port Authority Trans-Hudson (PATH) and subway tunnels was destroyed. The cable vault beneath 140 West St., where many of the cables were spliced into the switching system, was also severely damaged. Several I-beams crashed through the vault, slicing through conduits and cables. In addition, massive flooding-workers pumped 10 million gallons of water from the basement of 140 West St. alone-damaged another 300 cables.

Rerouting top priority
Verizon responded immediately, making emergency service restoration-rerouting traffic for the police and fire departments and for medical facilities in the area-its top priority. Ironically, the carrier's E-911 service was fed from the 140 West St. building. "As a matter of fact," says Grant, "the switch that fed E-911 was pretty much hanging in mid-air." The service on that switch was immediately redirected to the carrier's Bridge Street CO, however, and not a single 911 call was dropped.

Today, the work continues. "Right now, we are running service on damaged equipment in the sense that we have cleaned and repaired what we could," admits Grant, who adds that the equipment is being replaced in stages. "We need to do it independent of our customers' knowledge, meaning there should be no interruption of services." Verizon replaced one of its Class 5 switches in June, with another scheduled for replacement in December.

At press time, bypass cables still lined the streets around 140 West St., though Grant says crews are working to remove them as soon as possible.

Keys to faster service restoration
Verizon was able to restore much of its service quickly, thanks in part to the diversification of its physical plant, contends Dennis Elwell, executive director of Business Recovery and Continuity Services, part of Verizon's Enterprise Solutions Group. That diversification includes having more than one CO feed a locale, having SONET loops feed a CO, and having those loops take different paths, so there is diversity in the route of the path, in the entrance, and even in the risers.

Grant also cites Verizon's insistence on using Network Equipment Building Standards (NEBS)-certified vendor equipment as a key factor in speeding the immediate recovery effort. "For example, in some instances, we had racks that were hit by falling debris, which caused them to fall over, but the equipment right next to it was still bolted down, was still carrying traffic," she notes.

From a technology standpoint, SONET proved to be a winner. The carrier deployed 18 SONET rings to provide added capacity and ensure reliable service immediately following the disaster. "We would have to say that if we had a SONET ring with more than two nodes, something other than a point-to-point, every SONET ring in that configuration worked perfectly," admits Elwell. "And we share that assessment with folks like AT&T and WorldCom. SONET is not a leading-edge technology, but it is extremely resilient."

Different world
Today, all the major carriers should "be looking at their communities of interest, such as San Francisco, Chicago, St. Louis, etc., and asking themselves, 'What do we need to do now to prevent this from happening to us?'" contends CIR's Greenholtz. That said, he believes carriers are in a tough place given current economic conditions. "It is very difficult to spend that kind of money on a contingency and not on a growth factor, but I think they all realize they are going to have to do it."

From a physical standpoint, each CO should have a fallback office or the equivalent, he says, and fewer channels should be housed in a single office. "Over three million circuits went through 140 West St.," notes Greenholtz. "I don't think you'll see that happen again."

The folks at Verizon plan to evaluate the power and mechanical systems at their other major hubs as well as the physical layout of these spaces. "We're going to be looking at the way we have equipment in [major hubs] to ensure they are at the same level of survivability as the West Street building was and better," says Grant.

Immediately following the events of Sept. 11, WorldCom (Clinton, MS) established an executive task force to review its disaster recovery plans. "We needed to ensure that our plans are up to date," says Jennifer Baker of WorldCom's operations and technology public relations department. "Is there anything we need to update post-Sept. 11 that perhaps we know differently now? This effort continues today; the team meets on a regular basis."

There is certainly an incentive for carriers to reassess their network designs and recovery plans; Verizon expects it will spend about $1.4 billion on the restoration effort-work that may not be completed until 2004.

"It's a whole different world for the telecommunications industry," says Greenholtz. "It's changed the way [network] planning is going to be done forever."

The importance of service diversity is, for many business customers, a critical lesson learned in the wake of the Sept. 11, 2001 terrorist attacks on Manhattan and Washington, DC.

Businesses and consumers must ensure that their carrier's fiber plant is diverse-truly diverse, emphasizes Jennifer Baker of WorldCom's (Clinton, MS) operations and technology public relations department. "It can't be two different T1s into the same central office, for example, because if that central office is gone, as a customer, that doesn't help you out much," explains Baker.

She admits that this strategy may mean purchasing service from multiple carriers. "Maybe that means two different carriers providing your mission-critical services so that if one carrier does have an issue, you really do have a true backup," she suggests.

However, even if users choose to purchase service from separate carriers, they still need to do a little detective work, asserts Dennis Elwell, executive director of Business Recovery and Continuity Services, part of Verizon's Enterprise Solutions Group (New York City). The group has been touring the East Coast, conducting seminars on network protection in emergency situations. Elwell has heard frequent complaints from enterprise customers that they were not getting the diversified services they pay for.

"When there were so many of these terrible outages, people who thought they were receiving diversity from other carriers-certainly they were receiving a separate bill-only found that these carriers were, in fact, reselling services more than likely from a Verizon or an AT&T," he explains. "It was an eye-opener-disappointing too for customers and for Verizon."

Elwell admits that initially some consumers were angry and resentful toward Verizon for reselling its services so widely-at least 40 competitive local-exchange carriers were collocated at the carrier's 140 West St. central office (CO)-but he reiterates, "It's not necessarily Verizon's fault. Verizon is required to sell to whomever for those services."

Regardless, there has been an awakening among business customers that true facilities-based diversification is not only important, it also must be verified and consistently monitored, says Elwell. "These circuits can be swung over onto other circuit groups," he notes, "and what might be a completely diverse network one day might be rerouted. Customers have a greater awareness of that now."

Verizon's Enterprise Solutions Group is currently working with business customers to determine which COs they are fed from and how to strengthen that diversity.

REPRINTS
Whatever the purpose, all articles, products announcements, and advertisements in Lightwave can be reprinted to suit your informational needs.

For information, contact Kathleen McIntosh at:
(603) 891-9203, fax (603) 891-0587, or e-mail: [email protected].