CHAPTER 15
TELECOMMUNICATIONS
AND NETWORKING
Your Connection to the World

Everything is connected … no one thing can change by itself.
—Paul Hawken

INTRODUCTION

All of us are becoming more interconnected, as people and companies make increasing use of telecommunications networks to connect to suppliers, customers, business associates, and friends. Telecommunications has enabled our businesses and personal lives to be “just-in-time” no matter where we might be in the world. We use telecommunications networks to talk, share data, run applications, download files, organize schedules, exchange e-mail, and access the Internet. We use these networks to connect us with our coworkers and to collaborate with customers in doing the business of the firm. This chapter reviews the dangers that threaten our telecommunications networks and the attributes unique to these networks that require special processes to restore them after a disaster.

Most organizations have two distinct telecommunications networks: one for voice communication that uses the standard telephone system (also known as the public switched telephone network, or PSTN) and another for all electronic communication that uses one or more computer networks that are typically connected to the Internet. While the need for two distinct networks is slowly changing with the increasing use of VoIP (Voice-over-Internet Protocol) for sending voice communications over the Internet, many organizations will have to maintain both for the foreseeable future. Fortunately for our planning purposes, both types of networks have very similar issues that affect disaster recovery.

PUBLIC SWITCHED TELEPHONE NETWORK

How important can a telephone line be? What’s the cost of a telephone call? What is the value of a missed call? How much would you pay for 100 percent telecommunications reliability? Is such a thing even possible?

Consider this scenario. You are the Plant Materials Manager sitting at home in your living room. As you watch the evening news, you see a video of flames rising out of the roof of a key supplier’s main factory. You don’t need trouble like this. What should you do? You try calling your salesperson and the company offices but no one answers. First thing in the morning and all the next morning you try calling them. The company is more than an hour drive away, you can’t get away from the office for that long, and your calls still go unanswered.

Looking through your contact list, you select another supplier and place an order for a 2-month supply of goods to replace the supplier that had the fire. The set-up costs are a killer but it is worth the money if you can keep your factory running. At least this problem is contained. Trouble is, the fire you saw on TV was isolated to the front offices and the factory is fine. The warehouse is bulging. At a time like this, the supplier needs the cash, but employees can’t call out until the telephone switching room is replaced! The company took care of the fire but forgot all about its customers.

All companies strike a fine balance between the cost of reliable telephone service and the cost of downtime. In a time of tight budgets (which is always), you balance the cost of premium services against the potential loss of telephone service. The more reliable that you want your telephone network to be, the more you must spend. Telephone service is central to the conduct of business in most companies. A failure isolates customers and suppliers from your company—and that can quickly become a very lonely feeling.

Some companies are even more dependent on telephone service than others are. If you work at a factory, you use the telephone to conduct business, but your main concern is moving products down the line. The loss of telephone service does not slow down the assembly line one bit. However, if your facility is a call center for sales or service, then 100 percent reliable telephone service is crucial to your ongoing business. In the factory example, if the conveyor on the main assembly line breaks, the workers have nothing to do. In the call center, if the telephone service is interrupted, then they are likewise idled. How your facility utilizes its telephone service has a great deal to do with how crucial it is for your operations.

The North American telephone network is designed to carry traffic from about 10 percent of all telephones in a given area. This approximates its load at its busiest times. In a wide-area disaster, this capacity is quickly swamped, as people everywhere call to check on their loved ones. The cellular network doesn’t fare any better because portions of it also use landlines and it has its own capacity limits. In today’s fast-paced business environment, how long can your business afford to be without telephone service?

The disasters that can befall your delicate communication lines are legion. Anywhere along its path the wire can be broken, switching equipment can lose power, or problems can even occur within your building. The high level of telephone communications reliability of our modern telecommunications networks is the envy of the world. You must ensure that your company has reviewed the risks to its telecommunications pipeline and has taken steps to reduce the likelihood of failure. Our job here is to identify those risks and build a wish list of mitigation actions to make your company’s telephone communications even more rock solid than ever before.

The next few sections of this chapter review the basics of how your telephone system works. If you are already a telephone system expert, you can skip these sections. If you are charged with writing a plan in an area in which you are not a full-fledged expert, then you should read on. A basic understanding of the main components will make it easier for you to develop an appropriate plan.

PSTN Basics

The world of telephonic communications all begins with the telephone instrument on your desk. This modern marvel connects you to the world at large. The telephone is connected by wire to the wall jack, which in turn is connected to a wiring closet. A wiring closet may be one of those locked doors on your office building floor that you never get to look into. The same function can be served by running wires to a specific place on the wall of a factory (which is generally a wide-open space). In either place, you would see brightly colored wires running in spaghetti-like fashion to a telephone wire “punch-down block” or punch block.

If you look at a punch block, it seems to be a “rat’s nest” of colored wire routed in an orderly fashion, yet going in all directions. Each office telephone has at least one pair of wires running from its wall jack all the way back to the wiring punch block. The punch block also has wires running to the telephone switch for your building. This is where the two are connected to each other. If an office no longer needs a telephone line, the punch block is the place where it can be disconnected.

Private Branch Exchange

From the wiring closet, large bundles of wires run to the company’s telephone switching equipment. In a larger company this would be a Private Branch Exchange (PBX). A PBX replaces the long-gone company telephone operator who would connect internal calls with a plug patch panel. The patch panel physically connected the wires from one telephone instrument into the wires of another. This is really a basic operation, but until electronics matured, it was the only way to do it. This is now all done electronically. In today’s offices, a PBX takes these incoming wires and, using the signal on them, provides electronic switching of calls within the building. A simple way to think about a PBX is to consider it a special-purpose computer with all the support needs of a minicomputer.

The PBX determines which calls are intended for external telephone numbers and connects to the local telephone company central office using “trunk” lines. Trunk lines are used for inbound or outbound calls. The number of trunk lines the PBX has connected to the central office determines the maximum number of external (inbound or outbound) calls you are capable of supporting at a given moment. Modern PBX systems offer a wide range of additional services such as:

Voice mail.

Telephone conferencing.

Call transferring.

Music on hold.

Based on this, from a business continuity perspective, you have a large single point of failure device. You must plan to recover from a catastrophic failure of the PBX (e.g., burned to a cinder in a fire). A close examination of the PBX room and its ancillary equipment will show that it is essentially a computer room, requiring the same electrical and climate stabilization actions as any computer room. Backup copies of the configuration data must be made for each device and handled with the same care as the backup data from your computer system. These data should be securely stored off-site and available when needed. Up-to-date backed-up data are the key to a prompt recovery.

Internal to a company, the telephone signals can be analog or digital. Digital PBX systems provide a wide range of services beyond simply routing calls. Most PBX systems use digital signals to communicate with the telephones. This allows for additional services, such as one-touch dialing, preprogrammed telephone numbers, voice mail, etc. This is significant in the case where you want to use a modem to dial out of the office. A fax machine or a direct modem connection requires an analog line. If you have a digital PBX, then you will need separate analog lines.

Rather than run a multitude of analog telephone lines, most companies access the Internet via their external data network. This works fine for most office dwellers. However, some devices still need these analog lines and you should keep track of where they are. Examples of analog dial-out lines might be for an alarm service, which dials out to notify the repair service of an out-of-tolerance condition, or for validating a credit card in stores at the point of purchase.

OTHER VITAL EQUIPMENT LOCATED IN THE PBX ROOM Other important equipment is typically located in the same room as your PBX. After the critical devices are identified, make sure they are protected and draft a plan to fully recover them. These devices may include:

Interactive Voice Response (IVR). Gives callers information based on what the caller enters using the telephone keypad. You have heard the messages—please select 1 to talk to sales, select 2 to talk to … These audio tracks should be backed up and the queuing logic documented.

Automatic Number Identification. This is the business version of what is sold to consumers as “caller ID.”

Intelligent Port Selector. Connects the incoming call to the first available line. This is used when you have multiple people answering inbound calls, as in a hotel chain’s reservation center.

Call Management System. Used to monitor the volume of telephone calls during peak periods, to identify the number of telephone operators needed, and to track operator efficiency.

Call Accounting. Tracks calls made and assigns them to a billing account. This is also known as a “Station Message Detail Recording” (SMDR). This is used in various ways. Lawyers might use it to bill their time to a specific client. Some companies use this to track employees’ long distance usage, etc.

Call Monitoring. Tracks the level of call activity by showing the status of the trunk lines, the number of calls in progress, the number of calls waiting in queues, the wait time, the number of abandoned calls, and the status of the operators.

The Telephone Company’s Central Office

Soon after telephone service was first created, it became obvious that every person could not be wired to every other person they might possibly want to call. This would result in an impossible maze of wires. To call someone new, you would first need to run a wire from your telephone to their telephone. Imagine the problems in asking a girl for her telephone number in those days!

To simplify this problem, all the telephone lines were run into a centralized building and then the switchboard operators would physically patch your telephone line into the switchboard, making the connection. These buildings would be located at various places around the city and were the central place where connections were made. Eventually, automated switching made it easier, but buildings were still needed to switch the calls. In more recent years with the advent of solid-state circuits, the floor space required for these buildings has shrunk dramatically. You still see them around: small buildings without any windows, usually neatly trimmed grass, and a small telephone company sign by the front door.

The central office provides a service similar to your PBX by switching your call to another local telephone, to a different central office far away, or to a long-distance carrier’s point of presence (POP). The long-distance company then routes the call through its switching center and back to a distant central office and down to the far-away telephone.

Interexchange Carrier Point of Presence

With the breakup of the AT&T long-distance monopoly in 1982 came the creation of independent long-distance telecommunications providers. These non-AT&T long-distance providers are known as Interexchange Carriers (IXCs). Along with this choice came the opportunity to split your long-distance service across different carriers in hopes that all of them would not be knocked out in a disaster and therefore your communications traffic could flow out on the alternative pipeline. For that to be true, a lot of careful planning is necessary.

First, you must ensure cable separation so that you have a different wire path from your facility to the IXC’s point of presence. Most companies are a mix of owning the network in high-traffic areas and renting from another carrier in a low-traffic area. This means your traffic separation may only be on paper. If you are in a high-traffic area, the carrier may run a separate cable to your facility. If you are not, then you will probably connect to the IXC in the nearest central office. Do not take for granted that because you use two companies you are on two different wires. Ask them if they share lines and ask to see the route of the cable.

Some telecommunications experts believe that route separation is much more important than using multiple telecommunications companies. They feel it is easier to manage one supplier so long as the cable issue is addressed.

When evaluating networks, the first thing to consider is whether they rent their network or own it. Most are a mixture. The more of the network equipment that they own, the more control they can exercise over it. When choosing an IXC, things to consider include:

What is their system availability time, and what they will guarantee?

What are the consequences to the IXC of downtime? You can ill afford it. A few extra free minutes of service every month is poor recompense for missed customer calls.

What is the restoration priority for the sections of the network that you will be using?

What are their alternate routes for the places you normally communicate with? Don’t automatically assume a new carrier will be a better alternative to your existing carrier. Is your service route a spur (single-threaded) service? Is it a ring architecture, which at least gives you two paths in case one has a problem?

How often does the carrier practice its disaster recovery procedures?

How easily can the IXC shift your inbound calls to another site?

OK, well, this all sounds pretty straightforward. After all, telephone service has been around for well over 100 years and its technology is pretty well known. What could possibly go wrong?

COMPUTER NETWORKS

All but the smallest firms today have one or more networks within their organization. A network consists of one or more servers that are connected to one or more workstations and allows users to share information and resources. This connection can be by copper wire, by fiber-optic cable, or even by radio frequency (RF, or wireless). Rarely are workstations connected directly into a server. Typically they plug into the wall jack, which is connected to a wiring closet. In the closet will be one or more “hubs” that will connect the local devices to a hub in the computer room. This room is often the same closet as contains the telephone wiring.

In the computer room (either a central location or possibly sprinkled about the facility) sit the servers. A server is a computer that runs software to provide access to resources attached to the network, such as printers, disk storage, and network applications. A server can be any type of computer that supports the sharing of resources: it may be a standard desktop PC, or a dedicated device containing large amounts of memory and multiple storage devices that can support hundreds of PCs at the same time.

The next section reviews the basics of how your computer network works. If you are already a network expert, you can skip this section. If you are charged with writing a plan in an area in which you are not a full-fledged expert, then you should read on. A basic understanding of the main components will make it easier for you to develop an appropriate plan.

Computer Network Basics

The term local area network (LAN) is typically used to describe workstations and servers that are all physically in the same location. A LAN can be implemented using two main architectures:

Peer-to-Peer. Each computer on the network communicates directly with every other computer. Each computer is treated equally on the network. Communication between computers is coordinated using a network hub.

Client/Server. Each end-user PC is connected to a central server, which coordinates communication and supplies resources to the end users. Typically the server contains file storage, printer, backup, and other resources that are shared by the end-user client computers.

Figure 15-1 shows a typical client/server LAN, with five PCs connected to a server. The server has attached a network printer and a backup tape drive that can be accessed from any end-user client computer on the network.

A wide-area network (WAN) is a data communications network that consists of two or more local area networks connected to allow communication between geographically dispersed locations. Most companies have some sort of WAN connection, either connecting to other locations within the organization or to the public Internet. There are several different types of WAN communication links available. The most common ones are:

Dialup. This is the most basic type of connection, using a modem and a PSTN line. This is also the slowest and most unreliable type of connection available, supporting up to 56 kilobits per second (kbps).

FIGURE 15-1: Typical LAN configuration.

ISDN. An ISDN (integrated services digital network) connection uses a PSTN line using an enhanced communication standard to speed up the connection to 64 or 128 kbps.

T-1. A dedicated connection that supports data rates up to 1.5 megabits per second (mbps). A T-1 line consists of 24 individual channels, each supporting 64 kbps of data.

T-3. A dedicated connection that supports data rates up to 43 mbps. A T-3 line consists of 672 individual channels, each supporting 64 kbps of data.

Frame Relay. This is a packet switched protocol for connecting devices on the WAN. It supports data transfer rates up to T-3 speed.

Cable Modem. Using the same digital or fiber-optic network as cable TV, cable modem Internet is capable of speeds up to 60 mbps.

ATM. ATM (Asynchronous Transfer Mode) is a means of digital communications that is capable of very high speeds; suitable for transmission of images or voice or video as well as data.

Wireless. A wireless LAN bridge can connect multiple LANs for distances up to 30 miles with a direct line of sight.

VPN. A VPN (Virtual Private Network) uses encryption in the lower protocol layers to provide a secure connection through the public Internet.

Figure 15-2 shows a corporate wide area network, with two remote locations linked to the headquarters, which is in turn connected to the public Internet. Just as your phone system has a room for patching users into the telephone network, your computer network also has a central location where individual pieces of equipment are connected into one or more servers. This might be the same room as your telephone patch panel, or it could be part of a separate dedicated computer room.

It starts with each piece of equipment on the network being connected by wire to the wall jack, which is in turn connected to a wiring closet. A wiring closet for your computer network is typically located in or next to the computer room where your servers are located. The wiring closet consists of many different cables running in spaghetti-like fashion to a network cable “punch-down block” or punch block.

The network cable punch block is similar to the telephone wiring punch block discussed earlier. It has network cables routed in an orderly fashion, yet going in all directions. Each piece of equipment on the network has at least one cable running from its wall jack all the way back to the cable punch block. The punch block also has wires running to the computer servers. This is where the two are connected to each other. If a piece of equipment needs to be removed from the network, the punch block is the place where it can be disconnected.

FIGURE 15-2: Typical WAN configuration.

Internet Service Providers

Internet Service Providers (ISPs) connect your organization to the Internet. You are connected to your ISP via one of the network connection options listed in the previous section. Most organizations will have only one physical connection between its network and the ISP for Internet access. If Internet access is critical to the operation of your business, you should consider a connection to a second ISP to give you a second pathway to the Internet. There are, however, several things to evaluate and plan for when considering a redundant ISP. (This is similar to considering multiple telecommunications companies.)

First, you must ensure cable separation so that you have a different wire path from your facility to the ISP’s point of presence. Most ISPs are a mix of owning the network in high-traffic areas and renting from another ISP in a low-traffic area. This means your traffic separation may only be on paper. If you are in a high-traffic area, the ISP may run a separate cable to your facility. If you are not, then you will probably connect to the ISP’s nearest point of presence. Do not take for granted that because you use two ISPs you are on two different wires. Ask them if they share lines and ask to see the route of the cable.

Some telecommunications experts believe that route separation is much more important than using multiple ISPs. They feel it is easier to manage one ISP so long as the cable issue is addressed. When choosing an ISP, things to consider include:

What is their system availability time, and what they will guarantee?

What are the consequences to the ISP of downtime? You can ill afford it. A few extra free minutes of service every month is poor recompense for missed sales due to the Internet being unavailable.

What is the restoration priority for the sections of the network that you will be using?

What are ISP’s alternate routes for the places you normally communicate with?

How often does the ISP practice its disaster recovery procedures?

RISK ASSESSMENT

When developing your disaster recovery plan for your telecommunication services, look for single points of failure that will adversely impact your critical business processes. These critical processes should have been identified in the Business Impact Analysis. External threats to telecommunications include cables being cut, interference from electromagnetic sources, attacks by hackers and other intruders, and damage from natural hazards, such as fire or water. Internal threats include many of these same dangers; cables can be cut by remodelers and leaky water pipes can damage cables and equipment. In an ideal world, you would have duplicate service running to each telephone or desktop. Externally, an issue with multiple Internet connections is that there are now two points of potential entry for hackers; be sure to work closely with your security team to adequately protect your WAN. Internally, it is usually not cost-effective to run duplicate cables; an alternative is to install an extra jack every few cable drops. If a problem occurs in an end-user’s cable, the extra jack would be available as a backup until the problem cable is repaired.

It is also important to review your vulnerability to problems with the devices that connect your telecommunications network together, such as routers, switches, and hubs. Your Business Impact Analysis should determine the level of threat that these devices pose if they fail. Look at having redundant devices installed that can take over if the primary device fails.

Wireless telecommunication connections can provide an effective backup strategy if the wired network fails. Wireless networks have no wires to fail and can be quickly installed if needed after a failure in the wired system. The major drawback to using a wireless network is that the transmissions can be intercepted; make sure encryption and other security measures are implemented to avoid others intercepting your network traffic.

You should also consider using network monitoring software that can detect problems on the network. Network monitoring software can alert you immediately if a node on the network is having problems or has failed, which will allow you to restore service more quickly and help prevent problems from cascading. Most network monitoring software can be configured to look for system parameters that fall out of the desired range and then generate a message to a support person to take action.

Natural Hazards

Now that you know the basics of how your telecommunications systems work, what are the risks to these systems? Chapter 3 discussed natural hazards in detail, so this section only addresses those natural hazards that have a major impact on your telecommunications systems.

Ice Storms and Blizzards. Ice can coat cables strung from telephone poles and, if the weight is great enough, potentially bring them down.

Thunderstorms and Lightning. Severe rain can weaken the ground around a pole and cause it to sag when there are high winds at the same time pushing against it. Lightning can strike telephone poles and send a major charge flying down the line, burning up wire and equipment along the way.

Tornadoes. A powerful destructive force that can snap lines and rip up telephone poles. High-risk areas are prime candidates for buried lines.

Hurricanes and Floods. Can cover a wide swath of land and not only bring down a telecommunications network but also prevent crews from promptly addressing the problems.

Human-Created Hazards

People are just as big a danger to your telecommunications systems as nature can be. Human-created dangers include:

Breaks in Buried Lines. Sometimes emergency excavation is necessary (such as repairing a broken gas main). Sometimes the local government is cleaning trash out of the ditches alongside the road. Sometimes a well-meaning person just digs without asking (including on your own property). In any of these cases, there is the chance that your tiny little cable will be dug up and severed.

Accidents. Sometimes people miss the tight turn and break off a telephone pole. If this is your only line to the central office or your ISP, your service is gone until the pole is replaced.

Central Office or ISP Failure. A problem in a central office or with your ISP can quickly shut down your telecommunications unless you are wired to a second provider.

Denial of Service Attacks. Hackers can use various techniques to render your Internet service unusable. A Distributed Denial of Service (DDoS) attack uses the computing power of thousands of vulnerable, infected machines to overwhelm a target or a victim.

Telecommunications Equipment Room

The room that houses your telecommunications equipment may not be as safe as you think it is. Dangers to your telecommunications room include:

Temperatures that are too hot, too cold and that swing widely are all hard on your telephone switching equipment. Extreme temperatures stress the printed circuits. Large swings in temperature (hot to cold) cause expansion and shrinkage of circuit cards and again can weaken components over time.

Humidity has an effect on temperature and the growth of mold on your equipment.

Loss of electrical power will definitely stop a PBX, routers, switches, and other telecommunications equipment. Pay phones and direct lines out should still be operational since the telephone circuits supply their own power.

Water pipes overhead in your telecommunications room could release water onto your equipment. The same goes for your cable panels. Overhead pipes along external walls could potentially freeze and leak when thawing. Leaks from rooftop air conditioning compressors are also a problem.

Security may be a factor. This room is not set up to accommodate tourists and they should not be allowed in. Keep the door locked at all times.

Fire is a possible danger. This equipment generates heat and is in danger from fire or the sprinklers used to extinguish a fire. Typically this room is unattended and a slow-starting fire may go undetected. Gas fire suppression is expensive but may save your equipment.

Cabling

The discussions earlier about wiring closets had a purpose. Imagine the mess if an isolated fire in the wiring closet melted all these wires. In terms of structural damage, you got off pretty easy. But in terms of damage to the telecommunications system, that entire area of the facility will be without service for some time. New cable would have to be run from the PBX or Internet server to the closet and from the wall to the closet. The alternative is to splice an extension onto each cable and run it into a punch block. Either way is time consuming and expensive. So you can see how the wiring closet can be a single point of failure for your telecommunications systems. This makes it a good idea to keep everyone out of it and to not store anything in the closet that might cause a problem.

INTERNAL CABLING For disaster avoidance, concerns begin with the wiring closet and patch panels. Take a walk with your telecommunications specialist and identify the location of all telecommunication patch panels in your facility. Things you are looking for:

1. In a closet

The door and any windows are kept locked.

There is fire suppression equipment (usually fused link sprinklers).

There is nothing else in the closet except telecommunication and/or data communications equipment. Combustible materials stored in the closet threaten your equipment. This is not the place to store holiday decorations, old files, office supplies, janitor supplies, etc.

Very few people have keys—telecommunication support staff, security, and no one else.

There should be sufficient light to work in the closet.

There will often be data network wiring and telephone switching equipment together in these closets. They are a natural fit here. The issue is that where the patch panel is simply a physical connection for wires, the network equipment is energized electronics, which will generate heat and introduce a potential fire source.

2. Outside a closet

Be sure it is covered and the cover is locked. Same rules for the keys as for the wiring closets. Enough keys for the right people and none for anyone else.

All cables and wires leading into the external panel are encased in heavy conduit to inhibit tampering.

If this is located in a warehouse or factory, be sure it is strongly protected from environmental influences (leaky ceilings, etc.) and from being crushed by a forklift or toppling stacked material.

EXTERNAL CABLING The first rule of outside cabling is “cabling and backhoes don’t mix!” A chain is only as strong as its weakest link. Your servers and PBX are snugly locked up in their room, and the ISP’s office and the telephone company’s central office are also secure. But the wire in between is exposed to the ravages of weather, people, and machines.

External cabling, which is the cable that runs from your wiring closet to the telecommunication provider’s office, is your biggest concern. The wire runs from your building to an access point along the road, usually on a pole. Then the wire runs through the countryside (usually along a road or railroad) to a point of presence or central office. In the city, it might run through underground pipes to the point of presence or central office. You have no control over where the wire is run and no capability to protect it! In some areas, you will even have separate cable runs to be concerned about—one for the local telephone company, one for the Interexchange Carrier (long-distance service), and yet another for your Internet service.

A common term used when describing part of a telecommunications network is “the last mile.” The last mile describes the wire from the telephone company’s central office or the ISP’s point of presence to your structure (and is usually more than a mile). This is also known as the “local access” or the “local loop.” This part of the network is the most vulnerable. It is often carried on telephone poles (susceptible to ice storms and errant vehicles) or underground (susceptible to damage from digging).

ROUTE SEPARATION The best path to telecommunications reliability is redundancy. This can include redundant equipment, redundant technicians, and redundant cabling. The more alternate paths that a signal can be routed, the more likely it is to get through. The principle of cable route separation should be an integral part of your telecommunications network design. Essentially this means that you have more than one cable between your building and your telecommunications providers. This prevents a total communications outage from a single cable cut.

Few companies can afford to take this to the extreme, but you can look into some steps. Begin by asking the telecommunications provider to show you the route that your cable takes from the wall of your building to their office. This will also show you how exposed the cable is to auto accidents (are the poles close to the road), to backhoes (is the cable buried along the road in some places), or any other number of threats.

With this experience in mind, negotiate with the telecommunications provider a fresh cable run from your building (exit from the opposite end from the other cable) to a different central office or point of presence. This will keep you operational in case your usual point of presence is damaged or experiences an equipment failure, or if the cable to the point of presence was broken. How likely is that to happen?

To reduce the chance of a total system failure similar to the Hinsdale disaster, the telephone companies have gradually migrated their central office structure from a spoke-and-hub approach (with its obvious single point of failure) to a ring or mesh approach, in which multiple central offices are connected to each other. In this scenario, calls are routed around the damaged central office in a manner that is transparent to the caller. This is an ongoing process and may not yet have been completed in rural areas or small towns. If possible, you want routing separation with the wire running from your facility running to two different central offices on separate routes.

Route separation is more important than having multiple vendors. Most cable routes follow railroad rights of way and the major carriers’ lines commonly converge at bridges. Imagine the number of places there are to cross a major river, highway, etc. There aren’t many to choose from. The lines from various companies often come together here and cross under the same portion of the bridge. What could go wrong? Vehicles crossing bridges might catch on fire, river barges can break free and strike bridge pilings, and major bridges are tempting terrorist targets.

Many telecommunication providers define cable separation as a distance of 25 feet or more between cables. Others ensure cables are at least 100 feet apart and have at least a 2-foot separation at cable crossovers. Ask your Internet, local, and long-distance telephone carriers how they define cable route separation and how faithful they are to that standard. When using multiple vendors, even if their cables are separated, they share a common weakness if they join at the same point of presence or cross a river under the same bridge. Upon close investigation, you may even see that where you are using two companies, one is leasing part of the same wire from your other provider!

What to do? Find out how many access points there are for your telecommunication service. Does it enter the building at more than one place (again, trying to separate the cables)? Ask the telecommunication provider to identify the route that the cable takes from their office to your facility. Drive along this route and look for potential problems.

In terms of your long-distance provider, where is their point of presence located? Where is the point of presence of their competitors? You can consider running a separate connection to a second point of presence to provide for multiple circuits. Depending on the distance, this separate line may be quite expensive to run and maintain.

MAP IT OUT Now that you know the cable route to the telecommunication provider, map it out—from where it enters your wall to the provider’s point of presence. You might think, “Isn’t that the telecommunication provider’s responsibility?” The answers is yes, but type up that excuse and paste it to your wall in case the line is cut. Try it on your boss! If you drive to work using the same route as your cable, you might see construction crews digging near where the cable is to repair a water main, snapped poles, or sagging wire due to accidents or severe weather—any number of threats too close to the wire for comfort.

So draw out a map of where your wire runs from the telephone pole outside to the telecommunication provider’s office. Then make a detailed map of the run from the pole to the wall and on to the wiring closet room. Indicate which lines terminate in equipment provided by the telecommunication provider and which lines provide essential services.

DEVELOPING A PLAN

This sections describe issues to consider as your develop a disaster recovery plan for your telecommunication assets.

What Are We Protecting?

As you know, the first step in building a plan is to make an inventory of your telecommunications assets. You will assemble at least three lists. The first lists every major item and who to call if it breaks. The second shows where the main cable runs are located in the building. The third includes all the telephone numbers used by the facility. In a crisis, you may need to reroute some of these to another location.

1. Begin with a list of all major devices in your telephone switching room and network room, such as the PBX, IVR, Internet server, etc. Include on this list:

A description of each device.

The serial numbers of the main equipment and any major components.

The name, 24-hour telephone number, contract number, and contract restrictions for whomever you have arranged to service that item.

The location of every item, including a simple floor map of all telecommunication rooms.

Be sure to back up your entire configuration data, either on magnetic media or, if the file is small, print it off and store it safely away.

2. Now make a wiring inventory of all the cable runs within your facility. You do not need to show individual runs to the offices. In a crisis, you can always shift someone to another office. This is best accomplished with computer-aided drafting software on a digitized version of the floor plan.

Indicate the runs on maps of each building or floor of a multistory building. Knowing the location of these cable runs is important to quickly assess damage.

Indicate where the telecommunication service enters the building and its route to the telephone exchange and network rooms. Note any hazards along the cable path.

If you have pay phones, indicate them on the map.

If you have any independent direct lines that bypass your telephone switch, mark them on the map.

3. Make up a telephone number inventory with all the telephone numbers assigned to your building/facility.

DID (Direct Inward Dialing) lines.

Dedicated telephone lines that bypass your PBX, such as fax machines.

Pay telephones.

Foreign exchange lines.

Identify Critical Telephone Circuits

Telephones are used to communicate, and every part of the company uses them differently. In an emergency, salespeople will need to contact customers, the warehouse will need to call suppliers, and the people at the disaster site will want to call home. In a disaster, you won’t be able to please everyone right away, so you need a restoration priority guideline to know which circuits to recover (or protect) and in what order.

Start this analysis by reviewing the critical functions identified by your executives at the beginning of the business continuity planning project. Next, meet with representatives from each department and ask them to identify their critical telecommunications needs and at what point in the disaster containment/recovery they would be needed. For example, early in the emergency, the Human Resources team might need to notify employees to not report to work or to come in at certain times, etc. Each department must prioritize its communications needs.

With your critical communications functions identified, you can determine which circuits support these key functions. If you trace out the top three or four circuits, you may see some of the same hardware, some of the same cables, and circuit paths common to them all. Working down your list, you can see which of your hardware devices (or cable bundles) has the greatest benefit to being restored first.

You must also include in your plan how you will relocate operations to another site. This is a twofold issue. In the first case, if a department must be moved off-site due to damage to its offices, you need to be able to shift its inbound calls to the new location quickly. In the second case, if your telephone equipment room is destroyed, you must quickly restore a minimal level of service. Some people might plan to build a new telephone room onsite. Others will contract with a service company that will bring a telephone equipment room to your site already set up in a large trailer. The company needs to plug into your wiring (no trivial task). This could buy you time until your equipment room is rebuilt. If you elect to bring in the configured trailer, this agreement must be made in advance, and be aware that these might not be available in a wide-area disaster.

Review Network Security

Your network has some unique security issues to consider. The first is physical security. No one but the network support staff should be allowed in the network closets. The network administrator must work with the facility’s security manager to ensure that these doors are always locked. Depending on the airflow in the closet, some vents may be added to the doors. These rooms are not for storing holiday decorations, old documents, unwanted furniture, etc.

Sometimes network equipment is in freestanding cabinets instead of closets. These cabinets must be locked just like the closets and the keys held by only a few people. In a large facility, this can add up to a lot of keys. Your goal is to minimize the number of keys by using a submaster key for all closets and, if possible, a submaster key to all cabinets. These keys must be tracked and, if lost, a determination made whether to rekey the locks.

The next area to consider is logical security. The network software on the servers and in the network devices will be, in some cases, password protected. These passwords should be protected like any other and known only to the key network support staff. However, they should also be written down and locked in the data processing manager’s office in case the network staff is unavailable. (In a wide-area emergency, the network manager or key staff members may not be available or able to come in.)

Another issue to consider is having a policy that no one is permitted to plug anything into the network. That is the exclusive job of the network support team. This is to stop people from plugging in their home notebook PCs and bringing down your network. In addition, contract employees should never be permitted to connect into your network. The same policy should cover your wireless nodes.

Because servers can support a large number of users and are used to host critical applications, loss of a server can have a severe impact on the business. Some processes to ease the restoration of servers include:

Store Backup Tapes and Software Off-site. Follow the procedures discussed in Chapter 17 for handling and storing backups. Backups of data and application installation media should be stored off-site in a secure, environmentally controlled facility.

Standardize Hardware, Software, and Peripherals. Standard configurations of these items will make restoration much easier. The standard configurations for hardware, software, and peripherals should be documented in your plan.

Document the Network. The physical and logical network diagram must be kept up to date. The physical documentation should include a diagram of the physical facility, where the primary cables are routed. The logical diagram shows the network nodes and how they interconnect. Both diagrams are critical in restoring the network after a disaster.

Document Vendor Information. Maintaining up-to-date information on the vendors you use for hardware, software, and peripherals will make it easier to restore your operations.

Work with Your Security Team. The more secure your systems are, the less likely you are to have a data loss due to a security breach. You’ll also want to be sure you can restore the latest level of security if you must configure new servers after an emergency.

Telecommunications Mitigation Plan

With these risks in mind, along with due consideration of the identified priority circuits and equipment, you can assemble a mitigation plan to reduce the likelihood of a threat or its impact if it occurs. The key to telecommunications mitigation is redundancy. Redundancy in equipment in case one machine must be repaired. Redundancy in communications routes in case a cable is severed or interrupted. Redundancy in communication methods, such as radios, cellular telephones, or satellite communications, will provide at least basic communications support.

CABLE MITIGATION PLAN Your cable mitigation plan should consider the following.

Multiple paths for the “last mile.” Investigate the path from your telecommunications equipment room all the way to the central office or point of presence. Ask the telecommunications provider to make another connection from your PBX or Internet server out through the wall of your building at a point distant from the other exit point and on a different route to a different point of presence. Be sure that your in-house wiring staff understands what you are asking for before they start.

Multiple paths for Interexchange Carriers. Investigate the path from your telephone equipment room to your service provider’s point of presence. Avoid the same route as used for your local telephone service.

From the pole to your wall, if the cable is underground, ensure it is clearly marked with “do not dig” indicators or other obstacles to keep digging equipment away (or at least delay them until the cable can be marked).

TELECOMMUNICATIONS EQUIPMENT ROOM MITIGATION PLAN Think of your telecommunications room like a computer room. Their environmental and security needs are almost identical.

Uninterruptible Power Supply (UPS). When you lose electrical power, this is what will keep your PBX or Internet server active until external power is restored or until your facility’s electrical generators kick in. Conduct a power loss test to see which equipment is not connected to the UPS. If the device isn’t essential or time consuming to restart, take it off the UPS. The fewer machines on the UPS, the better. When conducting the power test, see how long the batteries can support the load. Be sure the UPS is properly maintained by your service company.

Fire. This room requires the same fire protection as a computer room. An early-warning fire alarm system and gas fire suppression system are highly recommended.

Security Access. This room is normally unattended. It should be locked as no one has any business strolling around in this room. The telecommunications system administrator can normally perform his or her administration duties via terminal over the network.

Structural Investigation. Although you can rarely select the room, a close inspection can identify problems. Water pipes running along the walls or along the ceiling are a potential source of problems. They may freeze or leak. They should be watched carefully. Consider installing a plastic shield attached to a drain placed under them to catch condensation or leaks. A roof-mounted air conditioner may cause a roof leak. External walls may stress the air-handling equipment as outside temperatures heat or cool the walls.

Temperature Variability. Your equipment must be in area that maintains a specific operating temperature and humidity range for maximum life. This is also essential to maintaining your service agreement coverage. Proper air conditioning, heating, and humidity control equipment must be in service and well cared for.

Alarms Such as Humidity, Temperature, Fire, and Electric. A bank of alarms will help you monitor the condition of the room. These alarms must sound within the room, as well as at the security guard station as the room is normally unattended. Early detection will reduce the likelihood of significant damage. Include automatic paging equipment for notifying the after-hours support team of problems. Consider installing automatic shutdown software for your equipment. This signals your hardware to shut itself down gracefully in the event of a problem.

Data Backups. Like a computer, your switch and configurable devices need to back up their configuration data whenever these change. Store copies of these files in a secure off-site location.

Housekeeping. Keeping the door locked will help to prevent others from using the switch room as a storage facility. Nothing should be stored in this room that does not pertain to the equipment. This will reduce the amount of combustibles available to a fire. Do not let this room become another storage or janitor’s closet.

ALTERNATE COMMUNICATIONS METHODS You must have written procedures for quickly routing Interexchange Carrier traffic to the local telephone service in the event that your IXCs have major problems. How hard is this to do, and then to switch it back again? Is it easier (or cheaper) to do this by splitting the load across two carriers and then shifting the entire load to the functional carrier? Other items to consider include:

Do you have company-owned cell phones and cellular modems for communicating when the telephone system is inoperable?

Develop written procedures for how to work with the telephone company to shift specific telephone numbers or all incoming calls to a different company site.

Do you have any employees who are ham radio operators? It is nice to have alternative channels when the primary ones are not available. In a wide-area emergency, radio communications will also get congested. Radio conversations are not secure and anyone with the proper equipment can listen to them. Radio communication is also slow and not suitable for large data volumes.

Other communications alternatives include satellite and microwave, both susceptible to problems if the antenna has been shifted by a storm or earthquake.

ACTION STEPS FOR YOUR PLAN

Most companies contract emergency recovery of their telecommunications equipment through the same company that services their equipment. A problem arises if there is more than one service company involved. An alternative is to arrange with a company to come onsite if your telecommunications equipment room is destroyed and to set up a trailer adjacent to your building with a ready-to-go telephone switch and Internet connection. To support this, companies often run cables to the front of the building for a quick connection.

Another consideration is that pay phones do not route through your PBX. They are direct lines to the outside service. So, if your PBX is inoperative, use pay phones. Keep a list of where they are and their telephone numbers.

Cell phones are also common. Cellular traffic can be somewhat limited in a wide-area emergency as the local cell towers become saturated with calls. Use this as an alternative communications channel, not as the primary backup.

A branch office is an ideal place to shift operations to until the disaster-struck site is recovered. The key is how far away the backup facility is located from the affected site. It should be far enough away to be unaffected by the same disaster. There is no set mileage distance but it should at least be on a different power grid and telephone central office. If it is too far, then you must also provide lodging for the relocated people.

One problem is how to relocate to these sites in a crisis. If this is during an area-wide disaster, mass transit such as air travel may be disrupted and driving there yourself may be difficult. One side aspect of the attack on the World Trade Center in September 2001 was that it shut down all air travel. Companies scrambling to activate their out-of-state recovery sites had a difficult time shifting key personnel and material to the site. No one had planned for a complete shutdown of the air transportation system.

Things to consider after a disaster:

Do not make any unnecessary calls—only for emergencies.

When calling, you may need to wait several minutes to get a dial tone. Do not hit the switch hook, because every time you do, you are placed at the end of the line for the next available dial tone.

When you receive a dial tone, quickly dial your number. In a time of low telephone service availability, the dial tone is offered for a much shorter time.

TESTING

Plans are written by people with the best of intentions but, unless they are tested, you will never know what you don’t know. Testing a plan exposes gaps and omissions in the process steps. It points out incorrect emergency telephone numbers and emergency equipment that no longer exists.

Test exercises are a great way to train the people on what to do. The more that they practice their emergency steps, the faster they will be able to perform them as they become familiar actions instead of something new. Where possible, include your emergency service providers. They will be a great source of recovery information.

When testing your plan, include some of your contract service providers. They may be able to point out some gaps in your planning or things that can be done in advance to make their assistance to you flow much easier.

Also, over time, your communications flows will shift. As this occurs, update your testing accordingly.

CONCLUSION

The modern business relies on telecommunications to perform its role in the marketplace. Like any resource, we need to be familiar with its role within our operation and how its absence will affect us. Redundancy is your best defense against a disaster removing your ability to communicate with customers and suppliers. A thorough understanding of the telecommunication requirements of your organization will help you to design the most cost-effective plan to protect against its absence.