9

WORK AREA RECOVERY PLAN

Getting the Office Up and Running

Ya gots to work with what you gots to work with.

—STEVIE WONDER

INTRODUCTION

Work area recovery means preparing workspace in which to temporarily recover business operations. It usually involves offices, but it could easily encompass call centers, retail space, or factories. Whatever its function, a plan is needed to establish a place for people to work. Every day that your business is out of service is another day where:

Your competitors’ sales force is active while yours is idle.

Bills are not sent to customers nor is there a place to receive funds.

Bills are not paid and potentially become overdue.

Customer orders are not received or processed, potentially leading to cancellation.

Some companies focus exclusively on recovering their IT operations and never think about applying the same effort to the people who are to use the IT services. Recovering one without the other will not restore service to your customers. Office space will not be recovered within the recovery time objective (RTO) without a tested plan.

This plan enables key personnel (such as the sales force and the tech support call center) to work during a disruptive event. Creating and testing this plan demonstrates corporate responsibility while simultaneously protecting your business reputation. A plan that promptly restores service minimizes the disruption of revenue and also protects customer relationships.

Work areas are more prone to disasters (small and large) than a data center. Rivers breach their banks, water pipes burst, small fires are turned into large water hazards by the fire department, and on and on. In a blizzard, the data center keeps chugging on, but the offices stop as employees are unable to come in to work. In a labor action, people cannot get into the workplace, but the IT system runs on and on. So, planning for a loss of work areas is more practical and has a more likely payback than planning for an IT loss.

Customers are sensitive to supplier interruptions. Many long-term contracts require that the supplier demonstrate that a tested plan is in place. Everyone has at one time or another been disappointed by the failure of someone to deliver promised goods or services when ordered. For most companies, their office workers are their “face to the customer.” When no one answers the phone during business hours, customers become very concerned about that company’s reliability and look elsewhere for goods and services.

A challenge in working in recovered facilities is security of company information. Computers are always a target for anyone looking to make quick cash, but in a recovered site, confidential company documents must also be safeguarded. Many people may not be known to others, enabling curious strangers and criminally minded people to enter the facility and wander around.

The easiest team to recover is the executive staff. Some of them are already in the Command Center. The rest can work out of a hotel conference room from which they can use company cell phones for outbound calls. In many cases, their key customers already know their company cell phone number for inbound calls. Furthermore, the hotel can provide an online connection.

Other departments, such as Customer Service, are heavily dependent on inbound telephone traffic. These groups must recover in a dedicated location so that the inbound telephone connections can be quickly changed. Like other transaction-based teams, the sales crew will require ready access to its data.

WRITING A WORK AREA RECOVERY PLAN

The first question to answer is, Why are you writing this plan? What problem are you solving? Is it to keep the Sales department always functioning? Is it a regulatory or contracted requirement? Whatever it is, write it down before starting your plan. This reason anchors the plan development and prevents it from drifting off target. It will vary from just someone to answer the phone to maintaining full customer service even during a disaster.

Based on this reason, select a strategy that describes the site where you will recover (in a company site, in a rented facility many miles away, or in local facilities acquired at the time they are needed). It details how many seats must be immediately available and, sometimes, how many more should be ready within an additional time period. Most important, it will identify the RTO for the facility to be ready.

The key elements of the plan are selecting a recovery site, determining who should be there, equipping the site, activating the site, and operating it during a company disaster. In a company-wide emergency, alternative work arrangements may be necessary, such as employees working from home or in scattered groups where facilities can be secured.

An important piece of the strategy is to enable employees to work from home. This requires advanced planning to provide them with Virtual Private Network (VPN) access. A VPN enables them to securely connect to the data center (or recovered data center) and continue their online work. To do this, companies must create a policy governing people working from home. Working from home should never involve company confidential material.

Recovered departments usually follow a hierarchy of importance. For example:

Executive management, legal team, corporate communications, (core) Human Resources team—typically recovered near the Command Center and the disaster site

Human Resources personnel (the remainder at the office recovery site)

Customer contact team (Sales)

Accounting (cash flow)

Company operations

Write a separate plan for each type of work area to be recovered, such as office, call center, warehouse, factory, and retail. Each will likely have its own recovery site and its unique requirements.

A Place to Work

Building the plan is a team effort. Include representatives from each of the critical departments during site selection and work area layout. If the departments don’t like the layout, they won’t use it. Their ideas and last-minute requirements are essential to make the site selection a success.

Review the vital business functions identified by the Business Impact Analysis with each department. Identify how many people need to be recovered from each team. Some will need to be recovered immediately and some later. Be sure to include space for the supervisors and managers necessary for the team to function. Most recovery sites are wide open spaces, so carefully assign the few offices available. Executives will expect individual offices.

For each person, estimate 70 to 80 square feet of space. Multiply this times the number of people, and you will have the approximate minimum floor space requirements. This amount of space also accounts for hallways, conference rooms, reception, break room, and common areas. You can cram people into tighter space but productivity, already constrained by the basic work environment, will be further reduced at a time when more is needed. Sometimes the limiting factor is the local building and safety code for occupancy limits and mandatory ratio of sanitary facilities to personnel.

To stretch your limited assets, consider changing some departments to night hours. In this approach, customer-facing team members, such as Customer Service, Technical Support, or Sales, are in the seats during daytime hours, while Accounting, Human Resources, etc.—team members who are internally focused—work in the same spaces at night.

There are three kinds of “seats” that can be set up for recovery. “Hot seats” are fully equipped and ready to go. These are the most expensive to set up and maintain. “Warm seats” are missing some of the equipment, usually the personal computer and telephone. “Cold seats” are floor space and may or may not have a table and chair. This space may be set aside for team members who can wait several days before recovering.

In general, a work surface should be 36 inches wide and 24 inches deep. This provides sufficient depth for the PC and keyboard and some space to the side for shuffling papers and writing. Of course, a comfortable chair to accompany the worktable is important. Adequate lighting either from desk lamps or overhead light (preferred) is essential. Take care that data, telephone, and power cords are carefully routed to avoid tripping anyone.

To allocate scarce resources in a chaotic time, establish a restoration priority for your recovery site. A restoration priority ensures that the recovery team is always focused on the highest-value actions for that moment. This is as simple as a sequence of what to do next. Be sure this priority list is well communicated to the teams.

Make sure to carefully consider the environmental control needs for people and technical equipment. The facility must be able to hold the temperature and humidity within an acceptable range. This avoids problems with equipment and with people becoming ill in the middle of a company disaster. How important can this be? Imagine someone trying to recover offices in an empty warehouse in the middle of winter. Everything else can be there except the ability to hold heat within the work area.

Other aspects to consider in selecting a recovery site include:

Easy access to airports, major highways, or public transportation

Adequate parking

Loading dock for deliveries

Nearby lodging and food establishments

Storage space for preprinted forms and reference materials onsite for ready use

The key to the success of the facility selection is the careful management of end-user expectations. Emphasize that it provides a basic work area with limited service. As the plan is tested, each department will better understand the situation and become your champion to finance facility improvements. The size, location, and desk setup will all drive the solution’s cost. The executive approval process often cuts back on user requests.

Recovery Options

The criteria used for selecting a site for recovering your workers is similar to that of selecting an IT recovery site. It should be far enough away to avoid damage from the same incident. Beyond that, it can be quite close or two states away. If the recovery will be in a commercial site, then it may be several states or more away. Commercial recovery sites are sprinkled around the country and the nearest available site may be across the country. The first company to declare a disaster has its pick of sites. In a wide-area disaster, the nearest available site may be far away. For this reason, when a hurricane strikes, some companies declare a disaster immediately. Figure 9-1 lists the issues you need to consider when evaluating recovery options.

DIFFERENT COMPANY SITE

Using a different company site is a simple way to go. This site should be close enough for people to drive to it. If it is also used for IT training classes, then it can be quickly converted to a recovery site by canceling the classes and reimaging the computers. Using a company site means you know it has an active network and telephone connections, the security is already in place, and you can pre-position materials for emergency use. For the desktop PCs, you might use the last generation of units that you were going to scrap.

Using a company site brings the risk that some executives will try to use it for an operational activity. When a disaster strikes, it will be difficult to kick them out. Therefore, closely guard the facility’s uses. Do not dress it up so much that it is too attractive to someone. Its layout should be dictated by how it will be used and is not designed for aesthetics.

CONTRACTED HOT SITE

A contracted hot site is the “least-grief approach” because you pay someone to take on all the maintenance. The terms of the agreement depend on the vendor and the level of service you hire, but it typically includes test time and a set number of seats in a recovery. Any variation from the standard desktop and single line telephone must be coordinated with the vendor. These sites are close to public transportation and already have arrangements for local lodging and food.

Contracted hot sites can be expensive. Testing time must be reserved far in advance and may be preempted if another customer declares a disaster and occupies the space. Testing time may also require an additional fee. Finally, in a disaster, the closest recovery facility may be occupied and the nearest available seats several states away.

MOBILE RECOVERY EQUIPMENT

Mobile recovery equipment comes to the disaster site. These are expandable trailers that contain almost everything needed in a disaster site. Each trailer includes its own generator, telephone switch, and a satellite uplink for communications. When a disaster is declared, the trailers are pulled to the customer site and activated. If local electricity and data network connections are available, then so much the better.

An immediate advantage to this approach is that all employees sleep at home. This enables a high level of employee participation (depending on the type of disaster). In an incident such as a structural fire, using trailers provides an onsite presence that may be comforting to customers.

FIGURE 9.1. Recovery options.

A trailer’s usefulness may be limited in a wide-area disaster like a flood. If the company’s building is inundated with flood water, so is the parking lot, so the trailers must be parked somewhere else. If the company lacks distant property, then a site must be rented. Still, employees can likely drive to the trailers and back home at night.

SCRAMBLE RIGHT AFTER THE INCIDENT

Some companies feel that the local real estate situation is such that buildings with adequate space and facilities can be found on short notice. Then, the workplace is set up with a damn-the-cost speed. Overall, the company then saves the annual expense of subscribing to recovery trailers or to a hot site.

The first problem with this approach is that it leaves all planning to the point of incident, when there is so much to do. Without a test site, the plan cannot be validated or the team members adequately trained. Second, it ignores that more is needed than four walls and a roof. External data and telephone connections are required, as well as properly configured desktop equipment. Finally, it underestimates the time required to settle the real estate details even if everyone is pressing for an immediate resolution.

Employee Notification

Immediately after a disaster is declared (an incident that activates this plan), the facility preparation crew is notified. In most cases, this is the team leader for each recovery team. This crew opens the facility and prepares it for the recovery teams. If the recovery is in a commercial site, some of this work may be done by the service provider, based on terms of the service agreement.

The crew is notified by the company’s automated notification “telephone blast” system, which sends a notification to every person on a particular list. (Alternatively, the slower and less reliable telephone call tree may be used.) This crew includes team leaders for:

Security

Facilities

Each supported department

Onsite IT support

Onsite telecommunications and network support

If there was an IT disaster as well, then there will be a time delay before the IT systems are usable and can be used. If so, the main crew for the recovery site should be alerted through the automatic notification system (or other process) to arrive about 24 hours before all systems are scheduled to be recovered. This minimizes idle time at the site waiting for IT application availability.

Tools to Work With

A personal computer with a network connection and a telephone are the primary tools provided to an office worker. The first challenge is to provide the equipment. Keep spare units onsite ready to exchange in case of a failure. In a chaotic recovery, there is no time to wait for a repairperson.

Work together with each recovery team to develop a “standard” workstation layout. This will ease unit setup and IT support. A standard unit will also make it easy to move equipment between the teams as the recovery team makeup shifts.

Each department must create checklists for activation of their teams. These lists will ensure that the required materials are stored in the facility for ready use. They will also provide a list of items to monitor for updates, such as procedure manuals, quality checklists, and routing forms.

DESKTOP PCS

With an unlimited budget, a quick trip around town can collect enough PCs to support all of your recovery sites. The challenge is to load the company’s desktop PC software image onto those devices. This can require an hour per unit (assuming that a copy of the image was maintained at the recovery site). If there are hundreds of units to load, the process can be quite tedious. The software image contains the operating system and its configuration settings, specific device drives for the onboard components, information security settings, and copies of all standard software.

If the PCs being used are not the exact same model as the company standard, then a software image must be created for each model (or variation on a model), potentially requiring several hours. A new image is required to ensure that all of the necessary hardware drivers are installed and configured, as well as all security settings installed. If the emergency purchasing process brought in many different models, then the problem grows.

Some companies bypass the PC imaging issue by using a virtual environment for their desktops. Virtual environments can run on any typical PC and configuration. Using this technology, the PC’s “desktop” is a session inside of a computer room server. Once that server is recovered (and the user’s data is restored), employees will see their electronic desktop as before. In essence, the PC runs a browser into the server, so little local configuration is required.

As an interim to the data center recovery, many companies establish a file server at the recovery site. This unit provides a local “public disk” for file sharing and local reference information. It can also be used to hold the latest software “image” and any other files that would be locally useful in a recovery. A maintenance item is to keep the information on this device current.

TELEPHONES

Few office workers can complete their daily tasks without ready access to a telephone, with its quick access to coworkers, customers, and suppliers. Given the close working quarters in the recovery site, speaker phones are impractical. Therefore, each desk should have a hands-free headset.

Some workers, such as those in Customer Service, depend heavily on inbound telephone traffic. They will require an Automated Call Director (ACD) device to route incoming calls to the proper place. (This is another reason for not allowing people to sit anywhere they wish.) Outbound callers can supplement the recovery center’s phones with company cell phones. This will also free the facility’s telephone trunks for inbound calls.

Because everyone working in the facility is new to the building, provide a preprinted telephone directory to each seat (but do not print it until the center is activated). This will be based on previous seating assignments and not on who actually shows up.

PRINTERS

There are times when a printed document or label is essential. Each department must identify its printing requirements in terms of types of printers, volume (so adequate printer materials and paper stock can be stored in the center), and workstations each must connect to. This may lead to requirements for special printers, which then leads to the requirement to store printing supplies (toner, ink cartridges, and ribbons) onsite.

The other half of printing is the material to print on. This could be special sizes or colors of paper, multiple part forms, and stickers. An alternative to preprinted forms is a template on which the employee enters the information. A laser printer then prints the completed form. This avoids the need to stockpile expensive preprinted documents that eventually become obsolete and must be refreshed. Printing supplies should be adequate for as long as it takes to order a resupply plus at least one day. These supplies will age and must be rotated back for use in the primary facility over time. The more specialized these materials are, the longer that it may take to obtain them.

A corporate companion to a printer is a shredder. This tool reduces the chance that confidential printed company information can be retrieved from the trash. The shredder location should be marked on the floor plans and placed somewhere to minimize its contribution to the noise level.

REFERENCE MATERIALS

Many office workers use reference materials in their work. This includes physical documents as well as electronically stored data, from sales tax tables to a catalog of industrial suppliers and telephone numbers for local trucking companies. Each department should include on its activation checklists the reference materials necessary to function. There may be an opportunity to consolidate some of these documents for fewer copies. The reference materials held in the recovery facility should be provided and maintained by each department.

Some of the company’s reference materials may be confidential “vital records.” They may include financial information, customer data, or even the health records of employees. In each case, the vital records must be protected from internal compromise just as they would be in the normal company offices. This may require locked doors or locked cabinets in the work areas. Where possible, obtain or convert reference documents to CDs or obtain them via the Internet. This is another point in favor of an onsite recovery facility file-sharing server.

Provide reference material detailing the recovery facility which:

Explains the layout of the facility.

Shows the arrangement of departments in the facility seating.

Includes a telephone number chart for the recovery area (especially valuable if each of the seats is designated for a specific person).

Collocate Interactive Teams

Selecting who sits where can be emotional for some people. Some people wish to sit by windows, others away from the door. Bring in representatives from each department that will work in the recovery center. Some groups naturally work closely together. They pass physical documents or exchange information on a routine basis. These teams should be located adjacent to each other. Other teams, such as the company legal department, require privacy and value isolation. When laying out your recovery area, keep in mind the specific needs of each team. A carefully designed and executive-approved floor plan can minimize arguments and political posturing.

Once the seating is settled, create signs for each of the work areas. When the recovery team begins arriving at the facility, its members will need to know where to work. If this information is hard to find, then people will start sitting wherever it strikes their fancy, disrupting all of the careful planning. To reduce this confusion, install lots of signs. You should:

Label every door in the building.

Label every cabinet as to its contents.

Label every desk in the recovery center as to the department using it and those dedicated desks that no one else should use.

Suspend signs from the ceiling to indicate the location of each department.

Telecommunications and Data Systems

A recovered work area site must have a data connection to the recovered IT site. The bandwidth must be adequate to support the number of workstations at the site. In addition, telephone service must be provided to the desktop. Adequate inbound service can be an issue. Inbound telephone lines must be rerouted to the recovery site for the duration of its use. Outbound traffic is less of an issue as cell phones can be used.

The difficult parts of the plan to execute are moving the inbound call lines from the damaged work site to the recovery location. This includes fax machines, inbound local telephone lines, and inbound toll-free numbers. The telecommunications support team must work with the telecom provider in advance to understand the steps. As always, the question is how long will it take and will it still meet your RTO? Specific information may be required, such as the phone numbers involved, the service provider’s contract number, and whom to call to request this immediate service.

Several important telephone features to include are conferencing and voice mail. These functions are common in offices, and their absence will be an employee productivity concern.

The recovery site activation team sets up signage so that everyone knows where to go, where to sit, and where to secure supplies. They verify that each workstation, telephone, printer, fax, copier, and shredder is ready to go. They pull from storage any reference materials and special forms so that each desk is ready to begin when the recovery team arrives.

Security

Activating a recovery facility transforms it from an empty or lightly occupied building into a hub of activity. It will attract a lot of attention. Few people in the company can recognize every employee. There will be many strange faces in the crowd. This is an opportunity for strangers to walk in and leave with company property under their arm. The recovery site requires both physical security to protect assets and information security to protect its data.

An easy way to control access is to use the same key card access as used in the primary facility. This will require adding employees to the local security server. If you keep a backup copy of the primary facility’s key card access list, it can be added locally along with the security zones into which they are welcome. Otherwise, post a security guard at the employee entrance. No one enters the building without a card key and a company ID card or without being escorted by a company employee.

In addition to security for the building, there is also security for company documents created during the recovery. This might be anything pertaining to legal compliance, call logs, and customer information. Ensure there is adequate shredder support onsite in sound-deadening rooms.

Another issue is that the workstations are close together. While in the primary facility there may be sound-deadening walls and cubicle barriers, everything is now wide open. Ask everyone to speak quietly since others may be on the telephone nearby. For security, be careful about what is spoken out in the open.

TESTING

Before a plan can be declared as operational, it must be tested. This trains the participants, as well as demonstrates that the plan can meet the RTO. Over time, be sure that each department has a team participating and observing the tests. This familiarizes them with the recovery site, the area around it, recovery processes, trade-offs, and limitations at the site.

Recovery site testing is often conducted in small groups. It is very difficult to swing inbound lines into the center, but teams that primarily use outbound lines can test the plan by working onsite for several days. This also works well for internally focused (noncustomer-facing) teams.

MAINTAINING THE RECOVERY SITE

If your company is providing its own recovery site, it will require regular maintenance. The revision level of the recovery site must match that of the site it is supporting. As the organization changes or as the emphasis of the business evolves, so must the recovery facility. This eliminates one more distraction when recovering the facility in this site. Maintenance should be performed at the following intervals:

Annually

Executives must determine if the Business Impact Analysis has materially changed from the previous year.

IT validates that the desktop computer hardware is adequate and still meets corporate standards.

Telecom support team reevaluates if the telecom arrangements need to be changed.

Quarterly

Each department reviews its desktop requirements with the IT team, which ensures that the desktop computers in the recovery facility still meet everyone’s needs.

The IT team verifies that the software in the recovery units is adequately patched with bug fixes and security patches.

Periodically conduct tours of the facility to acquaint employees with where they may work in an emergency and familiarize them with the facility’s layout. This is an opportunity to remind everyone that this facility’s value is to always be ready in an emergency and that it is not available for use as a “production” facility.

One idea is to “brand” the facility to help place its recovery function in people’s minds. For example, call it the “People and Infrastructure Recovery Facility (PIRF)” or “The ORB (Office Recovery Building).” Another idea is to create an employee information brochure that describes the facility’s capabilities. The brochure includes a map and driving directions to the recovery site.

Conclusion

Every company depends heavily on its IT department. Its prompt recovery is critical to the company’s continued viability. However, some companies overlook the fact that recovered computer databases are of little value if no one is available to use them. A work area recovery plan is an essential complement to any IT recovery plan.

Recovering a workplace is not trivial. People need an adequate place to do their jobs, with a minimum of environmental distractions. Adequate data lines must be installed in advance, as they cannot be prepared on short notice. Similarly, telephone service must be in place so that when the incident occurs, the inbound lines can be quickly redirected from the damaged facility to the recovery site.

Selecting a recovery strategy will likely follow the same strategy as the IT recovery plan. It may be another company site or a third-party facility, or the decision may be to bring fully equipped trailers to the disaster site. Each approach has its advantages and disadvantages in terms of convenience, cost, and capabilities.

Recovery sites are busy places. It should be very easy to find your way through the recovery site. Signs should be everywhere informing recovery team members where to go, where to find things, and where not to go.

Be sure to test everything at the recovery site as soon as you arrive. The RTO is measured from the time of the incident, not from when you were alerted. Early detection of problems enhances the chances of a timely recovery.

Finally, if you have designated a company site for recovery, never let your defenses down for a moment. Name it something that positions it in people’s minds as providing value as the recovery site. Never let it be used for even a small production function. Regularly scheduled testing helps to keep it active enough (and annoying enough) that production functions look elsewhere for a quieter setting.