Disaster Recovery: The Fed is About to Change the Rules

RFG believes the draft white paper issued by federal agencies, motivated by the September 11, 2001 attacks, will be the basis of new legislation or regulations to which all U.S. financial institutions will need to conform. IT executives in the financial services sector should immediately evaluate, and modify where necessary, their disaster recovery and business continuity plans in light of this draft. Financial services IT executives should expect to expand their commitment to and expenditures for business continuance plans in 2003, in order to support what will likely be mandated expenditures. Moreover, all IT executives should remain aware of the status of pending federal regulations, as these tend to become propagated across the entire IT industry, and plan accordingly.

Business Imperatives:

• If the definitions, audit points, and requirements enumerated in the draft white paper are, in fact, released as formal regulations, the impact upon disaster recovery (DR) and/or business continuity plans (BCP) could be enormous. While the suggested requirements for business recovery and process resumption times are expressed in terms of two to four hours, equally threatening is the suggestion that qualifying enterprises may have to maintain an alternative backup site that is 200 300 miles away from the primary site. IT executives should stop any work and cease spending on any efforts that are currently underway to construct a non-conforming backup site until the requirements of this proposal are sorted out and finalized in the form of official governmental regulation.
• The draft white paper does discuss the potential for utilizing outsourced business continuity providers. Rather than construct an entirely new backup data center hundreds of miles away from the current primary site, IT executives should consider a number of alternatives while reviewing and revising their DR/BCS plan.
• If the draft white paper is a good barometer into what the federal agencies are considering, the entire DR/BCS landscape could change dramatically, with relatively short lead times to build and implement new (or revised) plans. IT executives should take steps now to minimize the impact of these potential new regulations, keep their options open, minimize expenditures, and position their enterprise to become compliant in as short a time as possible.

Federal regulatory agencies have been meeting with financial services industry representatives and IT vendors in the aftermath of September 11, 2001, in an effort to develop sound practices that strengthen the resiliency of the U.S. financial system. As a result, three federal agencies, the Federal Reserve System (FRS), the Office of the Comptroller of the Currency (OCC) and the Securities and Exchange Commission (SEC), assisted by the Federal Reserve Bank of New York and the New York State Banking Department, have issued a draft white paper. The terms outlined by the paper could significantly affect IT executives and their disaster recovery and/or business continuity plans.

The agencies released the white paper in August 2002 and solicited comment from enterprises in the industry until October 22, 2002. Several dozen public agencies, private enterprises, and governmental entities have responded with comments many of which emphasize the necessity of these agencies to be very explicit and provide high degrees of specificity in their definitions, requirements and forthcoming regulations.

Entitled “Interagency Concept Release: Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System,” the full text of the white paper, and comments submitted, can be found on the web sites of any of the agencies. The SEC link is https://www.sec.gov/rules/concept/34-46432.htm. The document number is Release No. 34-464432, File No. S7-32-02.

The concepts in white papers of this sort tend to evolve into federal edicts, laws, or regulations. Because of the impact and magnitude of the recommendations made in the white paper, RFG believes IT executives and their partners in the corporate legal office should remain vigilant in monitoring formal actions of the agencies and be prepared to implement resulting regulations as they emerge.

As will be seen, the definitions contained in the draft white paper are sufficiently vague. It is possible, then, that any enterprise processing high value transactions, trading in securities, or materially affecting the economy could conceivably become included or affected by the regulations in the future. In that context, all IT executives, not just those in the financial services sector, are urged to maintain awareness of these regulatory activities. Since some of the activities urged could be considered “best practices” or “good business sense” anyway, it behooves IT executives to begin to adhere to the spirit, if not the discrete distance and time requirements, of these proposed regulations.

Definitions

• Wide Scale, Regional Disruption: a disruption that causes a severe loss of power, telecommunications, transportation, or other critical infrastructure component across a metropolitan or other geographic area and any economically integrated adjacent communities.
• Critical Markets: markets that provide the means for banks, securities firms, and other financial institutions to manage the cash and securities positions for themselves and their customers. Critical markets are those markets for federal funds, foreign exchange and commercial paper, and government, corporate and mortgage-backed securities. This definition seems to include all enterprises involved in financial services.
• Core Clearing and Settlement Organizations: those organizations that provide critical clearing and settlement services for financial markets and “large value payment system operators.” While determining which enterprises qualify in this context, the agencies discuss various measures (for example, market share, dollar value thresholds) with which to allow an enterprise to categorize itself. Since any resulting regulations or laws may well provide for multiple tiered plans (and IT requirements) based on the size or market impact of an enterprise, the metrics may well have a important bearing on how an enterprise categorizes itself and what level of resiliency is required of it. Additionally, these organizations provide settlement services in sufficient volume or value to present systemic risk in their sudden absence, and for whom there are no viable immediate substitutes.
• Systemic Risk: the risk that the failure of one participant in a transfer system or financial market to meet its required obligations will cause other participants to be unable to meet their obligations when due, causing significant liquidity or credit problems and threatening the stability of financial markets.

With these definitions in mind, the agencies estimate that the top 15-20 banks and the top five to 10 securities firms “and possibly others” would be impacted by the resulting regulations. RFG believes that the phrase “and possibly others” leaves the door open for the agencies to effectively include and affect most enterprises in the financial services sector, regardless of size. There is another implication here. There is the potential that insurance providers for those enterprises outside the financial services sector will require implementation of any resulting regulations. Insurance carriers may insist on compliance with these regulations as part of due diligence clauses for directors’ and officers’ liability policies, to ensure the continued viability of that enterprise in the face of a wide scale, regional disruption.

In response to the events of September 11, 2001, and the subsequent impact upon IT and the financial sector, the agencies are attempting to assure the resilience of critical U.S. financial markets when confronted with wide scale, regional disruptions. In addition, the white paper addresses question of the validity of the backup data (restore tapes, for example). It also questions the efficacy of a disaster recovery plan that involves moving people and IT goods with a disabled airline system, and the ability to have adequate numbers of trained IT staff at some remote backup site.

Scope

The white paper identifies three business continuity objectives:

• “Rapid recovery and timely resumption of critical operations following a wide scale, regional disruption;
• Rapid recovery and timely resumption of critical operations following the loss or inaccessibility of staff in at least one major operating location; and
• A high level of confidence, through ongoing use or robust testing, that critical internal and external continuity arrangements are effective and compatible.”

In an itemization of “Recovery of Critical Activities,” the paper defines those activities to include

• Completing, clearing, and settling large value transactions;
• Meeting material end-of-day funding and collateral obligations;
• Managing and communicating material open enterprise and customers’ risk positions;
• Performing all support and related functions.

The agencies indicate the expectation that the recovery and resumption of these critical activities will be accomplished within one business day. Subsequent discussions in the white paper, however, indicate that the agencies are thinking of imposing recovery time targets of no longer than four hours after an event, and a resumption time target of no more than two hours after an event.

The language “all support and related functions” used, the lack of definition of what constitutes “large” or “material,” and the mention of four and two hour recovery resumption targets are indicative of how all-encompassing or broad subsequent regulations may be. They also provide a glimpse of what stringent requirements may well be mandated in the months ahead.

Affected enterprises are expected to maintain sufficient “out-of-region” IT capability (out-of-region telecommunications capacity, data, equipment, staff, etc.) to meet the recovery and resumption objectives. The agencies offer suggestions that an enterprise may utilize its own remote, backup or outsourced facility. Emphasis is placed on the necessity of having two sites with totally separate and independent labor pools and infrastructure components (electric power, water supply, telecommunications, and transportation) such that no event can affect both sites simultaneously. Enterprises are expected to periodically and routinely test DR/BCS plans for the required volume capacity, connectivity, and functionality. Additionally, enterprises are expected to work cooperatively to conduct cross-organization tests to assure compatibility of plans and strategies across multiple enterprises in critical markets.

While urging enterprises to design (or re-design) DR/BCS plans as soon as possible, the agencies articulate the expectation that development of plans will be complete within six months after regulations are issued. The agencies also maintain that out-of-region backup resources will be in place within one year after issuance of the regulations.

Audit Points

The agencies clearly state their intentions to enforce compliance with regulations (when published):

• To require affected enterprises to implement the resulting recommendations or practices;
• To implement the practices in forthcoming regulations;
• To review (audit) enterprises’ DR/BCS plans for reasonableness, risk management, and an assessment of the probable effects of a disruption of one enterprise’s activities on the financial system as a whole;
• To review how an enterprise identifies its own critical activities, sets recovery and resumption objectives, and builds plans to meet those objectives;
• To analyze an enterprise’s recovery time, resumption objectives, and implementation schedules in light of market and peer expectations;
• To review an enterprise’s DR/BCS testing plans and test results to confirm that the enterprise is able to manage its business risks in the event of a wide scale, regional disruption.

From this list, it becomes patently clear that construction and periodic testing of a DR/BCS plan is no longer optional, but about to become mandatory. The agencies are dramatically expanding the breadth and depth of federally mandated DR/BCS plans. Further, requirements for a federally approved plan will be embedded in federal regulations, with the agencies auditing affected enterprises to ensure compliance with these regulations.

Finally, the agencies pose the rhetorical question- what will be the minimum distance required of affected agencies between primary and backup facilities? The agencies suggest that a 200-300 mile separation of such facilities may be required.

Alternatives

In short, IT executives should not rush out and construct new backup data centers to fulfill federal requirements, even if all the specifics were known. RFG believes IT executives should look first at exploiting the capabilities of another data center within the enterprise, perhaps one used by another line of business (LOB). Expanding an existing LOB data center to meet processing requirements of the corporate data center may be substantially less expensive than starting from scratch. There may be several benefits to be had by expanding one (or both) data centers so that they have parallel configurations and each can provide disaster backup support for the other. Further benefits may be accrued if both data centers can load share or load split.

Similarly, if there is no existing LOB data center that can be expanded, there may be facilities at a branch office that could be expanded to fulfill the business continuance role.

With the agencies discussing partnerships and cooperation, IT executives should investigate co-location facilities with shared infrastructures to mitigate the costs of building an entirely new, self-sufficient data center.

Any expansion of remote facilities should cause an IT executive to examine, once again, the advantages and disadvantages of contracting with an outsourced DR/BCS provider such as IBM Business Continuity and Recovery Services, SunGard Data Systems, and others.

RFG Recommendations

In light of the probability that new regulations will be forthcoming early in 2003, IT executives should position themselves and their enterprises to comply with these regulations as non-disruptively and inexpensively as possible. In the interim, IT executives should:

1. Eliminate or minimize any monies currently being expended to build remote data centers. A temporary “hold” is in order while the requirements are solidified.
2. Notify and seek proactive support of the enterprise’s senior management. Even the agencies suggest that involvement and support of senior management will be required to implement these substantive and comprehensive regulations.
3. Begin to identify critical activities that could be covered under the new regulations. Beyond data centers, these could include call centers and any other facility, operation, or process required to conduct business.
4. Begin an assessment of the current state of existing DR/BCS plans. If an effective DR/BCS planning team is not yet in place, this would be the opportune time to form a multi-disciplinary team and establish clear mission objectives, approval processes, reporting structures, etc. Having these administrative tasks completed before stringent deadlines are in force can save time and facilitate compliance.
5. Build an inventory of enterprise-wide data processing sites with a view toward identifying which one(s) could be expanded to fulfill the secondary, backup role in this expanded environment.
6. Initiate discussions with alternative vendors such that the DR/BCS team is armed with information necessary to compare alternatives.

RFG believes that the federal agencies are intent upon seizing the lessons learned from September 11, 2001, and increasing the resiliency of the nations’ entire financial industry and its infrastructures. While many enterprises had previously adopted sophisticated DR/BCS strategies, some worked as designed, others fell short of meeting expectations, and still other enterprises found themselves bereft of plans to cope with what seemed unimaginable. The broad, pervasive, all-encompassing scope of the draft white paper could cause alarm bells to go off all over the industry. While the specifics of the IT infrastructure requirements are debated over the coming months, IT executives should expect – and plan for increasing their commitment to and resources expended on DR/BCS in 2003.