The FDIC's Failed Bank Data Services Project

This is the accessible text file for FDIC OIG report number AUD-17-003 entitled 'The FDIC's Failed Bank Data Services Project' .

This text file was formatted by the FDIC OIG to be accessible to users with visual impairments.

We have maintained the structural and data integrity of the original printed product in this text file to the extent possbile. Accessibility features, such as descriptions of tables, footnotes, and the text of the Corporation’s comments, are provided but may not exactly duplicate the presentation or format of the printed version.

The portable document format (PDF) file also posted on our Web site is an exact electronic replica of the printed version.

Office of Audits and Evaluations

Report No. AUD-17-003

The FDIC's Failed Bank Data Services Project

March 2017

Executive Summary

Why We Did The Audit

The FDIC, as receiver for a failed financial institution, acquires control of the institution’s records and generally must maintain them in accordance with the Federal Deposit Insurance Act for at least 6 years. Maintaining these records is critically important as they are used by various internal and external stakeholders, including outside counsel, to support such activities as investigations, litigation, customer service, tax administration, research, and asset sales. The FDIC’s Failed Bank Data Services (FBDS) project established a new contract and system to facilitate this important task.

The objective of this performance audit was to determine (1) the status of the project, including progress and costs in relation to goals, budgets, and milestones; (2) factors contributing to the project’s progress; and (3) significant issues or risks that must be addressed to achieve project success.

Background

In November 2008, as the recent financial crisis was unfolding, the FDIC entered into a contract with Lockheed Martin Services, Inc. (Lockheed) to provide an information system and services—collectively referred to as Data Management Services (DMS)—that included collecting, storing, and searching data from failed financial institutions. The FDIC initially anticipated that the DMS contract with Lockheed would end on June 30, 2015, but for reasons discussed later, the FDIC extended the contract until October 31, 2016. On December 31, 2014, the FDIC awarded a $275 million contract to CACI-ISS, Inc. (CACI) to build and host a new system—FBDS—that would replace DMS. The FBDS contract also called for CACI to transition all legacy failed financial institution data and services from DMS to its new system. As of June 30, 2016, the FDIC had spent $20.4 million for services under the FBDS contract.

The FDIC’s Division of Resolutions and Receiverships (DRR), Legal Division, and several other divisions and offices are involved with the project, and, therefore, management and subject matter experts from these organizations represent key stakeholders. The Division of Administration handles contracting matters associated with FBDS, and DRR oversees contractor performance. An Executive Board and a Steering Committee comprised of stakeholder management are the primary governance bodies for the project.

Audit Results

At the time of our audit, the FDIC had a number of significant achievements associated with the FBDS project. Specifically, the FDIC had:

- established the initial FBDS infrastructure and implemented security controls that were deemed adequate to begin using the system;

- transitioned a large amount of data from the DMS system to FBDS;

- transferred data collection, processing, and litigation support services from Lockheed to CACI;

- used the FBDS system to process and load data for the unanticipated failure of a large, complex financial institution;

- conducted a user satisfaction survey for the FBDS system and services, the results of which indicate a generally favorable view of FBDS; and

- realized a significant reduction in data storage and related costs.

With respect to milestones and costs, there was a delay in implementing certain system capabilities and transitioning data from DMS to FBDS. The transition-related schedule delays caused the FDIC to extend the DMS contract into 2016. As a result of the contract extension, and other challenges and unforeseen activities, the FDIC absorbed about $14.6 million more in transition-related costs than estimated. Overall, total transition-related costs remained less than what was originally projected when the FDIC’s Board of Directors approved the project.

The following factors impacted project progress and costs:

- FDIC personnel did not fully understand the DMS system and what was involved in transitioning the data and services to a new contractor, or communicate related requirements to bidders in a comprehensive transition plan as part of the FBDS solicitation.

- The FDIC did not always establish clear expectations in contract documents. For example, the FBDS contract did not include a comprehensive set of requirements or a clear timeline for implementing certain key capabilities in FBDS.

- For non-information technology development projects like FBDS, the FDIC has not implemented a formal project management framework to guide and structure project activities. The FDIC had also not included important project management-related plans as requirements and deliverables in the FBDS contract. We noted that DRR personnel used certain other tools and techniques to manage the project.

Those involved in managing the project cited technical challenges, and the unanticipated failure of the financial institution referenced earlier, as other factors that impacted the transition.

At the time we concluded our work, the FDIC had completed or was working to complete actions to mitigate risks for the project, including:

- expanding contract requirements to facilitate more effective end-of-contract transition planning;

- establishing additional governance mechanisms;

- establishing a plan for validating that transition activities are completed in accordance with contract requirements; and

- formalizing FBDS-related contract oversight guidance.

Our report discussed additional steps to mitigate risks and provide greater assurance of project success, such as enhancing project management; determining the desired capacity of the FBDS system to handle additional data; evaluating FBDS processes for improvement opportunities; and establishing guidance for reviewing contract performance metrics.

Recommendations

Our office made seven recommendations to strengthen FBDS governance, project management, and contract oversight to reduce FBDS project-related risks going forward. FDIC management concurred with our recommendations and described planned actions that were responsive.

[End of Executive Summary]

Contents

Background

Audit Results

Status of the FBDS Project, Including Progress and Costs in Relation to Goals, Milestones, and Budgets

Transition-Related Goals Addressed

Other FBDS Goals Addressed

Remaining Work to Address Risks

Milestones

Transition-Related Costs

Factors Contributing to the Project's Progress

Understanding Project Scope and Requirements

Establishing Clear Expectations in Contract Documents Implementing a Project Management Framework

Other Factors

Addressing Issues or Risks to Provide Greater Assurance of Project Success

FDIC Corrective Actions Taken or in Process

Additional Risks Warranting Management Attention

Corporation Comments and OIG Evaluation

Appendices

1. Objective, Scope, and Methodology

2. Glossary of Terms

3. Abbreviations and Acronyms

4. Challenges Affecting Project Milestones 5. Corporation Comments

6. Summary of the Corporation’s Corrective Actions

Tables

1. Summary of Key FBDS Project Milestones as of June 30, 2016

2. Summary of Transition-Related Contractor Estimated and Actual Costs

Figures

1. The FDIC’s Governance Structure for the FBDS Project

2. FBDS System Overview – by Business Function

[End of Contents]

[FDIC OIG Letterhead, FDIC logo, Federal Deposit Insurance Corporation, Office of Inspector General, Office of Audits and Evaluations, 3501 Fairfax Drive, Arlington, Virginia 22226]

DATE: March 27, 2017

MEMORANDUM TO: Bret D. Edwards, Director, Division of Resolutions and Receiverships

FROM: Mark F. Mulholland, /Signed/, Assistant Inspector General for Audits

SUBJECT: The FDIC’s Failed Bank Data Services Project (Report No. AUD-17-003)

This report presents the results of our audit of the FDIC’s Failed Bank Data Services (FBDS) project.1 At the time of our review, the FDIC was in the process of completing the transition of data and services from its legacy system, the Lockheed Martin Services, Inc. (Lockheed)- developed Data Management Services (DMS), to a new established system, the CACI-ISS, Inc. (CACI)-developed FBDS. The FDIC is using FBDS to manage failed financial institution (FI) data for all legacy and new FI failures.

Footnote 1: For purposes of this report, the terms FBDS project and FBDS program are used interchangeably. [End of footnote]

The audit objective was to determine (1) the status of the FBDS project, including progress and costs in relation to goals, budgets, and milestones; (2) factors contributing to the project’s progress; and (3) significant issues or risks that must be addressed to achieve project success. To address our objective, we reviewed relevant status reports, contracting information, and project documentation, and interviewed FBDS project stakeholders and contractor personnel. We consulted A Guide to the Project Management Body of Knowledge (PMBOK® Guide) as a source for sound project management2 practices applicable to the FBDS project in conducting our work.

Footnote 2: 2 Certain terms that are underlined when first used in this report are defined in Appendix 2, Glossary of Terms. Appendix 3 contains a list of abbreviations and acronyms. [End of footnote]

We conducted this performance audit in accordance with generally accepted government auditing standards. Appendix 1 of this report includes additional details regarding our objective, scope, and methodology.

Background

When appointed as receiver for a failed FI, the FDIC acquires control of the institution’s records—both electronic and hard copy. The preservation of the FI’s records is critical to the FDIC’s obligation to conclude the affairs of the receivership. Such records include, for example, loan and deposit data, financial reports, email communications, file shares, suspicious activity reports, reports of examination, human resource records, and board of directors’ (Board) minutes. The FDIC is responsible for determining what data and documents belonging to the failed FI are records that need to be preserved.3 Maintaining these records is critically important as they are used by various internal and external stakeholders to support such activities as investigations, litigation, customer service, tax administration, research, and asset sales.4

Footnote 3: 12 United States Code (U.S.C.) § 1821(d)(15)(D). The Federal Deposit Insurance Act requires the FDIC, as receiver for a failed FI, to maintain the institution’s records (that are fewer than 10 years old as of the date of appointment) for at least 6 years after the date of appointment. [End of footnote]

Footnote 4: 12 U.S.C. § 1820(f). The Federal Deposit Insurance Act provides that any electronically stored data is deemed to be an original record for all purposes. Thus, the FDIC’s electronic database (currently FBDS) is deemed to contain original failed FI records and these records are admissible as evidence in all State and Federal courts or administrative agencies to prove any act, transaction, occurrence, or event therein recorded. [End of footnote]

Because each FI manages its data differently and many FIs use customized proprietary systems to manage their data, the task of collecting and converting failed FI data to a usable format is a significant challenge and expense for the FDIC. Further, the FDIC must be able to quickly collect and store data outside of the failed FI’s systems because acquiring institutions may not maintain all the failed FI’s legacy systems and data, and the failed FI’s information technology (IT) personnel may become unavailable within months of the failure.

Legacy Contract with Lockheed for Data Management Services

In November 2008, as the recent financial crisis was unfolding, the FDIC entered into a contract with Lockheed (DMS contract) to provide data management services. Under the terms of the contract, Lockheed was responsible for providing a standard method of maintaining failed FI data, including data collection, migration, conversion, cataloging, indexing, storage, security, and retrieval. DMS used iCONECT-nXT as its primary hosting, data preservation, and searching platform, but also used a Relativity e-discovery platform to maintain databases of a subset of litigation-related information. Under the terms of the contract, DMS was an outsourced information service, where Lockheed owned DMS and the FDIC owned the information stored in DMS.

At the time the DMS contract was awarded, the FDIC did not yet know the full scope of the banking crisis, including the massive volume of data associated with FI failures and the complexities of capturing and processing the data. Specifically, the FDIC anticipated that Lockheed would handle up to 45 terabytes of data from a projected 150 FI failures. By the time of the June 27, 2014 FBDS contract solicitation, Lockheed was managing approximately 900 terabytes of data from almost 500 bank failures. The FDIC initially anticipated that the DMS contract with Lockheed would end on June 30, 2015, but for reasons discussed later in the report, the FDIC extended the contract until October 31, 2016. As of June 30, 2016, the FDIC had awarded Lockheed 252 DMS-related task orders valued at $732.5 million and had spent $506 million against those task orders.

Contract with CACI for Failed Bank Data Services

In April 2014, the FDIC’s Board of Directors authorized the Division of Resolutions and Receiverships (DRR), the Division of Information Technology (DIT), and the Division of Administration (DOA), with the consent of the Acting General Counsel, to enter into a new contract to support the management of failed FI data. The intent of the contract was to leverage industry-leading technologies and, if possible, significantly reduce the costs associated with managing failed FI data.

Following a competitive process, the FDIC awarded a 10-year Basic Ordering Agreement (BOA) contract to CACI (FBDS contract) on December 31, 2014. The FBDS contract statement of objectives (SOO) required CACI to build and host the FBDS system, and to transition all legacy failed FI data and services from DMS to FBDS. Under the BOA and related task orders, CACI provides ongoing services such as data collection from new failed FI source systems, document scanning, processing, and business and litigation support. CACI loads the collected and processed information to a central data center accessible through a secure Website. CACI also provides infrastructure hosting services, user training, and help desk services such as phone support and research assistance. The FDIC has designated FBDS as an outsourced information system for security purposes.

The FDIC Board of Directors’ case for FBDS (FBDS Board Case) approved a not-to-exceed contract ceiling authority of $295 million. CACI’s winning bid for FBDS was approximately $75 million. The FDIC established a not-to-exceed value of $275 million for the BOA, which was $20 million less than the approved case amount. According to FDIC personnel, the $275 million provides contingency funding should there be an unanticipated increase in future FI failures above the estimates used in the FBDS contract. As of June 30, 2016, the FDIC had issued 430 task orders under the BOA, with a combined value of $126 million, and had spent $20.4 million against those task orders.

FBDS Governance Structure

Several FDIC divisions and offices are involved with the FBDS system, and, therefore, management and subject matter experts (SMEs) from these organizations represent key stakeholders of the project. Key project stakeholders include primary business users such as DRR Investigations, DRR Customer Service, and the Legal Division, among others. These entities use information from FBDS to investigate and pursue or defend claims, respond to failed FI customer requests, and support other litigation activities. In addition, FBDS information is used by the Division of Risk Management Supervision for enforcement action purposes and by the Division of Insurance and Research (DIR) for research purposes. Other key stakeholders are responsible for oversight of FBDS, including the DOA Acquisition Services Branch Contracting Officer who manages FBDS procurement activities, including task order issuance and modification; a DRR Oversight Manager (OM) who, along with 6 DRR Task Order Oversight Managers and 15 Technical Monitors (TMs) from DRR and other stakeholder divisions, provide contract oversight management; and the Chief Information Officer Organization, which certifies system security.

The FDIC established a governance structure for the FBDS project that incorporates key stakeholders mentioned above. Figure 1 provides an overview of the FDIC’s governance structure for the FBDS project.

Figure 1: The FDIC’s Governance Structure for the FBDS Project (organization chart)

Risk Manager advises FBDS Executive Board and the FBDS Steering Committee

Executiive Sponser is part of the FBDS Executive Board

FBDS Program Manager supports and informs the governance bodies (FBDS Executive Board and FBDS Steering Committee)

FBDS Program Manager interacts and coordinates with FBDS Core Team and SMEs

Source: FBDS Program Charter, dated January 15, 2016.

[End of Figure 1: The FDIC’s Governance Structure for the FBDS Project]

The FBDS Program Charter describes, at a high level, the organization, roles, responsibilities, and interrelationships of key stakeholders in the governance structure, which incorporates the following parties:

- Executive Board - comprised of FDIC executives, or their delegates, from the key stakeholder organizations. This group provides strategic direction, facilitates resources, and resolves corporate-level issues for FBDS. A DRR Deputy Director serves as the Executive Sponsor and chair of this board.

- Steering Committee - comprised of other leaders from the key stakeholder organizations. The committee is responsible for recommending, reviewing, and prioritizing projects and initiatives; reviewing and/or approving scope, schedule, and cost changes up to limits established by the Executive Board; providing regular project oversight; and ensuring adequate project resources are available.

- FBDS Program Manager – responsible for ensuring the project meets scope, schedule, and cost goals; leading an FBDS Program Office that supports and informs the governance bodies; and coordinating interaction with a Core Program team and other FDIC SMEs primarily dedicated to FBDS. The FBDS Program Manager is also a TM for the FBDS contract and interacts with the DRR OM and DOA Contracting Officer.

- Risk Manager - from the Division of Finance (DOF), who coordinates with the FBDS Program Manager and other stakeholders to identify and advise the governance bodies on project risks and risk mitigation strategies.

In addition, DRR management updates the FDIC Capital Investment Review Committee (CIRC) quarterly on the status of the FBDS project.5

Footnote 5: To ensure accountability and transparency, the FDIC Board approval of the FBDS project carried a stipulation that the Board receive frequent status updates through the CIRC and its quarterly report to the Board. Since the fourth quarter of 2014, the CIRC’s quarterly reports to the Board have included an auxiliary report containing the FBDS project status.

Audit Results

The FDIC had a number of significant achievements associated with the FBDS project. However, the FDIC did not meet key project milestones and project costs exceeded estimates reported to and reviewed by the project governance bodies. The FDIC’s achievements include:

- establishing the initial FBDS infrastructure and obtaining a security authorization for the system;

- transitioning a large amount of legacy failed FI, and litigation case-related data from DMS to FBDS;

- transferring services provided under DMS from Lockheed to CACI, including data collection and processing services as well as litigation support services;

- scaling the FBDS system to process and load data for the unanticipated failure of a large,6 complex FI with $5.6 billion in assets that failed within 2 months of FBDS contract award;

- conducting a user satisfaction survey for the FBDS system and services, the results of which indicate a generally favorable view of FBDS; and

- realizing a significant reduction in infrastructure hosting and related costs.

Footnote 6: The FBDS contract SOO defines a small FI as less than $1 billion in assets, a medium FI as $1 to $5 billion in assets, and a large FI as over $5 billion in assets.

With respect to milestones and costs:

- While the FBDS system was established in March 2015, shortly after the date required by the contract, there was a delay in implementing important functionality related to automated service request management and cost management. Activities were underway as of June 30, 2016 to implement that functionality.

- Data transition from DMS to FBDS was supposed to be completed by June 2015 but was subsequently revised to December 2015, and then to May 2016 as a result of project schedule delays. While substantial progress had been made, some residual transitionrelated activities remained as of June 30, 2016.

- The schedule delays led the FDIC to extend the DMS contract into 2016, resulting in additional DMS hosting costs of $6.7 million. As a result of that extension, and other transition-related challenges and unforeseen activities, transition-related costs are expected to total $24.4 million, which is $14.6 million higher than early contractor estimates. The FDIC absorbed the additional costs related to the schedule delays as DRR management and the DOA Contracting Officer concluded they were not contractually attributable to CACI or Lockheed. Of note, total transition-related costs of $24.4 million were less than the $31.7 million originally projected for this phase of the new contract in the FBDS Board Case.

The following factors impacted FBDS project progress and costs:

- Understanding project scope and requirements. FDIC personnel did not fully understand the DMS system and the requirements for transitioning failed FI data and services to a new contractor, or communicate these requirements to bidders in a comprehensive transition plan as part of the FBDS solicitation. FDIC personnel attributed the limited understanding of the DMS system and requirements to limitations on access to proprietary information about DMS.

- Establishing clear expectations in contract documents. The FDIC did not always establish clear expectations in contract documents. FDIC personnel indicated that the initial FBDS contract milestone to complete the transition by the end of June 2015 was unrealistic. Soon after the contract award, in January 2015, the FDIC, Lockheed, and CACI began discussing transition requirements and coordinating transition activities. It then became evident that the expected transition timeline needed to be significantly revised. However, it was not until December 2015 that the FDIC modified the Lockheed and CACI transition task orders to incorporate integrated project schedules to which the contractors could be held accountable. The FBDS contract also did not include a comprehensive set of requirements or a clear timeline for implementing service request and cost management functionality in FBDS.

- Implementing a project management framework. For non-IT development projects like FBDS, the FDIC has not implemented an industry best practices (formal) project management framework to guide and structure project activities. The FDIC had also not included important project management-related plans as requirements and deliverables in the FBDS contract. We noted that DRR personnel used certain other tools and techniques to manage the project.

DRR personnel identified technical challenges, and the unanticipated failure of a large, complex FI early in the contract, as other factors that impacted the transition.

At the time we concluded our work, DRR had completed or ongoing actions to mitigate risks for the project, including:

- expanding contract requirements to facilitate more effective end-of-contract transition planning;

- establishing additional review bodies to enhance governance over project changes and award fees;

- establishing a plan for validating that transition activities are completed in accordance with contract requirements; and

- formalizing FBDS-related contract oversight guidance.

Our report discusses additional steps to mitigate risks and provide greater assurance of project success, such as:

- enhancing project management activities;

- determining the desired scalability of the FBDS system;

- evaluating FBDS processes for improvement opportunities, including establishing business rules for automated archiving; and

- establishing guidance for the review of contract performance metrics.

Our report contains seven recommendations for the Director, DRR, in coordination with FBDS governance bodies, to strengthen FBDS governance, project management, and contract oversight to reduce FBDS project-related risks going forward.

Status of the FBDS Project, Including Progress and Costs in Relation to Goals, Milestones, and Budgets

As of June 30, 2016, the FDIC had a number of significant achievements associated with the goals for the FBDS project, such as addressing transition-related goals for establishing and securing the initial FBDS infrastructure; transitioning an enormous volume of data from the DMS system into FBDS; and transferring DMS services, such as the collection and processing of data from recently failed FIs, to FBDS. In addition, the FDIC addressed other FBDS goals by expanding the infrastructure to facilitate the processing of data for a large FI failure, conducting an initial survey of FBDS users to determine satisfaction with the new FBDS system and services, and realizing significant reductions in infrastructure hosting and related costs. However, more work remained to ensure that the goals of the project were achieved. Of note, the FDIC needed to implement automated service request and cost management functionality in ServiceNow and complete final quality control work over data transitioned from DMS.

Transition-Related Goals Addressed

Establishing the FBDS Environment. By mid-2015, CACI had constructed the initial FBDS infrastructure, developed security processes and artifacts, and obtained a security Authorization to Operate (ATO) from the FDIC Chief Information Officer (CIO).7 The FBDS system is comprised of two main applications – Relativity and ServiceNow. Relativity is the e-discovery platform that CACI uses to host databases of failed FI records and to search and retrieve information from those records. ServiceNow is a Web-based service automation tool that CACI uses for help desk management and to implement change management controls. Of note, CACI, under a task order dated June 14, 2016, was developing important service request management and cost management functionality in ServiceNow that FDIC personnel initially believed would be implemented in 2015. Figure 2 depicts a high-level overview of the FBDS system, from a business function perspective.

Footnote 7: The FDIC CIO approved the ATO on March 20, 2015, with the condition that all findings with a risk rating of moderate and low findings related to contingency planning must be fully remediated and verified prior to FBDS production deployment. CACI personnel reported that all such findings had been remediated as of May 19, 2015. [End of footnote]

Figure 2: FBDS System Overview – by Business Function (component diagram)

FBDS is composed of two subcomponents - (Relativity and ServiceNow)

Relativity - Failed Bank Database Hosting and Searching

ServiceNow (components)

Service Request Management (Not Implemented)

Cost Management (Not Implemented)

Help Desk Management

Change Management

Source: Office of Inspector General (OIG) analysis of FBDS system and contract documents, as of June 30, 2016.

[End of Figure 2: FBDS System Overview – by Business Function]

Transitioning DMS Data and Services. In the second quarter of 2015, CACI began work to transition data from the DMS disaster recovery (DR) system into FBDS. FBDS governance reports indicate that, as of March 4, 2016, CACI had transitioned almost 8.8 billion files and 19,823 databases for 509 failed FIs, as well as 139 litigation cases, from DMS to FBDS. In addition, from May 2015 to June 2016, CACI performed various litigation support and other services, such as data collection and processing services for the seven FI failures occurring during that period. As of June 30, 2016, the remaining transition-related work primarily included quality control activities, such as addressing file paths that were missing from DMS and reformatting certain images that were stored in DMS that are not compatible with FBDS.

Other FBDS Goals Addressed

Demonstrating Scalability. The FBDS Board Case projections did not envision a large FI failure until 2018. However, CACI demonstrated that FBDS was scalable by processing and loading8 into FBDS the data for Doral Bank, a large and complex FI with $5.6 billion in assets that failed in February 2015, just 2 months after the FBDS contract was awarded. In addition, at the end of 2015, CACI expanded the capacity of FBDS by purchasing additional data processing and server database licenses at a cost of $1.4 million. DRR personnel indicated that afterwards, FBDS was able to more efficiently process data for Doral Bank, as well as for similar large FI failures that may occur in the future.

Footnote 8: Lockheed performed the majority of the data collection work for Doral Bank because the FBDS contract was only recently awarded and the FDIC concluded that CACI was not yet prepared for this work at the time that the bank failed. CACI initially experienced delays in processing the data and making it available to FBDS users as it worked to enhance the system. [End of footnote]

Measuring User Satisfaction. CACI provided DRR with the results of CACI tests from May and June 2016 related to the speed of FBDS search functionality that appear to indicate the FBDS system returned search results more quickly than DMS. CACI also provided DRR the results of an initial April 2016 CACI-prepared general user satisfaction survey. In total, 44 out of 228 FDIC users of FBDS responded to the survey. The responses indicate a general satisfaction with the FBDS system, although some concerns about FBDS search capabilities were also expressed. Overall, the survey results appear to indicate a more positive view of the FBDS system when compared to the results of general user satisfaction surveys of the DMS system conducted in 2012 and 2015.9

Footnote 9: DRR personnel provided the information regarding user search functionality and user satisfaction to the OIG near the end of the audit fieldwork and, therefore, the OIG did not independently validate this information for accuracy. [End of footnote]

Realizing Cost Reductions. DRR personnel prepared an analysis to identify certain cost reductions associated with the operation of FBDS instead of DMS. Specifically, DRR personnel compared infrastructure hosting and related costs charged for DMS in early 2015 (before the DMS data migration began) to similar charges for FBDS in early 2016.10 According to DRR’s analysis, FBDS infrastructure hosting and related costs are estimated to be, on average, approximately $936,00011 lower per month than for DMS – a reduction of about 54 percent. According to DRR personnel, lower fixed unit prices for the storage of both bulk and litigationrelated data are a key reason for the cost reductions.

Footnote 10: The DRR analysis compared DMS iCONECT-nXT and Relativity hosting and related costs for licensing, maintenance, program management, and help desk for the months of March through June 2015 to similar FBDS Relativity costs for the months of March through June 2016. DRR prepared an initial analysis in August 2016 and revised the analysis in March 2017 based on feedback received from Lockheed. [End of footnote]

Footnote 11: The OIG spoke with DRR personnel to obtain an understanding of their methodology for estimating the cost reductions, and determined that it appeared reasonable. However, the OIG did not independently validate the underlying support for DRR’s calculations. [End of footnote]

Remaining Work to Address Risks Notwithstanding these achievements, as of the end of our fieldwork, the FDIC had work remaining to address certain risks for the project related to:

- enhancing governance over project changes and award fees;

- ensuring that transition activities are completed in accordance with contract requirements;

- formalizing FBDS-related contract oversight guidance;

- enhancing project management activities;

- determining the desired scalability of the FBDS system;

- evaluating FBDS processes for improvement opportunities; and

- establishing guidance for the review of contract performance metrics.

We discuss these on-going project risks and related FDIC efforts in more detail later in the Addressing Issues or Risks to Provide Greater Assurance of Project Success section of the report.

Milestones

The FDIC achieved the 2014 FDIC Performance Goal to award the FBDS contract by December 31, 2014. However, as illustrated in Table 1, the FDIC encountered challenges meeting key post-award FBDS project milestones established in contract documents and subsequent corporate and division performance goals.

Table 1: Summary of Key FBDS Project Milestones as of June 30, 2016

Row 1; Milestone: Award the FBDS contract by December 31, 2014.; Status: The FDIC and CACI signed the FBDS contract on December 31, 2014.;

Row 2; Milestone: Transfer the DMS DR system to CACI by February 10, 2015.; Status: CACI received the DMS DR system from Lockheed on April 22, 2015, but did not obtain access to the system for planning purposes until June 3, 2015.;

Row 3; Milestone: Establish the FBDS environment and prepare to collect, process, and host failed FI data by March 6, 2015.; Status: CACI established the FBDS environment, including the initial Relativity and ServiceNow functionality, on March 27, 2015. CACI began collecting, processing, and hosting data from new failed FIs during May 2015, beginning with a bank that failed in May 2015.;

Row 4; Milestone: Begin migrating DMS databases to FBDS on March 6, 2015.; Status: CACI began migrating DMS Relativity databases on July 7, 2015. CACI began migrating DMS iCONECT-nXT databases on August 4, 2015.;

Row 5; Milestone: Complete the transition of data and services from DMS to FBDS by: - June 29, 2015. - November 15, 2015. - December 31, 2015. - May 31, 2016.; Status: The FDIC, CACI, and Lockheed did not meet these milestones. As of June 30, 2016, CACI had reported that substantially all the legacy failed FI data had been transferred from the DMS system to the FBDS system, with the exception of copies of certain litigation-related productions, which were to be provided by Lockheed.;

Row 6; Milestone: Implement request and cost management services within FBDS by December 31, 2016.; Status: The FDIC and CACI entered into a task order on June 14, 2016 to implement these services within ServiceNow, with a final implementation date of April 30, 2017. FBDS Program Office personnel indicated that certain service request and cost management functionality would be available to FBDS users during the second half of 2016.;

Sources: CACI BOA; CACI Transition Task Order; Lockheed Transition Task Order; CACI ServiceNow Task Order; 2014, 2015, and 2016 FDIC Performance Goal summaries; DRR 2016 Division Goals; and CACI status reports.

[End of Table 1: Summary of Key FBDS Project Milestones as of June 30, 2016]

As noted below, delays in completing the transition by the established milestones resulted in significant unplanned costs to the FDIC, in large part due to additional Lockheed costs incurred to maintain and operate the DMS system during the extended transition period.

Transition-Related Costs

In the FBDS Board Case, DRR requested, and the Board authorized, a contract ceiling authority of $295 million. Included in the $295 million were DRR-projected cost estimates of $21.7 million for the DMS to FBDS transition and $10 million for data and software conversion, for a total estimated transition-related cost of $31.7 million for the FBDS project. These DRR estimates were formulated prior to the FBDS solicitation process, and, therefore, could not factor in vendor-proposed FBDS solutions and related cost estimates. During our audit work we requested DRR’s support for the $31.7 million estimate submitted to the FDIC Board, but DRR personnel were unable to provide supporting documentation. According to DRR personnel, the $31.7 million estimate was developed by FBDS Program Office personnel who possess significant experience developing similar cost estimates, and with input from industry experts.

Once the chosen FBDS vendor’s solution and estimated costs were known through a competitive bidding process, a key project cost control would be to establish an updated estimate of FBDS project transition expenditures, against which the contractors’ progress should be measured. Both the successful bidder, CACI, and Lockheed provided initial transition plans and related cost estimates totaling $7.7 million. DRR personnel communicated to both parties that after contract award, and once Lockheed and CACI had an opportunity to discuss the new system and coordinate their actions, the plans and estimates may need to be revised.

The coordination between the contractors began in January 2015, and resulted in an early May 2015 negotiated price increase in the Lockheed transition cost estimate to $6.5 million, which the DRR OM concluded was a realistic cost estimate. We found no evidence that CACI revised its initial transition cost estimate of $3.3 million during 2015. Therefore, the two contractor cost estimates for transition totaled $9.8 million. From April through November 2015, the FBDS Program Office reported these amounts to the FBDS governance bodies as the estimated cost, or project budget, for completing the transition. Table 2 on the following page provides a summary of transition-related cost estimates and projected actual costs for CACI and Lockheed.

Table 2: Summary of Transition-Related Contractor Estimated and Actual Costs (approximate amounts presented in millions)

Row 1; Cost Type: DMS to FBDS Transition;; Estimate to Complete Transition per FBDS Board Case: ; Estimate To Complete Transition by December 31, 2015 (1): ; Projected Actual Cost Through December 31, 2016 (2): ; Projected Actual Cost Compared to FBDS Board Case Estimate:; Projected Actual Cost Over Estimates Used by Governance Bodies: ;

Row 2; Cost Type: Data and Software Conversion; Estimate to Complete Transition per FBDS Board Case: $10.0; Estimate To Complete Transition by December 31, 2015 (1): ; Projected Actual Cost Through December 31, 2016 (2): ; Projected Actual Cost Compared to FBDS Board Case Estimate: ; Projected Actual Cost Over Estimates Used by Governance Bodies: ;

Row 3; Cost Type: CACI Transition; Estimate to Complete Transition per FBDS Board Case: ; Estimate To Complete Transition by December 31, 2015 (1): $3.3; Projected Actual Cost Through December 31, 2016 (2): $7.1; Projected Actual Cost Compared to FBDS Board Case Estimate: ; Projected Actual Cost Over Estimates Used by Governance Bodies: $(3.8) [in red];

Row 4; Cost Type: Lockheed Transition; Estimate to Complete Transition per FBDS Board Case: ; Estimate To Complete Transition by December 31, 2015 (1): $6.5; Projected Actual Cost Through December 31, 2016 (2): $10.6; Projected Actual Cost Compared to FBDS Board Case Estimate: ; Projected Actual Cost Over Estimates Used by Governance Bodies: $(4.1) [in red];

Row 5; Cost Type: Lockheed Additional DMS Hosting; Estimate to Complete Transition per FBDS Board Case: ; Estimate To Complete Transition by December 31, 2015 (1): ; Projected Actual Cost Through December 31, 2016 (2): $6.7; Projected Actual Cost Compared to FBDS Board Case Estimate: ; Projected Actual Cost Over Estimates Used by Governance Bodies: $(6.7) [in red];

Row 6; Cost Type: Summary; Estimate to Complete Transition per FBDS Board Case: $31.7; Estimate To Complete Transition by December 31, 2015 (1): $9.8; Projected Actual Cost Through December 31, 2016 (2): $24.4; Projected Actual Cost Compared to FBDS Board Case Estimate: $7.3; Projected Actual Cost Over Estimates Used by Governance Bodies: $(14.6) [in red];

Source: OIG summary of information from contracting documents, the FDIC New Financial Environment (NFE), and DRR Financial and Management Reporting.

(1) Amounts reported to the FBDS Executive Board and Steering Committee from April through November 2015.

(2) Amounts reflect actual costs through November 30, 2016 plus estimates for December 2016.

[End of Table 2: Summary of Transition-Related Contractor Estimated and Actual Costs]

The FDIC’s projected actual costs for transition-related services from 2015 and 2016 exceeded, by $14.6 million, the early cost estimates reported to and reviewed by the governance bodies and monitored by the FBDS Program Office. Amounts spent over the contractor estimates reflect costs associated with schedule delays; enhancements to the FBDS infrastructure to process a larger than expected data volume, including data from the unanticipated failure of a large FI in February 2015; and unforeseen work to address data quality issues with, or to provide copies of, the legacy DMS data. DRR personnel were unable to quantify the amount of cost increases that related to specific challenges or additional requirements.

Further, as reflected in Table 2, the delays in meeting the revised transition milestone resulted in significant additional DMS hosting costs of $6.7 million. In September 2015, the FBDS Executive Board approved the option to have Lockheed continue to host DMS into 2016, rather than transfer these operations to CACI. The purpose of extending the DMS contract through May 2016 was to minimize the potential for service disruptions, and reduce the reputational risk of any resulting delays in responding to subpoenas, litigation activities, and customer requests. The FDIC absorbed the additional costs related to the schedule delays, as DRR management and the DOA Contracting Officer concluded they were not contractually attributable to CACI or Lockheed.

Overall Costs. DRR management indicated it was important to put the transition costs discussed above in the context of its overall budgeted and actual costs for DMS and FBDS data support activities. For 2015, DRR requested Corporate budget funds of $95.1 million for data support, of which $63.4 million was for non-transition-related contract services and $31.7 million was for transition services, consistent with the FBDS Board Case.12 DRR reported 2015 actual data support costs of $72.7 million,13 which was $22.4 million less than the budgeted amount. Both the FBDS Board Case and the 2015 budget request were developed when DRR anticipated that the transition would be completed in 2015. However, due to the transition-related delays discussed earlier, project activities and costs that were expected to conclude in 2015 were extended into 2016. DRR’s 2016 budget for data support included $10.1 million for transition-related activities, and, as of June 30, 2016, the division reported spending $9.3 million on those activities. Overall, as of June 30, 2016, cumulative 2015 and 2016 total expected transition-related costs of $24.4 million, as summarized in Table 2, are $7.3 million less than the $31.7 million FBDS Board Case projection.

Footnote 12: Importantly, we noted that throughout 2015, the FBDS Program Office incorrectly reported to FBDS governance bodies the approved corporate budget amounts. During early 2016, DRR’s Financial and Management Reporting section assumed responsibility for FBDS financial reporting to the governance bodies. [End of footnote]

Footnote 13: This amount includes $6.6 million in costs associated with the unanticipated failure of the large, complex FI, Doral Bank. The $6.6 million figure represented non-transition related costs incurred under separate task orders awarded to Lockheed and CACI for Doral Bank-related activities. [End of footnote]

Factors Contributing to the Project's Progress

Our review identified several key factors that impacted FBDS project progress and costs:

- Understanding project scope and requirements.

- Establishing clear expectations in contract documents.

- Implementing a project management framework.

DRR personnel identified technical challenges, and the unanticipated failure of a large, complex FI early in the contract, as other factors that impacted the transition.

Understanding Project Scope and Requirements

FDIC personnel indicated that they faced significant challenges in obtaining a sufficient understanding of the legacy DMS system and the requirements for transitioning failed FI data and services to a new contractor. The FBDS Program Manager noted that because DMS was a contractor-provided service, Lockheed considered key information about the DMS system and services to be proprietary and, therefore, only limited information about DMS was made available to FDIC personnel as they prepared for the FBDS contract solicitation. Notably, we determined that the 2008 DMS contract required Lockheed to provide an exit strategy for transitioning all DMS data back to the FDIC, although the contract did not identify specific requirements for the exit strategy deliverable. The FDIC first requested this deliverable in January 2014, at the start of the second option period of the DMS contract, and in response Lockheed provided the FDIC a draft transition plan in February 2014.

The FBDS Board Case identified as a key project risk that if the information needed to transition the DMS data and services was not well documented for bidders, the transition could cost more or take longer than expected. The FBDS Board Case cited a mitigation strategy to address this project risk. Specifically, the FDIC would work closely with Lockheed to obtain information about its DMS system and services to allow the FDIC to prepare a comprehensive transition plan that could then be provided to the bidders. However, as noted above, the FDIC did not obtain sufficient information to develop or share a comprehensive transition plan as part of the June 27, 2014 contract solicitation, although certain information from the transition plan proposed by Lockheed in February 2014 was included in the contract solicitation.

As a result, FBDS bidders had questions about certain DMS system and transition requirements that the FDIC could not answer. The FDIC informed bidders that the precise details of FDIC’s transition-out agreement with the current DMS service provider (Lockheed) would not be finalized until after contract award, at which time the FDIC, the FBDS vendor, and the DMS service provider would meet to work out the transition details. The FDIC did not request additional information about Lockheed’s February 2014 proposed transition plan until mid- August 201414 and issued a transition task order to Lockheed in mid-December 2014, shortly before the FBDS contract was awarded.

Footnote 14: Lockheed provided an initial response to FDIC’s request on August 28, 2014 (the same day that the FDIC received bids on the FBDS contract solication) and a subsequent response on September 5, 2014. [End of footnote]

Establishing Clear Expectations in Contract Documents

Initial Transition Milestone. The FBDS Board Case identified as a key project risk that if the project schedule was overly aggressive, then quality could suffer and milestones could be missed, resulting in higher costs. FDIC personnel indicated that the 180-day FBDS contract milestone for completing the transition by the end of June 2015 was unrealistic, and the FDIC did not retain any written analysis explaining or supporting this timeframe. Both CACI and Lockheed developed initial transition project schedules to address the 180-day transition milestone. However, once FDIC, Lockheed, and CACI personnel began discussing transition requirements and coordinating transition activities after contract award, it became evident that the expected transition timeline needed to be significantly revised. As early as the March 6, 2015 FBDS Executive Board meeting, the planned project schedule for completing transition activities and shutting down DMS services had been extended by 4 months to November 9, 2015.

Integrated Project Schedules. The FDIC did not initially require Lockheed and CACI to develop, and implement through contract task orders, integrated project schedules reflecting updated project requirements, responsibilities, and milestones that may have helped assign accountability among the contractors for schedule delays or additional costs. It was not until December 2015 that the FDIC contractually established integrated project schedules with both Lockheed and CACI, through modifications of their respective transition task orders. FDIC personnel noted that, in hindsight, this coordination should have occurred sooner in the transition process. CACI attributed many of the earlier project delays to the lack of cohesiveness between the CACI transition-in plan and the Lockheed transition-out plan. We also noted that the FDIC did not establish service level agreements or service level credits for transition-related activities, perhaps because these activities were intended to be completed in a relatively short timeframe.

SOO Requirements. The FBDS Board Case also identified as a key project risk that if contract SOO requirements are not well-defined, then bidder responses to the solicitation will not meet the FDIC’s business needs and the cost will not be reliably estimated. While the SOO required the implementation of automated service request and cost management, FDIC personnel indicated the SOO did not establish a comprehensive set of requirements or a clear timeline for implementing that functionality, and instead focused on desired outcomes to encourage bidders to be innovative. As a result, some FBDS stakeholders presumed this functionality would be available early in the FBDS contract. However, FDIC personnel indicated that FDIC management had placed a priority on activities to address transition-related challenges over other project activities, to try to achieve the 2015 FDIC Performance Goal for transition. Therefore, a task order that included the agreed-upon requirements, and a project schedule and milestones for implementing those requirements in ServiceNow, was not issued until June 2016. In addition, the funding limit for the new ServiceNow task order of $1,013,084 was substantially higher than the 2016 budget estimate of $198,360 for this work, as it would have been difficult to adequately estimate the cost to implement these requirements prior to specifically defining them in early 2016.

Implementing a Project Management Framework FDIC personnel indicated that the FBDS project was not considered an IT development project that fell within the scope of established FDIC IT project management policies and procedures.15 For non-IT development projects, the FDIC has not implemented an industry best practices framework, such as the PMBOK® Guide, or established specific requirements for managing such projects. Such a framework is critical for complex and costly initiatives involving multiple divisions and offices, such as FBDS.

Footnote 15: See FDIC Circular 1300.7, Information Technology Development Policy, and the related FDIC Rational Unified Process system development life cycle methodology, which includes a project management approach influenced by the PMBOK® Guide generally accepted best practices in project management. [End of footnote]

In addition, the CACI proposal, which the FBDS contract incorporated by reference, indicated that CACI would prepare certain project management-related plans to support FBDS project activities. However, many of those plans are not identified as contract deliverables, and as a result, had not been requested for review by FDIC personnel. By not including these plans as deliverables in the contract, the FDIC missed an opportunity to more fully assess and leverage the results of CACI’s project management practices.

FBDS Program Office personnel indicated that they had not followed a formal project management framework, but did leverage several standard tools to manage the FBDS transition activities. The FBDS Program Office generally relied on CACI to develop project management documents and tools, such as the transition-related project schedules and communication plans. Nonetheless, we found evidence that FDIC personnel were also performing certain FBDS-related project management activities, including reviewing the CACI prepared transition project schedule and related status reports, tracking transition-related activities on spreadsheets, and conducting meetings with the contractor, stakeholders, and governance personnel. However, minutes of many of the transition-related meetings held in 2015 were not documented, which limited our insight into the results of those project management activities.

While we could not directly correlate the lack of a formal project management framework to specific project delays or additional costs, it is reasonable to conclude that the FDIC would have benefitted from such a framework to guide and structure FBDS activities. We have developed a recommendation to this effect in the Addressing Issues or Risks to Provide Greater Assurance of Project Success section of our report.

Other Factors

DRR personnel stated that the extent and nature of the technical challenges encountered, and the unanticipated failure of a large, complex FI early in the FBDS contract also had an adverse impact on project schedule and costs. In particular, CACI and Lockheed encountered multiple technical challenges subsequent to the FBDS contract award that impacted the timeline and associated costs of transitioning the failed FI data and services from DMS to FBDS. The FBDS Program Office believes that technical challenges are common for a project of this type and size. We describe these technical challenges in more detail, in relation to the three key transitionrelated milestones of June 2015, December 2015, and May 2016, in Appendix 4 of this report.

Another significant challenge was the failure of Doral Bank, a large, complex FI, within 60 days of the FBDS contract award, which DRR personnel indicated had not been anticipated. Because Doral Bank failed during the time CACI was in the process of establishing the FBDS environment and services, the FDIC contracted with Lockheed to collect the majority of Doral Bank data, and to process and load that data into DMS for contingency purposes. However, DRR also contracted with CACI to process and load the same data, as well as a large number of Doral Bank records scanned by CACI, into FBDS as the system of record. While DRR personnel were unable to quantify the impact, if any, of Doral Bank’s failure on the FBDS transition milestones and costs, the failure required significant unexpected contractor resources, comprised of CACI and Lockheed costs totaling $7.1 million as of June 30, 2016 that were incurred under non-transition-related task orders.

Addressing Issues or Risks to Provide Greater Assurance of Project Success

At the time we concluded our work, DRR had completed or ongoing actions to mitigate risks for the project, including: expanding contract requirements to facilitate more effective end-ofcontract transition planning, establishing additional review bodies to enhance governance over project changes and award fees, establishing a plan for validating that transition activities are completed in accordance with contract requirements, and formalizing FBDS-related contract oversight guidance. Our report discusses additional steps to mitigate risks and provide greater assurance of project success, such as enhancing project management activities, determining the desired scalability of the FBDS system, evaluating FBDS processes for improvement opportunities, and establishing guidance for the review of contract performance metrics.

FDIC Corrective Actions Taken or in Process

Transition Planning. The FDIC took action to mitigate the risk of future transition planning challenges by ensuring that the FBDS contract required CACI to provide a detailed plan for seamlessly transitioning FBDS data and services to a follow-on service provider, should the FDIC decide at a later time to pursue one. In addition, the contract requires CACI to provide the FDIC with system and service diagrams, source code, software configurations, process documentation, and other important artifacts developed to manage, implement, or sustain the FBDS project. These requirements are intended to provide the FDIC with more information about the FBDS system and services than it had about the DMS system and services. CACI provided an initial transition-out plan as part of the FBDS contract solicitation, and is required to revisit and refresh the plan annually to take into account any changes to its hardware, software, or procedures. CACI delivered the contractually-required update to the FBDS transition-out plan to the FDIC in March 2016.

Project Governance. The FBDS Program Charter established the governance structure for the FBDS project, but did not identify the FDIC’s FBDS Change Control Board (CCB), alternatively referred to as a Configuration Control Board, or define its composition or roles and responsibilities. During the audit, the FBDS Program Office took action to enhance the governance over FBDS system changes by establishing a draft CCB charter that provides a proposed membership and initial guidance.16 However, the FDIC needed to enhance the guidance for the FBDS change control process to more clearly define how key FBDS stakeholders will be incorporated into the process, and to clarify the authority of the CCB. In addition, the charter should be reviewed and accepted by the appropriate FBDS governing body. Such enhancements would provide greater assurance that proposed changes are cost effective, made in a consistent manner, and consider overall project objectives. At the end of the audit fieldwork, DRR personnel advised us that the CCB charter was being revised and would subsequently be presented to the FBDS Steering Committee.

Footnote 16: The CCB is to be convened when CACI has a proposed addition, deletion, or modification to FBDS servers and systems. [End of footnote]

Further, the FDIC was developing an FBDS contract Performance and Award Fee Review Board (PAFRB) to facilitate the review of contractor performance, in order to assess whether or not CACI met or exceeded contractually identified strategic objectives. While the FBDS Program Office had drafted a charter containing high-level process guidance for the PAFRB, this charter had also not yet been reviewed and approved by the appropriate FBDS governing body.

Recommendation 1. We recommend that the Director, DRR, in coordination with FBDS governance bodies, establish approved CCB and PAFRB charters that fully define the roles, responsibilities, and authorities of these Boards consistent with management expectations.

Transition Validation and Acceptance. As of June 30, 2016, CACI had reported that substantially all of the legacy DMS data had been transferred to the FBDS system, with the exception of copies of certain litigation-related productions that were to be provided by Lockheed. Other transition-related work that remained included addressing data quality issues identified by CACI in the legacy DMS data and ensuring that all transition work described in the contract and transition task order had been successfully completed. As a result, the FDIC extended the CACI transition task order through December 31, 2016.

We found that the FDIC did not have a documented plan or strategy for validating and accepting that the FBDS system incorporated all key transition-related requirements specified in the contract SOO, including the preservation of all legacy business functionality, FI data, and indexes so there is no disruption to the FDIC institution closing activities or any litigation activities. We requested documentation supporting that the FDIC had formally accepted the FBDS system and services from CACI as having met contract requirements. Such documentation should also indicate how the FDIC had validated that required system functionality existed within FBDS and would help establish a contractually agreed-upon system baseline against which to evaluate future proposed changes. DRR personnel could not provide documentation that indicated that the FBDS system had been formally validated and accepted by the FDIC.

CACI also performed certain quality control procedures to determine whether legacy data had been accurately transferred from DMS to FBDS. However, the FDIC did not develop test plans to document the approach and procedures for performing FDIC user acceptance testing for the new FBDS system. User acceptance testing usually involves structured testing to ensure the system meets desired functionality. FBDS stakeholders performed limited procedures to verify CACI’s testing of the legacy data transitioned from DMS to FBDS, and to test system functionality, but indicated that the guidance regarding what procedures to perform was unstructured and initially unclear.

The PMBOK® Guide recommends that projects incorporate validation and acceptance criteria. The FBDS contract SOO states that “It will be expected that the FBDS provider will develop comprehensive acceptance / transition /and validation plans.” However, the contract did not include acceptance and validation plans for transition phase activities as specific deliverables. At the end of the audit fieldwork, FBDS Program Office personnel indicated that they were working with CACI to develop a strategy for documenting how CACI met the significant contract requirements for the transition, and how FDIC personnel would validate and accept that CACI met those requirements.

Recommendation 2. We recommend that the Director, DRR, in coordination with FBDS governance bodies, coordinate with CACI to develop an appropriate validation and acceptance plan for ensuring that significant contract requirements are met and there is a sufficiently documented acceptance that FBDS transition-related activities have been concluded.

Contract Oversight. At the start of the audit, FDIC personnel had not prepared a Contract Management Plan (CMP) for the FBDS contract, as required by FDIC and DRR policy.17 The objective of the CMP is to provide the Contracting Officer, OM, and TMs a common understanding of both contractor and FDIC obligations under the contract. The CMP is important because it identifies the strategy for managing key contract vulnerabilities or performance risks that are inherent in the contract, as well as for managing any unique contract terms and conditions.

Footnote 17: FDIC Circular 3700.16, FDIC Acquisition Policy Manual, requires a CMP for all contracts involving the acquisition of services having a total estimated value of $1,000,000 and greater. DRR Circular 3700.16, DRR Contract Management, requires a CMP for all contracts over $100,000. [End of footnote]

In addition, FBDS stakeholders identified the need for more formalized guidance regarding FBDS-related contract oversight processes and controls. FDIC Circular 4010.3, FDIC Enterprise Risk Management Program, requires current and appropriately documented procedures, which can decrease the risk that control processes will not be consistently or effectively applied, in particular in situations where there are changes in key contract oversight personnel. We noted that DRR developed Excel spreadsheets to document the monitoring of project service requests, such as requests for data scanning, loading, searching, and exporting, among others. Monitoring activities included review of cost estimates, approval of project task initiation, and review and acceptance of related contract deliverables. DRR also developed Excel spreadsheets to document the review of contractor invoices. However, DRR had not developed written guidance that sufficiently described these control practices, to ensure effective and consistent application. FDIC personnel indicated that preparation of the CMP and other, more formalized, FBDS-related contract oversight process guidance was considered a lower priority early in the contract, as FBDS stakeholders worked to address the many transition-related challenges.

After we brought these issues to FDIC management’s attention during the audit, DRR and DOA personnel jointly developed a CMP using the FDIC-established template. At the end of our fieldwork, DRR personnel had also drafted high-level process guidance specific to FBDS contract oversight. Due to the timing of the CMP and draft process guidance, we did not assess these documents to determine if they were comprehensive, or were being effectively implemented. However, we did note that the draft guidance does not contain a process for comparing initial cost estimates with actual amounts invoiced by the contractor. FBDS stakeholders noted that this control is critical for ensuring that costs incurred for services do not exceed expectations, and identified its absence as a key risk. The draft guidance notes and assumes that once the new ServiceNow functionality is available, invoice reporting will be more accurate and will facilitate comparing estimates to actual costs for projects and requests.

Recommendation 3. We recommend that the Director, DRR, in coordination with FBDS governance bodies, assess and finalize FBDS-related contract oversight guidance and develop a plan to review and revise it, as appropriate, to reflect process changes and control enhancements resulting from the implementation of the ServiceNow service request and cost management functionality.

Additional Risks Warranting Management Attention

While the actions the FDIC has taken or planned are positive, this section of our report discusses additional issues or risks that management should address to help provide greater assurance of project success.

Project Management. We identified through interviews of FBDS stakeholders that project management practices being employed may not ensure that FBDS-related projects, such as the transition from DMS, are completed on schedule, within budget, and meet key FDIC requirements. As noted earlier in the report, the FDIC had not implemented a project management framework that might help mitigate this risk on other FBDS-related projects, such as the recently initiated ServiceNow task order. This project, which was estimated to cost $1 million, will automate and link service requests to the cost system, allowing the FDIC to better identify and assess the reasonableness of FBDS costs. ServiceNow implementation will involve multiple releases, requiring timely and effective communication and coordination among FBDS stakeholders and CACI personnel under tight timeframes. As a result, it is important for the FDIC to establish a project management framework that incorporates a project management plan and related documents that facilitate:

- Schedule management: In the last quarter of 2015, FDIC personnel found that changes were needed to the task dependencies18 in the CACI transition project schedule, which, once revised by CACI, helped identify a more appropriate critical path for project activities. FDIC personnel also determined that CACI had not developed a schedule baseline for the transition, which would have made it easier to understand where the project was falling behind schedule. The project management plan should address FDIC review of other project schedules, such as the CACI schedule for ServiceNow, to ensure they reflect the appropriate task dependencies and critical path, and are adequately baselined.

Footnote 18: Project activities may or may not have dependencies between them that can affect the application and use of resources.

- Communication management: Both the ServiceNow task order and another important FBDS-related project on the horizon – a significant upgrade of the Relativity software – will require coordination and communication among CACI personnel and FBDS stakeholders. The project management plan should address communication strategies for ensuring that the numerous meetings and other interactions between these groups are timely and effective.

- Stakeholder management: Certain FBDS stakeholders indicated a need for more formal and consistent identification and prioritization of FBDS-related issues, and the actions to address them. During the audit, the FBDS Program Office and CACI initiated an actions and issues log in response to this feedback, but had not yet formalized guidance for managing that process. Certain FBDS stakeholders also indicated a desire for greater involvement in defining project requirements and improving other FBDS processes. The project management plan should address strategies and guidance for ensuring the many FBDS stakeholders are effectively engaged and issues are effectively managed.

- Quality management: While FDIC TMs reviewed contract deliverables and related invoice billings from CACI, the FDIC had not defined its strategy for overseeing the contractor’s quality assurance activities, and for performing FDIC quality assurance activities. The project management plan should address such a strategy, including the approach and procedures for performing FDIC user acceptance testing to determine that ServiceNow functionality meets FDIC requirements.

- Human resource management: During the audit, some FBDS stakeholders indicated that FBDS monitoring activities would benefit from additional staff resources to support DRR contract oversight and the FBDS Program Office, which has a number of project responsibilities, and FDIC management took action to assign more FDIC personnel to the FBDS project. The project management plan should identify and document how the FDIC will continue to address the human resource requirements for FBDS-related projects, such as FDIC staff needed for ServiceNow requirements validation and user acceptance testing, and also outline the roles and responsibilities of assigned project resources.

While CACI had provided the FDIC a project schedule for the ServiceNow implementation, at the time of our audit, the FBDS Program Office had not documented plans for managing and monitoring ServiceNow project activities involving FBDS stakeholders, such as coordinating communication, validating that requirements meet stakeholder needs, and conducting FDIC user acceptance testing to ensure desired requirements are implemented in the system.

Recommendation 4. We recommend that the Director, DRR, in coordination with FBDS governance bodies, implement a project management framework reflecting industry best practices, to include a project management plan and other documents as appropriate for the ServiceNow task order and future FBDS-related projects.

System Scalability. The FBDS contract SOO generally required FBDS to have the capacity to process data for 10 small to medium FI failures per year, but to be sufficiently scalable to handle an unforeseen increase in the number of FI failures during the contract, potentially including large FI failures. However, while the SOO states that scalability is of critical importance to the FDIC, the SOO does not define metrics for determining whether the FBDS system is sufficiently scalable. In addition, we found that FDIC management had not yet established clear expectations for the scalability of the FBDS system, or a process for determining whether the FBDS system can meet any expectations that are established.

The FBDS Board Case stated that FDIC staff were conducting an analysis of how to collect, preserve, and use data in the event a Systemically Important Financial Institution (SIFI) or a Global-SIFI (G-SIFI) were to fail. The case noted that it was unclear at that time whether the FBDS contract would be utilized to perform these functions, or whether an alternative data collection strategy would be more appropriate. Nevertheless, the FDIC Board was told that, in procuring FBDS, the FDIC will require significant scalability so as to not preclude the possibility that FBDS could be used in a SIFI or G-SIFI resolution.

To address the FBDS goal of providing improved scalability, CACI developed the FBDS system to host failed FI data in one platform – Relativity – rather than the multiple platforms used with DMS. In addition, CACI is using Nuix eDiscovery (Nuix) software to process the failed FI data for loading into FBDS, which CACI indicated is one of the fastest information processing tools and can be quickly scaled using Nuix’s flexible licensing model. Further, CACI indicated that both the FBDS primary and DR sites would be available for production in the event there is a surge in FI failures, although this capability had not been tested at the time of the audit. At the staffing level, CACI plans to leverage approved vendors for rapid deployment to meet any FDIC surge requirements for data collection.

As noted earlier in the report, CACI demonstrated FBDS scalability by processing and loading data for Doral Bank, and expanding the FBDS capacity for future large and complex FI failures, although not specifically for a SIFI. However, CACI initially experienced delays in processing Doral Bank data and making that data available to FBDS users, as it worked to enhance the system. In addition, CACI and FDIC personnel believe that FBDS is the largest implementation of Relativity in existence; therefore, the limits to which the system could, or should, be effectively scaled further are unproven. Accordingly, until the FDIC defines or baselines the desired capacity for handling a complex FI failure, multiple concurrent FI failures, or a SIFI, there is a risk that the FBDS system may not be able to efficiently process failed FI data in such situations and, therefore, FBDS users may not have timely access to needed FI data.

One key FBDS stakeholder noted that scalability was difficult to measure, and the FDIC needs to balance the costs of unused infrastructure against the need to have timely access to failed FI data. This individual indicated that DRR generally has up to a year after an FI closing to make the FI’s data available in FBDS; however, the contract metrics for making failed FI data available in FBDS are generally 90 days or less. By defining or base-lining its desired capacity for handling a complex FI failure, or multiple concurrent FI failures, the FDIC gains greater assurance that the FBDS system meets the contract requirement to be sufficiently scalable, and that FBDS will be able to efficiently collect, process, and provide users with timely access to failed FI data.

Recommendation 5. We recommend that the Director, DRR, in coordination with FBDS governance bodies, conduct a feasibility study to determine the level of scalability desired for FBDS and, based on that determination, assess the ability of the FBDS system to scale to that level and what additional investment might be required.

System and Process Improvements. The FBDS contract SOO Section 6, Infrastructure Capabilities, addresses system improvement by requiring CACI to provide annually to the FDIC a roadmap of all FBDS services, as well as a supporting strategy that addresses industry trends related to technologies and capabilities to support FBDS. Due to transition-related efforts, CACI delivered an initial FBDS Annual Technology Briefing on June 24, 2016, rather than in 2015 as required by the contract. The FBDS Program Manager indicated that the FDIC will consider the information in this deliverable as part of the 2017 FBDS budget development process.

The FBDS contract SOO Section 5.4, Content Management, also addresses system improvement by requiring automated archiving of data to improve FBDS cost efficiency. We identified a risk that this functionality would not be implemented in a timely manner. At the time of the audit, the FDIC had not yet developed, and worked with CACI to implement, the business rules that will support the archiving capabilities of FBDS. However, the FBDS Program Manager indicated that the plan was to perform this work after an upgrade to the Relativity software, identified in the aforementioned FBDS Annual Technology Briefing, was completed. CACI completed the upgrade in January 2017.

While the FDIC has developed strategies for system improvement, FBDS stakeholders identified a risk that the FBDS project does not have a clear plan for reviewing FBDS processes to determine if they can be more cost efficient. Certain FBDS stakeholders identified data collection as one of the most critical areas for process review and improvement. Key decisions for this area include determining what, and how much data, to collect at an FI closing, who should be involved in the decision making process, and when such decisions should be made. Other areas identified for review and potential improvement included data processing, data storage, and data productions.

Certain FBDS stakeholders indicated that it will be important to ensure effective collaboration between DRR Investigations, DRR Customer Service, the Legal Division, and other FBDS stakeholders in reviewing and determining changes that are needed to key processes. They added that the FBDS project would benefit from a plan that defines which FBDS stakeholders are involved in the discussions and how this involvement is coordinated. Such a plan could help ensure that the FDIC timely identifies and implements process efficiencies that could result in significant cost savings.

The FBDS contract SOO Section 9.1, Service Level Agreements/Key Performance Indicators, requires CACI to provide the FDIC with service and technology recommendations throughout the life of the contract to continuously improve service delivery and capabilities while driving cost efficiencies. On January 19, 2016, the FBDS Program Manager initiated an FBDS Program Strategy and Innovation group, comprised of FBDS Program Office personnel, CACI personnel, the DRR OM, the DOA Contracting Officer, and other key FBDS stakeholders. The objective of this group, is to continually improve the FBDS environment and services in line with changes to technology, industry standards and practices, and the FDIC’s own growth. This group was developed to help analyze and prioritize improvement requests. While this was a positive development, it had not resulted in a plan for process improvement at the time of the audit.

The PMBOK® Guide recommends a plan that details steps for analyzing processes to identify activities that enhance their value. Although not identified as a deliverable in the contract, the CACI proposal for FBDS indicated CACI would prepare a Process Improvement Management Plan that would outline steps for analyzing program processes to identify and enhance the value of the process. At the time of the audit, the FDIC had not received such a plan. FDIC and CACI personnel noted that the transition has been the highest priority for CACI and indicated that the FBDS project would focus on archiving and other process improvement activities after the transition was completed.

Recommendation 6. We recommend that the Director, DRR, in coordination with FBDS governance bodies, request and receive a Process Improvement Management Plan from CACI within a reasonable timeframe, and in conjunction with that plan, develop a strategy and guidance for reviewing FBDS processes, approving and implementing needed improvements, and communicating the results of these efforts. This includes establishing and implementing business rules to support the automated archiving capabilities of FBDS.

Contract Metrics. FDIC processes for FBDS performance review management had not been formalized, which creates a risk that such processes may not be effectively implemented. The FBDS contract established 14 service level metrics and related performance measures, which include efficiency-focused metrics that assess how timely CACI responds to certain user requests or conducts CACI data collection activities. The CACI summary of monthly performance metrics through April 30, 2016 indicates that CACI data collection activities were generally timely while summary performance metrics for CACI responses to user requests identified mixed results.

During the audit, both FBDS stakeholders and CACI personnel identified the need to review and update or expand, as appropriate, the FBDS contract performance metrics to ensure they meet FBDS project needs, and to better represent the priorities of the FDIC as well as the services that are being provided by CACI. We found that the FDIC had not yet developed formalized process guidance for revising the metrics and related service level agreements, as appropriate, and for ensuring that the appropriate stakeholders are included in this process.

While FBDS infrastructure hosting costs are significantly less than DMS, some FBDS stakeholders noted that the FDIC had not yet established formal service level metrics specifically designed to evaluate the cost efficiencies of FBDS. Such metrics could include measuring variances between initial task cost estimates and the actual costs incurred to complete the tasks. Another metric could measure the costs of data collection. The DRR OM was developing an analysis of data collection costs under DMS and FBDS to assess the cost efficiencies of the FBDS processes, but that analysis was incomplete at the end of the audit fieldwork. We also noted that the FDIC had not developed efficiency-focused contract performance metrics for scenarios in which there is a large, complex failure, such as Doral, or multiple concurrent failures. Another area for potential metrics could be user satisfaction.

Metrics help determine whether the project satisfies the business needs for which it was undertaken, and whether the contractor is adhering to contractually required service level agreements. Failure to adequately adjust or revise metrics to reflect current project needs can have negative consequences for the project’s stakeholders. FBDS contract SOO Appendix 4, Service Level Metrics and Performance Indicators, requires the FDIC and CACI to annually review contract service levels and adjust their stringency as appropriate, taking into consideration current performance, possible changes in performance capabilities and productivity, and comparable industry information. In July 2016, near the end of the audit fieldwork, the FBDS Program Office initiated action to develop process guidance for updating service level agreements.

Recommendation 7. We recommend that the Director, DRR, in coordination with FBDS governance bodies, develop guidance for reviewing contract performance metrics, and revising the metrics and related service level agreements as appropriate.

Corporation Comments and OIG Evaluation

The Director, DRR, provided a written response, dated January 19, 2017, to a draft of this report. The response is presented in its entirety in Appendix 5. In the response, the Director, DRR, concurred with all seven of the report’s recommendations. In addition, the response describes planned corrective actions to address the recommendations. A summary of the Corporation’s corrective actions is presented in Appendix 6. The planned actions are responsive to the recommendations, and the recommendations are resolved.

We also provided a draft of the report to Lockheed and CACI. Both firms provided comments that we considered and incorporated into the final report as we deemed appropriate.

Appendix 1

Objective, Scope, and Methodology

Objective

Our audit objective was to determine (1) the status of the Failed Bank Data Services project, including progress and costs in relation to goals, budgets, and milestones; (2) factors contributing to the project’s progress; and (3) significant issues or risks that must be addressed to achieve project success.

We performed our audit fieldwork from November 2015 through August 2016. We issued a preliminary draft report in October 2016 and received informal comments and additional information for our consideration in December 2016. We performed steps to validate any feedback that resulted in additions or revisions to the report.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective.

Scope and Methodology

The scope of this audit included FBDS project-related activities that occurred during the period January 2014 through June 2016, and the status of project goals, milestones, and costs as of June 30, 2016.

To obtain an understanding of the FBDS project, we reviewed:

- The FBDS Board Case and related FDIC Board minutes that request and record, respectively, the authority to enter into a 10-year Basic Ordering Agreement to support the FDIC’s management of failed FI data.

- The Technical Evaluation Panel Report and Selection Recommendation Report documenting the rationale for the FBDS contract award.

- The FBDS Program Charter outlining the program goals and governance structure.

To accomplish the audit objective, we performed the following procedures and techniques:

- Reviewed and summarized pertinent information in the FBDS contract, the DMS contract, and relevant task orders;

- Examined documentation summarizing the FBDS governance activities, including presentations to, and meeting minutes for, the FBDS Executive Board, FBDS Steering Committee, and project-related working groups;

- Identified Corporation performance goals relevant to the FBDS project and determined the FDIC’s success in achieving them;

- Analyzed project budget and actual cost information obtained from FDIC personnel and system-generated reports from the FDIC’s core NFE financial system and from the Automated Procurement System;

o We did not perform audit procedures to assess the effectiveness of information system controls associated with this information because such procedures were not necessary to accomplish our audit objective. Rather, we corroborated the reliability of information obtained from automated systems, as appropriate, through discussions with FDIC personnel and our review of other relevant documentation.

- Interviewed FBDS project stakeholders from DRR, DOA, the Legal Division, DOF, DIR, DIT, and the CIO Organization about the risks, goals, status, challenges, costs, schedules, and decision-making related to the FBDS project;

- Interviewed contractor project management key personnel regarding the timeline of transition-related activities, the challenges encountered in achieving contract objectives, and the risks for the FBDS project;

- Obtained and reviewed selected contract deliverables, such as contract status reports and service level metric results;

- Reviewed relevant FDIC policies, procedures, and guidance, such as FDIC Circular 3700.16, FDIC Acquisition Policy Manual, and the related FDIC Acquisition Procedures, Guidance and Information; FDIC Circular 4010.3, FDIC Enterprise Risk Management Program; and FDIC Circular 1300.7, Information Technology Development Policy. In addition, we consulted the PMBOK® Guide as a source for sound project management practices applicable to the FBDS project in conducting our work.

We conducted our work at the FDIC’s offices in Dallas, Texas and Arlington, Virginia.

Appendix 2

Glossary of Terms

Capital Investment Review Committee (CIRC) - The purpose of the CIRC is to implement a systematic management review process that supports the FDIC’s capital investments and ensures regular monitoring and proper management once funded. The CIRC is responsible for overseeing investments deemed to have a significant corporate impact, such as those identified as critical by the FDIC Chairman or Board of Directors. The CIRC membership includes the FDIC Chief Financial Officer, Chief Information Officer, General Counsel, Division and Office Directors, and the Chief Risk Officer.

Change Management - The processes for monitoring and controlling modifications to a project.

Cost Management - The processes for project cost estimation, invoicing, accounting, and trend analysis.

Critical Path The sequence of activities on a project schedule that represents the longest path through a project, which determines the shortest possible project duration. Comparing the project’s progress along the critical path can help determine schedule status.

Database Index - A database index improves the speed at which data can be retrieved from a database table. Indexes are used to quickly locate data without having to search every row in a database table every time a database table is accessed.

Help Desk Management - The processes for setting up system users, password resets, basic troubleshooting, and escalating specific research, query, extract, and production requests to the appropriate business support unit.

Infrastructure Hosting - Services that include storing failed FI data and providing FBDS users access to that data. Infrastructure hosting costs are primarily a function of the volume of data stored in FBDS but also include the cost of software licenses and monthly FBDS user fees.

Production - The export of data, conversion of images into specific formats, and other services, as needed, to support litigation-related discovery requests.

Project Management - The application of knowledge, skills, tools, and techniques to project activities to meet project requirements. Project management typically includes identifying requirements, interacting with stakeholders, and balancing competing project constraints such as scope, quality, schedule, budget, resources, and risks.

Service Level Agreement - An agreement that establishes each aspect of the contractor’s performance to be monitored and reported on, and the associated performance metrics that will be used to measure the quality, efficiency, or other attributes of the contractor’s performance.

Service Level Credit - A monetary amount payable to the FDIC by the FBDS contractor for failure to achieve an acceptable quality level. Service Level Credits do not apply during the FBDS transition period.

Service Request Management - The processes to manage all work requests, such as user service requests, program office requests, upgrade activities, and maintenance activities.

Schedule Baseline - The approved version of a schedule model that can be changed only through formal change control procedures and is used as a basis for comparison to actual results. It is accepted and approved by the appropriate stakeholders as the schedule baseline, with baseline start dates and baseline finish dates. During monitoring and controlling activities, the approved baseline dates are compared to the actual start and finish dates to determine whether variances have occurred.

Appendix 3

Abbreviations and Acronyms

ATO - Authorization to Operate

BOA - Basic Ordering Agreement

CACI - CACI-ISS, Inc.

CCB - Change Control Board

CIO - Chief Information Officer

CIRC - Capital Investment Review Committee

CMP - Contract Management Plan

DIR - Division of Insurance and Research

DIT - Division of Information Technology

DMS - Data Management Services

DOA - Division of Administration

DOF - Division of Finance

DR - Disaster Recovery

DRR - Division of Resolutions and Receiverships

FBDS - Failed Bank Data Services

FDIC - Federal Deposit Insurance Corporation

FI - Financial Institution

G-SIFI - Global Systemically Important Financial Institution

IT - Information Technology

Lockheed - Lockheed Martin Services, Inc.

NFE - New Financial Environment

Nuix - Nuix eDiscovery

OIG - Office of Inspector General

OM - Oversight Manager

PAFRB - Performance and Award Fee Review Board

PMBOK® - Project Management Body of Knowledge

SIFI - Systemically Important Financial Institution

SME - Subject Matter Expert

SOO - Statement of Objectives

TM - Technical Monitor

U.S.C. - United States Code

[End of Appendix 3]

Appendix 4

Challenges Affecting Project Milestones

This Appendix provides additional details on technical challenges that resulted in project delays in relation to the three key transition-related milestones of June 2015, December 2015, and May 2016.

Challenges Affecting the June 2015 Transition Milestone

DMS Disaster Recovery System. Lockheed hosted DMS iCONECT-nXT data on its production system in Maryland and on its DR system in California. Lockheed updated the DMS DR system daily with a new copy of DMS iCONECT-nXT data. One of the initial transition-related activities entailed Lockheed transferring the DMS DR system to the CACI primary FBDS facility in Virginia so that CACI could transition data more efficiently and securely to FBDS. CACI’s initial project schedule anticipated that the DMS DR system would be transferred in early February 2015 and be accessible by early March 2015. However, CACI did not obtain access to the data on the DMS DR system until June 2015. This delay in the CACI schedule was primarily the result of two factors. The first was the need to agree on and establish an acceptable connection between the primary FBDS facility and the DMS production system before the transfer of the DMS DR system to CACI could occur. This connection was in place by early April 2015. The second was the need to complete and implement, through modifications to the FBDS and DMS contracts, a mutually agreed-upon Interconnection Security Agreement and Memorandum of Agreement before CACI could access data on the DMS DR system once it was transferred. The contract modifications to incorporate these documents were effective May 26, 2015.

Relativity Migration. Lockheed engaged a subcontractor to maintain the DMS Relativity data, which consisted of information related to litigation cases. CACI began planning for the migration of this information in January 2015. CACI’s initial project schedule anticipated starting the migration in March 2015. The targeted timeframe was subsequently revised to the end of April 2015, around the time the FBDS system was approved for production deployment. Additional time was then needed in May and June 2015 to understand and document the migration process, establish a schedule that prioritized the order in which the litigation case information would be migrated, and develop and evaluate the related communication plan. These efforts required coordination among multiple parties, including the FBDS Program Office, the Legal Division, CACI, Lockheed, and others. As a result of the time taken to complete these activities, CACI started the migration of the litigation cases on July 7, 2015.

Challenges Affecting the December 2015 Transition Milestone

DMS Back-ups. CACI had planned to transition data 24 hours a day from the DMS DR system. However, CACI was unable to do so during the times when Lockheed was backing-up data to the DMS DR system. CACI reported that this limited access resulted in at least 48 hours per week less data transition time, reducing CACI’s effectiveness by 28 percent. CACI personnel indicated that they requested, through the FDIC, a modification to the DMS back-up schedule in May 2015 to allow CACI more time to access data on the DMS DR system. FDIC coordination with Lockheed to modify the associated DMS contract requirements took longer than expected. It was not until September 2015 that Lockheed and the FDIC agreed to a modified back-up schedule that allowed CACI a 4-day uninterrupted window each week for transitioning data from the DMS DR system.

Oracle Data Pump. The CACI-proposed method to extract iCONECT-nXT data directly from the DMS DR system to FBDS was not accepted by Lockheed because it would require unacceptable changes to the DMS environment. Therefore, CACI changed its approach by requesting that iCONECT develop an Oracle Data Pump module for the DMS DR system that would facilitate very high-speed movement of data from one database to another. CACI received the Oracle Data Pump module from iCONECT on June 22, 2015, after which it went through testing by Lockheed and adjustments by CACI. According to CACI, the Oracle Data Pump was ready for use in transitioning data on August 4, 2015.

The data transition speeds using the Oracle Data Pump module were initially very slow. In the month that followed, iCONECT and CACI personnel, in coordination with Lockheed and Oracle, identified that the Oracle Data Pump module had to be configured to exclude the database indexes when exporting databases from the DMS DR system. CACI personnel indicated that the necessary indexes would be created and added to the databases once the databases were imported into FBDS. These efforts resolved the data transition speed issues with the Oracle Data Pump in mid-September 2015.

Challenge Affecting the May 2016 Transition Milestone

Copies of Relativity Productions. In early 2016, the FDIC determined that certain DMS Relativity productions may not have been included in the DMS Relativity data previously migrated to FBDS, because they were not on the DMS Relativity platform. However, it was not clear which productions might not have been included in the migration to FBDS. Therefore, on March 17, 2016, the FDIC requested that Lockheed provide the FDIC, as part of the transition, a copy of all of the DMS Relativity productions, including those that had been maintained by its subcontractor outside of the DMS Relativity platform. The initial target for completing this work was April 29, 2016, one month before the May 31, 2016 transition milestone.

On May 5, 2016, Lockheed notified the FDIC that its subcontractor had provided copies of only 5 percent of the DMS Relativity production data that the subcontractor was to provide. Because of its subcontractor’s limited capacity to copy the volume of data requested, Lockheed reported that it would not be able to provide the remaining productions by the May 31, 2016 contract expiration date. Subsequently, the FDIC extended the DMS contract and transition task order through October 2016 to provide additional time to complete the delivery of the Relativity productions at no additional cost to the FDIC. Lockheed met this revised contract milestone.

[End of Appendix 4]

Appendix 5

Corporation Comments

[FDIC Letterhead, FDIC logo, Federal Deposit Insurance Corporation, Division of Resolutions and Receiverships, 3501 North Fairfax Drive Arlington, VA 22226]

TO: Mark F. Mulholland, Assistant Inspector General for Audits, Office of Inspector General

FROM: Bret D. Edwards, Director /Signed/, Division of Resolutions and Receiverships

SUBJECT: Management Response to Draft Audit Report Entitled, The FDIC’s Failed Bank Data Services Project (Assignment No. 2016-019)

The Federal Deposit Insurance Corporation (FDIC) has completed its review of the Office of Inspector General’s (OIG) draft audit report entitled The FDIC’s Failed Bank Data Services Project (Assignment No. 2016-019) dated January 4, 2017.

In its report, the OIG indicates that the Division of Resolutions and Receiverships (DRR) FBDS project had a number of significant achievements, including the successful transfer of billions of failed bank records and tens of thousands of related databases. While we acknowledge that original key project milestones were missed and project costs exceeded initial estimates, we are pleased to report that the transition successfully concluded on October 31, 2016, and that the ongoing contractual relationship with CACI is producing the cost savings FDIC was seeking when it chose to re-compete this relationship.

We agree with the OIG about the need to address certain issues identified during the course of the audit and fully concur with all seven of its recommendations as set forth below.

Recommendation #1: We recommend that the Director, DRR, in coordination with FBDS governance bodies, establish approved CCB and PAFRB charters that fully define the roles, responsibilities, and authorities of these Boards consistent with management expectations.

DRR Response: DRR concurs with this recommendation.

Corrective Action: Change Control Board (CCB) and Performance Award and Fee Review Board (PAFRB) charters have been drafted by the FBDS Project Management Office (PMO) and DRR Contract Oversight and approved by the FBDS Steering Committee. DRR, in coordination with FBDS governance bodies, will provide final approvals for the CCB and PAFRB charters that fully define the roles, responsibilities, and authorities of these Boards consistent with management expectations.

Completion Date: April 30, 2017

Recommendation #2: We recommend that the Director, DRR, in coordination with FBDS governance bodies, coordinate with CACI to develop an appropriate validation and acceptance plan for ensuring that significant contract requirements are met and there is a sufficiently documented acceptance that FBDS transition-related activities have been concluded.

DRR Response: DRR concurs with this recommendation.

Corrective Action: DRR, in coordination with FBDS governance bodies, will ensure that the PMO and DRR Contract Oversight coordinate with CACI to provide appropriate validation and acceptance documentation for ensuring that significant contract requirements were met and there is a sufficiently documented acceptance that FBDS transition-related activities have been successfully concluded.

Completion Date: August 31, 2017

Recommendation #3: We recommend that the Director, DRR, in coordination with FBDS governance bodies, assess and finalize FBDS-related contract oversight guidance and develop a plan to review and revise it, as appropriate, to reflect process changes and control enhancements resulting from the implementation of the ServiceNow service request and cost management functionality.

DRR Response: DRR concurs with this recommendation.

Corrective Action: DRR, in coordination with FBDS governance bodies, will assess and finalize Contract Oversight FBDS-related guidance. DRR will also review and revise the guidance following the placement of ServiceNow in the production environment.

Completion Date: July 31, 2017

Recommendation #4: We recommend that the Director, DRR, in coordination with FBDS governance bodies, implement a project management framework reflecting industry best practices, to include a project management plan and other documents as appropriate for the ServiceNow task order and future FBDS-related projects.

DRR Response: DRR concurs with this recommendation.

Corrective Action: DRR, in coordination with FBDS governance bodies, will implement a project management framework reflecting industry best practices, to include a project management plan and other documents for ServiceNow and other future FBDS-related projects as appropriate.

Completion Date: September 30, 2017

Recommendation #5: We recommend that the Director, DRR, in coordination with FBDS governance bodies, conduct a feasibility study to determine the level of scalability desired for FBDS and, based on that determination, assess the ability of the FBDS system to scale to that level and what additional investment might be required.

DRR Response: DRR concurs with this recommendation.

Corrective Action: While the scalability of FBDS appears to be sufficient to meet near term resolution projections, DRR will conduct a feasibility study to identify the limits to which the system can be scaled. This study will use actual results from the FBDS processing and loading of data for the failed Doral Bank. Additionally, multiple hypothetical resolution scenarios will be assessed to identify the current scalability of FBDS and what measures CACI and FDIC could take to enhance its scalability.

Completion Date: December 15, 2017

Recommendation #6: We recommend that the Director, DRR, in coordination with FBDS governance bodies, request and receive a Process Improvement Management Plan from CACI within a reasonable timeframe, and in conjunction with that plan, develop a strategy and guidance for reviewing FBDS processes, approving and implementing needed improvements, and communicating the results of these efforts. This includes establishing and implementing business rules to support the automated archiving capabilities of FBDS.

DRR Response: DRR concurs with this recommendation.

Corrective Action: The FBDS PMO has asked CACI to provide a Process Improvement Plan for the program. Once received, the FBDS Program Team will develop a strategy and guidance for reviewing FBDS processes, approving and implementing needed improvements, and communicating results, to include establishing and implementing business rules to support the automated archiving capabilities of FBDS.

Completion Date: December 15, 2017

Recommendation #7: We recommend that the Director, DRR, in coordination with FBDS governance bodies, develop guidance for reviewing contract performance metrics, and revising the metrics and related service level agreements as appropriate.

DRR Response: DRR concurs with this recommendation.

Corrective Action: DRR has drafted and will finalize guidance through the FBDS Executive Board for reviewing contract performance metrics and revising the metrics and related service level agreements as appropriate.

Completion Date: June 30, 2017

[End of Appendix 5]

Appendix 6

Summary of the Corporation’s Corrective Actions

This table presents corrective actions taken or planned by the Corporation in response to the recommendations in the report and the status of the recommendations as of the date of report issuance.

Row 1; Rec. No.: 1; Corrective Action: Taken or Planned: CCB and PAFRB charters have been drafted by the FBDS PMO and DRR Contract Oversight and approved by the FBDS Steering Committee. DRR, in coordination with FBDS governance bodies, will provide final approvals for the CCB and PAFRB charters that fully define the roles, responsibilities, and authorities of these Boards consistent with management expectations.; Expected Completion Date: 4/30/2017; Monetary Benefits: No; Resolved;a Yes or No: Yes; Open or Closedb: Open;

Row 2; Rec. No.: 2; Corrective Action: Taken or Planned: DRR, in coordination with FBDS governance bodies, will ensure that the PMO and DRR Contract Oversight coordinate with CACI to provide appropriate validation and acceptance documentation for ensuring that significant contract requirements were met and there is a sufficiently documented acceptance that FBDS transition-related activities have been successfully concluded.; Expected Completion Date: 8/31/2017; Monetary Benefits: No; Resolved;a Yes or No: Yes; Open or Closedb: Open;

Row 3; Rec. No.: 3; Corrective Action: Taken or Planned: DRR, in coordination with FBDS governance bodies, will assess and finalize Contract Oversight FBDSrelated guidance. DRR will also review and revise the guidance following the placement of ServiceNow in the production environment.; Expected Completion Date: 7/31/2017; Monetary Benefits: No; Resolved;a Yes or No: Yes; Open or Closedb: Open;

Row 4; Rec. No.: 4; Corrective Action: Taken or Planned: DRR, in coordination with FBDS governance bodies, will implement a project management framework reflecting industry best practices, to include a project management plan and other documents for ServiceNow and other future FBDS-related projects as appropriate.; Expected Completion Date: 9/30/2017; Monetary Benefits: No; Resolved;a Yes or No: Yes; Open or Closedb: Open;

Row 5; Rec. No.: 5; Corrective Action: Taken or Planned: While the scalability of FBDS appears to be sufficient to meet near term resolution projections, DRR will conduct a feasibility study to identify the limits to which the system can be scaled. This study will use actual results from the FBDS processing and loading of data for the failed Doral Bank. Additionally, multiple hypothetical resolution scenarios will be assessed to identify the current scalability of FBDS and what measures CACI and FDIC could take to enhance its scalability.; Expected Completion Date: 12/15/2017; Monetary Benefits: No; Resolved;a Yes or No: Yes; Open or Closedb: Open;

Row 6; Rec. No.: 6; Corrective Action: Taken or Planned: The FBDS PMO has asked CACI to provide a Process Improvement Plan for the program. Once received, the FBDS Program Team will develop a strategy and guidance for reviewing FBDS processes, approving and implementing needed improvements, and communicating results, to include establishing and implementing business rules to support the automated archiving capabilities of FBDS.; Expected Completion Date: 12/15/2017; Monetary Benefits: No; Resolved;a Yes or No: Yes; Open or Closedb: Open;

Row 7; Rec. No.: 7; Corrective Action: Taken or Planned: DRR has drafted and will finalize guidance through the FBDS Executive Board for reviewing contract performance metrics and revising the metrics and related service level agreements as appropriate.; Expected Completion Date: 6/30/2017; Monetary Benefits: No; Resolved;a Yes or No: Yes; Open or Closedb: Open;

a Resolved – (1) Management concurs with the recommendation, and the planned, ongoing, and completed corrective action is consistent with the recommendation.

(2) Management does not concur with the recommendation, but alternative action meets the intent of the recommendation.

(3) Management agrees to the OIG monetary benefits, or a different amount, or no ($0) amount. Monetary benefits are considered resolved as long as management provides an amount.

b Recommendations will be closed when the OIG confirms that corrective actions have been completed and are responsive.

[End of Table]

[End of Report]