DATE:	May 18, 2005
MEMORANDUM TO:	Stephen M. Beard
	Deputy Assistant Inspector General for Audits
FROM:	Fred S. Selby
	Director
SUBJECT:	Draft Report Entitled , The FDIC’s New Financial Environment (NFE) Testing
	(Assignment Number 2005-008)

DOF appreciates the ongoing review work performed by the OIG and its recommendations to
enhance the success of our New Financial Environment (NFE) implementation effort. Specific
to this report, we thank OIG and KPMG for their feedback on opportunities to improve our
testing process and results.

We are pleased to note that the NFE core financials component (consisting of 14 PeopleSoft
modules/tools of functionality and 23 legacy systems renovated to accommodate changes
brought about by NFE) of our overall project went into production on May 2, 2005. This “go-
live” decision was made only after an extensive discussion and analysis of implementation risks
against processes and activities in place to address or mitigate those risks. Three “Go/No-Go”
Decision Checkpoints in April provided the focus for ensuring our decisions to proceed were
supported by the necessary progress along our implementation path and input from critical
business and technical parties. These Checkpoints targeted critical readiness activities with an
evaluation of input from multiple sources, to include a comprehensive Production Readiness
Review (PRR) for all systems within the NFE implementation (core modules and legacy),
business sign-off of User Acceptance Test results, system performance testing, business
operations readiness feedback, implementation checklist progress for both core and legacy
systems, and data conversion progress to name a few.

While we recognized all along that going live with a systems implementation of this magnitude
and complexity is an inherently risky venture, we firmly believe the sound project management
and implementation processes that have guided both our development efforts and this
implementation were instrumental in successfully mitigating these risks. As noted above, the
control framework we designed and followed afforded a high degree of confidence that a “go
live” decision was completely appropriate under these circumstances. Risks that were identified
as “show stoppers” were given special scrutiny and only when a knowledgeable, multi-
disciplinary team agreed that such risks were being well managed or properly contained did we
agree to move the new or renovated systems into production. At least to date, it appears this
decision was well-founded and, with the benefit of hindsight, the utility of further testing does
not appear to have been worth the likely cost and organizational impact of system
implementation delays.

MANAGEMENT DECISION

OIG Recommendation: Review the risks identified and develop a risk resolution and action
approach in accordance with the risk mitigation procedures outlined in the NFE risk management
plan.

Management Response: As noted above, we appreciate the feedback provided by the OIG on
our testing efforts. It was helpful to receive this input as issues were identified and developed as
it allowed for a more timely assessment of potential exposures and actions needed. Management
concurred with the recommendation to review the risks identified in the draft report and develop
a risk resolution and action approach. As OIG findings/recommendations were received, NFE
project management reviewed and discussed the input in light of overall project risks and other
mitigating efforts. Using the Summary of Findings table in the OIG draft report as a guide, we
have developed a risk assessment matrix to summarize the conclusions reached from our pre-go-
live internal assessment of the various conditions identified by OIG. This matrix is included as
an Attachment to this response.

We have also considered some of the applicable risk/finding points raised in planning for the
remaining components of the overall NFE implementation effort and have taken alternate actions
where beneficial and appropriate.

FINDINGS

In addition to our response to the specific OIG recommendation, we offer the following general
feedback on the four finding areas noted in the OIG draft report. These comments are included
to lend additional perspective on the conditions, causes, and effects cited by the OIG in its report.

Finding 1: Production Simulation Tests Needed

NFE underwent extensive testing over a ten month period starting in June 2004 and ending in
early April 2005. The testing was divided into three distinct phases, System Integration Test
(SIT), Independent Test, and User Acceptance Test (UAT). The five month SIT encompassed
the set up and execution of all NFE modules and interfaces. SIT tests were primarily conducted
by our systems implementer, and the results were reviewed and approved by the NFE team and
by business users. During SIT, over 640 test scripts were executed for the core modules and
over 1,500 for the supporting renovated legacy systems. SIT was followed by a three-week
Independent Test effort, which was conducted by FDIC’s Division of Information Technology
independent testing group. The Independent Test team reviewed and validated the results of SIT
and also conducted additional testing on high-risk NFE functions and interfaces (over 650 test
scripts were executed). The Independent Test effort was followed by a four month UAT, which
was conducted by business users, who specified the business activities that they deemed
necessary for additional testing. UAT included three separate test passes, in which some
functions were tested multiple times, using scripts that simulated actual business transactions.
Over 620 test scripts were executed in this testing phase. As with SIT and Independent Test,
UAT included tests of NFE interfaces, online processing, and batch processes.

Additionally, the NFE team conducted a series of performance tests in the actual NFE production
environment starting in December 2004 and ending in March 2005. The performance testing
encompassed five cycles covering on-line transactions, batch processing, query and report
processing, critical processes (e.g., accounts payable), and supplemental payments.

In summary, NFE testing has been comprehensive in terms of its scope, functional and technical
coverage, and business user participation. It has included tests in the quality assurance and
production environments and has devoted special attention to areas such as interfaces and high
volume processing where risks were deemed to be high. Tests of all critical functional areas
were executed multiple times. The tests uncovered numerous problems, which were
subsequently fixed and re-tested. A great deal of confidence and assurance in the viability and
readiness of the system was gained in the course of this ten month series of tests.

Nevertheless, management acknowledges that an additional round of production simulation tests
as recommended would have provided yet a further level of assurance that the system and
associated processes would indeed perform as required when placed in production. We did not
believe, however, that the additional level of planning, preparation, and coordination that would
be required for such a test would be justified by the reduction in risk that it would offer. The
NFE team estimated that it would take from six to eight weeks to plan for the test, prepare the
environment, coordinate the business groups, and execute the test. The resultant test would, for
the most part, cover conditions already tested multiple times and would be conducted by the
same group that had recently completed the four month UAT.

While we continue to face challenges as additional system issues are uncovered in production,
we know from the extensive interviews we conducted in planning for this implementation that
such occurrences are the norm. In the end, we believe we struck an appropriate balance between
engaging in extensive and in many cases redundant testing and our desire to move the systems
into production as quickly and cost effectively as possible.

Finding 2: Effectiveness of Accounting Verifications for Month- and Year-End Closings

Accounting verifications of business process results occurred for all processes during UAT and
were a fundamental part of validating test results. Accounting verifications for month- and year-
end closings were tested by the NFE team and approved by business users during SIT and again
in UAT. Knowledgeable staff that have had significant experience dealing with the closing
process in the general ledger area as well as financial reporting area were brought in to validate
the effectiveness and accuracy of the closing process.

During testing, the team reconciled internal module activity (for example, accounts payable,
accounts receivable, asset management, etc. to the general ledger). We used delivered
PeopleSoft reports as well as constructed special queries to assist us in this reconciliation
process. Because we took an end to end approach with our testing, with many scripts starting in
a legacy system, we also reconciled activity from the legacy feeder systems to the PeopleSoft
general ledger.

In testing the monthly and yearly closing processes, we were careful to include testing our ability
to process accruals and reversals, to perform complex allocations, and to unclose/process
adjustments/re-close the period. We verified that our closing rules worked properly. We tested
both the fiscal and calendar year closing processes.

In sum, we believe that we thoroughly tested our ability to properly close the accounting periods
in NFE. We do, however, recognize that we still need to further refine our month-end and year-
end closing processes and supporting procedures. We will be developing the monthly closing
checklist and a comprehensive year-end checklist as we have done in prior years, taking into
consideration the changes inherent in the PeopleSoft system.

Finding 3: Defect Consolidation, Traceability, and Documentation

FDIC is using a centralized and controlled defect tracking system (Test Director) not only for NFE,
but for all interfacing systems as well. Test Director Cr1)) allows the test activities to be partitioned by
domains, and within domains, by projects. NFE uses TD domains and partitions to separate test
activities by system and, if deemed necessary, by test phase (e.g., UA’I). The purpose of this
separation is to enable teams to focus closely on the test defects specific to the current phase of testing
for a particular system or related modules. Typically teams are dedicated to a specific system or
module during a test phase.

Combining all NFE-related defect reporting into a single domain and project could be
counterproductive in that it would require each team to filter out a large number of defects not likely
relevant to their current area of focus. While it would enable the use of the TD Find Similar Defect
function, the project team believes the tradeoff between the limited utility of that function and the need
to constantly filter out defects from different systems and test phases was not worthwhile, particularly
given the complexity of the NFE testing cycles. It should be noted that the Find Similar Defect
function is limited to comparison of keywords entered into the defect description field. The team also
believes that the relatively small number of defects (once limited to a system and a test phase) and the
expert knowledge of its test team members has been sufficient to find defects of a similar nature or
that have reoccurred. We believe the risk is relatively low that significant numbers of recurring or
similar test defects have gone undetected.

We do recognize that hi-directional tracing of test defects from their test origin to change request
documentation can be improved and would be of benefit to the testing effort. Going forward,
management will explore the inclusion of a standardized field in TD to cross-reference the test script
in which the defect was identified. It will also include relevant defect identification numbers in
change requests as appropriate. These initiatives will be undertaken for future NFE test phases, but
will not be retroactively applied to test phases that have been completed or that are now underway.

Finding 4: Effectiveness of Test Activities Performed

The scope of NFE UAT testing was determined by each business area after consideration of its
business priorities, transactional and control risks, and the adequacy/coverage of previous System
Integration Testing (SIT). These business owners and their subject matter experts drove the
identification of original requirements and actively participated in multiple process design sessions.
They were included in UAT test planning and. prior to testin2, reviewed the test scripts and expected
results. The end-users of the system conducted the UAT test script execution activity, and both the
NFE team members and the business areas reviewed and signed off on all UAT test script results.

Testing was necessarily risk-focused and relevant to important FDIC business functionality. We
began UAT in December of 2004 with a comprehensive initial test plan, but we recognized at
that time that additional business test scenarios would be identified as testing progressed and the
users became more familiar with how the system was working. As expected, we identified many
issues in testing that needed to be fixed, identified additional testing required in certain areas,
and repeated end-to-end process testing in several critical functional areas to ensure readiness for
go-live. We do not believe that a significant number of scenarios have been missed or that major
gaps existed in test coverage. We took under advisement the particular scenarios and areas of
concern noted in the OIG recommendations as we reviewed final test priorities.

We do intend to review our test development processes and document retention and control
practice. This review will be undertaken for future NFE test phases, but will not be retroactively
applied to test phases that have been completed or that are now underway.

Thank you for the opportunity to provide feedback on your draft report. If you have any
questions regarding our response, please contact Karen Hughes at (202) 416-7201.

Attachment

cc:	Steven App
	Karen Hughes
	Mark Brenneman
	James Anderson
	William Ortiz
	Connie Bundle

---

Attachment

NFE Risk Assessment Matrix (based on pre-implementation assessment of KPMG/OIG-identified risk areas)

Finding	Condition	Level of Risk Assigned by KPMG/OIG as of Feb 2005	Level of Risk Assigned by NFE as of mid-April 2005	Resolution and Action
1	Inadequate user training	High	Low	User Acceptance Testing has been completed and the intended scope of coverage was successfully achieved. Timeliness may have been improved if users were more familiar with the application prior to starting testing. Testers for the future releases are receiving more pre-UAT exposure to the modules; this will benefit those UAT test efforts.
	Inadequate chart of account tests	High	Low	Chart of Account and Combo Edit testing was extensive, especially given the legacy interface activity. Post-implementation chart of accounts problems will be handled via normal problem resolution procedures.
	Business processes were not tested sequentially fro start to finish without interruption	High	Medium	All key business processes were tested by the end of UAT on 4/15/2005. UAT testing was sequential, but, as the finding indicates, was not always uninterrupted. NFE post- implementation plans are to gradually start up and monitor business processes ever the first 30-60 days of operation. All key processes, with the exception of quarterly and yearly processes, will have been completed by June 30, 2005.
	UAT did not include all production simulation testing.	High	Medium	UAT and performance testing simulated the majority of production work. Production readiness validation for all NFE and legacy components and systems is part of our cutover work. Not having unscripted days of heavy volume activity in a pre-production set-up does introduce risk daily operational meetings post-implementation will help ensure start-up problems are quickly identified and resolved.
2	Documented evidence did not exist for accounting-based verifications performed for month- and year-end closings during SIT.	Medium	Low	Month- and year-end closing activities have been fully tested and verified during UAT.
	UAT month and year-end tests planned does not provide requirements for accounting verifications and reconciliations.	High	Low	Month- and year-end closing activities have been fully tested and verified during UAT. Post-implementation, Accounting Operations will be building new PS-specific steps as appropriate into existing month-end and year-end closing guidelines.
	Scripts for identifying or researching un-posted transactions and posting errors were omitted.	Medium	Low	Anomaly transactions were included within the scope of our UAT efforts.
	CQMS did not perform an independent review of the month-aid and year-end test processes.	Medium	Low	Month- and year-end closing activities have been fully tested and verified during UAT.
	Independent test activities excluded UAT validation activities	Medium	Low	UAT validation was performed by the NFE team and NFE business owners.
3	A centralized and controlled defect tracking system has not been established.	Medium	Low	Defects will continue to be tracked using Test Director during post-implementation and for NFE maintenance.
4	UAT documentation is not effectively organized to independently verify the accuracy and completeness of tests performed for NFE business processes.	Low	Low	Test documentation was effectively organized for purposes of managing and conducting the test but not necessarily for independent verification.
	Test activities and scenarios were missing from sampling test scripts applicable to compliance requirements of the FFMIA.	Medium	Low	The scope of our test scenarios and activities provided appropriate test coverage, especially in the higher risk functions.
	There are no plans at this time for testing about 50-70 customized FMS reports currently generated by the Walker GL.	Medium	Low	An extensive effort was conducted to identify needed reports, whether delivered vs. custom, or operational vs. management. A substantial number of operational and management reports are part of the NFE May 2 go-live effort. The subset of customized management reports not included in the go-live date will be tested prior to release to production environment.