Mitigating Risk Through Quality Assurance

RFG believes using outsourced application development services can expose enterprises to undesirable risks, including backdoors, data corruption and privacy, poor code quality, and viruses. To address these exposures, IT executives should develop processes and isolate applications and data in a manner to ensure code quality, data security, and privacy. Furthermore, IT executives will need to maintain internal control over acceptance, compliance, system, and volume testing. All these tests, and the associated test environments, should be under quality assurance (QA) supervision to ensure the sanctity and privacy of live data, or copies of customer data. Moreover, IT executives should perform all of these tests internally, and establish and enforce rigorous oversight and testing processes that provide quality control personnel with access to the necessary time and resources. Moreover, IT executives should provide these teams with access to outsourcer application developers and project managers, documentation, library versioning, time, and separated/secured testing facilities to validate that new applications meet the required standards and perform suitably.

Business Imperatives:

• Risk mitigation is an underlying concern forcing corporations to reevaluate how policies, procedures, and technologies are employed to reduce or eliminate exposures. Outsourced application development presents risks to the enterprise for which the enterprise remains accountable. Thus, IT executives must address concerns related to application development methods, how access to existing applications and data is extended, versioning control, and quality assurance. A formalized methodology should be used to inspect each version of delivered outsourcer code. IT executives should work with internal auditors and other executives to gauge exposure levels appropriately, develop a plan and timeline to mitigate risks based on compliance requirements and importance, and guarantee the quality and security of outsourced code while safeguarding enterprise information.
• Privacy and security concerns, especially for those corporations in highly regulated industries such as the financial and healthcare industries, require IT executives to take precautions to prevent outsourcers from gaining access to or corrupting customer data. Furthermore, IT executives need to verify that the code developed by outsourcers performs up to required specifications and is free from software that could potentially damage and/or expose other enterprise applications or data. IT executives should develop procedures that keep outsourcers and applications developed by outsourcers segregated until such time as their performance and quality characteristics can be confirmed by QA personnel.
• Outsourced developers should not have access to live or sensitive company data, and physical and electronic security should exist at the vendor’s facility to ensure that neither data nor code can leave protected areas. Furthermore, code testing (beyond unit and function tests) should always be performed internally regardless of the application being developed. Since QA teams will not control this process and need to rely on operations, user, and development testing staffs, QA should work with those units performing the testing to help validate completeness and thoroughness of databases, test criteria, and the tests themselves. IT executives should empower QA leaders to help establish, monitor, and enforce the oversight and controls necessary, both internally and at the vendor facilities, to reduce technology risks.

Assuring the quality of software code can be challenging, especially when developers lack familiarity with the business. Additional breakdowns can occur if development specifications are not rigorously documented, developer experience and skill sets are not validated, procedures are loosely defined, and security measures are not strict. Risks related to privacy and security are of particularly high priority these days due to compliance requirements from regulations such as the Gramm-Leach-Bliley Act of 1999 (GLBA), Health Insurance Portability & Accountability Act of 1996 (HIPAA), and Sarbanes-Oxley Act of 2002 (SOX). IT executives should ensure that development processes and QA testing are conducted in a clearly defined, documented, and auditable manner that is capable of delivering the required code quality and data security.

The first step in mitigating risk and assuring code quality and data privacy is to understand how existing development processes are conducted. As the enterprise likely has numerous application development projects underway internally and with more than one vendor, IT should establish a common development methodology, including libraries and tools. IT executives and QA teams should also incorporate procedural and technical evaluations into the examination process, looking at how closely outsourcers are able to conform to required access controls, deadlines, specifications, quality, and security metrics.

This assessment will presumably demonstrate that some elements are in compliance while others are not, and IT executives should attempt to determine the root causes of those differences. Inequalities may be derived from any combination of outsourcer capabilities, communications, development processes, and measurement metrics. Both the outsourcer and the enterprise should also have monitoring facilities and controls in place to facilitate development and catch problems early. This exercise should help IT begin to construct a list of best practices to concatenate, promote, and promulgate and the procedures necessary for proper communication, quality, and security.

Effective application development requires continuous amounts of formal and informal communication. This is especially the case when coordinating development efforts with geographically dispersed third parties, where there is often little opportunity to build and leverage informal relationships and exchanges. Moreover, English is often not the first language for many international software developers, and subtleties in definition, language, and usage commonly cause communication problems. Outsourcing applications that require industry and/or business knowledge add another layer of complexity to the task and provide another opportunity for miscommunication or outright failure due to unrealistic expectations.

IT executives should tightly control communications processes with outsourcers by formalizing communication schedules to include weekly discussions with application developers and biweekly or monthly reviews with project managers. These discussions should encompass overall progress, task clarifications, and time lines, and summaries should be provided in written form for confirmation and to prevent confusion. Outsourcers sometimes tend to perform only to the letter and not the spirit of the engagement contract, however, and therefore, IT executives should endeavor to incorporate all requirements and details fastidiously into the contract to prevent additional confusion, cost, and/or time overruns.

Beyond communication, proper oversight and controls are necessary to obtain the required level of code quality. Policies and procedures for the outsourcer, testing group, and QA team should be defined ahead of time based on best practices. Furthermore, automated technologies should be adopted to support collaborative and tracking efforts. Required capabilities should include centralized progress reporting, exceptions, issue tracking, suggestion and resolution, ownership responsibilities, standards adherence, and version control.

The role of the QA team is central in building the monitoring, process, technical, and quality controls that drive application development, establishing how the entities collaborate, and governing development and compliance efforts. The definition of these processes is often not clarified in the contract at the project’s inception, and some IT executives falsely believe that service level agreements (SLAs) are the only controls required. SLAs focus mainly on budget and costs issues, but ignore progress monitoring and quality measurements. Each development stage requires its own controls and monitoring to keep developers progressing properly and to ensure that applications and data are both physically and logically secured.

Sharing intellectual property with an outsourced provider presents unneeded risks to the enterprise whether that vendor is located domestically or internationally. One recommended method of coping with this challenge is to prevent sharing any sensitive data with third parties, if at all possible. One way to accomplish this is to perform all testing efforts internally. In fact, completing the entirety of the build and test cycle in-house not only helps keep enterprise data secure, but it also ensures quality. This topic will be expounded upon later in this note.

Physical security, system isolation, and technical security are also required to mitigate outsourcing development risks effectively. Obviously, both the outsourcing and in-house environments should have appropriate levels of restricted access to the areas, libraries, and systems being used for development. IT executives may wish to use and enforce lockdown procedures, and also have floppy drives removed, USB ports disabled, and printer access denied. Additionally, IT executives may wish to restrict some or all of the outsourcer development staff from accessing the enterprise system by allowing access to sensitive data only to particular employees, logging activities, and using thin client tools such as Citrix, Inc.s MetaFrame for data viewing.

Once applications have been functionally tested, application developers should be relinquishing code to a system build staff. This staff will be responsible for the build libraries and version control, configuration management, test data and inputs, and test systems. QA should be responsible for monitoring all the build and test activities and for certifying that all availability, code quality and functional specifications, documentation, exit criteria, privacy, scalability, and security issues are fully addressed. At a minimum, there should be integration, system, regression, stress and volume, and user acceptance tests, and exit criteria should exist for each. The development and execution of these tests can consume a significant amount of time, and it is important, therefore, that there be a test plan developed early in the project.

The plan should encompass all the elements and an adequate budget for hardware, software, and people, as well as test development and execution time. Projects frequently fail to stay on budget and schedule as a result of inadequately budgeted testing. Lastly, to minimize the risk exposure when bugs are encountered, there should be individuals from the development team and the test team associated with each coverage fix.

The QA group should be responsible for all change management, releases, and release management. Because QA ultimately has this responsibility, there are some organizations that have QA own the entire system build and test management staff. However, this is not a requirement and may not be desirable from a career standpoint. That is, similar to developers, testers should primarily develop application and system skills and over time the individuals in these positions may change roles. QA staff, on the other hand, should primarily have process and release/change management skills.

Some organizations make the mistake of having QA report directly to the development organization because they see QA as simply another application development life cycle step. This structure is a conflict of interest, and IT executives should keep the QA function separate from development and testing. This helps ensure that the proper level of business requirements are met, code access is granted, privacy and security concerns are addressed, and respect is afforded when additional requests are made by QA. RFG recommends that QA report to the operations organization and should be the gatekeeper protecting the enterprise from code released before its time.

RFG believes exposures quality, privacy, security, and other exposures exist whenever the enterprise undertakes application development efforts. However, those risks can be amplified when working with outsourcers who may lack business understanding, communication skills, and expertise. This is especially true if vendors are provided access to sensitive enterprise data. Properly addressing these concerns requires a combination of communication, oversight, processes, quality control, technology, and security mechanisms all of which should be based on internal best practices, established prior to the engagement, and vigorously enforced. IT executives should perform all quality control functions internally and ensure that QA reports to operations rather than application development to prevent any conflicts of interest. Moreover, IT executives should substantiate that the proper communication channels remain open and flowing throughout the engagement, and that quality control measurements beyond the standard service level agreement (SLA) budget and cost metrics are carefully defined in the contract.

RFG analyst Adam Braunstein wrote this Research Note. Interested readers should contact Client Services to arrange further discussion or an interview with Mr. Braunstein.