The Software Quality Debate Rages On

RFG believes the relationship between enterprise application software vendors and their customers is undergoing a metamorphosis. While not entirely complete, this morphing relationship is yielding more savvy IT executives who are perceiving software quality in a new light and who are requiring vendors to guarantee the integrity of their products. IT executives should not eschew challenging vendors on the terms and conditions (Ts & Cs) of their licensing agreements, especially when they appear to be egregiously one-sided. Moreover, IT executives should negotiate and renegotiate contracts to ensure inclusion of clauses holding vendors accountable for the quality of their productsbefore signing on the bottom line.

Business Imperatives:

• The new name of the software game is accountability and shared risk. Each year, organizations spend astronomical sums of money on application software and, often, they are forced to accept that software “as is,” is poorly tested, and not guaranteed to be reasonably free of defects. IT executives should demand that software licensing agreements reflect that vendors are stepping up to the plate and becoming accountable for the quality of their codeespecially as compliance regulations loom large.
• It will not be long before regulatory compliance will extend to the products developed and sold by enterprise application software vendors. IT executives and their line-of-business (LOB) and C-level counterparts should understand that, as it now stands, a source code error in a purchased application, which leads to unintentional misstatements by their company, does not exclude the liability of their company executives and officers. During 2004, as they re-engineer processes and prepare for new reporting requirements, IT executives should consider establishing a vendor oversight committee.
• Today, savvy hardware, middleware, and software vendors are beginning to understand that to become trusted partners to their customers, they will need to share in the risk/reward. Downtime, loss of business, and other complications due to faulty application code are outdated excuses and ones that should no longer be tolerated by LOB and C-level executives. IT executives should seek to work with senior executives at vendors with whom they are or would like to be doing business to ensure that the software code being delivered has passed internal quality tests and that the vendor has certified it as such. This includes, but is not limited to, the code being stable and containing no known errors that could render the application inaccessible, severely limited, or accessible, but malfunctioning.

In November 2003, RFG published the Research Note “Software Code Quality: Dotting the Is and Crossing the Ts & Cs.” In the Note, RFG scrutinizes several generic software contract Ts & Cs. The goal of the Note was to point out the potential traps or black holes that are written into software agreements that protect the vendor and are a caveat emptor for the customer. The Note espouses the opinion that IT executives should compel software vendors to take responsibility for the quality of their products and not be permitted to release faulty software with impunity.

Moreover, the Note was also a call to action for IT executives urging them to scrutinize vendor contracts and negotiate for the inclusion of “quality of code” language as well as challenge vendor claims forcing customers to purchase software “as is,” without the proper warranties in place. Moreover, the note urged IT executives to negotiate for amenities, penalties, and other protective clauses that mitigate enterprise risk and exposure.

Readers Respond

Along with the publication of the syndicated Note in the Analyst Corner, RFG received many responses from both its constituents and other readers. While many readers felt the Note was helpful, other readers had a different perspective. Some of the more critical responses, taken in aggregate, are listed below.

• Beta test customers rarely put the product through all the paces. They become beta testers for financial incentives and are not good sources of feedback when it comes to finding bugs. Bugs are actually found when putting software into production.
• Early adopters of new software releases run the risk of becoming beta test sites; therefore, IT executives should wait a reasonable amount of time after the release (i.e., a month or two) before purchasing to minimize risk.
• If vendors are pre-occupied with finding and fixing bugs, they will never be able to code new features or sell their products and ensure a steady revenue stream.
• It is impossible to determine every permutation of customer customization and then test each and every one.
• It is impossible to find every bug; bugs cannot be quantified.
• Software companies would be driven out of business if their products were being returned because of bugs in code.
• Software vendors who buy software from other vendors can easily fix any problems that arise. However, there are problems when IT executives, who do not know how to play the software game, think that everything is either black or white and if the software does not work, it must, de facto, be the fault of the vendor.
• The Research Note was one-sided in its approach to the relationship between the application software vendor and the customer.
• There is disagreement regarding RFG’s espousing of the opinion that the products developed and sold by software vendors could cost companies money and, therefore, these vendors should be accountable and pay a penalty. The contention is that the customer should also be responsible and penalized every time they call the vendor for support and the problem is the fault of the customer and not the vendor.
• Vendor testing of software code quality, fixing the bugs, and then retesting could create more bugs and become a never-ending cycle.

RFG Ripostes

While it is impossible for RFG to address every reader response, this remainder of this Research Note addresses most of the aforementioned points of contention. RFG concedes the point that there are times that customers’ IT environments may not be mature enough to handle all the aspects of the application software life cycle. Moreover, it is true that customers do make mistakes and do not properly read or understand manuals or misinterpret documentation on feature capabilities.

RFG has written in past Research Notes that it is a fallacy that problems only occur because of quality of code. Many other things can go wrong, including the customer not employing proper project/change management policies and procedures, poor client/vendor relations, or trying to customize the process or the code beyond its ability to be customized. Even basics can go wrong, such as poor definitions of requirements. RFG has entreated IT executives to ensure that they have internal IT governance processes in place, and that they understand how to appropriately implement and customize the application suites they purchase.

However, RFG maintains that customers have the privilege of making mistakes. For the most part, they are paying upwards of 15 to 22 percent of the original cost of the software in yearly maintenance fees, which means the vendor is receiving a steady annuity stream. Moreover, customers pay vendors for support and therefore have the right to contact them with whatever problems they have regardless of who is responsible for them.

RFG clients, who have been beta test sites, have, indeed, put the code through all the paces and have worked closely with vendors to correct code, and customize the product for a particular industry vertical. When all is said and done, the beta test customer has essentially enabled that vendor to sell the product throughout the industryeven to competitors. The problem with this is the financial incentives provided to those customers pale in comparison to the time and resources they dedicated and the intellectual capital they have contributed for little or no return on their investment.

The suggestion that a month or two is a reasonable amount of time for IT executives to wait before purchasing/implementing a new release falls short of best practices. To reduce risk, in the hope that waiting means a more stable solution, most IT executives wait from six months to one year, or moreand even then, oftentimes, the code is still buggy.

Many software licensing agreements/contracts that RFG reviews continue to reflect a bygone era. Usually they are anachronous and entirely one sidedin the vendor’s favor. Today, the name of the game is partnership and if vendor and customer are going to be mutual partners, then each must put some skin in the game and share in the responsibility of making the purchase and subsequent implementation successful.

Furthermore, the vendor has an obligation to provide as solid a solution as possible. Quality Guru, Philip B. Crosby, who published the best seller, Quality is Free, developed the “Four Absolutes of Quality Management,” which are quoted below from a United Kingdom Department of Trade and Industry (DTI) Web page.

• Quality is defined as conformance to requirements, not as ‘goodness’ nor ‘elegance’.
• The system for causing quality is prevention, not appraisal.
• The performance standard must be Zero Defects, not ‘that’s close enough’.
• The measurement of quality is the Price of Non-conformance, not indices.

Crosby also went onto develop the “Fourteen Steps to Quality Improvement.”

In most cases, enterprises are spending more on installing software than on the software itself. This could be upwards of four times the amount of the software. While it is very expensive for a vendor to release install non-buggy code, when the vendor does not provide good code, the customers’ costs can quickly escalate. However, based on RFG’s experience, software quality is sometimes sacrificed by the vendor to meet artificial deadlines and to add “must-have” features for prospective and important existing customers. Furthermore, development, marketing, and sales organizations in some vendor companies view the quality assurance group (QA) as a roadblock to release. Based on the experiences of RFG analysts, QA organizations in many software companies have a relatively low status.

Disruption to business and its processes can cost a company tens of thousands of dollars per hour. In many companies, CFOs have become enamored with outsourcing. IT executives understand that their departments could easily become downsized or entirely replaced. Therefore, they must take all necessary precautions to eliminate downtime and interruptionor risk being made redundant. RFG believes vendors should perform security tests such as adherence to the Common Criteria for Information Technology Security Evaluation from the National Institute of Standards and Technology (NIST). Adherence to such requests means that the vendor is certifying or guaranteeing its code, and will be accountable if software code errors are directly responsible for downtime or failures that interrupt business.

A previous, and companion, RFG Research Note entitled “Getting the Software Code Quality You Want,” discussed the two primary reasons why enterprises today are experiencing problems with the code quality of packaged application suites.

• Software suites are often designed and coded to satisfy a single customer (or customer sub-segment), and therefore is not generalized sufficiently to meet the individual needs of each customer.
• Vendors typically test their products under limited conditions, resulting in customers becoming beta test sites.

In the aforementioned Note, RFG admonished software vendors that are not monitoring defect statistics and have no process in place to pinpoint and deal immediately and effectively with code errors and mentions two cases in point. First, a New York Times article, entitled “Microsoft Reports Progress in Averting Computer Crashes,” reported that Microsoft Corp. found 1 percent of application software errors are responsible for 50 percent of the crashes experienced by users. (Paid Web site registration is required to view this article in its entirety.)

Second, in an article entitled “Larry Ellison’s One-Man Show,” BusinessWeek reported that when Oracle Corp. released its 11i E-Business Suite, industry analysts estimated there were from 5,000 to 7,000 bugs that required fixing. (Free Web site registration may be necessary to view this article.) Meanwhile, InformationWeek, in an article entitled, “Customers Clamor for Software Quality,” reported that Oracle finally acknowledged its error and was addressing the issue of the quality of its software.

Therefore, while it is difficult to identify all of the bugs, oftentimes this is more a function of the mentality of the people involved in, or influencing, the software lifecycle that leads to inadequate testing and not the nature of the challenge itself. As RFG has demonstrated, there are instances of a large percentage of bugs being generated by one group of programmers or a single programmer. This could be quickly and effectively dealt with if the vendor had correct quality controls in place.

RFG believes that enterprise application software vendors who rigorously quality test their code and partner with customers will emerge as champions over those who do not. Those vendors that do quality test their code should have no problem demonstrating to their customers and prospects that they have a rigorous system testing and release process, and that the code is stable and performs as claimed. Moreover, they will secure steady revenue and an annuity steam because satisfied customers will remain loyal and will buy additional products based on a vendor’s reputation for delivering quality software.

Finally, there is the issue of compliance. IT executives, along with other C-level executives and officers of enterprises, will soon be required to sign off on the accuracy of financial statements and other reporting instruments. It would be heinous if inaccurate reporting and misstatements made by C-level executives were, in fact, based on a software bug in an application that had gone undetected by vendor QA, or lack thereof. Savvy IT and C-level executives will compel their application software vendor partners to certify that the software code being delivered has passed internal quality tests, is stable, and contains no known errors that could limit or eliminate entirely the application’s accessibility or functionality and ability to perform. These executives do not want to be led away in handcuffs only to discover that a vendor-generated software bug was responsible for reporting errors.

RFG believes IT executives, their C-level and LOB counterparts, and the vendors themselves are beginning to understand that a company’s partnership with a few good vendors will be of greatest value to the organization. IT executives are becoming vocal with their IT vendors about the requirements of their organizations and compelling them to listen and take action or forfeit business. Moreover, IT executives are engaging legal counsel and knowledgeable third parties to negotiate for amenities, penalties, and other protective clauses in contracts that mitigate enterprise risk and exposure. IT executives should continue on this course, as in doing so they will continue to add value to the organization and elevate the role of IT. Application software vendors, who have not already done so, should take heed as those vendors who understand the value of partnership, quality, and shared risk/reward will ultimately prevail.

RFG analyst Maria DeGiglio wrote this Research Note. Interested readers should contact RFG Client Services to arrange further discussion or an interview with Ms. DeGiglio.