Security Testing of Custom Software Applications

Tools and techniques for testing applications - excerpted from Secure and Resilient Software Development by Mark Merkow and Lakshmikanth Raghavan

1 2 Page 2
Page 2 of 2

Automated tools tend to report a high number of false positives. Sometimes it will take an organization several months to fine-tune the tool to reduce these false positives, but some level of noise will always remain in the findings. Source code analyzers are poor at detecting business logic flaws. Some of the other types of attacks that automated analysis cannot detect are complex information leakage, design flaws, subjective vulnerabilities such as cross-site request forgery, sophisticated race conditions, and multistep-process attacks.

In a research paper written by James Kupsch and Barton Miller of the University of Wisconsin, the authors presented the results of their efforts to evaluate and quantify the effectiveness of automated source code vulnerability assessment tools by comparing such tools to the results of an in- depth manual evaluation of the same system.3 The key findings were the following.

  • Of the 15 serious vulnerabilities found in the study, Fortify Software found six and Coverity only one.
  • Both Fortify and Coverity had significant false positive rates, with Coverity having a lower false positive rate. The volumes of these false positives were significant enough to have a serious impact on the effectiveness of the analyst.
  • In the Fortify and Coverity results, they found no significant vulnerabilities beyond those identified by the study. Fortify Software and Coverity are two of the commercial automated code analyzers discussed below.

    8.6.2 Commercial and Free Source Code Analyzers

    Here is a sampling of some of the available source code analyzers, both commercial (with dedicated support) and free or open-source software. Commercial—Multilanguage

    Commercially available multilanguage source code analyzers include the following.
    • Armorize CodeSecure—Appliance with Web interface and built-in language parsers for analyzing ASP.NET, VB.NET, C#, Java/J2EE, JSP, EJB, PHP, Classic ASP, and VBScript (http://www.armorize. com/?link_id=codesecure)
    • Coverity Software Integrity—Identifies security vulnerabilities and code defects in C, C++, C#, and Java code (http://www.coverity. com/products)
    • Compuware Xpediter—For mainframe-based applications; offers analysis of COBOL, PL/I, JCL, CICS, DB2, IMS, and other popular mainframe languages ( xpediter.asp)
    • Fortify 360—Helps developers identify software security vulnerabilities in C/C++, .NET, Java, JSP, ASP.NET, ColdFusion, Classic ASP, PHP, VB6, VBScript, JavaScript, PL/SQL, T-SQL, and COBOL, as well as configuration files ( products/fortify-360)
    • Klocwork Insight and Klocwork Developer for Java—Provides security vulnerability and defect detection as well as architectural and build-over-build trend analysis for C, C++, C#, and Java (http://
    • Ounce Labs—Automated source code analysis that enables organizations to identify and eliminate software security vulnerabilities in languages including Java, JSP, C/C++, C#, ASP.NET, and VB.NET ( Open Source—Multilanguage

    Here are a few of the open-source products for source code analysis.
    • O2—A collection of open-source modules that help Web application security professionals maximize their efforts and quickly obtain high visibility into an application's security profile with the objective of "automating application security knowledge and work- flows"
    • RATS (Rough Auditing Tool for Security)—Can scan C, C++, Perl, PHP, and Python source code. ( rats.jsp)
    • YASCA—A plug-in-based framework for scanning arbitrary file types, with plug-ins for scanning C/C++, Java, JavaScript, ASP, PHP, HTML/CSS, ColdFusion, COBOL, and other file types; integrates with other scanners, including FindBugs, JLint, PMD, and Pixy ( .NET Support

    • FxCop—Free static analysis for Microsoft .NET programs that compile to CIL; stand-alone and integrated in some Microsoft Visual Studio editions ( bb429476%28VS.80%29.aspx)
    • StyleCop—Analyzes C# source code to enforce a set of style and consistency rules; can be run from inside Microsoft Visual Studio or integrated into an MSBuild project (http:// Java Support

    • Checkstyle—Besides some static code analysis, can be used to show violations of a configured coding standard (http://checkstyle.
    • FindBugs—An open-source static byte code analyzer for Java (based on Jakarta BCEL) from the University of Maryland (http://
    • PMD—A static rule set-based Java source code analyzer that identifies potential problems (

    Among the tools listed, we will examine in detail Fortify 260 as a commercial tool and O2 as an open-source tool.

    8.6.3 Fortify 360

    Fortify 360 provides the critical analytic, remediation, and management capabilities necessary for a successful, enterprise-class software security assurance (SSA) program.

    • Identification: Comprehensive root-cause identification of more than 400 categories of security vulnerabilities in 17 development languages
    • Remediation: Brings security, development, and management together to remediate existing software vulnerabilities
    • Governance: Monitors organization-wide SSA program performance and prevents the introduction of new vulnerabilities from internal development, outsourcers, and vendors through automating secure development life-cycle processes
    • Application defense: Contains existing vulnerabilities so they can't be exploited
    • Compliance: Demonstrates compliance with government and industry mandates as well as internal policies4
    • The architecture and context of how Fortify 360 is deployed and operated is shown in Figure 8.2.

      Fortify 360's static source code analyzer (SCA) provides root-cause identification of vulnerabilities in source code. SCA is guided by a comprehensive

      Fortify 360 architecture

      Figure 8.2 Fortify 360 Architecture

      set of secure coding rules and supports a wide variety of languages, platforms, build environments, and integrated development environments (IDEs), such as Eclipse, Visual Studio, and others.

      Figure 8.3 is screenshot of the results of a Fortify 260 source code analysis done on WebGoat, a deliberately insecure J2EE Web application that is maintained by OWASP and is designed to teach Web application security lessons. O2—OunceOpen

      O2 originated from work conducted by the OunceLabs Advanced Research Team (ART). O2 aims to push to the limit the power of multiple static analysis engines. These tools have been developed by security professionals for security professionals and are intended to help automate a security consultant's brain.
      Fortify audit workbench

      Figure 8.3 Fortify Audit Workbench

      Following is a list of O2 modules:

      • O2 Tool—XRules—O2's eXtended rules environment, which allows the execution and editing of complex security analysis work- flows
      • O2 Tool—SpringMVC—Support for Spring's Framework MVC
      • O2 Tool—RulesManager—Powerful viewer and editor for Ounce's Rules
      • O2_Tool_FindingsViewer—Powerful filter and editor for Ozasmt files
      • O2_Tool_CirViewer—View and create (for .NET) CIR (Common Intermediate Representation) objects
      • O2_Tool_SearchEngine—RegEx text search-based GUI
      • O2_Tool_CSharpScripts—Edit and debug C# scripts
      • O2_Tool_DotNetCallbacksMaker—Automatically create Ounce Rules for .NET callbacks
      • O2_Tool_FindingsQuery—Filter Ozasmt files using LAMDA-like queries
      • O2_Tool_JavaExecution—Write O2 scripts in Java
      • O2_Tool_JoinTraces—Join traces (e.g., .NET and Web and Web Services layer)
      • O2_Tool_Python—Write O2 scripts in Python
      • O2_Tool_O2Scripts—O2 scripts editor (includes O2 Object Model)
      • O2_WebInspect(PoC of Integrating Ounce's & WebInspect's assessment data)

      Figure 8.4 lists all the modules and their maturity to date.

      O2 modules

      Figure 8.4 O2 Modules

      Figure 8.5 is a screenshot of the results from the O2 source code analysis conducted on WebGoat.

      While we do not endorse or recommend any particular automated tool, we do recommend that all organizations perform an objective evaluation of

      O2 webgoat assessment

      Figure 8.5 O2 WebGoat Assessment

      available commercial software and free software to determine the best fit for their development language(s) and SDLC methodology. Organizations can also use a combination of tools to provide a high level of assurance in the security scanning process.

      Excerpted from Secure and Resilient Software Development by Mark S. Merkow and Lakshmikanth Raghavan and published by CRC Press. © 2010 Taylor and Francis Group, LLC. Reprinted with permission.

  • 1 2 Page 2
    Page 2 of 2
    FREE Download: Get the Spring 2019 digital issue of CSO magazine today!