DevOps puts an intense focus on automation - taking humans out of the loop whenever possible to allow frequent, incremental updates to production systems. However, thorough application testing often has multiple components - much of this can be automated, but manual testing is also required. This is inconvenient and not "DevOps-y," but is unfortunately an unavoidable requirement in the real world. In addition, managing these multiple sources of application vulnerability intelligence often requires manual interaction - to clear false positives, de-duplicate repeated results, and make decisions about triage and remediation.
Axway has rolled out an application security program that incorporates automated static and dynamic testing, attack surface analysis, component analysis, as well as inputs from 3rd parties including manual penetration testing, automated and manual dynamic testing, automated and manual static testing, and test results from vendors providing test data on their products. Automation has allowed Axway to increase the frequency of web application testing, thus reducing the cycle time in the application vulnerability "OODA loop." Moving beyond the identification of vulnerabilities, Axway has deployed ThreadFix to automatically aggregate the results of the automated testing and de-duplicate findings. 3rd party penetration testers are also finding vulnerabilities and reporting them in reasonably structured CSV files requiring Axway to convert this manual test data and incorporate it into the aggregated vulnerability model in ThreadFix. Centralizing this pipeline allows for metric tracking - both for the application security program as a whole as well as on a per-vulnerability-source basis. This automation and consolidation now covers 50% of Axway's application vulnerability review process - with plans to extend further.
This presentation walks through Axway's construction of their application security-testing pipeline and the decisions they were forced to make along the way to best maximize the use of automation while accommodating the reality of manual testing requirements. It then looks at how this testing regimen and the associated automation have allowed them to impact deployment practices as well as collect metrics on their assurance program. Finally, it looks at lessons learned along the way - the good and the bad - and identifies targeted next steps Axway plans to take to increase the depth and frequency of application security testing while dealing with the deployment realities placed on them to remain agile and responsive to business requirements.