by Phil Martin
Reconnaissance, or information gathering tools
Vulnerability scanners
Fingerprinting tools
Sniffers or protocol analyzers
Password crackers
Network security devices such as scanners, proxies and vulnerability management suites
Wireless security tools
Reverse engineering tools such as assemblers and disassemblers, debuggers and decompilers
Source code analyzers
Vulnerability exploitation tools
Security-oriented operating systems
Privacy testing tools
Test Data Management
While we can test individual input fields and ensure the data is properly stored all day long, validating that an application functionally works at the business level requires us to use a good set of data. For example, if we want to test performance of a search function, then we need to make sure the database has the correct tables populated, and with enough data to simulate the appropriate volume. Performing a search on a table with a single record will always be fast, but once we load that table with one million records, the results could be alarmingly slow. Additionally, if our test data has orphaned records and field values that are complete gibberish, then our testing will suffer as well.
The easiest way to load our test environment with good data is to simply export production data. From a security perspective, this is a terrible idea. Not only do we expose PII, PFI or PHI data to anyone with access to the test environment, but we are allowing people to view sensitive details about real customers that may not belong in those categories. I once had a testing team that decided to export production data to a test environment that had very few access controls for internal employees. The application dealt with retail stores, and one day one of my employees walked in to a store and let an employee there know that she knew what the actual cost for a product was and demanded a lower price. Imagine the ear full I received the next day when the customer called me to express his displeasure using a few…choice words, shall we say? And rightfully so – that data should never have found its way in such a raw manner to a test environment.
That leaves us with two choices on how to securely generate test data – either generate the data from scratch or import production data with a proper amount of scrubbing, masking and obfuscation applied. The alternative approach is perfectly legitimate, but risky – we must ensure that ALL sensitive data is scrubbed and anyone with access to that environment cannot reverse-identify production data.
The other approach – generating data from scratch – will require a sophisticated software package to carry out. This may be a home-grown version or be purchased from a third-party. There are a number of shrink-wrapped packages that can look at an existing database and capture relationships and data patterns. From this, such a product can generally do a fairly decent job of generating a large amount of quality data. Naturally, at time this will require extensive configuration to get the best quality from the generated data. The amount of storage space available in the test environment is often a limiting factor in how much data can be generated. Although it takes considerable time to setup, the benefits of using some type of a test data management system are the following:
Keeps ongoing data management costs low with smaller data sets that costs less to persist.
We are assured that sensitive data is not being disclosed.
We are not taking risks by importing data from a production environment.
Reduces the likelihood of insider threats and fraud.
When using test data, we often refer to executing synthetic transactions, or transactions that have no business value other than being used to test with. We can execute two different types of synthetic transactions – passive and active. A passive synthetic transaction does not result in the creation of more data. For example, we can query a database to find all inventory older than 90 days. This results in an in-memory data set which is tossed as soon as we are done with it. The database has not been altered in any way, and this can be seen as a ‘one-time’ transaction. An active synthetic transaction will result in some type of database alteration or update. As an example, if we simulate purchasing a product in our test environment, the test database now has a new purchase record, and the inventory has been depleted by a single item. The reason we are calling out passive vs. active synthetic transactions is that if we are not careful, an active synthetic transaction might be mistaken for a real transaction and be processed by the production system as such. For example, our test system creates a new purchase, and it leverages the same queuing bus as the production environment. The production system sees a new purchase and charges a customer’s credit card – whichever customer happened to have the same unique ID as the customer in the test system. Ooops! Now we have a ticked-off customer who will never shop with us again because they deleted their account.
Defect Reporting and Tracking
If we don’t have an official process around reporting, tracking and addressing defects, we will never have a stable and secure product. A defect is any behavior of an application that does not meet the intended functionality, regardless if the intended functionality has been documented or not. I can’t count the number of arguments I have witnessed between development and product teams when a developer uses the ‘It wasn’t documented’ defense to explain why he implemented buggy code. Yes, it is the responsibility of the product owners to properly document expected behavior, but there will always be ‘assumptions’ on both sides, and we must use common sense and not take the easy way out when it comes to quality software.
Defects can fall into five categories – bugs, flaws, behavioral anomalies, errors and vulnerabilities, as shown in Figure 121.
Figure 121: Software Defects
Reporting Defects
The most important rule when reporting a defect is to ensure the development team has enough information to reproduce the problem. If a developer cannot make the problem surface at-will, then she has little hope of ever truly fixing the problem. Let’s walk through the various defect report attributes that should be provided.
A unique defect identifier should be generated and associated with each and every new defect reported, even if the defect is a duplicate of a defect already reported. A common mistake with green support staff and product owners is to try and group multiple issues in a single defect – every unique issue deserves its own unique identifier. Not only does this make it easier for the developer to figure out the core issue, it increases the chance that a given issue will be addressed and deployed in less time as it does not have to wait for all problems in a single defect to be resolved.
The title should be concise yet descriptive. Instead of saying “When searching for ‘Quality One Nissan’ immediately after login the results are duplicated when sorting by inventory date“, simply state “Search results are duplicated when sorting by date”. A common problem is that a reported issue will morph into a different one, yet the original title never changes. Always ensure a defect title applies to the actual problem being reported and worked on.
The description should be a brief summary. The verbose title we just mentioned would be a great description for the defect.
Always include detailed steps that shows the developer how to reproduce the problem. As we stated, if a defect is not reproducible, it will not ever be fixed. The following is a good example of detailed steps to reproduce our search problem:
1) Provide a valid username and password and login.
2) Click on the ‘Search’ button at the top right.
3) Enter ‘Quality One Nissan’ into the search field.
4) Click the ‘Go’ button.
5) The search results will appear correctly with the default sort set to ‘By Name’.
6) Click the Sort drop down list and select ‘By Date’.
7) The list will reload.
8) Notice that items are duplicated. In my test the ‘2017 Nissan Versa’ item
showed up twice as the first two results in the list.
After a thorough description of how to reproduce the problem, we now need to describe the expected results. Without this bit of information, a developer might not understand how the current behavior is incorrect. The best way to do this is to include the requirement identifier from the original requirements traceability matrix, or RTM. It might be that the original requirements were incorrect, in which case our defect has now turned into a change request. If an RTM is not available, or the requirement was never documented, this is the place to spell out the desired behavior. In our example we might say ‘Items should never be duplicated regardless of the selected sort method.’
‘A picture is worth a thousand words’ goes the saying, and as far as I know this is never truer than with software requirements and defect reports. A screenshot of the result of the defect is infinitely better than a textual description for two reasons:
It visualizes the problem for the development team.
It provides proof that the reporter was able to reproduce the problem.
If a screenshot might reveal sensitive information, then care must be taken to redact or hide that information from the image before including it in the defect report.
Defects should be categorized by type indicating whether it is associated with functionality or security. In larger products, defects should be heavily categorized and prioritized so that the most important are worked first. Figure 122 provides an example.
The environment – such as development, test, staging or production - in which the defect was discovered should be recorded, along with the following related information:
If the issue could be reproduced in other environments.
If the environment was internal or external.
The OS and service pack in the environment.
The web address if the software was a web application.
Figure 122: Defect Categorization (Types)
The build number represents the version of the software that has been deployed and should be reported with the defect. This serves two purposes – it allows us to see if the defect is specific to a single version, and it helps us to determine the RASQ for a given version based on reported security defects.
The tester name must be recorded so that the development team knows who to contact with further questions.
The reported on date and time should be accurately recorded so that the time to resolution can be tracked. The elapsed time for a defect to be resolved is an important metric when looking for ways to improve a process.
The severity of the defect is gauged by the reporter, which may or may not reflect reality. However, the reported severity is useful in triaging and prioritizing incoming defects until the severity can be validated or updated. Severity ratings are very qualitative, and often follow a pattern very similar to the following scheme:
Critical defects prevent the software from running as expected, affecting all users.
Major defects prevent some functions of the software to be unavailable, with no work-around possible.
Minor defects affect some of the business functionality, but users have a work-around and can continue to be productive.
Trivial issues do not affect business functionality, but a change could enhance the user’s experience. This is where we might find UI enhancement requests.
Aside from using categories and severity to prioritize a list of defects, simply setting the priority is also used. Priority best represents how quickly a defect needs to be addressed and is usually a qualitative value such as Mission Critical, High, Medium and Low.
The status of a defect is used to follow the defect through a workflow, or a series of sequential steps. For example, the following is an example of a common workflow, with a defect’s status being updated with each step:
New
Confirmed
Assigned
In-Progress
Ready for Testing
Resolved
In addition, other non-linear states might be Reopened or Deferred.
Finally, as the ‘Assigned’ status might reflect, we need to record an assigned to value for each defect. This represents the development team member responsible for fixing the defect in code.
Figure 123: An Example Defect Life Cycle
Tracking Defects
As we noted when discussing the status attribute for a defect, each issue follows a pre-defined workflow that allows us to track the state of a defect at any given time. Defects should be tracked in a centralized system that all appropriate people have access to depending on their access rights. Without a single system to control defect flows, it will be impossible to reliably know the quality state of a product, and it will be difficult to assign issues so that they are not duplicated on multiple development team member’s to-do list. There are five core capabilities that any defect tracking solution should support – documentation, authentication, workflows, notification and auditing.
First the system must ensure that a defect is properly documented by requiring certain fields to be populated before a defect report is accepted. Every organization and product is unique and have different requirements, and therefore the tracking solution should support custom fields, even to the point of forcing issue reporters to populate required, custom fields.
Second, the solution must provide some type of authentication mechanism so that fields reflecting the identity of the reporter are automatically populated. This prevents errors, allows us to properly implement auditing, and provides the ability to track user activity as they work on a defect.
A customizable workflow is an absolute necessity, as every organization works differently. Workflow is represented by the various statuses that a defect passes through, and reporting will heavily use this capability. Figure 123 Shows an example of such a life cycle.
The fourth capability any decent tracking solution should provide is to notify users as defects move from status to status. This prevents a defect from being ‘stuck’ in a specific state due to the ‘assigned to’ person not knowing it is ready to be worked on. Notifications most often occur via email, text messages or alerts within the defect tracking solution itself.
Finally, a good tracking solution will always provide the ability to record user actions as they occur by creating an audit trail. This information must result in useful reports that are secured to the appropriate access levels.
Impact Assessment and Corrective Action
There is a saying that goes, “If everything is a priority, then nothing is.” This pithy statement reflects the reality that if we don’t bother prioritizing a list, then we will have no idea what to do next, and inaction sets in. All defects must be ordered by priority and severity, and occasionally by category. When using agile methodologies, defects must be added to the backlog, which itself represents a prioritization.
Before a defect is placed into a backlog or assigned to the development team, it must be triaged. This activity is named after the actions that an emergency medical ward goes through when experiencing a sudden influx of patients. In that scenario, medical personnel may evaluate the injury and choose one of three actions – turn the patient away, defer treatment or treat immediately. When triaging defects, we also have three options – replace the software to avoid the risk, defer treatment till a future version, or address the defect immediately. If you have some exposure to the information security world, these options map to avoid, transfer and mitigation. Figure 124 shows the various options.
Impact
Hospital
Software
InfoSec Term
Serious
Treat immediately
Fix in an emergency or current release
Mitigate
Moderate
Treat after serious injuries have been addressed
Fix in a future release
Transfer
Low
Turn patient away to another facility or department
&nb
sp; Replace the software
Avoid
Figure 124: Corrective Actions
Notice that none of the options are called ‘ignore’. Ignoring a risk is not a valid option, although purposefully deciding to do nothing is acceptable as long as the business is aware and agrees to such an action. If we ignore a risk, we don’t even bother evaluating whether or not to address it.
In this book, we often refer to various ‘environments’ in which our software under development can be deployed to. While every company is different, there are four different environments you should be familiar with, as shown in Figure 125.
Figure 125: Various Deployment Environments
The development environment is the development team’s sandbox to play in, and they often have complete say over what is deployed into this environment. Never expect an application to remain stable here, as things are always in flux, and developers will often try out new strategies in this environment to see how they work.
The testing environment is controlled by the testing team, and only builds that are ready to be tested are deployed here.
The user acceptance environment, sometimes called the staging environment, is normally controlled by the business owner, and only release candidates are usually deployed into this environment. It should mirror the production environment as closely as possible in terms of security, performance and scalability, but often does not due to the cost involved. If a lesser infrastructure is used for the staging environment, then the difference between staging and production must be a known and quantitative value. Why? Because will need to be able to extrapolate measured performance and scalability to the production environment before we get there.
