Standard operating procedure

A standard operating procedure (SOP) describing the overall workflow of how a penetration test or offensive security operation is created. It includes involved stakeholders, approvers, informed parties, other participants, and the objectives of the operations. An SOP is important in ensuring that a mature and repeatable process develops. Like the rules of engagement, it's advisable to seek legal counsel to ensure that the tactics and techniques highlighted do not violate company policy or laws.

There are considerations throughout an engagement, and procedures might vary depending on the service offering that the procedure discusses. The following diagram shows some of the possible cornerstones of a purple team service. The many stages and procedures are also present for red teams and penetration testing services.

It is useful to templatize the SOP in order to have a repeatable process so that the format can be reused:

Figure 1.1: Standard operating procedure

Leveraging attack plans to track an operation

The attack plan is the high-level master test plan to track the operation. It will contain all the information needed to track logistics, stakeholders, tasks, and findings, and share notes. It might contain the various pieces of data directly or point to other documents as needed.

Mission objective – what are we setting out to achieve or demonstrate?

The mission objective will clearly state the intent of the operation. A clear objective will be extremely helpful because it will help the pen testers focus on the right targets and apply the appropriate techniques.

For instance, the approach and operational mode of pen testers will significantly differ if the goal is a red team breach operation to assess detection readiness by acquiring a certain production database, compared to assessing the security state and detection capabilities of a dedicated website.

Objectives should also include intermediary objectives, such as the desire to evade detection, or solely using tactics of a certain well-known adversary. This helps to assess the operation by evaluating intermediary objectives and their detection/response capabilities in addition to the broad objective of the operation.

The high-level mission objective might be established during a planning meeting with business stakeholders or entirely defined by the offensive security team.

Stakeholders and their responsibilities

Here is a set of roles that need to be defined and articulated to ensure that well-established and effective operations are performed:

• Service/Product team: What targets are included?
• Business stakeholders: Who is the owner of the target?
• Blue team: Depending on the kind of operation, you might include the blue team.
• Red team: This is us!
• Authorization: Who authorized the operation? This usually includes business stakeholders.
• Informed parties: Others that should know about the activity, debriefs, and findings?

If your company does not run its own infrastructure but leverages other cloud systems, make sure to check for appropriate authorization from the cloud provider to engage in offensive security testing. Some companies, such as Amazon, might require some paperwork.

To define the stakeholders and their role, a simple RACI chart can be leveraged as well. There is some good information available in the Roles & Responsibility Charting document by Michael L Smith and James Erwin, which highlights this method. Details can be found here: https://pmicie.org/files/22/PM-Toolkit/85/racirweb31.pdf.

A simple RACI chart to define the roles and responsibilities for a red team operation is as follows:

Codenames

Each operation (especially red teaming ones) should have a codename assigned to refer to the activities and objectives to be performed and enable stakeholders to communicate the matter without having to reveal the targets or objectives constantly.

Good codenames also help create strong team cohesion and identity. And they help keep the work fun, and what is the world if you cannot have a little fun?

Timelines and duration

The duration of an offensive security operation might vary significantly. An operation might run for as little as a few days, all the way up to many weeks or even months. There is no rule or best practice. It solely depends on the mission objective and what the goal, strategic output, or organizational change should be.

It's good to plan in buffer time in case mission objectives are adjusted or new targets are being scoped in. This is something that we should entertain as an operation progresses and new intelligence and findings are discovered. It could even mean aborting a mission to reprioritize a more important one first.

For general security assessments, it's good to set aside a few weeks at least. It depends on the target's size and scope. Red teaming operations can be longer, even months, but I have run red teams that lasted just a couple of hours of actual execution.

Understanding the risks of penetration testing and authorization

A seasoned offensive security team will have a discussion about the risks of their actions and activities throughout an operation. It is advised to pose a set of questions to stakeholders ahead of time as well, to understand the maturity level of the business group or target being dealt with. The main goal of performing risk analysis at this stage is to ensure that the offensive security team does not accidentally cause an outage that could have been foreseen.

Some question that might arise during these conversations are as follows:

• Has the target ever been pen tested before?
• Has fuzzing been performed by the engineering team?
• How many concurrent connections can the system handle?
• Are production or non-production assets part of the target list?
• Is the system shared between multiple users, such as cloud services versus dedicated hosting?

Depending on the maturity of the system being targeted, different stakeholders might need to chime in and authorize the operation. This is really business-dependent. To enable effective offensive security, it's good that some actions can be authorized quickly, for example, by an engineering manager.

Organizations are different, so it's critical to understand what works well in yours. I have seen places that enable the red team to go off and freely work the way they see fit if they follow the rules of engagements and their SOP. At other places, CISO and other stakeholders might want to be much more involved.

Kick-off meeting

If time and logistics permit, it's great to have a kick-off meeting with all stakeholders involved so that various teams and groups get to know each other, and communication channels are established. This is important for purple teaming operations.

Deliverables

It should also be established what the expected deliverables will be before, during, and after the conclusion of the operation. This includes potential status updates, daily or weekly sync and triage meetings, as well as a final summary, reports, or debriefs to stakeholders.

Notifying stakeholders

Depending on the operation type, a notification is sent to stakeholders to highlight the outline of the test and its objective, as well as timelines. The notification should include a reference to the rules of engagement and the standard operating procedure. The notification might be sent out to a broad audience for a transparent operation but might also be sent to only need-to-know stakeholders for covert red teaming operations that have the intent to validate the internal security efforts.

These notifications are useful in many ways. First, they raise the awareness of the key stakeholders around testing being performed. If there is, for instance, an outage or service disruption, the business owner can reach out to ensure the activity is or is not related to offensive security work. Secondly, it raises the overall awareness of the team and the kind of testing that is being performed.

An example of a possible notification might look like this:

Figure 1.2: Example of a penetration testing notification

In the next section, we will learn how to track progress during an operation.

Attack plan during execution – tracking progress during an operation

This can be a simple list of tasks to ensure all the basic tests are performed during assessments and operations. Ideally, this is defined before the operation begins, highlighting some of the core objectives and tasks that need to be performed.

The value of having such a plan becomes apparent when running larger, more complex operations with four or more pen testers and a wide range of targets. The attack plan should be a living document, which means during every red team sync, which should occur regularly, the plan is reviewed, progress is discussed, and new items/tasks or scenarios to explore get added.

A great benefit is also that in the end, it's easy to see which avenues were not explored and should be revisited and covered in a future engagement. Such information should be included in the debrief.

Possible ways to store and track attack plans could be something like OneNote. If you start storing sensitive information in your attack plans, then ensure that the attack plan is encrypted. I have also seen teams leverage Mattermost (https://www.mattermost.org/) or Slack instances to help with collaboration.

Other information you might find in an attack plan is covered in the following subsections.

Reconnaissance tasks and results

This is the section where everyone tracks and stores insights into OSINT work and port scan results. If scanning is all automated and tracked over time as well, this might be a link to the database that contains the information, or a dashboard. The attack plan can also be seen as a place for collaboration to take notes and highlight findings.

Attack scenarios

This is where the attack scenarios to accomplish mission objectives are written down. This list will organically grow during the operation. It's common that staging multiple exploits together enables an overall scenario. At times, it might contain high-level brainstorming ideas to think about and explore.

Covering vulnerability classes

Have all common vulnerability classes been thought of? Attack methods differ from pen tester to pen tester, and the ultimate way to engage in reviewing code and applications should be left up to the pen tester. The key is not to discourage exploring new avenues of finding vulnerabilities. When it comes to comprehensive lists of tests to perform, there are some great resources, such as OSSTMM and OWASP.

Managing defects and incidents

It's important to highlight how defects are managed, including how the pen test team is going to protect the sensitive information before, during, and after an operation. It's likely that you want to archive your penetration testing findings and logs. Since findings and logs possibly contain sensitive information, the SOP should highlight steps to ensure that the information is adequately protected.

The SOP can go into great detail around the workflow and processes, and it might vary depending on what services the offensive security team provides.

Purple team sync and triage meetings

Depending on the kind of operation, the SOP might stipulate mandatory sync meetings between stakeholders. For purple team operations, defining timeslots for all the key stakeholders to get together regularly to review the work and share findings and brainstorm improvements should be formalized to ensure that it happens regularly. The operational model should be that the red team, the blue team, and some members of the engineering team are ideally collocated to enable continuous communication.

Documenting activities

At a minimum, a pen tester should document significant activity during operational engagements. Here are some considerations and ideas about what to capture and how.

Screenshots and logs

At a minimum, a pen tester should document significant activity via screenshots and keep a thorough access log of what actions were performed and what data was accessed. This is critical information that might be needed to distinguish friend from foe, and later for remediation and eviction. The blue team can cross-check detections and security alerts with red team logs as well.

The offensive security team will want to give an effective and memorable debrief at the end, so having supportive screenshots of the activity and findings is something that helps build out the full story.

At times, there is sensitive information in screenshots and logs, so be aware that they might need special handling. Many standard tools have great logging capabilities, but be aware that they usually log things in clear text, and further encryption of logs and data should be applied. This is especially important if you plan to archive reports in the long term.

Screen recordings

Some parts of operations should be recorded on video. This is for repudiation and educational reasons during debriefs. This is especially critical when accessing production resources. The idea is to protect the pen tester in case some other event happens or in case there is a dispute about what was performed, or what kind of information was accessed or exfiltrated.

Peer testing

With great power comes great responsibility, people say. As an offensive security engineer, it's not uncommon to be able to control the IT infrastructure of entire organizations. With these acquired privileges, a simple mistake on the keyboard or with the mouse could lead to service disruption or the deletion of enormous amounts of information. To avoid mistakes and accidents, it's recommended to run critical parts of an operation in peer mode to avoid accidents, and to screen record the activities.

At times, it might slow down one party, especially if a junior pen tester is teamed up with someone more senior, but it's also a great mentoring and educational opportunity to help grow individuals.

Wrapping up an operation

After an operation concludes, a report, summary, or debrief should be produced to ensure that all stakeholders receive the proper information.

Cleaning up and archiving

As part of wrapping up an operation, the offensive security might have to do some housekeeping, such as cleaning up attack environments, deleting unneeded infrastructure that might have been created during the operation, as well as archiving logs and attack plans. For long-term archival, I recommend encrypting the artifacts.

Eviction and remediation support

At times, the red team might be asked to help with eviction and remediation of findings. It is most likely a supportive role to ensure that no unnecessary user account additions remain. Depending on the rules of engagement, the red team might certainly be allowed (at times even encouraged) to remain present in an environment.

One area where the offensive team should also help is with user education. For instance, the pen testers can notify and talk to everyone whose credentials were compromised and inform them to change their password and reach out if they have questions. This grassroots hands-on approach is very effective, and users are very interested in how their machines were compromised.

Report and summaries

If the offensive team is integrated well in the engineering process and incident management, then a lengthy detailed report will not be needed. To raise awareness of the occurrence of an operation and its conclusion, produce a summary document or email that contains details about what was done and what the result was.

This executive summary highlights problematic patterns, the most important findings, and also what was not tested. It also includes references to the details of the issues. This reference is most likely a link to the findings, incidents, or a bug tracking system. In my opinion, issues and incidents should be directly tracked where the engineers, the blue team, and responders are working. There is little reason to create yet another place to persist the sensitive information about the details of the vulnerability.

I have seen that some companies track pen test findings with a different system or put the findings into a separate risk management tool, which most engineers don't even know about. This has the result that engineers never see pen test findings until much later and, unsurprisingly, issues are not fixed quickly in such cases.

A large how to document is collateral that needs protection. Adversaries might have special interest to acquire these reports. So, if there is no clear business reason to produce such lengthy aggregated reports, I advise against it. This is one of the differences when operating internally versus hiring an external consultant vendor for a black box test. In that case, you might just receive a single report, and then you must drive the resolution and mitigations yourself.

Remember that defects should have ideally been filed in the bug tracking system at this stage.

Debrief

The actual in-person debrief is one of the more significant meetings in which the offensive security team can impact and drive change in the organization.

Debriefs will include executive sessions in which the business owners, leadership, and key stakeholders are present to ensure that they have all the information to make the correct risk trade-offs. The debrief might also include a broader audience, such as all engineers and other team members. It could all be done in one debrief session or multiple sessions – it often depends on how your customer (the internal team you are working with) would like to handle the situation. Personally, I think it's great if execs and leadership are in the same meeting as the engineers because a lot of great conversations can take place. It also allows voices that often might not have an audience to be heard.

As a best practice, it's great to put a conceptual attack graph on the screen during a debrief session to discuss the stages and findings of the operation. The graph could also include detections that were triggered and improvements that were put in place during the operation. Later parts of the debrief will describe findings in more detail, but a conceptual overview is excellent for discussions.

Here is an example of what an attack graph that might be presented in a debrief could look like. It shows the most significant attacks, as well as any detections that might have occurred. To get the best attention, I find it most useful to build out the graph via animations step by step to slowly reveal the entire story:

Figure 1.3: Attack graph of a debrief summary

A debrief might also be leveraged as an education or training session so that others in the organization can learn from the findings and not repeat them. If your organization is okay with it, the offensive team can also entertain the idea of presenting the findings at a security conference. In the Appendix about Security Culture are some useful tips on how to build a training program.

Special consideration should be given to debriefs of purple team operations. Red team operations often solely project the view of the adversary, sometimes not highlighting the blue team's view at all. A purple team debrief focuses on the detections and improvements achieved and attempts to highlight the areas that need further investment.

Reflecting

After each operation, it's advisable to have a discussion about what went well and what could have been done better. This might include action items for the future, as well as potential ideas for further operations if it was not possible to assess certain aspects of the system due to time or resource constraints.

The team might also identify gaps in training or knowledge to brush up on.

Getting input and feedback from the other stakeholders will also allow the manager to assess the performance of the team and provide guidance to individuals on how their actions and communication with other stakeholders were perceived.

To gather feedback, I have always wanted to create a survey at the end of each pen test that everyone involved would have the chance to fill out. Practically, unfortunately, there was never enough time to work out the details for this. I do, however, believe such a feedback survey can provide great insights.

For security training sessions, I have established a survey process in the past. This survey helped me to understand how speakers were received, if the audience liked the content, if it was relevant to their work, and what other topics might be of interest for future training sessions.

I am, however, positive that applying the same methodology of gathering feedback at the end of a pen test will help to improve the effectiveness of the offensive security program.

Overarching information sharing via dashboards

Information that spans multiple operations is best made available via dashboards to stakeholders. More information about measuring the offensive program and security state of the organization is available in Chapter 3, Measuring an Offensive Security Program.

Contacting the pen test team and requesting services

One thing that we have not yet addressed is the way anyone in the organization can reach the penetration test team. Having an open ear is important if we want to make progress. How can we get a pen test on xyz done? In order to provide an official interface for anyone in the organization, it works to just set up an email group such as redteamrequest or something along those lines. That way, anyone has a direct channel to the red team and can ask for help or resources.

A more mature way is to create a portal or penetration test engagement system where someone can submit a request for services via a simple ticketing system. That way, it's directly in the triage queue of the pen test team.

The SOP should also contain some of the metadata to help progress the conversation. This includes information about the target and the service that is being requested, as well as approximate timelines.

The best way to track this information is by using a project management system such as Visual Studio or Jira – leverage whatever is commonly used in your organization.