Skip to main content

​
When things go wrong

No matter how strong your defenses are, breaches happen. Firewalls get bypassed. Employees click phishing links. Zero-day exploits appear without warning. The question is never if your organization will face a security incident - it is when. Incident response (IR) is the organized approach an organization uses to detect, contain, and recover from cybersecurity incidents. A well-prepared team with a tested plan can mean the difference between a minor disruption and a catastrophe that shuts down the entire business.
Every major organization - from Fortune 500 companies to government agencies - has an incident response plan. Many are required by law to have one. Without a plan, teams panic, make mistakes, and let incidents spiral out of control.
PICERL cycle showing Preparation, Identification, Containment, Eradication, Recovery, and Lessons Learned

The PICERL Incident Response Framework

​
Events, alerts, and incidents

Before you can respond to an incident, you need to understand what qualifies as one. Not every suspicious event is an emergency. Think of it as a funnel. Your systems generate millions of events every day. Security tools filter those into a much smaller number of alerts. Security analysts investigate alerts and determine which ones are real incidents that require a coordinated response.
Not every alert is an incident. Part of a security analyst’s job is separating real threats from false positives. If every alert triggered a full response, your team would burn out in a week.

​
The PICERL framework

PICERL is the industry-standard incident response framework. It defines six phases that every response team follows, in order, when handling a security incident. The name comes from the first letter of each phase: Preparation, Identification, Containment, Eradication, Recovery, Lessons Learned.
1

Preparation

Build your team, create your plan, and practice before an incident happens.
2

Identification

Detect that something is wrong and determine the scope of the incident.
3

Containment

Stop the incident from spreading to other systems.
4

Eradication

Remove the threat completely from your environment.
5

Recovery

Restore systems to normal operations and monitor for re-infection.
6

Lessons Learned

Analyze what happened, what worked, and what you need to improve.
Let’s go deep on each phase.

​
Phase 1: Preparation

Emergency Plan notebook representing incident response preparation

Every organization needs a documented incident response plan ready before an attack happens

Preparation is the most important phase - and it happens before any incident occurs. A good company should always be in the planning and preparation phase. Incident response is an ongoing, never-ending process, not something you build once and forget.

​
Build the incident response team (IRT)

Every organization needs a dedicated group of people who know exactly what to do when an incident hits. The IRT typically includes:

​
Define your communication plan

When an incident hits, everyone needs to know who to call and how. Your communication plan should answer:
  • Who gets notified first?
  • What communication channels are used? (Phone? Slack? In-person only?)
  • When do you escalate to executives? To legal? To law enforcement?
  • Who talks to the media?
During an incident, never use potentially compromised communication channels. If your email server is breached, don’t send incident updates over email. Use out-of-band communication - phone calls, personal devices, or pre-established backup channels.

​
Create playbooks

A playbook is a step-by-step guide for responding to a specific type of incident. You don’t want your team improvising during a crisis - you want them following a tested plan. Common playbooks include:
  • Ransomware response
  • Phishing email investigation
  • Insider threat detection
  • Distributed denial-of-service (DDoS) mitigation
  • Data breach notification

​
Set up tools

Your team needs the right technology in place before day one:
  • SIEM (Security Information and Event Management): Collects and correlates logs from across your network
  • IDS/IPS (Intrusion Detection/Prevention Systems): Monitor for suspicious activity
  • Forensic tools: Software for creating disk images, analyzing malware, and preserving evidence
  • Ticketing system: Track incidents from detection through resolution
  • Backup systems: Verified, tested, and ready to restore

​
Practice

Plans on paper are not enough. You need to practice.
  • Tabletop exercises (TTX): Discussion-based walkthroughs where your team talks through a scenario step by step. No actual systems are involved - just brains and whiteboards.
  • Simulations: Hands-on exercises using actual tools and realistic attack data. These test both your plan and your people under pressure.
The Donuts & Dragons wargame you are doing this week is essentially a tabletop exercise. You are walking through attack scenarios, discussing how your defenses respond, and identifying gaps - exactly what professional security teams do.

​
Phase 2: Identification

Identification is where you detect that an incident has occurred and figure out what you are dealing with.

​
Detection sources

Incidents can be discovered through many channels:
  • SIEM alerts: Automated correlation of log data triggers a warning
  • IDS/IPS notifications: Network or host-based sensors detect suspicious traffic or behavior
  • User reports: An employee notices something strange - “My computer is running slowly” or “I got a weird email from the CEO”
  • Log anomalies: A security analyst reviewing logs spots an unusual pattern
  • External notification: A partner, law enforcement agency, or even the attacker themselves notifies you
  • Threat intelligence feeds: Indicators of compromise (IOCs) matching your environment

​
Determine scope

Once you suspect an incident, you need to quickly answer:
  • What systems are affected? Is it one workstation, a server, or the entire network?
  • What data is at risk? Customer records? Financial data? Intellectual property?
  • How long has this been going on? Was the attacker in your network for hours or months?
  • Is the incident still active? Is the attacker still inside right now?
From this moment forward, document everything. Every action taken, every system examined, every decision made. This documentation becomes critical during forensics, legal proceedings, and the lessons learned phase.

​
Phase 3: Containment

Now that you know what you are dealing with, your immediate priority is to stop it from getting worse. Containment prevents the incident from spreading to additional systems.

​
Short-term containment

These are your emergency, right-now actions:
  • Isolate affected systems from the network (unplug the cable, disable the Wi-Fi adapter)
  • Disable compromised user accounts
  • Block malicious IP addresses at the firewall
  • Redirect DNS to prevent communication with command-and-control servers
The goal is speed - stop the bleeding before you worry about surgery.

​
Long-term containment

Once the immediate spread is stopped, you implement more durable containment measures:
  • Patch the vulnerability the attacker exploited
  • Strengthen access controls
  • Deploy additional monitoring on affected segments
  • Set up a clean staging environment for rebuilding systems
Preserve evidence. Do not wipe, reformat, or rebuild compromised systems yet. You need them intact for forensic analysis. Take forensic images before making any changes.

​
Phase 4: Eradication

With the incident contained, you now remove every trace of the threat from your environment. Eradication actions include:
  • Delete malware from all affected systems
  • Close backdoors the attacker installed for persistent access
  • Patch vulnerabilities that allowed the initial breach
  • Rebuild compromised systems from known-clean images (do not trust a system that was compromised - rebuild it from scratch)
  • Reset credentials for any accounts that may have been compromised
  • Scan again - verify the threat is truly gone by running thorough scans across your environment
Attackers frequently install multiple backdoors. If you find one, assume there are others. A partial cleanup leaves your door wide open for a return visit.

​
Phase 5: Recovery

Recovery is the process of returning your organization to normal operations. This phase requires patience - you don’t flip everything back on at once.
1

Restore from clean backups

Use verified, uncompromised backups to restore data and systems. Never restore from a backup that may contain the attacker’s malware.
2

Rebuild and validate

Bring systems back online one at a time. Validate each system before reconnecting it to the production network.
3

Monitor closely

Watch restored systems intensely for any signs of re-infection. Increase logging and alerting thresholds during this period.
4

Gradual return

Start with non-critical systems and work your way up. Full production operations resume only after you are confident the environment is clean.
This is where backups prove their worth. An organization with tested, offline backups can recover from ransomware in hours. An organization without backups may be looking at weeks of downtime - or paying the ransom.

​
Phase 6: Lessons learned

This is the phase most organizations skip - and it is the one that matters most for preventing the next incident. Within 1-2 weeks of closing the incident, gather your entire response team for a post-incident review. Walk through the complete timeline and answer:
  • What happened? Build a detailed timeline from initial compromise to full recovery.
  • How did the attacker get in? Identify the root cause, not just the symptoms.
  • What went well? Which parts of your plan worked? Which team members stepped up?
  • What failed? Where did the plan break down? What took too long? What was missing?
  • What needs to change? Update your playbooks, tools, training, and staffing based on what you learned.
This information gets documented in two formal deliverables (covered in the next section).
The lessons learned phase is what makes incident response a cycle, not a one-time event. Every incident teaches you something. The best security teams use every incident - even small ones - to continuously improve their preparation.

​
Post-incident reporting

After the dust settles, you need to formally document what happened. These reports serve multiple audiences: technical teams, executives, legal counsel, regulators, and insurance providers.
The IRR is the formal, comprehensive record of the entire incident. It documents what happened, what actions were taken, and the outcomes.A typical IRR includes:The IRR becomes part of the organization’s permanent records and may be requested by auditors, regulators, or courts.
The RCA digs deeper than the IRR. It identifies the fundamental cause of the incident - not just the immediate trigger, but the underlying weakness that allowed it to happen.For example:
  • Immediate cause: An employee clicked a phishing link
  • Root cause: The organization had no security awareness training program, no email filtering, and no multi-factor authentication
A strong RCA answers “why” multiple times until you reach the systemic issue. This is sometimes called the 5 Whys technique:
  1. Why was the data stolen? → The attacker accessed the database.
  2. Why could the attacker access the database? → They had valid credentials.
  3. Why did they have valid credentials? → An employee’s password was compromised.
  4. Why was the password compromised? → The employee entered it on a phishing page.
  5. Why did the employee fall for phishing? → There was no security awareness training.
Root cause: Lack of security awareness training and phishing defenses.
Executives don’t need technical deep-dives. They need a clear, concise summary that answers:
  • What happened (in plain language)?
  • What is the business impact (financial loss, customers affected, regulatory exposure)?
  • What are you doing to prevent it from happening again?
  • What resources do you need (budget, staffing, tools)?
This report is typically 1-2 pages and focuses on risk and business impact, not technical details.

​
Training types

Your incident response plan is only as good as the people executing it. Training ensures your team can perform under pressure.
Tabletop exercises are discussion-based practice sessions. Your team gathers around a table (or a video call) and walks through a hypothetical scenario step by step.How it works:
  1. A facilitator presents a scenario (e.g., “You discover ransomware on five workstations”)
  2. The team discusses what they would do at each stage
  3. The facilitator introduces complications (“The backup server is also encrypted”)
  4. The team adapts their response
  5. Everyone debriefs on what went well and what didn’t
Pros: Low cost, no disruption to operations, great for testing communication and decision-making.Cons: No hands-on practice with actual tools, may not reveal technical gaps.

​
Exercise: write an incident report

Put what you have learned into practice. Read the scenario below, then write a brief incident report.
The situation: On Tuesday at 9:47 AM, a barista at Donuts & Dragons received an email that appeared to be from the POS vendor. The email said there was a critical update available and included a link to download it. The barista clicked the link and entered their work credentials on a page that looked identical to the vendor’s login portal. By 10:15 AM, the attacker used the stolen credentials to access the POS system remotely and began exporting customer payment data.At 11:30 AM, the security monitoring service flagged unusual outbound data transfers from the POS system. The team was notified at 11:45 AM.Your task: Write a brief incident report that includes:
  1. Timeline of key events
  2. Systems affected and data at risk
  3. Response actions you would take (using PICERL phases)
  4. Root cause of the incident
  5. Recommendations to prevent recurrence
Discuss your report with your team. Compare your timelines and recommendations - did everyone catch the same details?

​
Key takeaways

PICERL is a cycle

Incident response is not a one-time event. Each incident feeds back into the preparation phase, making your team stronger for the next one.

Preparation is everything

The vast majority of incident response happens before an incident. Build your team, write your playbooks, and practice relentlessly.

Document everything

From the moment you suspect an incident, write it down. Documentation drives forensics, legal proceedings, and future improvements.

Never skip lessons learned

The post-incident review is what transforms a painful experience into lasting organizational improvement.