Incident Response

Security Incident Response Team

TODO: We should document who is part of the security incident response team, but this playbook may not be the best channel for that.

Tooling

Communication Channels

• Splunk Security Team Rotation: The on-call person is responsible for monitoring the following places to address any pressing issues and initiate the relevant security processes:
  • The Splunk/VictorOps alerts
  • The security@narrative.io emails
  • The alerts on the #auto-security Slack channel
  • The discussions on the #security` Slack channel
  • Any other security concerns adressed to them directly through Slack, email, or other communication mechanisms
• #security-incidents Slack Channel: Used by the security team to discuss security incidents
• Incident Log: Used to record the incident response for each incident.

Incident Log

The Incident Log contains a document per incident.

• Naming convention: YYYY-MM-DD - Incident Name
• Template

The security incident response team has access to the incident log.

Each log entry should contain the following information:

• Date
• List of people involved in the decision making process
• An evaluation of the Incident Response Plan: wether all interested parties are in the right communication channels, etc
• The forensics evidence gathered while executing the incident response plan
• An evaluation of the impact and consequences of the incident
• An analysis of the root causes as well as the lessons learned
• The action items to remediate the problem in the short and longer term if needed.

A shortcut ticket assigned to the Security Incidents label should be created with a link to the relevant incident log entry.

Process

Communicate a Security Incident

• Communicate a security incident by using the following channels:
  • The #security Slack channel
  • The security@narrative.io email list
  • The phone number of anyone in the security incident response team, or, as a fall back, anyone in the dev or executive team.
• To the extent possible, make yourself available to provide additional information/identification when requested.

Initiate the Incident Response Process

• The first person in the Security Incident Response Team aware of an incident should create a shortcut ticket assigned to the Security Incidents label
• The rest of the process is lead by the on-call person
• Create an entry in the Incident Log
• Add a link to the relevant incident log entry to the Shortcut ticket
• Perform appropriate video identification depending on the context. Access Change guidelines/procedures from the System Access Control playbook are particularly relevant here.
• It is important to keep in mind that when video identification is not possible, only ever proceed with revoking credentials, not emitting new ones.

Incident Response Guidelines

Every incident is different and requires its own assessment and reaction function. However, here are a few guidelines that can help to react quickly and avoid missing important steps.

Compromised Credentials

Here are some important steps to consider when credentials have (potentially) been compromised:

• Start from the list of systems that the person has access to.
• Revoke access to the most critical systems first, and to the least critical last
• Once access has been revoked, audit the systems to determine what has been accessed and the extent of the potential damage
• Some systems do not provide meaningful auditing capabilities. We have to accept the risk of not auditing these.
• Communicate internally and externally about the extent of the breach if it meaningfully impacts the business and its partners

Auditing compromised systems

• Narrative app: no audit log available
• Google Workspace
  • Provides a great way to audit logins
• AWS Console Cloudtrail Event History can be used with a username lookup
• Google Cloud
  • Google Workspace audit logs
  • Service-specific Cloud Audit Logs overview
  • Logs Explorer
• Github
  • Organization Log. The actor= filter can be used to restrict entries to a given user.
• Cloudflare
  • Audit Log that can be filtered by user
• Notion
  • The audit log is available in in Settings & Members
• Slack
  • Audit Log Help
• Shortcut
  • No audit log available
• Stripe
  • Unsure if the Request Logs provide a complete audit trail
• NPM
  • No audit log available?
• DataDog
  • Audit Trail available, can be activated as needed.
• Cron / Jenkins
  • No audit trail
• Splunk On Call
  • No audit trail
• Drata
  • No audit trail
• Hubspot
  • Audit Log Help
• LinkedIn Business
  • No audit trail
• Facebook Business
  • The audit trail can be exported from settings/people
• Instagram
  • No audit trail