Today, we will see how a user can automate the isolation of AWS EC2 instance and re-image a clean instance through Mindflow.
For organizations deployed on the cloud, the security of their cloud infrastructure is paramount. The need for quick and effective responses to potential security breaches is essential in maintaining the integrity of an organization’s data and systems. One such use case is the automation of the retrieval, isolation, and re-imaging of a compromised EC2 instance in AWS. This process mitigates the risks associated with security breaches and ensures minimal disruption to business operations.
The primary interest of this use case lies in its ability to streamline the incident response and remediation process. By automating the detection and handling of compromised EC2 instances, organizations can reduce the time and effort required by their teams to identify and address security issues manually. This improves the overall efficiency of incident management and allows IT teams to focus on more strategic tasks.
Through integration with Jira and Slack, this use case also enhances communication and collaboration among team members by providing real-time updates on the remediation process. Creating and updating Jira issue tickets ensure that relevant stakeholders are informed about the incident, while Slack notifications allow for immediate awareness and confirmation of actions taken.
As such, we’re going to orchestrate and automate the following tools:
In summary, automating the retrieval, isolation, and re-imaging of compromised EC2 instances offers numerous benefits to organizations, including improved security, faster incident response, and better collaboration among team members. By leveraging AWS native services along with third-party tools, organizations can effectively protect their cloud infrastructure and maintain the highest level of security and operational efficiency.
How can you perform isolation of AWS EC2 instances and re-image a new instance the old way?
Amazon GuardDuty generates alerts for unusual or suspicious activity and forwards them to AWS Security Hub, providing a centralized view of security threats. The security analyst monitors the GuardDuty events page, investigates alerts and affected resources, and creates a Jira issue for tracking the incident response process. Upon approval, the analyst isolates the compromised EC2 instance, deletes key pairs, stops the instance, and launches a new instance with the attached EBS volume. Throughout the process, the analyst updates the Jira ticket and notifies the team on Slack, ensuring transparency, communication, and collaboration during incident response.
How can it be automated through Mindflow?
Initial event ingestion
Three options:
- Either creating event rules straight on EventBridge to triage upstream.
- Â Creating an EventBridge triaging Flow to triage events that will trigger different Flows according to choosen data.
- Â Creating a Flow ingesting alerts from EventBridge and performing an all-in-one Flow about EC2 isolation and reimaging.
For the purposes of the demonstration, we’ll go with the last option.
An EventBridge event webhook will contain every data we need to perform the triage: the resource type in cause and the event details. Thus, we will create two conditions to triage between EC2-related events and others, and then between different events related to EC2 to select only the ones that acknowledge a potential compromise of the instance that will thus need to be stopped and re-imaged.
Automating the isolation of AWS EC2 instance and re-image
Here we go on the most exciting part of the Flow, where we are going to orchestrate and automate the different actions to perform isolation of the AWS EC2 instance, a re-imaging using an EBS snapshot from a sane point in time, and run and start this instance, before closing the issue. We’ll begin the demonstration after the initial triaging we saw just above.
- Retrieve the EC2 Instance: Use the AWS EC2 API to fetch the details of the desired EC2 instance based on the instance ID we will have in the Eventbridge webhook payload.
- Create an issue on Jira: Utilize the Jira API to create a new issue/ticket with the necessary details about the EC2 instance, including instance ID and event details.
- Notify and ask for confirmation on Slack: Send a notification message to the appropriate channel or user with the Jira issue link and request confirmation to proceed with the isolation of AWS EC2 instance and re-imaging a new clean instance.
- Start the EC2 instance isolation: After receiving confirmation, start the quarantine. Use the AWS EC2 API to modify the security group rules or apply a new security group to the instance, limiting its network access for isolation.
- Delete key pair associated with the EC2 instance: Use the API to remove the key pair associated with the instance, preventing unauthorized access.
- Get the snapshots attached to the EBS volume: Use the EC2 API to fetch the details of the EBS volume attached to the instance and then retrieve the snapshots associated with the volume.
- Stop the EC2 instance: Stop the running instance by pushing the instance ID in the InstanceID field.
- Create a new EBS volume: Utilize the API to create a new EBS volume with the desired size and configuration based on the retrieved snapshot details.
- Launch a new instance with this EBS volume attached: Use the AWS EC2 API to launch a new instance with the newly created EBS volume attached and configure the instance with new security groups and key pairs.
- Edit the issue: Update the Jira issue using the Jira API, adding information about the new instance, the attached EBS volume, and the process status.
- Notify on Slack: Send a final notification message on Slack using the Slack API, informing the team or user about the completion of the process and providing details about the new instance and the updated Jira issue.
Additional steps
Of course, the best would be to run a scan before closing the issue to be 100% sure that the newly mounted instance is clean and up for running. You can use different AWS or third-party tools, such as Sophos UTM, Crowdstrike Cloud Security, etc. Here, we will go with a native AWS service, AWS Inspector.
In short, we will run an AWS Inspector assessment scan on the newly created EC2 instance, checking for potential vulnerabilities and compliance issues, returning the findings, and gathering them in the Jira issue ticket before sending a final notification to Slack with the issues URL.
- Create a resource group using the EC2 instance key-value pair: Utilize the AWS Resource Groups API to create a new resource group, which includes the new instance, based on its key-value pair (tags).
- Create a new assessment target: Use the AWS Inspector API to create a new assessment target, specifying the recently created resource group as the target.
- Create an assessment template: Utilize the AWS Inspector API to create an assessment template, defining the rules packages for the assessment and the assessment target.
- Start an assessment run: Use the AWS Inspector API to initiate the assessment run, utilizing the created assessment template.
- Describe the findings from the assessment run: After the assessment run is completed, use the AWS Inspector API to retrieve the detailed findings and results of the security assessment.
- Create an issue or Edit an existing one in Jira: Use the Jira API to either create a new issue or edit an existing one, including the findings from the AWS Inspector assessment run and any necessary remediation actions or recommendations.
- Notify Slack with a high-level summary: Utilize the Slack API to send a notification message to the appropriate channel or user to provide a high-level overview of the AWS Inspector assessment findings and a link to the Jira issue for further details and action.
