preventing terminating EC2 by IAM policy

A major incident occurred at our company when the web application we support unexpectedly went down in production. We found that a junior DevOps engineer accidentally terminated the production EC2 instance, causing the disruption. Only Solutions Architects should be able to stop or terminate production instances. We also found that many developers have full access to our production AWS account.

Adding tags to the EC2 instances in the production environment and adding resource-level permissions to the developers with an explicit deny on terminating the instance which contains the tag is correct because it identifies the instances based on its environment using a tag which creates a resource level permission that explicitly denies anyone from terminating certain instances hosted in production.

ou can achieve this by using AWS Identity and Access Management (IAM) policies and AWS Resource Tags to implement resource-level permissions. Specifically, you’ll be creating a policy that allows developers to perform specific actions on EC2 instances with a particular tag, and an explicit “deny” to prevent them from terminating instances with that tag.

Here are the steps to implement this using the AWS CLI:

1. Create an IAM Policy that allows developers to perform specific actions on instances with a certain tag. You can use the ec2:CreateTags action to add tags, and other actions like ec2:StartInstances, ec2:StopInstances, or any other EC2-related actions you want to allow.

aws iam create-policy --policy-name DeveloperEC2Permissions --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:Describe*"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "ec2:ResourceTag/Environment": "Production"
                }
            }
        }
    ]
}'

2. Create an IAM Group for your developers and attach the policy to the group.

aws iam create-group --group-name Developers
aws iam attach-group-policy --group-name Developers --policy-arn <ARN of the DeveloperEC2Permissions policy>

3. Add developers to the IAM Group.

bashCopy codeaws iam add-user-to-group --group-name Developers --user-name <DeveloperUserName>

4. Explicit Deny Policy: Now, you need to create an explicit deny policy that prevents terminating instances with the “Production” tag. You can create a policy and attach it to the developers as well.

aws iam create-policy --policy-name DenyTerminateProductionInstances --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Action": "ec2:TerminateInstances",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "ec2:ResourceTag/Environment": "Production"
                }
            }
        }
    ]
}'

5. Attach the Deny Policy to the Developers Group.

aws iam attach-group-policy --group-name Developers --policy-arn <ARN of the DenyTerminateProductionInstances policy>

Now, developers in the “Developers” group will have permissions to perform specified actions on instances with the “Production” tag. However, they will be explicitly denied the ability to terminate instances with the “Production” tag.

Make sure to replace <ARN of the DeveloperEC2Permissions policy> and <ARN of the DenyTerminateProductionInstances policy> with the actual ARNs of the policies created.

Please ensure that you have appropriate IAM permissions to execute these commands and that you adapt the policy actions and resources as per your specific requirements and organizational structure.