My goal was clearly defined. I wanted to design and introduce a flow that won’t annoy my teammates. What annoys developers when working in an AWS multi-account setup? A constant need for reauthentication, looking for a phone when you are asked for MFA or figuring out how to use a given tool to work in the context of a given AWS account. This can be pretty cumbersome. And yes, you can set up an authentication strategy so that developers won’t even notice that they are changing an account. or even work in a multi-account setup.
Let’s list tools used by our team which require authentication towards AWS:
- AWS SDK, CLI, and management console
- Terraform
- Serverless
- Automation, generally CI/CD
In the article, I want to show what approach we took and how well it goes in line with the mentioned tools.
General Idea
I will spoil it a little bit, I proposed the following: one IAM role to rule them all. The idea is simple and easy to manage. We have one account that acts as an entry point. It is called a `bastion`. From it, you jump into a child account using the AssumeRole action. To log in to the bastion account we use OKTA + AWS SSO. That integration is set up by our cloud governance team. If you would like to more about how to configure check AWS user guide.
Here is a diagram that describes the idea
Diagram taken from internal documentation
On each child account, we defined IAM role called `AssumableAdmin`. Here it’s configuration (god mode):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}
We also need to define trust relationships, which says who can assume that role. The following policy document states that a given principal can assume a role from an account under consideration. This means: anything that originates from the bastion account (Not just federated users but as well IAM users or AWS services like EC2)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "AWS": "" },
"Action": "sts:AssumeRole"
}
]
}
How do you create such a role if you don’t yet have access to child accounts? You can do this manually via root user or any other approach that will work for you. In our case, we provide such things via administrative Terraform scripts. We also have access to child accounts via SSO, so we used those credentials when we run administrative scripts. Anyway, That was a one-off operation. For any other interactions with a given account, we use to do by AssumeRole.
MaxSessionDuration
That parameter is crucial. If you don’t change it you will make developers angry. There is nothing worse than being logout out in the middle of meaningful work. When you assume a role, you receive temporary credentials. The default lifespan of them is 1 hour. Change it to 8h, please.
Named profiles
Content of configuration file `~/.aws/config`:
[profile bastion] sso_start_url = https://foobar.awsapps.com/start sso_region = eu-north-1 sso_account_id = sso_role_name = FullAccess [profile dev] role_arn = arn:aws:iam:::role/AssumableAdmin source_profile = bastion duration_seconds = 28800 [profile pro] role_arn = arn:aws:iam:::role/AssumableAdmin source_profile = bastion duration_seconds = 28800
What is cool about such a configuration file is that you can freely share it between developers within a team. No secrets are stored. Not having credentials on a local machine is more secure than having them.
Two settings that deserve a few additional words. `source_profile` points on the profile that is the entry point. By default, AssumableAdmin can be assumed only for 1 hour, to change it to 8 hours, we need to set `duration_seconds = 28800`.
With tooling
I’m sure that you need more convincing. Let me show you how this idea works with our tools.
AWS CLI
Let’s see how we work in a multi-account environment from the command line:
# asp is a handy tool for switching profiles; with autocompletion # part of oh my zsh and aws plugin $ asp bastion dev pro $ asp bastion $ aws sso login # You will be redirected to AWS SSO sign-in page. # Everything is automated; you don't need to type anything # To do some work in the context of the dev account: $ asp dev $ aws dynamodb list-tables # And of the pro account: $ asp pro $ aws dynamodb list-tables
AWS SDK
Straightforward thing. If you are using the default credential provider resolver you don’t need to change your code. And if you load credentials from `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` environment variables, now you can get rid of them. Afterwards, you simply do:
$ asp dev $ npm start
Management console
To jump from bastion to a child account, you must visit one of the following pages. One to assume the dev role and the second one is for the pro. Please look carefully at the query params.
- `https://signin.aws.amazon.com/switchrole?account=<dev_account_id>&roleName=dev&displayName=dev`
- `https://signin.aws.amazon.com/switchrole?account=<pro_account_id>&roleName=pro&displayName=pro`
A page with a form will open. The form will be automatically populated with values from query parameters.
You just need to click on “Switch role” button.
After switching a role the given role will be saved in the role history. To access, while in AWS console, click on the top right corner, a submenu will appear, and at the bottom, you will see the last used roles. From now on, you can switch between accounts with just two clicks!
Terraform
Here our approach shines. Terraform can assume a role automatically. If you want to see a real case implementation check the configuration of the AWS provider on one of my repositories on GitHub.
provider "aws" {
assume_role {
role_arn = "arn:aws:iam:::role/AssumableAdmin"
}
}
Let’s see how the development of Terraform configuration can look in the multi-account deployments.
$ asp bastion $ aws sso login $ cd $ terraform init $ terraform plan # and now in a production context. $ cd $ terraform init $ terraform plan
As bastion profile can assume `AssumableAdmin` role on every child account we don’t even need to explicitly switch profiles as Terraform is doing this for ourselves. It’s just a matter of changing the working directory. If you are interested in details on how we work with Terraform check my last article Ultimate Terraform project structure.
Serverless framework
You can apply a similar strategy as for Terraform. Serverless gives you a chance to set a role that will be used for deployment. However, as stated in the docs.
It is important to understand that deploymentRole only affects the role CloudFormation will assume. All other interactions from the serverless CLI with AWS will not use that deploymentRole.
In practice, it means that whenever you do:
$ asp pro $ serverless info ... Serverless Error ---------------------------------------- AWS profile "pro" doesn't seem to be configured ...
Serverless will crash as it doesn’t work very well with profiles. Support for AWS SSO profiles is a long-awaited feature. This is the state for December 2022, the feature request is still open.
You have at least two options to omit this problem:
- Use Serverless plugin to extend its capabilities. Here is my fork of serverless-better-credentials. We use it and it works brilliantly with our approach. To reference to it in your project you will need to create an artifact and store it somewhere or use `npm link` capability. Then you just do:
$ asp pro $ serverless info # It works as expected!
- If you don’t want to trust random guys or libraries from the internet, you can do it the old way. By using `aws sts assume-role` to obtain temporary credentials and store them in environmental variables.
$ asp dev $ aws sts assume-role \ --role-arn $(aws configure get role_arn) \ --role-session-name "temporary-credentials" $ export AWS_ACCESS_KEY_ID="get_from_output_above" $ export AWS_SECRET_ACCESS_KEY="get_from_output_above" $ export AWS_SESSION_TOKEN="get_from_output_above" $ serverless info
CI/CD in Travis
Travis executors are ephemeral environments where creating `~/.aws/config` file would be problematic. More problematic would be obtaining temporary credentials as it is hard for me to imagine how you would log in to OKTA using a browser there. Still, we have old good IAM users. For Travis, we created IAM user on the bastion account that can do cross-account assume for `AssumableAdmin` roles. With that, all flows described before are applicable!
Bonus
Break glass
Can you guess what would happen if your SSO provider will be unavailable? Yes, you are right. You won’t be able to access your AWS account! To mitigate this problem we created one special IAM user. We called it `break-glass`. Credentials for it are stored in the password manager to which all developers have access.
Developer onboarding
Every new employee has an OKTA account from day one. We add she or he to the mailing group which is attached to our OKTA group, which is then attached to AWS accounts. Now, the new engineer just needs to copy and paste the config file settings that I presented at the beginning. This file can be shared in some common space as it contains no secrets. Can onboarding be easier than that?
Summary
There is one lesson worth taking after reading this article. Interoperability is neglected when making an architectural decision, choosing a design pattern, or using an authentication strategy. It’s worth asking how the new approach will work with my current toolset. To not adapt everything to the new solution as it should be done vice versa. The new element should suit well to the current landscape. This is what we made when choosing an authentication method for AWS. The approach that we have taken works brilliantly with all tools that developers use in our team.