How to Create AWS IAM Role or Instance Profile
This blog will walk you through the process of creation of a new AWS IAM role that is equivalent to AWS instance profile in the process of integrating Databricks with AWS.
Understanding the Integration
When we start a new cluster on Databricks we are essentially provisioning a new EC2 instance in AWS. For the Databricks cluster to access AWS services including S3, Glue Catalog and EC2 we have to create a role and an instance profile. In this particular case, the role and the instance profile are the same because when a role is created to provide permissions to an EC2 instance, automatically an instance profile is created.
Creating the Role and Instance Profile
We will create a role on top of the EC2 instance and we will see how to get an instance profile. We will then assign the permissions that are required for a role in order to access the Databricks’ clusters.
Step-by-Step Guide
- To create a role, it is necessary to go to the AWS Management Console.
- Search for IAM and select roles on the left side.
- Then click on create role and in the trusted entity type we will select AWS service.
- For the use case we will type EC2 and choose it.
- We will then click on next and add the permissions and in the following tutorial we shall be looking at that.
Reviewing the Role and Instance Profile
After creating the role, we shall check on the trusted entities and click on create role. We will also notice that a role has been created for us, and for Databricks, as well as an instance profile. To do this, we have to move to the nex blog where I will explain how to register this instance profile with Databricks.