Deploying Selenium Grid Using AWS CDK & Fargate

Hello, today we look deeper into deploying Selenium Grid with AWS CDK and Fargate. I find inspiration from this AWS blog post, I rewrote this in CDK 2 Python. I will explain each function, so you understand how they work together to make the infrastructure. We will use HTTP like in the old example. I have an advanced https version but I have also product code in it which I am using for product development, so unfortunatly I cannot provide this source. But I think the reader can easily make this http version scaled and with https!

In the end it should look like this picture underneeth, you should see a grid with VNC support!

How do you do this(important steps)

1- Making & Sending Docker Images to AWS ECR

This guide will show you how to use a shell script to make and send Docker images to AWS Elastic Container Registry (ECR).

What You Need Before Starting

Before you start, make sure you have:

• An AWS account with permissions to make repositories in ECR and send images.
• Docker installed on your local machine.
• AWS CLI installed on your local machine.

Step 2: Setting Up Environment Variables

First, you need to set up the environment variables for the script. You can change the script directly or make a separate .env file with these variables:

ECR_REPO_NAME_SEL_HUB="selenium-hub"
ECR_REPO_NAME_CHROME="chrome"
ECR_REPO_NAME_FIREFOX="firefox"
AWS_REGION="eu-west-2"
AWS_ACCOUNT="your-aws-account-id"

Note: Replace AWS_ACCOUNT with your actual AWS account ID.

Step 3: Running the Script

After setting up the environment variables, navigate to the directory containing the shell script and execute the following command:

./push-to-ecr.sh

To verify whether the images have been successfully pushed to ECR, visit the ECR console in the AWS Management Console. You should see the newly created repositories and the associated images.

Conclusion

Utilizing a shell script to create and push Docker images to AWS ECR is a time-saver. By adhering to this tutorial, you can effortlessly create and push Docker images to ECR. While this process can be integrated into any GitLab/GitHub pipeline, I opted to execute it manually in this instance, both as a cost-saving measure and for educational purposes.

The CDK2 Python Part

In the cdk2 we will create the infra as code in python, which also can be written as typescript or javascript. The essence is that it generates infra cloudformation like Terraform does, but then it is native like Bicep is for Azure.

How-> Here comes the very technical part, I think I lost already a lot of readers :-)

__init__ Function

The __init__ function acts as the constructor for the SeleniumStack class. It plays a pivotal role in initializing an array of AWS resources. These resources include:

• VPC (Virtual Private Cloud): Provides an isolated virtual network, allowing you to control your own network environment.
• Fargate Cluster: A serverless compute engine for containers.
• ECR Repositories (Elastic Container Registry): A fully-managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images.
• IAM Roles: These are AWS Identity and Access Management (IAM) entities that define a set of permissions for making AWS service requests.

After initializing these resources, the function proceeds to create the Selenium Hub service and node services for both Chrome and Firefox browsers.

create_selenium_hub_service Function

This function is dedicated to creating the Selenium Hub Fargate service. Here’s what it does:

• Task Definition: Sets up a Fargate task definition that includes the Selenium Hub container.
• Security Group: Establishes a security group to allow inbound access.
• Log Group: Initiates a log group for logging purposes.
• Application Load Balancer (ALB): Configures an ALB, which performs a health check and facilitates communication via HTTP.

create_selenium_node_service Function

This function crafts a Selenium Node Fargate service, specifically for a given browser—either Chrome or Firefox. The steps it follows are:

• Task Definition: Creates a task definition with the relevant browser container.
• Environment Variables: Configures environment variables essential for connecting to the Selenium Hub.
• Log Group & Security Group: Similar to the hub service, it sets up a log group for logging and a security group for inbound access.
• Fargate Service Registration: Finally, the function creates the Fargate service and registers it with the ECS cluster and Cloud Map for service discovery.

add_autoscaling_policy Function

This function is tasked with establishing an autoscaling policy for a specified Fargate service. The autoscaling is configured based on:

• CPU and Memory Utilization: The service scales up or down, adjusting to the load.
• Target Utilization Percentages and Cooldown Periods: These parameters are specified to control the scaling activities efficiently.

In essence, this function ensures that the service adapts to the varying loads by adjusting its scale based on the utilization of resources and specified parameters.

code can be found in the repo: https://github.com/learnautomatedtesting/cdk2seleniumpython

Dockerimage example to push to ecr

FROM selenium/node-chrome:4.8.3-20230404

USER root

# Install dependencies
RUN apt-get update && \
    apt-get install -y \
        unzip \
        wget \
        xvfb

# Download and install ChromeDriver
RUN CHROMEDRIVER_VERSION=$(curl -sS https://chromedriver.storage.googleapis.com/LATEST_RELEASE) && \
    wget -q "https://chromedriver.storage.googleapis.com/111.0.5563.64/chromedriver_linux64.zip" && \
    unzip chromedriver_linux64.zip && \
    rm chromedriver_linux64.zip && \
    mv chromedriver /usr/local/bin/

USER seluser

build an deploy proces first you bootstrap : if you do not have created a cdk in aws then: you do a cdk synth (synthesizes the repo) then: cdk deploy: this will deploy your infra as code

Bear in mind about the security aspects. There is a security hub in AWS and they can support you finding critical issues. luckily the open source community is very quick handling those tickets, even faster then the paid ones I must say. Also protect your grid so that not everybody can accces it (Via headers and or private subnet) Knowing that this article is not for all readers,mainly senior software developers in test and or infra developers I think it is good to have some understanding how easy you can provision AWS selenium in fargate, in a private subnet and or public.

repo source can be found here: https://github.com/learnautomatedtesting/cdk2seleniumpython

Hope you still have read everything but I understand that most already thought, what is this :-)