Serverless browser automation with AWS Lambda and Puppeteer

Manipulating a web browser environment with an API provides a wide range of automation capabilities for developers. It allows you to generate PDF files, screenshot webpages, or run health checks on a website, all from code. It can also enable you to automate form submissions, build UI tests, or diagnose performance issues. Headless Chromium is a popular package for operating a browser programmatically. Whether you are load testing a website or periodically fetching content, this can be configured with minimal code. 

You can run a headless browser on your local development machine or a remote server. However, many typical browser automation tasks are a good fit for AWS Lambda. You can configure a Lambda function to start on a schedule, or in response to an event. You can also configure Lambda to scale up for load testing operations, making it a cost effective alternative to managing a fleet of instances.

In this blog post, I show how you can deploy a browser automation task to Lambda. This example uses the AWS Serverless Application Model (AWS SAM) to simplify the deployment of cloud resources. You can download the code for this blog post from the companion GitHub repository. To deploy to your AWS account, follow the instructions in the README file.

Get started with ACG and transform your career with courses and real hands-on labs in AWS, Microsoft Azure, Google Cloud, and beyond.

In the example application, a Lambda function is invoked every 15 minutes to take a screenshot of a webpage and save the image to an S3 bucket. The architecture looks like this:

1. An Amazon EventBridge rule invokes the Lambda function using a schedule expression.
2. The Lambda function uses Chromium to load the target webpage. Once the page is loaded and rendered, it takes a screenshot.
3. The screenshot is saved to an Amazon S3 bucket.

This Node.js example uses an npm package called Puppeteer, which exposes a high-level API to control the Chromium browser. A snippet of the Lambda function shows how this works:

browser = await chromium.puppeteer.launch({
     args: chromium.args,
     defaultViewport: chromium.defaultViewport,
     executablePath: await chromium.executablePath,
     headless: chromium.headless,
     ignoreHTTPSErrors: true,
});

let page = await browser.newPage()
await page.goto(pageURL)
const buffer = await page.screenshot()

This uses JavaScript async/await syntax to avoid callbacks and use sequential code flow. Once the browser object is defined, the code instructs Chromium (via Puppeteer) to fetch the webpage. After the page is loaded and the DOM is rendered in the headless Chromium instance, it stores a screenshot of the page in a buffer variable. Finally, the image is written to the S3 bucket:

const s3result = await s3
   .upload({
      Bucket: process.env.S3_BUCKET,
      Key: `${Date.now()}.png`,
      Body: buffer,
      ContentType: 'image/png',
      ACL: 'public-read'
    })
    .promise()

console.log('S3 image URL:', s3result.Location) 

This uses the S3 upload method in the AWS SDK for JavaScript. It defines the bucket and key, sets the content type to PNG, and then configures the access control list (ACL) so the object is publicly viewable. Finally, it logs the public URL of the stored object.

AWS Lambda allows you to package dependencies together with your code as a zip file. The deployment package can be up to 250 MB (unzipped) or 50 MB (zipped, for direct upload). For larger packages, it’s recommended that you use the container packaging format for Lambda functions, which allows packages of up to 10 GB. 

If you use a tool like AWS SAM or Serverless framework, the packages are created on your development machine, then zipped and uploaded to the Lambda service. At runtime, these files are unzipped in the Lambda execution environment when the function is run. These tools can simplify this packaging process and help streamline your deployments.

However, when you use a dependency that contains a binary, you must ensure that you package a version of the binary that is compatible with the operating system used by Lambda. The Puppeteer package contains an entire Chromium browser bundled into the deployment. Since the browser relies on binaries, npm installs the binary that matches the operating system of your local development machine. However, Lambda requires the binary that matches its underlying operating system, which is Amazon Linux 2.

To help with this, many popular packages have been converted into Lambda layers by the community. You can define up to five layers per Lambda function and these are copied into your deployment package when you create or update a function. For Chromium, this makes it easier to run one binary version in development and use another binary version at runtime. Learn more about creating and using Lambda layers to simplify your development process.

One developer has packaged a Chromium binary for AWS Lambda and published this to GitHub. You can install this with Puppeteer in your Lambda function by including the libraries in package.json. You can also bundle both with an existing public Lambda layer. This GitHub repo frequently publishes new versions of the chrome-aws-lambda package, which you can include directly in your Lambda function.

Layers are only available in the AWS Region where they are published. The maintainers of this public repo have published this layer to 16 Regions and provided layer ARNs in the README file. These ARNs are public and you can include in any Lambda function in those Regions in any AWS account. 

There are many popular libraries that have been bundled into Lambda layers by the community. This GitHub repo aggregates layers for commonly used utilities like GeoIP, MySQL, OpenSSL, Pandas, scikit-learn, and many others. To use these in your Lambda functions from a compatible runtime, you only need to include the layer ARN in a supported Region.

The Lambda function in this example could have been defined directly in the AWS Management Console. However, by using AWS SAM, you can define the same infrastructure as code. This helps create repeatable deployments quickly and reduces human error from clicking around the console.

The AWS SAM template defines all the AW resources used by the application. First, it declares an S3 bucket:

  S3Bucket:
    Type: AWS::S3::Bucket  

Next, the template defines the Lambda function and where the code can be found. Since it runs an entire browser within the function, the memory is set to 4096 MB. The timeout is configured to 15 seconds to ensure that the function ends if the target webpage is unresponsive. 

SnapshotFunction:
     Type: AWS::Serverless::Function
     Description: Invoked by EventBridge scheduled rule
     Properties:
       CodeUri: src/
       Handler: app.handler
       Runtime: nodejs12.x
       Timeout: 15
       MemorySize: 4096 

The template includes a reference to the publicly available Chromium layer and substitutes the Region code at deployment time. Providing that the example is deployed in one of those 16 Regions where the layer is available, the layer ARN is valid:

Layers:
  - !Sub 'arn:aws:lambda:${AWS::Region}:764866452798:layer:chrome-aws-lambda:22'

Environment variables are used to set the target website URL and define the bucket name to store the image. Finally, since the function only writes data to S3, it uses an AWS SAM policy template to provide write permissions to the single bucket. This follows the principle of least privilege:

Environment:
  Variables:
   TARGET_URL: 'https://serverlessland.com'
   S3_BUCKET: !Ref S3Bucket
 Policies:
   - S3WritePolicy:
       BucketName: !Ref S3Bucket 

Lambda functions are invoked in response to events. In this case, the function runs at a timed interval, which is managed by EventBridge. Using a schedule expression, the template configures the function to run every 15 minutes:

Events:
  CheckWebsiteScheduledEvent:
    Type: Schedule
    Properties:
      Schedule: rate(15 minutes) 

Anytime you make changes to the Lambda function or the resource in this template, run sam deploy again to deploy the new version to the AWS Cloud. The AWS SAM CLI detects the differences between versions and deploys the new code and resources automatically. 

After deployment the example application, navigate to the Lambda console. Open SnapshotFunction deployed by AWS SAM. The function is invoked automatically every 15 minutes but you can trigger the function by choosing Test:

The Log output contains details of the function duration and the public URL of the image that is generated and stored in the S3 bucket. Navigate to this URL in a browser to view the screenshot:

After the function has been deployed for a few hours, it has been invoked multiple times by the scheduled event. You can monitor its performance using Amazon CloudWatch metric. From the Lambda function, choose Monitoring to see the number of invocations, average duration, and any errors:

From the S3 console, open the S3 bucket created by the AWS SAM deployment to see the date-stamped objects created by each Lambda invocation:

Programmatically controlling a web browser enables you to automate many useful tasks with code. For many of these, you can use AWS Lambda to minimize infrastructure overhead and simplify scaling. This blog post shows how to deploy an example application that uses a headless browser to take periodic screenshots of a webpage.

For commonly used libraries or packages with operating-system specific binaries, Lambda layers can help simplify deployment. Many libraries have publicly maintained layers you can include in your Lambda functions. With infrastructure as code tools like AWS SAM, you can define your code and layers together in YAML, to help create repeatable deployments and accelerate development.

For more serverless learning resources, visit Serverless Land and check out our Serverless comparison below.