Integration Testing AWS Lambda Functions with Localstack

Working with AWS locally can be incredibly difficult without the right setup. Even more so with lambda functions. However, despite these challenges, testing remains an absolute fundamental to crafting professional software.

The biggest challenge is that when running any local tests that involve any AWS software, the internals of AWS are proprietary and inaccessible without deploying and interacting with said feature live.

It isn’t uncommon either for developers to have their own AWS account for development, but this can get quite expensive in terms of managing and controlling this from an organisational standpoint. Consistently deploying to an external environment also extends out the time to receive feedback on changes- locally previewing changes will always cut the feedback loop when it comes to local development.

We therefore are presented with 2 choices: mocking or emulating. Mocking involves mocking the calls that are outside of your function code to call stubs instead of the AWS service.

To Mock or To Emulate? 🔗

Let’s take this trivial example, a lambda function that uploads an object to S3 (a .txt file) and returns its contents:

/*
 * handler.ts
*/

import { S3 } from "aws/s3";

const s3Client = new S3();

export default function handler() {
    const putObjectParams: PutObjectParams = {
        key: "textfile.txt",
        bucketName: 1
    }
    
    const textFile = s3Client.put(putObjectParams);
    
    console.log("File saved!");

    return {
        statusCode: 200
    }
}

If we ran the above code as is through as a unit test, it would fail with the following error:

Unable to reference S3. Unable to fetch credentials

s3Client is trying to connect to AWS’ actual S3 service, but it wouldn’t ever work without passing in credentials explicitly- something we’re actively avoiding, we wouldn’t want to be charged excees fees just for running a unit test!

We could however mock the S3 class, so that when .txt is called, it returns back the shape of data that .get() would usually return.

Using any package that can handle mocking, jest, in this instance, we can mock the return type.

/*
* handler.spec.ts
*/

import jest from "jest";
import handler from "./handler";

jest.mock("@aws/client", () => {
	S3: {
		upload: jest.fn()
    }
})

it("should return back 200", () => {
    handler();
});

Running the test again now passes with flying colours. But, that’s not the end of the story.

Mocking Limitations 🔗

Mocking works to a certain extent, but what happens if we add an object lock to the S3 bucket?

/*
* serverless.ts
* 
* Using Serverless Framework to handle the provisioning of our infrastructure.
*/

const serverless = {
    s3: {
        bucketName: "example_bucket",
        versioning: true,
        Properties: {
            objectLock: true
        }
    }
}

One of the rules in S3 is that when a bucket has an object lock, any objects that get PUT in have to have a precomputed md5 hash attached to that same payload request.

However, in theory our tests should still pass. This is because we’ve mocked the S3 class, so that it wouldn’t matter if our bucket had a lock or not. And they do indeed pass.

    [tick] 1 test passed

But is that what we really want to find out when it comes to hitting our test environments? Should that be slipping through code reviews? Surely the tests that we write as developers should yield enough confidence to deploy without configuration issues like this being apparent.

Emulating through LocalStack 🔗

Enter LocalStack. LocalStack is an open source application that allows anybody to spin up AWS locally. It works through emulating the AWS services. It’s worth noting that emulation will never be exactly the same as AWS, but its as close as you’ll get to the real deal and suffices our purposes here.

LocalStack is free and supports the most commonly used AWS service, such as S3, lambda functions, API gateways, however it works on a licensing model, where certain services that are less commonly used, such as Step Functions, are only supported in the pro version. A full list of supported/non supported services can be found here.

Having looked deeper into LocalStack, it runs off of python under the hood, and has regular automated checks to ensure parity with AWS- loss of parity could happen at any time if AWS updated their internals, LocalStack is an emulation after all.

Setting up LocalStack is straight forward, it’s packaged in docker and therefore makes it incredibly easy to put in a docker-compose file.

# docker-compose.yml

version: "3.8"

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack
    ports:
      - "127.0.0.1:4566:4566"            # LocalStack Gateway
      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
    environment:
      - DEBUG=1
      - SERVICES=S3 # Only start up S3
      - PERSISTENCE=${PERSISTENCE-}
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR-}
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

Running docker-compose up -d will start up LocalStack within a matter of seconds.

We’ll need to update our s3Client to now point to LocalStack.

/*
 * handler.ts
*/

import { S3 } from "aws/s3";

const s3LocalConfig = process.env.IS_LOCAL && {
	region: "us-east-1",
	endpoint: "http://127.0.0.1:4566",
	forcePathStyle: true
}
const s3Client = new S3(s3LocalConfig);

export default function handler() {
	const putObjectParams: PutObjectParams = {
		key: "textfile.txt",
		bucketName: "example_bucket"
	}

	const textFile = s3Client.put(putObjectParams);

	console.log("File saved!");

	return {
		statusCode: 200
	}
}

We’ll then start our tests again but though passing the IS_LOCAL=true npx jest command to run the tests again.

Successfully failed! 🔗

An error should now appear:

Unable to PUT item, need MD5 hash when calling .put()

This is exactly what we wanted. The error reflects what would have happened if we tried this against the real AWS environment.

MD5 digests are required when object locks are applied to ensure that the object made it through from the client to the S3 in one peace.
If the object only got partially uploaded, then there would be no way to rectify that without tampering with the object lock settings.

We can now confidently update the application code so that it also computes an MD5 digest prior to uploading.

/*
 * handler.ts
*/

import { S3 } from "aws/s3";

const s3LocalConfig = process.env.IS_LOCAL && {
	region: "us-east-1",
	endpoint: "http://127.0.0.1:4566",
	forcePathStyle: true
}
const s3Client = new S3(s3LocalConfig);

export default function handler() {
	const key = "textfile.txt";
	const fileContents = fs.readFile(key)
	const md5Digest = crypto.createHash('md5').update(fileContents).digest("hex");

	const putObjectParams: PutObjectParams = {
		key,
		bucketName: 1, 
		md5: "12345"
	};

	const textFile = s3Client.put(putObjectParams);

	console.log("File saved!");

	return {
		statusCode: 200
	}
}

When we re-run our tests again, they now pass with the MD5 being passed through.

    [tick] 1 test passed

What's actually happened here though? 🔗

LocalStack under the hood has emulated the S3 service, through the custom local endpoint we’ve specified in our application code, the PUT request was made to that endpoint, which matches the exact request that LocalStack’s S3 was expecting, which is parity to AWS’ S3 service.

To prove this, we can query the S3 bucket like we would with any other service through the AWS CLI.

aws s3 ls testing-bucket/my_file.txt --endpoint-url=http://localhost:4572

The results:

-rw-r--r--@   1 jimmywei  staff    16388 23 Sep 10:48 .DS_Store

Conclusion: A Hard Left Shift Approach 🔗

As software development nowadays is focused on more cloud native solutions, this challenge of maintaining confident and reliable tests increases, however LocalStack is the happy medium between achieving that.

Having a reliable suite of unit and integration tests plays a fundamental part in how we can deliver software reliably from our local machines to production, but more so, is having tests that we can have confidence in and trust.

The further left within the software development lifecycle, that is, the closer to developers, where we can detect these issues, then the faster we can iterate on fixing them before it becomes more costly.