The serverless approach to testing is different and may actually be easier

I’ve been thinking a lot about testing recently. At work we have recently increased the number of our lambda functions by a significant amount due to additions of client applications and addition of features. This isn’t a massive deal to develop new features, but there has been something that has been beginning to bug me (if you’ll excuse the pun).

I’m all for creating tests. Whether it’s true “Test Driven Development” — or whatever the testing methodology du jour is now — is immaterial to me. Sometimes in a startup, you just have to deploy something fast, and write a test later (I know, I know — but I’m just giving people who’ve never worked in a startup the real world scenarios). And sometimes, the tests never get written because you think that your use case is already caught (it isn’t).

Often tests get written because of bugs occurring in the production environment. This will always occur unless you have endless money and time — which you won’t in a startup.

Tests are vitally important.

But if you’re using the prevailing testing wisdom — serverless is hard.

Serverless architecture uses a lot of services — hence why some prefer to call the architecture “service-full” instead of serverless. Those services are essentially elements of an application that are independent of your testing regime.

An external element.

A good external service will be tested for you. And that’s really important. Because you shouldn’t have to test the service itself. You only really need to test the effect of your interaction with it.

Here’s an example …

Let’s say you have a Function as a Service (e.g. Lambda function) and you utilise a database service (e.g. DynamoDB). You’ll want to test the interaction with the database service from the function to ensure your data is saved/read correctly, and that your function can deal with the responses from the service.

Now, the above scenario is relatively easy because you can utilise DynamoDB from your local machine, and run unit tests to check the values stored in the database. But have you spotted something with this scenario? It’s not the live service — it’s a copy of it. But the API is the same. So, as long as the API doesn’t change we’re ok, right?

To be honest, I’ve reached a point where I’m realising that if we use an AWS service, the likelihood is that AWS have done a much better job of testing it than I have. So we mock the majority of our interactions with AWS (and other) services in unit tests. This makes it relatively simple to develop a function of logic and unit test it — with mocks for services required.

This is similar to when using a framework such as Rails. You shouldn’t be testing that the ORM works. That’s the ORM maintainers job, not yours. So it stands to reason that if a service provides an interface and documentation about how the interface works, then it should be fine — right?

Hopefully…

Here’s where there is a problem with serverless… sort of. Unit tests are easy with a FaaS function because the logic is often tiny. There is a tendency to an over reliance on mocks in my view but it works.

All other forms of testing are hard. In fact, I’d say we’ve possibly moved into needing a different paradigm to discuss this.

Through years of building monolithic applications, we’ve got absolutely obsessed that certain types of testing are absolutely vital — and if we don’t have them we’re “wrong”.

So let’s just step back a bit.

We’ve actually been having the discussion about distributed systems and testing for a while. The microservice patterns have shown us that it’s not always appropriate and often expensive to try to test everything in the way we do a monolith.

The key for integration testing with a microservice pattern is that you test the microservice and it’s integration with external components. Which is interesting, because you’re still imagining some sort of separation here.

In Lambda, in this context, every single Lambda needs to be treated as a microservice then for testing. Which means that your function’s unit tests (with mocks) need to be expanded to integration tests by removing the mocks, and using the actual service or stubbing the service in some way.

Unfortunately not every external service is easily testable in this way. And not every service provides a test interface for you to work with — nor do some services makes it easy to stub themselves. I would suggest that if a service can’t provide you with a relatively easy way to test the interface in reality, then you should consider using another one.

This is especially true when a transaction is financial. You don’t want a test to actually cost you any real money at this point!

For me, the easiest way to test a serverless system as a whole is to generate a separate system in a non-linked AWS account (or other cloud provider). Then make every external service essentially link to a “test” service, or as best we can limit our exposure to cost.

This is how I’ve approached it — and it relies on Infrastructure as Code to make it happen. Hence, the use of something like Terraform or CloudFormation.

But interestingly, when you go beyond a single function like this in a microservice approach, you get onto things like component testing and then system testing. Essentially testing is about increasing the test boundary each time. Start with a small test boundary and work out.

Unit testing, then integration, and so on …

But interestingly, our unit tests are doing the job of testing the boundary of each function reasonably well, plus doing the unit test, and also testing the function’s relation to external services reasonably well. So the next step is to test a combination of the services together.

But since we’re using external services for the majority of our interactions, and not invoking functions from within functions very often, then the test boundaries are actually relatively uncoupled.

Hmm… so basically, the more uncoupled a function’s logic is from other function’s logic, the closer the test boundary is as we move outwards in tests.

So after good unit and integration tests on a Function by Function basis, what comes next? Is it simply end to end testing next? This becomes really interesting, since that means testing the entire “distributed system” in a staging style environment with reasonable data.

Basically, what it seems to happen with a Function as a Service approach is that the suite of tests seem a lot simpler than you would normally do with a monolithic or even a microservice approach.

The test boundaries for unit testing a FaaS Function appears to be very close to an integration test versus a component test within a microservice approach.

Quick Caveat: if you do lots of function to function invocations, then you are coupling those functions and then test boundaries will change. Functions invoking functions make a separate test boundary to worry about.

Which comes back to something else very interesting. If you build functions, and develop an event driven approach utilising third party services (e.g. SNS, DynamoDB Triggers, Kinesis, SQS in the AWS world) as the event connecting “glue” — then you may be able to essentially limit yourself to testing the functions separately and then the system.

Not exactly, but close.

I would suggest the system testing is harder. If you’re purely using an API Gateway with Lambdas behind it, then you can use third party tools to test the HTTP endpoints and build a test suite that way. It’s relatively understood.

But if you’re doing a lot of internal event triggering, such as DynamoDB triggers setting of a chain of events and multiple lambdas, then you have to do something different. This form of testing is harder, but since everything is a service — including the Lambda — it should be relatively simple to do.

The person that builds the tool for this kind of system testing with serverless will do very well. At present, the CI/CD tools we have and testing tools around it are not (quite) good enough.

When I started thinking about this article, I was expecting to figure out a lot of things around how to fit better testing regimes into our workflow.

As this article has come together, what’s happened is an identification of why serverless approaches are different to monolithic and microservice approaches. As a result, I’ve realised the inherent advantages for testing of smaller bits of uncoupled logic.

You can’t just drag your old “Testing for Monoliths Toolbox” into the serverless world and expect it to work any more.

Testing in serverless is different.

Testing in serverless may actually be easier.

In fact, testing in Serverless may actually be easier to maintain as well.

But we’re currently lacking the testing tools to really drive home the value. — looking forward to when they arrive.

I’m often a reluctant test writer. I like to hack and find things out as I go before building things to “work”. I’ve never been one of the kinds of people to force testing in any scenario so I may be missing the point in some of this. There are definitely people more qualified to talk about testing than me, but these are simply thoughts on testing.

Master modern tech skills, get certified, and level up your career. Whether you’re starting out or a seasoned pro, you can learn by doing and advance your career in cloud with ACG.

Additional Resources

Testing Strategies in a Microservice Architecture — There has been a shift in service based architectures over the last few years towards smaller, more focussed "micro… — martinfowler.com

bliki: TestPyramid — tags: The test pyramid is a concept developed by Mike Cohn, described in his book Succeeding with Agile. Its essential… — martinfowler.com

Just Say No to More End-to-End Tests — by Mike Wacker At some point in your life, you can probably recall a movie that you and your friends all wanted to see… — testing.googleblog.com