Locust statistics in AWS Cloudwatch

I developed a simple locust plugin and open sourced it. Your questions answered:

What are you talking about?

Let me give you some context. I needed a performance test setup for one of our important services in Simpl. My tool of choice for load generation was Locust. I had my reasons and I will discuss that in a different post.

Ok, so what? Why a plugin?

Locust is a great load generation tool. It has been around for a long time, has a good community, fairly performant and supports a lot of features for load generation. But, there was still one problem that needed to be solved for our case - performance test reporting the way we wanted it.

Locust does allow us to generate reports from the UI, but that approach did not work for us since we were running it in headless mode (part of the CI setup). I also wanted the reports to live for much longer than the duration of test runs since it helps in comparitive analysis. So I decided that we need to send the metric information elsewhere to store and track them. AWS Cloudwatch metrics was a good solution for Simpl's setup - a lot of our metrics go there.

To make this happen, we needed some software which will take the metrics measured by locust and send it to AWS Cloudwatch. The idea is not a new one. If we look at listeners in the locust plugins repo, we can see solutions for this for other metrics collection systems. Timescale is one such example. I decided that a cloudwatch listener plugin is the way to go.

What does this plugin do?

The plugin is simple. It hooks into the events generated by locust, gathers the data from the event hooks, converts them into metrics and uses a client (boto3) to push it into Cloudwatch.

Give me more details buddy...

Locust provides a set of event hooks, which can be used to track various actions & events of locust.

The cloudwatch plugin works primarily using 2 event hooks (it actually uses 4, but 2 are important):

• When locust initialises, it calls the init event hook. Our plugin is bootstrapped as part of the event hook. The user of the plugin does this step. Checkout an example for Application Insights.

The cloudwatch plugin also does it in similar manner:

• When each request is completed (successful or unsuccessful), the request event hook is called. All the data related to the request like request type, URL of the request, response time etc. are passed as parameters to this call. This is the primary source of data for creating and publishing metrics of a performance test. The plugin registers this event hook when it is instantiated as part of bootstrapping. Application Insights example here - focus on line 26 (last line).

The cloudwatch plugin does it this way (line 168):

Now the data is all available, so all the plugin has to do is to push it to Cloudwatch. Sounds easy right. It is easy, but the plugin does a little trick:

Instead of just passing the metrics to Cloudwatch at the end of each request, the plugin batches them. Why? Because posting metrics to Cloudwatch costs money at request level. So, instead, it is better to batch them & send it. Cost reduction is always a good idea!😄🤩 Also, this approach optimises the time spent in publishing these metrics.

The request data is encapsulated into an object RequestResult which provides methods to convert locust raw data to meaningful Cloudwatch metrics.

Dig more using view raw.

This object is stored in gevent.queue.Queue - line 200.

Once a batch of requests has been queued in, the plugin will create metrics out of them and send them to cloudwatch in batches (lines 203-204). Two constants govern the limits of this behaviour.

The first one focuses on the limit of actual requests batched. The second one is focused on batch size of metrics sent to Cloudwatch - (one request can lead to more than one metric record - request count, response time, failure count etc.).

That is it! If you want to look at the code in full, check it here

Where does the plugin run now?

The plugin (a version of it) has been running in Simpl perf environments for one of the services for more than a year. More teams/services within Simpl are adopting it now.

The version which is running inside Simpl has some support for configuring AWS Cloudwatch response time alarms as well. But we have not open sourced that yet (hopefully can do it soon).

Where is the code now?

The pull request was submitted to the main github repo, and it got approved and merged into the master branch. It was released on 18th Oct as part of Release 4.1.0. You can track all releases here. The locust plugin repo has other useful plugins as well. So you should check it out fully.

How did the process of open source contribution go?

I had thought of contributing this plugin sometime back in January 2023. I raised a Github issue and discussed with my CTO Puneet Singh to get his approval- He was all for it. But before taking the next step, life got in the way and I dropped the ball; the issue got closed.

I picked it back up in the last couple of weeks and actively started the contribution process. Lars Holmberg, the maintainer of the plugins repo and the main locust repo helped me through the entire process.

We had very useful and timely interactions on the locust Slack channel. Apart from the plugin code, I created an example usage file (referred above) as well as a basic test for the plugin. Lars suggested and helped me with creating these. He also helped me improve the code of the plugin to make it easier to understand and to conform to the policies followed by the locust development process. And in quick time, the plugin code got accepted. It was a swift, constructive and satisfying process.

I have not made too many open source contributions till date, but this experience makes me want to do a lot more of it. Thanks Lars Holmberg for helping with this.

Conclusion

This is my first significant open source contribution. I know it is very late in the game, but it better late than never. This experience has made me believe that this process is a worthwhile one. My journey has only started, and I hope to do more. Also, hopefully this post will encourage you to do more of this and not wait for long like me. Wish you a happy time doing open source!