Parting Thoughts on Performance Testing

After spending the last four months diving into performance testing and hunting down memory leaks, this series comes to an end. Let’s take a moment to reflect on the key insights and explore some ideas you can try on your own.

Time-box everything

Performance testing can go on forever - there is always another bottleneck to fix. Set a time limit for each session to avoid getting stuck in endless optimizations. That way, you can improve performance while still having time for everything else.

Much more to learn

Starting with performance testing can feel overwhelming. There will be moments when everything clicks, and others when it feels like an avalanche of unknowns. That is normal. Knowledge builds up in waves, sometimes smooth, sometimes rough. Take breaks, stick to your time boxes, and push through the uncertainty. It takes time to make sense of everything, but each challenge you overcome adds to your expertise.

Performance testing should be continuous

Performance testing is not a one-time task - it should be a habit. By testing early and often, we can catch issues before they impact users. This proactive approach helps prevent performance problems long before they become critical. The earlier we find bottlenecks, the easier they are to fix.

Minimalistic test scenarios take you a long way

While realistic test scenarios are great, a minimalistic test that simply floods an endpoint with requests can also be useful. These minimalistic tests will not catch every issue, but they provide a quick and cost-effective way to identify major bottlenecks. Fix the obvious problems first, then move on to more sophisticated test scenarios.

Identical infrastructure for infrastructure tests

If we want to test the infrastructure itself, we need to have an identical set-up. That is especially true when we try to reduce the infrastructure and keep our response times at the same level.

For everything else we can interpolate the results. We could run our tests on the local developer machine, check that our code improvements are indeed an improvement and deploy them to production. There we can measure again and see how the improvement in the production relates to the improvement on our developer machine.

Async/await and caching

The focus on this series was on performance testing and not on changing the architecture of our applications to improve speed. I made this decision to focus on the methodology and the tooling, without diving into application rewrites. That said, async/await and caching are two approaches that you should explore when you have a performance-critical application.

Implementing async/await in an existing application requires significant effort. It can enhance throughput by avoiding blocking operations, particularly for I/O-heavy tasks like database queries or API calls. Poorly managed asynchronous code can lead to race conditions or excessive resource consumption, that will destroy the performance gains.

Caching can reduce response times by storing frequently accessed data in memory, avoiding repeated expensive computations or database queries. It takes significantly less effort to introduce caching into an existing application. But be aware that cache-invalidation is a hard problem and outdated or improperly invalidated caches can serve stale data.

Async/Await and caching can help us to make our applications faster, but both approaches come with costs and require knowledge.

Conclusion

Performance testing is fun as soon as you dive through the obstacles and gained enough knowledge to be dangerous. The goal is to keep your focus, use time-boxes and keep learning, then there is always more going on than what we can observe.

I hope this series helped you in improving your performance testing skills. It is by no means complete and for all I wrote about are many different approaches you could take. The ones I described helped me to hunt down the problems in my applications and they may be a good starting place for your activities.