Observability For Automation Tester

Reading Time: 5 minutes

It is very common yet quite new to hear the term ‘Observability’. But what does that mean?

In this current modern technology world where we are working with so many different types of systems – microservices, distributed systems and many others which are kind of huge spider webs. Imagine while testing these distributed systems and you have no clue of what’s going on under the hood. Gone are the days where testers had to rely only on testing the user interface or the API’s.

There are many cases in distributed systems where no one had an answer to what’s going wrong whenever there was a production issue each time. We had some monitoring and logging in place but we had no clue where to look at when things went wrong on production. There was a need to have more powerful insights of the internals of the system and more than that there was a need for visibility to understand what’s happening under the hood of the system which is giving the team the superpowers to predict the future.

Observability and Test

From a tester’s eye, There is no replacement for the deeper level of information that a observable system can provide. Although on a practical level observability has three pillars : logs, metrics and traces that we will discuss later in this blog thoroughly.

Imagine you join a new team where you work on a new product or service and the domain is totally new for you, so as a tester what is the first thing you try to do?

You try to explore the product or service to understand so that once you understand you can start building your test strategy and start working on your test suite and test cases.

So what if, the more you are trying to understand it and feels that the product is very complex and whenever there is a production issue, Developers on your teams marking those tickets blocked/Invalid because they could not find any information and they could not debug it to fix the issue. There might be a lot of reasons but the most important thing is the lack of visibility. When there is no visibility into the system then anyone who is trying to debug they can not find any information.

So how can we get visibility in the system? Here comes OBSERVABILITY in the picture.

It’s just not about getting effective metrics, well-structured logs and extensive tracing. we also need a productive mindset that is inquisitive, exploratory and hunger to learn and the processes that can make all of those things meaningful and impactful.

Observability : Definition

Imagine a scenario whenever the systems are down and we need to find answers by asking questions as quickly as possible. So the system needs to be observable so that we can find what is happening inside the system by just observing from the outside but the question is how can we make the system observable.

So the answer is by using the DATA.

But how can we get the data and what type of data do we need to have an observable system?

We can get the data to make the system observable by adding instrumentation. And that instrumentation can give us the data that can be in the form of logs, it can be in the form of metrics or it can be in the form of traces/events.

So it is all about collecting the data whenever there are request and services running, Visualising those in the form of Logs, Metrics and Traces. Then use that data to find out the root cause of any given issue.

How Observability Helps Test Engineer

So now you might be thinking why should automation testers care about observability or how it could help automation testers. So we automation testers are great at being creative and we are good at asking questions and we are also good at exploration by performing exploratory testing.

By having an observable system it helps automation testers to find more information and it empowers our ability to understand the production which can also help us uncover the behaviour of real uses. Like we don’t have to guess or estimate that our end users might be using our systems. What are the user journeys they are taking or which are the critical part of the system the users are using regularly.

So ultimately we don’t have to guess all those things and by gathering all this information or data it doesn’t mean that we are just collecting the information but by collecting that information it allows us to design our test methods. By having access to these observability tools for automation testers especially when we are trying to test something or maybe we are trying to reproduce a production bug we don’t have to go blind and assume things.

By accessing these logs , Metrics and any of the traces we can find more information and this information can be used while you might be testing or you might re-producing any of the bugs or you can add more information on bugs which you are trying to raise and that way you might help developers to debug things easily.

How to maintain Observability in Test

So, We have understood that observability can help automation tester in many ways that we have seen above. but how a tester can start with observability with respect to test ? If you are in a team that has already a bad observability graphs. In such scenarios you can atleast kick-off while asking high level of questions about product/service.

Although, There are some tools like Grafana, Prometheus, AppDynamics etc. which allows testers to see their data via charts and graphs that are unified into one dashboard (or multiple dashboards!) for easier interpretation and understanding.

But initially you can get started with some fundamental questions a QA must asks while preparing to test a new feature.

You can ask about any existing logs team is having.
You can ask about it’s format and how it is structured.
You can ask about alerting state if having and if not than what about alerts if this feature goes live.
You can ask who should be get alerts.
You can ask about format of dashboard team is needed and what attributes it should contain.
You can ask about implementing tracing.

While asking these questions atleast we move into glove of observability and getting answers of those questions are not just important from a testing perspective but it is critical for everyone.

Conclusion

To end this blog I would say observability not only testers but it gives more power to the entire team to get the visibility of the system or product when needed. So it empowers everyone on the team whether it’s for developer, It’s for offshore engineer or its SRE or it’s automation testers meanwhile it’s just not useful for one particular role.