Centralized Log Management with ELK Stack


Logging and Log Management

Organizations that adopt multiple systems, servers and applications may find it difficult to track security logs that they generate. And with the evolution of microservice architectures, logging has become increasingly important. Security logs can help developers analyze errors, identify attacks, and gather insights. Logging allows organizations to improve their servers and systems and are essential to troubleshoot application/ infrastructure performance. Actively reviewing the security log keeps cybercriminal activities at bay. A comprehensive log management system can be tailored to alert users regarding malware detection, unauthorized login attempts, DoS attacks, data export, and other such events

Choosing the Right Logging Tool

Centralized logging assists organizations to gather, analyze, and display their event logs at a single location. Different types of logging tools are available on the internet such as Loggly, Sumo Logic, Splunk, etc. While these are some of the popular options, the cheapest alternative is maintaining an ELK (Elasticsearch, Logstash, and Kibana) Stack. They all have more or less the same features to offer. 

What is ELK Stack?

ELK Stack is a combination of Elasticsearch, Logstash, and Kibana, and is the most popular open-source log analysis platform. Logstash aggregates the logs, transforms/ parses data -> Elasticsearch stores and indexes incoming logstash data -> Kibana analyses and visualizes the data from Elasticsearch. In addition to that, Beats ships log data to Elasticsearch and Logstash, using various types of shippers for different types of files – Filebeats, Metricbeat, etc.

Well-known companies like Netflix, Stack Overflow, LinkedIn, etc. opted for ELK Stack. This shouldn’t come as a surprise considering all of the critical capabilities and services that this stack provides:

  1. A central logging system for all microservices, with real-time logging analytics and alerting system.
  2. Simplified, scaled deployment, vertically and horizontally.
  3. Data visualization that captures and displays the analytics.

Configuring ELK Stack

In the following demo, we’ll analyse NginX and Docker logs using Filebeats and visualize them in Kibana.

We first set up ELK Stack 7.8.1 on docker. You can find the file here. If you want to install the system directly please see this.

To set up the docker, run:

$ sudo docker-compose up -d

You are all set to proceed if the local host http://localhost:80 returns a positive response.

ELK Stack works


*username – admin; password – admin

Elastic – http://localhost:80/elastic

Elastic local host


Kibana – http://localhost:80/kibana

ELK Stack Kibana


Now that you’re all set up, let’s have a look at the logs in Kibana.

Go to Kibana ->Stack Management -> Index Pattern -> Add Index.

Add logstash-server-* and logstash-logs-*

Choose @timestamp field as time filer

Kibana Index


Now go to the Discover panel to see your logs.

ELK Stack Discover


Once you are able to see the logs, you can create visualizations to represent critical business metrics.

To create a new visualization, find the option ‘Visualize’ on the side panel and follow the instructions mentioned here. You can add more fields in the Logstash pipeline config based on your requirements and visualize daily/ monthly/ yearly/ custom time range. Here is an example:

ELK Stack example


Other features of ELK Stack

  • Define the structure of your logs and create visualizations accordingly.
  • Subscribe to Slack/ email alerts to be notified about ERROR logs.
  • Monitor your services such as MySQL, Kafka, Mongo, EC2 system, etc., using Metricbeat.
  • Set alerts for a daily summary of your infrastructure, based on the log data. Eg. distinct new users login.
  • Add ML pipelines in between to analyse the logs and take decisions accordingly. For instance, take a look at the load on service and predict the future load. Based on which you can scale your services in advance.

ELK Stack allows users to analyze and visualize data from any source, in any format. The stack is owned by the company Elastic that combines their three open source products Elasticsearch, Logstash, and Kibana. Which means that the stack’s centralized logging capabilities and its supplemental features are available to anyone, free of cost. This makes ELK Stack a popular choice among developers, for log analysis.

Why programming skills are essential for pen-testers

Why programming skills are essential for penetration testers


Some security professionals across the world would say that one does not need to learn coding to hunt for bugs in web applications. In fact, some experienced security professionals would go even further to suggest that entry-level positions in cybersecurity and hacking does not require extensive knowledge of programming.

Although this holds true to some extent, a career in hacking and pen-testing web applications demands in-depth knowledge in programming.


Where do many researchers go wrong?

In case of Cross-Site Scripting (XSS) attacks, for instance, researchers report the bugs by triggering an alert. This clearly does not call for advanced understanding of programming. 

But they may lack the skills to exploit the same bug to create a javascript code so as to steal cookies or leverage the XSS bug to carry out other malicious activities. 

Inspired by such bounty hunters, beginners in the field assume that all they have to do is fire up Burp Intruder, add a list of payloads, and prompt an alert on the browser to earn a quick buck. 


Why do you need to learn programming in security testing?

Understanding the application:

Awareness and proficiency in programming can help a researcher understand an application’s infrastructure and the implementation of its many functionalities. Once you are familiar with the workings and technicalities of web applications, even entry-level programmers can certainly outsmart amateur coding enthusiasts. 


Attack automation:

Hackers use tools such as Nmap, Metasploit, Amass, etc. to automate enumeration and exploitation processes. Automation of enumeration attacks saves them a lot of time and effort. By learning how to code, you are also opening yourself up to vast knowledge, which can guide a beginner to build such tools on their own. Apart from that, while pen-testing, a programmer at some point will have to write a code that can exploit a vulnerability; for instance, when you have to pass the current timestamp along with a request, you need to automate it using coding. This requires that you are well versed with programming.



Programming is said to be the future of innovations, and a necessary skill to master. Therefore, a security professional should undergo training and have adequate knowledge regarding programming. Anyone pursuing a career in penetration testing should consider programming as an essential part of their occupation. It does not merely set you apart from peers, but also gives you a competitive advantage over them. 


Happy Automation!