Centralized Log Management with ELK Stack

 

Logging and Log Management

Organizations that adopt multiple systems, servers and applications may find it difficult to track security logs that they generate. And with the evolution of microservice architectures, logging has become increasingly important. Security logs can help developers analyze errors, identify attacks, and gather insights. Logging allows organizations to improve their servers and systems and are essential to troubleshoot application/ infrastructure performance. Actively reviewing the security log keeps cybercriminal activities at bay. A comprehensive log management system can be tailored to alert users regarding malware detection, unauthorized login attempts, DoS attacks, data export, and other such events

Choosing the Right Logging Tool

Centralized logging assists organizations to gather, analyze, and display their event logs at a single location. Different types of logging tools are available on the internet such as Loggly, Sumo Logic, Splunk, etc. While these are some of the popular options, the cheapest alternative is maintaining an ELK (Elasticsearch, Logstash, and Kibana) Stack. They all have more or less the same features to offer. 

What is ELK Stack?

ELK Stack is a combination of Elasticsearch, Logstash, and Kibana, and is the most popular open-source log analysis platform. Logstash aggregates the logs, transforms/ parses data -> Elasticsearch stores and indexes incoming logstash data -> Kibana analyses and visualizes the data from Elasticsearch. In addition to that, Beats ships log data to Elasticsearch and Logstash, using various types of shippers for different types of files – Filebeats, Metricbeat, etc.

Well-known companies like Netflix, Stack Overflow, LinkedIn, etc. opted for ELK Stack. This shouldn’t come as a surprise considering all of the critical capabilities and services that this stack provides:

  1. A central logging system for all microservices, with real-time logging analytics and alerting system.
  2. Simplified, scaled deployment, vertically and horizontally.
  3. Data visualization that captures and displays the analytics.

Configuring ELK Stack

In the following demo, we’ll analyse NginX and Docker logs using Filebeats and visualize them in Kibana.

We first set up ELK Stack 7.8.1 on docker. You can find the file here. If you want to install the system directly please see this.

To set up the docker, run:

$ sudo docker-compose up -d

You are all set to proceed if the local host http://localhost:80 returns a positive response.

ELK Stack works

 

*username – admin; password – admin

Elastic – http://localhost:80/elastic

Elastic local host

 

Kibana – http://localhost:80/kibana

ELK Stack Kibana

 

Now that you’re all set up, let’s have a look at the logs in Kibana.

Go to Kibana ->Stack Management -> Index Pattern -> Add Index.

Add logstash-server-* and logstash-logs-*

Choose @timestamp field as time filer

Kibana Index

 

Now go to the Discover panel to see your logs.

ELK Stack Discover

 

Once you are able to see the logs, you can create visualizations to represent critical business metrics.

To create a new visualization, find the option ‘Visualize’ on the side panel and follow the instructions mentioned here. You can add more fields in the Logstash pipeline config based on your requirements and visualize daily/ monthly/ yearly/ custom time range. Here is an example:

ELK Stack example

 

Other features of ELK Stack

  • Define the structure of your logs and create visualizations accordingly.
  • Subscribe to Slack/ email alerts to be notified about ERROR logs.
  • Monitor your services such as MySQL, Kafka, Mongo, EC2 system, etc., using Metricbeat.
  • Set alerts for a daily summary of your infrastructure, based on the log data. Eg. distinct new users login.
  • Add ML pipelines in between to analyse the logs and take decisions accordingly. For instance, take a look at the load on service and predict the future load. Based on which you can scale your services in advance.

ELK Stack allows users to analyze and visualize data from any source, in any format. The stack is owned by the company Elastic that combines their three open source products Elasticsearch, Logstash, and Kibana. Which means that the stack’s centralized logging capabilities and its supplemental features are available to anyone, free of cost. This makes ELK Stack a popular choice among developers, for log analysis.

Development of a Language-Independent Microservice Architecture

 

There is a proliferation of modern programming languages now more than ever. Nevertheless, development organizations tend to stick to one or two languages, allowing them to manage their stack properly and to keep it clean. 

Programming languages are essential tools that if not chosen appropriately, could have a crippling effect on the business. For example, developers prefer Python over Java to set up machine learning algorithms, because Python has better libraries required for machine learning and artificial intelligence applications. 

Every language has a particular set of capabilities and properties. For instance, some languages are faster than others but may take time to write. Some languages consume more resources and yet make development faster. And, there are other languages that fall in between.

Microservice multilingual

Building a language-independent microservice architecture 

While it is not quite common, a microservice architecture can be built to suit multiple languages. It provides developers with the freedom to choose the right tool for their task. In the final phase of the software development lifecycle, if Docker can deploy containers irrespective of its language, its architecture can be replicated in the initial stages of building microservices as well.

However, managing multiple languages can be a huge task and a developer should definitely consider the following aspects when they plan to add more languages to their technology stack:

  • New boiler plate code, 
  • Defining models/data structures again, 
  • Documentation,
  • The environment it works on, 
  • New framework, etc.

 

Why RESTful API is not the right choice

REST API requires a new framework for every new language such as Express for Node.js, Flask for Python, or Gin for Go. Also, the developer may have to code their models/ data structures over and over for each service as well. This makes the whole process redundant and can result in several errors.

For example, it is very common to have one database model for multiple services. But, when the database is updated, it has to undergo changes for each service. To build a multilingual microservice, we need a common framework that supports multiple languages and platforms and one that is scalable.

 

Make way for gRPC and Protocol Buffers

gRPC 

gRPC is Google’s high performing, open-source remote procedure calls framework. It’s compatible with multiple languages such as Go, Node.js, Python, Java, C-Sharp, and many more. It uses RPC to provide communication capabilities to microservices. Moreover, it is a better alternative to REST. gRPC is much faster than REST. And this framework uses Protocol Buffers as the interface definition language for serialization and communication instead of JSON/ XML. It works based on server-client communication.

An in-depth analysis of gRPC is given here.

 

Protocol Buffers 

Protocol buffers are a method of serializing data that can be transmitted over wire or be stored in files. In short, Protocol buffer or Protobuf is the same as what JSON is with regards to REST. It is an alternative to JSON/ XML, albeit smaller and faster. Protobuf defines services and other data and is then compiled into multiple languages. The gRPC framework is then used to create a service.

gRPC uses the latest transfer protocol – HTTP/2, which supports bidirectional communication along with the traditional request/ response. In gRPC the server is loosely coupled with the client. In practice, the client opens a long-lived connection with the gRPC server and a new HTTP/2 stream will be opened for each RPC call.

 

How do we use Protobuf?

We first define our service in Protobuf and compile it to any language. Then, we use the compiled files to create our gRPC framework to create our server and client, as shown in the diagram below.

Protobuffers for microservices

To explain the process:

  1. Define a service using Protocol Buffers
  2. Compile that service into other languages as per choice
  3. Generate boiler-plate code in each language 
  4. Use this to create gRPC server and clients

 

Advantages of using this architecture:

  • You can use multiple languages with a single framework.
  • Single definition across all the services, which is very useful in an organisation.
  • Any client can communicate with any server irrespective of the language.
  • gRPC allows two-way communication and uses HTTP/2 which is uber fast.

 

Creating a microservice with gRPC

Let’s create a simple microservice – HelloWorldService. This is a very basic service that invokes only one method – HelloWorld – which would return the string “hello world”. 

These are the 4 steps to follow while creating a microservice: 

  1. Define the service (protocol buffer file)
  2. Select your languages
  3. Compile the defined service into selected languages
  4. Create a gRPC server and client, using the compiled files

For this simple service, we are opting 2 languages: Go and Node.js. Since gRPC works on a client-server architecture, we shall use Go on the server (since it is fast and resource efficient) and Node.js for clients (since most of the apps these days are React/ Angular).

**One can also decide to run gRPC servers with REST API too, if they do not want to create clients. They can do this by creating a REST Proxy server. Although it sounds laborious, it is actually pretty simple and will be dealt with in the ‘Compiling’ section that will follow.

 

Step 1:  Defining a service

Messages and services are defined using Protobuf in a proto file (.proto file).

The syntax is quite simple; the messages are defined and are used as Request and Response for the RPC calls, for each service. Here is a language guide for Protocol Buffers.

We can create a new folder for each service under the name of that particular service. And, for ease of accessibility, we can store all these services under the folder “services.”

The service we are defining here is “helloworld.” And each folder should contain 2 files, namely:

  • service.proto – contains definition 
  • .protolangs – contains the languages that this service should be generated in.

 

Services folder
Folder structure

Now, let’s define our service:

gRPC hello world

 

As shown above, we have defined an empty Request, a string Response and a HelloWorldService with a single RPC call HelloWorld. This call accepts requests and returns responses. You can see the full repo here.

 

Step 2: Selecting languages

Once the service has been defined, we will have to choose the languages to compile the services into. This choice is made based on service requirements and usage, and also the developer’s comfort. The different languages that one can choose are Go, Node.js, Python, Java, C, C#, Ruby, Scala, PHP, etc. As mentioned earlier, in this example, we’ll be using Go and Node.js. We’ll then add these languages to the .protolangs file, mentioning each language in a new line.

adding languages for microservices

 

Step 3: Compiling

Compiling is the most interesting part of this entire process. In step 3, we’ll compile the .proto file to the selected languages, Go and Node.js.

The Protocol Buffer comes with a command line tool called “protoc”, which compiles the service definition for use. But for each language we’ll have to download and install plugins. This can be achieved by using a dockerfile.

Namely is a docker file which includes all of this and is available to all. It has the “proto compiler” with support for all languages and additional features like documentation, validator, and even a REST Proxy server that we talked about in step-1.  

For example,


$ docker run -v `pwd`:/defs namely/protoc-all -f myproto.proto -l ruby

It accepts a .proto file and language, both of which we already have. The following is a simple bash script that will loop through our services folder, and pick up the service.proto file to compile to the languages in the .protolangs file.


#!/bin/bash
echo "Starting ... "
set x
 
REPO="`pwd`/repo"
 
function enterDir {
 echo "Entering $1"
 pushd $1 > /dev/null
}
 
function leaveDir {
 echo "Exiting"
 popd > /dev/null
}
 
function complieProto {
   for dir in */; do
       if [ -f $dir/.protolangs ]; then
           while read lang; do
               target=${dir%/*}
               mkdir -p $REPO/$lang
               rm -rf $REPO/$lang/$target
               mkdir -p $REPO/$lang/$target
               mkdir -p $REPO/$lang/$target/doc
 
               echo "  Compiling for $lang"
               docker run -v `pwd`:/defs namely/protoc-all -f $target/service.proto -l $lang --with-docs --lint $([ $lang == 'node' ] && echo "--with-typescript" || echo "--with-validator")
 
               cp -R gen/pb-$lang/$target/* $REPO/$lang/$target
               cp -R gen/pb-$lang/doc/* $REPO/$lang/$target/doc
               sudo rm -rf gen
 
           done < $dir/.protolangs
       fi
   done
}
 
function complie {
 echo "Starting the Build"
 mkdir -p $REPO
 for dir in services/; do
   enterDir $dir
   complieProto $dir
   leaveDir
 done
}
 
complie

 

  • The scripts run a loop for each folder in /services
  • It then picks up the .protolangs file and loops them again with each of the languages written in the folder. 
  • It then compiles service.proto with the language. 
  • The docker generates the files in gen/pb-{language} folder. 
  • We simply copy the content to repos/{language}/{servicename} folder.

We, then, run the script:

$ chmod +x generate.sh

$ ./genetare.sh

The file generated appears in the /repos folder.

Tip: You can host these definitions in a repository, and use the script generated in a CI/CD Pipeline to automate this process.

Microservice node file

The successfully generated service_pb for both Node.js and Go, along with some docs and validators, constitutes the boiler-plate code for our server and clients that we are about to create.

**As discussed earlier, if you don’t want to use the client and want REST-JSON APIs instead, you can create a REST Proxy by adding a single tag in the namely/protoc-all dockerfile i.e. — with-gateway. For this we’ll have to add api paths in our protofiles. Have a look at this for further information. Now, run this gateway and the REST Proxy server will be ready to serve the gRPC server. 

 

Tip: You can also host this protorepo on github as a repository. You can have a single repo for all the definitions in your organisation, in the same way as google does.

 

Step 4: gRPC Server and Client

Now that we have service_pb code for Go and Node.js, we can use it to create a server and a client. For each language the code will be slightly different, because of the differences in the languages. But the concept will remain the same.

For servers: We’ll have to implement RPC functions.

For clients: We’ll have to call RPC functions.

 

You can see the gRPC code for all languages here. With only a few lines of code we can create a server and with fewer lines of code we can create a client.

 

Server (Go): 

package main
 
import (
   "context"
   "fmt"
   "log"
   "net"
 
   helloworld "github.com/rohan-luthra/service-helloworld-go/helloworld"
   "google.golang.org/grpc"
)
 
type server struct {
}
 
func (*server) HelloWorld(ctx context.Context, request *helloworld.Request) (*helloworld.Response, error) {
   response := &helloworld.Response{
       Messsage: "hello world from go grpc",
   }
   return response, nil
}
 
func main() {
   address := "0.0.0.0:50051"
   lis, err := net.Listen("tcp", address)
   if err != nil {
       log.Fatalf("Error %v", err)
   }
   fmt.Printf("Server is listening on %v ...", address)
 
   s := grpc.NewServer()
   helloworld.RegisterHelloWorldServiceServer(s, &server{})
 
   s.Serve(lis)
}

 

 

 

As you can see, we have imported the service.pb.go that was generated by our shell script. Then we implemented the function HelloWorld – which returns the Response “hello world from go grpc.” Thus creating a gRPC server.

 

Client (Node.js):

var helloword = require('./helloworld/service_pb');
var services = require('./helloworld/service_grpc_pb');
 
var grpc = require('grpc');
 
function main() {
 var client = new services.HelloWorldServiceClient('localhost:50051',
                                         grpc.credentials.createInsecure());
 var request = new helloword.Request();
 var user;
 if (process.argv.length >= 3) {
   user = process.argv[2];
 } else {
   user = 'world';
 }
 client.helloWorld(request, function(err, response) {
     if(err) console.log('Node Client Error:', err)
     else console.log('Node Client Message:', response.getMesssage());
 });
}
 main();  

 

We have imported the service_pb.js file which was generated by our shell script. Add the gRPC server address and call the HelloWorld function and output response to console.

 

Test the code

Run the server and make sure that the code works. 

Testing code Microservice

Now that our server is running, let’s make a call from our Node.js client: 

Testing Node.js

 *Ignore the warning.

 

When we receive the console output “hello world from go grpc,” we can conclude that everything worked out as expected.

Thus, we have successfully created a gRPC server and client with a single proto definition and a shell script. This was a simple example of an RPC call returning a “hello world” text. But you can do almost anything with it. For example, you can create a CRUD microservice that performs add, get, update RPC calls, or an operation of your choice. You just have to define it once and run the shell script. You can also call one service from another, by creating clients wherever you want or in any language you want. This is an example of a perfect microservice architecture.

 

Summary

Hope this blog helped you build a microservice architecture with just Protocol Buffers, gRPC, a docker file, and a shell script. This architecture is language-independent which means that it suits multiple programming languages. Moreover, it is almost 7 times faster than the traditional REST server, with additional benefits of documentation and validations. 

Not only is the architecture language-independent, this allows you to manage all the data structures in your organisation under a single repository, which can be a game changer. Protocol buffers also support import, inheritance, tags, etc.

You just have to follow these steps when you create a new microservice:

  1. Define your service
  2. Select languages
  3. Compile i.e. run the shell script (automated – can be done on CI/CD pipeline)
  4. Finally, code your servers and clients.
  5. Deploy however you want (Suggestion: use docker containers)

Now you are equipped to create multiple microservices, enable communication between microservices, or just use them as it is and also add a REST Proxy layer to build APIs. Remember that all services use a single framework irrespective of their programming language.

You can find the whole code repo here.

 

Bonus: 

You can also publish the generated proto files of a service as packages for each language, for e.g. Node.js – npm, Python – pip, golang – GitHub, Java – Maven and so on. Run “npm install helloworld-proto to call the .pb files. And if someone updates the Proto definition you just have to run “npm update helloworld-proto”.