Tag: Research

There is a proliferation of modern programming languages now more than ever. Nevertheless, development organizations tend to stick to one or two languages, allowing them to manage their stack properly and to keep it clean. 

Programming languages are essential tools that if not chosen appropriately, could have a crippling effect on the business. For example, developers prefer Python over Java to set up machine learning algorithms, because Python has better libraries required for machine learning and artificial intelligence applications. 

Every language has a particular set of capabilities and properties. For instance, some languages are faster than others but may take time to write. Some languages consume more resources and yet make development faster. And, there are other languages that fall in between.

Building a language-independent microservice architecture 

While it is not quite common, a microservice architecture can be built to suit multiple languages. It provides developers with the freedom to choose the right tool for their task. In the final phase of the software development lifecycle, if Docker can deploy containers irrespective of its language, its architecture can be replicated in the initial stages of building microservices as well.

However, managing multiple languages can be a huge task and a developer should definitely consider the following aspects when they plan to add more languages to their technology stack:

• New boiler plate code, 
• Defining models/data structures again, 
• Documentation,
• The environment it works on, 
• New framework, etc.

Why RESTful API is not the right choice

REST API requires a new framework for every new language such as Express for Node.js, Flask for Python, or Gin for Go. Also, the developer may have to code their models/ data structures over and over for each service as well. This makes the whole process redundant and can result in several errors.

For example, it is very common to have one database model for multiple services. But, when the database is updated, it has to undergo changes for each service. To build a multilingual microservice, we need a common framework that supports multiple languages and platforms and one that is scalable.

Make way for gRPC and Protocol Buffers

gRPC 

gRPC is Google’s high performing, open-source remote procedure calls framework. It’s compatible with multiple languages such as Go, Node.js, Python, Java, C-Sharp, and many more. It uses RPC to provide communication capabilities to microservices. Moreover, it is a better alternative to REST. gRPC is much faster than REST. And this framework uses Protocol Buffers as the interface definition language for serialization and communication instead of JSON/ XML. It works based on server-client communication.

An in-depth analysis of gRPC is given here.

Protocol Buffers 

Protocol buffers are a method of serializing data that can be transmitted over wire or be stored in files. In short, Protocol buffer or Protobuf is the same as what JSON is with regards to REST. It is an alternative to JSON/ XML, albeit smaller and faster. Protobuf defines services and other data and is then compiled into multiple languages. The gRPC framework is then used to create a service.

gRPC uses the latest transfer protocol – HTTP/2, which supports bidirectional communication along with the traditional request/ response. In gRPC the server is loosely coupled with the client. In practice, the client opens a long-lived connection with the gRPC server and a new HTTP/2 stream will be opened for each RPC call.

How do we use Protobuf?

We first define our service in Protobuf and compile it to any language. Then, we use the compiled files to create our gRPC framework to create our server and client, as shown in the diagram below.

To explain the process:

1. Define a service using Protocol Buffers
2. Compile that service into other languages as per choice
3. Generate boiler-plate code in each language 
4. Use this to create gRPC server and clients

Advantages of using this architecture:

• You can use multiple languages with a single framework.
• Single definition across all the services, which is very useful in an organisation.
• Any client can communicate with any server irrespective of the language.
• gRPC allows two-way communication and uses HTTP/2 which is uber fast.

Creating a microservice with gRPC

Let’s create a simple microservice – HelloWorldService. This is a very basic service that invokes only one method – HelloWorld – which would return the string “hello world”. 

These are the 4 steps to follow while creating a microservice: 

1. Define the service (protocol buffer file)
2. Select your languages
3. Compile the defined service into selected languages
4. Create a gRPC server and client, using the compiled files

For this simple service, we are opting 2 languages: Go and Node.js. Since gRPC works on a client-server architecture, we shall use Go on the server (since it is fast and resource efficient) and Node.js for clients (since most of the apps these days are React/ Angular).

**One can also decide to run gRPC servers with REST API too, if they do not want to create clients. They can do this by creating a REST Proxy server. Although it sounds laborious, it is actually pretty simple and will be dealt with in the ‘Compiling’ section that will follow.

Step 1:  Defining a service

Messages and services are defined using Protobuf in a proto file (.proto file).

The syntax is quite simple; the messages are defined and are used as Request and Response for the RPC calls, for each service. Here is a language guide for Protocol Buffers.

We can create a new folder for each service under the name of that particular service. And, for ease of accessibility, we can store all these services under the folder “services.”

The service we are defining here is “helloworld.” And each folder should contain 2 files, namely:

• service.proto – contains definition 
• .protolangs – contains the languages that this service should be generated in.

Now, let’s define our service:

As shown above, we have defined an empty Request, a string Response and a HelloWorldService with a single RPC call HelloWorld. This call accepts requests and returns responses. You can see the full repo here.

Step 2: Selecting languages

Once the service has been defined, we will have to choose the languages to compile the services into. This choice is made based on service requirements and usage, and also the developer’s comfort. The different languages that one can choose are Go, Node.js, Python, Java, C, C#, Ruby, Scala, PHP, etc. As mentioned earlier, in this example, we’ll be using Go and Node.js. We’ll then add these languages to the .protolangs file, mentioning each language in a new line.

Step 3: Compiling

Compiling is the most interesting part of this entire process. In step 3, we’ll compile the .proto file to the selected languages, Go and Node.js.

The Protocol Buffer comes with a command line tool called “protoc”, which compiles the service definition for use. But for each language we’ll have to download and install plugins. This can be achieved by using a dockerfile.

Namely is a docker file which includes all of this and is available to all. It has the “proto compiler” with support for all languages and additional features like documentation, validator, and even a REST Proxy server that we talked about in step-1.  

For example,

$ docker run -v `pwd`:/defs namely/protoc-all -f myproto.proto -l ruby

It accepts a .proto file and language, both of which we already have. The following is a simple bash script that will loop through our services folder, and pick up the service.proto file to compile to the languages in the .protolangs file.

#!/bin/bash
echo "Starting ... "
set x
 
REPO="`pwd`/repo"
 
function enterDir {
 echo "Entering $1"
 pushd $1 > /dev/null
}
 
function leaveDir {
 echo "Exiting"
 popd > /dev/null
}
 
function complieProto {
   for dir in */; do
       if [ -f $dir/.protolangs ]; then
           while read lang; do
               target=${dir%/*}
               mkdir -p $REPO/$lang
               rm -rf $REPO/$lang/$target
               mkdir -p $REPO/$lang/$target
               mkdir -p $REPO/$lang/$target/doc
 
               echo "  Compiling for $lang"
               docker run -v `pwd`:/defs namely/protoc-all -f $target/service.proto -l $lang --with-docs --lint $([ $lang == 'node' ] && echo "--with-typescript" || echo "--with-validator")
 
               cp -R gen/pb-$lang/$target/* $REPO/$lang/$target
               cp -R gen/pb-$lang/doc/* $REPO/$lang/$target/doc
               sudo rm -rf gen
 
           done < $dir/.protolangs
       fi
   done
}
 
function complie {
 echo "Starting the Build"
 mkdir -p $REPO
 for dir in services/; do
   enterDir $dir
   complieProto $dir
   leaveDir
 done
}
 
complie

• The scripts run a loop for each folder in /services. 
• It then picks up the .protolangs file and loops them again with each of the languages written in the folder. 
• It then compiles service.proto with the language. 
• The docker generates the files in gen/pb-{language} folder. 
• We simply copy the content to repos/{language}/{servicename} folder.

We, then, run the script:

$ chmod +x generate.sh

$ ./genetare.sh

The file generated appears in the /repos folder.

Tip: You can host these definitions in a repository, and use the script generated in a CI/CD Pipeline to automate this process.

The successfully generated service_pb for both Node.js and Go, along with some docs and validators, constitutes the boiler-plate code for our server and clients that we are about to create.

**As discussed earlier, if you don’t want to use the client and want REST-JSON APIs instead, you can create a REST Proxy by adding a single tag in the namely/protoc-all dockerfile i.e. — with-gateway. For this we’ll have to add api paths in our protofiles. Have a look at this for further information. Now, run this gateway and the REST Proxy server will be ready to serve the gRPC server. 

Tip: You can also host this protorepo on github as a repository. You can have a single repo for all the definitions in your organisation, in the same way as google does.

Step 4: gRPC Server and Client

Now that we have service_pb code for Go and Node.js, we can use it to create a server and a client. For each language the code will be slightly different, because of the differences in the languages. But the concept will remain the same.

For servers: We’ll have to implement RPC functions.

For clients: We’ll have to call RPC functions.

You can see the gRPC code for all languages here. With only a few lines of code we can create a server and with fewer lines of code we can create a client.

Server (Go): 

package main
 
import (
   "context"
   "fmt"
   "log"
   "net"
 
   helloworld "github.com/rohan-luthra/service-helloworld-go/helloworld"
   "google.golang.org/grpc"
)
 
type server struct {
}
 
func (*server) HelloWorld(ctx context.Context, request *helloworld.Request) (*helloworld.Response, error) {
   response := &helloworld.Response{
       Messsage: "hello world from go grpc",
   }
   return response, nil
}
 
func main() {
   address := "0.0.0.0:50051"
   lis, err := net.Listen("tcp", address)
   if err != nil {
       log.Fatalf("Error %v", err)
   }
   fmt.Printf("Server is listening on %v ...", address)
 
   s := grpc.NewServer()
   helloworld.RegisterHelloWorldServiceServer(s, &server{})
 
   s.Serve(lis)
}

As you can see, we have imported the service.pb.go that was generated by our shell script. Then we implemented the function HelloWorld – which returns the Response “hello world from go grpc.” Thus creating a gRPC server.

Client (Node.js):

var helloword = require('./helloworld/service_pb');
var services = require('./helloworld/service_grpc_pb');
 
var grpc = require('grpc');
 
function main() {
 var client = new services.HelloWorldServiceClient('localhost:50051',
                                         grpc.credentials.createInsecure());
 var request = new helloword.Request();
 var user;
 if (process.argv.length >= 3) {
   user = process.argv[2];
 } else {
   user = 'world';
 }
 client.helloWorld(request, function(err, response) {
     if(err) console.log('Node Client Error:', err)
     else console.log('Node Client Message:', response.getMesssage());
 });
}
 main();  

We have imported the service_pb.js file which was generated by our shell script. Add the gRPC server address and call the HelloWorld function and output response to console.

Test the code

Run the server and make sure that the code works. 

Now that our server is running, let’s make a call from our Node.js client: 

 *Ignore the warning.

When we receive the console output “hello world from go grpc,” we can conclude that everything worked out as expected.

Thus, we have successfully created a gRPC server and client with a single proto definition and a shell script. This was a simple example of an RPC call returning a “hello world” text. But you can do almost anything with it. For example, you can create a CRUD microservice that performs add, get, update RPC calls, or an operation of your choice. You just have to define it once and run the shell script. You can also call one service from another, by creating clients wherever you want or in any language you want. This is an example of a perfect microservice architecture.

Summary

Hope this blog helped you build a microservice architecture with just Protocol Buffers, gRPC, a docker file, and a shell script. This architecture is language-independent which means that it suits multiple programming languages. Moreover, it is almost 7 times faster than the traditional REST server, with additional benefits of documentation and validations. 

Not only is the architecture language-independent, this allows you to manage all the data structures in your organisation under a single repository, which can be a game changer. Protocol buffers also support import, inheritance, tags, etc.

You just have to follow these steps when you create a new microservice:

1. Define your service
2. Select languages
3. Compile i.e. run the shell script (automated – can be done on CI/CD pipeline)
4. Finally, code your servers and clients.
5. Deploy however you want (Suggestion: use docker containers)

Now you are equipped to create multiple microservices, enable communication between microservices, or just use them as it is and also add a REST Proxy layer to build APIs. Remember that all services use a single framework irrespective of their programming language.

You can find the whole code repo here.

Bonus: 

You can also publish the generated proto files of a service as packages for each language, for e.g. Node.js – npm, Python – pip, golang – GitHub, Java – Maven and so on. Run “npm install helloworld-proto” to call the .pb files. And if someone updates the Proto definition you just have to run “npm update helloworld-proto”.

Let us assume search engines, like Google, never existed! How would you find what you need from across 4.2 billion web pages? Web crawlers are programs written to browse the internet, to gather information, index and parse the collected data, to facilitate quick searches. Crawlers are, thus, a smart solution to big data sets and a catalyst to major advancements in the field of cyber security.

In this article, we will learn:

1. What is crawling?
2. Applications of crawling
3. Python modules used for crawling
4. Use-case: Fetching downloadable URLs from YouTube using crawlers
5. How do CloudSEK Crawlers work?

What is crawling?

Crawling refers to the process of scraping/ extracting data from websites/ the internet using web crawlers. For instance, Google uses spider bots (crawlers) to read the content of billions of web pages and posts. Then, it gathers data from these sites and arranges them in the Google Search index.

Basic stages of crawling: 

1. Scrape data from the source
2. Parse the collected data
3. Clean the data of any noise or duplicate entries
4. Structure the data as per requirement

Applications of crawling

Organizations crawl and scrape data off of web pages for various reasons that may benefit them or their customers. Here are some lesser known applications of crawling: 

• Comparing data for market analysis
• Monitoring data leaks
• Preparing data sets for Machine Learning algorithms
• Fact-checking information on social media

Python modules used for crawling

• Requests – Allow you to send HTTP requests to web pages
• Beautifulsoup – Python library that retrieves data from HTML and XML files, and parses its elements to the required format
• Selenium – Open source testing suite used for web applications. It also performs browser actions to retrieve data.

Use-case: Fetching downloadable URLs from YouTube using crawlers

A single YouTube video may have several downloadable URLs, based on: its content, resolution, bitrate, range and VR/3D. Here is a sample API and CLI code to get downloadable URLs on YouTube along with their Itags:

Project structure 

youtube
|
|---- app.py
|---- cli.py
`---- core.py


The project will contain three files:

app.py: For the api interface, using flask micro framework

cli.py: For command line interface, using argparse module

core.py: Contains all the core (common) functionalities which act as helper functions for app.py and cli.py.

# youtube/app.py
import flask
from flask import jsonify, request
import core
app = flask.Flask(__name__)
app.config["DEBUG"] = True
@app.route('/', methods=['GET'])
def get_downloadable_urls():
if 'url' not in request.args:
return "Error: No url field provided. Please specify an youtube url."
url = request.args['url']
urls = core.get_downloadable_urls(url)
return jsonify(urls)
app.run()


The flask interface code to get downloadable URLs through API.

Request url - localhost:<port>/?url=https://www.youtube.com/watch?v=FIVPlraNgXs

# youtube/cli.py
import argparse
import core
my_parser = argparse.ArgumentParser(description='Get youtube downloadable video from url')
my_parser.add_argument('-u', '--url', metavar='', required=True, help='youtube url')
args = my_parser.parse_args()
urls = core.get_downloadable_urls(args.url)
print(f'Got {len(urls)} urls\n')
for index, url in enumerate(urls, start=1):
print(f'{index}. {url}\n')

Code snippet to get downlodable urls through comand line interface (using argparse to parse command like arguments)

Command line interface - python cli.py -u 'https://www.youtube.com/watch?v=aWPYw7iVBg0'

# youtube/core.py
import json
import re
import requests
def get_downloadable_urls(url):
html = requests.get(url).text
RE = re.compile(r'ytplayer[.]config\s*=\s*(\{.*?\});')
conf = json.loads(RE.search(html).group(1))
player_response = json.loads(conf['args']['player_response'])
data = player_response['streamingData']
return [{'itag': frmt['itag'],'url': frmt['url']} for frmt in data['adaptiveFormats']]

This is the core (common) function for both API and CLI interface. 

The execution of these commands will:

1. Take YouTube url as an argument
2. Gather page source using the Requests module
3. Parse it and get streaming data
4. Return response objects: url and itag

How to use these URLs?

• Build your own YouTube downloader (web app)
• Build an API to download YouTube video

Sample result 

[{
'itag': 251,
'Url': 'https://r2---sn-gwpa-h55k.googlevideo.com/videoplayback?expire=1585225812&ei=9Et8Xs6XNoHK4-EPjfyIiA8&ip=157.46.68.124&id=o-AGeDi3DVtAbmT5GiuGsDU7-NPLk23fOXNnY16gGQcHWu&itag=251&source=youtube&requiressl=yes&mh=Av&mm=31%2C26&mn=sn-gwpa-h55k%2Csn-cvh76ned&ms=au%2Conr&mv=m&mvi=1&pl=18&initcwndbps=112500&vprv=1&mime=audio%2Fwebm&gir=yes&clen=14933951&dur=986.761&lmt=1576518368612802&mt=1585204109&fvip=2&keepalive=yes&fexp=23882514&c=WEB&txp=5531432&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ADKhkGMwRAIgK4L4VVHAlWMPVPEcmdkhnb2u8UM6eYhFz16kGruxZjUCIFXZJM9ejVK7OZJFqx7YwBqa3CrDvVakuU86vcIyMv-a&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=ABSNjpQwRAIgKBhJytjv73-c7eMWbVkb-X8_rNb7_xApZvaPfw7wGcMCIHqJ405fQ3Kr-e_5fV8gokMUNi0rrrLG8T85sLGTQ17W'
}]

What is ITag?

ITag gives us more details about the video such as the type of video content, resolution, bitrate, range and VR/3D. A comprehensive list of YouTube format code ITags can be found here.

How do CloudSEK Crawlers work?

CloudSEK’s digital risk monitoring platform, XVigil, scours the internet across surface web, dark web, and deep web, to automatically detect threats and alert customers. After configuring a list of keywords suggested by the clients, CloudSEK Crawlers:

1. Fetch data from various sources on the internet
2. Push the gathered data to a centralized queue
3. ML Classifiers group the data into threats and non-threats
4. Threats are immediately reported to clients as alerts, via XVigil. Non-threats are simply ignored.

Introduction

The process of threading allows the execution of multiple instructions of a program, at once. Only multi-threaded programming languages like Python support this technique. Several I/O operations running consecutively decelerates the program. So,  the process of threading helps to achieve concurrency.

In this article, we will explore:

1. Types of concurrency in Python
2. Global Interpreter Lock
3. Need for Global Interpreter Lock
4. Thread execution model in Python 2
5. Global Interpreter Lock in Python 3

Types of Concurrency In Python

In general, concurrency is the parallel execution of different units of a program, which helps optimize and speed up the overall process. In Python, there are 3 methods to achieve concurrency:

1. Multi-Threading
2. Asyncio
3. Multi-processing

We will be discussing the fundamentals of thread-execution model. Before going into the concepts directly, we shall first discuss Python’s Global Interpreter Lock (GIL).

Global Interpreter Lock

Python threads are real system threads (POSIX threads). The host operating system fully manages the POSIX threads, also known as p-threads.

In multi-core operating systems, the Global Interpreter Lock (GIL) prevents the parallel execution of p-threads of a multi-threaded Python process. Thus, ensuring that only one thread runs in the interpreter, at any given time.

Why do we need Global Interpreter Lock?

GIL helps to simplify, the implementation of the interpreter, and memory management. To understand how GIL does this, we need to understand reference counting.

For example: In the code below, ‘b’ is not a new list. It is just a reference to the previous list ‘a.’

>>> a = []
>>> b = a
>>> b.append(1)
>>> a
[1]
>>> a.append(2)
>>> b
[1,2]

Python uses reference counting variables to track the number of references that point to an object. The memory occupied by the object is released if the value of the reference counting variable is zero. If threads, of a process sharing the same memory, try to access this variable to increment and decrement simultaneously, it can cause leaked memory that is never released, or releasing the memory incorrectly. And this leads to crashes.

One solution is to have a lock for the reference counting variable object memory by using semaphores so that it is not modified simultaneously. However, adding locks to all objects increases the performance overhead in the acquisition and release of locks. Hence, Python has a GIL which gives access to all resources of a process to only one thread at one time. Apart from GIL there are other solutions, such as garbage collection used in JPython interpreter, for memory management.

So, the primary outcome of GIL is, instead of parallel computing you get pre-emptive (threading) and co-operative multitasking (asyncio).

Thread Execution Model in Python 2

import time
def countdown(n):
     while n > 0:
     n -= 1
count = 50000000
start = time.time()
countdown(count)
end = time.time()
print('Time taken in seconds -', end-start)

import time
from threading import Thread
COUNT = 50000000
def countdown(n):
     while n>0:
     n -= 1
t1 = Thread(target=countdown, args=(COUNT//2,))
t2 = Thread(target=countdown, args=(COUNT//2,))
​start = time.time()
t1.start()
t2.start()
t1.join()
t2.join()
end = time.time()
​print('Time taken in seconds -', end - start)

Ideally, the par.py execution time should be half of the seq.py execution time. However, in the above example we can see that the par.py execution time is slightly higher than that of seq.py. To understand the reduction in performance, despite the sharing the work between two threads which run in parallel, we need to first discuss CPU-bound and I/O-bound threads.

CPU-bound threads are threads performing CPU intense operations such as matrix multiplication or nested loop operations. Here, the speed of program execution depends on CPU performance.

I/O-bound threads are threads performing I/O operations such as listening to a socket or waiting for a network connection. Here, the speed of program execution depends on factors including external file systems and network connections.

Scenarios in a Python multi-threaded program

When all threads are I/O-bound

If a thread is running, it holds the GIL. When the thread hits the I/O operation, it releases the GIL, and another thread acquires it to get executed. This alternate execution of threads is called multi-tasking.

Where one thread is CPU bound, and another thread is IO-bound:

A CPU bound thread is a unique case in thread execution. A CPU bound thread releases the GIL after every few ticks and tries to acquire again. A tick is a machine instruction. When releasing the GIL, it also signals the next thread in the execution queue (ready queue of the operating system) that the GIL has been released. Now, these CPU-bound and I/O-bound threads are in a race to acquire the GIL. The operating system decides which thread needs to be executed. This model of executing the next thread in the execution queue, before completing the previous thread, is called pre-emptive multitasking.

In most of the cases, operating system gives preference to the CPU bound thread and allows it to reacquire the GIL, leaving the I/O-bound thread starving. In the below diagram, the CPU-bound thread has released GIL and signaled thread 2. But even before thread 2 tries to acquire GIL, the CPU-bound thread has reacquired the GIL.  This issue has been resolved in Python 3 interpreter’s GIL.

In a single core operating system, if the CPU bound thread reacquires GIL, it pushes back the second thread to the ready queue, assigning it some priority. This is because Python doesn’t have control over the priorities assigned by the operating system.

In a multicore operating system, if the CPU bound thread reacquires the GIL then it does not push back the second thread, but continuously tries to acquire the GIL using another core. This characterizes thrashing. Thrashing reduces the performance if many threads try to acquire the GIL, using different cores of the operating system. Python 3 also addresses the issue of thrashing.

Global Interpreter Lock in Python 3

The Python 3 threat execution model has a new GIL. If there is only one thread running, it continues to run, until it hits an I/O operation or other thread requests, to drop the GIL. A global variable (gil_drop_request) helps to implement this.

If gil_drop_request = 0, running thread can continue until it hits I/O

If gil_drop_request = 1, running thread is forced to give up the GIL

Instead of CPU-bound thread check after every few ticks, the second thread is sending a GIL drop request by setting the variable gil_drop_request = 1 after reaching a timeout. The first thread will then immediately drop the GIL. Additionally, to suspend the first thread’s execution, the second thread sends a signal. This helps to avoid thrashing. This check is not available in Python 2.

Missing Bits in the New GIL

While the new GIL does address issue such as thrashing, it still has some areas of improvement:

Waiting for time out

Waiting for timeout can make the I/O response slow. Especially, when there are multiple, recurrent I/O operations. I/O-bound Python programs take considerable time to add a GIL, followed by a time out. This happens after every I/O operation, and before the next input/output is ready.

Unfair GIL acquiring

As seen below, the thread that makes the GIL drop request is not the one that gets the GIL. This type of situation can reduce the performance of I/O, where response time is critical.

Prioritizing threads

There is a need for the GIL to distinguish between CPU-bound and I/O-bound threads and then assign priorities. High priority threads must be able to immediately preempt low priority threads. This will help improve the response time considerably.  This issue has already been resolved in operating systems. Operating systems use timeout to automatically adjust task priorities. If a thread is preempted by a timeout, it is penalized with a low priority. Conversely, if a thread suspends early, it is rewarded with raised priority. Incorporating this in Python will help improve thread performance.