Engineering Archives

Posted on November 21, 2020 | by Theertharaja B | Leave a Comment Engineering

The Rise of Cybercrime on Telegram and Discord and the Need for Continuous Monitoring

Instant messaging, popularly called IM or IM’ing, is the exchange of near real-time messages through a stand-alone application or embedded software. Unlike chat rooms with many users engaging in multiple and overlapping conversations, IM sessions usually take place between two users in private.

One of the core features of many instant messenger clients is the ability to see whether a friend or co-worker is online on the service — a capability known as presence. As the technology has evolved, many IM clients have added support for exchanging more than just text-based messages, allowing actions like file transfers and image sharing within the IM session.

The top three messaging apps by the number of users are WhatsApp – 2 billion users, Facebook Messenger – 1.3 billion users, and WeChat at 1.12 billion users. Messenger is the top messaging app in the US. In 2017, approximately 260 million new conversations were taking place each day on the app. In total, 7 billion conversations were occurring daily.

The power of social media platforms lies in their ability to connect users and create unique avenues for interaction. For individuals, enterprises, and governments, they facilitate new ways of reaching their audience, promoting a product, and fostering communities.

The growing presence of cybercriminals on social media platforms

The universal appeal of social media platforms makes it equally attractive to cybercriminals. Yet the growing range of criminal risks encountered across social media remains significantly under-researched.

Cybercriminals, it seems, aren’t that different from consumers and enterprise users—they want tools that are easy to use and widely available. They prefer services that are simple, have a clean graphical user interface, are intuitive to use, and are not buggy. Localization and language support also make a difference. Cybercriminals are very careful about who they let into their exclusive club, but they also don’t want to jump through excessive (and costly) hoops to communicate with each other.

The rise of Telegram and Discord

The point of serious concern is that many Telegram and Discord groups are being used by cybercriminals to perform illegal activities such as selling exploits and botnets, offering hacking services, and advertising stolen data.

The double-edged sword of Telegram’s end-to-end encryption

The end-to-end encryption provided by Telegram has paved the way for a host of illegal activities, turning coveted online privacy into a breeding ground for crime. Telegram claims to be more secure than mass market messengers such as WhatsApp and Line. It allows, among other things, anonymous forwards, which means your forwarded messages will no longer lead back to your account. You can also unsend messages and delete entire chats from not just your own phone, but also the other person’s. In addition, it allows you to set up usernames through which you can talk to Telegram users without revealing your phone number. These two features differentiate it from WhatsApp.

Telegram’s secret chat option

Telegram’s secret chats feature uses end-to-end encryption, which means it leaves no trace on servers, it supports self-destructing messages, and it doesn’t allow forwarding. Voice calls are end-to-end encrypted as well. It even allows bots to be set up for specific tasks. Due to its rich feature set and rapid adoption, Telegram has become a sought after tool on the fraud scene. According to Telegram’s website, the app allows users to create private groups containing up to 200,000 members as well as public channels that can be accessed by anyone who has the app.

The double edged sword of Telegram — The double-edged sword of Telegram

Telegram reported in April 2020 that it was logging 1.5 million new users daily. It added that it was the most-downloaded social app in 20 markets globally. The platform has been widely adopted globally and is available in 13 languages.

How threats actors are exploiting Telegram

Until recently, fraudsters mainly utilized Telegram groups and channels to organize their communities. Groups can be best described as chat rooms in which all members can read, comment, and post. This is where fraudsters advertise, connect, and share knowledge and compromised information, akin to dark web forums. Channels, on the other hand, are groups in which only the administrator is authorized to post and regular members have access to view, similar to blogs. Fraudsters mainly use Telegram channels to advertise fraud services and products in bulk. In this way Telegram serves as a ‘Dark Web lite’ for shady businesses.

Examples of a Telegram channel being used as ‘Dark web lite”

The discovery of an exploit is not in itself illegal. Indeed, it is often rewarded by software companies or related businesses that may be affected. But if an exploit is sold, knowing that it is going to be used to commit a crime, then there is a possibility of being charged as an accomplice. The legal ambiguities have generated another grey economy in the trading of exploits. Several sites on social media platforms have been found to be openly vending exploits, including accounts such as Injector exploits database, Exploit Packs.

Telegram channels selling exploits and — Telegram channels selling exploits

Unprotected databases are one of the primary reasons for the rise in the exposed user records. Reports indicate that the data posted by hackers contain authentic databases that could lead to serious concerns for affected individuals and organizations. After the disclosed data breach, potential threat actors could discuss over the telegram channels and hacking forums. An attacker can further use the data to gather sensitive information and facilitate further attacks.

Threat actors trading information and databases via Telegram

Around 30-40% of the social media sites inspected offered some form of hacking service. Very often there was an emphasis on ‘ethical’ hacking services, though there were no obvious ways to corroborate this. These groups offer tools for hacking websites, hackers for hire, and hacking tutorials. Cybercriminals in fact offer everything necessary to arrange a fraud or to conduct a personal attack. The offers are usually very specific and include malicious code and software that can help get access to personal accounts.

Hacking services being advertised and solicited on Telegram

Discord is now where the world hangs out

Discord is a real-time messaging platform that bills itself as an “all-in-one voice and text chat for gamers,” due to its slick interface, ease of use, and extensive features. The platform has made it easy to communicate with friends and create and sustain communities via text, voice, and video.

The app allows users to set up their own servers where they can chat with their friends or with others who share their interests. Discord was originally created for gamers to collaborate and communicate, but has now been widely adopted by other groups and communities ranging from local hiking clubs to art communities and study groups.

The rising popularity of Discord as a communication channel

Discord has garnered 100 million active users per month, 13.5 million active servers per week, and 4 billion servers with people talking for upwards of 4 hours per day. Discord is now where the world talks, hangs out, and builds relationships with their communities and friends. There are servers set up to function as online book clubs, fan groups for television shows or podcasts, and science discussions, to name a few. All this sounds harmless, but does Discord have a dark side? Yes, there are servers that promote illegal activities using the platform.

A convenient way to create and sustain communities and friendships

How cybercriminals are leveraging Discord

Being an encrypted service, Discord hosts numerous chat channels that promote illicit practices. Besides the obvious gaming chats, Discord is exploited to carry out other nefarious activities, like selling credit and loyalty cards, drugs, hacker resources, and doxing services. Much of the popularity has to do with the secure, encrypted, peer-to-peer communications available on the platform, allowing criminals to transact openly while avoiding scrutiny from law enforcement.

Is Discord the new dark web?

Illicit markets on Discord work much like “conventional” Dark Web markets on TOR. First, a buyer must locate a seller, join their network, and pay in bitcoin. One of the most popular goods on Discord marketplaces are credit and loyalty points. Some of these markets, having been kicked off TOR by law enforcement, have relocated their services to Discord.

Stolen credit card data, when sold on Discord or across other dark web sites, often include other identifying information such as names, email addresses, phone numbers, physical addresses, and passwords. These cards can be used to make purchases online and offline, or to buy untraceable gift cards. Loyalty points, another very popular item on Discord, can be purchased for just a few dollars (paid in bitcoin) and these can be exchanged for cash, or for goods and gift cards.

Besides the purchase of credit cards and loyalty points, some powerful hacking tools have found their way to Discord, making it possible for buyers to compromise accounts directly. One prominent example is OpenBullet, released on Microsoft’s GitHub code platform. Originally intended as a testing tool for security professionals, it was modified by hackers and spread quickly. It is easy to use, configure, and deploy, and helps the server owner set up DDOS services, carding methods, and malware sales.

It is easy to monitor paste websites like Pastebin because we know the structure of websites; what type of data is pasted, etc. But monitoring discussions on Discord, while not as simple, is critical for organizations. And time is of the essence when it comes to detecting and alerting organizations to information being exchanged or discussed, that pertains to their data and assets.

Cybercriminals using Discord to communicate and exchange data and services

Cybercriminals also tend to use these platforms to share news, exchange vulnerability and exploit information, and cite research work from within the cybersecurity community.

The need for continuous monitoring

This is just the beginning of cybercriminals using instant messaging platforms to further their businesses. And with the rising popularity of encrypted messaging apps, we can expect illegal activities to flourish on these platforms. Given the quick turnaround time on IM platforms, as opposed to forums where criminals first post their needs/ services and then have to wait for a reply, it is only a matter of time before cybercriminals shift their transactions to these platforms. And tools like chatbots allow for automated replies and advertising, helping threat actors achieve more in less time.

Which is why real time monitoring of dark web markets, Telegram channels, and Discord servers is no longer a luxury but a basic requirement for organizations to secure their data and assets. And this is where CloudSEK’s proprietary digital risk monitoring platform XVigil can help you stay ahead of cybercriminals and their increasingly sophisticated schemes. XVigil scours the internet, including surface websites, dark web marketplaces, and messaging platforms like Telegram and Discord. It detects malicious mentions and exchanges pertaining to your organization’s digital assets and provides you real-time alerts. Thus giving you enough time to take proactive measures to prevent costly breaches and attacks.

Posted on November 6, 2020 | by Piyush Goyal Engineering

MongoDB Sharding 101: Creating a Sharded Cluster

What is Sharding?

Sharding is a process in MongoDB, that divides large data sets into smaller volumes of data and distributes them across multiple MongoDB instances. When the data within MongoDB is huge, the queries run against such large data sets can lead to high CPU usage. MongoDB leverages the feature of Sharding to tackle this. During this process, the large database is split into shards (subsets of data) that logically function as a collection. Shards are implemented through clusters that consist of:

Shard(s): They are MongoDB instances that host subsets of data. In production, all shards need to be a part of replica sets that maintain the same data sets. Replica sets provide redundancy, high availability and are the basis for all production deployments.
Config server: It is a mongod instance that holds metadata about various mongoDB instances. A config server runs small mongod processes. In a production environment, usually, there are 3 config servers, wherein each server consists of a copy of metadata, that are kept in sync. And as long as one config server is alive, the cluster will remain active.
Query router: This is a mongoDB instance that is responsible for re-directing the client’s commands to the right servers. It doesn’t have a persistent state. It acts as the interface between the client application and the relevant shard. The query router gathers information that is needed to answer a query, reduces false positives and false negatives in the data, and routes the query to the most accurate information sources.

In a two-part series about sharding in MongoDB, we explain:

How to create a sharded cluster
How to select a shard key

In the first part we provide a tutorial on how to create a sharded cluster, explaining all the basics of sharding.

Sharded Cluster Tutorial

To create a cluster, we need to be clear about:

The number of shards required initially
Replication factor and other parameters for the replica set

mongod and mongos are two different processes in MongoDB. While mongod is the primary daemon that handles data requests, data access and performs background management operations, mongos is a shard utility that processes and routes user queries and determines the location of data in the sharded cluster. The shards and config servers run the mongod process, whereas the query router server runs the mongos process.

For our illustration, let’s take 4 shards (namely a, b, c, and d) and a replication factor of 3. So in total there will be 12 shard servers (mongod processes). We’ll also run 4 query routers (mongos processes) and 3 config servers (small mongod processes) as one would in a production environment; running all of this in one server simulating a cluster.

In actuality, these shards along with their replica sets are run on 12 different machines. However, config servers are lightweight processes and with only 4 shards. So, the load will be considerably low. This means you can run config servers on any one of the shard servers. The mongos processes can be run either on a shard server or directly on a client application machine.

The benefit of running mongos on a client application machine is that it runs on a local interface without having to go outside of the network. However, you should remember to enable the right security options. On the other hand, if you run it on the shard server or any other server, the client will not have to go through the process of setting the cluster up.

Data directories

The first step in the process is to create data directories for our shard servers, config servers, and logs.

Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p data/a0
Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p data/a1
Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p data/a2

Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p data/b0
Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p data/b1
Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p data/b2

Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p data/c0
Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p data/c1
Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p data/c2

Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p data/d0
Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p data/d1
Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p data/d2

Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p config/cfg0
Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p config/cfg1
Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir -p config/cfg2

Tue Oct 27 piyushgoyal mongo_tutorial $ mkdir logs

Config server and Shard server

Use the mongod command, as shown below, to start the config server. We have, thus, passed the configsvr parameter that declares the config server for that cluster.

Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --configsvr --replSet cfg --dbpath config/cfg0 --port 26050 --fork --logpath logs/log.cfg0 --logappend
Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --configsvr --replSet cfg --dbpath config/cfg1 --port 26051 --fork --logpath logs/log.cfg1 --logappend
Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --configsvr --replSet cfg --dbpath config/cfg2 --port 26052 --fork --logpath logs/log.cfg2 --logappend

Now let’s get on with our shard servers. Here, we run the mongod command to pass the shardsvr parameter which in turn declares that it’s part of a sharded cluster. We then rename the replica set with the name of the shard.

Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --shardsvr --replSet a --dbpath data/a0 --port 27000 --fork --logpath logs/log.a0 --logappend

Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --shardsvr --replSet a --dbpath data/a1 --port 27001 --fork --logpath logs/log.a1 --logappend

Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --shardsvr --replSet a --dbpath data/a2 --port 27002 --fork --logpath logs/log.a2 --logappend


Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --shardsvr --replSet b --dbpath data/b0 --port 27100 --fork --logpath logs/log.b0 --logappend

Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --shardsvr --replSet b --dbpath data/b1 --port 27101 --fork --logpath logs/log.b1 --logappend

Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --shardsvr --replSet b --dbpath data/b2 --port 27102 --fork --logpath logs/log.b2 --logappend


Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --shardsvr --replSet c --dbpath data/c0 --port 27200 --fork --logpath logs/log.b0 --logappend

Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --shardsvr --replSet c --dbpath data/c1 --port 27201 --fork --logpath logs/log.b1 --logappend

Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --shardsvr --replSet c --dbpath data/c2 --port 27202 --fork --logpath logs/log.b2 --logappend


Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --shardsvr --replSet d --dbpath data/d0 --port 27300 --fork --logpath logs/log.c0 --logappend

Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --shardsvr --replSet d --dbpath data/d1 --port 27301 --fork --logpath logs/log.c1 --logappend

Tue Oct 27 piyushgoyal mongo_tutorial $ mongod --shardsvr --replSet d --dbpath data/d2 --port 27302 --fork --logpath logs/log.c2 --logappend

Mongos Server

In the next step, we run the mongos command to start mongos processes. In order to tell where the config servers are, we pass config servers IP using the –configdb parameter. We’ll run a single mongos process on a standard mongo TCP port to adhere to best practices.

Tue Oct 27 piyushgoyal mongo_tutorial $ mongos --configdb cfg/127.0.0.1:26050,127.0.0.1:26051,127.0.0.1:26052 --logpath logs/log.mongos0 --port 27017 --bind_ip 0.0.0.0 --fork --logappend
Tue Oct 27 piyushgoyal mongo_tutorial $ mongos --configdb cfg/127.0.0.1:26050,127.0.0.1:26051,127.0.0.1:26052 --logpath logs/log.mongos1 --port 26061 --bind_ip 0.0.0.0 --fork --logappend
Tue Oct 27 piyushgoyal mongo_tutorial $ mongos --configdb cfg/127.0.0.1:26050,127.0.0.1:26051,127.0.0.1:26052 --logpath logs/log.mongos2 --port 26062 --bind_ip 0.0.0.0 --fork --logappend
Tue Oct 27 piyushgoyal mongo_tutorial $ mongos --configdb cfg/127.0.0.1:26050,127.0.0.1:26051,127.0.0.1:26052 --logpath logs/log.mongos3 --port 26063 --bind_ip 0.0.0.0 --fork --logappend

Note that we never run the config server and the sharded server on the same standard mongo TCP port.

Now, check if all the processes are running smoothly.

Tue Oct 27 piyushgoyal mongo_tutorial $ ps -A | grep mongo

26745 ??         0:00.95 mongod --configsvr --replSet cfg --dbpath config/cfg0 --port 26050 --fork --logpath logs/log.cfg0 --logappend

26748 ??         0:00.90 mongod --configsvr --replSet cfg --dbpath config/cfg1 --port 26051 --fork --logpath logs/log.cfg1 --logappend

26754 ??         0:00.90 mongod --configsvr --replSet cfg --dbpath config/cfg2 --port 26052 --fork --logpath logs/log.cfg2 --logappend

26757 ??         0:00.77 mongod --shardsvr --replSet a --dbpath data/a0 --port 27000 --fork --logpath logs/log.a0 --logappend

26760 ??         0:00.77 mongod --shardsvr --replSet a --dbpath data/a1 --port 27001 --fork --logpath logs/log.a1 --logappend

26763 ??         0:00.76 mongod --shardsvr --replSet a --dbpath data/a2 --port 27002 --fork --logpath logs/log.a2 --logappend

26766 ??         0:00.76 mongod --shardsvr --replSet b --dbpath data/b0 --port 27100 --fork --logpath logs/log.b0 --logappend

26769 ??         0:00.77 mongod --shardsvr --replSet b --dbpath data/b1 --port 27101 --fork --logpath logs/log.b1 --logappend

26772 ??         0:00.75 mongod --shardsvr --replSet b --dbpath data/b2 --port 27102 --fork --logpath logs/log.b2 --logappend

26775 ??         0:00.73 mongod --shardsvr --replSet c --dbpath data/c0 --port 27200 --fork --logpath logs/log.b0 --logappend

26784 ??         0:00.75 mongod --shardsvr --replSet c --dbpath data/c1 --port 27201 --fork --logpath logs/log.b1 --logappend

26791 ??         0:00.74 mongod --shardsvr --replSet c --dbpath data/c2 --port 27202 --fork --logpath logs/log.b2 --logappend

26794 ??         0:00.77 mongod --shardsvr --replSet d --dbpath data/d0 --port 27300 --fork --logpath logs/log.c0 --logappend

26797 ??         0:00.75 mongod --shardsvr --replSet d --dbpath data/d1 --port 27301 --fork --logpath logs/log.c1 --logappend

26800 ??         0:00.71 mongod --shardsvr --replSet d --dbpath data/d2 --port 27302 --fork --logpath logs/log.c2 --logappend

26803 ??         0:00.00 mongos --configdb cfg/127.0.0.1:26050,127.0.0.1:26051,127.0.0.1:26052 --logpath logs/log.mongos0 --port 27017 --bind_ip 0.0.0.0 --fork --logappend

26804 ??         0:00.24 mongos --configdb cfg/127.0.0.1:26050,127.0.0.1:26051,127.0.0.1:26052 --logpath logs/log.mongos0 --port 27017 --bind_ip 0.0.0.0 --fork --logappend

76826 ??         8:58.30 /usr/local/opt/mongodb-community/bin/mongod --config /usr/local/etc/mongod.conf

26817 ttys009    0:00.00 grep mongo

26801 ttys016    0:00.01 mongos --configdb cfg/127.0.0.1:26050,127.0.0.1:26051,127.0.0.1:26052 --logpath logs/log.mongos0 --port 27017 --bind_ip 0.0.0.0 --fork --logappend

Status and Initiation of Replica Sets

Connect to one of the shard servers. For instance, let’s connect to 0 shard

Tue Oct 27 piyushgoyal mongo_tutorial $ mongo --port 27000

Run the following command to check the status of the replica set

> rs.status()

{

"operationTime" : Timestamp(0, 0),

"ok" : 0,

"errmsg" : "no replset config has been received",

"code" : 94,

"codeName" : "NotYetInitialized",

"$clusterTime" : {

"clusterTime" : Timestamp(0, 0),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}

What follows is a dialogue box that says “run rs.initiate() is not yet done for the set”. To initiate the replica set, run the following command:

> rs.initiate()

{

"info2" : "no configuration specified. Using a default configuration for the set",

"me" : "localhost:27000",

"ok" : 1,

"$clusterTime" : {

"clusterTime" : Timestamp(1604637446, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

},

"operationTime" : Timestamp(1604637446, 1)

}

If you run rs.status() once again, you’ll see that a single member has been added. Now, let’s add others as well to this replica set.

a:PRIMARY> rs.add("127.0.0.1:27001")

{

"ok" : 1,

"$clusterTime" : {

"clusterTime" : Timestamp(1604637486, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

},

"operationTime" : Timestamp(1604637486, 1)

}
a:PRIMARY> rs.add("127.0.0.1:27002")

{

"ok" : 1,

"$clusterTime" : {

"clusterTime" : Timestamp(1604637491, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

},

"operationTime" : Timestamp(1604637491, 1)
}

Run rs.status() to check the status of the initiation. Once it has been initiated, repeat the same process for other shards and the config server.

Add Shards

Connect to the mongos process so as to add shards to the cluster.

Tue Oct 27 piyushgoyal mongo_tutorial $ mongo --port 27017

Run the command below to add shards:

mongos> sh.addShard("a/127.0.0.1:27000")

{

"shardAdded" : "a",

"ok" : 1,

"operationTime" : Timestamp(1604637907, 8),

"$clusterTime" : {

"clusterTime" : Timestamp(1604637907, 8),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}

mongos> sh.addShard("b/127.0.0.1:27100")

{

"shardAdded" : "b",

"ok" : 1,

"operationTime" : Timestamp(1604638045, 6),

"$clusterTime" : {

"clusterTime" : Timestamp(1604638045, 6),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}

mongos> sh.addShard("c/127.0.0.1:27200")

{

"shardAdded" : "c",

"ok" : 1,

"operationTime" : Timestamp(1604638065, 4),

"$clusterTime" : {

"clusterTime" : Timestamp(1604638065, 4),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}

mongos> sh.addShard("d/127.0.0.1:27300")

{

"shardAdded" : "d",

"ok" : 1,

"operationTime" : Timestamp(1604638086, 6),

"$clusterTime" : {

"clusterTime" : Timestamp(1604638086, 6),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}

Note: When you run sh.addShard(‘a/127.0.0.1:27000’), if you get an output as shown below, try running sh.addShard(‘a/127.0.0.1:27001’) or sh.addShard(‘a/127.0.0.1:27002’).

{
 "ok" : 0,
 "errmsg" : "in seed list a/127.0.0.1:27000, host 127.0.0.1:27000 does not belong to replica set a; found { hosts: [ \"localhost:27000\", \"127.0.0.1:27001\", \"127.0.0.1:27002\" ], setName: \"a\", setVersion: 3, ismaster: true, secondary: false, primary: \"localhost:27000\", me: \"localhost:27000\", electionId: ObjectId('7fffffff0000000000000001'), lastWrite: { opTime: { ts: Timestamp(1604637886, 1), t: 1 }, lastWriteDate: new Date(1604637886000), majorityOpTime: { ts: Timestamp(1604637886, 1), t: 1 }, majorityWriteDate: new Date(1604637886000) }, maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, maxWriteBatchSize: 100000, localTime: new Date(1604637894239), logicalSessionTimeoutMinutes: 30, connectionId: 21, minWireVersion: 0, maxWireVersion: 8, readOnly: false, compression: [ \"snappy\", \"zstd\", \"zlib\" ], ok: 1.0, $clusterTime: { clusterTime: Timestamp(1604637886, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, operationTime: Timestamp(1604637886, 1) }",
 "code" : 96,
 "codeName" : "OperationFailed",
 "operationTime" : Timestamp(1604637888, 1),
 "$clusterTime" : {
 "clusterTime" : Timestamp(1604637888, 1),
 "signature" : {
 "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
 "keyId" : NumberLong(0)
 }
 }
}

Sharding a collection
Run sh.status() to check if the shards have been added.
Now that we have set up the cluster, let’s test it.

When you shard a collection, you should choose a specific shard key. Without which sharding is impossible. It helps to distribute documents among members of the cluster.
The data stored by the key is called a chunk. The chunk size is only allowed to be between 1 and 1024 megabytes.
For example, let’s assume a collection of the following documents:

{
    "_id": ObjectId("5f97d97eb7a0a940f157ebc8"),
    "x": 1
}

And we use key x to shard the collection. The data is distributed as shown below:

x_low x_high shard

0    1000    S2    ----> chunk

1000  2000    S0    ----> chunk

2000  3000    S1    ----> chunk

The collection’s documents are distributed based on the shard key assigned. Documents of a particular shard key belong to the same shard.

The process of sharding can be divided into two:

Range-based sharding
Hashed sharding

To keep this tutorial simple, we prefer range-based sharding. Range-based sharding is helpful for queries involving a range. Against the backdrop of these processes, the system also carries out these two key operations:

Split
Migration (it is been handled by cluster balancer)

For example, if a chunk size exceeds the prescribed size, the system, on its own, will attempt to find out the appropriate median key to divide the chunk into 2 or more parts. This is known as split, which is an inexpensive, metadata change. However, migration is responsible for maintaining the balance, for instance, moving a chunk from one shard to another. This is an expensive operation which usually involves transferring data worth ~1024MB.
You can still read and write to the database while the migration is in process. There’s a liveliness property to these migrations which keeps the database alive for operations, so a big lock doesn’t occur.
For our example, we’re creating a database named tutorial with collection foo and are going to insert some documents.

Tue Oct 27 piyushgoyal mongo_tutorial $ mongo --port 27017
mongos> use tutorial
mongos> for(var i=0; i<999999; i++) { db.foo.insert({x:i}) }

To allow sharding for the database, we have to manually enable it. Connect to the mongo instance and run sh.enableSharding(“<dbname>”).

mongos> sh.enableSharding("tutorial")

{

"ok" : 1,

"operationTime" : Timestamp(1604638168, 21),

"$clusterTime" : {

"clusterTime" : Timestamp(1604638168, 21),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}

If you run sh.status() in the database, it returns “partitioned = True” in the tutorial database.

Now we shard our collection using key x. To shard a collection, we have to first create an index on the key and then run sh.shardCollection(“<db_name>.<collection_name>”, <key>).

mongos> db.foo.ensureIndex({x: 1})

{

"raw" : {

"b/127.0.0.1:27101,127.0.0.1:27102,localhost:27100" : {

"createdCollectionAutomatically" : false,

"numIndexesBefore" : 1,

"numIndexesAfter" : 2,

"ok" : 1

}

},

"ok" : 1,

"operationTime" : Timestamp(1604638185, 9),

"$clusterTime" : {

"clusterTime" : Timestamp(1604638185, 9),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}

mongos> sh.shardCollection("tutorial.foo", {x: 1})

{

"collectionsharded" : "tutorial.foo",

"collectionUUID" : UUID("b6506a90-dc0f-48d2-ba22-c15bbc94c0d6"),

"ok" : 1,

"operationTime" : Timestamp(1604638203, 39),

"$clusterTime" : {

"clusterTime" : Timestamp(1604638203, 39),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}

The collection is now sharded. To check the shard distribution, number of chunks (nchunks), run the following command:

mongos> use tutorial
mongos> db.foo.stats()

To know the distribution of chunks across all databases, run the command below:

mongos> sh.status()

Let’s run some queries.

mongos> use tutorial
mongos> db.foo.find({x: 55})

To view the query plan or and understand how the cluster queries through different servers to get the document, append .explain() at the end of the query.

mongos> db.foo.find({x: 55}).explain()

{

"queryPlanner" : {

"mongosPlannerVersion" : 1,

"winningPlan" : {

"stage" : "SINGLE_SHARD",

"shards" : [

{

"shardName" : "b",

"connectionString" : "b/127.0.0.1:27101,127.0.0.1:27102,localhost:27100",

"serverInfo" : {

"host" : "Piyushs-MacBook-Pro.local",

"port" : 27100,

"version" : "4.2.8",

"gitVersion" : "43d25964249164d76d5e04dd6cf38f6111e21f5f"

},

"plannerVersion" : 1,

"namespace" : "tutorial.foo",

"indexFilterSet" : false,

"parsedQuery" : {

"x" : {

"$eq" : 55

}

},

"queryHash" : "716F281A",

"planCacheKey" : "0FA0E5FD",

"winningPlan" : {

"stage" : "FETCH",

"inputStage" : {

"stage" : "SHARDING_FILTER",

"inputStage" : {

"stage" : "IXSCAN",

"keyPattern" : {

"x" : 1

},

"indexName" : "x_1",

"isMultiKey" : false,

"multiKeyPaths" : {

"x" : [ ]

},

"isUnique" : false,

"isSparse" : false,

"isPartial" : false,

"indexVersion" : 2,

"direction" : "forward",

"indexBounds" : {

"x" : [

"[55.0, 55.0]"

]

}

}

}

},

"rejectedPlans" : [ ]

}

]

}

},

"serverInfo" : {

"host" : "Piyushs-MacBook-Pro.local",

"port" : 27017,

"version" : "4.2.8",

"gitVersion" : "43d25964249164d76d5e04dd6cf38f6111e21f5f"

},

"ok" : 1,

"operationTime" : Timestamp(1604638250, 30),

"$clusterTime" : {

"clusterTime" : Timestamp(1604638255, 27),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}

In the second part of this two-part series, we will explore various methods by which we can select a shard key.

Posted on October 28, 2020 | by Palash Engineering

Contemporary Single-Page Applications and Frontend Security

The past decade witnessed a meteoric rise in the number of applications adopting the Single-Page Application (SPA) model. Such applications are designed in such a way that the content on every new page within the application appears on a single web page, without having to load new HTML pages. Single-Page Applications leverage JavaScript capabilities to manipulate Document Object Model (DOM) elements, allowing to update all the content on the same page.

A more traditional web page architecture loads different pages as the user attempts to open a new web page. In this case, each HTML page is usually linked to other pages. Upon each page load, the browser fetches and displays a fresh page.

Whereas, Single-Page Applications are enabled by JavaScript frameworks such as React, Angular, and VueJS. These frameworks help to conceal the complex functions that SPAs perform. They also provide additional benefits such as modular reusable components, state management, etc. Such modern frameworks help SPAs execute web pages effortlessly, as compared to multi-page applications that use Vanilla JavaScript. Using plain JavaScript makes it difficult to keep the UI updated with dynamic state changes.

The emergence of such frameworks causes changes in the security implications it may have on the frontend. Therefore, it is necessary to understand their internal machinery while disassembling client-side code and how modern frameworks change the attack vectors.

Build Process

As is customary, a developer creates a web page by defining the page structure in an HTML file and its styling in a CSS file. Then, it is linked in HTML using tags such as <style> or <link>, by embedding JavaScript code with <script> tags. However, building Single-Page Applications is more complicated than this.

Frameworks such as React and Vuejs provide a virtual DOM, which allows you to steer clear of raw HTML. A virtual DOM, unlike a normal DOM, is a concept in which the representation of the UI is stored in memory, instead of rendering it on the browser with every change. This allows faster changes to DOM. The code is also written in modular JavaScript files which is then processed in the build process.

Here’s a diagram overviewing the build process:

Transpiling

JavaScript is a dialect of ECMAScript and it isn’t a fixed standard; new features are added to ECMAScript with newer standards, every few years. While newer standards are released frequently, no browser JavaScript engine (Chromium V8, Safari Javascript Core, Firefox SpiderMonkey) fully implements all the ECMA specifications, each having certain differences in features they support. Also your code needs to be compatible with older browsers released to support older specifications alone.

Transpiling enables you to write modern code which is then converted to standards and features that are supported everywhere, for e.g. ES6 -> ES5. You might use const, let to define your variables but the transpiler helps to convert it to var, internally.

Transpiling is also used to convert code in JavaScript dialects such as Typescript and Coffeescript to plain JavaScript. Each dialect contains its own transpiler such as tsc for Typescript. Babel is the most prevalent transpiler.

Bundling

A typical modern app contains hundreds of external dependencies when you look at it recursively and it isn’t practical to load each of those separately in a script tag.

Through the process of bundling, you can take each import or require statement, find the source files and bundle them all into one single JavaScript file, applying appropriate scoping.

With the help of this process, all of the code that makes the application work is bundled together; the business logic code as well as the boiler-plate code are bundled together.

Minification/ Obfuscation

The final JavaScript output usually can be very large due to extra spaces or unnecessary, redundant data. Therefore, the final step in the build process is to minify the code. During the process of minification, comments/ whitespaces are removed, identifiers are renamed to shorter variants, and certain routines are simplified. This process then leads to obfuscation, wherein the final code is unreadable and differs highly from the source code.

Bundling and minification is usually done using Webpack.

Single-page application: Original code - easy to read — Original code – easy to read

Single-page application: Transpiled, Bundled, Minified final output - unreadable — Transpiled, Bundled, Minified final output – unreadable

Reverse engineering code

Once the code has been minified in the build process, security researchers will have to de-minify it to study the app. As most of the information such as variable names are lost during the process of minification, there isn’t a straightforward way to do this. However, there are certain tools to aid you in the process such as jsnice.org.

This tool uses machine learning to restore minified variable, function names and infer type information. It also formats the code and adds comments.

After this step, you are still left with a bundled code, but the main logic would be readable.

To debundle it into modules, we need to know how Webpack or any other bundler works.

Debundling

A bundler starts with an entry point file – the root of your application logic – and traces import and require statements. It then builds a dependency graph, module A requires B which requires C and D, and so on.

If you look through the Webpack chunks after they have been passed through jsnice, you’ll find a lot of calls to “__webpack_require__(<number>).”

“ __webpack_require__” is similar to the require JavaScript syntax in functionality, except it contains the logic to load modules from the chunks.

The only way to unbundle a bundle is to construct the abstract syntax tree (AST) manually, as there have been attempts to debundle that aren’t maintained anymore.

You could use these resources to study in depth how a bundle file works and to know the internals of Webpack. In this video, Webpack founder Tobias Koppers shows us how bundling is done manually.

Security Tip

How do these frameworks change attack vectors?

Reduced XSS

React does not render dynamic HTML; it sanitizes content coming from all variables, even if they do not contain dynamic content. Here XSS is all but eradicated unless the developer uses an unsafe function such as dangerouslySetInnerHTML.

In that case, even if you find data reflection you would not be able to insert HTML.

CSP-Bypasses

You could use certain gadgets available in Angular to bypass Content Security Policy (CSP) in certain cases.

<img src=”/” ng-on-error=”$event.srcElement.ownerDocument.defaultView.alert($event.srcElement.ownerDocument.domain)”

Here, if CSP is blocking inline scripts but the page is using Angular, you could use the ng-on-error attribute to get Angular to execute the JavaScript. These types of gadgets are often patched but discovered regularly in Vuejs and Angularjs.

Posted on October 15, 2020 | by Rohan Luthra | Leave a Comment Engineering

Development of a Language-Independent Microservice Architecture

There is a proliferation of modern programming languages now more than ever. Nevertheless, development organizations tend to stick to one or two languages, allowing them to manage their stack properly and to keep it clean.

Programming languages are essential tools that if not chosen appropriately, could have a crippling effect on the business. For example, developers prefer Python over Java to set up machine learning algorithms, because Python has better libraries required for machine learning and artificial intelligence applications.

Every language has a particular set of capabilities and properties. For instance, some languages are faster than others but may take time to write. Some languages consume more resources and yet make development faster. And, there are other languages that fall in between.

Building a language-independent microservice architecture

While it is not quite common, a microservice architecture can be built to suit multiple languages. It provides developers with the freedom to choose the right tool for their task. In the final phase of the software development lifecycle, if Docker can deploy containers irrespective of its language, its architecture can be replicated in the initial stages of building microservices as well.

However, managing multiple languages can be a huge task and a developer should definitely consider the following aspects when they plan to add more languages to their technology stack:

New boiler plate code,
Defining models/data structures again,
Documentation,
The environment it works on,
New framework, etc.

Why RESTful API is not the right choice

REST API requires a new framework for every new language such as Express for Node.js, Flask for Python, or Gin for Go. Also, the developer may have to code their models/ data structures over and over for each service as well. This makes the whole process redundant and can result in several errors.

For example, it is very common to have one database model for multiple services. But, when the database is updated, it has to undergo changes for each service. To build a multilingual microservice, we need a common framework that supports multiple languages and platforms and one that is scalable.

Make way for gRPC and Protocol Buffers

gRPC

gRPC is Google’s high performing, open-source remote procedure calls framework. It’s compatible with multiple languages such as Go, Node.js, Python, Java, C-Sharp, and many more. It uses RPC to provide communication capabilities to microservices. Moreover, it is a better alternative to REST. gRPC is much faster than REST. And this framework uses Protocol Buffers as the interface definition language for serialization and communication instead of JSON/ XML. It works based on server-client communication.

An in-depth analysis of gRPC is given here.

Protocol Buffers

Protocol buffers are a method of serializing data that can be transmitted over wire or be stored in files. In short, Protocol buffer or Protobuf is the same as what JSON is with regards to REST. It is an alternative to JSON/ XML, albeit smaller and faster. Protobuf defines services and other data and is then compiled into multiple languages. The gRPC framework is then used to create a service.

gRPC uses the latest transfer protocol – HTTP/2, which supports bidirectional communication along with the traditional request/ response. In gRPC the server is loosely coupled with the client. In practice, the client opens a long-lived connection with the gRPC server and a new HTTP/2 stream will be opened for each RPC call.

How do we use Protobuf?

We first define our service in Protobuf and compile it to any language. Then, we use the compiled files to create our gRPC framework to create our server and client, as shown in the diagram below.

To explain the process:

Define a service using Protocol Buffers
Compile that service into other languages as per choice
Generate boiler-plate code in each language
Use this to create gRPC server and clients

Advantages of using this architecture:

You can use multiple languages with a single framework.
Single definition across all the services, which is very useful in an organisation.
Any client can communicate with any server irrespective of the language.
gRPC allows two-way communication and uses HTTP/2 which is uber fast.

Creating a microservice with gRPC

Let’s create a simple microservice – HelloWorldService. This is a very basic service that invokes only one method – HelloWorld – which would return the string “hello world”.

These are the 4 steps to follow while creating a microservice:

Define the service (protocol buffer file)
Select your languages
Compile the defined service into selected languages
Create a gRPC server and client, using the compiled files

For this simple service, we are opting 2 languages: Go and Node.js. Since gRPC works on a client-server architecture, we shall use Go on the server (since it is fast and resource efficient) and Node.js for clients (since most of the apps these days are React/ Angular).

**One can also decide to run gRPC servers with REST API too, if they do not want to create clients. They can do this by creating a REST Proxy server. Although it sounds laborious, it is actually pretty simple and will be dealt with in the ‘Compiling’ section that will follow.

Step 1: Defining a service

Messages and services are defined using Protobuf in a proto file (.proto file).

The syntax is quite simple; the messages are defined and are used as Request and Response for the RPC calls, for each service. Here is a language guide for Protocol Buffers.

We can create a new folder for each service under the name of that particular service. And, for ease of accessibility, we can store all these services under the folder “services.”

The service we are defining here is “helloworld.” And each folder should contain 2 files, namely:

service.proto – contains definition
.protolangs – contains the languages that this service should be generated in.

Now, let’s define our service:

As shown above, we have defined an empty Request, a string Response and a HelloWorldService with a single RPC call HelloWorld. This call accepts requests and returns responses. You can see the full repo here.

Step 2: Selecting languages

Once the service has been defined, we will have to choose the languages to compile the services into. This choice is made based on service requirements and usage, and also the developer’s comfort. The different languages that one can choose are Go, Node.js, Python, Java, C, C#, Ruby, Scala, PHP, etc. As mentioned earlier, in this example, we’ll be using Go and Node.js. We’ll then add these languages to the .protolangs file, mentioning each language in a new line.

Step 3: Compiling

Compiling is the most interesting part of this entire process. In step 3, we’ll compile the .proto file to the selected languages, Go and Node.js.

The Protocol Buffer comes with a command line tool called “protoc”, which compiles the service definition for use. But for each language we’ll have to download and install plugins. This can be achieved by using a dockerfile.

Namely is a docker file which includes all of this and is available to all. It has the “proto compiler” with support for all languages and additional features like documentation, validator, and even a REST Proxy server that we talked about in step-1.

For example,


$ docker run -v `pwd`:/defs namely/protoc-all -f myproto.proto -l ruby

It accepts a .proto file and language, both of which we already have. The following is a simple bash script that will loop through our services folder, and pick up the service.proto file to compile to the languages in the .protolangs file.


#!/bin/bash
echo "Starting ... "
set x
 
REPO="`pwd`/repo"
 
function enterDir {
 echo "Entering $1"
 pushd $1 > /dev/null
}
 
function leaveDir {
 echo "Exiting"
 popd > /dev/null
}
 
function complieProto {
   for dir in */; do
       if [ -f $dir/.protolangs ]; then
           while read lang; do
               target=${dir%/*}
               mkdir -p $REPO/$lang
               rm -rf $REPO/$lang/$target
               mkdir -p $REPO/$lang/$target
               mkdir -p $REPO/$lang/$target/doc
 
               echo "  Compiling for $lang"
               docker run -v `pwd`:/defs namely/protoc-all -f $target/service.proto -l $lang --with-docs --lint $([ $lang == 'node' ] && echo "--with-typescript" || echo "--with-validator")
 
               cp -R gen/pb-$lang/$target/* $REPO/$lang/$target
               cp -R gen/pb-$lang/doc/* $REPO/$lang/$target/doc
               sudo rm -rf gen
 
           done < $dir/.protolangs
       fi
   done
}
 
function complie {
 echo "Starting the Build"
 mkdir -p $REPO
 for dir in services/; do
   enterDir $dir
   complieProto $dir
   leaveDir
 done
}
 
complie

The scripts run a loop for each folder in /services.
It then picks up the .protolangs file and loops them again with each of the languages written in the folder.
It then compiles service.proto with the language.
The docker generates the files in gen/pb-{language} folder.
We simply copy the content to repos/{language}/{servicename} folder.

We, then, run the script:

$ chmod +x generate.sh

$ ./genetare.sh

The file generated appears in the /repos folder.

Tip: You can host these definitions in a repository, and use the script generated in a CI/CD Pipeline to automate this process.

The successfully generated service_pb for both Node.js and Go, along with some docs and validators, constitutes the boiler-plate code for our server and clients that we are about to create.

**As discussed earlier, if you don’t want to use the client and want REST-JSON APIs instead, you can create a REST Proxy by adding a single tag in the namely/protoc-all dockerfile i.e. — with-gateway. For this we’ll have to add api paths in our protofiles. Have a look at this for further information. Now, run this gateway and the REST Proxy server will be ready to serve the gRPC server.

Tip: You can also host this protorepo on github as a repository. You can have a single repo for all the definitions in your organisation, in the same way as google does.

Step 4: gRPC Server and Client

Now that we have service_pb code for Go and Node.js, we can use it to create a server and a client. For each language the code will be slightly different, because of the differences in the languages. But the concept will remain the same.

For servers: We’ll have to implement RPC functions.

For clients: We’ll have to call RPC functions.

You can see the gRPC code for all languages here. With only a few lines of code we can create a server and with fewer lines of code we can create a client.

Server (Go):


package main
 
import (
   "context"
   "fmt"
   "log"
   "net"
 
   helloworld "github.com/rohan-luthra/service-helloworld-go/helloworld"
   "google.golang.org/grpc"
)
 
type server struct {
}
 
func (*server) HelloWorld(ctx context.Context, request *helloworld.Request) (*helloworld.Response, error) {
   response := &helloworld.Response{
       Messsage: "hello world from go grpc",
   }
   return response, nil
}
 
func main() {
   address := "0.0.0.0:50051"
   lis, err := net.Listen("tcp", address)
   if err != nil {
       log.Fatalf("Error %v", err)
   }
   fmt.Printf("Server is listening on %v ...", address)
 
   s := grpc.NewServer()
   helloworld.RegisterHelloWorldServiceServer(s, &server{})
 
   s.Serve(lis)
}

As you can see, we have imported the service.pb.go that was generated by our shell script. Then we implemented the function HelloWorld – which returns the Response “hello world from go grpc.” Thus creating a gRPC server.

Client (Node.js):


var helloword = require('./helloworld/service_pb');
var services = require('./helloworld/service_grpc_pb');
 
var grpc = require('grpc');
 
function main() {
 var client = new services.HelloWorldServiceClient('localhost:50051',
                                         grpc.credentials.createInsecure());
 var request = new helloword.Request();
 var user;
 if (process.argv.length >= 3) {
   user = process.argv[2];
 } else {
   user = 'world';
 }
 client.helloWorld(request, function(err, response) {
     if(err) console.log('Node Client Error:', err)
     else console.log('Node Client Message:', response.getMesssage());
 });
}
 main();

We have imported the service_pb.js file which was generated by our shell script. Add the gRPC server address and call the HelloWorld function and output response to console.

Test the code

Run the server and make sure that the code works.

Now that our server is running, let’s make a call from our Node.js client:

*Ignore the warning.

When we receive the console output “hello world from go grpc,” we can conclude that everything worked out as expected.

Thus, we have successfully created a gRPC server and client with a single proto definition and a shell script. This was a simple example of an RPC call returning a “hello world” text. But you can do almost anything with it. For example, you can create a CRUD microservice that performs add, get, update RPC calls, or an operation of your choice. You just have to define it once and run the shell script. You can also call one service from another, by creating clients wherever you want or in any language you want. This is an example of a perfect microservice architecture.

Summary

Hope this blog helped you build a microservice architecture with just Protocol Buffers, gRPC, a docker file, and a shell script. This architecture is language-independent which means that it suits multiple programming languages. Moreover, it is almost 7 times faster than the traditional REST server, with additional benefits of documentation and validations.

Not only is the architecture language-independent, this allows you to manage all the data structures in your organisation under a single repository, which can be a game changer. Protocol buffers also support import, inheritance, tags, etc.

You just have to follow these steps when you create a new microservice:

Define your service
Select languages
Compile i.e. run the shell script (automated – can be done on CI/CD pipeline)
Finally, code your servers and clients.
Deploy however you want (Suggestion: use docker containers)

Now you are equipped to create multiple microservices, enable communication between microservices, or just use them as it is and also add a REST Proxy layer to build APIs. Remember that all services use a single framework irrespective of their programming language.

You can find the whole code repo here.

Bonus:

You can also publish the generated proto files of a service as packages for each language, for e.g. Node.js – npm, Python – pip, golang – GitHub, Java – Maven and so on. Run “npm install helloworld-proto” to call the .pb files. And if someone updates the Proto definition you just have to run “npm update helloworld-proto”.

Posted on October 7, 2020 | by Mayank Satnalika | Leave a Comment Engineering

How does CloudSEK’s XVigil detect rogue, fake applications

Brand monitoring is one of the premium services offered by CloudSEK’s flagship digital risk monitoring platform – XVigil. This functionality is comprehensive of a wide range of use cases such as:

Fake domain monitoring
Rogue/ fake application detection
VIP monitoring

Threat actors deploy fake or rogue apps that masquerade as the official application, by infringing on our client’s trademark and copyrighted material. Upon seeing our client’s familiar trademark, their customers are tricked into installing such apps on their devices, thereby running a malicious code that allows threat actors to exfiltrate data. Just this year, XVigil reported and alerted our clients regarding over 2.4 lakh fake apps from various third party app stores.

Classification through similarity scoring

Classification of such threats forms a major part of XVigil’s threat monitoring framework. The platform identifies and classifies the app as fake or rogue based on whether it impersonates our client’s apps, or if the uploaded APK files are different from our client’s official APKs.

For any machine learning problem, you need training data, where you expect your test data to be similar to the training data. In this case, however, the data is different for each client, and you would need a separate model for every new client.

We approach the problem as a similarity score problem. We compare the suspected app with all the official apps of the client and see how similar the suspected app is to the official app in terms of the app-title, description, screenshots provided, logos, etc. Here, a greater concern arises when we don’t have all the information related to the client, to compare it with the suspicious app.

CloudSEK’s Knowledge Extraction Module

To gather more information about the client, we built a knowledge extraction module. When a new client signs up, this module is triggered and it tries to collect all the information it can about the client. The knowledge extraction module was built as a generic module that tries to obtain every piece of information it can about the client. The client’s name and their primary domain are the only inputs required for the process.

With these details, the knowledge extraction module can identify the industry that our client operates in, their various products and services, competitors of the client, the main tech-stack/ technologies the client works with, their official apps, and so on. These details are sourced from Google Play store, or by crawling and parsing the client’s website, various job listings posted by the clients themselves, etc. The gathered information is then passed through custom Named Entity Recognition (NER) models or static rules on them to get the client details in a structured manner.

When we monitor for malicious applications, we run a text/ image similarity model on the gathered information (client’s logo, official app, competitors, etc.) and the information present in the suspicious app. For example, text similarity checks how contextually similar the client’s app description is to that of the suspicious app. Another module tries to find if the client’s logos appear on the screenshots provided within the malicious app. The different scores are then summarised to recommend an ensemble score.

Finally…

If the similarity score is greater than a certain threshold, we can safely say that the app resembles our client’s brand/ app. Now we need to check if the APK uploaded is a modified APK or the original client’s APK itself. The only challenge here is to maintain the APKs released by the client, for any and all versions of their apps, for all devices/ regions. To work around this issue, we compare the suspicious app’s certificate with the certificates present in the official app of the client present on Google Play store. This removes all the apps’ APKs from the official app developers of the client. This leaves us with malicious apps, which is then reported to our clients.

Posted on July 15, 2020 | by Sparsh Kulshrestha | Leave a Comment Engineering

Quickstart Shodan: What is it and how does it work

Rated the best search engine for hackers, Shodan was referred to as the scariest search engine on the internet, back in 2013. While Shodan is similar to Google, in that they are both search engines that use crawlers, it crawls the entire internet to map and index internet-connected IoT devices. To put this into perspective, Google’s crawlers only index a part of the internet that is publicly accessible, the World Wide Web. And although Shodan is infamous for being instrumental in blackhat hacking, in the right hands, this search engine is effective in the process of vulnerability assessment and pentesting IoT devices.

In this article we delve into Shodan and its working, and discuss the features that make this search engine useful for pentesters and bug bounty hunters.

What is Shodan?

As mentioned earlier, Shodan (Sentient Hyper-Optimized Data Access Network) is a search engine for devices connected to the internet. Designed and developed by Web Developer John Matherly, this search engine crawls the entire internet, parses the service banners of IoT devices, and indexes them for future searches. It maps and reports any device, such as webcams, smart devices, medical devices, traffic lights, C2 systems of devices, etc., that is connected to the internet and is not secure.

The service banners of IoT devices contain metadata relevant to the devices, such as:

Geo-location
Make and model
IP address
Default username and password
Software version

How does Shodan work?

Upon scanning the entire internet, the search engine sends queries to connected IoT devices for publicly available information related to them. The servers of such devices return their service banners to the user. Shodan also supports customized queries using filters like city, country, hostname, OS, etc. to find out the corresponding details.

The Basics

Similar to other search engines, Shodan also utilizes a search box, to which users can enter search terms that adhere to its search query syntax. Moreover, search results can be narrowed down to be as accurate as possible, by making use of quotation marks and other such operators.

For instance, boolean operators + or — can be used to include or exclude terms in the query.

Shodan Filters

This search engine only searches for the data property on the IoT devices’ banners. Hence, it employs search filters for refined results. Here are some basic search filters you can use:

city: find devices in a particular city
country: find devices in a particular country
geo: you can pass it coordinates
hostname: find values that match the hostname
net: search based on an IP or /x CIDR
os: search based on operating system
port: find particular ports that are open
before/after: find results within a timeframe

Search examples:

org:”Amazon” ssl:”target”

ssl:”target”

html:”Dashboard Jenkins”

http.component:”jenkins”

http.title:”302 Found”

http.component:”java”

ssl.cert.subject.cn:”target”

hostname:”target”

http.favicon.hash:-335242539

product:elastic port:9200

Continuous Monitoring

This IoT search engine can be used to monitor your networks or products continuously, to help you stay informed of any threats to them. For this, you can make use of the Shodan Command-line Interface (CLI).

Use the following command to set up an alert for notifications related to your IP range:

shodan alert create “Bug-Bounty-Target” 198.172.0.0/24

The following command allows you to create a trigger to send the alert:

shodan alert triggers

When Shodan detects a malware or a new CVE against your product, get a notification using this:

shodan alert enable {ALERTID} malware

Automation

Every new trick boils down to automation these days. Fortunately, with this search engine you can automate various tasks that are within its range of activities. There are the three main ways to automate this search engine:

Shodan-API
Shodan Python Module
Shodan CLI

Check out this handy tool that makes the entire process easier and hassle-free: m4ll0k/Shodanfy.py

Conclusion

Although it seems quite perverse, security professionals, researchers, and even government agencies trust Shodan to alert them on unmanaged IoT devices that may present vulnerabilities. Such devices could potentially expose critical data belonging to an enterprise, organization or individual, to attacks. This search engine helps prevent this. By following the methods prescribed above, you can also explore Shodan and monitor and protect your network or product from exploitation.

Posted on July 8, 2020 | by Prince Kumar | Leave a Comment Artificial Intelligence

With Herbie, humanoid robots are finally here

The concept of anthropomorphic robots has always captured the human imagination, starting with playwright Karel Čapek, who introduced the word “Robot” in the 1920s, to Isaac Asimov’s incredible science fiction. But, in reality, robots have been relegated to doing menial tasks such as lifting things or executing repetitive actions, as is the case with industrial robots.

However, the latest developments in Artificial intelligence (AI) and advanced automation could give us advanced humanoid robots, capable of carrying out multiple functions that not only replace manual labour in manufacturing industries, but also have significant domestic, everyday uses. While AI enables machines to mimic remarkable human cognitive functions, the developments in the field of automation have widened the scope of its usage across different areas such as agriculture, manufacturing, business, analytics, and more.

More significantly, AI has been able to transform elementary robotics to intelligent humanoids capable of interpreting voice commands and gestures, interacting with human beings and its own environment, locomoting, and other computational abilities.

In this article we will discuss one such humanoid known as Herbie and delve into its mechanical structure and sensory faculties.

Herbie

This humanoid model is a type of Socially Assistive Robot (SAR), developed to facilitate the rehabilitation of children with Cerebral Palsy (CP). CP is a congenital disorder caused by abnormal brain development. It affects muscular movement, motor skills, and results in various disabilities. While Herbie is not the first attempt to provide assistance in the field of medicine and therapy, it is a multi-functional humanoid built to address real-time challenges by leveraging different faculties such as vision, speech and hearing, smell, etc.

Mechanical Structure of Herbie

The structure of Herbie is made of acrylic sheets which are light-weight. Herbie consists of a skeleton and various chambers, as well as a base and a head. It also has different subsections that hold electronic circuits and enable power supply. The skeletal structure consists of an acrylic tube that is 200mm in diameter and 400mm in height. The base of Herbie is made up of a circular acrylic sheet that has a radius of 192mm and a thickness of 5mm.

The base also consists of two clamps that hold the motor, and two castor wheels attached beneath it. The motors are DC motors and run at 60rpm. And each motor is individually powered and controlled by a driver through Raspberry Pi that functions as Herbie’s brain.

The latest upgraded version of Herbie is more efficient and stable, with comparatively low power consumption.

Application of Intelligence

The artificial cognition of Herbie facilities:

Image processing through the faculty of vision
Speech processing through faculties of hearing and speaking
Gaseous detectors through the faculty of smell
Ultrasonic sensors through the faculty of touch

Faculty of vision and image processing

Herbie’s faculty of vision is used particularly for image processing that can help with:

Surveillance or obstacle detection
Navigation or path detection
Face recognition

Obstacle detection using Herbie

The camera affixed to Herbie captures images and identifies obstacles as well as the path. The image captured is then mapped and labeled using a Planner. Based on the label, the subsequent motion commands are configured. For instance, if the label is “X,” then a motion command to move forward is configured. And if the label is “Y,” it should move to the right.

Algorithm used for this purpose

The first step in the algorithm is to detect obstacles from a captured image.
All the images are then processed as grayscale images as it is a single-layer image and reduces the quantity of information obtained from the image, as compared to a colour image (RGB).
The image is then resized and divided into M x N matrices.
The edge of the image represents the outline of the obstacle. The surface and the shadow of the image will be composed of darker or brighter pixels, which helps Herbie tell the difference.
The edge of the obstacle is extracted by calculating the variance in a small square window which has the width w[pixel].
The variance value V(p,q) is calculated for each pixel. A Threshold (Th) is fixed for the calculation

Th = b[V(p,q)] +tσ; where σ is a standard deviation of V(p,q), t is decided on an experimental basis.

If the result of the calculation is greater than the threshold itself, Herbie identifies it as an obstacle. However, if the result of the calculation is lower than the threshold figure, Herbie continues to process images, moving in the default direction.
The surface of obstacles is determined by extracting bright and dark pixels.
Areas that do not generate edge or surface data are considered to be minute areas of irregularity on the surface. Herbie rejects this as it does not detect any obstacle.

Faculty of Smell and Gas Sensors

Gas sensors are capable of detecting various gases as well smoke. Such sensors consist of sensing material that has lower conductivity when the air is clear of high concentrations of different gases such as LPG, Propane, Hydrogen, Methane, Smoke, Carbon Monoxide, etc.

Similarly, the sensing material becomes highly conductive when the concentration of such gases are higher in the atmosphere.

In the case of Herbie, an MQ-2 gas sensor is used.
The detection range is 300-10,000ppm.
MQ-2 can function between the temperature range of -20℃ to 50℃.
The sensing material of MQ-2 is SnO2.
The response time is less than 10 seconds.
The analog reading is the converted to sensor voltage using the following equation:

Sensor Voltage = Analog Reading*3.3V/4095

The concentration of gas in PPM is then calculated using the equation:

PPM = 10.938*e˄ (1.7742*sensor voltage)

Faculty of touch and ultrasonic sensor

Ultrasonic sensors are used to measure the distance to an object. These sensors are usually environmentally independent. Its module transmits sound pulses and picks up the echo generated. It measures the time lapse between sending and receiving these pulses to calculate the distance to an object or obstacle.

Herbie uses HC-SR04 ultrasonic sensors.
It has a resolution of 3mm and the ranging distance between 2-50cm
As soon as the module detects an object it transmits the information to a remote station using a wireless transceiver.
A pair of ZigBee is used to transmit and receive data.
When a high pulse of 10μs hits the trigger pin, the sensor transmits 8 pulses of 40KHz each.
The range is then calculated using the difference in the time taken by the pulse/ signal to leave and return as shown in the equation below:

Distance(cm) = Time(sec)/58

Faculty of speech and hearing and speech recognition

The speech recognition process is composed of three major modules:

Acoustic analysis
Training
Testing

Multiple utterances of the same words that are part of the vocabulary are recorded.
These acoustic signals are then processed using the acoustic analysis module.
A knowledge base for the speech recognition system is developed using the training module.
And the testing module is employed for system testing.
A multilingual speech recognition system for associated words is developed using Hidden Markov Model Toolkit (HTK).
The system uses Mel Frequency Cepstral Coefficients (MFCC) as feature vectors for each speech unit.
The speech samples are recorded at a sampling rate of 16,000Hz and are represented using 16 bits.
For Herbie we chose three languages for recognition namely, Hindi, English, and Kannada.
The system’s vocabulary consists of 75 different commands, 25 from each of these three languages.
Commands include words such as front, backward, left, right, etc.
The confusion matrix for speech recognition in English was calculated at 98% accuracy.

Conclusion

Soon, we plan to carry out the training and testing of Herbie in challenging environments and probably extend the recognition capabilities to multiple languages. The estimated market size of assistive technology has expanded over the past couple of years. There is a growing demand for Socially Assistive Robots in the field of healthcare. And although it is not entirely possible to replace human beings, humanoid robots are here to stay and to transform the future of mankind.

Posted on July 2, 2020 | by Mayank Satnalika | Leave a Comment Engineering

Here’s how we proactively monitor XVigil’s 50+ Machine Learning models

Machine Learning (ML) models are essential to identifying patterns and making reliable predictions. And at CloudSEK, our models are trained to derive such predictions across data collected from more than 1000 sources. With over 50 different models running in production, monitoring these Machine Learning models is a daunting task and yet indispensable.

The ML development life cycle consists of training and testing models, their deployment to production, and monitoring them for improved accuracy. A lack of adequate monitoring could lead to inaccurate predictions, obsolete models, and the presence of unnoticed bugs in them.

CloudSEK’s Data Engineering team works together with Data Scientists to deploy ML models and track their performances continuously. To achieve this, we ensure that the following requirements are fulfilled:

Model versioning: Enable multiple versions of the same models
Initializing observer patterns using incoming data
Comparing results from different versions

At CloudSEK, different Machine Learning models and their multiple versions classify a document across various stages. Whereby, the client is alerted only to the most accurate results from efficient models or an ensemble of results by combining different versions.

What constitutes a version upgrade?

At its core, all machine learning modules are composed of 2 parts. The output of an ML module depends on both of these components:

The core ML model weights file which is generated upon training a model.
The surrounding code statements that represent preprocessing, feature extraction, post-processing, etc.

As a rule of thumb, any significant modifications made to these two components qualify as a version upgrade. However, minor changes or bug fixes or even static rules additions don’t lead to an upgrade, and are simply considered as regular code updates, which we track via Git.

Deploying and Monitoring Models

Generally, Machine Learning models are hosted on stateless docker containers. Such containerized models listen to queues for messages, as soon as the docker container runs on a system. The container maintains a configuration file with information about the type of models, their versions, and whether these models are meant for production.

When the docker container is built, you can pass the latest Git repository Git commit hash to it, to be set as an environment variable. The diagram explains the data flow between ML models and their different versions:

When the container is run, data is consumed from a message queue. The model name present in the configuration file determines the data that is consumed. Once it is processed, the predictions are returned as a dictionary which is then persisted into a database.

The ML modules can also possibly return optional metadata that contains information such as the actual prediction scores, functions triggered inside, etc.

Given below is a sample of a document after processing the results from all the models:

{

"document_id" : "root-001#96bfac5a46", 

"classifications_stage_1_clf_v0" : {

"answer" : “suspected-defaced-webpage",

"content_meta" : null,

"hit_time" : ISODate("2019-12-24T14:54:09.892Z"),

"commit_hash" : "6f8e8033"

},

"classifications_stage_2_clf_v0" : {

"answer" : {

"reason" : null,

"type" : "nonthreat",

"severity" : null

},

"content_meta" : null,

"hit_time" : ISODate("2019-12-24T15:40:46.245Z"),

"commit_hash" : null

},

"classifications_stage_2_clf_v1" : {

"answer" : {

"reason" : null,

"type" : "nonthreat",

"severity" : null

},

"content_meta" : null,

"hit_time" : ISODate("2019-12-24T15:40:46.245Z"),

"commit_hash" : null

}

}

How this helps us

This process allows us to find, for any given document, the exact state of all the models that classified a particular document. We can rollback between the model versions and a minor change in the value provided in the configuration file should allow us to set the main production model apart from the test models.

A Metabase instance can be leveraged to visualize key metrics and the performance of each classifier, on a dashboard. It may also contain details about the documents that are processed by each model, or how many documents were classified with category X, category Y, etc. ( in the case of classification tasks), and more.

Screenshot of the internal dashboard which helps in visualisations key metrics — Screenshot of the internal dashboard which helps in visualizing key metrics

Monitoring also allows data scientists to study and compare the results of the various versions of the models, given that the particulars of version outputs are retrieved. This data provides them with a set of documents that reveal which output may have been influenced by a new model. This data is then added to the training data to calibrate the models.

Posted on June 24, 2020 | by Shashank Gupta | Leave a Comment Engineering

A peek into the black-box: Debugging deep neural networks for better predictions

Deep learning models are often criticized for being complex and opaque. They are called black-boxes because they deliver predictions and insights, but the logic behind their outputs is hard to comprehend. Due to its complex multilayer nonlinear neural networks, Data Scientists find it difficult to ascertain the factors or reasons for a certain prediction.

This unintelligibility makes people wary of taking important decisions based on the models’ outputs. As human beings, we trust what we understand; what we can verify. And over time, this has served us well. So, being able to show how models go about solving a problem to produce insights, will help build trust, even among people with cursory knowledge of data science.

To achieve this it is imperative to develop computational methods that can interpret, audit, and debug such models. Debugging is essential to understanding how models identify patterns and generate predictions. This will also help us identify and rectify bugs and defects.

In this article we delve into the different methods used to debug machine learning models.

Source: interpretable-ml-book — Source: https://christophm.github.io/interpretable-ml-book/terminology.html

Permutation Importance

Also known as permutation feature importance, it is an algorithm that computes the sensitivity of a model to permutations/alterations in a feature’s values. In essence, feature importance evaluates each feature of your data and scores it based on its relevance or importance towards the output. While permutation feature importance, measures each feature of the data after it has been altered, and scores it based on its importance towards generating an output.

For instance, let us randomly permute or shuffle the values of a single column in the validation dataset with all the other columns intact. If the model’s accuracy drops substantially and causes an increase in error, that feature is considered “important”. On the other hand, a feature is considered ‘unimportant’ if shuffling its values doesn’t affect the model’s accuracy.

Debugging ML models using ELI5

ELI5 is a Python library that assists several ML frameworks and helps to easily visualize and debug black-boxes using unified API. It helps to compute permutation importance. But it should be noted that the permutation importance is only computed on test data after the model is built.

After our model is ready, we import ELI5 to calculate permutation importance.

The output for above code is shown below:

The features that are on top are the most important, which means that any alterations made to these values will reduce the accuracy of the model significantly. The features that are at the bottom of the list are unimportant in that any permutation made to their values will not reduce the accuracy of the model. In this example, OverallQual was the most important feature.

Debugging CNN-based models using Grad-CAM (Gradient-weighted Class Activation Mapping)

Grad-CAM is a technique that produces visual explanations for outputs to deliver transparent Convolutional Neural Network (CNN) based models. It examines the gradient information that flows into the final layer of the neural network to understand the output. Grad-CAM can be used for image classification, image captioning, and visual question answering. The output provided by Grad-CAM is a heatmap visualization, which is used to visually verify that your model is trained to look at the right patterns in an image.

Debugging ML models using SHAP (SHapley Additive exPlanations)

SHAP is a game theoretic approach that aims to explain a prediction by calculating the importance of each feature towards that prediction. The SHAP library uses Shapley values at its core and explains individual predictions. Lloyd Shapley introduced the concept of Shapley in 1953 and it was later applied to the field of machine learning.

Shapley values are derived from Game theory, where each feature in the data is a player, and the final reward is the prediction. Depending on their contribution towards the reward, Shapley values tell us how to distribute this reward fairly among the players.

We use SHAP quite often, especially for the models in which explainability is critical. The results are really quite accurate.

SHAP can explain:

General feature importance of the model by using all the data
Why a model calculates a particular score for a specific row/ record
The more dominant features for a segment/ group of data

Shapley values calculate the feature importance by comparing two predictions, one with the feature included and the other without it. The positive SHAP values affect the prediction/ target variable positively whereas the negative SHAP values affect the target negatively.

Here is an example to explain the same. For this purpose, I am taking a red wine quality dataset from kaggle.

Now, we produce variable importance plots, which lists the most significant variable in descending order. Wherein, the top variable would contribute more to the model.

In the above figure, all the variables are plotted in descending order. Color of the variables represent the feature value, whether they are high (in red) or low (in blue) in that observation. A high level of the “sulphates” content has a high and positive impact on the quality rating. X-axis represents the “positive” impact. Similarly, we can say that “chlorides” is negatively correlated with the target variable.

Now, I would like to show you how SHAP values are computed in individual cases. Then, we execute such values on several observations and choose a few of the observations randomly.

After choosing random observations, we initialize our notebook with initjs().

Explanation for some of the terms seen in the plot above:

Output value: Prediction for that observation, which in this case is 5.35
Base value: Base value is the mean prediction, or mean (yhat), here it is 5.65
Blue/ red: Features that can impact the prediction more are shown in red, and those that have the least impact are in blue color.

For more information and to test your skills, check out kaggle.

Posted on June 19, 2020 | by CloudSEK Threat Intelligence | Leave a Comment Engineering

[Quiz] Weekly Cyber Trivia Friday #1

Cyber Trivia Friday is here!

In our first-ever cyber quiz, we want to find out if you’re up-to-date on your cybersecurity news from across the world.

If you’re behind on the news, fret not. We’ve sprinkled in some hints to help you along.

The winners will be the first 3 people to submit the quiz and get all 5 questions right.

Loading…

The growing presence of cybercriminals on social media platforms

The rise of Telegram and Discord

The double-edged sword of Telegram’s end-to-end encryption

Telegram’s secret chat option

How threats actors are exploiting Telegram

Discord is now where the world hangs out

How cybercriminals are leveraging Discord

Is Discord the new dark web?

The need for continuous monitoring

What is Sharding?

Sharded Cluster Tutorial

Data directories

Config server and Shard server

Mongos Server

Status and Initiation of Replica Sets

Add Shards

Build Process

Transpiling

Bundling

Minification/ Obfuscation

Reverse engineering code

Debundling

Security Tip

How do these frameworks change attack vectors?

Reduced XSS

CSP-Bypasses

Building a language-independent microservice architecture

Why RESTful API is not the right choice

Make way for gRPC and Protocol Buffers

How do we use Protobuf?

Advantages of using this architecture:

Creating a microservice with gRPC

Step 1: Defining a service

Step 2: Selecting languages

Step 3: Compiling

Tip: You can host these definitions in a repository, and use the script generated in a CI/CD Pipeline to automate this process.

Step 4: gRPC Server and Client

Server (Go):

Client (Node.js):

Test the code

Summary

Bonus:

Classification through similarity scoring

CloudSEK’s Knowledge Extraction Module

Finally…

What is Shodan?

How does Shodan work?

The Basics

Shodan Filters

Continuous Monitoring

Automation

Conclusion

Herbie

Mechanical Structure of Herbie

Application of Intelligence

Faculty of vision and image processing

Obstacle detection using Herbie

Algorithm used for this purpose

Faculty of Smell and Gas Sensors

Faculty of touch and ultrasonic sensor

Faculty of speech and hearing and speech recognition

Conclusion

What constitutes a version upgrade?

Deploying and Monitoring Models

How this helps us

Permutation Importance

Debugging ML models using ELI5

Debugging CNN-based models using Grad-CAM (Gradient-weighted Class Activation Mapping)

Debugging ML models using SHAP (SHapley Additive exPlanations)