Tag: Research

Let us assume search engines, like Google, never existed! How would you find what you need from across 4.2 billion web pages? Web crawlers are programs written to browse the internet, to gather information, index and parse the collected data, to facilitate quick searches. Crawlers are, thus, a smart solution to big data sets and a catalyst to major advancements in the field of cyber security.

In this article, we will learn:

1. What is crawling?
2. Applications of crawling
3. Python modules used for crawling
4. Use-case: Fetching downloadable URLs from YouTube using crawlers
5. How do CloudSEK Crawlers work?

What is crawling?

Crawling refers to the process of scraping/ extracting data from websites/ the internet using web crawlers. For instance, Google uses spider bots (crawlers) to read the content of billions of web pages and posts. Then, it gathers data from these sites and arranges them in the Google Search index.

Basic stages of crawling: 

1. Scrape data from the source
2. Parse the collected data
3. Clean the data of any noise or duplicate entries
4. Structure the data as per requirement

Applications of crawling

Organizations crawl and scrape data off of web pages for various reasons that may benefit them or their customers. Here are some lesser known applications of crawling: 

• Comparing data for market analysis
• Monitoring data leaks
• Preparing data sets for Machine Learning algorithms
• Fact-checking information on social media

Python modules used for crawling

• Requests – Allow you to send HTTP requests to web pages
• Beautifulsoup – Python library that retrieves data from HTML and XML files, and parses its elements to the required format
• Selenium – Open source testing suite used for web applications. It also performs browser actions to retrieve data.

Use-case: Fetching downloadable URLs from YouTube using crawlers

A single YouTube video may have several downloadable URLs, based on: its content, resolution, bitrate, range and VR/3D. Here is a sample API and CLI code to get downloadable URLs on YouTube along with their Itags:

Project structure 

youtube
|
|---- app.py
|---- cli.py
`---- core.py


The project will contain three files:

app.py: For the api interface, using flask micro framework

cli.py: For command line interface, using argparse module

core.py: Contains all the core (common) functionalities which act as helper functions for app.py and cli.py.

# youtube/app.py
import flask
from flask import jsonify, request
import core
app = flask.Flask(__name__)
app.config["DEBUG"] = True
@app.route('/', methods=['GET'])
def get_downloadable_urls():
if 'url' not in request.args:
return "Error: No url field provided. Please specify an youtube url."
url = request.args['url']
urls = core.get_downloadable_urls(url)
return jsonify(urls)
app.run()


The flask interface code to get downloadable URLs through API.

Request url - localhost:<port>/?url=https://www.youtube.com/watch?v=FIVPlraNgXs

# youtube/cli.py
import argparse
import core
my_parser = argparse.ArgumentParser(description='Get youtube downloadable video from url')
my_parser.add_argument('-u', '--url', metavar='', required=True, help='youtube url')
args = my_parser.parse_args()
urls = core.get_downloadable_urls(args.url)
print(f'Got {len(urls)} urls\n')
for index, url in enumerate(urls, start=1):
print(f'{index}. {url}\n')

Code snippet to get downlodable urls through comand line interface (using argparse to parse command like arguments)

Command line interface - python cli.py -u 'https://www.youtube.com/watch?v=aWPYw7iVBg0'

# youtube/core.py
import json
import re
import requests
def get_downloadable_urls(url):
html = requests.get(url).text
RE = re.compile(r'ytplayer[.]config\s*=\s*(\{.*?\});')
conf = json.loads(RE.search(html).group(1))
player_response = json.loads(conf['args']['player_response'])
data = player_response['streamingData']
return [{'itag': frmt['itag'],'url': frmt['url']} for frmt in data['adaptiveFormats']]

This is the core (common) function for both API and CLI interface. 

The execution of these commands will:

1. Take YouTube url as an argument
2. Gather page source using the Requests module
3. Parse it and get streaming data
4. Return response objects: url and itag

How to use these URLs?

• Build your own YouTube downloader (web app)
• Build an API to download YouTube video

Sample result 

[{
'itag': 251,
'Url': 'https://r2---sn-gwpa-h55k.googlevideo.com/videoplayback?expire=1585225812&ei=9Et8Xs6XNoHK4-EPjfyIiA8&ip=157.46.68.124&id=o-AGeDi3DVtAbmT5GiuGsDU7-NPLk23fOXNnY16gGQcHWu&itag=251&source=youtube&requiressl=yes&mh=Av&mm=31%2C26&mn=sn-gwpa-h55k%2Csn-cvh76ned&ms=au%2Conr&mv=m&mvi=1&pl=18&initcwndbps=112500&vprv=1&mime=audio%2Fwebm&gir=yes&clen=14933951&dur=986.761&lmt=1576518368612802&mt=1585204109&fvip=2&keepalive=yes&fexp=23882514&c=WEB&txp=5531432&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ADKhkGMwRAIgK4L4VVHAlWMPVPEcmdkhnb2u8UM6eYhFz16kGruxZjUCIFXZJM9ejVK7OZJFqx7YwBqa3CrDvVakuU86vcIyMv-a&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=ABSNjpQwRAIgKBhJytjv73-c7eMWbVkb-X8_rNb7_xApZvaPfw7wGcMCIHqJ405fQ3Kr-e_5fV8gokMUNi0rrrLG8T85sLGTQ17W'
}]

What is ITag?

ITag gives us more details about the video such as the type of video content, resolution, bitrate, range and VR/3D. A comprehensive list of YouTube format code ITags can be found here.

How do CloudSEK Crawlers work?

CloudSEK’s digital risk monitoring platform, XVigil, scours the internet across surface web, dark web, and deep web, to automatically detect threats and alert customers. After configuring a list of keywords suggested by the clients, CloudSEK Crawlers:

1. Fetch data from various sources on the internet
2. Push the gathered data to a centralized queue
3. ML Classifiers group the data into threats and non-threats
4. Threats are immediately reported to clients as alerts, via XVigil. Non-threats are simply ignored.

Introduction

The process of threading allows the execution of multiple instructions of a program, at once. Only multi-threaded programming languages like Python support this technique. Several I/O operations running consecutively decelerates the program. So,  the process of threading helps to achieve concurrency.

In this article, we will explore:

1. Types of concurrency in Python
2. Global Interpreter Lock
3. Need for Global Interpreter Lock
4. Thread execution model in Python 2
5. Global Interpreter Lock in Python 3

Types of Concurrency In Python

In general, concurrency is the parallel execution of different units of a program, which helps optimize and speed up the overall process. In Python, there are 3 methods to achieve concurrency:

1. Multi-Threading
2. Asyncio
3. Multi-processing

We will be discussing the fundamentals of thread-execution model. Before going into the concepts directly, we shall first discuss Python’s Global Interpreter Lock (GIL).

Global Interpreter Lock

Python threads are real system threads (POSIX threads). The host operating system fully manages the POSIX threads, also known as p-threads.

In multi-core operating systems, the Global Interpreter Lock (GIL) prevents the parallel execution of p-threads of a multi-threaded Python process. Thus, ensuring that only one thread runs in the interpreter, at any given time.

Why do we need Global Interpreter Lock?

GIL helps to simplify, the implementation of the interpreter, and memory management. To understand how GIL does this, we need to understand reference counting.

For example: In the code below, ‘b’ is not a new list. It is just a reference to the previous list ‘a.’

>>> a = []
>>> b = a
>>> b.append(1)
>>> a
[1]
>>> a.append(2)
>>> b
[1,2]

Python uses reference counting variables to track the number of references that point to an object. The memory occupied by the object is released if the value of the reference counting variable is zero. If threads, of a process sharing the same memory, try to access this variable to increment and decrement simultaneously, it can cause leaked memory that is never released, or releasing the memory incorrectly. And this leads to crashes.

One solution is to have a lock for the reference counting variable object memory by using semaphores so that it is not modified simultaneously. However, adding locks to all objects increases the performance overhead in the acquisition and release of locks. Hence, Python has a GIL which gives access to all resources of a process to only one thread at one time. Apart from GIL there are other solutions, such as garbage collection used in JPython interpreter, for memory management.

So, the primary outcome of GIL is, instead of parallel computing you get pre-emptive (threading) and co-operative multitasking (asyncio).

Thread Execution Model in Python 2

import time
def countdown(n):
     while n > 0:
     n -= 1
count = 50000000
start = time.time()
countdown(count)
end = time.time()
print('Time taken in seconds -', end-start)

import time
from threading import Thread
COUNT = 50000000
def countdown(n):
     while n>0:
     n -= 1
t1 = Thread(target=countdown, args=(COUNT//2,))
t2 = Thread(target=countdown, args=(COUNT//2,))
​start = time.time()
t1.start()
t2.start()
t1.join()
t2.join()
end = time.time()
​print('Time taken in seconds -', end - start)

Ideally, the par.py execution time should be half of the seq.py execution time. However, in the above example we can see that the par.py execution time is slightly higher than that of seq.py. To understand the reduction in performance, despite the sharing the work between two threads which run in parallel, we need to first discuss CPU-bound and I/O-bound threads.

CPU-bound threads are threads performing CPU intense operations such as matrix multiplication or nested loop operations. Here, the speed of program execution depends on CPU performance.

I/O-bound threads are threads performing I/O operations such as listening to a socket or waiting for a network connection. Here, the speed of program execution depends on factors including external file systems and network connections.

Scenarios in a Python multi-threaded program

When all threads are I/O-bound

If a thread is running, it holds the GIL. When the thread hits the I/O operation, it releases the GIL, and another thread acquires it to get executed. This alternate execution of threads is called multi-tasking.

Where one thread is CPU bound, and another thread is IO-bound:

A CPU bound thread is a unique case in thread execution. A CPU bound thread releases the GIL after every few ticks and tries to acquire again. A tick is a machine instruction. When releasing the GIL, it also signals the next thread in the execution queue (ready queue of the operating system) that the GIL has been released. Now, these CPU-bound and I/O-bound threads are in a race to acquire the GIL. The operating system decides which thread needs to be executed. This model of executing the next thread in the execution queue, before completing the previous thread, is called pre-emptive multitasking.

In most of the cases, operating system gives preference to the CPU bound thread and allows it to reacquire the GIL, leaving the I/O-bound thread starving. In the below diagram, the CPU-bound thread has released GIL and signaled thread 2. But even before thread 2 tries to acquire GIL, the CPU-bound thread has reacquired the GIL.  This issue has been resolved in Python 3 interpreter’s GIL.

In a single core operating system, if the CPU bound thread reacquires GIL, it pushes back the second thread to the ready queue, assigning it some priority. This is because Python doesn’t have control over the priorities assigned by the operating system.

In a multicore operating system, if the CPU bound thread reacquires the GIL then it does not push back the second thread, but continuously tries to acquire the GIL using another core. This characterizes thrashing. Thrashing reduces the performance if many threads try to acquire the GIL, using different cores of the operating system. Python 3 also addresses the issue of thrashing.

Global Interpreter Lock in Python 3

The Python 3 threat execution model has a new GIL. If there is only one thread running, it continues to run, until it hits an I/O operation or other thread requests, to drop the GIL. A global variable (gil_drop_request) helps to implement this.

If gil_drop_request = 0, running thread can continue until it hits I/O

If gil_drop_request = 1, running thread is forced to give up the GIL

Instead of CPU-bound thread check after every few ticks, the second thread is sending a GIL drop request by setting the variable gil_drop_request = 1 after reaching a timeout. The first thread will then immediately drop the GIL. Additionally, to suspend the first thread’s execution, the second thread sends a signal. This helps to avoid thrashing. This check is not available in Python 2.

Missing Bits in the New GIL

While the new GIL does address issue such as thrashing, it still has some areas of improvement:

Waiting for time out

Waiting for timeout can make the I/O response slow. Especially, when there are multiple, recurrent I/O operations. I/O-bound Python programs take considerable time to add a GIL, followed by a time out. This happens after every I/O operation, and before the next input/output is ready.

Unfair GIL acquiring

As seen below, the thread that makes the GIL drop request is not the one that gets the GIL. This type of situation can reduce the performance of I/O, where response time is critical.

Prioritizing threads

There is a need for the GIL to distinguish between CPU-bound and I/O-bound threads and then assign priorities. High priority threads must be able to immediately preempt low priority threads. This will help improve the response time considerably.  This issue has already been resolved in operating systems. Operating systems use timeout to automatically adjust task priorities. If a thread is preempted by a timeout, it is penalized with a low priority. Conversely, if a thread suspends early, it is rewarded with raised priority. Incorporating this in Python will help improve thread performance.

In their quest for new attack surfaces, threat actors find easy targets among the ~2.7 million Android applications, ad infinitum.

What makes Android applications irresistible targets? 

1. The ease with which an attacker can acquire the entire source code of an Android application.
2. The source codes often contain API keys, secret tokens, sensitive credentials, and endpoints, which developers forget to remove after staging.

Developers often lack awareness about attack vectors on a mobile app. Owing to a false sense of security, provided by the sandboxed and permission-oriented operating systems. However, mobile apps have the same attack vectors as web apps, albeit with different exploitation techniques. 

When an attacker meets an Android app

After getting their hands on an Android app’s source code, attackers first decompile it, analyse it for weaknesses, and then exploit it.

Decompiling the source code

The attacker first extracts files and folders from the apk file of an Android app, which is similar to unzipping a zip file. However, the files are compiled. To read the source code, the files are decompiled using:     

• Apktool: To read the AndroidManifest.xml file. It also disassembles to small code and allows to repack the apk after modifications.
• Dex2jar: To convert the Dex file into a Jar file.
• Jadx/Jd-gui: To read the code in GUI format.

Analysing the source code

The attacker analyses the source code using 2 methods: 

• Static Analysis
• Dynamic Analysis

Static Analysis

In this approach, the attacker examines the source code for secret tokens, API keys, credentials, and secret paths. They also understand the source code, check for activities, content providers, broadcast receivers, vulnerable permissions, local storage, sensitive files, etc.

Here are some examples of what attackers look for during static analysis:

Sensitive Information 

As we can see, the strings.xml file below exposes sensitive information such as API keys, bucket name, Firebase database URL, etc.

Working Credentials

Often, usernames and passwords are hidden in the source code, because developers forget to remove them. 

Content Providers

Content Providers allow applications to access data from other applications. And in order to access a Content Provider, you need its URI. 

Attackers check if the Content Provider attribute is ‘exported=true,’ which implies that it can be accessed by third-party applications.

Below we are accessing the content provider declared by the vulnerable application through the tool i.e. content, which acts as a third-party application.

Activities

An activity implements a screen/window in an app. So, an app usually invokes only an activity i.e. a particular screen in another app, and not the app as a whole.  

Attackers check for weaknesses in activities in the AndroindManifest.xml file. Where, if an activity is marked as ‘exported=true,’ a third-party app can initiate that activity.

For example, the screen below has a functionality to submit a password. And only on submitting the password, we can see the dashboard. 

However, if the activity responsible for showing the dashboard is marked ‘exported= true,’ an attacker can use an Activity Manager (AM) tool to run it. 

And by doing this the attacker can access the dashboard without a password. 

Broadcast Receivers

Broadcast receivers listen for system-generated or custom generated events from other applications or from the system itself. 

Attackers check for weaknesses in Broadcast Receivers in the AndroindManifest.xml file. And if the intent-filter tag is declared ‘exported=true,’ a third-party application can easily access it. To prevent this, we need to explicitly declare ‘exported=false.’

After checking for the parameters that the receiver can accept, attackers can write a command that will trigger the receiver on behalf of the application.  

For example: The following command triggers the receiver to send a message to a phone number. 

Following which, the message is delivered to the phone number:

Dynamic analysis

In this approach, the attacker uses a binary toolkit such a Frida to hook to a targeted application and change its implementation. It also allows the attacker to bypass root detection and SSLs. 

Since Frida has the capability to access classes and functions of a targeted process/ application, the attacker injects their own JavaScript (JS) payload at runtime, to analyze the behavior of the code. 

Bypassing root detection

The following application has root detection. Hence, to access all the capabilities of the application, the attackers need to bypass root detection. 

First, the attacker needs to identify the function that is responsible for root detection. Which they can get from the source code:

If one of the functions i.e. ‘doesSuperuserApkExist(‘’) and doesSuExist(‘’)’ returns True, it will be identified as a Rooted Device.

So, the attacker needs to change the implementation of these two functions to False, in order to bypass the rooting. And this is where Frida comes to use. 

By injecting the following JS payload, the attacker changes the implementation of the functions responsible for root detection. 

With the help of use(), the attacker accesses the PostLogin class. Here, they mention the function i.e. ‘doesSUexist or doesSuperuserApkExist,’ that they need to hook. After which, the function will start returning False instead of True. 

Frida Client then sends the server this JS payload. And the server makes it a thread and sends it to the JS Engine. So, whenever the application calls this function, Frida will call the thread instead of the actual function declared in source code.

Conclusion

Given the increasing sophistication of cyber-attacks, it is important that Android apps undergo proper vulnerability assessments before publishing. Also, developers should not leave sensitive information such as API keys and credentials exposed in the source code. 

The purpose of content, regardless of the choice of font, size, colour, and design, is to communicate one’s thoughts. The aesthetic of the text is just as important, as the composition of sentences and paragraphs, in conveying ideas, to the reader, with clarity. So, it’s not just what, but also how, a writer presents their content, that determines its impact on the reader. Better typography produces better content. 

How does typography affect content?

• Typography is crucial in invoking and sustaining a reader’s interest from the first line to the last. 
• It can enhance the overall readability of the text. 
• It makes the reading experience less effortful, thus allowing the reader to absorb the content with ease. 
• 90% of the content we consume on the web, in books, posters, and emails, is text. So, typography is not a mere afterthought. It is a primary factor that determines the reach and impact of the content it presents. 

In essence, typography is:

Where type refers to letters and other characters which are arranged to form textual content.

How to choose a Typeface?

Writers are often confronted with the task of choosing the appropriate font to deliver their content. 

Here are a few factors to consider when choosing a typeface:

• Choosing a typeface with multiple weights to create textual contrast

Varying the text weights will add contrast to the content. Apart from beautifying the text, it also helps to distinguish key text fragments from the rest of the content. 

• Tailor the content to suit your audience

While creating content, the writer should have the intended audience in mind. The readers’ age, interests, and cultural backgrounds determine what they read.

For instance, children and adults prefer different types of fonts.  

To stimulate an interest in reading, young readers need well-shaped and legible letters, such as Sassoon Primary or Gill Sans. Similarly, commonly used font styles such as Roboto or Futura are more appealing to adults.

How to add variety and flair to text content?

Contrasting helps to draw the focus of readers to certain words or phrases. It also establishes hierarchy within the content and creates seamless content flow, by building relationships. Thus making it easy to navigate the different sections of the content. 

Techniques of contrasting

Variation in text is achieved by:

• Adjusting the weight of text 
• Underlining text
• Styling text using italics
• Changing size and color of the font
• And many more 

Writers can define their own style of contrasting, as well.

Tips: To reflect hierarchy in the content, font size of the text can be adjusted by 1.5x to 2x. This together with a variation in the weight of the text, will help the writer create visible gradations within, and between, sections.

How to make text more engrossing?

Creating a style guide for typography will help writers standardize type and enhance legibility across their content.

Keeping the right line height for the text content

The vertical space between lines is crucial to the visual impact of the content. Narrow line spacing makes the content look crammed and will prove tedious to the reader. While increasing line spacing improves readability drastically, it adds to the content real estate.

Tips: For better legibility, the line height of the text should be between 1.2 to 1.5 times the size of the font.

Takeaway

Typography plays a significant role in the process of writing textual content. Good typography makes reading effortless, whereas poor typography is off putting. Since typography is as much an art as it is a technique, the scope for experimentation is unlimited.  Writers should play with different styles and patterns, before settling on the one that suits their content best.