A Beginner's Guide To Web Scraping With Python

on August 12, 2020


A Beginner’s Guide To Web Scraping With Python


A Beginner Guide To Web Scraping With Pythong


Extracting chunks of data from web pages is a job on its own, one that can be very resource and time-intensive. At least that’s what newbie developers or data scientists go through, manually combing websites to maintain hoards of excel sheets. Owing to accuracy constraints, this also turns out to be a counterproductive precursor while building any Machine Learning model.

Enter Web Scraping - an automated technique that can extract massive amounts of data from websites in a fraction of the time.

Feels like Deja Vu? You might have used (or heard of) APIs to do the same. In fact, it remains the best method to access structured website data. But keep in mind that not all websites provide APIs. In such cases, Web Scraping comes to our rescue as a powerful alternative strategy!


Use Cases of Web Scraping


Web Scraping with Python can be used for a multitude of reasons. Top use cases of the practice include:

  • Creating a lead generation engine with rich market data
  • Monitoring approval ratings of brands by scraping reviews
  • Conducting technical SEO audits with keyword analysis
  • Building Machine Learning models with large data sets
  • Conducting market research and analysis at scale
  • Gathering feed data for end-to-end testing bots
  • Collecting product information for price comparisons
  • Analyzing social media data to understand what’s trending
  • Collecting R&D data to carry out surveys and data analysis
  • Curating data from multiple sources to create a unified platform

The list goes on!


Also Read:Top 10 Uses of Python in Business


Is Web Scraping Legal?


This is arguably the most common query that novice web scrapers come across. The short answer - it depends; a lot on the website that you are scraping and the reason for the activity.

To better understand if a website allows scraping, your best bet is to read the robots.txt file (and tie this up with the terms and conditions to better understand your chances). They are created by webmasters and instruct the search engine bots about how to crawl pages on the website. This includes indicators for whether specific web-crawling software is allowed or disallowed to crawl any section of the website.

To access the robots.txt file, ‘simply navigate to this link on your target website’: www.example.com/robots.txt


Here is what you need to find (and steer clear of) in such as file:

1. Allow full access

This is denoted by the following and states that all pages are crawlable by bots.


Python for beginner - Allow full access


2. Block all access

This states that no part of the website can be accessed by an automated web crawler. It is denoted by:


Python for beginner - bloc all access


3. Partial access

This declares specific sections or files on the site that are accessible and the ones that are not. Denoted by:


Python for beginner - partial access


How To Scrape Websites With Python


Packed with a large collection of libraries and an easy-to-understand syntax, scraping with Python can be intuitive and fun, especially for beginners. Moreover, libraries such as Numpy, Matlplotlib, and Pandas open the gates for further manipulation of the extracted data.

So let’s get to the chase with the basics!


Required Installations


We would be using the Google Chrome Browser and Ubuntu Operating System for this tutorial. In the case you are on a different OS, that’s perfectly fine. Python is a high-level, cross platform language. This means that a code written on one OS can run on another OS without any hiccup.


Just head on to one of these links (based on your operating system) to download and install Python:

If you don’t have Python installed on your Ubuntu system, simply head on to the terminal and type in this command:


Python for beginner - Install python on your system


Next, we need to install the Python PIP package management system in order to install and manage software packages that are written in Python.


Python for beginner - Install python PIP


Additionally, we will be using the following libraries in the tutorial. Here are some details of why we need them and how they can be installed through the terminal:

  • Beautiful Soup: To extract the data from markup languages such as HTML and XML.

Python for beginner - Beautiful soup


  • Pandas: To extract data and store it in the desired format for data analysis.

Python for beginner - Install pandas


  • Selenium: To automate website testing and performing actions such as element clicks, refresh page, goto website link, and more.

Python for beginner - Install selenium


Next, it is important to set the path to chromedriver. This will configure the webdriver to use the Chrome browser.

The best way to do this is to use the Webdriver Manager for Python with two simple steps:

  • Install the manager:

Python for beginner - Webdriver manager


Once you are set, follow this step by step process to begin scraping with Python:


1. Finding and Selecting the URL that You Want to Scrape


Begin the scraping process by identifying the URL or website that you aim to extract the data from. This can be any URL on the Internet, as long as it does not violate any organizational policy against data security or web scraping.

For this tutorial and illustration purposes, we are going to scrape the URL: example.com/product-link

You can follow the steps discussed here for any URL of your choice.


2. Inspect the Page that We Want to Scrape


The next step is to inspect the data on the web page and accordingly devise the approach to scaping. For instance, if the data on the web page is nested in tags (which is the most common scenario), you would need to pinpoint these tags.

To do this, simply right click on the web page and click on ‘Inspect’.


Python for beginner - Inspect the page


Notice that the ’<‘div’>’ tag has multiple nested tags, each representing a set of data.


3. Finding and Finalizing the Data We Want to Extract


Select the type of data that you want to extract from the web page. For instance, if you are gathering details of products on an e-commerce website, you may want to extract product details such as Price, Name, and Rating.


4. Writing the Code to Extract the Data


And now, we code!

Begin by creating a Python file by opening the terminal and typing gedit with a .py extension. For instance:


Python for beginner - extract data


Before we go any further and begin writing code in this file, we need to import the requisite libraries first. Simply follow these commands:


Python for beginner - Import the requiste libraries


If you’re using chrome, run this:


Python for beginner - Import chrome driver manager


For other browsers, you can refer to this link.

The last command will open the URL and let it load completely before returning the control to the script.

Next, we need to open the target URL and specify the data lists that need to be created while scraping the web page.

We will be creating two lists, one for the product names and another for their prices.


Python for beginner - create list


content creates an instance, representing the content of the entire page source.

soup lets you navigate and search through the HTML code for the data that is to be scraped.


In our scenario, suppose that the web page that we are scraping is of an e-commerce product where following tags are being used for our target data sets:

  • < H1 > Tag: Name of the products
  • < span > Tag: Price of the products

For every data set, we need to first find the tags that have the respective class-names and extract the target data before storing them in variables. The following code is used to execute these steps.


Python for beginner - Extract target data


Here, ‘s-iteminfo’, ‘s-productname’, and ‘s-product_price’ refer to the respective classes under which our target tags (data location) are nested.


5. Storing the Data in the Most Appropriate Format


The next step is to run the code with the following command:


Python for beginner - Store data


It returns with our target data sets. It is now time to organize and store it for better readability and retrieval. This is where the Pandas data frame comes into the picture.

The following code does the trick.


Python for beginner - Store data in pandas


‘pd.DataFrame’ creates a two-dimensional tabular data set that contains labeled axes (rows and columns). This essentially means that the extracted data is converted into a table with two columns - Product Name and Price.

The resulting Pandas data frame is then written to a .csv file with specifications for index and encoding. index=False passes a False boolean value to the index parameter since we generally do not need to store the preceding indices of every row.

The output CSV file is then created with columns and rows of the defined data sets. It will the name ‘products.csv’ should look something like this upon running:


Python for beginner - Output csv file

*prices are just indicative as examples and do not represent any real-world market indications.


Why Do We Use Selenium


To put it simply, Selenium helps us automate testing for web applications. Browser interactions such as link clicks and form submissions can be quickly automated with specific Python scripts. It even helps in the case of multiple ajax requests where data needs to be scraped from Javascript generated content. While this can be done with BeautifulSoup too, the difference in the way the data on the web page is rendered calls for more options.


Looking Forward


We hope this guide was useful to you to get started with scraping with Python. Like every advanced Python concept, we have merely grazed the surface here. If you are planning to dive deeper into the universe of Python and its infinite capabilities, a structured course can help you better navigate the uncertainties and complexities. Our Python Training Course fits the bill in such situations and has been created to help novice developers become skilled at handling Python code!


Our magic sauce? Hybrid learning and real-life projects! Learn more about how such a course can help accelerate your career.


Also Read:Python for Beginners with Examples