Web Scraping using Python: Tutorial on how to extract data from website using Python

What is Web Scraping?

In simple words, Web Scraping is an extraction of data from a website. For example, some websites contains an extensive amount of invaluable data like stock prices, product’s details and sports stats, so if you want to access their information, either you will copy or paste information manually or write a code (Web Scrapping) that automates it. Web scraping is also known as web data mining or web harvesting.

It is difficult to gather the information automatically from webpage as it is scatter all over and layered in various structures and codes. By using Web scraping it become so easy to structure the data.

Why Python for Web Scraping?

Following reasons makes Python more suitable for Web Scraping:

• Python has an Enormous collection of Libraries.
• It has Easily Understandable Syntax.
• Python has one of the biggest and most active communities, where you can seek help from.
• Python code is relatively smaller compared to codes of other languages. Means less time to build web scrapper.

How does it Works?

To understand it well, lets scrap Wikipedia to get a detail of S&P 500 Companies. However, you can use the method described here to any extract data from any website.

First things is to establish a connection between Python code and website and get page contents. Every web page is a html page which browser renders to give beautiful looking website.

We can subdivide entire process of web scrapping into following broader areas:

• Connecting the website and fetching its HTML.
• Navigating and searching HTML tags for the relevant data and extracting its contents

Python has many libraries that automates all this tasks, so you don’t have to write code for it. Let’s get started.

Connecting the website and fetching its HTML.

This is the first step of every web scrapping process. You don’t have to write lengthy code for establishing a connection to the website and fetching its HTML. Beauty of Python and its open source library is that we can do it in just one line of code.

Get method of requests library helps to get html contents of the desired url.

Code: Following code, stores the html content of Wikipedia page having details of S&P 500 Companies into a variable named page_html.

#Step-1: Connecting to Website and getting its contents
from requests import get
url = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
page_html = get(url) # To see page content, print(page_html.content)

Navigating and searching HTML tags for the relevant data.

Installing BeautifulSoup4 (BS4)

After getting html page source, getting a list of S&P 500 companies boils down to extracting the contents of appropriate html tags. Although you can write a program that does this task, but Beautiful Soup library of Python makes it super easy. It provides idiomatic ways of navigating and searching HTML and XML. You can pip instal beautifulsoup4 package using the following command. For more details about installation and details of Beautiful Soup package, you can visit here.

pip install beautifulsoup4

Parsing page contents

After importing BeautifulSoup from bs4 library, the next step is to parse the page contents with html parser of BeautifulSoup. This returns a beautiful soup object which gives the ease to navigate and search HTML tags based on various conditions like class and id.

Code: Following code stores the BeautifulSoup object after parsing content of page_html into variable soup.

#Step-2: Parsing page contents using html parser to understand it
from bs4 import BeautifulSoup
soup = BeautifulSoup(page_html.text, 'html.parser')

Navigating and searching required data

Wikipedia page contains two tables of which table with constituents as id contains the required data. To extract contents of html tag, Beautiful soup provides two methods i.e. find and find_all. As the name suggest find method will find only the first match however, find_all method finds all the matches. Both the method takes html tag name as an argument that needs to be searched. To narrow down the search, optional HTML/CSS parameters (like class, id, borders, etc) can be passed as a dictionary.

Code: Following code stores the content of table tag with id=’constituents’ into a variable table.

# Step-3: Extracting Table with table id = "constituents" 
table = soup.find('table', {'id': 'constituents'})

Extracting Table Data

It is necessary to understand the structure of the table before extracting its contents. Each row to table is inside ‘tr’ (table row) tag. The first row of table contains the header’s data while each of the remaining rows contains details about a S&P 500 company. Following html code shows the structure of a table.

# Structure of Table containing list of S&P 500 stocks
<tbody><tr>
<th><a href="/wiki/Ticker_symbol" title="Ticker symbol">Symbol</a>
</th>
<th>Security</th>
<th><a href="/wiki/SEC_filing" title="SEC filing">SEC filings</a></th>
<th><a href="/wiki/Global_Industry_Classification_Standard" title="Global Industry Classification Standard">GICS</a> Sector</th>
<th>GICS Sub Industry</th>
<th>Headquarters Location</th>
<th>Date first added</th>
<th><a href="/wiki/Central_Index_Key" title="Central Index Key">CIK</a></th>
<th>Founded
</th></tr>
<tr>
<td><a rel="nofollow" class="external text" href="https://www.nyse.com/quote/XNYS:MMM">MMM</a>
</td>
<td><a href="/wiki/3M" title="3M">3M Company</a></td>
<td><a rel="nofollow" class="external text" href="https://www.sec.gov/cgi-bin/browse-edgar?CIK=MMM&action=getcompany">reports</a></td>
<td>Industrials</td>
<td>Industrial Conglomerates</td>
<td><a href="/wiki/St._Paul,_Minnesota" class="mw-redirect" title="St. Paul, Minnesota">St. Paul, Minnesota</a></td>
<td>1976-08-09</td>
<td>0000066740</td>
<td>1902
</td></tr>
</table>

Extracting Rows of Table

To extract all the rows again, find_all method of BeautifulSoup is used. <tr>…</tr> tags contain data of each row of which the first row contains table header and the remaining rows contain table data.

Code: The following code extracts contents of all the ‘tr’ (table row) tag in table into variable table_rows

# Extracting all table rows
table_rows = table.find_all('tr')

Extracting Table Headers

Header of table contains, Symbol, Security, SEC filings, GICS Sector, GICS Sub Industry, Headquarters Location, Date First added. CIK and Founded. To extract these header data, use find_all method on first row.. Text attribute is used to retrieve the text content.

Code: In the following code, first row is stored in variable header_row. Then all the ‘th’ (table head) tag contents of the first row is stored in variable ths. Using for loop, contents of each table head of ths is appended into header list.

# Getting Table Header Elements from First rows
header = []
header_row = table_rows[0]
ths = header_row.find_all('th')
for th in ths:
    # th contains html 
    # for e.g. ths[0] -> <th><a href="/wiki/Ticker_symbol" title="Ticker symbol">Symbol</a>  </th> 
     # To extract contents (Ticker Symbol) use .text attribute
    header.append(th.text)

Extracting Table Data for Remaining Rows

After getting header’s data from first row, the next step is to extract details of each S&P 500 company. Each row has details about one company. The procedure is very much similar to extracting table header. It is done using nested for loops. The outer for loop navigates through each row and the inner loop navigates through each table data.

Code: Outer for loop navigates through table_rows and stores each table row in variable row. For each row, all the ‘td’ (table data) tags is extracted in variable tds. Nested for loop is used to store text of each ‘td’ into list. Contents of each in variable SP_500.

# Getting Table Data for Remaining Rows
SP_500 = []
# assigning header to data
SP_500.append(header)
for row in table_rows[1:]:
    td_contents = []
    tds = row.find_all('td')
    for td in tds:
        td_contents.append(td.text)
    SP_500.append(td_contents)

Now we have all the details of S&P 500 companies given in Wikipedia in variable SP_500.

Entire Code

from requests import get
from bs4 import BeautifulSoup

url = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
page_html = get(url) # To see page content, print(page_html.content)

#Step-2: Parsing page contents using html parser to understand it
soup = BeautifulSoup(page_html.text, 'html.parser')

# Step-3: Extracting Table with table id = "constituents" 
table = soup.find('table', {'id': 'constituents'})

# Step-4: Extracting all table rows
table_rows = table.find_all('tr')

# Step 5: Extracting Table Header Elements from First rows
header = []
header_row = table_rows[0]
ths = header_row.find_all('th')
for th in ths:
    # th contains html for e.g.
    # ths[0] -> <th><a href="/wiki/Ticker_symbol" title="Ticker symbol">Symbol</a> </th> 
     # To extract contents (Ticker Symbol) use .text attribute
    header.append(th.text)
    
# Step 6: Extracting Table with table id = "constituents" 
table = soup.find('table', {'id': 'constituents'}) 

# Step 7: Extracting Table Data for Remaining Rows
SP_500 = []
# assigning header to data
SP_500.append(header)
for row in table_rows[1:]:
    td_contents = []
    tds = row.find_all('td')
    for td in tds:
        td_contents.append(td.text)
    SP_500.append(td_contents)

Extracted Data: