How can a Data Scientist Easily Use ScRapy on Python Notebook

Introduction

Web-Scraping is one of the easiest and cheapest ways to gain access to a large amount of data available on the internet. We can easily build structured datasets and that data can be further used for Quantitative Analysis, Forecasting, Sentiment Analysis, etc. The best method to download the data of a website is by using its public data API(fastest and reliable), but not all websites provide APIs. Sometimes the APIs are not updated regularly and we may miss out on important data.

Hence we can use tools like Scrapy or Selenium for web-crawling and scraping as an alternative. Scrapy is a free and open-source web-crawling framework written in Python. The most common way of using scrapy is on Python terminal and there are many articles that can guide you through the process.

Although the above process is very popular among python developers it is not very intuitive to a data scientist. There’s an easier but unpopular way to use scrapy i.e. on Jupyter notebook. As we know Python notebooks are fairly new and mostly used for data analysis purposes, creating scrapy functions on the same tool is relatively easy and straightforward.

Basics of HTML tags

Installing Scrapy on Python Notebook

The following block of code installs and import the necessary packages needed to get started with scrapy on python notebook:

!pip install scrapy

import scrapy

from scrapy.crawler import CrawlerRunner

!pip install crochet

from crochet import setup

setup()

import requests

from scrapy.http import TextResponse

Crawler Runner will be used to run the spider we create. TextResponse works as a scrapy shell which can be used to scrape one URL and investigate HTML tags for data extraction from the web-page. We can later create a spider to automate the whole process and scrape data up-to n number of pages.

Crochet is set up to handle the ReactorNotRestartable error. Let us now see how we can actually extract data from a web-page using CSS selector and X-path. We will scrape yahoo news site with the search string as tesla in this as an example. We will scrape the web-page to get the title of the articles. 

Inspect element to check HTML tag

Right clicking on the first link and selecting the inspect element will give us the above result. We can see that the title is part of <a> class. We will use this tag and try to extract the title information on python. The code below will load the Yahoo News website in a python object. 

r=requests.get(“https://news.search.yahoo.com/search;_ylt=AwrXnCKM_wFfoTAA8HbQtDMD;_ylu=

X3oDMTEza3NiY3RnBGNvbG8DZ3ExBHBvcwMxBHZ0aWQDBHNlYwNwYWdpbmF0aW9u?p=tesla&nojs=1&ei=UTF-8&b=01&pz=10&bct=0&xargs=0”)

response = TextResponse(r.url, body=r.text, encoding=’utf-8′)

The response variable stores the web-page in html format. Let us try to extract information using <a> tag. The code line below uses CSS extractor that is using the <a> tag to extract titles from the webpage.

response.css(‘a’).extract()

Output of CSS selector on response variable

As we can see that there are more than just article details under <a> tag. Hence <a> tag is not properly able to extract the titles. The more intuitive and precise way to obtain specific tags is by using selector gadget. After installing the selector gadget on chrome, we can use it to find the tags for any specific parts of the webpage that we want to extract. Below we can see the tag suggested by the Selector Gadget:

HTML tag suggested by Selector Gadget

Let us try to extract information using ‘#web a’ selector. The code below will extract the title information using the CSS extractor. 

response.css(‘#web a’).extract()

Output after using ‘#web a’ tag

After using the tag suggested by Gadget Selector we get more concise results using this tag. But still, we just need to extract the title of the article and leave other information apart. We can add X-path selector on this and do the same. The code below will extract the title text from the CSS tags:

response.css(‘#web a’).xpath(‘@title’).extract()

List of all the titles

As we can see we are successfully able to extract the titles of the articles from the Yahoo News website. Although it is a little bit of trial and error procedure, we can definitely understand the process-flow to boil down to correct tags and xpath address required to extract any specific information from a webpage. 

We can similarly extract the link, description, date, and publisher by using the inspect element and selector gadget. It is an iterative process and may take some time to obtain desirable results. We can use this code snippet to extract the data from the yahoo news web-page.
upGrad’s Exclusive Data Science Webinar for you –

ODE Thought Leadership Presentation

title = response.css(‘#web a’).xpath(“@title”).extract()

media = response.css(‘.cite-co::text’).extract()

desc = response.css(‘p.s-desc’).extract()

date = response.css(‘#web .mr-8::text’).extract()

link = response.css(‘h4.s-title a’).xpath(“@href”).extract()

Read: Career in Data Science

Building the spider

We now know how to use tags to extract specific bits and pieces of information using the response variable and css extractor. We now have to string this together with Scrapy Spider. Scrapy Spiders are classes with predefined structure which are built to crawl and scrape information from the websites. There are many things which can be controlled by the spiders:

1. Data to be extracted from response variables.
2. Structuring and Returning the data.
3. Cleaning data if there are unwanted pieces of information in extracted data.
4. Option of scraping websites until a certain page number.

Hence the spider essentially is the heart of building a web scraping function using Scrapy. Let us take a closer look at every part of the spider. The part of data to be extracted is already covered above where we were using response variables to extract specific data from the webpage.

Our learners also read: Free Python Course with Certification

Cleaning data

The code block below will clean the description data.
TAG_RE = re.compile(r'<[^>]+>’)  # removing html tags
j = 0
#cleaning Description string
for i in desc:
   desc[j] = TAG_RE.sub(”, i)
   j = j + 1

This code uses regular expression and removes unwanted html tags from the description of the articles. 

Before:

<p class=”s-desc”>In a recent interview on Motley Fool Live, Motley Fool co-founder and CEO

Tom Gardner recalled meeting Kendal Musk — the brother of <b>Tesla</b> (NASDAQ: TSLA)… </p>

After Regex:

In a recent interview on Motley Fool Live, Motley Fool co-founder and CEO Tom Gardner recalled meeting Kendal Musk — the brother of Tesla (NASDAQ: TSLA)..

We can now see that the extra html tags are removed from the article description.

Structuring and Returning data

# Give the extracted content row wise

for item in zip(title, media, link, desc, date):

   # create a dictionary to store the scraped info

   scraped_info = {

   ‘Title’: item[0],

   ‘Media’: item[1],

   ‘Link’ : item[2],

   ‘Description’ : item[3],

   ‘Date’ : item[4],

   ‘Search_string’ : searchstr,

   ‘Source’: “Yahoo News”,

   }

# yield or give the scraped info to scrapy

yield scraped_info

The data extracted from the web pages are stored in different list variables. We are creating a dictionary which is then yielded back outside the spider. Yield function is a function of any spider class and is used to return data back to the function. This data can then be saved as json, csv or different kinds of dataset.

Must Read: Data Scientist Salary in India

Navigating to n number of pages

The code block below uses the link when one tries to go to the next page of results on Yahoo news. The page number variable is iterated until the page_limit reaches and for every page number, a unique link is created as the variable is used in the link string. The following command then follows to the new page and yields the information from that page.

All in all we are able to navigate the pages using a base hyperlink and edit information like the page number and keyword. It may be a little complicated to extract base hyperlink for some pages but it generally requires understanding the pattern on how the link is updating on next pages.

pagenumber = 1       #initiating page number

page_limit = 5   #Pages to be scraped

#follow on page url initialisation  next_page=”https://news.search.yahoo.com/search;_ylt=AwrXnCKM_wFfoTAA8HbQtDMD;_ylu=

X3oDMTEza3NiY3RnBGNvbG8DZ3ExBHBvcwMxBHZ0aWQDBHNlYwNwYWdpbmF0aW9u?p=”+MySpider.keyword+”&nojs=1&ei=UTF-8&b=”+str(MySpider.pagenumber)+”1&pz=10&bct=0&xargs=0″

if(MySpider.pagenumber < MySpider.res_limit):

MySpider.pagenumber += 1

      yield response.follow(next_page,callback=self.parse) 

#navigation to next page 

Final Function

The code block below is the final function when called will export the data of the search string in .csv format. 

def yahoo_news(searchstr,lim):

  TAG_RE = re.compile(r‘<[^>]+>’)  # removing html tags

  class MySpider(scrapy.Spider):
      cnt = 1
      name = ‘yahoonews’   #name of spider
      pagenumber = 1       #initiating page number
      res_limit = lim   #Pages to be scraped
      keyword = searchstr          #Search string
      start_urls = [“https://news.search.yahoo.com/search;_ylt=AwrXnCKM_wFfoTAA8HbQtDMD;_ylu=X3oDMTEza3NiY3RnBGNvbG8DZ3ExBHBvcwMxBHZ0aWQDBHNlYwNwYWdpbmF0aW9u?p=
“+keyword+“&nojs=1&ei=UTF-8&b=01&pz=10&bct=0&xargs=0”]
      def parse(self, response):
          #Data to be extracted from HTML
          title = response.css(‘#web a’).xpath(“@title”).extract() 
          media = response.css(‘.cite-co::text’).extract()
          desc = response.css(‘p.s-desc’).extract()
          date = response.css(‘#web .mr-8::text’).extract()
          link = response.css(‘h4.s-title a’).xpath(“@href”).extract()
          j = 0
          #cleaning Description string
          for i in desc:
              desc[j] = TAG_RE.sub(”, i)
              j = j + 1
          # Give the extracted content row wise
          for item in zip(title, media, link, desc, date):
            # create a dictionary to store the scraped info
            scraped_info = {
                ‘Title’: item[0],
                ‘Media’: item[1],
                ‘Link’ : item[2],
                ‘Description’ : item[3],
                ‘Date’ : item[4],
                ‘Search_string’ : searchstr,
                ‘Source’: “Yahoo News”,
            }
            # yield or give the scraped info to scrapy
            yield scraped_info
          #follow on page url initialisation
          next_page=“https://news.search.yahoo.com/search;_ylt=AwrXnCKM_wFfoTAA8HbQtDMD;_ylu=X3oDMTEza3NiY3RnBGNvbG”\
                    “8DZ3ExBHBvcwMxBHZ0aWQDBHNlYwNwYWdpbmF0aW9u?p=”+MySpider.keyword+“&nojs=1&ei=”\
                    “UTF-8&b=”+str(MySpider.pagenumber)+“1&pz=10&bct=0&xargs=0”
                   if(MySpider.pagenumber < MySpider.res_limit):
              MySpider.pagenumber += 1
              yield response.follow(next_page,callback=self.parse) #navigation to next page 
  if __name__ == “__main__”:   
    #initiate process of crawling
    runner = CrawlerRunner(settings={
      “FEEDS”: {“yahoo_output.csv”: {“format”: “csv”}, #set output in settings
      },
    })

    d=runner.crawl(MySpider) # the script will block here until the crawling is finished

We can spot different blocks of codes which are integrated to make the spider. The main part is actually used to kick-start the spider and configure the output format. There are various different settings which can be edited regarding the crawler. One can use middlewares or proxies for hopping. 

runner.crawl(spider) will execute the whole process and we’ll get the output as a comma separated file.

Output 

Conclusion

We can now use Scrapy as an api on Python notebooks and create spiders to crawl and scrape different websites. We also looked at how different tags and X-path can be used to access information from a webpage. 

Learn data science courses from the World’s top Universities. Earn Executive PG Programs, Advanced Certificate Programs, or Masters Programs to fast-track your career.