How can a Data Scientist Easily Use ScRapy on Python Notebook
Updated on Nov 30, 2022 | 10 min read | 6.4k views
Share:
For working professionals
For fresh graduates
More
Updated on Nov 30, 2022 | 10 min read | 6.4k views
Share:
Table of Contents
Web-Scraping is one of the easiest and cheapest ways to gain access to a large amount of data available on the internet. We can easily build structured datasets and that data can be further used for Quantitative Analysis, Forecasting, Sentiment Analysis, etc. The best method to download the data of a website is by using its public data API(fastest and reliable), but not all websites provide APIs. Sometimes the APIs are not updated regularly and we may miss out on important data.
Hence we can use tools like Scrapy or Selenium for web-crawling and scraping as an alternative. Scrapy is a free and open-source web-crawling framework written in Python. The most common way of using scrapy is on Python terminal and there are many articles that can guide you through the process.
Although the above process is very popular among python developers it is not very intuitive to a data scientist. There’s an easier but unpopular way to use scrapy i.e. on Jupyter notebook. As we know Python notebooks are fairly new and mostly used for data analysis purposes, creating scrapy functions on the same tool is relatively easy and straightforward.
The following block of code installs and import the necessary packages needed to get started with scrapy on python notebook:
!pip install scrapy
import scrapy
from scrapy.crawler import CrawlerRunner
!pip install crochet
from crochet import setup
setup()
import requests
from scrapy.http import TextResponse
Crawler Runner will be used to run the spider we create. TextResponse works as a scrapy shell which can be used to scrape one URL and investigate HTML tags for data extraction from the web-page. We can later create a spider to automate the whole process and scrape data up-to n number of pages.
Crochet is set up to handle the ReactorNotRestartable error. Let us now see how we can actually extract data from a web-page using CSS selector and X-path. We will scrape yahoo news site with the search string as tesla in this as an example. We will scrape the web-page to get the title of the articles.
Inspect element to check HTML tag
Right clicking on the first link and selecting the inspect element will give us the above result. We can see that the title is part of <a> class. We will use this tag and try to extract the title information on python. The code below will load the Yahoo News website in a python object.
r=requests.get(“https://news.search.yahoo.com/search;_ylt=AwrXnCKM_wFfoTAA8HbQtDMD;_ylu=
X3oDMTEza3NiY3RnBGNvbG8DZ3ExBHBvcwMxBHZ0aWQDBHNlYwNwYWdpbmF0aW9u?p=tesla&nojs=1&ei=UTF-8&b=01&pz=10&bct=0&xargs=0”)
response = TextResponse(r.url, body=r.text, encoding=’utf-8′)
The response variable stores the web-page in html format. Let us try to extract information using <a> tag. The code line below uses CSS extractor that is using the <a> tag to extract titles from the webpage.
response.css(‘a’).extract()
Output of CSS selector on response variable
As we can see that there are more than just article details under <a> tag. Hence <a> tag is not properly able to extract the titles. The more intuitive and precise way to obtain specific tags is by using selector gadget. After installing the selector gadget on chrome, we can use it to find the tags for any specific parts of the webpage that we want to extract. Below we can see the tag suggested by the Selector Gadget:
HTML tag suggested by Selector Gadget
Let us try to extract information using ‘#web a’ selector. The code below will extract the title information using the CSS extractor.
response.css(‘#web a’).extract()
Output after using ‘#web a’ tag
After using the tag suggested by Gadget Selector we get more concise results using this tag. But still, we just need to extract the title of the article and leave other information apart. We can add X-path selector on this and do the same. The code below will extract the title text from the CSS tags:
response.css(‘#web a’).xpath(‘@title’).extract()
List of all the titles
As we can see we are successfully able to extract the titles of the articles from the Yahoo News website. Although it is a little bit of trial and error procedure, we can definitely understand the process-flow to boil down to correct tags and xpath address required to extract any specific information from a webpage.
We can similarly extract the link, description, date, and publisher by using the inspect element and selector gadget. It is an iterative process and may take some time to obtain desirable results. We can use this code snippet to extract the data from the yahoo news web-page.
upGrad’s Exclusive Data Science Webinar for you –
ODE Thought Leadership Presentation
title = response.css(‘#web a’).xpath(“@title”).extract()
media = response.css(‘.cite-co::text’).extract()
desc = response.css(‘p.s-desc’).extract()
date = response.css(‘#web .mr-8::text’).extract()
link = response.css(‘h4.s-title a’).xpath(“@href”).extract()
Read: Career in Data Science
We now know how to use tags to extract specific bits and pieces of information using the response variable and css extractor. We now have to string this together with Scrapy Spider. Scrapy Spiders are classes with predefined structure which are built to crawl and scrape information from the websites. There are many things which can be controlled by the spiders:
Hence the spider essentially is the heart of building a web scraping function using Scrapy. Let us take a closer look at every part of the spider. The part of data to be extracted is already covered above where we were using response variables to extract specific data from the webpage.
Our learners also read: Free Python Course with Certification
The code block below will clean the description data.
TAG_RE = re.compile(r'<[^>]+>’) # removing html tags
j = 0
#cleaning Description string
for i in desc:
desc[j] = TAG_RE.sub(”, i)
j = j + 1
This code uses regular expression and removes unwanted html tags from the description of the articles.
Before:
<p class=”s-desc”>In a recent interview on Motley Fool Live, Motley Fool co-founder and CEO
Tom Gardner recalled meeting Kendal Musk — the brother of <b>Tesla</b> (NASDAQ: TSLA)… </p>
After Regex:
In a recent interview on Motley Fool Live, Motley Fool co-founder and CEO Tom Gardner recalled meeting Kendal Musk — the brother of Tesla (NASDAQ: TSLA)..
We can now see that the extra html tags are removed from the article description.
# Give the extracted content row wise
for item in zip(title, media, link, desc, date):
# create a dictionary to store the scraped info
scraped_info = {
‘Title’: item[0],
‘Media’: item[1],
‘Link’ : item[2],
‘Description’ : item[3],
‘Date’ : item[4],
‘Search_string’ : searchstr,
‘Source’: “Yahoo News”,
}
# yield or give the scraped info to scrapy
yield scraped_info
The data extracted from the web pages are stored in different list variables. We are creating a dictionary which is then yielded back outside the spider. Yield function is a function of any spider class and is used to return data back to the function. This data can then be saved as json, csv or different kinds of dataset.
Must Read: Data Scientist Salary in India
The code block below uses the link when one tries to go to the next page of results on Yahoo news. The page number variable is iterated until the page_limit reaches and for every page number, a unique link is created as the variable is used in the link string. The following command then follows to the new page and yields the information from that page.
All in all we are able to navigate the pages using a base hyperlink and edit information like the page number and keyword. It may be a little complicated to extract base hyperlink for some pages but it generally requires understanding the pattern on how the link is updating on next pages.
pagenumber = 1 #initiating page number
page_limit = 5 #Pages to be scraped
#follow on page url initialisation next_page=”https://news.search.yahoo.com/search;_ylt=AwrXnCKM_wFfoTAA8HbQtDMD;_ylu=
X3oDMTEza3NiY3RnBGNvbG8DZ3ExBHBvcwMxBHZ0aWQDBHNlYwNwYWdpbmF0aW9u?p=”+MySpider.keyword+”&nojs=1&ei=UTF-8&b=”+str(MySpider.pagenumber)+”1&pz=10&bct=0&xargs=0″
if(MySpider.pagenumber < MySpider.res_limit):
MySpider.pagenumber += 1
yield response.follow(next_page,callback=self.parse)
#navigation to next page
The code block below is the final function when called will export the data of the search string in .csv format.
def yahoo_news(searchstr,lim):
TAG_RE = re.compile(r‘<[^>]+>’) # removing html tags
class MySpider(scrapy.Spider):
cnt = 1
name = ‘yahoonews’ #name of spider
pagenumber = 1 #initiating page number
res_limit = lim #Pages to be scraped
keyword = searchstr #Search string
start_urls = [“https://news.search.yahoo.com/search;_ylt=AwrXnCKM_wFfoTAA8HbQtDMD;_ylu=X3oDMTEza3NiY3RnBGNvbG8DZ3ExBHBvcwMxBHZ0aWQDBHNlYwNwYWdpbmF0aW9u?p=
“+keyword+“&nojs=1&ei=UTF-8&b=01&pz=10&bct=0&xargs=0”]
def parse(self, response):
#Data to be extracted from HTML
title = response.css(‘#web a’).xpath(“@title”).extract()
media = response.css(‘.cite-co::text’).extract()
desc = response.css(‘p.s-desc’).extract()
date = response.css(‘#web .mr-8::text’).extract()
link = response.css(‘h4.s-title a’).xpath(“@href”).extract()
j = 0
#cleaning Description string
for i in desc:
desc[j] = TAG_RE.sub(”, i)
j = j + 1
# Give the extracted content row wise
for item in zip(title, media, link, desc, date):
# create a dictionary to store the scraped info
scraped_info = {
‘Title’: item[0],
‘Media’: item[1],
‘Link’ : item[2],
‘Description’ : item[3],
‘Date’ : item[4],
‘Search_string’ : searchstr,
‘Source’: “Yahoo News”,
}
# yield or give the scraped info to scrapy
yield scraped_info
#follow on page url initialisation
next_page=“https://news.search.yahoo.com/search;_ylt=AwrXnCKM_wFfoTAA8HbQtDMD;_ylu=X3oDMTEza3NiY3RnBGNvbG”\
“8DZ3ExBHBvcwMxBHZ0aWQDBHNlYwNwYWdpbmF0aW9u?p=”+MySpider.keyword+“&nojs=1&ei=”\
“UTF-8&b=”+str(MySpider.pagenumber)+“1&pz=10&bct=0&xargs=0”
if(MySpider.pagenumber < MySpider.res_limit):
MySpider.pagenumber += 1
yield response.follow(next_page,callback=self.parse) #navigation to next page
if __name__ == “__main__”:
#initiate process of crawling
runner = CrawlerRunner(settings={
“FEEDS”: {“yahoo_output.csv”: {“format”: “csv”}, #set output in settings
},
})
d=runner.crawl(MySpider) # the script will block here until the crawling is finished
We can spot different blocks of codes which are integrated to make the spider. The main part is actually used to kick-start the spider and configure the output format. There are various different settings which can be edited regarding the crawler. One can use middlewares or proxies for hopping.
runner.crawl(spider) will execute the whole process and we’ll get the output as a comma separated file.
Output
We can now use Scrapy as an api on Python notebooks and create spiders to crawl and scrape different websites. We also looked at how different tags and X-path can be used to access information from a webpage.
Learn data science courses from the World’s top Universities. Earn Executive PG Programs, Advanced Certificate Programs, or Masters Programs to fast-track your career.
Get Free Consultation
By submitting, I accept the T&C and
Privacy Policy
Start Your Career in Data Science Today
Top Resources