Comprehensive Guide On Requests Module in Python: Examples, Tips, and Best Practices

Have you ever wondered why Python is one of the most popular programming languages today? Its versatility is a major reason, as it can be used for tasks ranging from data science to web development. In fact, Python is so widely used that 51% of developers include it in their projects.

What sets Python apart in web development is its robust ecosystem of libraries, such as requests, which simplifies communication with web servers. This library allows you to interact with APIs securely, making it an essential tool for web developers.

If you're looking to use Python for web development, understanding the requests module in Python is crucial. This blog will guide you through the request library in Python and show you how to use its features to build strong and reliable web applications. Dive in!

What Is the Requests Module in Python, and Why Is It Important?

The requests module in Python is a popular HTTP library that makes working with HTTP requests more straightforward and user-friendly. It simplifies the process of sending HTTP/1.1 requests, such as GET, POST, PUT, and DELETE, without the need to manage low-level details like URL encoding or socket connections.

The main objective of the requests module is to enable interactions with web services or APIs, allowing you to retrieve and send data over the web easily. Here are some of the features of the requests module.

• Sending HTTP requests with minimal code.
• Handling URL parameters, JSON data, and form data efficiently.
• Managing HTTP headers and cookies.
• Supporting SSL/TLS for secure connections.
• Built-in handling of timeouts, sessions, and retries.

The widespread adoption of the requests module is due to its simplicity and ease of use, making it one of the most popular third-party Python libraries. Here’s why it is preferred.

• Readable Code: Its API is designed to be human-readable and Pythonic.
• Comprehensive Documentation: Extensive resources and examples make it beginner-friendly.
• Active Community: A wide user base ensures continuous updates, troubleshooting, and community support.

The requests module’s simplicity and ease of use make it a popular tool for developers building web applications, data integrations, or API consumers. Here’s why it is important.

• Easing API Integration

The requests module allows you to send HTTP requests and consume APIs with minimal effort, allowing seamless integration with services like payment gateways.

• Improve Developer Productivity

The requests module helps you focus on the application's logic instead of managing details like URL encoding, headers, or handling raw sockets.

• Reliability

The requests module handles various scenarios, like automatic handling of HTTP redirects, timeout and retry configurations, and secure connections with SSL/TLS.

• Standardized Library

The requests module is widely supported, actively maintained, and compatible with different frameworks and tools, making it an industry standard.

• Supports Modern Applications

Modern applications rely on cloud services and microservices architecture. The requests module allows you to build, test, and scale such systems with ease.

Now that you have understood the requests module in Python, let’s explore ways to install it in Python.

How Do You Install the Requests Library in Python?

You can install the requests library using pip, the Python package manager. It is compatible with Python versions 2.7, 3.6, and later.

Here are the steps to install the request library python.

1. Check if Python is installed: Make sure Python and pip are installed on your system. You can check the version of Python and pip by running the following command.

python --version
pip --version

Output:

Python 3.x.x
pip x.x.x from /path/to/pip

If Python or pip is not installed, download Python from its official website.

2. Install: Use the following command to install the requests library.

pip install requests

Output:

Collecting requests
Downloading requests-x.x.x-py2.py3-none-any.whl (xx.x kB)
Installing collected packages: requests
Successfully installed requests-x.x.x

3. Check Availability: Run the following command to verify that the requests library is available.

python -c "import requests; print(requests.__version__)"

Output:

X.x.x

If you are using a virtual environment, you can install the requests module using the following steps.

1. Virtual environment: Create a virtual environment using the following command.

python -m venv myenv

2. Start virtual environment: Activate the virtual environment by running the following commands.

• Windows

myenv\Scripts\activate

• Mac/Linux

source myenv/bin/activate

3. Setting up requests: Install requests in a virtual environment through this command.

pip install requests

4. Check installation: Verify installation using the following command.

python -c "import requests; print(requests.__version__)"

Output:

2.28.1

Also Read: How to Run a Python Project? Installation & Setup, Execution [2024]

Now that you know how to install the requests module in Python, let’s explore how to make your first HTTP Python requests.

How Can You Make Your First HTTP Request Using Python Requests?

You can make your first HTTP request using the requests library to perform a simple GET request. Here are the steps to make a GET request.

• Import the requests Library: Ensure the library is installed and import it into your Python script.
• Send a GET Request: Use the requests.get() method to send a request to a URL.
• Inspect the Response: Access the properties of the response object, such as status code, headers, and content.

Code Snippet:

import requests
# Step 1: Specify the URL
url = "https://jsonplaceholder.typicode.com/posts/1"
# Step 2: Send a GET request
response = requests.get(url)
# Step 3: Inspect the response
print("Status Code:", response.status_code)
print("Headers:", response.headers)
print("Content:", response.text)

Output:

Status Code: 200
Headers: {'Content-Type': 'application/json; charset=utf-8', ...}
Content: {
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}

The GET method in Python's requests library supports various arguments that allow you to customize the request. Here are some of the arguments, along with sample code syntax.

• url: The URL to which the GET request is sent

response = requests.get("https://example.com")

• params: It is a dictionary, list of tuples, or bytes to send as query parameters in the URL.

params = {"search": "python", "page": 2}
response = requests.get("https://example.com", params=params)

• headers: It is a dictionary of HTTP headers to include in the request

headers = {"User-Agent": "my-app/1.0", "Authorization": "Bearer token"}
response = requests.get("https://example.com", headers=headers)

• cookies: It is a dictionary of cookies to include in the request.

cookies = {"sessionid": "12345"}
response = requests.get("https://example.com", cookies=cookies)

• timeout: The maximum time (in seconds) to wait for the server's response. 

response = requests.get("https://example.com", timeout=5)

• auth: It is a tuple for basic HTTP authentication (username, password).

from requests.auth import HTTPBasicAuth
response = requests.get("https://example.com", auth=HTTPBasicAuth("user", "pass"))

• allow_redirects: It is a boolean to determine whether redirects are followed. The default is set to True.

response = requests.get("https://example.com", allow_redirects=False)

• stream: When the argument is set to True, the response content is not immediately downloaded. This is useful for large responses.

response = requests.get("https://example.com", stream=True)

• proxies: It is a dictionary that specifies proxy servers for the request.

proxies = {"http": "http://proxy.com:8080", "https": "https://proxy.com:8080"}
response = requests.get("https://example.com", proxies=proxies)

• verify: it is a boolean or path to control SSL certificate verification. The default is set to True.

response = requests.get("https://example.com", verify=False)  # Skips SSL verification (not recommended)

• cert: It is a path to a client-side SSL certificate file (PEM format)

response = requests.get("https://example.com", cert=("path/to/cert.pem", "path/to/key.pem"))

Now that you’ve explored how to make HTTP Python requests using GET, let’s check out some of the popular HTTP methods.

What Are the Common HTTP Methods and When Should You Use Them?

The HTTP methods define the action to be performed on a resource. Some common HTTP methods include GET, POST, PUT, DELETE, HEAD, and OPTIONS. Here are the ways to use these methods.

• GET: It is used to fetch data or request information without changing any resource on the server.

Example: 

response = requests.get("https://api.example.com/data")

• POST: Used when submitting data to the server, e.g., submitting forms or creating new resources.

Example: 

response = requests.post("https://api.example.com/data", json={"name": "Raj"})

• PUT: This method is used when you need to update an existing resource or create it if it does not exist.

Example: 

response = requests.put("https://api.example.com/data/1", json={"name": "Raj Updated"})

• DELETE: This method removes a resource from the server.

Example: 

response = requests.delete("https://api.example.com/data/1")

• HEAD: Used when you need to check if a resource exists or to inspect headers.

Example: 

response = requests.head("https://api.example.com/data")

• OPTIONS: Requests information to check what methods are allowed for a resource.

Example: 

response = requests.options("https://api.example.com/data")

Let’s explore how you can work with query parameters in a GET request.

How Can You Work with Query Parameters and URL Encoding?

In GET requests, query parameters are added to the URL after a question mark (?), and multiple parameters are separated by an ampersand (&). 

The requests library allows you to add these parameters using the params argument, which automatically handles URL encoding.

Here’s a code example of a GET request with query parameters.

import requests
# Define query parameters
params = {"search": "Python", "page": 2}
# Send GET request with query parameters
response = requests.get("https://api.example.com/search", params=params)
# Print URL with encoded query parameters and response
print("Request URL:", response.url)
print("Response Status Code:", response.status_code)
print("Response Text:", response.text)

Output:

Request URL: https://api.example.com/search?search=Python&page =2
Response Status Code: 200
Response Text: {"results": [{"title": "Learn Python", "author": "Kumar D"}]}

The URL encoding makes sure that special characters in parameters (e.g., spaces, ampersands, etc.) are safely transmitted over the web. The requests library automatically encodes parameters according to the need. 

Here’s a code example of URL encoding with special characters.

import requests
# Define query parameters with special characters
params = {"search": "Python programming & tutorials", "page": 2}
# Send GET request with query parameters
response = requests.get("https://api.example.com/search", params=params)
# Print the URL with encoded parameters
print("Request URL:", response.url)
print("Response Status Code:", response.status_code)
print("Response Text:", response.text)

Output:

Request URL: https://api.example.com/search?search=Python%20programming%20%26%20tutorials&page=2
Response Status Code: 200
Response Text: {"results": [{"title": "Learn Python", "author": "John Doe"}]}

In the following section, let's explore different methods for submitting data using various types of HTTP requests.

How Do You Submit Data Using POST, PUT, and DELETE Requests?

You can use POST and PUT requests to send data to the server. While POST creates new resources, PUT is used to update existing resources.

Here are code examples of using POST and PUT requests.

1. POST request to create data

import requests
# Define data to be submitted
data = {"username": "Kumar_D", "email": "kumar@yahoo.com"}
# Send POST request
response = requests.post("https://api.example.com/users", json=data)
# Output response
print("Response Status Code:", response.status_code)
print("Response JSON:", response.json())

Output:

Response Status Code: 201
Response JSON: {"id": 123, "username": "kumar_d", "email": "kumar@yahoo.com"}

2. PUT request to update data

import requests
# Define data to update
data = {"email": "kumar_new@yahoo.com"}
# Send PUT request to update user with ID 123
response = requests.put("https://api.example.com/users/123", json=data)
# Output response
print("Response Status Code:", response.status_code)
print("Response JSON:", response.json())

Output:

Response Status Code: 200
Response JSON: {"id": 123, "username": "kumar_d", "email": "kumar_new@yahoo.com"}

The DELETE request is used to remove a resource from the server. It usually does not require any data in the request body, as the resource to be deleted is specified in the URL.

Here’s an example of a DELETE request.

import requests
# Send DELETE request to remove user with ID 123
response = requests.delete("https://api.example.com/users/123")
# Output response
print("Response Status Code:", response.status_code)

Output:

Response Status Code: 204

Now that you’ve looked at the common HTTP methods. Let’s explore ways to handle responses in the requests module in Python.

upGrad’s Exclusive Data Science Webinar for you –

How upGrad helps for your Data Science Career?

How Do You Handle Responses: Status Codes, Headers, and Content?

When you make an HTTP request, the server responds with information such as the status code, headers, and content. You can access all these elements of the response using the requests library.

Here’s how you can handle status codes, headers, and content in HTTP requests.

• Status codes

You can access the status code with the status_code attribute. Here’s a sample code snippet to access the status code.

import requests
# Send a GET request
response = requests.get("https://api.example.com/data")
# Access status code
print("Status Code:", response.status_code)

Output:

Status Code: 200

• Response Headers

The response header contains data about the response, such as content type, server information, and caching directives. You can use the headers attribute to access the headers. 

Here’s a sample code snippet to access headers.

import requests
# Send a GET request
response = requests.get("https://api.example.com/data")
# Access headers
print("Headers:", response.headers)

Output:

Headers: {'Content-Type': 'application/json', 'Server': 'nginx', 'Cache-Control': 'no-cache'}

• Response

The response content comes in formats such as JSON, plain text, or HTML. Depending on the Content-Type header, you need to handle the content. 

Here’s a code snippet to handle JSON content.

import requests
# Send a GET request
response = requests.get("https://api.example.com/data")
# Check if the response is JSON
if response.status_code == 200:
    data = response.json()  # Parse JSON content
    print("JSON Data:", data)

Output:

JSON Data: {"results": [{"name": "Kumar D", "age": 30}]}

Also Read: How to Open JSON File?

In the following section, let’s check out ways to handle timeouts and errors in requests.

How Can You Handle Timeouts and Errors in Requests?

You need to set a timeout to prevent your program from hanging indefinitely if the server is unresponsive. The requests library allows you to set a timeout for the request. 

Here’s a code snippet to set timeout.

import requests
try:
    # Send a GET request with a timeout of 3 seconds
    response = requests.get("https://api.example.com/data", timeout=3)
    print("Response Status Code:", response.status_code)
except requests.Timeout:
    print("The request timed out.")

Output:

If the server takes more than 3 seconds to respond, you’ll get the following output.

The request timed out.

You can experience common exceptions while making HTTP requests. You need to handle these exceptions to make your code more robust. 

Here’s a code snippet to handle multiple exceptions.

import requests

try:
    # Send a GET request with a timeout
    response = requests.get("https://api.example.com/data", timeout=3)
    
    # Raise an error for HTTP status codes 4xx or 5xx
    response.raise_for_status()  
    
    print("Response Status Code:", response.status_code)
    print("Response Content:", response.text)

except requests.Timeout:
    print("The request timed out.")
except requests.ConnectionError:
    print("A network error occurred.")
except requests.HTTPError as http_err:
    print(f"HTTP error occurred: {http_err}")
except requests.RequestException as err:
    print(f"An error occurred: {err}")

Output for timeout error: 

The request timed out.

Output for connection error:

A network error occurred.

Output for HTTP error:

HTTP error occurred: 404 Client Error: Not Found for url: https://api.example.com/data

Now that you’ve explored ways to handle responses in the HTTP requests module in Python, let’s look at some of its advanced features.

What Advanced Features Does the Requests Module Offer?

The requests module offers several features that can help you handle complex use cases, such as basic authentication, proxies, and SSL certificate verification.

Here’s how you can use these features.

• Basic Authentication

Basic authentication allows you to send a username and password in the HTTP request headers to authenticate the user. You can use this feature with the requests library by using the auth parameter.

Here’s the code snippet for using basic authentication.

import requests
from requests.auth import HTTPBasicAuth
# Send GET request with basic authentication
response = requests.get(
    "https://api.example.com/data", 
    auth=HTTPBasicAuth('username', 'password')
)
# Print the status code and response content
print("Status Code:", response.status_code)
print("Response Content:", response.text)

Output:

Status Code: 200
Response Content: {"data": "Here is the protected data"}

• Setting up Proxies

You can route requests through a proxy server by using the proxies parameter, which the requests library offers.

Here’s a code snippet for using proxies.

import requests

# Define proxy settings
proxies = {
    "http": "http://proxy.example.com:8080",
    "https": "https://proxy.example.com:8080"
}

# Send GET request through the proxy
response = requests.get("https://api.example.com/data", proxies=proxies)

# Print the status code and response content
print("Status Code:", response.status_code)
print("Response Content:", response.text)

Output:

Status Code: 200
Response Content: {"data": "Fetched via proxy"}

• Managing SSL Certificate Verification

The requests verifies SSL certificates by default. If you want to disable SSL certificate verification for certain requests, you can disable using the verify parameter.

Here’s a code snippet to disable SSL certificate verification.

import requests
# Send GET request without verifying SSL certificates
response = requests.get("https://self-signed.badssl.com/", verify=False)
# Print the status code and response content
print("Status Code:", response.status_code)
print("Response Content:", response.text)

Output:

Status Code: 200
Response Content: (Content of the page)

In the following section, you can check out ways to manage cookies and sessions.

What Are Sessions and Cookies, and How Do You Manage Them?

A session allows you to use parameters like cookies, headers, and authentication across multiple requests. This is particularly useful when you want to make a series of requests to the same server without re-sending the same information each time.

Here’s how you can use session to maintain persistent parameters.

import requests
# Create a session
session = requests.Session()
# Set a session header (this header will be sent with all future requests in this session)
session.headers.update({"User-Agent": "my-app"})
# Send a GET request
response = session.get("https://httpbin.org/get")
# Print session cookies and response data
print("Cookies:", session.cookies)
print("Response Content:", response.text)

Output:

Cookies: <RequestsCookieJar[<Cookie sessionid=12345678 for httpbin.org/>]>
Response Content: {"args": {}, "headers": {"User-Agent": "my-app", ...}, "origin": "IP_ADDRESS", "url": "https://httpbin.org/get"}

When you send a request to a server that returns cookies, the requests session will automatically handle those cookies, storing them and returning them in subsequent requests.

Here’s how you can manage cookies within a session.

import requests
# Create a session
session = requests.Session()
# Send a request to set cookies
response = session.get("https://httpbin.org/cookies/set?name=KumarD")
# Print the cookies set by the server
print("Cookies Set by Server:", session.cookies)
# Send another request to check if the cookie is sent back
response = session.get("https://httpbin.org/cookies")
print("Cookies Sent in Subsequent Request:", response.text)

Output:

Cookies Set by Server: <RequestsCookieJar[<Cookie name=KumarD for httpbin.org/>]>
Cookies Sent in Subsequent Request: {"cookies": {"name": "KumarD"}}

Now that you’ve understood the advanced features of the requests module in Python, let’s explore the best practices for using it effectively.

What Are the Best Practices for Using the Requests Module Effectively?

The requests module is a powerful and user-friendly tool for making HTTP requests in Python. However, to make it efficient, secure, and maintainable, you need to follow best practices.

Here are some of the best practices for using the requests module in Python.

Now that you’ve explored some of the best practices while dealing with the requests module in Python, let's now focus on ways to enhance your Python skills further.

Enhance Your Python Skills with upGrad's Courses

The Python programming language has diverse applications spanning various domains such as Artificial Intelligence (AI), Machine Learning (ML), data analytics, and web application development.

To implement Python in various applications, it's crucial to have a strong foundation in Python programming. This can be achieved through various means, including online courses.

upGrad offers comprehensive courses that provide foundational knowledge of Python and equip you with practical skills to succeed in a wide range of fields.

Here are some of the courses offered by upGrad related to Python.

• Learn Basic Python Programming
• Learn Python Libraries: NumPy, Matplotlib & Pandas
• Case Study using Tableau, Python and SQL

Similar Read:

• Use Axios NPM to Generate HTTP Requests [Step-by-Step]
• Introduction to Package JSON Scripts in Node.js: A Guide for Optimizing Your Development Workflow

Unlock the power of data with our popular Data Science courses, designed to make you proficient in analytics, machine learning, and big data!

Explore our Popular Data Science Courses

Elevate your career by learning essential Data Science skills such as statistical modeling, big data processing, predictive analytics, and SQL!

Top Data Science Skills to Learn

Stay informed and inspired  with our popular Data Science articles, offering expert insights, trends, and practical tips for aspiring data professionals!

Read our popular Data Science Articles

References:
https://www.statista.com/statistics/793628/worldwide-developer-survey-most-used-languages/
https://flatironschool.com/blog/python-popularity-the-rise-of-a-global-programming-language/