Skip to main content

Crawl Website in Python

Crawl Website in Python

This page explains how to use Python and extract (title) information from a LIVE website. The code provided is fairly simple and to use it we need to be comfortable using a terminal and have basic programming knowledge. Resources and libraries used:

  • A terminal window
  • Python3 installed and accessible via the terminal window
  • PIP, the official Python package manager
  • requests - a popular and simple HTTP library
  • Beautiful Soup - a library used to parse HTML and extract information with ease
  • 10minutes to understand and type the commands

✅ Sample Code​

Check Python is installed​

Python is installed by default in MacOS and Linux systems and should be downloaded and installed in all Windows versions. Once is properly installed, we can start the Python console by typing python in the terminal.

$ python
Python 3.8.4 (tags/v3.8.4:dfa645a, Jul 13 2020, 16:46:45) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

Install libraries​

  • Request - simple HTTP library for Python, built for human beings.
  • Beautiful Soup - Python library for pulling data out of HTML and XML files.
$ pip install requests
$ pip install BeautifulSoup4

Write Code using Python CLI​

The first step is to import the libraries used in our code:

>>> import requests                        # import the library
>>> from bs4 import BeautifulSoup as bs    # import the library

Once the libraries are imported we can use all helpers exposed. The following code snippet defines a variable that holds the website address and download the page using requests library.

>>> site = 'https://google.com'            # define the website we want to process
>>> page = requests.get( site )            # download the page

At this point, the page should be injected and used via BeautifulSoup4.

>>> soup = bs(page.content, 'html.parser') # Parse the downloaded page with BeautifulSoup
>>> soup.title                             # Print the title
<title>Google</title>

This simple tutorial should provoke curious minds to search other Python hot topics and try to code more useful things. We will provide a short-list with suggestions:

  • List all images of a web page
  • List the inner links (to other pages, the same domain)
  • List the outer links (external websites)

✅ Resources​