Crawl Website in Python

This page explains how to use Python and extract (title) information from a LIVE website. The code provided is fairly simple and to use it we need to be comfortable using a terminal and have basic programming knowledge. Resources and libraries used:
A terminal window
Python3 installed and accessible via the terminal window
PIP, the official Python package manager
requests - a popular and simple HTTP library
Beautiful Soup - a library used to parse HTML and extract information with ease
10minutes to understand and type the commands 
​
Let's start writing code.
Check Python is installed
Python is installed by default in MacOS and Linux systems and should be downloaded and installed in all Windows versions. Once is properly installed, we can start the Python console by typing python in the terminal.
1
$ python
2
Python 3.8.4 (tags/v3.8.4:dfa645a, Jul 13 2020, 16:46:45) [MSC v.1924 64 bit (AMD64)] on win32
3
Type "help", "copyright", "credits" or "license" for more information.
4
>>>
Copied!
Install libraries
​Request - simple HTTP library for Python, built for human beings.
​Beautiful Soup - Python library for pulling data out of HTML and XML files.
1
$ pip install requests
2
$ pip install BeautifulSoup4
Copied!
Write code in Python Console
The first step is to import the libraries used in our code:
1
>>> import requests                        # import the library
2
>>> from bs4 import BeautifulSoup as bs    # import the library
Copied!
Once the libraries are imported we can use all helpers exposed. The following code snippet defines a variable that holds the website address and download the page using requests library.
1
>>> site = 'https://google.com'            # define the website we want to process
2
>>> page = requests.get( site )            # download the page
Copied!
At this point, the page should be injected and used via BeautifulSoup4.
1
>>> soup = bs(page.content, 'html.parser') # Parse the downloaded page with BeautifulSoup
2
>>> soup.title                             # Print the title   
3
<title>Google</title>
Copied!
This simple tutorial should provoke curious minds to search other Python hot topics and try to code more useful things. We will provide a short-list with suggestions:
List all images of a web page
List the inner links (to other pages, the same domain)
List the outer links (external websites)
​
Links & Resources
​Python - the official website
​Python Cheatsheet - this site should make you curious
Join AppSeed - For support and production-ready starters 

Content - Previous

How To

Setup CentOS for coding

Last modified 1yr ago

Copy link

Contents

Let's start writing code.