01 / 10

Intro to Computational Journalism

From datasets to websites

TransformationData → HTML
df

Rows and columns inside a dataframe.

→
<html>

Web pages are structured text too.

The routine

Errors are clues

When something breaks, start with the last line, find the exact error, and change one thing at a time.

NameError KeyError SyntaxError AttributeError No module named ...
Debugging routineRead → Google → Change → Run
Read the LAST line of the traceback first.
Google the exact error.
Change one thing.
Run again.

What is Python telling us?

NameError
Did the variable or cell run?
KeyError
Does the column exist? Check df.columns.
SyntaxError
Check brackets, quotes, and commas.
AttributeError
Check spelling or whether the method exists.
No module named
Install or import the package.

Group exercise

Use a new dataset and produce one filter, one sort, and one summary statistic.

1 filter
1 sort
1 summary

Not all data arrives as a CSV.

Spreadsheet

Rows and columns in a file.

→
Dataframe

Python can work with it.

→
Webpage

Structured text in the browser.

→
HTML

The blueprint underneath.

A webpage may look visual, but underneath it is structured text.

books.toscrape.com
Book listings
https://books.toscrape.com/
A Light in the Attic
£51.77★★★★☆
In stockfiction
Tipping the Velvet
£53.74★★★☆☆
In stockfiction
Soumission
£50.10★★★★☆
In stockfiction
Sharp Objects
£47.82★★★★☆
In stockmystery

HTML is the structure behind the page

HTMLone simple page
<html> <body> <article class="product"> <h3>A Light in the Attic</h3> <p class="price">£51.77</p> </article> </body> </html>
What to noticelabels
tag
element
attribute
class
parent
child
└── webpage
└── section
└── ol.row
└── li
└── article.product_pod
├── div.image_container
├── p.star-rating
├── h3 > a[title]
└── div.product_price
├── p.price_color
└── p.instock.availability
Scraping starts by finding the repeated container, then extracting fields inside it.

Anatomy of a webpage

A book card is nested HTML.

What we see as one book card is stored as a set of nested HTML elements.

article.product_pod └── p.price_color
article.product_podone row
h3 > a[title]title column
p.price_colorprice column
p.instock.availabilityavailability column

Scraping begins by finding the repeated container, then extracting fields inside it.

If one card has a pattern, Python can find all of them.

One book cardpattern
title price availability rating

A card is easier to scrape when the pieces repeat in the same order.

Book one
£21.00★★★★☆
In stocksame pattern
Book two
£18.99★★★☆☆
In stocksame pattern
Book three
£12.50★★★★★
In stocksame pattern
Book four
£27.10★★★☆☆
In stocksame pattern

Scraping starts by noticing repetition.

Right click → Inspect

books.toscrape.com
Browser view
Page on the left. Structure on the right.
A Light in the Attic
£51.77★★★★☆
In stockbook card
Sharp Objects
£47.82★★★★☆
In stockbook card
HTML panelinspect
<article class="product_pod"> <h3>A Light in the Attic</h3> <p class="price_color">£51.77</p> </article>
headline
book card
price
class name

Next: turn HTML back into a table

HTML page

Text in the browser.

↓
Beautiful Soup

Find the pieces we care about.

↓
Rows and columns

Turn the page into data.

↓
Dataframe

Work with it like a table.

Teaserpreview only
from bs4 import BeautifulSoup