How To Scrape Multiple Pages on a Website (BeautifulSoup Pagination)

🧠 Don’t miss out! Get FREE access to my Skool community — packed with resources, tools, and support to help you with Data, Machine Learning, and AI Automations! 📈 https://www.skool.com/data-and-ai-aut...

Learn how to handle pagination with BeautifulSoup in this Python web scraping tutorial! 🚀
In this video, we’ll show you how to extract data from websites with multiple pages by automating the pagination process. Whether you're scraping product listings, articles, or any paginated content, this guide will help you scrape data efficiently and responsibly.

Code: https://ryanandmattdatascience.com/be...

🚀 Hire me for Data Work: https://ryanandmattdatascience.com/da...
👨‍💻 Mentorships: https://ryanandmattdatascience.com/me...
📧 Email: ryannolandata@gmail.com
🌐 Website & Blog: https://ryanandmattdatascience.com/
🖥️ Discord:   / discord  
📚 *Practice SQL & Python Interview Questions: https://stratascratch.com/?via=ryan
📖 *SQL and Python Courses: https://datacamp.pxf.io/XYD7Qg

🍿 WATCH NEXT
Python Web Scraping Playlist:    • Python Website Scraping  


Learn how to scrape multiple pages on websites using Python in this comprehensive web scraping pagination tutorial. In this video, I walk you through scraping paginated websites, setting up timers to avoid detection, and building both basic and random timers for ethical web scraping. We cover two real examples: Books to Scrape and a hockey stats website with multiple pages of data.

I explain how to identify URL patterns across pages, build for loops and while loops to navigate pagination, and use Python libraries like BeautifulSoup, Requests, and Pandas. You'll learn how to extract data from tables, format URLs dynamically, and create functions to make your web scraping code reusable. We also cover important concepts like handling 404 errors, setting delays between requests, and building random timers using Python's random module to make your scraper less detectable.

By the end of this tutorial, you'll know exactly how to scrape websites with pagination, whether they have 10 pages or 1,000 pages. All code examples are available on the website linked in the description, and make sure to check out the full Python web scraping series for more tutorials on extracting data from the web.

TIMESTAMPS
00:00 Introduction to Web Scraping Pagination
00:46 Understanding Website Pagination Patterns
02:30 Introduction to Timers for Web Scraping
03:50 Setting Up a Basic Timer
05:14 Setting Up a Random Timer
06:56 Using Strings to Format URLs
08:16 Example 1: Books to Scrape Website
13:17 Example 2: Hockey Stats Website Setup
15:17 Building the Scraping Function
18:25 Adding Random Timer to Function
19:41 Debugging and Testing the Code
21:00 Filtering and Exporting Data to CSV
22:07 Recap: Understanding URL Patterns
23:18 Final Tips and Best Practices

OTHER SOCIALS:
Ryan’s LinkedIn:   / ryan-p-nolan  
Matt’s LinkedIn:   / matt-payne-ceo  
Twitter/X: https://x.com/RyanMattDS

Who is Ryan
Ryan is a Data Scientist at a fintech company, where he focuses on fraud prevention in underwriting and risk. Before that, he worked as a Data Analyst at a tax software company. He holds a degree in Electrical Engineering from UCF.

Who is Matt
Matt is the founder of Width.ai, an AI and Machine Learning agency. Before starting his own company, he was a Machine Learning Engineer at Capital One.

*This is an affiliate program. We receive a small portion of the final sale at no extra cost to you.