Python and Requests-HTML - Web Scraping Dynamic Content from JavaScript applications

In this video, we'll learn how to scrape content that is NOT present in initial page loads, but instead is loaded dynamically by JavaScript.

This is a common problem with scraping the modern web: the initial response contains minimal HTML and a SPA-based JavaScript app (React, Vue, Angular, etc). The data that we want to scrape on the page is therefore not present, but is rendered later via API calls from the SPA application.

We will look at how we can use requests-html to solve this issue in Python when scraping such sites. We'll also look at using this with BeautifulSoup in order to find data on the page.

This video makes use of the following sample website (a React application):
https://react-amazon-bestsellers-book...

📌 𝗖𝗵𝗮𝗽𝘁𝗲𝗿𝘀:
00:00 Intro
02:15 Sending GET request using Python requests library
04:00 Finding objects with BeautifulSoup
05:15 Installing requests-html
06:38 Executing JavaScript on page using requests-html

☕️ 𝗕𝘂𝘆 𝗺𝗲 𝗮 𝗰𝗼𝗳𝗳𝗲𝗲:
To support the channel and encourage new videos, please consider buying me a coffee here:
https://ko-fi.com/bugbytes

𝗦𝗼𝗰𝗶𝗮𝗹 𝗠𝗲𝗱𝗶𝗮:
📖 Blog: https://bugbytes.io/posts/
👾 Github: https://github.com/bugbytes-io/
🐦 Twitter:   / bugbytesio  

📚 𝗙𝘂𝗿𝘁𝗵𝗲𝗿 𝗿𝗲𝗮𝗱𝗶𝗻𝗴 𝗮𝗻𝗱 𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻:
requests-html: https://pypi.org/project/requests-html/
BeautifulSoup: https://beautiful-soup-4.readthedocs....
requests: https://pypi.org/project/requests/
Sample website: https://react-amazon-bestsellers-book...

#python #webscraping #datascience