Python Web Scraping

• Zac Clancy. What is Web Scraping? kite, 4/8/19.
• Danshu Jain. How to Scrape Websites Using Python. freecodecamp, 4/8/19.
• Colin OKeefe. Practical Introduction to Web Scraping in Python. Real Python.
• Kevin Sahn. Web Scraping 101 with Python. ScrapingBee, 8/2019.
• Pierre de Wulf. Web Scraping 101 with Python. Daolf, 8/2019.

Tools

Scrapy

• Scrapy – Stars: 39.4k – Updated: 1/2021 – Checked: 1/2021 – “An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.”
  • Daniel Ni. The Ultimate Guide to Building Scalable Web Scrapes with Scrapy. smashing magazine, 7/2019.
  • Zac Clancy. Web Scraping with Scrapy: Advanced Examples. kite, 4/2019.
  • Portia – Visual scraping for Scrapy. 7/2019.
  • Web Scraping with Scrapy and MongoDB. Real Python.
  • Web Scraping and Crawling with Scrapy and MongoDB. Real Python.

Beautiful Soup

• Beautiful Soup – Launchpad – Updated: 10/2020 – Checked: 1/2021 – “Beautiful Soup is a Python library designed for quick turnaround projects like screen-scraping.”
  • Philip Kiely. DIY Data: Web Scraping with Python and Beautiful Soup. floydhub, 2019.
  • Martin Breus. Beautiful Soup: Build a Web Scraper with Python. real python, 12/2019.
    • This article covers using BS with requests and pprint on Monster’s job site. Good tutorial.
  • Robley Gori. Web Scraping with Python: A Tutorial on the BeautifulSoup HTML Parser. kite, 10/2018.
  • Julia Kho. How to Web Scrape with Python in 4 Minutes. towardsdatascience, 9/2018.

Other Tools

• Newspaper3k – Stars: 10.6k – Updated: 9/2020 – Checked: 1/2021 – Article scraping and curation.
• PySpider – Stars: 14.8k – Updated: 8/2020 – Checked: 1/2021 – “A Powerful Spider(Web Crawler) System in Python.”