Python Book 9 - Python for Web Scraping for Rookies/ Beautiful Soup and Scrapy

Table of Contents

1. Introduction

1.1 What is Web Scraping?

1.2 Why Web Scraping?

1.3 Ethical and Legal Issues of Web Scraping

1.4 Python for Web Scraping

1.5 Installing and Setting Up Python

1.6 Summary

1.7 Exercises

2. Getting Started with Beautiful Soup

2.1 What is Beautiful Soup?

2.2 Parsing HTML with Beautiful Soup

2.3 Navigating the HTML Tree

2.4 Searching and Filtering Elements

2.5 Modifying and Extracting Data

2.6 Handling Exceptions and Errors

2.7 Summary

2.8 Exercises

3. Getting Started with Scrapy

3.1 What is Scrapy?

3.2 Creating and Running a Scrapy Project

3.3 Defining Items and Fields

3.4 Writing Spiders and Requests

3.5 Extracting Data with Selectors and XPath

3.6 Storing and Exporting Data

3.7 Summary

3.8 Exercises

4. Advanced Web Scraping Techniques

4.1 Handling Dynamic Content with Selenium

4.2 Scraping JavaScript-Rendered Pages with Scrapy-Splash

4.3 Scraping Multiple Pages and Links with Scrapy

4.4 Scraping Forms and Login Pages with Scrapy

4.5 Scraping AJAX and JSON Data with Scrapy

4.6 Summary

4.7 Exercises

5. Web Scraping Projects

5.1 Scraping Quotes from Goodreads with Beautiful Soup

5.2 Scraping News Articles from CNN with Scrapy

5.3 Scraping Product Reviews from Amazon with Scrapy and Selenium 5.4 Scraping Weather Data from NOAA with Scrapy-Splash

5.5 Scraping Tweets from Twitter with Scrapy and Twisted

5.6 Summary

5.7 Exercises

6. Conclusion

6.1 Key Takeaways

6.2 Further Resources