Easy Python Advanced Crawling [Scrapy, Selenium, Headless Chrome]

Hello. This is Dave Lee from Janjaemi Coding.

We are sharing with you an updated version of our easy-to-follow advanced Python crawling course.

Recently, the selenium library was updated, and the method name was changed. Of course, even if the method name is changed, it is common to use the existing method in the same way. This is for 'compatibility'. However, the selenium library development team? decided not to support the existing method at all. If this happens, there will be cases where the codes that used selenium will not work depending on the selenium library version, so it seems like a big decision.

Along with this, I changed two things, as shown below, since it doesn't affect the code but it shows the code I want to change:

1. All find_element_by_***() methods have been changed to find_element(By.***, ***).

- Example: find_element_by_id('myid') --> find_element(By.ID, 'myid')

- To declare By.ID, etc., the following additional declarations are also required.

from selenium.webdriver.common.by import By

2. Selenium 4.xx and above seems to want to change the initial setup code as well, although it doesn't affect the code.

- Previously, you could directly pass the path as an argument (e.g. webdriver.Chrome(chromedriver))

- It seems that it will change in the future to put it in a Service object. (Example: webdriver.Chrome(service=Service(chromedriver))

- For this, the webdriver_manager library also needs to be installed separately.

 

The above changes are actually just method name changes, but the core method names have changed, so all related codes and materials have been updated. All materials, including the codes in the next chapter, have been updated, so it would be a good idea to download them again when necessary.

- Understanding the big picture and goals of Selenium and Scrapy frameworks

- Let's solve it with practice problem 1: Headless Chrome and Selenium crawling

- Understanding the big picture of XPATH

- Understand XPATH by solving practice problems

- Multi-page crawling using Selenium

- Various uses of Scrapy: Crawling dynamic web pages by combining Selenium and Scrapy

 

I also found out about it in a hurry today and made a hasty correction. I hope there are no issues with the progress.

thank you