The Most Powerful Crawling Technology Today: Mastering Scrapy and Selenium

# What is the fastest and most stable crawling technique?

The most basic Python technology is the technique of using the requests and BeautifulSoup libraries. However, for each website, there are many cases where crawling is difficult with only related technologies. In this case, the technologies used are Selenium and Headless Chrome. Using related technologies, crawling is possible even in environments that require special operations, such as sites that require login. Also, when crawling a large amount of data stably rather than simply crawling a single web page, a framework called Scrapy is used. Scrapy is the fastest and most stable crawling technology.

# Is it difficult? How much do I need to know? Can I do it on my PC?

If you can understand Python programming and the basics of crawling, we have organized it so that you can crawl examples that are actually crawlable, focusing on the core parts, and use them right away. Also, since the related technology is closer to a technology that you need to use rather than a technology that you need to understand in depth, if you set a scope and learn it step by step, focusing on the parts that you absolutely need to understand, you can learn it more easily and in a shorter period of time.

* This course is designed to help you learn potentially complex related technologies easily in a short period of time .

# I don't have any basic knowledge of Python programming and crawling. Can I learn it systematically?

This course is designed to systematically learn from the basics to crawling technology or data analysis/science technology sequentially by learning it together with the Python Introduction and Crawling Basics Bootcamp in accordance with the data analysis/science roadmap described at the bottom of this detailed introduction.

In particular, Python programming and crawling basics are provided separately through the Python Introduction and Crawling Basics bootcamp classes. Through these two subjects, even those who have no knowledge of Python or crawling can follow along from the bottom, explaining everything from program installation in each Windows/Mac environment to background knowledge such as understanding of the web.

# I want to learn all the various technologies related to crawling, a data collection technology. How can I do that?

You can learn all the existing crawling technologies through the Python Introduction and Crawling Basics Bootcamp and this course.

# To what extent should I learn crawling techniques for data analysis/science?

It is sufficient to learn up to the level covered in the Python Introduction and Crawling Basics Bootcamp and this lecture. In addition, the background knowledge (HTML/CSS that compose the web) required to learn related technologies, various data formats required to handle data in programming, and even how to use Open API are all covered in both lectures. Through this, you can naturally become familiar with how to handle data in programming.

* Also, when doing data analysis/scientific work in the future, the most difficult thing is actually getting the data I want. If you are equipped with the most powerful weapon to get the data I want through this lecture and the Python Introduction and Crawling Basics Bootcamp, you can grow into a much more competitive and capable data analyst/scientist.