강의

멘토링

커뮤니티

BEST
Data Science

/

Data Engineering

The Most Powerful Crawling Technology Today: Mastering Scrapy and Selenium

For those who want to learn data science, big data, and crawling - We have prepared a variety of examples to help you quickly learn the latest and best crawling techniques available.

(4.9) 112 reviews

1,261 learners

Level Intermediate

Course period Unlimited

  • funcoding
Web Crawling
Web Crawling
Scrapy
Scrapy
Selenium
Selenium
Web Crawling
Web Crawling
Scrapy
Scrapy
Selenium
Selenium

Reviews from Early Learners

Reviews from Early Learners

4.9

5.0

ez2shd2p

100% enrolled

Through the crawling bootcamp lecture and the scrapy selenium lecture, I was able to basically do the data collection work that I wanted. After listening to the two lectures and collecting data myself, I realized why I need to understand and be able to handle the database to some extent... I'm going to listen to the next lecture. It was a lecture that collected only the minimum but not insufficient content that was necessary to obtain the results I wanted. Thank you for the great lecture, and I will complete the remaining two roadmaps.

5.0

allrounder

100% enrolled

I learned the beginner level so much that I had fun learning it, and it was so good! I started a little after the lecture came out, so there were a lot of parts that changed (I tried hard to apply it and try it myself, but I failed a few of them until the endㅠ). Anyway, it was great for learning the basic flow. Now I'm moving on to the advanced lecture, and I hope that this lecture will be an opportunity for me to overcome the parts that I failed at in the intermediate level due to the server change.

5.0

김상원

62% enrolled

I've been taking the class for 3 years!! The best instructor in my life~ Friendliness 5 points / Explanation 5 points / Voice 4.5 points -> Rounding 5 points / Example selection 5 points

What you will gain after the course

  • Crawling using Selenium/Headless Chrome/PhantomJS

  • Crawling by programming the browser

  • Crawling with Scrapy

  • Dynamic Web Page Crawling

<center><h3 style="text-align:start">20,000 people enrolled in online and offline classes!<br /> Easily learn the most powerful crawling (data collection) technology available<br /> <span style="color:#ba372a">Highly complete lecture</span> </h3></center>

This course is a systematic course that learns crawling and Python data collection technology along with Python introduction and crawling basics boot camp . It is a special course that was created with the students' perspective in mind as much as possible , and is conducted in parallel with the field, and is a course that allows you to learn the functions one by one in a short period of time along with the lecture, and to further build data processing capabilities .

# What is the fastest and most stable crawling technique?

The most basic Python technology is the technique of using the requests and BeautifulSoup libraries. However, for each website, there are many cases where crawling is difficult with only related technologies. In this case, the technologies used are Selenium and Headless Chrome. Using related technologies, crawling is possible even in environments that require special operations, such as sites that require login. Also, when crawling a large amount of data stably rather than simply crawling a single web page, a framework called Scrapy is used. Scrapy is the fastest and most stable crawling technology.

# Is it difficult? How much do I need to know? Can I do it on my PC?

If you can understand Python programming and the basics of crawling, we have organized it so that you can crawl examples that are actually crawlable, focusing on the core parts, and use them right away. Also, since the related technology is closer to a technology that you need to use rather than a technology that you need to understand in depth, if you set a scope and learn it step by step, focusing on the parts that you absolutely need to understand, you can learn it more easily and in a shorter period of time.

* This course is designed to help you learn potentially complex related technologies easily in a short period of time .

# I don't have any basic knowledge of Python programming and crawling. Can I learn it systematically?

This course is designed to systematically learn from the basics to crawling technology or data analysis/science technology sequentially by learning it together with the Python Introduction and Crawling Basics Bootcamp in accordance with the data analysis/science roadmap described at the bottom of this detailed introduction.

In particular, Python programming and crawling basics are provided separately through the Python Introduction and Crawling Basics bootcamp classes. Through these two subjects, even those who have no knowledge of Python or crawling can follow along from the bottom, explaining everything from program installation in each Windows/Mac environment to background knowledge such as understanding of the web.

# I want to learn all the various technologies related to crawling, a data collection technology. How can I do that?

You can learn all the existing crawling technologies through the Python Introduction and Crawling Basics Bootcamp and this course.

# To what extent should I learn crawling techniques for data analysis/science?

It is sufficient to learn up to the level covered in the Python Introduction and Crawling Basics Bootcamp and this lecture. In addition, the background knowledge (HTML/CSS that compose the web) required to learn related technologies, various data formats required to handle data in programming, and even how to use Open API are all covered in both lectures. Through this, you can naturally become familiar with how to handle data in programming.

* Also, when doing data analysis/scientific work in the future, the most difficult thing is actually getting the data I want. If you are equipped with the most powerful weapon to get the data I want through this lecture and the Python Introduction and Crawling Basics Bootcamp, you can grow into a much more competitive and capable data analyst/scientist.

<center><h3 style="text-align:start"> To help you become familiar with relevant application technologies in a short period of time<br /> <span style="box-sizing:inherit;font-weight:inherit;font-style:inherit;color:#ba372a">We provide lectures based on various codes and materials, step by step/technology-wise.</span> </h3><p><img title="selenium_scrapy_materials_ads.gif" src="https://cdn.inflearn.com/public/files/courses/324146/e81a1253-2739-4d78-baf3-9868b795e837/selenium_scrapy_materials_ads.gif" alt="" width="720" height="405" /></p></center>

Don't waste your time!
It's not that we can't do something because we don't have the information!
Learn with proven lectures!

Have there been people who have paid millions of won for a lecture that only teaches grammar? While listening to the lecture, you can naturally repeat it, and it is inevitably ingrained in your head, and it is even patterned so that it can be used in practice, so that it can be used as is! It can be changed to fit various cases, and it is possible to properly build up the basics. At the same time, it is a curriculum that considers liberal arts students and even time. It is a lecture that was created after much thought due to an attachment to online lectures.

<center><div id="text-block-20" class="mk-text-block "><div class="clearboth"> <strong style="color:#bf360c">Crawling by manipulating browser and mouse with Selenium</strong> <strong style="color:#bf360c">(Next News Comment Crawling Example)</strong> </div><div class="clearboth"><img src="https://cdn.inflearn.com/public/files/courses/324146/e5e8aeae-5967-4266-a214-35ffd0c3ba08/selenium3.gif" alt="" width="800" /></div><div class="clearboth"></div><div class="clearboth"><p style="text-align:start"> If you are curious about the systematic curriculum, I recommend that you also take the crawling lecture, data analysis, and database lectures. </p><div class="vc_col-sm-12 wpb_column column_container vc_custom_1443015009332 _ height-full" style="box-sizing:inherit;color:#000a12;font-family:'Noto Sans KR', -apple-system, system-ui, system-ui, 'Apple SD Gothic Neo', 'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif, Oxygen, Ubuntu, Cantarell, 'Fira Sans', 'Droid Sans', Helvetica;font-size:16px;text-align:start;background-color:#ffffff"><div class="vc_col-sm-12 wpb_column column_container vc_custom_1443015009332 _ height-full" style="box-sizing:inherit"><center style="box-sizing:inherit"><div style="box-sizing:inherit;width:536px;padding:0.5rem;margin-top:2rem;margin-bottom:2rem"><p style="box-sizing:inherit;margin-top:0px;margin-right:0px;margin-left:0px;padding:0px;line-height:27.808px;font-size:1.1rem;color:#ba372a;font-weight:bold"> This is a series of IT lectures that we have carefully created so that you can feel, 'Ah! It's really different!'<br style="box-sizing:inherit" /> Only those who are rational, considerate of each other, and can build good relationships<br style="box-sizing:inherit" /> Please take the class! </p></div></center><p style="box-sizing:inherit;margin-top:0px;margin-right:0px;margin-left:0px;padding:0.4rem 0rem 0.4rem 1rem;font-size:1.1rem;background-color:#ba372a"> <span style="box-sizing:inherit;color:white;font-weight:bold">Courses currently open or scheduled to open on Inflearn</span> </p><p style="box-sizing:inherit;margin-top:0px;margin-right:0px;margin-left:0px;padding:0.4rem 0rem 0.4rem 1rem;font-size:1.1rem;background-color:#eeeeee"> <span style="box-sizing:inherit;color:inherit;font-weight:bold">Full Stack Course: Tech Tree that will help you create the latest web/app services from A to Z on your own</span> </p><center style="box-sizing:inherit"><img style="box-sizing:inherit;height:auto;display:block" title="fullstack_roadmap.jpg" src="https://cdn.inflearn.com/public/files/courses/325804/5251d2ba-69a9-4a9c-a96c-c3dac4e9665e/fullstack_roadmap.jpg" alt="" width="720px" /></center><p style="box-sizing:inherit;margin-top:0px;margin-right:0px;margin-left:0px;padding:0px">They are numbered in order of ripening. </p><p style="box-sizing:inherit;margin-top:0px;margin-right:0px;margin-left:0px;padding-right:0px;padding-bottom:0px;padding-left:0px"> <a style="box-sizing:inherit;text-decoration-line:none;cursor:pointer" href="https://www.inflearn.com/course/python-crawling-basic?inst=71325257" target="_blank" rel="noopener noreferrer">1. <span style="box-sizing:inherit;font-weight:inherit;font-style:inherit;color:#1976d2">Python and data collection (crawling) basics (Python and web, data understanding basics)</span><br style="box-sizing:inherit" /></a> <a style="box-sizing:inherit;text-decoration-line:none;cursor:pointer" href="https://www.inflearn.com/course/SQL-DB-MYSQL-%ED%8C%8C%EC%9D%B4%EC%8D%AC-%EB%8D%B0%EC%9D%B4%ED%84%B0%EB%B6%84%EC%84%9D?inst=7abfe3b0" target="_blank" rel="noopener noreferrer">2. <span style="box-sizing:inherit;font-weight:inherit;font-style:inherit;color:#1976d2">MySQL and Data Storage/Analysis Basics (SQL Database Basics)</span></a><br style="box-sizing:inherit" /> <a style="box-sizing:inherit;text-decoration-line:none;cursor:pointer" href="https://www.inflearn.com/course/nosql-%ED%8C%8C%EC%9D%B4%EC%8D%AC-%EB%AA%BD%EA%B3%A0DB-%EC%9E%94%EC%9E%AC%EB%AF%B8%EC%BD%94%EB%94%A9?inst=c98270ed" target="_blank" rel="noopener noreferrer">3. <span style="box-sizing:inherit;font-weight:inherit;font-style:inherit;color:#1976d2">NoSQL(mongodb) Big Data Basics (NoSQL Database Basics)</span></a><br style="box-sizing:inherit" /> <a style="box-sizing:inherit;text-decoration-line:none;cursor:pointer" href="https://www.inflearn.com/course/%ED%8C%8C%EC%9D%B4%EC%8D%AC-%ED%92%80%EC%8A%A4%ED%83%9D-1?inst=4d5f4fb3" target="_blank" rel="noopener noreferrer"><span style="box-sizing:inherit;font-weight:inherit;font-style:inherit;color:#1976d2">4. Fastest Full Stack: Python Backend and Web Technology Basics [Full Stack Part 1]</span></a><br style="box-sizing:inherit" /> <a style="box-sizing:inherit;text-decoration-line:none;cursor:pointer" href="https://www.inflearn.com/course/%ED%94%84%EB%9F%B0%ED%8A%B8%EC%97%94%EB%93%9C-%ED%92%80%EC%8A%A4%ED%83%9D-2?inst=c3f5cbf0" target="_blank" rel="noopener"><span style="box-sizing:inherit;font-weight:inherit;font-style:inherit;color:#1976d2">5. Solid Front-end Fundamentals for Full Stack: Javascript (Vanilla JS and ES6+) and Latest Web Technologies [Full Stack Part 2]</span></a><br style="box-sizing:inherit" /> <a style="box-sizing:inherit;text-decoration-line:none;cursor:pointer" href="https://www.inflearn.com/course/%EC%84%9C%EB%B2%84%EA%B8%B0%EC%88%A0-%ED%92%80%EC%8A%A4%ED%83%9D-3?inst=b03c6f9f" target="_blank" rel="noopener"><span style="box-sizing:inherit;font-weight:inherit;font-style:inherit;color:#1976d2">6. Docker and the latest server technology for full stack (Linux, nginx, AWS, HTTPS, flask deployment) [Full Stack Part 3]</span></a><br style="box-sizing:inherit" /> 7. Flutter Basics for Full Stack App Development (Full Stack Part 4, scheduled to open in the second half of 2021)<br style="box-sizing:inherit" /> 8. Basic Vue or React Framework for Full Stack (Full Stack Part 5, scheduled for the second half of 2021)<br style="box-sizing:inherit" /><br style="box-sizing:inherit" /> <b style="box-sizing:inherit">As app/web technologies are rapidly changing, we have adjusted our priorities. In order to take the lead in more recent technologies, we will first proceed with flutter, the latest technology that supports both web and apps.</b> </p><p style="box-sizing:inherit;margin-top:0px;margin-right:0px;margin-left:0px;padding-right:0px;padding-bottom:0px;padding-left:0px"> <span style="box-sizing:inherit;font-weight:inherit;font-style:inherit;color:#e03e2d">* Full stack course packages are also available at a discounted price. (Discounts will be reduced soon.)<br style="box-sizing:inherit" /> <a style="box-sizing:inherit;text-decoration-line:none;cursor:pointer" href="https://www.inflearn.com/roadmaps/66" target="_blank" rel="noopener noreferrer">[Beginner~Intermediate] The easiest and most up-to-date full-stack roadmap</a> <span style="box-sizing:inherit;color:inherit;font-weight:bold"><a style="box-sizing:inherit;text-decoration-line:none;cursor:pointer" href="https://www.inflearn.com/roadmaps/49" target="_blank" rel="noopener noreferrer">(shortcut)</a></span></span> </p><p style="box-sizing:inherit;margin-top:2rem;margin-right:0px;margin-left:0px;padding:0.4rem 0rem 0.4rem 1rem;font-size:1.1rem;background-color:#eeeeee"> <span style="box-sizing:inherit;color:inherit;font-weight:bold">Data Analysis/Science Course: The latest tech tree that can bring in the data you want, analyze it, and even make predictions.</span> </p><p style="box-sizing:inherit;margin-top:0px;margin-right:0px;margin-left:0px;padding-right:0px;padding-bottom:0px;padding-left:0px"> They are numbered in order of ripening. </p><p style="box-sizing:inherit;margin-top:0px;margin-right:0px;margin-left:0px;padding-right:0px;padding-bottom:0px;padding-left:0px"> <a style="font-family:sans-serif" href="https://www.inflearn.com/course/python-crawling-basic?inst=71325257" target="_blank" rel="noopener noreferrer">1. <span style="color:#1976d2">Python and data collection (crawling) basics (Python and web, data understanding basics)</span><br /></a> <a style="font-family:sans-serif" href="https://www.inflearn.com/course/crawling-scrapy-selenium?inst=469beff7" target="_blank" rel="noopener noreferrer">2. <span style="color:#1976d2">Conquering Scrapy and Selenium (Currently the most advanced crawling intermediate technology and related IT knowledge)</span><br /></a> <a style="font-family:sans-serif" href="https://www.inflearn.com/course/sql-db-mysql-파이썬-데이터분석?inst=7abfe3b0" target="_blank" rel="noopener noreferrer">3. <span style="color:#1976d2">SQL and Data Storage/Analysis Basics (Data Storage/Analysis)</span><br /></a> <span style="color:#1976d2"><a href="https://www.inflearn.com/course/nosql-파이썬-몽고db-잔재미코딩?inst=c98270ed" target="_blank" rel="noopener">4. NoSQL(mongodb) Big Data Basics (Big Data Storage/Analysis)</a></span><br style="font-family:sans-serif" /> <span style="color:#1976d2"><a href="https://www.inflearn.com/course/파이썬-데이터분석-전처리-판다스-시각화?inst=65936339" target="_blank" rel="noopener">5. First Python Data Analysis (Data Preprocessing and Pandas, Latest Visualization) [Data Science Part 1]</a></span><br style="font-family:sans-serif" /> <a style="font-family:sans-serif" href="https://www.inflearn.com/course/처음-파이썬-머신러닝-입문?inst=8f7d0ec8" target="_blank" rel="noopener"><strong><span style="color:#1976d2">6. Python Machine Learning Bootcamp for Beginners (Easy! Learn concepts/applications with real problems) [Data Science Part 2]</span></strong></a><br style="font-family:sans-serif" /> <span style="color:#1976d2">7. AI Artificial Intelligence Bootcamp (Data Prediction Automation, First Half of 22') [Data Science Part 3]</span> </p><p style="box-sizing:inherit;margin-top:0px;margin-right:0px;margin-left:0px;padding-right:0px;padding-bottom:0px;padding-left:0px"> <span style="box-sizing:inherit;font-weight:inherit;font-style:inherit;color:#e03e2d">* We are also offering our current data science course packages at a discounted price. (The discount rate will be reduced soon.)<br style="box-sizing:inherit" /> <a style="box-sizing:inherit;text-decoration-line:none;cursor:pointer" href="https://www.inflearn.com/roadmaps/66" target="_blank" rel="noopener noreferrer">[Beginner~Beginner] Learn the basic data analysis techniques for employment easily and thoroughly</a> <span style="box-sizing:inherit;color:inherit;font-weight:bold"><a style="box-sizing:inherit;text-decoration-line:none;cursor:pointer" href="https://www.inflearn.com/roadmaps/66" target="_blank" rel="noopener noreferrer">(Shortcut)</a></span></span> </p><p><span style="box-sizing:inherit;font-weight:inherit;font-style:inherit;color:#e03e2d"> </span></p></div></div></div></div></center>

Recommended for
these people

Who is this course right for?

  • For those who want to learn crawling properly with basic data science

  • Those who want to learn the latest crawling technology

Need to know before starting?

  • Python Introduction and Crawling Basics Bootcamp (More than 5,000 people have taken the course online and offline) Pre-registration required

Hello
This is

33,899

Learners

2,504

Reviews

1,949

Answers

4.9

Rating

13

Courses

Janjemi Coding, Dave Lee

  • About Janjaemi Coding Introduction Blog [Click]

  • Key Experience: Senior Engineering Manager/Principal Product Manager at Coupang, Engineering Manager at Samsung Electronics (approx. 15 years of experience)

  • Education: BA in Japanese Language and Literature from Korea University / MS in Computer Science from Yonsei University (A total mix)

  • Key Development History: Samsung Pay, E-commerce search services, RTOS compiler, Linux Kernel Patch for NAS

  • Books: Linux Kernel Programming, Understanding and Developing Linux Operating Systems, IT Core Technologies for Everyone to Read and Understand, Python Programming Primer for Absolute Beginners

  • Official Website: Fun-Coding (http://www.fun-coding.org) [Click]

  • This site shares free resources related to full-stack development, data science, and AI.

  • Others: Fun-Coding YouTube Channel [Click]

    • I'm starting to share tips and short free lectures little by little to help with IT learning!

For eight years, I have been consistently creating solid full-stack, data science, and AI courses while balancing my work in the industry with teaching IT.

I am gradually starting to share my lectures. Balancing my current industry experience with teaching, I have been consistently creating solid full-stack, data science, and AI courses for 8 years.

Curriculum

All

42 lectures ∙ (8hr 57min)

Course Materials:

Lecture resources
Published: 
Last updated: 

Reviews

All

112 reviews

4.9

112 reviews

  • kwansoonkr0678님의 프로필 이미지
    kwansoonkr0678

    Reviews 5

    Average Rating 4.0

    5

    98% enrolled

    In my opinion, this is the best among the crawling basics courses. I had to acquire the skills in a hurry because of the thesis, but it was not as easy as I thought. I listened to other (overseas) lectures and read books.... After listening to this course twice, I can now collect the data I need! Thank you! Fighting! Instructor~

    • funcoding
      Instructor

      Hello, thank you so much for writing such a great review. One of the goals of this course was to help students quickly master the skills they need in a short period of time, which would take a long time to learn on their own. I am so happy and honored to have received such a valuable review that fits this purpose perfectly. Thank you.

  • vangogh36590933님의 프로필 이미지
    vangogh36590933

    Reviews 2

    Average Rating 5.0

    5

    62% enrolled

    I've been taking the class for 3 years!! The best instructor in my life~ Friendliness 5 points / Explanation 5 points / Voice 4.5 points -> Rounding 5 points / Example selection 5 points

    • funcoding
      Instructor

      ^^ Thank you for the high score You are a student who always stays with me~! I will work even harder~

  • iamjaylee2195322님의 프로필 이미지
    iamjaylee2195322

    Reviews 2

    Average Rating 5.0

    5

    100% enrolled

    I learned the beginner level so much that I had fun learning it, and it was so good! I started a little after the lecture came out, so there were a lot of parts that changed (I tried hard to apply it and try it myself, but I failed a few of them until the endㅠ). Anyway, it was great for learning the basic flow. Now I'm moving on to the advanced lecture, and I hope that this lecture will be an opportunity for me to overcome the parts that I failed at in the intermediate level due to the server change.

    • funcoding
      Instructor

      Thank you~^^ You are improving. I support your growth journey!!

  • choihyeok님의 프로필 이미지
    choihyeok

    Reviews 19

    Average Rating 4.9

    5

    100% enrolled

    Best lecture

    • funcoding
      Instructor

      Thank you for your kind review. I understand that it was helpful. Thank you.

  • keving님의 프로필 이미지
    keving

    Reviews 4

    Average Rating 5.0

    5

    81% enrolled

    I completely understand

funcoding's other courses

Check out other courses by the instructor!

Similar courses

Explore other courses in the same field!