Python Web Crawling & Automation to Do My Work (feat. Stock, Real Estate Data / Instagram)

With Python
Take on the challenge of web crawling and automation!

🙌 Who is this lecture for?

Goldman Sachs, a global investment bank, boasts a workforce comprised of top talent in fields like economics, management, mathematics, and statistics. Surprisingly, more than a third of its employees are IT professionals. Despite being a financial institution, the vast majority of its employees are proficient in coding. What's even more surprising is that this statistic is from several years ago. I'm confident that many of the existing employees have been replaced by computers, and the proportion of IT professionals has increased, reaching at least half.

As times change, coding skills are becoming increasingly important, regardless of major, field, or job type. Recently, more and more companies are demanding coding skills for hiring, regardless of whether they're in the humanities or sciences. At the national level, the importance of coding education is being emphasized, and various educational policies are being actively promoted.

Why is coding skills increasingly in demand as a fundamental skill? One reason is the growing global demand for AI, with the development and application of advanced algorithms using deep learning on the rise. Another reason is that many (repetitive) tasks in the workplace can be replaced or automated by software.

Life is too short to waste our precious time on repetitive tasks. When we live a life filled with such repetition, it becomes difficult to focus on more important, priority tasks. If we take a closer look at our daily lives, we find that there are many tasks that could be replaced with programming. For example, tasks like periodically retrieving desired data from a website or continuously monitoring specific content might be possible. What if computers could perform these tasks faster and more accurately than humans?

In this course, you'll learn how to create Python programs that can handle these tasks for you. Rather than offering a few lines of code and a cursory lecture that simply says, "If you run this code, you can scrape this data from site A," this course begins with the fundamentals of how the web works, presents various perspectives on crawling-related issues, and demonstrates the process of solving them through programming. This course aims to empower you to implement a variety of crawling-based automated tasks on any website .

🙌 Say goodbye to ctrl+c and ctrl+v! Create your own crawler using Python.

When I worked on outsourcing projects for non-IT companies in the past, I was shocked more than once. I often saw highly skilled individuals with master's and doctoral degrees working overtime, even through the night, to collect data from the web for research or reports. Most of them were laboriously retrieving data using mouse clicks and Ctrl+C and Ctrl+V, copying it one by one into Excel, and even manually handling formatting errors. This was a waste of time and effort.

The reason I was shocked was quite understandable, because these tasks could easily be automated with just a few lines of Python code . Unfortunately, while these individuals fully recognized the importance of coding, they were at a loss as to how to learn and apply it. Furthermore, the tasks at hand were more urgent and important, so they inevitably repeated this vicious cycle. In reality, many office workers have experienced similar things in the past or are currently wasting time on these meaningless tasks, causing stress. In this course, we will learn how to automate repetitive web-based tasks (such as crawling and periodic checking) by logically coding them in Python, and we will help you reclaim your precious time.

[Stock-Specific Minute, Daily, and Financial Statement Data Crawling & Preprocessing]

Real Estate Complex Information/Market Price Data & Gap Analysis Within the Complex

[Naver Blog List Crawling]

Just a few lines of Python code can change your work life and even your life.
Don't you want to live a life of leisure every day?

🙌 "No 40-hour workweek, no mistakes, no complaints, no grievances, no pay!" Create your own personal assistant.

'If I had employees under me, I could delegate work and take a break...'

Many working people (especially those just starting out) have probably thought about this at least once. Having direct reports who handle my work for me . And not just one, but several. Just imagining it makes you think it would be a happy work life, doesn't it?

With Python, you can do it too. Computers are much faster than humans and, because they never make mistakes, they're incredibly efficient in many ways. Plus, there's no need for breaks, no sleep, and no pay (except for a small electricity bill), so you don't have to worry about ever-increasing hourly wages. All we have to do is define the problem well in a programming language so we can do our job correctly .

So, what tasks can Python replace? Almost any repetitive task can be replaced.

[Automated Instagram Follow & Like]

[Download PDF file]

[Slack Messenger Notifications]

In addition to this, you can program and automate numerous tasks, including retrieving data at specific times/dates, such as Naver Cafe/blog posts, online news articles, stock prices, and financial statement information, or tasks that require periodic monitoring, such as movie ticket sales start notifications.

I personally create workers to do my work for me.
How about taking a break from your busy daily life and finding some more leisure?

🙋 Top 6 Questions Students Ask the Most

Q1. Is it mandatory to take other courses (e.g., "Python for Everyone," "Pandas Stock Quantitative Investment Part 1") to take this course? Is it better to take this course first and then take the other courses?

Q2. Looking at the table of contents, it seems like stock data is the main focus. Is this content limited to crawling stock and financial data?

Q3. I've completed the course, and I'm feeling confident because I learned so much. However, I'm not sure what topics or data to crawl. Do you have any tips for this?

Q4. In the previous class, the pandas version was 0.25.3, but this class uses pandas version 1.x. How do I handle version conflicts?

Q5. I'm having trouble understanding a part of the lecture. What should I do?

=> The answers to the above questions (Q1 ~ Q5) are covered in detail in the video " Section 0. FAQ of Orientation".

Q6. How can I view the table of contents within a Jupyter file?

A6. Please refer to https://www.inflearn.com/course/생초보-입문-PYTHON/lecture/73183

📚 We've prepared this orientation video with as much care as the course content itself. While it's a bit long, it covers the direction and purpose of not only this course but also future courses, and covers topics you might be curious about. Let this orientation spark your desire to learn!

📚 No more one-man lectures where the instructor stands in front of students! Learn the principles and learn together! Let's get started!