Inflearn Community Q&A

mrfreiheit6284

asked

[New Revised Edition] This is Real Crawling - Practical Edition (AI Monetization)

Finding tags containing text (feat. Regular Expressions)

첫페이지 크롤링 이 안되네요.

Written on

479

쿠팡 첫페이지 크롤링이 안됩니다.

ㅜㅜ 이건 어떻게 해결을 해야 할까요?

오류 메세지 조차 안나오고 멈춰버려요 ㅠㅜ

터미널에서 확인했을때 커서만 깜빡이고 진행을 안합니다.ㅡㅜ

from itertools import product
import requests
from bs4 import BeautifulSoup


main_url = "https://www.coupang.com/np/search?component=&q=%EA%B2%8C%EC%9D%B4%EB%B0%8D+%EB%A7%88%EC%9A%B0%EC%8A%A4"

#헤더에 유저에이전트 추가
response = requests.get(main_url, headers={'User-Agent' : 'Mozila/5.0'})
html = response.text
soup = BeautifulSoup(html, 'html.parser')

links = soup.select("a.search-product-link")
#print(links)

for link in links :
    sub_url = "https://www.coupang.com/" + link.attrs['href']

    response = requests.get(sub_url, headers={'User-Agent' : 'Mozila/5.0'})
    html = response.text
    soup = BeautifulSoup(html, 'html.parser')

    #브랜드명
    brand_name = soup.select_one("a.prod-brand-name").text
    #상품명
    product_name = soup.select_one("h2.prod-buy-header__title").text
    #가격
    product_price = soup.select_one("span.total-price > strong").text

    print(brand_name, product_name ,product_price)
 
 