인프런 커뮤니티 질문&답변

강의에서 사용하신 네이버 검색 API가 안되는듯 하여 카카오 API로 사용하고 있습니다. 코드는 거의 동일합니다.
네이버를 사용 못한 이유는 아래처럼 적혀있어서 바뀐것 같던데 강의대로 진행하니 접속 안되더라구요
(아래 API들은 개발자센터 API 제휴신청을 받지 않습니다. 대상: 검색, 캡차(음성), 캡차(이미지), Papago 번역, 한글인명로마자 변환, 지도(웹), 지도(모바일), 단축URL ...이후 생략)

아래에 코드 전체와 터미널 메세지 올려드립니다.

카카오API를 사용할때 현재 증상은 코드를 계속 반복 실행하다보면 이미지가 거의 저장되지 않고 어쩌다 딱 1개 저장되고 그러는데요.

이미지 저장이 제대로 안되는 이유는 첫째로 카카오 API로 Document에서 url을 가져올경우 아래처럼 
http://cfile211.uf.daum.net/image/99C2724D5BFC51620F146D
이런식으로 .jpg .png로 끝나지 않는 주소들이 많더라구요. 이럴때 .jpg로 받을 수 있는 방법이 있을까요?
(해당 url의 이미지 오른쪽클릭하고 저장 누르면 catcat4302c29fd7b6b7b78c357dbafaf8e6b7.jpg 처럼 jpg 이미지로 받아지는데 주소에는 다르게 되어있네요)
추가로 kr.storage.gettyimagesbank.com << 여기에도 이유는 모르겠는데 접속 안된다고 뜨고 있구요. 뭔가 연결이 불안정한것 같은데 처음 배우다보니 잘 모르겠습니다..

그나마 .jpg로 끝나는 주소들은 손상된 이미지 파일이 다운되고 수십번 코드 반복실행하니까 정상적인 이미지가 1개 받아졌습니다. 손상된 이미지들도 전체 url 갯수만큼이 아닌 몇개씩만 받아지다가 끊기고 그러네요. 정상적인 이미지가 수십번 코드 돌리다보니 딱 1개 받아졌는데 받아진 것도 신기하네요..

(구글링 해보니 이런 코멘트가 있네요 Does your network require the use of a proxy server? Minor differences in reading the proxy configuration could explain why requests works, but aiohttp doesn't, all on the same computer.)

 

어디가 잘못되었을지 봐주시면 감사하겠습니다.

 

 

(venv) C:\repos\drf-project>python 06-scraping.py

1

2

3

4

['http://cfile211.uf.daum.net/image/99C2724D5BFC51620F146D', 'http://cfile272.uf.daum.net/image/2241933F565A896D0B3DF8', 'http://imagescdn.gettyimagesbank.com/500/15/183/277/0/470624520.jpg', 'http://imagescdn.gettyimagesbank.com/500/15/133/212/0/497283946.jpg', 'http://postfiles8.naver.net/20141001_279/hankukilbo_1412142133250i7s1r_JPEG/architects-for-animals-cat-shelters-fixnation-designboom-08.jpg?type=w2', 'https://k.kakaocdn.net/dn/cv3Kiz/btq2ytNi9XL/VANConlQulYNVK4wp6luj1/img.jpg', 'http://cfile218.uf.daum.net/image/99EA5A33598E086229AA1D', 'http://cfile220.uf.daum.net/image/998E2433598E094C2C792D', 'http://imagescdn.gettyimagesbank.com/500/21/021/938/0/1319877684.jpg', 'http://imagescdn.gettyimagesbank.com/500/201705/a10816517.jpg', 'http://cfile254.uf.daum.net/image/231BBD3856C70225024607', 'http://imagescdn.gettyimagesbank.com/500/21/913/673/0/1310254926.jpg', 'http://cfile225.uf.daum.net/image/2333A03854CE570F1C692F', 'http://cfile208.uf.daum.net/image/255CE64757B5E5B02AABB7', 'http://cfile245.uf.daum.net/image/261F55395527AC222D1A76', 'http://cfile249.uf.daum.net/image/21564C3A5583C9802817A5', 'https://k.kakaocdn.net/dn/cLlRZg/btq5fDl7GHg/NXzeW2ksvn7q8ARkzB49Q0/img.png', 'http://kr.storage.gettyimagesbank.com/thumb/201907/jv11407647.jpg', 'https://image-notepet.akamaized.net/seimage/20160608%2F16%EC%8A%A4%EC%BD%94%ED%8B%B0%EC%8B%9C_%ED%8F%B4%EB%93%9C.jpg', 'http://imagescdn.gettyimagesbank.com/500/18/728/560/0/1028301566.jpg']

['http://postfiles1.naver.net/data5/2005/4/14/96/Samantha_%28the_Cat%29-love_me____.jpg?type=w3', 'http://postfiles9.naver.net/MjAxOTA0MjlfMTg2/MDAxNTU2NTMwNzA3NzQ3.1k0bcz6MSA2KJGIbqB76UMUBgKDZfc-FjEy9CcAQSUcg.-P3nz27RxJffodHD2Isrq39B-87u7Gq2-6trteNDsXMg.JPEG.j9fbm93/HP_PCC_lg_cat_training_hero.jpg?type=w966', 'http://cfile234.uf.daum.net/image/99C04633599EDE401765C3', 'http://imagescdn.gettyimagesbank.com/500/20/345/909/0/1267584560.jpg', 'http://imagescdn.gettyimagesbank.com/500/15/166/064/0/484403996.jpg', 'http://cfile296.uf.daum.net/image/23225639569E64000AAA7F', 'http://imagescdn.gettyimagesbank.com/500/16/819/541/0/506298170.jpg', 'http://imagescdn.gettyimagesbank.com/500/11/041/065/0/140018638.jpg', 'http://kr.storage.gettyimagesbank.com/thumb/201907/jv11516337.jpg', 'http://cfile276.uf.daum.net/image/2210183F54E97EEF23E6D8', 'http://cfile222.uf.daum.net/image/2246E24F57B5E5DA07A8FF', 'https://k.kakaocdn.net/dn/o9Fz9/btq6xZ8ZXg3/X14guc4ZVUq16RqHEHLtkk/img.png', 'http://www.lifeanddogue.com/images/magazine/201606/style/style03_05.jpg', 'http://imagescdn.gettyimagesbank.com/500/21/661/306/0/1346632438.jpg', 'http://cfs6.blog.daum.net/image/10/blog/2007/11/23/07/18/4746004cc4f81&filename=2.jpg', 'http://cfile212.uf.daum.net/image/9999B033599EDE431AEB75', 'https://k.kakaocdn.net/dn/bAT9NC/btq6tTUG4X2/fSLRFyUD0Xn5dw3KkWCOd0/img.png', 'http://cfile205.uf.daum.net/image/991A3E33598E08312939C3', 'http://postfiles10.naver.net/MjAxODA0MTJfMTIw/MDAxNTIzNTMyMDI0MTQy.IazlEIGrTjwsWQ2hroqCIYp5y0sRMQRk7FMkj3bSdxgg.Xslw7oDhpq-HAd5UH9uUwXMIXS5VIAKPLhw9NsnN_pkg.JPEG.zino0011/creative-and-cozy-cat-beds-4-554x408.jpg?type=w966', 'http://imagescdn.gettyimagesbank.com/500/21/480/550/0/1322719077.jpg']

['https://k.kakaocdn.net/dn/cvuvFC/btq352VsUBO/N8n1t86pk7GLxZk3HEjfm1/img.png', 'http://cfile232.uf.daum.net/image/99D4B033598E09502B118A', 'https://images-na.ssl-images-amazon.com/images/M/MV5BMTY5MTI1MzE5Nl5BMl5BanBnXkFtZTgwMjQzNjEzOTE@._V1_UX182_CR0,0,182,268_AL_.jpg', 'http://imagescdn.gettyimagesbank.com/500/12/452/156/0/154413607.jpg', 'http://cfile208.uf.daum.net/image/140AB13F4EBBCC231C7FF4', 'http://postfiles6.naver.net/20160314_85/feb1992_1457966212670EHAy5_JPEG/funny-pictures-your-cat-is-bursting-with-joy1.jpg?type=w1', 'http://cfile247.uf.daum.net/image/257A303D566AC9D007DCB0', 'https://k.kakaocdn.net/dn/cQ7TTA/btq3PQTYMqu/XjZ0YVbp0CEhk7xkiDwLnk/img.png', 'http://cfile207.uf.daum.net/image/2748F55057B5E5AC1628F2', 'http://cfile269.uf.daum.net/image/224AFD365583C958205E7D', 'http://cfile276.uf.daum.net/image/272953355583C9642BCD64', 'http://cfile264.uf.daum.net/image/21563F3456580736192F5D', 'https://k.kakaocdn.net/dn/HnVnF/btq3k8in2g7/uBK3eRtR0V0rzGD0PPWBZ1/img.png', 'http://cfile252.uf.daum.net/image/21320F4B54BC1F5E2AC850', 'http://imagescdn.gettyimagesbank.com/500/21/958/661/0/1357810001.jpg', 'http://cfile253.uf.daum.net/image/2126AD3652724CF41169B1', 'http://imagescdn.gettyimagesbank.com/500/16/088/209/0/534002900.jpg', 'https://k.kakaocdn.net/dn/diyAqX/btq5apIjPBB/SDQAbzkTOhDklkBgWkWq31/img.png', 'http://cfile216.uf.daum.net/image/2525A44B59233783163FF6', 'http://t1.daumcdn.net/cafeattach/1VCne/c2e11f0119b73afc62b228480579f35500f02d13']

['https://k.kakaocdn.net/dn/nW8gC/btq2yvyJiS4/3VLSkiKWarzPv65fY1G2X1/img.jpg', 'http://cfile242.uf.daum.net/image/26399A3852724CEF1F003B', 'http://imagescdn.gettyimagesbank.com/500/21/923/905/0/1340658899.jpg', 'http://postfiles4.naver.net/20130714_179/ddong__ko_1373732445771PT2by_PNG/cat_bowtie_model.png?type=w2', 'http://cfile273.uf.daum.net/image/216A5747545E1C1A27A267', 'http://cfile211.uf.daum.net/image/1366DD3F4EBBCC223E12AF', 'https://postfiles.pstatic.net/MjAyMTEyMDVfMTM3/MDAxNjM4NjMzOTI1NzM4.LocBFkzcEC50Hb03Rn5ecGGCsun28skNrcL1ptxpJGAg.XPBJjjVyIMmrCnPngYyrCtL03hS5uxS8BfJeVoUbbBMg.JPEG.yklego2004/Screenshot%EF%BC%BF20211204%EF%BC%8D000458%EF%BC%BFInstagram.jpg?type=w966', 'http://cafe74.daum.net/_c21_/pds_down_hdn?grpid=W1Em&fldid=Sl8b&dataid=22&fileid=1&regdt=20040303072031&realfile=%C8%A3%B7%AF.jpg&ln=7&grpcode=hitchdog&dncnt=N&.jpg', 'http://cfile214.uf.daum.net/image/99E97833598E090B2B861A', 'http://postfiles1.naver.net/20140224_208/hjy910730_1393170993156tdoAd_JPEG/Animal-Cat_232.jpg?type=w2', 'http://postfiles7.naver.net/20140610_150/wongj81_1402332780792TL21s_JPEG/600px-boy-and-cat.jpg?type=w2', 'http://imagescdn.gettyimagesbank.com/500/20/040/730/0/1217059948.jpg', 'http://cfile238.uf.daum.net/image/992AEA33598E094C2B02FF', 'http://imagescdn.gettyimagesbank.com/500/21/301/533/0/1313511695.jpg', 'http://cfile256.uf.daum.net/image/24269A4156166F4E348A93', 'http://kr.storage.gettyimagesbank.com/thumb/201907/jv11740169.jpg', 'http://cfile254.uf.daum.net/image/2445404D54BC1F2F1EF6B1', 'https://k.kakaocdn.net/dn/crOiXc/btq16RvrhyK/yLPlPuL7GRuQXAKNwmTfm0/img.png', 'http://cfile225.uf.daum.net/image/99D09033599EDE930DEE73', 'https://k.kakaocdn.net/dn/8mAo2/btq3zojysoc/TqhwFj0Ou1wU5anonkFdO1/img.png']

Traceback (most recent call last):

  File "C:\repos\drf-project\venv\lib\site-packages\aiohttp\connector.py", line 999, in _create_direct_connection

    hosts = await asyncio.shield(host_resolved)

  File "C:\repos\drf-project\venv\lib\site-packages\aiohttp\connector.py", line 865, in _resolve_host

    addrs = await self._resolver.resolve(host, port, family=self._family)

  File "C:\repos\drf-project\venv\lib\site-packages\aiohttp\resolver.py", line 31, in resolve

    infos = await self._loop.getaddrinfo(

  File "C:\Users\choi\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 855, in getaddrinfo

    return await self.run_in_executor(

  File "C:\Users\choi\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\thread.py", line 58, in run

    result = self.fn(*self.args, **self.kwargs)

  File "C:\Users\choi\AppData\Local\Programs\Python\Python310\lib\socket.py", line 955, in getaddrinfo

    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):

socket.gaierror: [Errno 11001] getaddrinfo failed

 

The above exception was the direct cause of the following exception:

 

Traceback (most recent call last):

  File "C:\repos\drf-project\06-scraping.py", line 50, in <module>

    asyncio.run(main())

  File "C:\Users\choi\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py", line 44, in run

    return loop.run_until_complete(main)

  File "C:\Users\choi\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 641, in run_until_complete

    return future.result()

  File "C:\repos\drf-project\06-scraping.py", line 45, in main

    await asyncio.gather(*[fetch(session, url, i) for i, url in enumerate(urls)])

  File "C:\repos\drf-project\06-scraping.py", line 37, in fetch

    await asyncio.gather(*[img_downloader(session, img) for img in image_url])

  File "C:\repos\drf-project\06-scraping.py", line 17, in img_downloader

    async with session.get(img) as response:

  File "C:\repos\drf-project\venv\lib\site-packages\aiohttp\client.py", line 1117, in __aenter__

    self._resp = await self._coro

  File "C:\repos\drf-project\venv\lib\site-packages\aiohttp\client.py", line 520, in _request

    conn = await self._connector.connect(

  File "C:\repos\drf-project\venv\lib\site-packages\aiohttp\connector.py", line 535, in connect

    proto = await self._create_connection(req, traces, timeout)

  File "C:\repos\drf-project\venv\lib\site-packages\aiohttp\connector.py", line 892, in _create_connection

    _, proto = await self._create_direct_connection(req, traces, timeout)

  File "C:\repos\drf-project\venv\lib\site-packages\aiohttp\connector.py", line 1011, in _create_direct_connection

    raise ClientConnectorError(req.connection_key, exc) from exc

aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host kr.storage.gettyimagesbank.com:80 ssl:default [getaddrinfo failed]