inflearn logo
강의

講義

知識共有

Airflow完全ガイド - Part 1

実習コードをダウンロードした後、Astro CLIでAirflowコンテナ実習環境を構築する

astro dev start - python 라이브러리 설치 fail

22

inflearner

投稿した質問数 1

0

[+] Building 353.8s (9/10)                                                                                                                                                                docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                      0.0s
 => => transferring dockerfile: 81B                                                                                                                                                                       0.0s
 => [internal] load metadata for astrocrpublic.azurecr.io/runtime:3.2-2                                                                                                                                   2.2s
 => [internal] load .dockerignore                                                                                                                                                                         0.0s
 => => transferring context: 113B                                                                                                                                                                         0.0s
 => [1/1] FROM astrocrpublic.azurecr.io/runtime:3.2-2@sha256:466a3e0a33a285fcde43340d6b10dc529284c19ea306e004c2e47d8295e6754d                                                                             0.0s
 => => resolve astrocrpublic.azurecr.io/runtime:3.2-2@sha256:466a3e0a33a285fcde43340d6b10dc529284c19ea306e004c2e47d8295e6754d                                                                             0.0s
 => [internal] load build context                                                                                                                                                                         0.0s
 => => transferring context: 1.21kB                                                                                                                                                                       0.0s
 => CACHED [2/8] ONBUILD COPY packages.txt .                                                                                                                                                              0.0s
 => CACHED [3/8] ONBUILD RUN /usr/local/bin/install-system-packages                                                                                                                                       0.0s
 => CACHED [4/8] ONBUILD COPY requirements.txt .                                                                                                                                                          0.0s
 => ERROR [5/8] ONBUILD RUN /usr/local/bin/install-python-dependencies                                                                                                                                  351.0s
------
 > [5/8] ONBUILD RUN /usr/local/bin/install-python-dependencies:
0.349 + /usr/local/bin/install-python-dependencies
0.360 Installing python dependencies using uv
0.580 Using Python 3.13.13 environment at: /usr/local
63.58 Resolved 198 packages in 1m 00s
63.68 Downloading flask-appbuilder (2.1MiB)
63.68 Downloading polars-runtime-32 (43.1MiB)
64.31 Downloading botocore (14.3MiB)
64.31 Downloading pyarrow (46.6MiB)
64.32 Downloading lxml (5.0MiB)
64.32 Downloading adbc-driver-manager (4.5MiB)
64.37 Downloading sqlean-py (2.9MiB)
64.37 Downloading connectorx (41.7MiB)
64.38 Downloading adbc-driver-sqlite (1.5MiB)
133.1  Downloaded adbc-driver-sqlite
149.2  Downloaded flask-appbuilder
181.5  Downloaded sqlean-py
206.7    Building pandas==2.1.4
223.0  Downloaded adbc-driver-manager
233.4  Downloaded lxml
345.7  Downloaded botocore
348.0    Building numpy==1.26.4
350.4   × Failed to build `numpy==1.26.4`
350.4   ├─▶ The build backend returned an error
350.4   ╰─▶ Call to `mesonpy.build_wheel` failed (exit status: 1)
350.4
350.4       [stdout]
350.4       + /tmp/.tmpsTSE7P/builds-v0/.tmpEbhjPZ/bin/python
350.4       /tmp/.tmpsTSE7P/sdists-v9/index/dac0d3b231508420/numpy/1.26.4/XkQbLab9UzRfRV7eP-5zy/src/vendored-meson/meson/meson.py
350.4       setup
350.4       /tmp/.tmpsTSE7P/sdists-v9/index/dac0d3b231508420/numpy/1.26.4/XkQbLab9UzRfRV7eP-5zy/src
350.4       /tmp/.tmpsTSE7P/sdists-v9/index/dac0d3b231508420/numpy/1.26.4/XkQbLab9UzRfRV7eP-5zy/src/.mesonpy-39t2gzuy
350.4       -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md
350.4       --native-file=/tmp/.tmpsTSE7P/sdists-v9/index/dac0d3b231508420/numpy/1.26.4/XkQbLab9UzRfRV7eP-5zy/src/.mesonpy-39t2gzuy/meson-python-native-file.ini
350.4       The Meson build system
350.4       Version: 1.2.99
350.4       Source dir:
350.4       /tmp/.tmpsTSE7P/sdists-v9/index/dac0d3b231508420/numpy/1.26.4/XkQbLab9UzRfRV7eP-5zy/src
350.4       Build dir:
350.4       /tmp/.tmpsTSE7P/sdists-v9/index/dac0d3b231508420/numpy/1.26.4/XkQbLab9UzRfRV7eP-5zy/src/.mesonpy-39t2gzuy
350.4       Build type: native build
350.4       Project name: NumPy
350.4       Project version: 1.26.4
350.4
350.4       ../meson.build:1:0: ERROR: Unknown compiler(s): [['cc'], ['gcc'],
350.4       ['clang'], ['nvc'], ['pgcc'], ['icc'], ['icx']]
350.4       The following exception(s) were encountered:
350.4       Running `cc --version` gave "[Errno 2] No such file or directory: 'cc'"
350.4       Running `gcc --version` gave "[Errno 2] No such file or directory:
350.4       'gcc'"
350.4       Running `clang --version` gave "[Errno 2] No such file or directory:
350.4       'clang'"
350.4       Running `nvc --version` gave "[Errno 2] No such file or directory:
350.4       'nvc'"
350.4       Running `pgcc --version` gave "[Errno 2] No such file or directory:
350.4       'pgcc'"
350.4       Running `icc --version` gave "[Errno 2] No such file or directory:
350.4       'icc'"
350.4       Running `icx --version` gave "[Errno 2] No such file or directory:
350.4       'icx'"
350.4
350.4       A full log can be found at
350.4       /tmp/.tmpsTSE7P/sdists-v9/index/dac0d3b231508420/numpy/1.26.4/XkQbLab9UzRfRV7eP-5zy/src/.mesonpy-39t2gzuy/meson-logs/meson-log.txt
350.4
350.4       hint: This usually indicates a problem with the package or the build
350.4       environment.
------
Dockerfile:1
--------------------
   1 | >>> FROM astrocrpublic.azurecr.io/runtime:3.2-2
   2 |
--------------------
ERROR: failed to build: failed to solve: process "/bin/bash -o pipefail -e -u -x -c /usr/local/bin/install-python-dependencies" did not complete successfully: exit code: 1

What's next:
    Debug this build failure with Gordon → docker ai "help me fix this build failure"


Error: an error was encountered while building the image, see the build logs for details

 

위 같은 에러가 나서 명령어 docker ai "help me fix this build failure" 로 docker ai 에 진입하고 Gordon 이 requirement.txt 의 버전을 변경했습니다. 바뀐 버전으로 astro dev start 명령어로 build 성공했습니다.

 

# Astro Runtime includes the following pre-installed providers packages: https://www.astronomer.io/docs/astro/runtime-image-architecture#provider-packages
apache-airflow-providers-fab
apache-airflow-providers-http==5.4.0
apache-airflow-providers-amazon==9.20.0
apache-airflow-providers-mysql==6.4.1
apache-airflow-providers-postgres==6.5.1
numpy>=2
pandas>=2.2
polars[database]==1.37.1
#adbc-driver-manager 
#adbc-driver-postgresql

ai 진단시 메세지 보니까 python 3.13 기반으로 의존성 맞춰주는거 같더라구요.

이렇게 변경된 의존성으로 실습 진행해도 될까요?

 

 

데이터-엔지니어링 airflow orchestration

回答 1

0

dooleyz3525

안녕하십니까,

container에서 python 3.13 으로 설치가 되어서 numpy 1.26.4 가 설치가 안되는군요.

제가 실습 코드를 다 테스트 해보지는 않았지만, numpy, pandas 가 설치가 되면 실습에는 문제가 없을 것 같습니다.

다만 실습환경 동일하게 버전을 설정하시면 좋을 것 같습니다. 제가 강의에서는 astro version을 1.34.0 을 설치합니다. 해당 버전이 맞는지 확인 부탁드립니다.

command 창에서 astro version 하시면 설치하신 버전이 나옵니다. 현재 버전이 어떻게 되는지 여기에 글 부탁드립니다.

 

감사합니다.

 

맥북 환경구성 에러

0

19

2

라이브 운영중인 환경의 테이블에 인덱스 추가시 고려사항

0

36

2

virtual box 설치 문제

0

59

2

맥북용 자료 부탁드립니다.

0

37

2

nosql -> sql 비교

0

34

1

주제영역3 데이타 조회 문의

0

36

2

studio 3t 설치시 문의

0

37

1

카프카 서버 구축 관련 문의

0

62

2

astro project 실무 적용 질문

0

48

1

제 3정규화 관련 질문

0

49

3

pykrx 회원제 전환으로 인한 실습 불가

0

103

2

오라클

0

55

1

'레벨 유지'의 의미에 대한 질문

0

52

2

48강 csv

1

61

2

강사님의 로그포맷과 저의 로그포맷이 차이가 있는것 같습니다.

0

74

3

GROUP BY 절 문제 4번에서 집계함수가 반드시 포함? 필수?

1

63

2

7 강에서 astro dev start를 하는데, localhosst8080화면이 안뜹니다

0

59

2

astro dev start에서 building 시간이 깁니다.

0

59

2

Live SQL 스크립트 설치건

1

70

2

수료증 발급

0

78

3

consume 이후 lag가 줄어들지 않음

0

69

2

mac erwin 설치 관련

1

125

2

dbt run 이후 에러 발생(customer_status 없음 )

0

72

3

모바일 앱

1

53

1