宿Šäžå¿ïŒ AWS ã¢ããªã±ãŒã·ã§ã³ã®å®æçã
ããžãã¹ã§AWSãã©ã®ããã«äœ¿çšããŠããã®ãå¿é
ã§ããïŒ
ð€ AWSå
¥éè
/äžäžçŽè
è¬çŸ©ãçµãããããŸã 1%ãè¶³ããªãããã§ãã
ð AWSã·ã§ãŒã±ãŒã¹ãéããŠäŒç€Ÿã§æé²ãçã£ãŠããŸãã
ðAWSãæäŸãããªãœãŒã¹ã§ããŸããŸãªãã€ãã©ã€ã³ãæ§ç¯ãããã§ãã
𥺠AWSã®æ§ã
ãªäœ¿ãæ¹ãäœéšããŠã¿ããã§ãã
ð€ã¯ã©ãŠãã®äžçã«ãã£ãšæ·±ãé¥ãããã§ãã
ð AWSã·ãªãŒãºã®çµçé§
ïŒããªãã®æ©ã¿ã«è§£æ±ºçãæç€ºããŸãã
AWSå®è·µç·šã®ããã®è¬çŸ©ã¯ããŸãAWSå
¥éè
ã äžçŽ/äžçŽè
ã®ããã®è¬çŸ©ãæããŠãã ããçããããããªãã£ããã§ããŸããã§ãããçããã®ç±çãªå¿æŽã𿝿ã®ãããã§ãã®è¬çŸ©ãã§ãããšããŠãéèšã§ã¯ãããŸããã
AWSãæ¬çªã§çŽ æ©ãå¿çšãããæ¹ã®ããã®è¬çŸ©ã§ãã
ãã®ã¬ãã¹ã³ã¯ããšã³ãžãã¢ãçŸæ¥ã§ã©ã®ããã«AWSã䜿çšããŠETLãã€ãã©ã€ã³æ§ç¯ãããã«ãŒã³ã³ãããAirflowã¹ã±ãžã¥ãŒã©ãæ§ç¯ããã®ãã鿥çã«äœéšã§ããããã«æ§æãããŠããŸããã¿ã€ãã«ã§ãããããã«ãæŠå¿µã®æŽçã¯æå°éã«æããå®ç¿äžå¿ã«é²ã¿ãŸãã
1ã€ã®ã»ã¯ã·ã§ã³ããšã®ã¯ã©ã¹ã®é·ãã¯çããŠ30åãé·ããŠ1æéããããŸããã°ã³ããŒããŒã¯ãã¹ãŠé€ããŠéèŠãªå
容ã ããå
¥ããŠå±æ§ã§çããã«äŒããããã§ãã
AWSãçŽ æ©ãå¿çšããŠäœ¿ãããæ¹ãAWSå
¥éè
ã®ããã®è¬çŸ©ãèããããæ·±ãã¯ã©ãŠããæãäžããŠã¿ããæ¹ã«åŒ·ããå§ãããŸãã
å®åã§çºçããå¯èœæ§ã®ããäŸå€ç¶æ³ã«å¯Ÿãã解決çãæç€ºããŸãã
è¬çŸ©å¶äœã®éçšã§æ°ã«ãªãéšåã®1ã€ã¯ãã©ãã«ã·ã¥ãŒãã£ã³ã°ïŒTroubleshootingïŒã»ã¯ã·ã§ã³ã§ããç§ãAWSãåãç°å¢ïŒãªãã¬ãŒãã£ã³ã°ã·ã¹ãã ãç°å¢èšå®ãªã©ïŒã¯ããªããšåžžã«åãã§ã¯ãããŸãããã ããç§ã¯ããæ»ã£ãŠãçããã¯ãšã©ãŒãæµ®ãã¶éšåããããšæããŸãããã¹ãŠã®ãšã©ãŒããã¹ãŠèª¬æããããšã¯äžå¯èœã§ããããã°ãã°çºçãããšã©ãŒãéããã©ã®ããã«ããªããçŽæ¥åé¡è§£æ±ºãããããšãã§ãããã«ã€ããŠã®æ©ã¿ãããããããŠã¿ãŸãããããã§ããã®ã»ã¯ã·ã§ã³ãäœãããšã«ãªããŸããã
è¬çŸ©ãäœããšãããç§ã¯ããŸããããŸãããä»ã®äººãã¡ã¯ãã®ãããªãšã©ãŒãçºçããŸãããïŒããšããèããæã¡ããã®æãã®å¯èœæ§ã®é«ãäŸå€ç¶æ³ãæŽçããããã«å¯Ÿãã解決çãçããã«æç€ºããŸãã
ã ãããåŠç¿åŸã¯...
- â
AWS ãèªåšã«äœ¿çšã§ããŸãã
- â
ãã€ã©ãã§ã©ã®ãªãœãŒã¹ãã©ã®ããã«äœ¿çšã§ãããã«ã€ããŠã®æèŠãçãŸããŸãã
- â
ãŸããç¶æ³ã«åãããŠé·æãšçæãåãèœåãã§ããŸãã
- â
ã¯ã©ãŠãã«å¯Ÿããçè§£åãé«ãŸããŸãã
Simon KimãäœæããAWSã·ãªãŒãºã®çµçã¹ããŒã·ã§ã³
æ¬è¬çŸ©ã¯å®ç¿90ïŒ
ãçè«10ïŒ
ã§æ§æãããŠããŸãããããã£ãŠãéžæã®ç¥èã¯AWSå
¥éè
ã®ããã®è¬çŸ©ã§ãã äžçŽ/äžçŽè
ã®ããã®è¬çŸ©ã¯ãªãã·ã§ã³ã§ããã宿Šç·šã®ããã®è¬çŸ©ãšäžç·ã«èŠèŽããŠãç¡çã¯ãããŸããã
AWS å®è·µç·šã®è¬çŸ©ã¯ AWS ã·ãªãŒãºã®çµç圹ã§ãã宿Šç·šãé 匵ã£ããããšãã£ãŠãAWSã®ãã¹ãŠããã¹ãŠçè§£ãããšã¯èšããŸãããããããä»åŸAWSã䜿çšããæ°ããã¹ãã«ãåŠã¶éã«å€§ããªå©ãã«ãªããšç¢ºä¿¡ããŠããŸãã
ã·ãªãŒãºéžæã³ãŒã¹ããã§ãã¯ããŠãã ããã
åŠç¿å
容ã®ãã¬ãã¥ãŒ
1. AWSãšAirflowã®åºäŒã
AirflowãAWSã§åãããšã¯ã§ããŸããïŒã¯ãã AWS ãš Airflow ãäžç·ã«äœ¿çšããŠãAWS ãªãœãŒã¹ã®å©ããåããŠãããå¹
åºããã€ãã©ã€ã³ãæ§ç¯ã§ããŸãã
(1) ã€ã³ã¹ã¿ã³ã¹äœæåŸã« Airflow ãåãã®ã«å¿
èŠãªããã±ãŒãžãã€ã³ã¹ããŒã«ããŸãã
ïŒ2ïŒAirflowããŒã¿ããŒã¹ãäœæããŠãŠãŒã¶ãŒãäœæããŸãã
(3) Airflow DAG ãå®è¡ããŸãã
2. AWS Batch & ECR
ããããžã§ããšãã¥ãŒãäœæããECRã«ããã·ã¥ããã€ã¡ãŒãžãå®è¡ããç·Žç¿ãããŸãã AWS Batch ã¯å
¥éè
ãäžçŽ/äžçŽè
ã®è¬çŸ©ã§æ±ãããŠããªãããããã®ã»ã¯ã·ã§ã³ã§ç°¡åã«ç޹ä»ããŸãã
(1) ããã«ãŒã€ã¡ãŒãžãä¿åãã ECR ãªããžããªãäœæããŸãã
(2) ããã«ãŒç»åãäœæããECR ãªãã«ç»åãããã·ã¥ããŸãã
ïŒ3ïŒããããžã§ãïŒJobïŒãšãã¥ãŒïŒQueueïŒãäœæããŠãžã§ããå®è¡ããŸãã
3. ETLãã€ãã©ã€ã³
AWSãªãœãŒã¹ïŒS3ãGlueãAthenaïŒã䜿çšããŠç°¡åãªETLãã€ãã©ã€ã³ãæ§ç¯ããŸãã
ïŒ1ïŒããŒããŒã¿ãå
¥ããã±ãããäœæããããŒã¿ãä¿æããAthenaããŒã¿ããŒã¹ãäœæããŸãã
ïŒ2ïŒGlue CrawlerãäœæããŠããŒããŒã¿ãæœåºããŸãã
(3) Athenaã§ããŒã¿æ€èšŒéçšãçµãŸãã
Q&A ð¬
Q. ãã®è¬çŸ©ãèããªããã°ãªããªãçç±ã¯äœã§ããïŒ
ä»ã§ã¯ãITç£æ¥ã ãã§ãªããããŸããŸãªå Žæã§ã¯ã©ãŠããèšåãã䜿çšãããŠããŸããã¯ã©ãŠãã€ã³ãã©ã¹ãã©ã¯ãã£ã¯çŸåšãå®ç掻ããé¢ããªããã°ãªããªããããã°ããŒã¿æä»£ã«äœãã§ããç§ãã¡ã¯ãæ¯æ¥å€æ°ã®ããŒã¿ãçæãåŠçãåæããŸããããã¯æ±ºããŠèª°ããããããšã¯ã§ããŸãããã¯ã©ãŠãã®å©ããåããŠãã ããã AWSã¯ãŸãã«ãã®åœ¹å²ãæ
ããŸãã
AWSã®å®è·µã®ããã®ã¬ãã¹ã³ã¯ãå®éã®çŸæ¥ã§ãã䜿ãããã¢ãŒããã¯ãã£ãããã€ã玹ä»ããããªãããããã£ãŠå®è·µã§ããè¬çŸ©ã§æ§æãããŠããŸããç·Žç¿ã«åŸãããªãããã§ãã®ãããªãªãœãŒã¹ã䜿çšãããããè¯ãæ¹æ³ããªãã®ããèªåã§è³ªåãæããªããè¬çŸ©ã«åŸãããšããå§ãããŸããã¯ã©ãŠãã«æ£è§£ã¯ãããŸããããã ãããè¯ãæ¹æ³ãååšããã ãã§ããäžããããç°å¢ã§ããªãã¯ããè¯ãæ¹æ³ãæ¢æ±ãèŠã€ããªããã°ãªããŸããããã®è¬çŸ©ã¯ãã®æ¹æ³ãæ¯æŽããŸãã
Q. éå°æ»è
ãèãããšãã§ããŸããïŒ
ãã¡ããã§ããã³ã³ãã¥ãŒã¿ã®å°æ»è
ã§ãªããŠãAWSã¯ã©ãŠãã®çµéšãããå Žåã¯ãååã«åè¬ã§ããŸãã
åè¬åã«ç¢ºèªããŠãã ããð¢
ç·Žç¿ã®ããã®OSãšããŒã«
- Windows/macOSã§åè¬ã§ããŸãã
- AWSã¢ã«ãŠã³ãã®äœæãå®äºããããè¬çŸ©ãåè¬ããããšããå§ãããŸãã
åŠç¿è³æ
- ç·Žç¿æã«äœ¿çšãããåœä»€ããŸãšããããã¹ããã¡ã€ã«ãPythonã³ãŒããã¡ã€ã«ãªã©ãåŠç¿è³æãšããŠæäŸãããŸãã
- ææ¥ååŸã«èªãã§ã¿ããšã圹ã«ç«ã€ãªã³ã¯ã¯è¬çŸ©æ²ç€ºæ¿ã§ç¢ºèªã§ããŸãã
ð¡éžæã®ç¥èãšæ³šæäºé
- ãã®è¬çŸ©ã¯å®è·µã®ããã®è¬çŸ©ã§ããã AWSå
¥éè
ã®æ¹ã«ã¯é©ããŠããŸããã AWSãšã¯ã©ãŠãã®çµéšããªãå Žåã¯ãã·ãªãŒãºAWSå
¥éè
ã®ããã®ã¬ãã¹ã³ãæåã«åŠã¶ããšã匷ããå§ãããŸãã ïŒ AWSäž/äžçŽè
è¬çŸ©ãšäžŠè¡ããŠåè¬ããããšã¯å€§äžå€«ã§ããïŒ
- ç·Žç¿ã«åŸãéã«å¿
èŠãªãã¡ã€ã«ã¯ææ¥æ·»ä»ãã¡ã€ã«ãããŠã³ããŒãããŠã䜿çšããã ããŸãã
- è¬çŸ©å
容ãåã蟌ãå Žåãåºæãæããã«ããŠããã ãããããšãããããŸãã