å®è·µäžå¿ã«å
å®ããŠåŠã¶
ããã°ããŒã¿ãã€ãã©ã€ã³ãã¹ã¿ãŒïŒ
special thanks to my lovely students ðšð»âð
* appreciate it, believe you'll do well anywhere ð©ð»âð
ããŒã¿åŠççè«ãšå®è·µ
ããã°ããŒã¿ãã€ãã©ã€ã³ã®äžæ žïŒ
ããã«ã¡ã¯ã J.PHILã§ãðð»
è¯ãæ©äŒãè¿ããInflearnã§åè¬çŸ©ã§ããã°ããŒã¿ã·ã¹ãã ã®æ§ç¯ãšåæã«é¢å¿ã®ããå
¥éè
ã®ããã«ãããŒã¿åŠççè«ãšå®ç¿ãè¬çŸ©ãé²ããŸããã
äžç®ã§èŠãããŒã¯ãŒã
Mastering Big Data Processing: Tools and Techniques for Success
忣ã·ã¹ãã
Apache Spark
HDFS
Elasticsearch
Logstash
ããã
Crawler
Scraping
Selenium
AWS S3
Node.js
Docker
ç§ãã¡ã¯ãªããã®è¬çŸ©ã
èãã¹ãã§ããïŒ ð
è¿å¹ŽçŽ10幎éã®æ¥é²çãªæè¡çºå±ã«ãããããŸããŸãªãã©ãããã©ãŒã ãšãµãŒãã¹ãçãŸããããŸããŸãªé¡§å®¢ããã®äžã§è³ªã®é«ãçæŽ»ãå¶ãã§ããŸãããã®ããããŠãŒããã§ã¯ã¹ã³ã§çºçããããŒã¿ã®äžã§ãå€ãã®äŒæ¥ããã®äžã§äŸ¡å€ããããŒã¿ãçºæããŠæœåºããBM ïŒBusiness ModelïŒãèšèšããããšã§ãããå°ãç§ãã¡ã«äŸ¡å€ããçæŽ»ãæäŸããŠããŸãã
ãã®ãããªç°å¢ã§ç§ãã¡ã®ãšã³ãžãã¢ãã¡ãããããŠæªæ¥ãäºæž¬ããŠå¯Ÿå¿ãããªãã What and Howæºåãã¹ãã§ããããïŒããã«ããŒã¿ã管çããåŠçããèœåãè²ãŠãå¿
èŠããããŸããéã«ãããªããããŒã¿ãããŸãæ±ãããšã«ãªãããã衚çŸããããšãã§ããã°ãç£æ¥ã«ã©ããªç¹ãå¯äžã§ããã®ã§ããããã
Data-driven decision-making
ð¡ããã°ããŒã¿åæã«ãããçµç¹ã¯ããŒã¿äžå¿ã®æ±ºå®ãäžãããšãã§ããããžãã¹çµæãåäžãããããšãã§ããŸãã
Increased efficiency and productivity
ð¡ããã°ããŒã¿åæã«ãããçµç¹ã¯éçšãç°¡çŽ åããã³ã¹ããåæžããçç£æ§ãåäžãããããšãã§ããŸãã
ã€ãããŒã·ã§ã³
ð¡ããã°ããŒã¿åæã¯ãäŒæ¥ãæ°ãã補åãšãµãŒãã¹ãéçºããæ¢åã®è£œåãšãµãŒãã¹ãæ¹åããæ°ããããžãã¹ã¢ãã«ãäœæã§ããããã«ããããšã§é©æ°ãä¿é²ããŸãã
ã ãããã®è¬çŸ©ã¯ã©ã®ããã«
æ§æãããŠããŸããïŒ ð
ð
Data Top-Tier Conference è«æãæ±ããããçµéš
ðšð»âðŒ
çŸæ¥ã§åŸã䟡å€ããããã°ããŒã¿ã·ã¹ãã ã®æ§ç¯ãšåæçµéš
ð§ð»âð«
é·ãéã倧åŠã§è¯ãåŒåãè²æããçµéš
ãã®ãããªè²Žéãªçµéšãããšã«ãããã®åéã«é¢å¿ã®ãã誰ã«ã§ããè¯ãåºçºç¹ã«ãªãããã«ãããã°ããŒã¿ããã»ã¹ã®4ã€ã®æ®µéã«ã€ããŠ14é±ç®ä»¥äžã®å¹
åºããªãããå
å®ããã³ãŒã¹ã§è¬çŸ©ãæ§æããŸããð
ããªãã¯äžã§ç޹ä»ããæè¡ã䜿ã£ãŠããŒã¿åéâ¶ããŒã¿ä¿åâ¶ããŒã¿åæâ¶è¡šçŸã«ã€ããŠçè«30ïŒ
ãšå®ç¿70ïŒ
ã®ã³ãŒãã©ãã圢åŒã§åŠç¿ããŸããçŽ6幎éãå倧ãªåŒåãã¡ã®å€§åãªãã£ãŒããã㯠ð åé¡§ããŠåæ ããã§ããã ãç°¡åã§è³ªã®é«ãã³ã³ãã³ãã§è¬çŸ©ãæ§æããã ãã«å
¥éè
ã®æ¹ã
ã«å¿
ã倧ãã«åœ¹ç«ã€ã§ãããã
ããïŒåèãŸã§ã«ãè¬çŸ©è³æã¯ä»åŸæ§ã
ãªReferenceãæ¢ãããããã®æ©äŒãéããŠResearchåéãããè¯ãäŒæ¥ã«è¡ã£ããããéã«åœ¹ç«ãŠãããã«ãã§ããã ãè±èªã§äœæããŸãã ð§ð»ââïž
ç§ãã¡ã¯äœãåŠã³ãŸããïŒ ð§ð»âð«
äžèšã®Big Data Processing 4 Stepsã«åºã¥ããŠã以äžã®ããã«ã«ãªãã¥ã©ã ãæ§æããŸããã ïŒ1é±éç¡ææ ååç
§ïŒ
ããã°ããŒã¿ãã€ãã©ã€ã³ã«èå³ããã
誰ã§ãåè¬ã§ããŸãð§ð»âð
PythonãšLinuxã®ã³ãã³ããããŒã¿ããŒã¹ã«é¢ããåºæ¬çãªç¥èã ããç¥ã£ãŠããã°ã誰ã§ãåè¬ã§ããŸãã
[ããã¢ãŒã·ã§ã³]åŠçã就任çã®æ¹ã«è¬çŸ©ã®éé¡ãæ¯æŽããŸãðª
åå
¥ã®ãªãåŠçãå°±åŠè
ã« çŽ20ïŒ
å²åŒãæäŸããŸããäžèšãªã³ã¯ã§ãç³ã蟌ã¿ããã ããã·ãŒã ã¬ã¹ãªcommãããããã²åè¬åãåãåããã¿ãã«ã${èªå·±ç޹ä»}åŠç/å°±åŠçãããã¢ãŒã·ã§ã³ãç³ã蟌ã¿ãŸãããã®ããã«ãã°æ®ããŠãã ãã:)
ã¢ããªã±ãŒã·ã§ã³ãªã³ã¯/ åè¬åã®ãåãåãã
[ããã¢ãŒã·ã§ã³] +200åè¬çèšå¿µãæéå»¶é·ããã¢ãŒã·ã§ã³ðª
+100åã®ããã¢ãŒã·ã§ã³ç¹å
žãåããåè¬çåãé€ããæ°èŠåè¬çåã«è¬çŸ©3ã¶æç¡æå»¶é·ããã¢ãŒã·ã§ã³ãè¡ããŸããäžèšã®ãªã³ã¯ãããç³ã蟌ã¿ãã ãã:)
ã¢ããªã±ãŒã·ã§ã³ãªã³ã¯
ç·Žç¿ç°å¢ã¯ã©ããªããŸããïŒ ð»
äžèšã®ãããªæ°è»œãªç°å¢ããçšæããã ããŠããååã«ææ¥ã«åŸãããšãã§ããŸãã
ïŒå®éã®Clusteræ§æã¯çŸåšè£œäœäžã®[åå¿è
]çè«è¬çŸ©ã§é²è¡ããäºå®ã§ã ðð»)
- OS: Ubuntu or Linux
- ãã·ã³ä»æ§
- Aws t2.medium 2 Core 4GB // ec2 free.tier åå ã¯å¯èœ
- Virtualboxãå©çšããŠäžèšOSã§åå å¯èœ
[å
¥éè
DOCKERè¬çŸ©]ããã¢ãŒã·ã§ã³ã€ãã³ãð
Dockerã«ã€ããŠè©³ããå匷ãããæ¹ã¯ããå
¥éè
ã®ããã®DockerãšDockerizingããã¹ã¿ãŒãããããšã匷ããå§ãããŸãã [ããã°ããŒã¿ãã€ãã©ã€ã³ãã¹ã¿ãŒ]åè¬ãããæ¹ã«ããã¢ãŒã·ã§ã³*é©çšããããŸãã
[ããã°ããŒã¿ã¯ã©ã¹ã¿æ§ç¯ããã±ãŒãž]ããŒã³ãããã¢ãŒã·ã§ã³ã€ãã³ãð
ãã£ããããã³ãŒãã©ããã§é«å¯çšæ§ãä¿èšŒãããããã°ããŒã¿ã¯ã©ã¹ã¿ãŒçŽæ¥æ§ç¯ãããæ¹ã«ããããã§ããæšªè¬çŸ©ãªã³ã¯ãã¯ãªãã¯åŸã [åè¬åãåãåãã]æ¬ã«ãID/Eã¡ãŒã«/ããã¢ãŒã·ã§ã³ç³è«ããŸãããšæ®ããŠãã ããã
ãã®è¬çŸ©ãäœã£ã人
J.PHILãã玹ä»ããŸãâïž