도저히 모르겠습니다.......왜이런지 알수가 없습니다.
hue -> workflow 에서 작업을 진행 하였습니다.
작업은 sqoop을 이용해서 만들어 보았습니다. 명령어는 아래와 같습니다.
sqoop import --connect jdbc:mysql://디비서버:3306/디비이름 --username root --password "비밀번호" --table SQOOP_TEST1 --hive-import --hive-database jun --hive-table sqoop_test1 -m 1
위의 명령어를 등록하여 실행 했습니다.
그런데 hdfs에는 db에서 조회한 결과물 데이터가 저장이 되는 것을 확인했습니다.
하지만 hive에 데이터를 넣는 부분에서 문제가 발생합니다.
참고로 hue/oozie를 통하지 않고 서버 접속후 shell에서 명령어를 실행하면 정상 작동 합니다.
오로지 oozie사용의 문제 같습니다.아래의 에러 로그도 oozie의 log링크를 클릭해 추출한 오류 메세지 입니다.
2022-12-02 08:15:27,597 [main] INFO org.apache.sqoop.orm.CompilationManager - Writing jar file: /tmp/sqoop-hadoop/compile/5941ad26fedc6f6eda6f548c6ec6d322/SQOOP_TEST1.jar
2022-12-02 08:15:27,604 [main] WARN org.apache.sqoop.manager.MySQLManager - It looks like you are importing from mysql.
2022-12-02 08:15:27,604 [main] WARN org.apache.sqoop.manager.MySQLManager - This transfer can be faster! Use the --direct
2022-12-02 08:15:27,604 [main] WARN org.apache.sqoop.manager.MySQLManager - option to exercise a MySQL-specific fast path.
2022-12-02 08:15:27,604 [main] INFO org.apache.sqoop.manager.MySQLManager - Setting zero DATETIME behavior to convertToNull (mysql)
2022-12-02 08:15:27,613 [main] INFO org.apache.sqoop.mapreduce.ImportJobBase - Beginning import of SQOOP_TEST1
2022-12-02 08:15:27,645 [main] WARN org.apache.sqoop.mapreduce.JobBase - SQOOP_HOME is unset. May not be able to find all job dependencies.
2022-12-02 08:15:27,987 [main] INFO org.apache.sqoop.mapreduce.db.DBInputFormat - Using read commited transaction isolation
2022-12-02 08:15:39,838 [main] INFO org.apache.sqoop.mapreduce.ImportJobBase - Transferred 14 bytes in 12.1888 seconds (1.1486 bytes/sec)
2022-12-02 08:15:39,841 [main] INFO org.apache.sqoop.mapreduce.ImportJobBase - Retrieved 1 records.
2022-12-02 08:15:39,841 [main] INFO org.apache.sqoop.mapreduce.ImportJobBase - Publishing Hive/Hcat import job data to Listeners for table SQOOP_TEST1
2022-12-02 08:15:39,858 [main] INFO org.apache.sqoop.manager.SqlManager - Executing SQL statement: SELECT t.* FROM `SQOOP_TEST1` AS t LIMIT 1
2022-12-02 08:15:39,873 [main] INFO org.apache.sqoop.hive.HiveImport - Loading uploaded data into Hive
2022-12-02 08:15:40,000 [main] ERROR org.apache.sqoop.tool.ImportTool - Import failed: java.io.IOException: Cannot run program "hive": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:501)
at org.apache.sqoop.util.Executor.exec(Executor.java:76)
at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:382)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:165)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:155)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:412)
at org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:54)
at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:225)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:219)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:155)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:143)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 34 more
<<< Invocation of Sqoop command completed <<<
No child hadoop job is executed.
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:412)
at org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:54)
at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:225)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:219)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:155)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:143)
Caused by: java.lang.SecurityException: Intercepted System.exit(1)
at org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
at java.lang.Runtime.exit(Runtime.java:101)
at java.lang.System.exit(System.java:987)
at org.apache.sqoop.Sqoop.main(Sqoop.java:253)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:165)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:155)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
... 16 more
Intercepting System.exit(1)
안녕하세요! "kbg4676" 님!
기본좋은 월드컵 주말 즐기 시는 지요!!??
대~한민국!! 화이팅! 입니다. ^^
이제 파일럿 프로젝트 막판까지 오셨네요!!! "kbg4676"님도 끝까지 화이팅!!! 입니다.
혹시.. Oozie의 shared 라이브러리에 사용중인 DBMS의 jdbc 클라이언트 jar 파일을 업로드 하셨는지요?!
파일럿 환경에서 사용중인 Postgresql DB의 경우, 해당 jdbc 클라이언트를 /user/oozie/share/lib/lib_*/sqoop 경로에 아래의 명령처럼 업로드후 사용해야 합니다.
$ hdfs dfs -put /opt/cloudera/parcels/CDH/jars/postgresql-42.2.5.jar /user/oozie/share/lib/lib_20220907170653/sqoop
그리고 아래 명령은 우지 워크플로우에서 테스트 했던 import 명령입니다. 2번째 DB to Hive가 문의 주신 것과 같은 테스트 입니다.
이때 우지 액션을 사용하시면서 주의 하실것은요..
우지의 sqoop 액션에 아래처럼 "import.." 명령을 시작으로 입력 하셔야 합니다. 실수로 "sqoop import.."로 입력 하면 아니 되옵니다.
<DB to HDFS>
import --connect jdbc:postgresql://192.168.56.101:7432/postgres --username cloudera-scm --password vsqBur5ygT --table smartcar_symptom_info --target-dir /sqoop-test/sqoop-data-2 -m 1
<DB to Hive>
import --connect jdbc:postgresql://192.168.56.101:7432/postgres --username cloudera-scm --password vsqBur5ygT --table smartcar_symptom_info --hive-import --hive-database default --hive-table smartcar_symptom_info_2 -m 1
참고로 우지의 shared에 라이브러리를 추가하여 사용하는 경우 log4j 에러가 발생하는 경우가 있습니다.
추가한 jar가 log4j에 의존성 때문에 발생하는 문제 인데요...
관련 에러가 발생하면 log4j2.xml 기본 파일을 작성해서, 우지의 shared 라이브러리 경로(/user/oozie/share/lib/lib_20220907170653/sqoop)에 업로드 해주시면 됩니다~
-빅디 드림
답글
Our Oracle 1z0-1071-22 practice test is the most reliable solution to quickly prepare for your Oracle Designing Oracle Azure Infrastructure Solutions. We are certain that our Oracle 1z0-1071-22 practice exam will guide you to get certified on the first try. Here is how we serve you to prepare successfully.
답글