스쿱 에러....

Question

안녕하세요 스쿱에서 자꾸 에러가 발생하여 문의드립니다. 전 과정까지는 문제없이 진행하였으나, sqoop export --connect jdbc:postgresql://127.0.0.1:7432/postgres --username cloudera-scm --password acjZt03D1Q --table smartcar_symptom_info --export-dir /user/hive/warehouse/managed_smartcar_symptom_info 위 명령어를 진행하는데 계속 exception이 발생합니다... 원인도 불분명하여 일단 질문 남깁니다... sqoop export --connect jdbc:postgresql://127.0.0.1:7432/postgres --username cloudera-scm --password vsqBur5ygT --table smartcar_symptom_info --export-dir /user/ hive/warehouse/managed_smartcar_symptom_info Warning: /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 22/04/14 21:11:43 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.3.2 22/04/14 21:11:43 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 22/04/14 21:11:43 INFO manager.SqlManager: Using default fetchSize of 1000 22/04/14 21:11:43 INFO tool.CodeGenTool: Beginning code generation 22/04/14 21:11:44 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM "smartcar_symptom_info" AS t LIMIT 1 22/04/14 21:11:44 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce 22/04/14 21:11:49 ERROR orm.CompilationManager: Could not rename /tmp/sqoop-root/compile/6f489f0c3f57d16ddd9b99ee38d3c452/smartcar_symptom_info.java to /root/./smartcar_symptom_in fo.java. Error: Destination '/root/./smartcar_symptom_info.java' already exists 22/04/14 21:11:49 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/6f489f0c3f57d16ddd9b99ee38d3c452/smartcar_symptom_info.jar 22/04/14 21:11:50 INFO mapreduce.ExportJobBase: Beginning export of smartcar_symptom_info 22/04/14 21:11:50 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 22/04/14 21:11:52 WARN mapreduce.ExportJobBase: IOException checking input file header: java.io.EOFException 22/04/14 21:11:52 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 22/04/14 21:11:52 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 22/04/14 21:11:52 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 22/04/14 21:11:52 INFO client.RMProxy: Connecting to ResourceManager at server01.hadoop.com/192.168.56.101:8032 22/04/14 21:11:53 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/root/.staging/job_1649933706082_0003 22/04/14 21:11:54 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:54 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:54 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:54 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:55 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:55 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:55 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:55 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:55 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:56 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:57 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:57 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:58 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:58 WARN hdfs.DataStreamer: Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:986) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:640) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:810) 22/04/14 21:11:58 INFO input.FileInputFormat: Total input files to process : 2 22/04/14 21:11:58 INFO input.FileInputFormat: Total input files to process : 2 22/04/14 21:11:58 INFO mapreduce.JobSubmitter: number of splits:5 22/04/14 21:11:58 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 22/04/14 21:11:58 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 22/04/14 21:11:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1649933706082_0003 22/04/14 21:11:58 INFO mapreduce.JobSubmitter: Executing with tokens: [] 22/04/14 21:11:59 INFO conf.Configuration: resource-types.xml not found 22/04/14 21:11:59 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 22/04/14 21:11:59 INFO impl.YarnClientImpl: Submitted application application_1649933706082_0003 22/04/14 21:11:59 INFO mapreduce.Job: The url to track the job: http://server01.hadoop.com:8088/proxy/application_1649933706082_0003/ 22/04/14 21:11:59 INFO mapreduce.Job: Running job: job_1649933706082_0003 22/04/14 21:12:10 INFO mapreduce.Job: Job job_1649933706082_0003 running in uber mode : false 22/04/14 21:12:10 INFO mapreduce.Job: map 0% reduce 0% 22/04/14 21:12:31 INFO mapreduce.Job: map 100% reduce 0% 22/04/14 21:12:32 INFO mapreduce.Job: Job job_1649933706082_0003 failed with state FAILED due to: Task failed task_1649933706082_0003_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 22/04/14 21:12:32 INFO mapreduce.Job: Counters: 12 Job Counters Failed map tasks=2 Killed map tasks=3 Launched map tasks=4 Data-local map tasks=4 Total time spent by all maps in occupied slots (ms)=74538 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=74538 Total vcore-milliseconds taken by all map tasks=74538 Total megabyte-milliseconds taken by all map tasks=76326912 Map-Reduce Framework CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 22/04/14 21:12:32 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 22/04/14 21:12:32 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 40.4452 seconds (0 bytes/sec) 22/04/14 21:12:32 INFO mapreduce.ExportJobBase: Exported 0 records. 22/04/14 21:12:32 ERROR mapreduce.ExportJobBase: Export job failed! 22/04/14 21:12:32 ERROR tool.ExportTool: Error during export: Export job failed! at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:444) at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:930) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:93) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:112) at org.apache.sqoop.Sqoop.run(Sqoop.java:146) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:182) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:233) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:242) at org.apache.sqoop.Sqoop.main(Sqoop.java:251)

Big.D · Answer

안녕하세요! 고준호님! 이제 거의 파일럿 프로젝트 마지막 단계까지 진행 하셨군요!!! 조금만더 화이팅 입니다. ㅎㅎ 우선 Sqoop이 PostgreSQL로 Connection 문제가 아닌가 의심스러운데요.. Sqoop 익스포트 명령전 PostgreSQL JDBC라이브러리를 Sqoop으로 Copy 하는 아래 명령에 문제가 없었는지 확인 부탁 드립니다. $ cp /opt/cloudera/parcels/CDH/jars/postgresql-*.jar /opt/cloudera/parcels/CDH/lib/sqoop/lib - 빅디 드림