kafka consumer에서 데이터를 못 읽는데요.

너무 감사합니다^^ 드디어 kafka에서 실행이 되네요. 얼마 남지 않은 강의 빨리 듣고, 수강평 남기도록 하겠습니다. 좋은 하루 되세요. ^^

kafka consumer에서 데이터를 못 읽는데요.

회사 프로젝트 때문에 뒷편 부터 강의를 신청했지만, 여전히 kafka가 말썽입니다. 초기설치를 하고 해도, flume batch는 전송을 잘 받는데, kafka한테 메시지를 주지 않는 것은 왜 일까요? virtual box 의 문제가 있을 수도 있다고 구글링을 해봤는데, 혹시 그 문제일까요? 현재 전 6.01을 쓰고 있거든요. 정말 이 문제 때문에, 정말 듣고 싶은 후속 강의를 듣지도 못하고 있네요... 구글링을 해봤는데, 어쩌면 아래 link에 있는cloudera 사용자와 마찬가지로, real time 적재가 불가능한 거 같습니다. https://community.cloudera.com/t5/Support-Questions/Problem-about-Configuring-Flume-as-Kafka-Consumer/td-p/61018/page/2

kafka consumer에서 데이터를 못 읽는데요.

자꾸 귀찮게 해드려서 죄송해서, 일단 그냥 계속 강의 진행하겠습니다.근데 SmartCar-Topic2를 했는데도, 여전히 real-time 데이터가 kafka에게 전송이 안되는 거 같습니다. 데이터 분석일을 3년 하고 있지만, 데이터를 수집 적재하는 것이 이렇게 어렵다는 것이 피부로 다가오는 군요.

kafka consumer에서 데이터를 못 읽는데요.

(사진) unkown error가 뜨는 것을 확인할 수가 있습니다. 두 개의 topic (SmartCar-Topic, SmartCar-Topic2) 둘 다 똑같은 unknown error를 가지고 있습니다.

kafka consumer에서 데이터를 못 읽는데요.

저 나름대로 정리를 해서, 처음부터 끝 까지 올려보았습니다. #create a kafka topic under the name of 'SmartCar-Topic' kafka-topics --create --zookeeper server02.hadoop.com:2181 --replication-factor 1 --partitions 1 --topic SmartCar-Topic #command lines for the list of Topics kafka-topics --zookeeper server02.hadoop.com:2181 --list(사진) #generate both batch and real data java -cp bigdata.smartcar.loggen-1.0.jar com.wikibook.bigdata.smartcar.loggen.CarLogMain 20160101 3 & java -cp bigdata.smartcar.loggen-1.0.jar com.wikibook.bigdata.smartcar.loggen.DriverLogMain 20160101 3 & #double-check the status of those generated data cd /home/pilot-pjt/working/SmartCar/ tail -f /home/pilot-pjt/working/driver-realtime-log/SmartCarDriverInfo.log #move the current file to the assigned directory mv /home/pilot-pjt/working/SmartCar/SmartCarStatusInfo_20160101.txt /home/pilot-pjt/working/car-batch-log #restart flume (사진) (사진)

선생님, 질문이 있습니다!

(사진) 일단 batch-log는 잘 받아들여 지는 거 같습니다. agent 이름의 오류를 수정하고 나서는 데이터가 들어왔다는 것은 확인했습니다. 근데 여전히 카프카가 말썽이네요. 오류도 없는데, 당췌 데이터가 올라오는 것을 볼 수가 없습니다. 그냥 batch로만 받는 방법은 없는 건가요? 정말 이 부분때문에 저도 많이 지치네요 ㅠ.ㅠ

kafka consumer에서 데이터를 못 읽는데요.

vim으로 실행하니 이러한 메시지가 나오더군요. 참고로 여러 방식으로 구글링도 해보고, 말씀해주신 되로 해보았는데도, 여전히 flume은 말썽이고, 반응이 없습니다. Kafka consumer도 real-time data를 읽어오지도 못하고 있습니다. 2021-04-26 01:33:16,126 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource 2021-04-26 01:33:16,126 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource 2021-04-26 01:33:16,127 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource 2021-04-26 01:33:16,129 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_Channel 2021-04-26 01:33:16,131 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource 2021-04-26 01:33:16,131 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource 2021-04-26 01:33:16,132 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_LoggerSink 2021-04-26 01:33:16,132 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource 2021-04-26 01:33:16,133 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource 2021-04-26 01:33:16,136 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource 2021-04-26 01:33:16,137 INFO org.apache.flume.conf.FlumeConfiguration: Added sinks: SmartCarInfo_LoggerSink DriverCarInfo_KafkaSink Agent: SmartCar_Agent 2021-04-26 01:33:16,138 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource 2021-04-26 01:33:16,138 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource 2021-04-26 01:33:16,139 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource 2021-04-26 01:33:16,139 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource 2021-04-26 01:33:16,140 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_Channel 2021-04-26 01:33:16,141 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_KafkaSink 2021-04-26 01:33:16,143 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_Channel 2021-04-26 01:33:16,143 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_KafkaSink 2021-04-26 01:33:16,147 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_Channel 2021-04-26 01:33:16,147 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource 2021-04-26 01:33:16,148 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_KafkaSink 2021-04-26 01:33:16,148 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource 2021-04-26 01:33:16,149 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource 2021-04-26 01:33:16,150 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_KafkaSink 2021-04-26 01:33:16,150 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_Channel 2021-04-26 01:33:16,150 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_KafkaSink 2021-04-26 01:33:16,151 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource 2021-04-26 01:33:16,153 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_Channel 2021-04-26 01:33:16,153 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource 2021-04-26 01:33:16,153 WARN org.apache.flume.conf.FlumeConfiguration: Agent configuration for 'SmartCar_Agent' has no configfilters. 2021-04-26 01:33:16,295 INFO org.apache.flume.conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [SmartCar_Agent] 2021-04-26 01:33:16,298 WARN org.apache.flume.node.AbstractConfigurationProvider: No configuration found for this host:tier1 2021-04-26 01:33:16,307 INFO org.apache.flume.node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} } 2021-04-26 01:33:16,462 INFO org.eclipse.jetty.util.log: Logging initialized @5107ms 2021-04-26 01:33:17,025 INFO org.eclipse.jetty.server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-05T06:11:46+09:00, git hash: 3ce520221d0240229c862b122d2b06c12a625732 2021-04-26 01:33:17,334 INFO org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@35ce6302{HTTP/1.1,[http/1.1]}{0.0.0.0:41414} 2021-04-26 01:33:17,343 INFO org.eclipse.jetty.server.Server: Started @5991ms 2021-04-26 01:33:17,398 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE=[flume-AGENT-08fa03562e178ba513af21531a0d067e], HOSTS=[server02.hadoop.com], ROLE_TYPE=[AGENT], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[flume], SERVICE_TYPE=[FLUME], LOG_LEVEL=[WARN], HOST_IDS=[28755065-292e-43b9-8e72-6e7cbbb66351], SEVERITY=[IMPORTANT]}, content=Agent configuration for 'SmartCar_Agent' has no configfilters., timestamp=1619368396153} 2021-04-26 01:41:26,101 INFO org.apache.flume.node.Application: Shutting down configuration: { sourceRunners:{} sinkRunners:{} channels:{} } 2021-04-26 01:41:26,116 INFO org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@35ce6302{HTTP/1.1,[http/1.1]}{0.0.0.0:41414} 2021-04-26 01:41:26,128 INFO org.apache.flume.lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 11 2021-04-26 01:41:26,134 INFO org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Configuration provider stopping 2021-04-26 01:45:14,770 INFO org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 2021-04-26 01:45:14,820 INFO org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/var/run/cloudera-scm-agent/process/36-flume-AGENT/flume.conf 2021-04-26 01:45:14,857 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_LoggerSink 2021-04-26 01:45:14,862 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource 2021-04-26 01:45:14,863 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_KafkaSink 2021-04-26 01:45:14,867 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource