sd4beatles

너무 감사합니다^^ 드디어 kafka에서 실행이 되네요. 얼마 남지 않은 강의 빨리 듣고,&nbsp; 수강평 남기도록 하겠습니다. 좋은 하루 되세요. ^^

kafka consumer에서 데이터를 못 읽는데요.

회사 프로젝트 때문에 뒷편 부터 강의를 신청했지만, 여전히 kafka가 말썽입니다. 초기설치를 하고 해도,&nbsp; flume batch는 전송을 잘 받는데, kafka한테 메시지를 주지 않는 것은 왜 일까요? virtual box 의 문제가 있을 수도 있다고 구글링을 해봤는데, 혹시 그 문제일까요? 현재 전 6.01을 쓰고 있거든요. 정말 이 문제 때문에, 정말 듣고 싶은 후속 강의를 듣지도 못하고 있네요...
구글링을 해봤는데, 어쩌면 아래 link에 있는cloudera 사용자와 마찬가지로, real time 적재가 불가능한 거 같습니다.&nbsp;
https://community.cloudera.com/t5/Support-Questions/Problem-about-Configuring-Flume-as-Kafka-Consumer/td-p/61018/page/2

자꾸 귀찮게 해드려서 죄송해서, 일단 그냥 계속 강의 진행하겠습니다.근데&nbsp; SmartCar-Topic2를 했는데도, 여전히 real-time 데이터가 kafka에게 전송이 안되는 거 같습니다. 데이터 분석일을 3년 하고 있지만, 데이터를 수집 적재하는 것이 이렇게 어렵다는 것이 피부로 다가오는 군요.

<img src="https://cdn.inflearn.com/public/files/posts/f38d450f-e026-4524-9910-98b46511b0ec/blob" alt="" />
unkown error가 뜨는 것을 확인할 수가 있습니다. 두 개의 topic (SmartCar-Topic, SmartCar-Topic2) 둘 다 똑같은 unknown error를 가지고 있습니다.&nbsp;

저 나름대로 정리를 해서, 처음부터 끝 까지 올려보았습니다.
#create a kafka&nbsp; topic under the name of 'SmartCar-Topic'
kafka-topics --create --zookeeper server02.hadoop.com:2181 --replication-factor 1 --partitions 1 --topic SmartCar-Topic
#command lines for the list of Topics
kafka-topics&nbsp; --zookeeper server02.hadoop.com:2181 --list<img src="https://cdn.inflearn.com/public/files/posts/da029e7a-f301-40bf-88b7-4b13efffd326/blob" alt="" />
#generate both batch and real data
java -cp bigdata.smartcar.loggen-1.0.jar com.wikibook.bigdata.smartcar.loggen.CarLogMain 20160101 3 &
java -cp bigdata.smartcar.loggen-1.0.jar com.wikibook.bigdata.smartcar.loggen.DriverLogMain 20160101 3 &

#double-check the status of those generated data
cd /home/pilot-pjt/working/SmartCar/
tail -f /home/pilot-pjt/working/driver-realtime-log/SmartCarDriverInfo.log
#move the current&nbsp; file to the assigned directory&nbsp;
mv /home/pilot-pjt/working/SmartCar/SmartCarStatusInfo_20160101.txt /home/pilot-pjt/working/car-batch-log
#restart flume
<img src="https://cdn.inflearn.com/public/files/posts/a6b0722e-15b2-4574-99a2-95d849d0cbd8/blob" alt="" />

<img src="https://cdn.inflearn.com/public/files/posts/6dfcbdef-651f-4b1a-a483-e46ff356a943/blob" alt="" />

<img src="https://cdn.inflearn.com/public/comments/c1e6a9a6-3b3f-49c3-a69f-97674cd5fb2c/blob" alt="" />

일단 batch-log는 잘 받아들여 지는 거 같습니다. agent 이름의 오류를 수정하고 나서는 데이터가 들어왔다는 것은 확인했습니다. 근데 여전히 카프카가 말썽이네요. 오류도 없는데,&nbsp; 당췌 데이터가 올라오는 것을 볼 수가 없습니다. 그냥 batch로만 받는 방법은 없는 건가요?&nbsp; 정말 이 부분때문에 저도 많이 지치네요 ㅠ.ㅠ

선생님, 질문이 있습니다!

vim으로 실행하니 이러한 메시지가 나오더군요. 참고로 여러 방식으로 구글링도 해보고, 말씀해주신 되로 해보았는데도, 여전히 flume은 말썽이고, 반응이 없습니다. Kafka consumer도 real-time data를 읽어오지도 못하고 있습니다.&nbsp;
2021-04-26 01:33:16,126 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource
2021-04-26 01:33:16,126 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource
2021-04-26 01:33:16,127 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource
2021-04-26 01:33:16,129 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_Channel
2021-04-26 01:33:16,131 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource
2021-04-26 01:33:16,131 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource
2021-04-26 01:33:16,132 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_LoggerSink
2021-04-26 01:33:16,132 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource
2021-04-26 01:33:16,133 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource
2021-04-26 01:33:16,136 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource
2021-04-26 01:33:16,137 INFO org.apache.flume.conf.FlumeConfiguration: Added sinks: SmartCarInfo_LoggerSink DriverCarInfo_KafkaSink Agent: SmartCar_Agent
2021-04-26 01:33:16,138 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource
2021-04-26 01:33:16,138 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource
2021-04-26 01:33:16,139 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource
2021-04-26 01:33:16,139 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource
2021-04-26 01:33:16,140 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_Channel
2021-04-26 01:33:16,141 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_KafkaSink
2021-04-26 01:33:16,143 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_Channel
2021-04-26 01:33:16,143 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_KafkaSink
2021-04-26 01:33:16,147 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_Channel
2021-04-26 01:33:16,147 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource
2021-04-26 01:33:16,148 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_KafkaSink
2021-04-26 01:33:16,148 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource
2021-04-26 01:33:16,149 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_TailSource
2021-04-26 01:33:16,150 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_KafkaSink
2021-04-26 01:33:16,150 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_Channel
2021-04-26 01:33:16,150 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_KafkaSink
2021-04-26 01:33:16,151 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource
2021-04-26 01:33:16,153 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_Channel
2021-04-26 01:33:16,153 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource
2021-04-26 01:33:16,153 WARN org.apache.flume.conf.FlumeConfiguration: Agent configuration for 'SmartCar_Agent' has no configfilters.
2021-04-26 01:33:16,295 INFO org.apache.flume.conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [SmartCar_Agent]
2021-04-26 01:33:16,298 WARN org.apache.flume.node.AbstractConfigurationProvider: No configuration found for this host:tier1
2021-04-26 01:33:16,307 INFO org.apache.flume.node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
2021-04-26 01:33:16,462 INFO org.eclipse.jetty.util.log: Logging initialized @5107ms
2021-04-26 01:33:17,025 INFO org.eclipse.jetty.server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-05T06:11:46+09:00, git hash: 3ce520221d0240229c862b122d2b06c12a625732
2021-04-26 01:33:17,334 INFO org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@35ce6302{HTTP/1.1,[http/1.1]}{0.0.0.0:41414}
2021-04-26 01:33:17,343 INFO org.eclipse.jetty.server.Server: Started @5991ms
2021-04-26 01:33:17,398 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE=[flume-AGENT-08fa03562e178ba513af21531a0d067e], HOSTS=[server02.hadoop.com], ROLE_TYPE=[AGENT], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[flume], SERVICE_TYPE=[FLUME], LOG_LEVEL=[WARN], HOST_IDS=[28755065-292e-43b9-8e72-6e7cbbb66351], SEVERITY=[IMPORTANT]}, content=Agent configuration for 'SmartCar_Agent' has no configfilters., timestamp=1619368396153}
2021-04-26 01:41:26,101 INFO org.apache.flume.node.Application: Shutting down configuration: { sourceRunners:{} sinkRunners:{} channels:{} }
2021-04-26 01:41:26,116 INFO org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@35ce6302{HTTP/1.1,[http/1.1]}{0.0.0.0:41414}
2021-04-26 01:41:26,128 INFO org.apache.flume.lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 11
2021-04-26 01:41:26,134 INFO org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Configuration provider stopping
2021-04-26 01:45:14,770 INFO org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
2021-04-26 01:45:14,820 INFO org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/var/run/cloudera-scm-agent/process/36-flume-AGENT/flume.conf
2021-04-26 01:45:14,857 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_LoggerSink
2021-04-26 01:45:14,862 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource
2021-04-26 01:45:14,863 INFO org.apache.flume.conf.FlumeConfiguration: Processing:DriverCarInfo_KafkaSink
2021-04-26 01:45:14,867 INFO org.apache.flume.conf.FlumeConfiguration: Processing:SmartCarInfo_SpoolSource

sd4beatles

Posts