안녕하세요, 빅디님! 강의 듣다가 궁금한점이 생겼는데요, 처음에 클라우데라 매니저로 하둡환경을 구성한 후에 클라우데라 매니저만 삭제하면 주키퍼, 얀, 스파크 등 설치/설정한 아키텍처가 함께 삭제되나요? 서비스를 각각 설치하고 구성하려니 많이 번거로워서 처음에만 클라우데라 매니저를 이용하고 무료사용기간 만료되면 삭제 후에 각 서비스를 따로 관리할수있을까 해서요 ㅎㅎ
안녕하세요, 강사님. 빅데이터 파일럿 프로젝트 강의를 수강 중입니다. 현재 강의 내용대로 Server01과 Server02를 모두 실행했고, http://server.hadoop.com:7180 주소로 클라우데라 매니저에 접속을 시도했는데 웹 페이지 자체가 열리지 않습니다. 또한 강의 자료에서 로그인 아이디와 패스워드 정보도 찾을 수 없었습니다. 혹시 클라우데라 매니저 기본 접속 주소 기본 로그인 아이디와 패스워드 접속이 되지 않을 때 점검해야 할 설정 을 알려주실 수 있을까요? 처음 환경을 구성하는 단계라서 안내해 주시면 큰 도움이 될 것 같습니다. 감사합니다.
안녕하세요 빅디님 ! gcc 설치 중에 오류가 나서 yum repository 삭제 후 다시 시도해 보았는데, 계속 오류가 나서 질문 드립니다. ㅠㅠ 어떤게 문제일까요..? yum repo 삭제는 다음과 같이 진행 하였습니다. [root@server02 ~]# cd /etc/yum.repos.d/ [root@server02 yum.repos.d]# rm -rf remi.* remi-* [root@server02 yum.repos.d]# [root@server02 yum.repos.d]# cd /var/cache/yum/ [root@server02 yum]# rm -rf x86_64 [root@server02 yum]# [root@server02 yum]# yum clean headers Loaded plugins: fastestmirror, refresh-packagekit, security Cleaning repos: base cloudera-manager extras updates 0 header files removed [root@server02 yum]# yum clean packages Loaded plugins: fastestmirror, refresh-packagekit, security Cleaning repos: base cloudera-manager extras updates 0 package files removed [root@server02 yum]# yum clean metadata Loaded plugins: fastestmirror, refresh-packagekit, security Cleaning repos: base cloudera-manager extras updates 0 metadata files removed 0 sqlite files removed 0 metadata files removed yum install -y gcc* 명령어 입력시 발생하는 오류 입니다. [root@server02 ~]# yum install -y gcc* Loaded plugins: fastestmirror, refresh-packagekit, security Setting up Install Process Loading mirror speeds from cached hostfile Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os&infra=stock error was 14: PYCURL ERROR 6 - "Couldn't resolve host 'mirrorlist.centos.org'" Error: Cannot find a valid baseurl for repo: base 추가로, CentOS-Base.repo 파일 내용 첨부드립니다. [root@server02 yum.repos.d]# cat CentOS-Base.repo # CentOS-Base.repo # # The mirror system uses the connecting IP address of the client and the # update status of each mirror to pick mirrors that are updated to and # geographically close to the client. You should use this for CentOS updates # unless you are manually picking other mirrors. # # If the mirrorlist= does not work for you, as a fall back you can try the # remarked out baseurl= line instead. # # [base] name=CentOS-$releasever - Base mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra #baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/ gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 #released updates [updates] name=CentOS-$releasever - Updates mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra #baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/ gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 #additional packages that may be useful [extras] name=CentOS-$releasever - Extras mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra #baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/ gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 #additional packages that extend functionality of existing packages [centosplus] name=CentOS-$releasever - Plus mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra #baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/ gpgcheck=1 enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 #contrib - packages by Centos Users [contrib] name=CentOS-$releasever - Contrib mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib&infra=$infra #baseurl=http://mirror.centos.org/centos/$releasever/contrib/$basearch/ gpgcheck=1 enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 감사합니다.
안녕하세요 선생님 클러스터 설정 시 오류가 발생하여 질문 드립니다. ㅜㅜ 다른 질문 글들을 참고하여 Cluster 1 삭제 후 재설치도 해보았고, putty로 접속하여 rm -rf /dfs/nn 명령어로 디렉토리 삭제 후 cluster 재설치도 해보았는데 계속 오류가 발생합니다. 원인과 해결 방법이 있을지 문의 드립니다.. * stderr로그 일부 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /tmp. Name node is in safe mode. The reported blocks 0 has reached the threshold 0.9990 of total blocks 0. The number of live datanodes 0 needs an additional 1 live datanodes to reach the minimum number 1. Safe mode will be turned off automatically once the thresholds have been reached. NamenodeHostName:server01.hadoop.com at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.newSafemodeException(FSNamesystem.java:1448) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1435) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3100) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1123) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:696) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1499) at org.apache.hadoop.ipc.Client.call(Client.java:1445) at org.apache.hadoop.ipc.Client.call(Client.java:1355) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy9.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:640) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy10.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2339) ... 18 more
select car_number, avg(battery) as battery_avg from SmartCar_Status_Info where battery < 60 group by car_number; Hue에서 이 쿼리 돌리면 아래 에러가 뜹니다.. chat gpt 통해서 hue_safety_valve.ini에 대한 Hue 서비스 고급 구성 스니펫에 아래 내용도 추가했는데 해결이 안됩니다. 이것 때문에 더이상 진도를 못 나가고 있어서 도움 부탁드려요. [yarn_clusters] [[default]] log_url_template= http://server02.hadoop.com:8042/node/containerlogs/${container_id}/${user} 500 Server Error: Server Error for url: http://server02.hadoop.com:8042/node/containerlogs/admin?doAs=admin Error 500 Server Error HTTP ERROR 500 Problem accessing /node/containerlogs/admin. Reason: Server ErrorCaused by:java.lang.ArrayIndexOutOfBoundsException: 4 at org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebAppFilter.containerLogPageRedirectPath(NMWebAppFilter.java:81) at org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebAppFilter.doFilter(NMWebAppFilter.java:62) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829) at com.google .inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at com.google .inject.servlet.GuiceFilter$ 1.call (GuiceFilter.java:133) at com.google .inject.servlet.GuiceFilter$ 1.call (GuiceFilter.java:130) at com.google .inject.servlet.GuiceFilter$ Context.call (GuiceFilter.java:203) at com.google .inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.security .http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.security .authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644) at org.apache.hadoop.security .authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1553) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security .SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:539) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io .FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io .SelectChannelEndPoint$ 2.run (SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run (ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$ 2.run (QueuedThreadPool.java:589) at java.lang.Thread.run (Thread.java:748) (error 500)
안녕하세요, 강사님. 강의대로 진행했음에도 불구하고 계속 사진처럼 빨갛게 뜹니다. 제가 처음에 클라우데라 매니저 설치할 때 실수를 반복하여 클러스터를 세 번 정도 설치했다가 삭제하고 마지막으로 한 번 더 제대로 설치했는데, 이전에 설치했던 parcel 혹은 다른 잔여 파일이 남아서 그런 걸까요? parcel을 설치할 때마다 c 디스크 공간이 줄어들었던 걸 보면 잔여 파일이 남은 것 같긴 한데 어떻게 삭제해야 할까요?
25/02/26 18:11:20 INFO zookeeper.ZooKeeper: Session: 0x0 closed 25/02/26 18:11:20 INFO zookeeper.ClientCnxn: EventThread shut down 25/02/26 18:11:20 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Closed. Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:242) at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:238) at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:238) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:238) at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96) at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1824) at kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:262) at kafka.admin.TopicCommand$.main(TopicCommand.scala:53) at kafka.admin.TopicCommand.main(TopicCommand.scala) 카프카 토픽생성에서 이런 에러가 뜹니다...