Я добавил в свою настройку spark пакет (в spark-default.conf), но когда я создаю новый сеанс с livy < / strong> это вызывает у меня проблемы (см. ошибку ниже) и сеанс и смерть. ps: когда я удаляю этот пакет, все работает нормально.
20/05/04 00:17:35 WARN RSCClient: Error stopping RPC.
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@6d493840(uncancellable)
at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:394)
at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
...........
Exception in thread "Thread-32" java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
........
at org.apache.livy.utils.LineBufferedStream$$anon$1.run(LineBufferedStream.scala:46)
20/05/04 00:17:36 WARN ContextLauncher: Child process exited with code 143.
20/05/04 00:17:36 ERROR SparkProcApp: job was killed by user
20/05/04 00:17:36 INFO InteractiveSession: Stopped InteractiveSession 0.
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleting InteractiveSession 0 because it was inactive for more than 3600000.0 ms.
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleting session 0
20/05/04 00:28:17 INFO InteractiveSession: Stopping InteractiveSession 0...
20/05/04 00:28:17 INFO InteractiveSession: Stopped InteractiveSession 0.
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleted session 0
Использую: cloudera hdp2.6.5:
- искра 2.3
- Ливий 0.7.0
- Hadoop 2.7
- lib без учителя (https://github.com/unsupervise/spark-tss)
шаг :
- livy conf => livy.spark.master пряжа-кластер
- spark-default conf => spark.jars.repositories https://dl.bintray.com/unsupervise/maven/
- spark-defaultconf => spark.jars.packages com.github.unsupervise: spark-tss: 0.1.1
