spark用kafka批量读取时的相关问题-创新互联

语法书写错误报错:查了半天最后发现这句写错了,盯着挨个对才发现冒号写成了逗号,改好即可

公司专注于为企业提供做网站、成都网站设计、微信公众号开发、商城建设，成都小程序开发，软件按需搭建网站等一站式互联网企业服务。凭借多年丰富的经验，我们会仔细了解各客户的需求而做出多方面的分析、设计、整合，为客户设计出具风格及创意性的商业解决方案，创新互联更提供一系列网站制作和网站推广的服务。

pyspark.sql.utils.IllegalArgumentException: Expected e.g. {"topicA":{"0":23,"1":-1},"topicB":{"0":-2}}, got {"aaa":{"0":10,"1":700,"2",700}}

分区数目不对,偏移量不对报错:更改分区数目,偏移量位置即可,或者将failOnDataLoss设置为False,此时程序结果会出现但依然会报错

java.lang.IllegalStateException: Cannot fetch offset 1 (GroupId: spark-kafka-relation-928e8f8b-5271-4295-a3e2-7eeb7965940f-executor, TopicPartition: aaa-0). 
Some data may have been lost because they are not available in Kafka any more; either the
 data was aged out by Kafka or the topic may have been deleted before all the data in the
 topic was processed. If you don't want your streaming query to fail on such cases, set the
 source option "failOnDataLoss" to "false".

批量读取时

py4j.protocol.Py4JJavaError: An error occurred while calling o45.showString.
: java.lang.AssertionError: assertion failed: If startingOffsets contains specific offsets, you must specify all TopicPartitions.

分区数目必须写全,否则会出现这个错误

你是否还在寻找稳定的海外服务器提供商？创新互联www.cdcxhl.cn海外机房具备T级流量清洗系统配攻击溯源，准确流量调度确保服务器高可用性，企业级服务器适合批量采购，新人活动首月15元起，快前往官网查看详情吧

标题名称：spark用kafka批量读取时的相关问题-创新互联
URL分享：http://csruizhi.cn/article/ijphp.html