文档建议反馈控制台
首页
学习
活动
专区
圈层
工具
MCP广场
文章/答案/技术大牛
发布
首页
学习
活动
专区
圈层
工具
MCP广场
返回腾讯云官网
社区首页 >问答首页 >kafka jdbc接收器连接器:在创建或更新一个连接器时重新创建所有其他连接器

问kafka jdbc接收器连接器:在创建或更新一个连接器时重新创建所有其他连接器
EN

Stack Overflow用户
提问于 2019-06-12 14:38:33
回答 1查看 146关注 0票数 0

我有大约500个连接器，每次我创建一个新的连接器，我必须等待很长时间才能重新创建以前的连接器。它太慢了。

sink.properties如下

代码语言：javascript
运行
复制
{
    "name": "saas-saas_order-worder_fee_price-sink",
    "config": {
        "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
        "errors.log.include.messages": "true",
        "tasks.max": "1",
        "topics": "saasosvc-worder_fee_price",
        "transforms": "unwrap",
        "auto.evolve": "true",
        "name": "saas-saas_order-worder_fee_price-sink",
        "transforms.unwrap.type": "io.debezium.transforms.UnwrapFromEnvelope",
        "auto.create": "true",
        "connection.url": "jdbc:postgresql://postgres:5432/saas?user=postgresuser&password=postgrespw",
        "errors.log.enable": "true",
        "insert.mode": "upsert",
        "pk.mode": "record_value",
        "pk.fields": "id"
    }
}

部分日志如下

代码语言：javascript
运行
复制
> 2019-06-12 05:44:42,242 INFO   || 
> WorkerSourceTask{id=saas-com_dyrs_mtsp_furnituresaleservice-source-0}
> flushing 0 outstanding messages for offset commit  
> [org.apache.kafka.connect.runtime.WorkerSourceTask]
>     2019-06-12 05:44:45,950 INFO   ||  WorkerSinkTask{id=saas-appconstruction-appconstruction_requestrecord-sink-0}
> Committing offsets asynchronously using sequence number 20376:
> {appssvc-appconstruction_requestrecord-0=OffsetAndMetadata{offset=2340,
> leaderEpoch=null, metadata=''}}  
> [org.apache.kafka.connect.runtime.WorkerSinkTask]
>     2019-06-12 05:45:18,242 INFO   ||  Connector saas-saas_order-worder_fee_price-sink config updated  
> [org.apache.kafka.connect.runtime.distributed.DistributedHerder]
>     2019-06-12 05:45:18,745 INFO   ||  Rebalance started   [org.apache.kafka.connect.runtime.distributed.DistributedHerder]
>     2019-06-12 05:45:18,747 INFO   ||  Stopping connector saas-order_base-product_vs_install-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,747 INFO   ||  Stopping connector saas-dyrs_complainservice-complain_type_record-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-appcustomerself-appcustomerself_appversionuser-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-com_dyrs_mtsp_changeservice-replace_product-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-com_dyrs_mtsp_designservice-version_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-dyrs_settlementservice-source  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-appcustomerself-appcustomerself_advice-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-im-im_accesstoken-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,761 INFO   ||  172.31.206.219 - - [12/Jun/2019:05:45:18 +0000] "POST /connectors/ HTTP/1.1" 201 604  549
> [org.apache.kafka.connect.runtime.rest.RestServer]
>     2019-06-12 05:45:18,775 INFO   ||  Stopped connector saas-im-im_accesstoken-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,776 INFO   ||  Stopping connector saas-saas_order-dd_order-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,778 INFO   ||  Stopped connector saas-dyrs_settlementservice-source  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,780 INFO   ||  Stopping connector saas-dyrs_settlementservice-balance_info_detail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,781 INFO   ||  Stopped connector saas-com_dyrs_mtsp_designservice-version_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,781 INFO   ||  Stopping connector saas-dyrs_authorityservice-pro_city_area-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,781 INFO   ||  Stopped connector saas-dyrs_complainservice-complain_type_record-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,781 INFO   ||  Stopping connector saas-constructionconfig-constructionconfig_checkmanagedetailstandard-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,799 INFO   ||  Stopped connector saas-saas_order-dd_order-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,799 INFO   ||  Stopping connector saas-saas_order-goods-sink   [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,857 INFO   ||  Stopped connector saas-saas_order-goods-sink   [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,857 INFO   ||  Stopping connector saas-com_dyrs_mtsp_businessopportunityservice-business_process_operator-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,860 INFO   ||  Stopped connector saas-appcustomerself-appcustomerself_appversionuser-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,860 INFO   ||  Stopping connector saas-com_dyrs_mtsp_changeservice-custom_wood_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,862 INFO   ||  Stopped connector saas-order_base-product_vs_install-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,863 INFO   ||  Stopping connector saas-com_dyrs_mtsp_quotelistservice-pre_quote_detail_tab_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,865 INFO   ||  Stopped connector saas-com_dyrs_mtsp_changeservice-replace_product-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,868 INFO   ||  Stopping connector saas-com_dyrs_mtsp_quoteservice-program_template_tab-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,870 INFO   ||  Stopped connector saas-dyrs_settlementservice-balance_info_detail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,870 INFO   ||  Stopping connector saas-dyrs_settlementservice-balance_config-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,866 INFO   ||  Stopped connector saas-appcustomerself-appcustomerself_advice-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,873 INFO   ||  Stopping connector saas-dyrs_authorityservice-account_login_fail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,872 INFO   ||  Stopped connector saas-dyrs_authorityservice-pro_city_area-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,873 INFO   ||  Stopping connector saas-com_dyrs_mtsp_dataexpansionservice-dynamic_tabpk-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,875 INFO   ||  Stopped connector saas-constructionconfig-constructionconfig_checkmanagedetailstandard-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,875 INFO   ||  Stopping connector saas-com_dyrs_mtsp_promotionservice-offer_content-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,884 INFO   ||  Stopped connector saas-com_dyrs_mtsp_businessopportunityservice-business_process_operator-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,885 INFO   ||  Stopping connector saas-com_dyrs_mtsp_businessopportunityservice-personal_clue_limit-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,891 INFO   ||  Stopped connector saas-com_dyrs_mtsp_changeservice-custom_wood_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,891 INFO   ||  Stopping connector saas-dyrs_settlementservice-balance_operation-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,902 INFO   ||  Stopped connector saas-com_dyrs_mtsp_quotelistservice-pre_quote_detail_tab_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,902 INFO   ||  Stopping connector saas-order_base-unit_vs_unit-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,904 INFO   ||  Stopped connector saas-com_dyrs_mtsp_dataexpansionservice-dynamic_tabpk-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,905 INFO   ||  Stopped connector saas-com_dyrs_mtsp_quoteservice-program_template_tab-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,907 INFO   ||  Stopping connector saas-com_dyrs_mtsp_fieldassociationservice-source  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,904 INFO   ||  Stopped connector saas-dyrs_authorityservice-account_login_fail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,907 INFO   ||  Stopping connector saas-customerself-customerself_msgsendrecorddetail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,908 INFO   ||  Stopping connector saas-finereport-constructionprocess_constructioninfoprocesscheck-sink 
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,913 INFO   ||  Stopped connector saas-dyrs_settlementservice-balance_config-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,134 INFO   ||  Stopped connector saas-order_base-unit_vs_unit-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,136 INFO   ||  Stopping connector saas-com_dyrs_mtsp_furnituresaleservice-fur_sale_contract_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,139 INFO   ||  Stopping connector saas-com_dyrs_mtsp_businessopportunityservice-call_record-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,154 INFO   ||  Stopped connector saas-com_dyrs_mtsp_furnituresaleservice-fur_sale_contract_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,154 INFO   ||  Stopping connector saas-com_dyrs_mtsp_fieldassociationservice-pre_filed-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,166 INFO   ||  Stopped connector saas-com_dyrs_mtsp_promotionservice-offer_content-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,166 INFO   ||  Stopping connector saas-runmonitoring-runmonitoring_browsinghistory-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,167 INFO   ||  Stopped connector saas-com_dyrs_mtsp_fieldassociationservice-pre_filed-sink  
> [org.apache.kafka.connect.runtime.Worker]

谢谢

apache-kafka
apache-kafka-connect
confluent-platform
EN

回答 1

Stack Overflow用户

回答已采纳

发布于 2019-06-12 16:46:41

“停止世界”的再平衡是Kafka Connect的一个已知问题。好消息是，在Apache Kafka 2.3中到期的KIP-415中，有了一个新的增量重新平衡功能，这应该会让事情变得更好。

与此同时，唯一的其他选择是对您的Kafka Connect工作者进行分区，并拥有单独的集群，将500个连接器划分为不同的集群(例如，按功能、类型或其他任意因素)。

票数 0
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：

https://stackoverflow.com/questions/56556007

复制
相关文章

相似问题

领券

腾讯云开发者

扫码关注腾讯云开发者

扫码关注腾讯云开发者

领取腾讯云代金券

热门产品

热门推荐

更多推荐

Copyright © 2013 - 2025 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有 

深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 深公网安备号 44030502008569

腾讯云计算（北京）有限责任公司 京ICP证150476号 |  京ICP备11018762号 | 京公网安备号11010802020287

问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档