文档建议反馈控制台
首页
学习
活动
专区
圈层
工具
MCP广场
文章/答案/技术大牛
发布
首页
学习
活动
专区
圈层
工具
MCP广场
返回腾讯云官网
社区首页 >问答首页 >Debezium服务器错误: java.lang.OutOfMemoryError: Java堆空间

问Debezium服务器错误: java.lang.OutOfMemoryError: Java堆空间
EN

Stack Overflow用户
提问于 2022-06-01 21:50:40
回答 2查看 403关注 0票数 0

Debezium服务器：v1.9.0

MongoDB地图集：v4.2.20

运行在AWS上的与Fargate w/ 1GB CPU和4GB内存

概述： Debezium启动了一个初始快照，并将一些数据发送给，但是它在完成快照之前遇到了一个错误(如下)。我尝试将容器的内存增加到4GB，但不确定这是否是问题所在。我正在用28‘m的总数和1100万个文档来测试这个集合。

Debezium配置(在Terraform中)：

代码语言：javascript
运行
复制
      environment = [
        {
          "name" : "DEBEZIUM_SINK_TYPE",
          "value" : "kinesis"
        },
        {
          "name" : "DEBEZIUM_SINK_KINESIS_REGION",
          "value" : "us-east-1"
        },
        {
          "name" : "DEBEZIUM_SINK_KINESIS_CREDENTIALS_PROFILE",
          "value" : "default"
        },
        {
          "name" : "DEBEZIUM_SINK_KINESIS_ENDPOINT",
          "value" : "https://kinesis.us-east-1.amazonaws.com"
        },
        {
          "name" : "DEBEZIUM_SOURCE_CONNECTOR_CLASS",
          "value" : "io.debezium.connector.mongodb.MongoDbConnector"
        },
        {
          "name" : "DEBEZIUM_SOURCE_OFFSET_STORAGE_FILE_FILENAME",
          "value" : "data/offsets.dat"
        },
        {
          "name" : "DEBEZIUM_SOURCE_OFFSET_FLUSH_INTERVAL_MS",
          "value" : "0"
        },
        {
          "name" : "DEBEZIUM_SOURCE_MONGODB_NAME",
          "value" : "test"
        },
        {
          "name" : "DEBEZIUM_SOURCE_MONGODB_HOSTS",
          "value" : "test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017"
        },
        {
          "name" : "DEBEZIUM_SOURCE_MONGODB_SSL_ENABLED",
          "value" : "true"
        },
        {
          "name" : "DEBEZIUM_SOURCE_MONGODB_MEMBERS_AUTO_DISCOVER",
          "value" : "true"
        },
        {
          "name" : "DEBEZIUM_SOURCE_DATABASE_INCLUDE_LIST",
          "value" : "test"
        },
        {
          "name" : "DEBEZIUM_SOURCE_COLLECTION_INCLUDE_LIST",
          "value" : "test.testCollection"
        },
        {
          "name" : "DEBEZIUM_SOURCE_CAPTURE_MODE",
          "value" : "change_streams_update_full"
        },
        {
          "name" : "DEBEZIUM_SOURCE_DATABASE_HISTORY",
          "value" : "io.debezium.relational.history.FileDatabaseHistory"
        },
        {
          "name" : "DEBEZIUM_SOURCE_DATABASE_HISTORY_FILE_FILENAME",
          "value" : "history.dat"
        },
        {
          "name" : "QUARKUS_LOG_CONSOLE_JSON",
          "value" : "false"
        }
      ]
      secrets = [
        {
          "name" : "DEBEZIUM_SOURCE_MONGODB_USER",
          "valueFrom" : "${data.aws_secretsmanager_secret.test-debezium-read.arn}:username::"
        },
        {
          "name" : "DEBEZIUM_SOURCE_MONGODB_PASSWORD",
          "valueFrom" : "${data.aws_secretsmanager_secret.test-debezium-read.arn}:password::"
        }
      ]

Stacktrace:

代码语言：javascript
运行
复制
2022-06-01 18:22:23,976 ERROR [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Error while attempting to sync 'test-mongodb-shard-0.test.testCollection': : java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
    at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:172)
    at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538)
    at java.base/java.lang.StringBuffer.append(StringBuffer.java:317)
    at java.base/java.io.StringWriter.write(StringWriter.java:106)
    at org.bson.json.StrictCharacterStreamJsonWriter.write(StrictCharacterStreamJsonWriter.java:368)
    at org.bson.json.StrictCharacterStreamJsonWriter.writeStartObject(StrictCharacterStreamJsonWriter.java:204)
    at org.bson.json.LegacyExtendedJsonDateTimeConverter.convert(LegacyExtendedJsonDateTimeConverter.java:22)
    at org.bson.json.LegacyExtendedJsonDateTimeConverter.convert(LegacyExtendedJsonDateTimeConverter.java:19)
    at org.bson.json.JsonWriter.doWriteDateTime(JsonWriter.java:129)
    at org.bson.AbstractBsonWriter.writeDateTime(AbstractBsonWriter.java:394)
    at org.bson.codecs.DateCodec.encode(DateCodec.java:32)
    at org.bson.codecs.DateCodec.encode(DateCodec.java:29)
    at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
    at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:203)
    at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:217)
    at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:200)
    at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:217)
    at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:200)
    at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:217)
    at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:159)
    at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:46)
    at org.bson.Document.toJson(Document.java:453)
    at io.debezium.connector.mongodb.JsonSerialization.lambda$new$0(JsonSerialization.java:57)
    at io.debezium.connector.mongodb.JsonSerialization$$Lambda$521/0x0000000840448840.apply(Unknown Source)
    at io.debezium.connector.mongodb.JsonSerialization.getDocumentValue(JsonSerialization.java:89)
    at io.debezium.connector.mongodb.MongoDbSchema$$Lambda$580/0x00000008404ce840.apply(Unknown Source)
    at io.debezium.connector.mongodb.MongoDbCollectionSchema.valueFromDocumentOplog(MongoDbCollectionSchema.java:90)
    at io.debezium.connector.mongodb.MongoDbChangeSnapshotOplogRecordEmitter.emitReadRecord(MongoDbChangeSnapshotOplogRecordEmitter.java:68)
    at io.debezium.connector.mongodb.MongoDbChangeSnapshotOplogRecordEmitter.emitReadRecord(MongoDbChangeSnapshotOplogRecordEmitter.java:27)
    at io.debezium.pipeline.AbstractChangeRecordEmitter.emitChangeRecords(AbstractChangeRecordEmitter.java:42)
    at io.debezium.pipeline.EventDispatcher.dispatchSnapshotEvent(EventDispatcher.java:163)

我注意到，在快照期间，发送的记录数量和最后记录的偏移量似乎没有变化，而每条消息之间经过的时间越长。这似乎是一个指数退避的事情，但我不完全确定。

​

​

示例：

代码语言：javascript
运行
复制
2022-06-01 16:20:37,789 INFO  [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Beginning snapshot of 'test-mongodb-shard-0' at {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:20:37,804 INFO  [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0)   Exporting data for collection 'test-mongodb-shard-0.test.testCollection'
2022-06-01 16:20:42,983 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 717 records sent during previous 00:00:06.159, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:20:57,417 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:00:14.434, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:21:05,107 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
2022-06-01 16:21:16,624 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:00:19.207, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:21:35,107 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
2022-06-01 16:21:53,130 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:00:36.505, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:22:05,107 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:23:17,521 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:01:24.391, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:23:35,106 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:26:06,523 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:02:49.003, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:26:35,107 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:31:18,075 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:05:11.552, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:31:35,106 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:42:07,711 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:10:49.636, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:42:35,106 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 17:03:12,872 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:21:05.161, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 17:03:35,117 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 17:45:58,637 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:42:45.765, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 17:46:05,106 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 18:22:23,976 ERROR [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Error while attempting to sync 'test-mongodb-shard-0.test.testCollection': : java.lang.OutOfMemoryError: Java heap space
mongodb
amazon-kinesis
debezium
debezium-server
EN

回答 2

Stack Overflow用户

发布于 2022-06-06 22:16:20

除了将容器内存增加到4GB之外，还可以设置更大堆大小，例如，最大堆大小和初始堆大小可以设置为2GB：

代码语言：javascript
运行
复制
  -Xms2048m -Xmx2048m

如果问题继续下去，请执行以下步骤：

  1. 使用参数-XX:+HeapDumpOnOutOfMemoryError启动JVM --当程序进入OOM时，这将给您一个堆转储。
  2. 使用像visualVM这样的工具来分析获得的堆转储。这将有助于识别内存泄漏。
票数 3
EN

Stack Overflow用户

发布于 2022-07-20 17:25:11

不是真的回答我的任务，只是想分享我的经验。我也偶尔收到java.lang.OutOfMemoryError，并想找出是什么原因造成的。

我的设置：

代码语言：javascript
运行
复制
Debezium 1.9.5
Kafka 2.8
Docker container memory - 6Gi
Java heap - 4Gi both min and max
max.queue.size.in.bytes - 512Mi
max.batch.size - 16384

来自stdout的错误：

代码语言：javascript
运行
复制
2022-07-20T16:47:16.348943181Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - cdc.config"
2022-07-20T16:47:27.628395682Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - cdc.status"
2022-07-20T16:47:28.970536167Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | connector-producer-REDACTED-202207200823-0"
2022-07-20T16:47:33.787361085Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-3"
2022-07-20T16:47:45.067373810Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "server-timer"
2022-07-20T16:47:46.987669188Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | REDACTED-dbhistory"
2022-07-20T16:48:03.396881812Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-2"
2022-07-20T16:48:04.017710798Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-coordinator-heartbeat-thread | production"
2022-07-20T16:48:09.709036280Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "prometheus-http-1-3"
2022-07-20T16:48:14.667691706Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "mysql-cj-abandoned-connection-cleanup"
2022-07-20T16:48:17.182623196Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-62"
2022-07-20T16:48:25.227925660Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-58"
2022-07-20T16:48:43.598026645Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HTTP-Dispatcher"
2022-07-20T16:48:45.543984655Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-1"
2022-07-20T16:48:52.284810255Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "SourceTaskOffsetCommitter-1"
2022-07-20T16:48:56.992674380Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DistributedHerder-connect-1-1"
2022-07-20T16:49:18.691603140Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Session-HouseKeeper-47a3d56a-1"
2022-07-20T16:49:19.350459393Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - cdc.offset"
2022-07-20T16:49:26.256350455Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-admin-client-thread | adminclient-8"
2022-07-20T16:49:33.154845201Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-59"
2022-07-20T16:49:34.414951745Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-60"
2022-07-20T16:49:40.871967276Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-61"
2022-07-20T16:49:56.007111292Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "debezium-sqlserverconnector-REDACTED-change-event-source-coordinator"
2022-07-20T16:50:00.410800756Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-admin-client-thread | connector-adminclient-REDACTED-0"

一些背景：

我传输一个每行都是1000字节的表。有一个有用的max.queue.size.in.bytes设置，我希望它可以作为连接器的上限，但是我的仪表板显示，队列大小在大多数情况下只有122 my在30米到60英里之间波动。我的计算是1000*16384永远不会达到512米。

在这种情况下，连接器已经正常工作了大约3个小时，然后在几分钟内没有新的流数据。不久之后，OutOfMemory就出现了。

我还注意到，每次发生这种情况时，容器都会获得其允许的最大CPU (在我的例子中是4个核)。

票数 0
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：

https://stackoverflow.com/questions/72468443

复制
相关文章

相似问题

领券

腾讯云开发者

扫码关注腾讯云开发者

扫码关注腾讯云开发者

领取腾讯云代金券

热门产品

热门推荐

更多推荐

Copyright © 2013 - 2025 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有 

深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 深公网安备号 44030502008569

腾讯云计算（北京）有限责任公司 京ICP证150476号 |  京ICP备11018762号 | 京公网安备号11010802020287

问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档