OGG-15051|OGG 同步 Oracle 到 Kafka 时遇到的一个错误

七月 28, 2022 2:28:43 下午 oracle.goldengate.datasource.UserExitDataSource createColumnValue
严重: Unable to decode column 57 : Input length = 1
java.nio.charset.MalformedInputException: Input length = 1
  at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
  at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:816)
  at oracle.goldengate.datasource.UserExitDataSource.createColumnValue(UserExitDataSource.java:1019)

Exception in thread "main" oracle.goldengate.util.GGException: Unable to decode column 57 : Input length = 1
  at oracle.goldengate.datasource.UserExitDataSource.createColumnValue(UserExitDataSource.java:1098)

Source Context :
  SourceModule            : [gglib.ggdal.adapter.java]
  SourceID                : [/scratch/aime/adestore/views/aime_adc4150560/oggcore/OpenSys/src/gglib/ggdal/Adapter/Java/JavaAdapter.cpp]
  SourceMethod            : [HandleJavaException]
  SourceLine              : [246]
  ThreadBacktrace         : [20] elements
                          : [/goldengate/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f7b0b15c0ae]]
                          : [/goldengate/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x6ac) [0x7f7b0b14c9bc]]
                          : [/goldengate/libgglog.so(_MSG_String(CSourceContext*, int, char const*, CMessageFactory::MessageDisposition)+0x39) [0x7f7b0b13ad19]]
                          : [/goldengate/libggjava.so(+0x2e9e7) [0x7f7b013d99e7]]
                          : [/goldengate/replicat(GenericImpl::Write(ObjectMetadata*, std_rec_hdr_def const*, ggs::gglib::ggdal::CDALRecord&)+0x52) [0x816832]]
                          : [/goldengate/replicat(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x5e) [0x7e2d8e]]
                          : [/goldengate/replicat(main+0x3b) [0x6e7e0b]]
                          : [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f7b037c63d5]]
                          : [/goldengate/replicat() [0x550831]]

2022-07-28 14:28:43  ERROR   OGG-15051  Java or JNI exception:
oracle.goldengate.util.GGException: Unable to decode column 57 : Input length = 1.

--源端删除表定义文件
rm -rf /goldengate/dirdef/goldengate.def
--重新生成表定义文件
/goldengate/defgen paramfile /goldengate/dirprm/test_ogg.prm

--test_ogg.prm 文件中提前写好需要同步到目标端的表
more /goldengate/dirprm/test_ogg.prm
defsfile /goldengate/dirdef/goldengate.def,FORMAT RELEASE 11.2
userid goldengate,password ogg12345
TABLE ds.test01;
TABLE prod.T_CRE;
TABLE prod.T_CRE_DOC_DATUM_RELATION;
TABLE prod.T_CRE_DOC_AUDIT_DETAIL;
TABLE prod.T_CRE_DOC_PROGRESS;
TABLE prod.T_CRE_DOC_DATUM;
---------------------------------
然后将生成的 goldengate.def scp 到目标端，然后重启 start rep1 即可。

--源端
Oracle GoldenGate Command Interpreter for Oracle
Version 12.3.0.1.4 OGGCORE_12.3.0.1.0_PLATFORMS_180415.0359_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Apr 15 2018 21:16:09
Operating system character set identified as UTF-8.

--目标端
Oracle GoldenGate for Big Data
Version 12.3.2.1.1 (Build 005)

Oracle GoldenGate Command Interpreter
Version 12.3.0.1.2 OGGCORE_OGGADP.12.3.0.1.2_PLATFORMS_180712.2305
Linux, x64, 64bit (optimized), Generic on Jul 13 2018 00:46:09
Operating system character set identified as UTF-8.

The message is 2885830 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.

vi custom_kafka_producer.properties

max.request.size = 5024000
send.buffer.bytes = 5024000

原因：
21-06-12 23:53:24 ERROR OGG-15051 Java 或 JNI exception: oracle.goldengate.util.GGException: Unable to decode column 1: Input length = 2。

我们需要继续采用这种策略:通过 CHARMAP 指定 UTF-16 到 UTF-16 映射的覆盖，以便将 U+FFFE 更改为 U+FFFD(替换字符)。
这应该用替换字符替换坏字符 U+FFFE。

The following is how we do it:

Step 1: Create a text file and name it myfile containing the following three
----步骤1:创建一个文本文件，命名为 myfile，包含以下三个文件(PS:文中没有说明此文件存放于何处？)
lines:
SOURCECHARSET utf-16be
TARGETCHARSET utf-16be
\xff\xfe \xff\xfd

Step 2: In replicat .prm file, add this via CHARMAP prior to the
REPLACEBADCHAR SUBSTITUTE ? FORCECHECK:
----步骤2:在 replicate.prm 文件中，通过 CHARMAP 将其添加到 REPLACEBADCHAR SUBSTITUTE ? FORCECHECK:

CHARMAP myfile
REPLACEBADCHAR SUBSTITUTE ? FORCECHECK

This should replace the bad character U+FFFE with a replacement character.
----这应该用替换字符替换坏字符 U+FFFE。

more ./dirprm/myfile
SOURCECHARSET utf-16be
TARGETCHARSET utf-16be
\xff\xfe \xff\xfd


REPLACEBADCHAR SUBSTITUTE ? FORCECHECK:

-- 进入 OGG 查看当前的 trail 文件编号及 RBA 号。
./ggsci
GGSCI (jieke-ogg-test) 4> Info rep1
REPLICAT   REP1     Last Started 2022-07-25 10:44   Status ABENDED
Checkpoint Lag       00:00:00 (updated 00:00:07 ago)
Process ID           8960
Log Read Checkpoint  File /soft/dirdat/pd000000016
                     2022-07-29 23:06:46.000355  RBA 128554753
查看到当前 trail 文件编号是 16，RBA 是 128554753,记住这个信息，然后使用 logdump 工具查看。
./logdump
----开启详细日志显示
Logdump 1277 > ghdr on
Logdump 1278 > detail on
----打开 trail 文件
Logdump 1279 > open /soft/dirdat/pd000000016
----跳到上面查看到的异常的 RBA 号 128554753，使用 n 查看当前正在经历的 DDL 或者 DML 操作。
Logdump 1280 > pos 128554753
Logdump 1281 > n

----再次按 n 回车，查看下一条操作及 RBA 是多少（多数情况下会多次执行 n，跳过多条操作），记录下当前的 RBA 是 131749957

./ggsci
----跳过之前的 128554753 操作，跳到 131749957
GGSCI (jieke-ogg-test) 2> alter rep rep1 extseqno 16,extrba 131749957
GGSCI (jieke-ogg-test) 3> start rep1 with NOFILTERDUPTRANSACTIONS
GGSCI (jieke-ogg-test) 4> start rep1