Help & Documentation>TencentDB for MariaDB>Development Guide>Binlog Consumption Format

Binlog Consumption Format

Last updated: 2023-08-31 09:38:26

On this page:

Data subscription helps you get incremental data from TencentDB for MariaDB and TDSQL, so that you can flexibly process real-time incremental data based on your actual business needs.

Feature List

Data subscription is supported for TencentDB for MariaDB and TDSQL instances in public cloud.
Data subscription is supported for TencentDB for MariaDB and TDSQL instances in private cloud.

Data source type

TencentDB for MariaDB and TDSQL.

Message Format

The data subscription feature parses the binlog (row format) of instances and encapsulates binlog events into JSON-formatted messages uploaded to the Kafka cluster. Message types include DML events, GTID events, XID events, and QUERY events. DML events consist of insert, update, and delete events, representing changes to data rows; GTID events signify the start of a transaction; XID events indicate the transaction's commitment; QUERY events represent DDL statements.
The DML message format is as follows:
{
    "logtype": "mysqlbinlog",          // Log type, unique value: mysqlbinlog
    "eventtype": 23,                   // Event type code, corresponding to the binlog event type code in MySQL

    "eventtypestr": "insert",          /*Event type string, including insert, update, delete, gtid, xid, and query events.
                                        The INSERT, UPDATE, and DELETE events correspond to DML statements, while the TID event indicates the start of a transaction.
                                        xid event indicates the end of a transaction; query event indicates a DDL statement. */
    "db": "testdb",                    // Database name
    "table": "testtable",              // Table name
    "localip": "000.00.000.000",       // IP of the machine where the instance is located
    "localport": 0000,                 // Instance port
    "begintime": 1511350073,           // Transaction start time, the start time of the transaction containing the current event
    "gtid": "0-2670193178-726233561",  // GTID, the gtid of the transaction where the current event is located
    "event_index":"4",                // Represents the sequence number of this event within the transaction  
    "where":[                         //where field, indicating the values of each field before the row change

    ],
    "field": [                         // The field attribute represents the values of each field after the row has been modified.
        "1",
        "'name1'"
    ]
}
GTID message format is as follows:
{                                    // A GTID event represents the start of a transaction
    "logtype":"mysqlbinlog",
    "eventtype":33,
    "eventtypestr":"gtid",
    "db":"sysdb",
    "table":"statustableforhb",
    "localip":"10.231.23.241",
    "localport":8810,
    "begintime":1511419963,
    "gtid":"35be190b-d019-11e7-ab7a-a0423f32c225:469",
    "event_index":"1"
}
XID message format is as follows:
{                                    // An XID event represents that a transaction has been committed
    "logtype":"mysqlbinlog",
    "eventtype":16,
    "eventtypestr":"xid",
    "db":"testsummer",
    "table":"test_table1",
    "localip":"10.231.23.241",
    "localport":8810,
    "begintime":1511419963,
    "gtid":"35be190b-d019-11e7-ab7a-a0423f32c225:469",
    "event_index":"5",
    "xid":"11866"
}
QUERY message format is as follows:
{
    "logtype":"mysqlbinlog",                          
    "eventtype":2,
    "eventtypestr": "query", // QUERY event corresponds to DDL statements
    "db":"testsummer",
    "table":"statustableforhb",
    "localip":"10.231.23.241",
    "localport":8810,
    "begintime":1511419941,
    "gtid":"35be190b-d019-11e7-ab7a-a0423f32c225:452",
    "event_index":"2",
    "sql":"create table test_table1 (id int primary key,name varchar(20))"
}

Subscription method

Customers can obtain real-time data by accessing message events stored in the Kafka cluster. They can use Tencent Cloud's data subscription API to fetch messages in real-time and process them accordingly.
中国站 - English