The content of this page has been automatically translated by AI. If you encounter any problems while reading, you can view the corresponding content in Chinese.
Reconciliation is an auxiliary system that any billing system needs. Whether reconciliation is the main or side system for payment, it is necessary to reconcile during or after the payment to ensure billing accuracy. By introducing TDMQ for Pulsar, the timeliness of reconciliation is ensured and the critical path of transactions is not affected.
Encountered Issues
1. System Decoupling
Transactions involve multiple systems, and decoupling between systems is necessary to avoid mutual influence.
2. Data Arrival Time Difference
There is a time difference in data arrival between systems, ensuring that data arriving at different times can be aggregated and calculated.
3. Data Consistency
Ensuring data is not lost, avoiding abnormal reconciliation results due to data loss.
4. Cross-Region Data Transmission
Systems are deployed in different regions, requiring cross-regional data transmission.
Deployment Architecture Diagram
Problem Resolution
We will use TDMQ for Pulsar's solution based on the issues mentioned above.
1. decoupling of systems issue
Intuitively, to achieve reconciliation between various systems, we can directly report messages to the reconciliation system, which is responsible for receiving messages and conducting reconciliation processing. However, there are some challenges here. There are many systems to be integrated, and new systems will continue to be added. This would require a considerable amount of time spent on system integration, and it would greatly intrude upon the existing system workflows. This design is highly unreasonable. By introducing TDMQ for Pulsar, we enable systems to only need to integrate with the MQ, so issues within individual systems will not affect other services.
2. data arrival time difference issue
Reconciliation requires data aggregation between various systems. To achieve real-time data aggregation, ideally, the arrival times of data should not differ significantly. However, due to the sequential nature of system processes, if data from one system is delayed, we need to control the data reading speed to ensure that a large amount of data does not accumulate in the reconciliation system while waiting for processing. By using TDMQ for Pulsar's temporary message storage, data arriving at approximately the same time allows for smoother data aggregation.
3. data consistency issue
TDMQ for Pulsar provides highly consistent and reliable data storage to ensure that data will not be lost. It also provides highly available services and can quickly and automatically repair exceptions.
4. cross-region data transmission issue
TDMQ for Pulsar offers two solutions for achieving data replication between multiple cities, providing real-time data replication channels for the business layer.
For data of utmost importance requiring cross-city disaster recovery, TDMQ supports strong consistency data synchronization between multiple regions, enabling message storage to be distributed across different areas.
For scenarios where strong consistency is not necessary, TDMQ for Pulsar offers an asynchronous replication solution between multiple cities to achieve eventual consistency of data across different locations.
The comparison between the two cross-city synchronization solutions is as follows:
Cross-City Solution
Production Time
Disaster Recovery Capability
Storage Cost
Multi-city Strong Consistency
High
High
Less
Multi-city Eventual Consistency
Low
Low
More
By introducing TDMQ for Pulsar and real-time computing capabilities, we transitioned transaction reconciliation from a daily mode to a real-time reconciliation mode, speeding up the accuracy detection of transactions.
