Customer Master Data Sync from Kingdee Cloud to Qeasy: Single-Strategy Incremental Integration

What This Strategy Solves

Customer master data is the "root" of every subsequent document in the supply chain. In one retail engagement, we kept hitting this scenario: thousands of distributor stores were being maintained in Kingdee Cloud, and another set was maintained in the marketing cloud. Three months later, the reconciliation showed over 200 stores with mismatched codes — one side had renamed without syncing, the other had created new customers without notification.

This strategy uses the Qeasy data integration platform for a single-responsibility sync: incrementally pulling customer records from Kingdee Cloud into the Qeasy middleware layer based on modification time, making it the single authoritative source for downstream distribution and reconciliation. It looks simple, but if the code mapping, time window, and incremental starting point aren't well designed, troubleshooting later becomes extremely painful.

Data Flow and Field Mapping

The data flow is unidirectional: Kingdee Cloud (source) → Qeasy datahub middleware → downstream targets.

The source calls the /jdy/v2/bd/customer customer archive query API, where the page and page_size parameters control the pull cadence. The two core fields for incremental identification are modify_start_time and modify_end_time, both millisecond-level timestamps.

Key field mapping (middleware layer is Qeasy datahub):

The datahub target node is configured as "write empty operation" — this step does not actually persist data, but rather deposits customer data into the Qeasy middleware layer first, to be consumed on demand by other distribution strategies. This is the typical "header-body phased" approach used by Qeasy customers.

How to Configure in Qeasy

For the source platform, select Kingdee.YXC, choose the API /jdy/v2/bd/customer, set effect to QUERY, and method to GET. Enable idCheck, disable buildModel and autoFillResponse, so the response structure is dynamically constructed in the subsequent processing stage.

For the middleware platform, select datahub, set effect to EXECUTE, method to POST, and perform a "write empty operation" first — its role is to trigger the receive event in the Qeasy middleware layer, allowing customer records to enter the unified data pool for subscription by downstream distribution strategies. This is the "centralized code mapping management" pattern most commonly used by Qeasy customers: all customer records are first aggregated in datahub, the code mapping is maintained in only one place here, and each downstream business line (orders, inventory, reconciliation) fetches data using the unified code.

The four request parameters are fixed: modify_end_time takes {{CURRENT_TIME}}*1000, modify_start_time takes {{LAST_SYNC_TIME}}*1000, page defaults to 1, and page_size is set to 50. This combination determines the incremental sync window size and the volume pulled per request.

Implementation Steps

Step 1: Determine the incremental starting point. On the first run, LAST_SYNC_TIME is usually empty or 0. At this point, a full pull must be triggered to fetch all customers and establish the initial baseline timestamp. A reliable approach is to manually specify a historical time point, such as the millisecond value corresponding to "2024-01-01 00:00:00", as the start of the first run window.

Step 2: Configure source-side scheduling. Set the source-side crontab to */16 8-22 * * *, which means execution every 16 minutes between 8 AM and 10 PM daily. The design rationale for this frequency is: Kingdee Cloud customer record modifications are concentrated during daytime business hours, so high-frequency polling ensures data latency stays within 16 minutes; during nighttime low-frequency periods, ineffective polling is avoided to prevent rate-limiting triggers.

Step 3: Configure middleware scheduling. Set the middleware crontab to 23 2 * * *, meaning execution at 2:23 AM daily. The source-side high-frequency polling writes data into the datahub staging area, while the middleware layer performs low-frequency batch consumption, preventing the downstream from being overwhelmed by high-frequency events.

Step 4: Build the code mapping table. Maintain a customer_mapping table in the Qeasy middleware layer, with the key being Kingdee's number and the value being the downstream unified code. New customers follow the "Kingdee → mapping table → downstream" chain; code changes follow "Kingdee modification → mapping table version update → downstream sync".

Step 5: Trial run and reconciliation. Run for 3 days first, observing the number of customers pulled each day, the distribution of modification times, and whether there are unfinished pagination requests. A typical experience is that the first two days will reveal some historical dirty data with empty modify_time, which needs to be filtered out during the cleaning stage.

Lessons Learned

1. The millisecond conversion at the incremental starting point is the most common pitfall. Kingdee returns modify_time as a string timestamp, but modify_start_time expects the "millisecond" level. The {{LAST_SYNC_TIME}}*1000 step must not be missed, otherwise what is pulled will always be an empty set.

2. Pagination termination condition is not checked. The script only checks whether page is empty, but not whether page_size is less than the requested value. As a result, when the last page has fewer than 50 records, it falls into an infinite loop. A reliable approach is: when the number of returned records < page_size, break out of the loop and refresh LAST_SYNC_TIME.

3. LAST_SYNC_TIME update timing is wrong. A typical mistake is updating while pulling — after a mid-way failure, the timestamp has already advanced, and the next restart will lose data. The correct approach is to update LAST_SYNC_TIME only after all pagination is complete and all data is written to datahub.

4. Dirty data with missing modification time fields. Some early Kingdee customer records have empty modify_time. This kind of data will neither appear in the incremental window nor in the full pull. A reliable approach is to force-assign a fallback value (such as creation time) to empty modify_time during the cleaning stage, otherwise these customers will never be synced to downstream.

5. High-frequency polling triggers Kingdee rate-limiting. The */16 frequency is fine when the number of customer records is small, but when the records exceed 10,000, combined with downstream distribution events, it may trigger QPS rate-limiting on the Kingdee side. It is recommended to monitor the response codes of source-side calls and automatically reduce the frequency when encountering 429.

Applicable and Non-Applicable Scenarios

Applicable: scenarios where customer record change frequency is moderate (dozens to hundreds of modifications per day), the coding system is relatively stable, and unified customer master data needs to be distributed to multiple downstream systems. Not applicable: one-time full-import customer record scenarios (using ETL tools directly is more efficient), and scenarios requiring bidirectional sync with complex conflict merging rules (it is recommended to introduce an MDM master data management platform).