Field Formatting Processing

Feature Overview

The motivation behind this feature is straightforward: in a data integration pipeline, the field formats returned by source systems rarely match what downstream applications, reports, or APIs expect. A common case is when a source system returns a shipping date as a long date-time string such as YYYYMMDD HH:MM:SS, while the downstream only requires a short date like YYYYMMDD. Handling every such discrepancy with custom scripts or manual cleanup quickly becomes expensive.

To address this, the Qeasy DataHub provides a built-in formatResponse function at the field mapping layer. It allows users to declare a set of "original field → new field → target format" mappings directly inside an integration strategy, and the platform automatically performs the transformation during data flow, injecting the result as a new field into subsequent processing steps. No additional scripting or external transformation service is required.

Use Cases

Typical scenarios include, but are not limited to:

1. Date and time formatting: A source system returns 20240115 14:30:00, and the downstream needs it split into a date 20240115 and a time 14:30:00, or expressed as a relative description such as "three days ago" or "one week ago".
2. Amount with thousands separators: Raw numeric strings from the source need to be displayed as currency-formatted values like ￥1,000.00 in financial reports for easier reconciliation.
3. Long text truncation: Detail fields that are too long can be shortened to a ...-style preview, avoiding data bloat in list views or logs.
4. Numeric normalization: Source values are string-typed numbers, and the downstream needs strict integers or fixed-precision floats, achievable through intval or round(2).
5. Array-to-text conversion: Multi-value fields such as tags or categories need to be joined into comma-separated strings for export or API delivery.

Configuration

formatResponse is an array structure; each element describes a single formatting rule. Example:

"formatResponse": [
  {
    "old": "delivery_statusInfo.delivery_date",
    "new": "modify_date_new",
    "format": "date"
  }
]

Field descriptions:

• old: The original field path returned by the source system. Nested paths are supported using dot notation, e.g., delivery_statusInfo.delivery_date.
• new: The name of the new field generated after formatting. Downstream nodes will read from new directly.
• format: The target format. Supported values are listed below.

Currently supported format types:

Formats with parentheses (such as longText(15), round(2), implode(',')) accept a custom parameter that overrides the default.

Notes

1. The new field does not overwrite the original: new is an additional field generated by the platform. The original old field is preserved for traceability and reconciliation.
2. Field paths must match the source structure: old must follow the exact JSON structure returned by the source system, including hierarchy and field names. Otherwise, the value cannot be retrieved.
3. Use format parameters carefully: Formats such as longText, round, and implode accept parameters whose values directly affect the output. Validate them with sample data before going live.
4. Multiple formattings can be chained: If the same field needs to undergo several transformations (for example, converting to a short date first and then to a relative description), you can configure multiple rules in the array, each operating on the new field produced by the previous step.
5. Relative time depends on the reference point: When using dateDescription, the result depends on the gap between the data ingestion time and the original timestamp. Ensure that the source time field carries correct time zone information.
6. Currency symbol and locale awareness: amount outputs a ￥ symbol by default. If the downstream uses a different currency or format, consider whether an additional transformation is needed in a later node.

With formatResponse, we aim to encapsulate this frequent yet tedious field-formatting work into platform capabilities, making integration strategies more readable and easier to maintain.