[Core] Fix JQ Transform Failures Caused By Datetime And Date Values by emekanwaoma · Pull Request #3046 · port-labs/ocean

emekanwaoma · 2026-04-03T13:38:44Z

Description

What -

Add a shared Ocean core utility to normalize raw data into JSON-compatible types (convert date/datetime to ISO strings).
Use it in both async and subprocess JQ processors before calling jq.input_value(...).
Add unit tests for the normalizer.

Why -

Prevent mapping/transformation failures like Object of type datetime is not JSON serializable when integrations (e.g. YAML parsing) introduce Python-native date/datetime values into raw payloads.
Improve reliability across all integrations without requiring per-integration workarounds.

How -

Introduced port_ocean/core/utils/json_compat.py with make_json_compatible(...).
Added tests in port_ocean/tests/core/utils/test_json_compat.py.
Efficiency note: This is the most efficient long-term approach because it centralizes the fix in Ocean core at the single JQ boundary (covering all integrations/mappings) and avoids scattered per-integration parsing changes or expensive serialize→parse workarounds.

Type of change

Please leave one option from the following and delete the rest:

Bug fix (non-breaking change which fixes an issue)

All tests should be run against the port production environment(using a testing org).

Core testing checklist

Integration able to create all default resources from scratch
Resync finishes successfully
Resync able to create entities
Resync able to update entities
Resync able to detect and delete entities
Scheduled resync able to abort existing resync and start a new one
Tested with at least 2 integrations from scratch
Tested with Kafka and Polling event listeners
Tested deletion of entities that don't pass the selector

Integration testing checklist

Preflight checklist

Handled rate limiting
Handled pagination
Implemented the code in async
Support Multi account

Screenshots

Include screenshots from your environment showing how the resources of the integration will look.

API Documentation

Provide links to the API documentation used for this integration.

qodo-code-review · 2026-04-03T13:38:59Z

Review Summary by Qodo

Fix JQ transform failures from datetime and date values

🐞 Bug fix

Walkthroughs

Description

• Add make_json_compatible() utility to normalize date/datetime to ISO strings
• Apply normalization in JQ processors before passing data to jq
• Prevents JSON serialization failures from Python-native date/datetime values
• Add comprehensive unit tests for the normalizer function

Diagram

flowchart LR
  A["Raw Data with<br/>date/datetime"] -->|make_json_compatible| B["JSON-compatible<br/>ISO strings"]
  B -->|input_value| C["JQ Processor"]
  C -->|Success| D["Mapped Entity"]

File Changes

1. port_ocean/core/utils/json_compat.py ✨ Enhancement +18/-0

Add JSON compatibility normalizer utility

• New utility module with make_json_compatible() function
• Recursively converts datetime and date objects to ISO format strings
• Handles nested dictionaries and lists for deep normalization
• Preserves all other JSON-compatible types unchanged

port_ocean/core/utils/json_compat.py

2. port_ocean/core/handlers/entity_processor/jq_entity_processor.py 🐞 Bug fix +3/-2

Apply JSON normalization in async JQ processor

• Import make_json_compatible from new utils module
• Wrap data with make_json_compatible() in _search() method before JQ processing
• Wrap data with make_json_compatible() in _search_as_bool() method before JQ processing
• Ensures async JQ processor handles date/datetime values correctly

port_ocean/core/handlers/entity_processor/jq_entity_processor.py

3. port_ocean/core/handlers/entity_processor/jq_entity_processor_sync.py 🐞 Bug fix +3/-2

Apply JSON normalization in sync JQ processor

• Import make_json_compatible from new utils module
• Wrap data with make_json_compatible() in _search() method before JQ processing
• Wrap data with make_json_compatible() in _search_as_bool() method before JQ processing
• Ensures sync JQ processor handles date/datetime values correctly

port_ocean/core/handlers/entity_processor/jq_entity_processor_sync.py

View more (2)

4. port_ocean/tests/core/utils/test_json_compat.py 🧪 Tests +16/-0

Add unit tests for JSON compatibility normalizer

• Add test for datetime and date conversion to ISO format strings
• Add test for recursive normalization of nested dictionaries and lists
• Validates correct handling of timezone-aware datetime objects
• Ensures all nested structures are properly normalized

port_ocean/tests/core/utils/test_json_compat.py

5. CHANGELOG.md 📝 Documentation +7/-0

Update changelog with version 0.38.28 release

• Add version 0.38.28 release notes
• Document improvement for normalizing JQ input data with date/datetime types
• Highlight prevention of transformation failures from JSON serialization issues

CHANGELOG.md

qodo-code-review · 2026-04-03T13:39:01Z

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX Issues (0)

1. ~~Repeated payload normalization~~ ☑ 🐞 Bug ➹ Performance

Description

make_json_compatible(data) is executed inside every _search/_search_as_bool call, while
_search_as_object triggers _search once per mapped leaf field, causing the entire raw payload to
be recursively rebuilt many times per entity. This can significantly increase CPU/memory usage and
(in the async path) block the event loop during mapping for large payloads or large mappings.

Code

port_ocean/core/handlers/entity_processor/jq_entity_processor.py[R103-106]

        try:
            compiled_pattern = self._compile(pattern)
-            func = compiled_pattern.input_value(data)
+            func = compiled_pattern.input_value(make_json_compatible(data))
            return func.first()

Evidence
In the async processor, every _search and _search_as_bool wraps jq.input_value(...) with
make_json_compatible(data), and _search_as_object schedules one _search(data, value, ...) per
mapping leaf, reprocessing the full payload each time. The sync processor has the same structure:
_search_as_object calls _search per leaf, and _search/_search_as_bool normalize the full
payload on each call. make_json_compatible recursively allocates new dicts/lists (not in-place),
so each invocation walks and rebuilds the entire payload subtree.
port_ocean/core/handlers/entity_processor/jq_entity_processor.py[99-178]
port_ocean/core/handlers/entity_processor/jq_entity_processor.py[200-213]
port_ocean/core/handlers/entity_processor/jq_entity_processor_sync.py[75-151]
port_ocean/core/handlers/entity_processor/jq_entity_processor_sync.py[154-168]
port_ocean/core/utils/json_compat.py[5-18]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`make_json_compatible(...)` currently runs inside every JQ query execution. Because mapping runs many JQ queries per entity (selector + each mapped field), the raw payload gets recursively traversed and rebuilt repeatedly, increasing CPU/memory and potentially blocking the async event loop.

## Issue Context
The goal (prevent pyjq failures on `date`/`datetime`) is correct, but the normalization should happen once at the JQ boundary per entity, then be reused for all subsequent JQ evaluations for that entity.

## Fix Focus Areas
- Compute `normalized_data = make_json_compatible(data)` once per entity (e.g., at the start of `_get_mapped_entity` / `_calculate_entity`), and pass `normalized_data` to `_search_as_bool` and `_search_as_object`.
- Update `_search`/`_search_as_bool` to assume the input is already normalized (or add an internal optional parameter / small wrapper) so normalization is not repeated.
- Apply the same approach in the sync processor so the subprocess path doesn’t normalize per field.

### Suggested focus locations
- port_ocean/core/handlers/entity_processor/jq_entity_processor.py[99-219]
- port_ocean/core/handlers/entity_processor/jq_entity_processor_sync.py[75-174]
- port_ocean/core/utils/json_compat.py[5-18]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

ⓘ The new review experience is currently in Beta. Learn more

github-actions · 2026-04-03T14:06:11Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/23948158664/artifacts/6260379203

Code Coverage Total Percentage: `90.59%`

…7570

github-actions · 2026-04-05T08:10:00Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/23997195562/artifacts/6276111613

Code Coverage Total Percentage: `90.59%`

github-actions · 2026-04-05T08:15:26Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/23997276483/artifacts/6276136915

Code Coverage Total Percentage: `90.58%`

github-actions · 2026-04-05T08:18:17Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/23997337011/artifacts/6276151962

Code Coverage Total Percentage: `90.58%`

matan84 · 2026-04-05T08:24:13Z

port_ocean/core/utils/json_compat.py

+    return value
+
+
+def jq_input_value(compiled_pattern: Any, data: Any) -> Any:


Suggested change

def jq_input_value(compiled_pattern: Any, data: Any) -> Any:

def compile_jq(compiled_pattern: Any, data: Any) -> Any:

matan84 · 2026-04-05T08:24:38Z

pyproject.toml

 [tool.poetry]
 name = "port-ocean"
-version = "0.38.27"
+version = "0.38.28"


Suggested change

version = "0.38.28"

version = "0.39.0"

github-actions · 2026-04-05T08:50:19Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/23997829024/artifacts/6276296135

Code Coverage Total Percentage: `90.58%`

fix: handle json incompatible types

3c3ee49

emekanwaoma requested a review from a team as a code owner April 3, 2026 13:38

github-actions bot added the size/M label Apr 3, 2026

emekanwaoma added 4 commits April 5, 2026 08:51

Merge branch 'main' of https://github.com/port-labs/ocean into PORT-1…

fced692

…7570

fix: retry type error

8f030c8

fix: update changelog

2fe8b16

fix: bump integration

9dea7dd

matan84 approved these changes Apr 5, 2026

View reviewed changes

fix: attend to comments

65fe9eb

emekanwaoma merged commit cf329e3 into main Apr 5, 2026
18 checks passed

emekanwaoma deleted the PORT-17570 branch April 5, 2026 09:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] Fix JQ Transform Failures Caused By Datetime And Date Values#3046

[Core] Fix JQ Transform Failures Caused By Datetime And Date Values#3046
emekanwaoma merged 6 commits intomainfrom
PORT-17570

emekanwaoma commented Apr 3, 2026

Uh oh!

qodo-code-review bot commented Apr 3, 2026

Uh oh!

qodo-code-review bot commented Apr 3, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 3, 2026

Uh oh!

github-actions bot commented Apr 5, 2026

Uh oh!

github-actions bot commented Apr 5, 2026

Uh oh!

github-actions bot commented Apr 5, 2026

Uh oh!

matan84 Apr 5, 2026

Uh oh!

matan84 Apr 5, 2026

Uh oh!

github-actions bot commented Apr 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		return value


		def jq_input_value(compiled_pattern: Any, data: Any) -> Any:

	def jq_input_value(compiled_pattern: Any, data: Any) -> Any:
	def compile_jq(compiled_pattern: Any, data: Any) -> Any:

Conversation

emekanwaoma commented Apr 3, 2026

Description

Type of change

All tests should be run against the port production environment(using a testing org).

Core testing checklist

Integration testing checklist

Preflight checklist

Screenshots

API Documentation

Uh oh!

qodo-code-review bot commented Apr 3, 2026

Review Summary by Qodo

Walkthroughs

File Changes

Uh oh!

qodo-code-review bot commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

github-actions bot commented Apr 3, 2026

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/23948158664/artifacts/6260379203

Code Coverage Total Percentage: 90.59%

Uh oh!

github-actions bot commented Apr 5, 2026

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/23997195562/artifacts/6276111613

Code Coverage Total Percentage: 90.59%

Uh oh!

github-actions bot commented Apr 5, 2026

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/23997276483/artifacts/6276136915

Code Coverage Total Percentage: 90.58%

Uh oh!

github-actions bot commented Apr 5, 2026

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/23997337011/artifacts/6276151962

Code Coverage Total Percentage: 90.58%

Uh oh!

matan84 Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

matan84 Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 5, 2026

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/23997829024/artifacts/6276296135

Code Coverage Total Percentage: 90.58%

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

qodo-code-review bot commented Apr 3, 2026 •

edited

Loading

Code Coverage Total Percentage: `90.59%`

Code Coverage Total Percentage: `90.59%`

Code Coverage Total Percentage: `90.58%`

Code Coverage Total Percentage: `90.58%`

Code Coverage Total Percentage: `90.58%`