Build an application that synchronizes external data
Build a Jotsu application that synchronizes paginated external API data safely with schedules, idempotent updates, and useful logs.
What you will build#
An application that imports records from an external API and reports every synchronization run.
Part 1: Plan the application#
Define an external stable ID, mapped fields, update rules, deletion policy, and a SyncRun record with timestamps, counts, cursor, and outcome.
Part 2: Connect the API#
Store the credential as a secret. Fetch one page on the server, validate it, and display a read-only preview.
At this point, one validated API page appears without storing records.
Part 3: Store records idempotently#
Upsert by external ID and record created, updated, skipped, and failed counts. Repeating the same page must not create duplicates.
Part 4: Add pagination and scheduling#
Handle the API’s cursor, page, or offset scheme. Add a scheduled job in code with bounded retries and rate-limit backoff. Include a development-only run-now path.
At this point, two identical runs produce the same records and separate successful run history.
Part 5: Test failures#
Test an expired credential, rate limit, malformed record, and interrupted page. Confirm each failure is logged and can resume safely.
Part 6: Deploy#
Deploy with production credentials, invoke one controlled run, and confirm its SyncRun result.
Extend the application#
Add webhook updates, drift reports, or a manual reconciliation queue.