Download all docs

Recipe: Scheduled Data Sync

This recipe keeps a local table in step with an external API on a timer. A schedule fires on a cron cadence, an http element pulls the latest data from the upstream service, and a sql element absorbs it — inserting new rows and updating changed ones. Your copy of the data stays fresh without anyone refreshing it.

The problem it solves

You depend on data that lives somewhere else — a CRM, a pricing feed, an inventory API — and you need a current local copy to query, join, and build on. Calling the API live on every request is slow and rate-limited; copying it once goes stale immediately. This recipe syncs on a schedule: the data is local and fast to read, and a timer keeps it from drifting out of date.

Elements

ElementRole
scheduleFires the sync on a cron cadence.
httpPulls the latest data from the upstream API.
sqlThe local copy — insert new rows, update changed ones.
automationSequences fetch → upsert as one flow.

Flow

  1. Create a schedule. Set spec.cron to your sync cadence; it emits a schedule event downstream each time it fires. Use trigger_now to force an immediate sync while testing and next_runs to confirm the upcoming fire times.
  2. Fetch the upstream data with an http element’s request operation (canonical alias call) — an outbound GET to the API, returning the current payload.
  3. Land it in a sql element. insert rows that are new; update rows that changed (match on the upstream’s stable id). Shape the table once with migrate, and read the synced data back with query.
  4. Sequence the two steps with an automation, which references schedule, http, and sql as steps — the schedule event triggers the run, the http step fetches, and the sql step writes. A condition child can skip the write when the upstream returned nothing new.
  5. Confirm it is running from the schedule’s history — a record of every sync, so a stalled feed is visible rather than silent.

What this shows

Each concern is a separate element doing its one job: the schedule owns the cadence, the http element owns the outbound fetch, and the sql element owns the durable local copy with explicit insert/update semantics. Because the trigger is declarative, changing “hourly” to “every five minutes” is one cron edit, and the sync’s history makes reliability observable. The pattern generalizes: point the http step at a different API or the sql step at a different schema and the same skeleton syncs anything on a timer.

Next pages