As businesses adopt more tools, the same data ends up in several places: customers in the CRM, the billing platform and the application database; inventory in the warehouse system, the store and the marketplace; orders in the commerce platform, accounting and analytics. When those copies disagree, people stop trusting the data, make wrong decisions and waste time investigating. Keeping systems in sync in near real time is a classic engineering challenge with well-understood solutions. This guide describes the options and the design principles that make synchronisation dependable.
Decide how fresh is fresh enough
"Real time" is expensive; "right time" is what you need. Ask what delay each use case can tolerate. A dashboard that updates every fifteen minutes may be perfectly fine. Stock levels on a high-demand product may need seconds. A financial ledger may need strict consistency but not instant propagation. Classify data flows by freshness requirement and criticality, then choose the simplest technique that meets each one. Over-engineering everything for sub-second latency multiplies cost and failure modes.
The main synchronisation techniques
Scheduled batch jobs. Export changes periodically and load them elsewhere. Simple and robust for reporting and slow-moving data, but latency is measured in minutes or hours, and large loads can strain systems.
Polling. Repeatedly ask an API for records changed since a timestamp or cursor. Suitable when a system has no push mechanism. Poll sensibly to respect rate limits, use incremental cursors, and beware of records that change without updating their timestamp.
Webhooks. The source pushes notifications when things change. Low latency and efficient, but delivery is at-least-once and unordered, so consumers must be idempotent and sometimes re-fetch current state. See our integration guide and our webhook design guide.
Change data capture (CDC). Tools read a database's transaction log and emit every insert, update and delete as an event, without changing the application. CDC provides accurate, ordered change streams with low overhead, ideal for replicating data to search indexes, caches, analytics stores and other services.
Event streaming platforms. Durable, partitioned logs let many consumers read the same stream of events independently and replay history. They suit high-volume, multi-consumer architectures, at the price of operational complexity.
Make events and state changes atomic
A frequent bug is updating the database and then failing to publish the event, or publishing an event for a change that was rolled back. The transactional outbox pattern solves this: within the same database transaction as the change, insert an event record into an outbox table. A separate relay process reads unpublished outbox rows and publishes them to a queue or stream, marking them as sent. Because the state change and the event commit together, you never lose or invent events. CDC on the outbox table is a robust way to implement the relay.
Design consumers to be idempotent
In distributed systems you get at-least-once delivery; exactly-once is mostly an illusion. Consumers must therefore tolerate duplicates and reordering. Techniques include storing processed event IDs and skipping repeats, using upserts keyed by a stable identifier so replaying an event has no additional effect, comparing version numbers or timestamps to ignore stale updates, and designing operations as "set to this state" rather than "increment by one." Handle deletes explicitly, with soft-delete markers or tombstone events, because a missing record is ambiguous.
Handle ordering carefully
Some updates must be applied in sequence, such as create before update, or debit before credit. Partitioning a stream by entity ID ensures events for the same customer or order are processed in order, while allowing parallelism across entities. When ordering cannot be guaranteed, make consumers order-tolerant by fetching the latest state on receipt or by holding events until dependencies arrive.
Resolve conflicts
If two systems can update the same data, conflicts will happen. Avoid multi-master editing where possible by giving each field a single owner, as described in our CRM integration article. When unavoidable, define a rule: last write wins based on synchronised timestamps, source priority, field-level merge, or manual review for sensitive data. Record conflicts so you can see how often they occur and adjust the design. For collaborative editing, more sophisticated techniques exist, which we touch on in our real-time collaboration article.
Sync across databases and clouds
Replicating between databases in different regions or clouds adds network latency, partial failure and consistency questions. Managed replication features handle many cases, such as read replicas and multi-region databases; for heterogeneous systems, CDC and streams bridge the gap. Decide the consistency model you can accept: strong consistency for money and stock, eventual consistency for search indexes and analytics. Document which data is eventually consistent and how long it may lag, so support staff and developers know what to expect.
Initial load and backfill
Synchronisation starts with getting the existing data across. Do a snapshot load while capturing ongoing changes so nothing is missed between the snapshot and the start of streaming, then verify counts and checksums. Build the ability to re-sync a single record, a time range or an entire dataset on demand, because you will need it after bugs, outages or schema changes.
Reconcile continuously
Even a perfect design meets reality: a bug, a manual edit, a partner outage. A reconciliation job compares systems on a schedule, checking counts, totals and sampled records, and flags or repairs differences. It is your safety net and also an early warning system. Alert when discrepancies exceed a threshold, and track drift over time as a quality metric.
Observe the pipeline
Track end-to-end latency from change to visibility in the destination, backlog size, error rates, retry counts and dead-letter contents. Give each event a correlation ID and log it through every hop so you can trace a single change across systems. Dashboards and alerts, as in our monitoring guide, turn silent drift into visible, fixable problems.
Security and privacy
Synchronisation copies data, and every copy is a liability. Encrypt data in transit and at rest, restrict access to streams and queues, filter sensitive fields, and ensure deletion requests propagate to every copy; see our privacy overview.
Start simple
Begin with one flow that hurts, choose the simplest method that meets its freshness need, make it idempotent and observable, and add reconciliation. Extend the approach as needs grow. If your data is scattered and unreliable, our software team can design a synchronisation layer that gets your systems agreeing, and keep it healthy with monitoring and a maintenance plan.
Put this into practice with CodeLuma
CodeLuma designs synchronisation architectures that keep your systems consistent in near real time, using proven patterns like change data capture, outboxes and reconciliation, without brittle nightly scripts.
- Custom software development - tailored systems, integrations and internal tools.
- Managed hosting - monitored, backed-up, Canadian-friendly hosting.
- Maintenance and support plans - updates, monitoring and ongoing improvement.
Start a conversation. Tell us about your project and we will reply with practical next steps, or browse all CodeLuma services. CodeLuma Development Inc. is based in Nova Scotia and works with teams across Canada and remotely.
