Skip to end of metadata
Go to start of metadata

Most applications typically require access to a remote data source, either on their own application data service or public data services. The access to these data services is highly dependent on data connectivity. When the data connectivity is not stable or hard to come by, the application will be not functional.

To overcome this issue, most applications now support an offline data storage that keep all or portion of data in the device's storage. When the remote server is unreachable due to connection issues, users can still work with the application by reading the local data. This scenario introduces the needs of two-way data synchronization to sync the modified local data to the server and vice versa.

Crosslight provides a set of components that you can use to enable data synchronization in your application. 

  • Crosslight SQLite to store data locally in your application.
  • Entity services to support data synchronization request.
  • RestClient to access the Entity Service.

On this page:

Overview

Synchronizing data can be done by doing two distinctive process sequentially.

  • Saving all pending changes from offline storage to online storage through an entity service.
  • Retrieving data based on last synchronized data.

These processes expect the entity to have a date time column that represents the last modified date of the entity. You can then retrieve the entity where the modified date is larger than last synchronized date.

You can create the entity services that handle data synchronization using Crosslight Entity Designer Extensions. The entity services will apply the data synchronization metadata that you can set in the entity designer. To learn more how to set the synchronization metadata, see Data Synchronization Metadata.

Crosslight provides components that allow you enable data synchronization in your application.

  • At server side
    • Entity Services
      Performs data synchronization to online storage based on accepted request data. 
  • At client side
    • SQLite Service
      Store synchronized data as well as new pending changes when the application is offline.
    • Rest Client
      Perform data transaction between offline and online service. 
    • Synchronization Service
      Complies all the data synchronization process from retrieving data using RestClient, storing data and pending changes using SQLiteService, performing queue management for multiple sync channel scenario and also resolving sync stall. 

Synchronizing Data Using Entity Service

To synchronize data in the server-side, you will need to call SynchronizeChanges method in the entity service.

This service accepts a SynchronizationRequest object that contains the modified entities and SynchronizationInfo.

The following code shows you how to create the  SynchronizationRequest that contains SynchronizationInfo and pending changes and send it to entity service using RestClient.

The above code shows a simple example to perform a synchronization request to the WebAPI. However, there are many aspects to be considered in the synchronization process such as the queue management, error handling, conflict resolution and much more. Fortunately, Crosslight Enterprise App Framework already provides a comprehensive synchronization implementation that takes account every little detail of the sync process. It's highly recommended that you use Enterprise App Framework to quickly get started with data synchronization in your apps. For more information, see Enabling Data Synchronization.

Adding Custom Filter To Synchronization Query

You can filter the query used for data synchronization by intercepting the entity context provider query process. Since your entity controller is generated by Entity Designer Extensions, you will need to create partial class to implement the query interception.

The following code shows you how to create query interceptor using a partial class.

Storing and Retrieving Pending Changes from SQLite

Another aspect of data synchronization is storing pending changes when there are no data connectivity, then retrieve it again later for syncing to server. Crosslight implements this functionality out-of-the-box in the SQLite component. For more information, see Data Synchronization Support.

Multiple Synchronization Channel Support

When building an enterprise application with several modules, synchronizing data for all modules in a bulk could hinder your application performance significantly. When facing this requirement, it is best to separate the data synchronization into several synchronization channels. The synchronization channel can then focus on specific entities only, for example you might want to gather all the master data in one synchronization channel, and transactional data per module in other synchronization channel.

At the heart of the synchronization channel are two key components, SynchronizationChannel and SynchronizationService, see the following figures.

Synchronization channel defines the entity types that will be processed by that channel. In general, you might want to separate your complex entities into several synchronization channels. One of the approaches that you can use is by separating it per module based. The following code shows you how to define synchronization channels.

The above example creates two synchronization channels, one for MasterData and another for Data. You can notice that the synchronization channel has various settings that you can customize which controls the synchronization behaviors, such as whether paging is enabled, auto retry, and more.

Synchronization Queue Management

Each channel can start data synchronization process by calling Start or StartImmediately method as follows.

The Start method will send the synchronization channel into a synchronization queue. This means that the channel will be handled subsequently based on the queue priority order. Note that each synchronization channel has priority level (lower number means higher priority). On the other hand, the StartImmediately method will send the synchronization channel in immediate channel which will be executed immediately.

For the queue management, the SynchronizationService uses four different containers to indicate the status of the synchronization channels.

  • Running Channels
    Indicates which synchronization channel is currently running (being synchronized) using Start method.
  • Immediate Channels
    Indicates which synchronization channel is currently running (being synchronized) using StartImmediately method. 
  • Synchronization Queue
    Indicates which synchronization channel is currently being queued and waiting to be passed to running channels to be synchronized.
  • Pending Channels
    Indicates which synchronization channel that currently pending. This might happens if user synchronizing the same channel subsequently and each subsequent request has newer pending changes. The SynchronizationService will need to process the first request first, which mean the second request will be put in pending channels and will be executed directly after the first request is finished. Any subsequent call after the synchronization channel put in pending channels will be ignore because its already represented.

The following table shows how queue management works when the Start method is used.

Stage 
  1. Determine whether the synchronization channel should be added to synchronization queue
  • If the running channel and synchronization queue does not contains the synchronization channel, add the synchronization channel to synchronization queue.

 

  • If the running channel does not contains the synchronization channel and synchronization queue has the synchronization channel, do nothing.

  • If the running channels contains the synchronization channel and the synchronization channel has newer pending changes while its not exist in synchronization queue, add to synchronization queue.

  • If the running channels contains the synchronization channel and the synchronization channel does not have newer pending changes, do nothing.

  • If the immediate channels contains the synchronization channel and the synchronization channel has newer pending changes while its not exist in synchronization queue, add to synchronization queue.

  • If the immediate channels contain synchronization channel and the synchronization channel does not have newer pending changes, do nothing.

 

2. If passed the validation, add the synchronization queue based on priority
3. Run the synchronization queue
  • Pop the first synchronization channel in synchronization queue
  • Add the synchronization channel to running channels, indicating the the channel is currently in progress
  • Execute the synchronization channel
  • After the process is finished remove the synchronization channel from running channels
  • If the pending channels contains the synchronization channel, execute the synchronization channel immediately using StartImmediately
  • If the queue has items, re-run this process

The following table shows how queue management when StartImmediately method is used.

Stage 
  1. Determine whether the synchronization channel should be added to immediate channels
  • If the immediate channels and running channels does not contains the synchronization channel, add to immediate channels and execute the synchronization channel.
    If the synchronization queue contains the synchronization channel, remove it from synchronization queue because it will be executed immediately.

  • If the immediate channels contains the synchronization channel, and the synchronization has newer pending changes, add the synchronization channel to pending channels.

  • If the immediate channels contains the synchronization channel, and the synchronization does not have newer pending changes, do nothing

  • If the running channels contains the synchronization channel, and the synchronization has newer pending changes, add the synchronization channel to pending channels.

  • If the running channels contains the synchronization channel, and the synchronization does not have newer pending changes, do nothing

2. If passed the validation, add the synchronization channel into immediate channels and execute the synchronization channel. Otherwise, check whether the synchronization channel has newer pending changes. If so, insert the synchronization channel to pending channels to be executed directly after the current process is done.

3. After the process is finished remove the synchronization channel from immediate channels.

4. If the pending channels contains the synchronization channel, execute the synchronization channel immediately using StartImmediately

Resolving Stall Synchronization

When synchronizing, you might encounter connection shortage which interrupt the synchronization process. When this happened, most chance you will lost critical information regarding the state of synchronization.

  • Is local pending changes already saved at online storage?
  • Which page of data that already transferred?

To resolve this situation, Crosslight provides a mechanism that able to identify whether the synchronization process has been stalled. When stalled, it will re-fetch the data first before starting the synchronization process again. This re-fetch process will fix the synchronization state such as ensuring the local pending changes states and also finalizing data transfer.

Performing in-depth re-fetch process can fix the following scenarios.

ScenarioDescription
Submitting new records with auto increment key (identity)

When you create new records with auto increment key (identity) at client side, the records are stored using temporary auto increment key that will be finalized after synchronized to server. In successful scenario, the synchronization process will return the finalized key from server side that can be used to finalize the temporary key at client side, thus finalizing the synchronization state at client side.

When this process is interrupted due to connection shortage, the client does not know whether the new records are successfully saved at server side. To overcome this issue, the key mapping between temporary key and finalized key is stored in synchronization state column of the entity. You can use Entity Designer Extensions to indicate which column in your table that will serve as synchronization state. To learn more how to use Entity Designer Extensions see, Creating Entity Model and Services with Crosslight Entity Designer Extensions.

Afterward, the data is re-fetched and applied the key mapping in the synchronization state to finalize the new entity state at client side.

Submitting new records followed by deleting the newly added records

When you're offline, creating new records then followed by deleting the newly added records basically leaving the client with no pending changes. However when you're online and the process is interrupted due to connection shortage, the client does not know whether the new records are successfully saved at server side. This creates confusion what should be done when deleting the newly added records.

To overcome this, re-fetch data will be performed. If the new records are successfully saved, the data will be retrieved during the re-fetch process, which will then properly marked as deleted. The deleted records will then be synchronized to the server subsequently, making server state equally identical with the client state.

Updating records

When you update existing records at client side, the particular records will be marked as modified. When the process is interrupted due to connection shortage, the client does not know whether the records are successfully updated at server side.

To overcome this, re-fetch data will be performed which allows the pending records to be finalized with the following conditions:

  • If the pending records have newer pending changes date than modified date from re-fetch data, merge the values and maintain the modified state.
  • If the pending records do not have newer pending changes date than modified date from re-fetch data, finalize the entity state to unchanged.

 

Note that the re-fetch process is implemented in the Enterprise App Framework which is automatically enabled when you migrated to Crosslight 4. There are no additional code necessary to enjoy these great reliability features. To ensure your project is successfully migrated to version 4, please refer to Crosslight 4 Upgrade Guide.

Data Synchronization Using Intersoft Application Framework

Crosslight also provides complete implementation of data synchronization process that you can utilize by implementing Crosslight App Framework. This framework already implements the common data synchronization processes such as:

  • Creating SQLite table for entity that contributed in data synchronization
  • Perform data synchronization and storing synchronization info after the process is completed
  • Support multi-user data synchronization
  • Support multiple sync channel
  • Queue management
  • Resolve stall mechanism
  • Provides API to store pending changes

To learn more how to enable data synchronization using the App Framework, see Enabling Data Synchronization.