Most applications typically require access to a remote data source, either on their own application data service or public data services. The access to these data services is highly dependent on data connectivity. When the data connectivity is not stable or hard to come by, the application will be not functional.
To overcome this issue, most applications now support an offline data storage that keep all or portion of data in the device's storage. When the remote server is unreachable due to connection issues, users can still work with the application by reading the local data. This scenario introduces the needs of two-way data synchronization to sync the modified local data to the server and vice versa.
Crosslight provides a set of components that you can use to enable data synchronization in your application.
- Crosslight SQLite to store data locally in your application.
- Entity services to support data synchronization request.
- RestClient to access the Entity Service.
Synchronizing data can be done by doing two distinctive process sequentially.
- Saving all pending changes from offline storage to online storage through an entity service.
- Retrieving data based on last synchronized data.
These processes expect the entity to have a date time column that represents the last modified date of the entity. You can then retrieve the entity where the modified date is larger than last synchronized date.
You can create the entity services that handle data synchronization using Crosslight Entity Designer Extensions. The entity services will apply the data synchronization metadata that you can set in the entity designer. To learn more how to set the synchronization metadata, see Data Synchronization Metadata.
Crosslight provides components that allow you enable data synchronization in your application.
- At server side
- Entity Services
Performs data synchronization to online storage based on accepted request data.
- Entity Services
- At client side
- SQLite Service
Store synchronized data as well as new pending changes when the application is offline.
- Rest Client
Perform data transaction between offline and online service.
- Synchronization Service
Complies all the data synchronization process from retrieving data using RestClient, storing data and pending changes using SQLiteService, performing queue management for multiple sync channel scenario and also resolving sync stall.
- SQLite Service
Synchronizing Data Using Entity Service
To synchronize data in the server-side, you will need to call SynchronizeChanges method in the entity service.
Adding Custom Filter To Synchronization Query
You can filter the query used for data synchronization by intercepting the entity context provider query process. Since your entity controller is generated by Entity Designer Extensions, you will need to create partial class to implement the query interception.
The following code shows you how to create query interceptor using a partial class.
Storing and Retrieving Pending Changes from SQLite
Another aspect of data synchronization is storing pending changes when there are no data connectivity, then retrieve it again later for syncing to server. Crosslight implements this functionality out-of-the-box in the SQLite component. For more information, see Data Synchronization Support.
Multiple Synchronization Channel Support
When building an enterprise application with several modules, synchronizing data for all modules in a bulk could hinder your application performance significantly. When facing this requirement, it is best to separate the data synchronization into several synchronization channels. The synchronization channel can then focus on specific entities only, for example you might want to gather all the master data in one synchronization channel, and transactional data per module in other synchronization channel.
Synchronization channel defines the entity types that will be processed by that channel. In general, you might want to separate your complex entities into several synchronization channels. One of the approaches that you can use is by separating it per module based. The following code shows you how to define synchronization channels.
The above example creates two synchronization channels, one for MasterData and another for Data. You can notice that the synchronization channel has various settings that you can customize which controls the synchronization behaviors, such as whether paging is enabled, auto retry, and more.
Synchronization Queue Management
Each channel can start data synchronization process by calling Start or StartImmediately method as follows.
The Start method will send the synchronization channel into a synchronization queue. This means that the channel will be handled subsequently based on the queue priority order. Note that each synchronization channel has priority level (lower number means higher priority). On the other hand, the StartImmediately method will send the synchronization channel in immediate channel which will be executed immediately.
For the queue management, the SynchronizationService uses four different containers to indicate the status of the synchronization channels.
- Running Channels
Indicates which synchronization channel is currently running (being synchronized) using Start method.
- Immediate Channels
Indicates which synchronization channel is currently running (being synchronized) using StartImmediately method.
- Synchronization Queue
Indicates which synchronization channel is currently being queued and waiting to be passed to running channels to be synchronized.
- Pending Channels
Indicates which synchronization channel that currently pending. This might happens if user synchronizing the same channel subsequently and each subsequent request has newer pending changes. The SynchronizationService will need to process the first request first, which mean the second request will be put in pending channels and will be executed directly after the first request is finished. Any subsequent call after the synchronization channel put in pending channels will be ignore because its already represented.
The following table shows how queue management works when the Start method is used.
|2. If passed the validation, add the synchronization queue based on priority|
|3. Run the synchronization queue|
The following table shows how queue management when StartImmediately method is used.
|2. If passed the validation, add the synchronization channel into immediate channels and execute the synchronization channel. Otherwise, check whether the synchronization channel has newer pending changes. If so, insert the synchronization channel to pending channels to be executed directly after the current process is done.|
3. After the process is finished remove the synchronization channel from immediate channels.
|4. If the pending channels contains the synchronization channel, execute the synchronization channel immediately using StartImmediately|
Resolving Stall Synchronization
When synchronizing, you might encounter connection shortage which interrupt the synchronization process. When this happened, most chance you will lost critical information regarding the state of synchronization.
- Is local pending changes already saved at online storage?
- Which page of data that already transferred?
To resolve this situation, Crosslight provides a mechanism that able to identify whether the synchronization process has been stalled. When stalled, it will re-fetch the data first before starting the synchronization process again. This re-fetch process will fix the synchronization state such as ensuring the local pending changes states and also finalizing data transfer.
Performing in-depth re-fetch process can fix the following scenarios.
|Submitting new records with auto increment key (identity)|
When you create new records with auto increment key (identity) at client side, the records are stored using temporary auto increment key that will be finalized after synchronized to server. In successful scenario, the synchronization process will return the finalized key from server side that can be used to finalize the temporary key at client side, thus finalizing the synchronization state at client side.
When this process is interrupted due to connection shortage, the client does not know whether the new records are successfully saved at server side. To overcome this issue, the key mapping between temporary key and finalized key is stored in synchronization state column of the entity. You can use Entity Designer Extensions to indicate which column in your table that will serve as synchronization state. To learn more how to use Entity Designer Extensions see, Creating Entity Model and Services with Crosslight Entity Designer Extensions.
Afterward, the data is re-fetched and applied the key mapping in the synchronization state to finalize the new entity state at client side.
|Submitting new records followed by deleting the newly added records|
When you're offline, creating new records then followed by deleting the newly added records basically leaving the client with no pending changes. However when you're online and the process is interrupted due to connection shortage, the client does not know whether the new records are successfully saved at server side. This creates confusion what should be done when deleting the newly added records.
To overcome this, re-fetch data will be performed. If the new records are successfully saved, the data will be retrieved during the re-fetch process, which will then properly marked as deleted. The deleted records will then be synchronized to the server subsequently, making server state equally identical with the client state.
When you update existing records at client side, the particular records will be marked as modified. When the process is interrupted due to connection shortage, the client does not know whether the records are successfully updated at server side.
To overcome this, re-fetch data will be performed which allows the pending records to be finalized with the following conditions:
Data Synchronization Using Intersoft Application Framework
Crosslight also provides complete implementation of data synchronization process that you can utilize by implementing Crosslight App Framework. This framework already implements the common data synchronization processes such as:
- Creating SQLite table for entity that contributed in data synchronization
- Perform data synchronization and storing synchronization info after the process is completed
- Support multi-user data synchronization
- Support multiple sync channel
- Queue management
- Resolve stall mechanism
- Provides API to store pending changes
To learn more how to enable data synchronization using the App Framework, see Enabling Data Synchronization.