How to create a synchronized Online/Offline data application with EntityFramework, JavaScript and JayData

19 June 2013

featured post

 

This post is a part of a series, find more here:

 

There are quite a few scenarios when an offline and an online version of the database (or a subset of it) is required to provide some application features. Such scenario can be an application that is not all the time connected to the net but needs to collect data (offline) for later submission. Another example can be an offline cache on the client for faster data access or to preserve network bandwidth or to provide all time access to some data regardless of the network state.

 

In this post we will focus on the first scenario, “collect locally, push to server for shared access”.  Upcoming posts will detail Online to Offline, and Local Data Cache scenarios.

 

The architecture

In our example we will create an offline capable HTML5 Application that will collect some user input (Task items) and store it in its persistent local database. Every once in a while we push all the new items to the server using REST and the HTTP protocol to store data online. Collecting user input and submitting new items to the server will be two decoupled workflows. This allows for seamless user experience even if network status goes offline or changes frequently.

 

Data model

The “TodoItems” table

TodoItems will be our only entity type for now. It will have a Task and a Completed field to store use input, and an InSync boolean field to keep track of new items.

The key field

Synchronization can be implemented with any key type, but there is one type that is especially optimal for it: the UUID/ GUID / $data.Guid, it has many names. The GUID type lets us share auto generated primary key values between the client and server.

Client and Server data model

We are about to share the very same model definition between the server and client store when creating the application. This simplifies a lot in the application code, as TodoItems will be able to come and go freely between the layers, so we don’t have to transform offline TodoItem to online TodoItem. And also their item ID’s will not collide either.

 

Server tier

We need to build an OData compatible REST service. This can be done with Entity Framework and WCF Data services, or ASPNET WebAPI. Or we can use a cloud based OData service as well.

Client tier

 

On the client we will use JayData as the data abstraction layer to provide a common developer experience over handling data regardless it’s being online or offline or in between. Under the storage agnostic EntityContext surface four JayData providers will be engaged, depending the client environment capabilities IndexedDB, webSQL or HTML5 localStore for storing offline data and the OData provider to communicate with the server.

 

Let’s build the application

Create the online data store

We will create an OData compatible REST service endpoint to accept Todo items from the clients. It has a number of possible ways, pick the one that fits your needs the best. Currently only the EF + WCF has a detailed walkthrough here.

  • Entity Framework 5.0 and WCF Data Services to store data in an SQL Server and if you don’t really want to write lot’s of code.
  • ASP.NET WebAPI if you need a code based backend solution.
  • JayStorm Open if you need an easy to use cloud solution.

 

No matter which option we take, we will conclude in a service with the following metadata:

todo-service-odata

Create The Client Application

Our client app will be a simple Single Page Application. If you created your online service with Visual Studio, simply open that project and start extending it with the following. If your OData endpoint is a cloud service then just start a new Empty Web project.

Open package manager console and add two packages: JayData and jQuery.

 

PM>Install-Package JayData; Install-Package jQuery 

 

Along with a bunch of script files with JayData package you get the JaySvcUtil.exe that creates JavaScript client classes from a WCF Data Service / OData endpoint metadata. We will use this utility to generate the TodoItem and the TodoContainer classes. We will eventually highjack types generated from the OData endpoint and will use them for the webSql storage as well.

Use JaySvcUtil to generate client data classes

Open a command prompt and navigate to you website project folder and invoke JaySvcUtil.exe with the –metadataUri param pointing to http://youlocalsite/OnlineDB.svc/$metadata

 

C:…\> >JaySvcUtil.exe -m http://localhost:30536/TodoService.svc/$metadata -o TodoDB.js

CropperCapture[91]

 

A new file TodoDB.js is generated and placed in your web site folder. You may have to press “Show all items” in the solution explorer to see. An occasional refresh may also help. Right click on the file to include in the project. Peeking into it you should see this:

Contents of the TodoDB.js file

//////////////////////////////////////////////////////////////////////
// Auto generated by JaySvcUtil.exe http://JayData.org for more info //
//            OData v3                                              //
//////////////////////////////////////////////////////////////////////
(function(global, $data, undefined) {

   $data.Entity.extend('TodoDBModel.TodoItem', {
    'Id': { key:true,type:'Edm.Guid',nullable:false,required:true },
    'Task': { type:'Edm.Binary',nullable:false,required:true,maxLength:200 },
    'Completed': { type:'Edm.Boolean',nullable:false,required:true },
    'InSync': { type:'Edm.Boolean',nullable:false,required:true }
   });

  $data.EntityContext.extend('OfflineDataCollector.TodoDBEntities', {
    TodoItems: { type: $data.EntitySet, elementType: TodoDBModel.TodoItem }
  });
  
  OfflineDataCollector.context = new OfflineDataCollector.TodoDBEntities({
      name: 'oData',
      oDataServiceHost: 'http://localhost:30536/TodoService.svc'
  });
})(window, $data);

We could use these classes without any further ado, but to make the following parts easier to read, let’s do some modifications. Rename TodoDBModel.Item to just Item, OfflineDataCollector.TodoDBEntities to TodoDB, make sure to update the elementType as well, and remove the context instantiation part from the end. You should have to come to this:

Contents of the modified TodoDB.js file

//////////////////////////////////////////////////////////////////////
// Auto generated by JaySvcUtil.exe http://JayData.org for more info//
//            OData v3                                              //
//////////////////////////////////////////////////////////////////////
(function(global, $data, undefined) {

   $data.Entity.extend('TodoItem', {
    'Id': { key:true,type:'Edm.Guid',nullable:false,required:true },
    'Task': { type:'Edm.Binary',nullable:false,required:true,maxLength:200 },
    'Completed': { type:'Edm.Boolean',nullable:false,required:true },
    'InSync': { type:'Edm.Boolean',nullable:false,required:true }
   });

  $data.EntityContext.extend('TodoDB', {
    TodoItems: { type: $data.EntitySet, elementType: 

TodoItem

 }
  });
  
})(window, $data);

Add a new HTML page

Create and index.html page and reference jQuery, datajs, JayData and the newly forged TodoDB.js  plus add the following additional script block to instantiate a local and a remote context from these classes:

 

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
<script src="Scripts/jquery-1.7.2.js"></script>
<script src="Scripts/datajs-1.0.3.js"></script>
<script src="Scripts/jaydata.js"></script>
<script src="TodoDB.js"></script>
<script>
    var onlinedb = new TodoDB({
        name: 'oData',
        oDataServiceHost: '/TodoService.svc'
    });

    var offlinedb = new TodoDB({
        name: 'sqLite',
        databaseName: 'TodoDB'
    });
</script>
</head>
<body>
</body>
</html>

 

Navigate to your index.html with a browser that supports WebSql: Chrome or Safari. We will get into supporting other browsers later in this article.

 

If all is well you should see the new local database in your browsers console by press F12 or going into the Develop menu in Safari and selecting Resources.

image

Generate and display local data

With our online and offline services now set up, we just need to create some offline data, storing it many TodoItem in our offlinedb.

 

Create the UI to collect user input and list local TodoItems

<body>
    <form style="width:200px;float:left" id="addTaskForm">
        <fieldset>
            <legend>Add Todo</legend>
            <input id="taskInput" />
            <input type="submit" value="Add item" />
        </fieldset>
    </form>
    <ul id="TaskList" style="float:left">
    </ul>
</body>

 

Create the code to add new TodoItems and list existing ones

 

<script>
    var onlinedb = new TodoDB({
        name: 'oData',
        oDataServiceHost: '/TodoService.svc'
    });

    var offlinedb = new TodoDB({
        name: 'sqLite',
        databaseName: 'TodoDB'
    });

    function showTodoItem(todoItem) {
        var li = $('<li>').text(todoItem.Task);
        li.append($('<div>').text('Synchronized: ' + todoItem.InSync));
        $('#TaskList').append($(li));
    }


    function listLocalTodoItems() {
        $('#TaskList').empty();
        offlinedb.TodoItems.forEach(showTodoItem);
    }

    $(function () {
        offlinedb.onReady(listLocalTodoItems);

        $('#addTaskForm').submit(function (evt) {

            offlinedb.TodoItems.add({
                Id: $data.createGuid(),
                Task: $('#taskInput').val()
            });

            offlinedb.saveChanges(function () {
                evt.target.reset();
                listLocalTodoItems();
            });
            //prevent actual form submit
            return false;
        });
    });
</script>

 

Launch your single page application, and add some tasks:

image

 

Submit new data to server

First some rudimentary UI: display two columns of tasks. First column shows local database content, while the second will display online items. We revisit the view we created before:

<body>
    <form class="column" id="addTaskForm">
        <fieldset>
            <legend>Add Todo</legend>
            <input id="taskInput" />
            <input type="submit" value="Add item" />
        </fieldset>
    </form>
    <div class="column">
        <h3>Task List, <a href="#" id="sync">Synchronize</a></h3>
        <ul id="TaskList">
        </ul>
    </div>
    <div class="column">
        <h3>Remote Task List</h3>
        <ul id="RemoteTaskList">
        </ul>
    </div>
</body>

 

Let’s create code for pushing new (InSync === false) items to the server, and setting InSync to true for items that has been successfully saved online. The essence of pushing items to the server is this routine:

    function synchronizeData() {
        offlinedb
            .TodoItems
            .filter("it.InSync === false")
            .toArray(function (todoItems) {
                onlinedb.addMany(todoItems);
                onlinedb.saveChanges(function () {
                    todoItems.forEach(function (todoItem) {
                        offlinedb.attach(todoItem);
                        todoItem.InSync = true;
                    });
                    offlinedb.saveChanges(function () {
                        listLocalTodoItems();
                        listRemoteTodoItems();
                    });
                });
            })
    }

 

This is where we arrive. Try adding new items and pressing Synchronize a number of times to see how it works.

image

 

This is how the index.html page should look like inside:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
 <style>
     .column {
         width:240px;
         float:left;
     }
 </style>
<script src="Scripts/jquery-1.7.2.js"></script>
<script src="Scripts/datajs-1.0.3.js"></script>
<script src="Scripts/jaydata.js"></script>
<script src="TodoDB.js"></script>
<script>
    var onlinedb = new TodoDB({
        name: 'oData',
        oDataServiceHost: '/TodoService.svc'
    });

    var offlinedb = new TodoDB({
        name: 'sqLite',
        databaseName: 'TodoDB'
    });

    function showTodoItem(todoItem, list) {
        var li = $('<li>').text(todoItem.Task);
        li.append($('<div>').text('Synchronized: ' + todoItem.InSync));
        list.append($(li));
    }


    function listLocalTodoItems() {
        $('#TaskList').empty();
        offlinedb.TodoItems.forEach( function(todoItem) {
            showTodoItem(todoItem, $('#TaskList'));
        });
    }

    function listRemoteTodoItems() {
        $('#RemoteTaskList').empty();
        onlinedb.TodoItems.forEach(function (todoItem) {
            showTodoItem(todoItem, $('#RemoteTaskList'));
        });
    }

    function synchronizeData() {
        offlinedb
            .TodoItems
            .filter("it.InSync === false")
            .toArray(function (todoItems) {
                onlinedb.addMany(todoItems);
                onlinedb.saveChanges(function () {
                    todoItems.forEach(function (todoItem) {
                        offlinedb.attach(todoItem);
                        todoItem.InSync = true;
                    });
                    offlinedb.saveChanges(function () {
                        listLocalTodoItems();
                        listRemoteTodoItems();
                    });
                });
            })
    }

    $(function () {
        offlinedb.onReady(listLocalTodoItems);
        onlinedb.onReady(listRemoteTodoItems);

        $("#sync").click(synchronizeData);

        $('#addTaskForm').submit(function (evt) {

            offlinedb.TodoItems.add({
                Id: $data.createGuid(),
                Task: $('#taskInput').val()
            });

            offlinedb.saveChanges(function () {
                evt.target.reset();
                listLocalTodoItems();
            });
            //prevent actual form submit
            return false;
        });

    });

</script>
</head>
<body>
    <form class="column" id="addTaskForm">
        <fieldset>
            <legend>Add Todo</legend>
            <input id="taskInput" />
            <input type="submit" value="Add item" />
        </fieldset>
    </form>
    <div class="column">
        <h3>Task List, <a href="#" id="sync">Synchronize</a></h3>
        <ul id="TaskList">
        </ul>
    </div>
    <div class="column">
        <h3>Remote Task List</h3>
        <ul id="RemoteTaskList">
        </ul>
    </div>
</body>
</html>
Go

Going offline capable

Now that you have a fully working HTML5 application that is able to store data locally, the next step is to turn it into an app that is able to run offline, without network connection present. To do that you have a number of options. You can build a Cordova app from these files to deploy as an App Store application. Or you can use a cache manifest file so that you HTML5 app is persisted locally on the device and can start regardless of being online or offline.

<!DOCTYPE html>
<html 

manifest="cache.manifest"

>
<head>
<title></title>

 

cache.manifest:

CACHE MANIFEST
NETWORK: 
/TodoService.svc
CACHE:
/index.html
/Scripts/jquery-1.7.2.js
/Scripts/datajs-1.0.3.js
/Scripts/jaydata.js
/TodoDB.js
/Scripts/jaydataproviders/oDataProvider.js
/Scripts/jaydataproviders/sqLiteProvider.js

Share this

Comments

  • Missing user avatar
    link

    Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point. You obviously know what youre talking about, why throw away your intelligence on just posting videos to your weblog when you could be giving us something enlightening to read?

Leave a comment

comments powered by Disqus