Cloud Sync or Local Data - why not both?
I finally got tired enough of searching for the perfect personal finance net-worth tracker that I decided to just bite-the-bullet (what a strange expression?) and build my own. I specifically wanted:
IMO most services end up trying to do too much and are over-complicated and laborious to use as a result. Synchronizing account balances via “connection” services is risky because you’re giving credentials to a 3rd part and they simply never work reliably anyway, but requiring entry of every transaction to keep balances updates is too much of a chore, and I also don’t want to have to dedicate a server to hosting it. I just want to track the balances of accounts over time, and update them every week or month or so to get an idea of big-picture progress.
It’s been a fun side-project to work on and has come together really well, here’s how it looks:
It’s built using SvelteKit with the as-of-yet unrelease Svelte 5 (labelled as Release Candidate) so I could get some practical experience of the new Runes / Signals (I believe you should always be learning via your side projects). It’s a client-side app so takes advantage of free Firebase Hosting and uses Firestore for data storage. I always try to develop projects as though they might become well-used, and design them to be efficient and cost-effective, especially anything “cloud” so I don’t go viral due to a horror story of runaway costs. An entire portfolio of 10 years worth of daily balances for 20+ accounts can easily be stored in a single Firestore document with the added benefit that it is then super-simple to secure correctly with a clear and straightforward security rule, and there is a generous free daily usage tier so it would be easy to make it self-supporting if / when I ever did decide to make it public.
One part that I thought would be important if it was made pubic is local-first storage. Not everyone will want to sync their data to the cloud, although there are of course benefits to doing so (as a backup, and to allow secure access from any device). But I get that people might not want to share it (a surprising number don’t care). In fact, I wouldn’t want them to share it by default because even though it’s unlikely to cost me much, if anything, I didn’t want to pay for an app for my own use, so I certainly don’t want to end up paying for anyone else! Allowing people to store their data locally first seems like a good option, and would enable people to use it entirely offline, possibly as a trial or free tier option. Even people who did want to synchronize things might want to have control over exactly when it did and didn’t sync, and of course it’s important to provide clear indication on the state of any cloud syncing.
It turned out to be way easier to accomplish than I expected thanks to Svelte 5 and Firestore. I’ll describe the moving parts to give you a better idea of how it works if you want to use this technique in your own applications. This is what we’ll build:
NOTE: I recorded the video before adding a little more to the example code - apologies for the slight-mismatch.
Any use of Firestore begins with the Firebase Client SDK. My firebase setup for most apps always ends up similar, with small pieces to configure and initialize the individual services used and keep things tidy should any one of them need to become more complex. The firebase config can be hard-coded but I like to keep it as .env
variables which SvelteKit makes easy to consume. Note, these do have to be publicly accessible on the client - it’s totally fine that people can see them! Whether you use static
env variables (these are compiled into the JavaScript for the site) or the dynamic
env (more suited to automated infrastructure deployments, which may create the project instances, or when you want to deploy the same code to different test, staging, and live environments) doesn’t change a whole lot, it’s just slightly different import and syntax.
/src/lib/firebase/config.ts
import {
PUBLIC_FIREBASE_API_KEY,
PUBLIC_FIREBASE_APP_ID,
PUBLIC_FIREBASE_AUTH_DOMAIN,
PUBLIC_FIREBASE_MEASUREMENT_ID,
PUBLIC_FIREBASE_MESSAGING_SENDER_ID,
PUBLIC_FIREBASE_PROJECT_ID,
PUBLIC_FIREBASE_STORAGE_BUCKET,
} from '$env/static/public'
export const config = {
apiKey: PUBLIC_FIREBASE_API_KEY,
authDomain: PUBLIC_FIREBASE_AUTH_DOMAIN,
projectId: PUBLIC_FIREBASE_PROJECT_ID,
storageBucket: PUBLIC_FIREBASE_STORAGE_BUCKET,
messagingSenderId: PUBLIC_FIREBASE_MESSAGING_SENDER_ID,
appId: PUBLIC_FIREBASE_APP_ID,
measurementId: PUBLIC_FIREBASE_MEASUREMENT_ID,
}
The config is used to initialize the app which is the base instance for any client-side Firebase SDK service:
/src/lib/firebase/app.ts
import { initializeApp } from 'firebase/app'
import { config } from './config'
export const app = initializeApp(config)
The Firestore service is then initialized from the app being sure to enable the local cache feature which stores data in IndexedDB and manages synchronizing data between separate browser tabs and between the client and the server. This is the part that enables local / offline use with Firestore.
/src/lib/firebase/firestore.ts
import { initializeFirestore, persistentLocalCache, persistentMultipleTabManager } from 'firebase/firestore'
import { app } from './app'
export const firestore = initializeFirestore(app, {
localCache: persistentLocalCache({
tabManager: persistentMultipleTabManager()
}),
})
I also initialize firebase auth
, for sign-in, and storage
for blobs / icons, but those aren’t important for this discussion so I won’t bore you with them. They are even simpler.
If you’re unfamilar with Firestore, it is a cloud document Database-as-a-Service that allows you to subscribe to receive updates to single documents, collections, or queries. One of the added benefits of subscribing to the data rather than just trying to do a fetch is that you immediately get some protection from offline issues - if there is no network connection the user can still be shown a cached copy of the data from when it was last read (it’s cached in IndexedDB) and the user can even write updates to it while disconnected, and when connectivity is restored things are synched up with the server.
Normally you just let this happen and get this amazing functionality for free but in our case (and if you want to provide a more informative UI showing the state of the synching) you can request extra meta-data as part of the subscription that tells you whether the data is from the local cache or the server, and whether there are any pending writes still waiting to be pushed to the server.
This is a basic Firestore subscription including the additional metadata - the onSnapshot
function returns an unsubscribe function which makes wrapping Firestore with a Svelte Store very easy. Svelte 5 is a little more cumbersome to use, but we’ll show how to do that later:
// create a reference to the document we want to subscribe to
// (a security rule will limit users to only access their own data)
const ref = doc(firestore, `portfolio/${user_id}`)
// create a subscription that will receive metadata changes:
onSnapshot(ref, { includeMetadataChanges: true }, (snap) => {
// check if the data exists before using it ...
if (snap.exists()) {
// it's good practice to use serverTimestamp when writing dates
// but they are only known once data has been written to the server
// so if using local-first writes, we can estimate the values
const data = snap.data({ serverTimestamps: 'estimate' })
}
})
So we can subscribe to a document and the callback will execute anytime the data is read from the local cache, is updated from the server (which could be us on a different device) or is written to. If we’re online and write data it will run twice - once when the data is written locally and again once it’s persisted on the server. This is called “latency compensation” and avoids any delay in the UI being updated from having to do a round-trip to the server. The metadata
object flags make reflecting pending writes in the UI very easy - we can use these flags to show the precise status.
Which is of course what we’re going to do here …
Let’s think about the different states our app data synching can be in. As well as Firestore transparently handling offline use due to transient connectivity issues (i.e. we’re on “LieFi”) we can also explicitly enable and disable the network ourselves. So we’ll have a flag to indicate whether we should be set to local
mode or cloud
mode, and as we need to know this any time the app runs, it will need to be stored in localStorage
to be accessible at startup so we can set it before we subscribe. This won’t show in the metadata
flags, we’ll maintain this state ourselves and call enableNetwork(firestore)
or disableNetwork(firestore)
when it changes. NOTE: these don’t change the network state of the device itself, just the Firestore client behavior - you can still be online and connected to the network but have Firestore work offline.
Seems easy - we’ll have called disableNetwork(firestore)
to turn it off so we’re working offline. At this point we know that we haven’t written anything, but we don’t know if we’re up-to-date or not because changes may have been written to the server from another device. Our callback will include:
{
fromCache: true
hasPendingWrites: false
}
If we do write to the document while we’re offline, the write will “succeed” locally and be reflected in the UI, but those changes won’t have been persisted on the server yet. The metadata will indicate this with:
{
fromCache: true
hasPendingWrites: true
}
If synching is enabled by calling enableNetwork(firestore)
, then any pending writes will be written to the server and any server updates since we were last online will be pulled down, so once enabled the status will be one of:
Before any local changes have been pushed to the server the metadata
will be:
{
fromCache: true
hasPendingWrites: true
}
The observant among you will see that these flags are the same as when we have pending writes in local
mode. Of course, until they are written to the server the status is the same, the only difference is that we enabled the write to the server.
If we have no local changes or our local changes have been written, the client still needs to check that there were no new writes that we need to read. While this is happening we’ll see:
{
fromCache: true
hasPendingWrites: false
}
Again, the flags are the same as when the network is disabled and we have no pending writes and the difference is that we’re in cloud
mode so it does the check with the server to fetch any writes that happened.
Finally, we’ve done the two-way sync, our writes are written and our reads are read. We get an indication that we’re fully up-to-date from:
{
fromCache: false
hasPendingWrites: false
}
So there are 5 different states that we might be in. Instead of dealing with the three separate values, let’s derive a simpler Status
type from the mode
, metadata.fromCache
, and metadata.hasPendingWrites
flags.
First, the types - we’re not using a Typescript enum
because they suck, I find this is a much nicer approach:
export const Status = [
'disabled',
'pending',
'uploading',
'downloading',
'synchronized',
] as const
export type Status = (typeof Status)[number]
export type Mode = 'local' | 'cloud'
We’re going to wrap everything into a class so our flags will be internal props and we’ll derive the status from them - the switch statements represent the status logic described above.
export class Storage {
// internal state
private _mode = $state<Mode>('local')
private fromCache = $state(false)
private hasPendingWrites = $state(false)
// public status based on the internal state
status = $derived.by<Status>(() => {
switch (this.mode) {
case 'local':
switch (this.hasPendingWrites) {
case true:
return 'pending'
case false:
return 'disabled'
}
case 'cloud':
switch (this.hasPendingWrites) {
case true:
return 'uploading'
case false:
switch (this.fromCache) {
case true:
return 'downloading'
case false:
return 'synchronized'
}
}
}
})
}
The reason that _mode
is internal and isn’t mode
(and public) is because we’ll be adding our own accessors so we can call the appropriave methods to enable or disable the network when it is set. We could have used an effect for this, but I like the directness of the setter.
export class Storage {
// internal state
private _mode = $state<Mode>('local')
private fromCache = $state(false)
private hasPendingWrites = $state(false)
// public status based on the internal state
status = $derived.by<Status>(() => {
switch (this.mode) {
case 'local':
switch (this.hasPendingWrites) {
case true:
return 'pending'
case false:
return 'disabled'
}
case 'cloud':
switch (this.hasPendingWrites) {
case true:
return 'uploading'
case false:
switch (this.fromCache) {
case true:
return 'downloading'
case false:
return 'synchronized'
}
}
}
})
+ constructor() {
+ // disable the network until we know it should be enabled
+ disableNetwork(firestore)
+
+ // TODO: subscribe to data here
+
+ // read the mode from localStorage, defaulting to local
+ this._mode = <Mode>localStorage.getItem('mode') || 'local'
+
+ // if cloud mode is enabled, turn on the network
+ // this will change the behavior of the subscription
+ if (this._mode === 'cloud') {
+ enableNetwork(firestore)
+ }
+ }
+
+ get mode() {
+ return this._mode
+ }
+
+ set mode(mode: Mode) {
+ // persist the new mode for next startup
+ localStorage.setItem('mode', mode)
+
+ // updated the state
+ this._mode = mode
+
+ // set network based on mode
+ switch (mode) {
+ case 'local':
+ disableNetwork(firestore)
+ break
+ case 'cloud':
+ enableNetwork(firestore)
+ break
+ }
+ }
}
We just need to plug in our subscription so it loads the data and sets the appropriate state flags. Before that though, some “best practice” that I like to use when working with Firestore is to define types for the in-memory version as well as the database version plus a converter to translate between them. The reason for this is two-fold: Firestore has some specific types for highprecision timestamps, binary blobs etc… and also to make it easy to do schema versioning. I created a lib to make writing client-side and server-side compatible Firestore data converters easier which you may find useful but for this example we just need a client-side converter and will just handle a date timestamp:
import { Timestamp, type FirestoreDataConverter } from 'firebase/firestore'
// in memory data model
export interface Data {
count: number
updated: Date
}
// database model (using a Firestore Timestamp)
export interface DBData {
count: number
updated: Timestamp
}
// converter for translating between them
export const dataConverter: FirestoreDataConverter<Data, DBData> = {
toFirestore(data: Data) {
const { count, updated } = data
return {
count,
updated: Timestamp.fromDate(updated), // convert JS Date to Timestamp
} as DBData
},
fromFirestore(snapshot, options) {
const data = snapshot.data(options) as DBData
const { count, updated } = data
return {
count,
updated: updated.toDate(), // convert Timestamp to JS Date
} as Data
},
}
Here’s where the data is managed in our class along with some methods to update it.
export class Storage {
// internal state
private _mode = $state<Mode>('local')
private fromCache = $state(false)
private hasPendingWrites = $state(false)
// public status based on the internal state
status = $derived.by<Status>(() => {
switch (this.mode) {
case 'local':
switch (this.hasPendingWrites) {
case true:
return 'pending'
case false:
return 'disabled'
}
case 'cloud':
switch (this.hasPendingWrites) {
case true:
return 'uploading'
case false:
switch (this.fromCache) {
case true:
return 'downloading'
case false:
return 'synchronized'
}
}
}
})
+ // the data being stored (frozen, as it's replaced when updated)
+ data = $state.frozen<Data>({ count: 0, updated: new Date() })
+
+ // reference to the document
+ ref: DocumentReference<Data, DBData>
+
- constructor() {
+ constructor(uid: string) {
+ this.ref = doc(firestore, `data/${uid}`).withConverter(dataConverter)
// disable the network until we know it should be enabled
disableNetwork(firestore)
- // TODO: subscribe to data here
+ // subscribe to data
+ $effect(() => {
+ let timeout = 0
+ return onSnapshot(this.ref, { includeMetadataChanges: true }, (snap) => {
+ this.fromCache = snap.metadata.fromCache
+
+ // debouncing the `hasPendingWrites` makes for a more pleasant UI without flickering
+ if (timeout) window.clearTimeout(timeout)
+ timeout = window.setTimeout(
+ () => (this.hasPendingWrites = snap.metadata.hasPendingWrites),
+ snap.metadata.hasPendingWrites ? 60 : 240
+ )
+
+ if (snap.exists()) {
+ this.data = snap.data({ serverTimestamps: 'estimate' })
+ }
+ })
+ })
// read the mode from localStorage, defaulting to local
this._mode = <Mode>localStorage.getItem('mode') || 'local'
// if cloud mode is enabled, turn on the network
// this will change the behavior of the subscription
if (this._mode === 'cloud') {
enableNetwork(firestore)
}
}
get mode() {
return this._mode
}
set mode(mode: Mode) {
// persist the new mode for next startup
localStorage.setItem('mode', mode)
// updated the state
this._mode = mode
// set network based on mode
switch (mode) {
case 'local':
disableNetwork(firestore)
break
case 'cloud':
enableNetwork(firestore)
break
}
}
+ async increment() {
+ await setDoc(this.ref, {
+ count: increment(1),
+ updated: serverTimestamp(),
+ }, { merge: true })
+ }
+
+ async decrement() {
+ await setDoc(this.ref, {
+ count: increment(-1),
+ updated: serverTimestamp(),
+ }, { merge: true })
+ }
}
NOTE: the normal way to “update” a Firestore document is with the updateDoc
method. We’re using setDoc
with the merge
option so we can do a partial update, leaving other fields unchanged, and also ensure a new document is created if it doesn’t already exist (i.e. perform an upsert). Also we never update our data directly - we always update Firestore and that will change the $state
data via the subscription (which is why the data is a frozen instance). In the real app there is $derived
state to apply exchange rates to foreign currency balances and to aggregate the account category totals and overall asset vs liability totals and ultimately the total net-worth, all over time for charting.
We can create an instance of our storage, passing in the current users unique user id:
const storage = new Storage(auth.currentUser.uid)
One very important piece is to ensure that each user can only see and update their own data. The Firestore security rule to enforce this is very straightforward, so it’s hard to mess up. Tip: you can unit test your Firestore security rules, for extra confidence that security is always correctly configured - there’s simnply no excuse to get this wrong!
rules_version = '2';
service cloud.firestore {
match /databases/{database}/documents {
// disable all access to all documents by default
match /{document=**} {
allow read, write: if false;
}
// allow the authenticated user to access their own data
match /data/{uid} {
allow read, write: if request.auth != null && request.auth.uid == uid;
}
}
}
Our UI component is surprisingly simple. We just need to output different status icons based on the storage state, and bind some UI elements to call the functions to update it:
<script lang="ts">
import { user } from '$lib/firebase'
import { disabled, downloading, pending, synchronized, uploading } from './icons'
import { Storage, Status } from './storage.svelte'
const storage = new Storage($user.uid)
const statusIcon: Record<Status, string> = {
disabled: disabled,
pending: pending,
uploading: uploading,
downloading: downloading,
synchronized: synchronized,
}
const statusLabel: Record<Status, string> = {
disabled: 'Disabled',
pending: 'Disabled (Pending)',
uploading: 'Updating',
downloading: 'Synchronizing',
synchronized: 'Synchronized',
}
// style the buttons to highlight the active mode
let localColor = $derived(storage.mode === 'local' ? 'bg-slate-600 text-white' : 'bg-gray-100')
let cloudColor = $derived(storage.mode === 'cloud' ? 'bg-slate-600 text-white' : 'bg-gray-100')
</script>
<div class="px-12 py-16">
<p class="flex items-center gap-2">
Cloud Storage:
{@html statusIcon[storage.status]}
{statusLabel[storage.status]}
</p>
<div class="my-4 flex items-center gap-4">
<button class="rounded-md px-4 py-1.5 {localColor}" onclick={() => (storage.mode = 'local')}>
Disable
</button>
<button class="rounded-md px-4 py-1.5 {cloudColor}" onclick={() => (storage.mode = 'cloud')}>
Enable
</button>
</div>
<div class="my-4 flex items-center gap-4">
<button class="rounded-md bg-gray-50 px-4 py-1.5" onclick={() => storage.increment()}>
Inc
</button>
<span class="text-center font-semibold">
{storage.data.count}
</span>
<button class="rounded-md bg-gray-50 px-4 py-1.5" onclick={() => storage.decrement()}>
Dec
</button>
</div>
</div>
The icons used are just SVG strings, but for completeness:
export const synchronized = /* svg */`<svg class="size-5 text-green-600" height="24px" viewBox="0 -960 960 960" width="24px" fill="currentColor"><path d="m414-280 226-226-58-58-169 169-84-84-57 57 142 142ZM260-160q-91 0-155.5-63T40-377q0-78 47-139t123-78q25-92 100-149t170-57q117 0 198.5 81.5T760-520q69 8 114.5 59.5T920-340q0 75-52.5 127.5T740-160H260Zm0-80h480q42 0 71-29t29-71q0-42-29-71t-71-29h-60v-80q0-83-58.5-141.5T480-720q-83 0-141.5 58.5T280-520h-20q-58 0-99 41t-41 99q0 58 41 99t99 41Zm220-240Z"/></svg>`
export const disabled = /* svg */`<svg class="size-5 text-gray-500" height="24px" viewBox="0 -960 960 960" width="24px" fill="currentColor"><path d="M792-56 686-160H260q-92 0-156-64T40-380q0-77 47.5-137T210-594q3-8 6-15.5t6-16.5L56-792l56-56 736 736-56 56ZM260-240h346L284-562q-2 11-3 21t-1 21h-20q-58 0-99 41t-41 99q0 58 41 99t99 41Zm185-161Zm419 191-58-56q17-14 25.5-32.5T840-340q0-42-29-71t-71-29h-60v-80q0-83-58.5-141.5T480-720q-27 0-52 6.5T380-693l-58-58q35-24 74.5-36.5T480-800q117 0 198.5 81.5T760-520q69 8 114.5 59.5T920-340q0 39-15 72.5T864-210ZM593-479Z"/></svg>`
export const pending = /* svg */`<svg class="size-5 text-red-600" height="24px" viewBox="0 -960 960 960" width="24px" fill="currentColor"><path d="M792-56 686-160H260q-92 0-156-64T40-380q0-77 47.5-137T210-594q3-8 6-15.5t6-16.5L56-792l56-56 736 736-56 56ZM260-240h346L284-562q-2 11-3 21t-1 21h-20q-58 0-99 41t-41 99q0 58 41 99t99 41Zm185-161Zm419 191-58-56q17-14 25.5-32.5T840-340q0-42-29-71t-71-29h-60v-80q0-83-58.5-141.5T480-720q-27 0-52 6.5T380-693l-58-58q35-24 74.5-36.5T480-800q117 0 198.5 81.5T760-520q69 8 114.5 59.5T920-340q0 39-15 72.5T864-210ZM593-479Z"/></svg>`
export const uploading = /* svg */`<svg class="size-5 text-red-600" height="24px" viewBox="0 -960 960 960" width="24px" fill="currentColor"><path d="M260-160q-91 0-155.5-63T40-377q0-78 47-139t123-78q25-92 100-149t170-57q117 0 198.5 81.5T760-520q69 8 114.5 59.5T920-340q0 75-52.5 127.5T740-160H520q-33 0-56.5-23.5T440-240v-206l-64 62-56-56 160-160 160 160-56 56-64-62v206h220q42 0 71-29t29-71q0-42-29-71t-71-29h-60v-80q0-83-58.5-141.5T480-720q-83 0-141.5 58.5T280-520h-20q-58 0-99 41t-41 99q0 58 41 99t99 41h100v80H260Zm220-280Z"/></svg>`
export const downloading = /* svg */`<svg class="size-5 text-red-600" height="24px" viewBox="0 -960 960 960" width="24px" fill="currentColor"><path d="M260-160q-91 0-155.5-63T40-377q0-78 47-139t123-78q17-72 85-137t145-65q33 0 56.5 23.5T520-716v242l64-62 56 56-160 160-160-160 56-56 64 62v-242q-76 14-118 73.5T280-520h-20q-58 0-99 41t-41 99q0 58 41 99t99 41h480q42 0 71-29t29-71q0-42-29-71t-71-29h-60v-80q0-48-22-89.5T600-680v-93q74 35 117 103.5T760-520q69 8 114.5 59.5T920-340q0 75-52.5 127.5T740-160H260Zm220-358Z"/></svg>`
One thing you want to pay attention to is the design of your schema. You have to put some thought into it to ensure that updates can be replayed in order to support offline use. In particular you can’t do any updates that require transactions as those can only be done online.
As an example of the difference for the counter: a transaction would work by reading the existing document, incrementing the value, and writing it back. Not only can this only be done online but it also suffers from contention if multiple devices are trying to update the same record at the same time (more of an issue in a multi-user scenario of course). Firestore will handle this by retrying transactions but by using the atomic increment update we no longer have the same constraints - multiple updates could increment or decrement a counter and it wouldn’t matter which order they executed in.
Often it just requires a little thought …
In this particular example one of the most common actions will be updating balances for accounts. How can that be modelled so that you can do it with single update operations and not transactions? You might assume that balances would be an array with a date and value for each entry? It would be impossible to update balances with an atomic operation if that were the case. The trick was to model the schema so that each day for each account was directly addressable - each account was in an object map, and daily balances were also an object map where the key was an encoded string based on the date. Not only could a single daily balance be updated with a non-transactional update, it also helped to store the data more efficiently too as the dates were stored in a much more compact format (redix 36 encoded string of the unix date). Here’s how that looks in the database:
{
accounts: {
tKm: {
category: "bank",
currency: "CAD",
history: {
sf76o0: 2500,
sfcqo0: 2500.75,
sfteo0: 3500.75,
sfyyo0: 3512.56,
},
institution: "u5l",
name: "Joint",
type: "savings",
}
}
}
IDs for accounts and institutions are 3 character nano-ids, and it’s really better to store and work with integer values when dealing with money, only adding the decimal places needed for display.
Offline first is a useful feature that Firestore can enable and the great thing is that if you use it correctly, it doesn’t cost a small fortune. If you do the math, it can be substantially less expensive to use than alternatives such as Replicache which also require you to build and host your own backend (including security) - you get all that included as part of Firestore with 50,000 reads and 20,000 writes per day for free so only need to pay for usage beyond that. Even if you do go past the free tier it can be as low as $0.03 / $0.09 per 100,000 reads and writes. Pretty cost effective for the functionality and reliability you get.