Skip to main content

IndexedDB and the Cache API — large data, and the conditions for disappearing

None of the three mechanisms from the previous chapter suits large data. Web Storage has a 5 MiB ceiling on the whole area per origin, and its API is synchronous. Cookie limits are smaller still, and cookies attach to requests automatically.

This chapter covers the two mechanisms that lie beyond them. But it is not only about capacity. Boundaries fail to line up here too. The way the spec divides storage and the way implementations divide it disagree, and "stored" and "still there" disagree as well.

What you'll learn in this chapter​

  • That the spec defines the storage key as a tuple of an origin, while implementations also key on the embedding site
  • Why "stored" and "still there" are not the same, and what navigator.storage.persist() changes
  • That IndexedDB is asynchronous and transactional
  • That the Cache API holds pairs of Request and Response
Prerequisites

Builds on the five axes from Browser storage. The Cache API builds on the ideas in HTTP caching and CDNs, and handling asynchronous APIs builds on The event loop.

What this chapter does not cover​

TopicWhere it lives
Choosing among localStorage / sessionStorage / cookiesBrowser storage
Caching the server directs (Cache-Control / ETag)HTTP caching and CDNs
Execution order of async work and microtasksThe event loop
The Partitioned cookie attributeCookies

The spec's boundary and the implementations' boundary do not line up​

The Storage Standard defines the key that divides storage — the storage key — as "a tuple consisting of an origin". That is scheme + host + port.

Yet the same spec places a note right beside that definition.

This is expected to change; see Client-Side Storage Partitioning

The spec itself says this is expected to change. And implementations have already moved ahead.

Implementations do not key on origin alone​

All three browser engines divide storage in third-party contexts — an iframe embedded in another site, for instance — per embedding site.

ImplementationHow it divides
Firefox"double-keys all client-side state by the origin of the resource being loaded and by the top-level site"
Chromium"that data is now isolated and only available to contexts that share both the same origin and the same top-level site". Enabled for all users from Chrome 115
WebKit"Third-party LocalStorage and IndexedDB are partitioned per first-party website and also made ephemeral"

So when widget.example is embedded in both a.com and b.com, the two get separate areas. From the embedded side it is the same origin, yet the contents are not shared.

widget.example embedded in a.com → area 1
widget.example embedded in b.com → area 2
widget.example opened directly   → area 3

The division exists to prevent tracking across sites. Even when the same script is embedded in many sites, it cannot stitch the visits together.

Differences between implementations remain​

What they agree on stops at "the embedding site is part of the key". The details are not aligned.

DifferenceDetail
PersistenceWebKit does not write third-party storage to disk (ephemeral). It goes away when the browser exits
NestingChromium adds an ancestor bit to the key. It is set when a cross-site document sits anywhere in between
CookiesFirefox partitions cookies dynamically. The Storage Access API can request access to the unpartitioned area

"Major browsers divide it" is safe as a shared premise; past that point you have to check the implementation. If you ship a third-party widget, the safe assumption is that storage does not work in a third-party context.

The Partitioned attribute from chapter 26 is the same story on the cookie side. Cookies have a route for declaring it explicitly through an attribute; Web Storage and IndexedDB have none, and implementations divide them silently.

"Stored" and "still there" are not the same​

A successful write does not guarantee a later read. Browsers discard data when storage runs tight.

best-effort and persistent​

The default treatment is best-effort. As the name says, the browser does its best, but there is no guarantee.

ModeSubject to eviction?
best-effort (default)Yes
persistentNo

Eviction goes in order of least recently used. An origin that calls navigator.storage.persist() and is granted permission drops out of that pool.

const persisted = await navigator.storage.persist();

A return value of true means you are persistent. The browser decides whether to grant it, so calling it does not mean it goes through. Some implementations look at whether the site is installed to the home screen, bookmarked, or visited often.

Some implementations delete what goes unused​

When cross-site tracking prevention is on, Safari deletes script-created data for an origin that has had no user interaction in the last seven days of browser use. This has nothing to do with storage pressure; time alone triggers it. Cookies set by the server are exempt.

For a service visited at long intervals, data cached for offline use may be disappearing every time. It surfaces as "it never reproduced in development, but reports come in from production".

What this means for design​

Do not make browser storage the only place the data lives. Build so that losing it is recoverable.

  • Use it as a copy of data that lives on the server
  • Keep a path that re-fetches when it is gone
  • For data you cannot afford to lose, request persistent and check the return value

IndexedDB — asynchronous and transactional​

MDN describes IndexedDB as "a transactional database system, like an SQL-based RDBMS" and says operations are "done asynchronously, so as not to block applications".

Two things separate it from Web Storage.

AspectWeb StorageIndexedDB
APISynchronousAsynchronous
Unit of consistencyNone (one operation at a time)Transaction
Value typesStrings onlyAnything structured-cloneable (objects / Blobs / files)

The synchronous localStorage stalls the main thread while it reads or writes. Even within the 5 MiB ceiling, handling a large value freezes painting and interaction. IndexedDB is asynchronous, so this does not happen.

Because it has transactions, writing several records together leaves no half-finished state when something fails midway. The raw API is event-based and awkward to write against, so in practice you wrap it in a thin Promise-based library. What follows is that form.

const tx = db.transaction('drafts', 'readwrite');
await tx.objectStore('drafts').put({ id: 1, body: '...' });
await tx.done;

It follows the same-origin policy, so another origin cannot read it. As the first half showed, though, third-party contexts divide further by the embedding site.

The Cache API — pairs of request and response​

What the Cache API holds is a pair of Request and Response. The request is the key and the response is the value.

const cache = await caches.open('assets-v1');
await cache.add('/styles/main.css');
const hit = await cache.match('/styles/main.css');

MDN states that "An origin can have multiple, named Cache objects" and that "they don't expire unless deleted". You can hold several named areas, and nothing expires until you delete it explicitly.

That is where it differs from HTTP caching. With Cache-Control the server dictates the lifetime and the browser decides automatically. With the Cache API your application code puts things in, and they stay until removed. The usual pattern is a versioned name (assets-v1), deleting the old area once the new version is in.

How this relates to service workers​

The Cache API is discussed alongside service workers, but it is not service-worker-only. Page scripts can use caches too.

A service worker is a script that sits between the page and the network. It can intercept the fetch a page issues and decide whether to answer from cache or let it through to the network.

page → [service worker] → network
             ↓
         Cache API

Sitting there, it can answer even when offline. With the network down, answering from cache keeps the application running.

Service workers are registered per origin and outlive the page. They are also subject to third-party partitioning — they appear both in Firefox's list and in WebKit's description.

Measuring how much you can use​

The Storage Standard's table of registered storage endpoints assigns 5 MiB to localStorage and sessionStorage, and assigns no value to caches and indexedDB.

Do not read that as "no limit". The absence is a per-endpoint limit; the origin as a whole still carries a quota the browser decides.

ImplementationRough best-effort ceiling
FirefoxThe smaller of 10% of disk and 10 GiB. The 10 GiB applies as a group limit across origins on the same site
Chromium60% of disk
WebKitAround 60%. Around 15% for other apps that embed web content

It is the same shape as the 4096 bytes for cookies. The value the spec writes and the limit an implementation enforces are different things, and looking at only one of them trips you up.

You can measure what is actually available at runtime.

const { quota, usage } = await navigator.storage.estimate();

The values are approximate. Browsers round them to frustrate tracking, so they are not usable for decisions right at a boundary.

Common misconceptions​

"IndexedDB and the Cache API have no size limit" — The spec merely assigns no per-endpoint value. The origin as a whole carries the browser's quota, calculated differently by each implementation.

"A successful write means it is still there" — The default is best-effort and subject to eviction. There is no guarantee until navigator.storage.persist() is granted.

"Calling persist() makes it permanent" — You can call it; whether it is granted is the browser's decision. You have to check the return value.

"Storage divides by origin" — That is what the spec defines, but the spec itself notes the definition is expected to change, and all three implementations also key third-party contexts on the embedding site.

"The Cache API needs a service worker" — Page scripts can use caches as well. What needs a service worker is intercepting fetch.

Exercises​

Q1. You ship an analytics widget. An iframe of widget.example is embedded in both a.com and b.com, and it writes a visitor identifier to IndexedDB. Can the identifier written on a.com be read on b.com?

Answer and explanation

Answer: No.

All three implementations divide third-party storage by the embedding top-level site. From widget.example's point of view it is the same origin, but inside a.com and inside b.com are separate areas.

Preventing exactly this kind of cross-site tracking is why the division was introduced in the first place. It is working as intended.

On WebKit, third-party storage is additionally not written to disk. It is gone once the browser exits.

If you ship a third-party widget, design for storage not working in third-party contexts. When you need identity, build a path where the embedding site hands it to you explicitly.

Q2. An offline-capable notes app stores drafts in IndexedDB. After showing "Saved", users report the data is gone when they open the app the following week. Before suspecting an implementation bug, what do you check?

Answer and explanation

Answer: Whether the storage is persistent, and which browser the reports come from.

Even when the write succeeded, the default best-effort mode is subject to eviction. There are two things to check.

CheckDetail
EvictionUnder storage pressure, the least recently used origins are discarded first
Time-based deletionSafari deletes script-created data for origins with no interaction in seven days of browser use when cross-site tracking prevention is on

"Gone when I open it the following week" fits the second one well. It fires on time alone regardless of capacity, so it never reproduces during development.

The response is to request navigator.storage.persist() and check the return value. It is not always granted, so pair it with a path that re-fetches from the server when the data is gone.

Not making browser storage the only home for the data is this chapter's design guidance.

Q3. You are making an image gallery viewable offline. Images are around 2 MB each and there are 100 of them. Do you use IndexedDB or the Cache API?

Answer and explanation

Answer: The Cache API suits this better.

What decides it is "what you look things up by".

MechanismKeySuits
Cache APIRequest (URL)Reusing what came off the network as-is
IndexedDBAny key, with indexesSearching by condition / updating structured data

Images are looked up by URL, which matches the Cache API's shape. You can store a Response directly, so the result of a fetch goes in without transformation. Put a service worker in front and it answers offline without changing the page's code.

IndexedDB suits cases like "show the ones taken last year" where you narrow by condition. It can hold Blobs, so images fit, but if you only ever look up by URL, its key expressiveness is overkill.

The 200 MB total far exceeds the Web Storage ceiling, but the Cache API and IndexedDB are judged against the origin-wide quota, so it fits if there is disk to spare. navigator.storage.estimate() lets you check beforehand.

Summary​

  • The spec defines the storage key as a tuple of an origin while noting itself that this is expected to change
  • All three implementations divide third-party storage per embedding site. The details are not aligned — WebKit does not even write it to disk
  • "Stored" and "still there" are not the same. The default is best-effort and subject to eviction
  • Safari has a time-based deletion. Data goes even when there is capacity to spare
  • IndexedDB is asynchronous and transactional. Unlike the synchronous Web Storage, it does not stall the main thread
  • The Cache API holds pairs of Request and Response and keeps them until you delete them explicitly. It is not service-worker-only
  • The spec assigning no per-endpoint capacity is not "no limit". The origin as a whole carries a quota
Related references