How can I reduce the size of transit encoded data in ClojureScript

Question

I'm working on a react native app in cljs, and have been storing our product catalog as transit json in the db. It has been working well so far but it is now pushing against the 6MB limit on android as our Christmas products come online.

I was thinking key caching might help - but I am worried it would be impossible to use it without sacrificing responsiveness.

At the moment, in order to keep the ui responsive, I store the data as a single transit map per product (js being single threaded, I need to keep deserialization / serialization chunks small so I don't get caught deserializing a massive datastructure when the user wants to press a button).

I think this means that I can't use transit key caching, because the repeated keys wouldn't occur until the next product.

Is there anyway I can share a key cache somehow - or a better way of reducing the size of the transit json?

Answer 1

One way would be to define custom product semantic type, which consists from common keys + extra keys.
https://github.com/cognitect/transit-format#extensibility

You will save some size, but will have to pay some maintenance effort and extra CPU time (which might be ok, since you are going to deserialize few at once).

This is an example of using Product only to wrap product maps, to not force you to use records in the rest of the codebase

(ns foo
  (:require 
   [cognitect.transit :as tr])
  (:import
   [com.cognitect.transit WriteHandler]
   [java.io ByteArrayOutputStream]))

(defrecord Product [m])
(def product-tag "pro")
(def common-keys [:product/foo :product/bar :product/baz])
(def custom-writers
  {Product (reify WriteHandler
             (getVerboseHandler [_] nil)
             (stringRep [_ kw] nil)
             (tag [_ _] product-tag)
             (rep [_ p] (let [common (mapv (:m p) common-keys)
                              custom (reduce dissoc (:m p) common-keys)]
                          (into [custom] common))))})

(def custom-readers
  {product-tag (fn [rep]
                 (let [[m & common-vals] rep]
                   (merge m (zipmap common-keys common-vals))))})

(defn write [writers product]
  (let [out (ByteArrayOutputStream. 4096)
        wr  (tr/writer out :json writers)]
    (tr/write wr product)
    (.toString out)))

(write {:handlers custom-writers}
  (Product. {:product/foo 1 :product/bar 2 :product/baz 3 :custom/foo 4}))
;; "[\"~#pro\",[[\"^ \",\"~:custom/foo\",4],1,2,3]]"

(write {}
  (Product. {:product/foo 1 :product/bar 2 :product/baz 3 :custom/foo 4}))
;; "[\"^ \",\"~:m\",[\"^ \",\"~:product/foo\",1,\"~:product/bar\",2,\"~:product/baz\",3,\"~:custom/foo\",4]]"

(write {}
  {:product/foo 1 :product/bar 2 :product/baz 3 :custom/foo 4})
;; "[\"^ \",\"~:product/foo\",1,\"~:product/bar\",2,\"~:product/baz\",3,\"~:custom/foo\",4]"