How do you approach data (database) modeling in Clojure?

Question

Hello everyone!

Recently i decided to create a copy of a moderate-sized project (which is written in Java with Spring Boot and Hibernate) in Clojure. And i wonder how to model data, and models, and relations in Clojure in idiomatic way?

I don't think it necessarily should be objects or ORM, but i do think that there should be some kind of a schema of DB or data spec reflected in code (so a developer can know what to query from database and what to expect to come from that query).

So, what my options are?

EDIT: As for now, database is PostgeSQL

Thanks!

EDIT2: Thank you all for your answers, they are very helpful and i definitely able to solve my problem now!

Answer 1

What database will you be using?

If you can choose one, as Adrian also mentioned, seriously consider Datomic as it has a simple and highly flexible data model.

If you have to use SQL, the most straightforward way is next-jdbbc which will return query result as list of maps. If you don't have to do any complex transformations on the data, simplest is to simply use Clojure extensive set of functions to access and manipulate the data. Don't worry too much about creating 'domain objects'. Clojure Applied provides some practical advice on how to structure domain data.

If you need to do complex transformations, a common technique is to 'normalise' the data, basically create create a big map with 'domain model name' as first level key, and id for each record as second level key, and then the model/table rows as list of maps. This allows you to quickly get access to different bits of data. This technique is also often used on the client side, if you need to write a Single Page Application (look at re-frame and fulcro).

DataScript is a datomic-like database, but in-memory. It is lightweight enough to use on the server and client-side. You define a schema, add your data and can then perform datalog queries and pulls on it, making use of it's simple and flexible data model. So even if you use SQL database, you can query some data, load the result into DataScript and manipulate and query further in memory before writing out results.

You can also look at specter, a DSL for performing data transformations, although in my mind it creates complexity that something like DataScript avoids. Whatever works for you.

Comments

I'll just add that we often use clojure.spec to describe our database schemas in code so that we have 1) documentation 2) validation code 3) a way to generate random, conforming data for tests (so we don't have to mock a database) 4) a "system of record" from which to derive the list of columns etc -- we generate CRUD functions from Specs using macros.

---

Highly recommend normalization. However, I don't think it's a common practice among re-frame people. It's briefly mentioned in re-frame's wiki but no one seems to be really using it https://github.com/Day8/re-frame/blob/master/docs/FAQs/DB_Normalisation.md
In contrast, normalization is the core part of om.next/fulcro.

Answer 2

I'm on mobile at the moment so can't explain much
but check out https://docs.datomic.com/on-prem/schema.html

Answer 3

First of all, as other people have mentioned, learn you some Datomic or Datascript. That will change the way you think about database.

There's this SQL library that is heavily influenced by Datomic pull api http://walkable.gitlab.io
It can help you build APIs quickly. It provides an expressive way to describe the relationship between your tables so your mind won't explode for remembering implementation details when more and more tables are added to the system.

Btw, imo, the latest idiomatic way of writing Clojure code is data-driven.

Answer 4

i do not think there is a single right answer to this question.... but i suggest you do consider the following.... you decided to go with a relational database ( i think that is perfectly fine! ) .... so one could argue... well... first and foremost your problem is related to ER-modeling etc... and not really about clojure.. :-)....

... what do i mean.... well... you have already mentioned ORM... since when you do java, the "impedance / paradigm mismatch" between the relational and OO worlds, often times, this leads to a huge amount of clutter / headache.... etc.

alright... but you do pay a huge price ( i think ) in terms of added complexity, when going with an ORM... i.e. if you are not careful about what your doing, hibernate, for example, will come up with the wildest things / queries... for example when naively fetching some kind of tree structure... the generated sql could be quite silly... now obviously hibernate is really powerful.... you can integrate very sophisticated caching etc. etc.... still ... to do hibernate well / right.... is not that easy / straightforward ... i think....

so... even when doing java i would suggest to consider spring jdbc data access first... since.... as i said... JPA certainly adds a good deal of complexity.... (.. so ask yourself if you really need JPA... )

... iiin any case... what i am saying is this.... relational databases are great... sql is great... postgres is great!!!... the problem with many programming languages is just that it is often times really painful to interact with the relational technology... so you start building all kinds of tooling like ORMs etc. to somehow make your life more bearable.... ( ... so we are talking about a therapy, not about a cure.... )

... now i want to argue that with clojure the entire mismatch problem is much less serious, since clojure is not build around classes like Person etc. but instead you have a few powerful collections and you are good to go :-)....

... so lets make this more concrete.... i have build this flashcard SAP using clojure(script)....

... now... there are lots of things i am not too happy about..... for example i really wanna change from bootstrap over to bulma etc. etc...... it was the first clojure project of non trivial size i have ever done.... so obviously i have made tons of mistakes.... and i will try to refactor / improve things as soon as possible.... the main point however is that what i am least worried about is the postgres / rdbms part....

.... i used the luminus template for setting the postgres project up... that setup includes hugsql ( A Clojure library for embracing SQL. ).... then you can do your ddl / schema stuff... your postgres queries.... stored procs... etc. etc.... in a way that is completely independent from any application programming language.... just use your postgres tooling / know-how... how great is that!!!!..... also.... say i wanna switch from clojure to java / node whatever.... well my entire database / persistence tier stuff can pretty much stay the way it is... since with hugsql you are really just doing sql..... plus very little meta configuration stuff that you put in comments... from those comments you will get the clojure functions you can use the run the queries..... you can pass args into those prepared statements in form of a map... and you get your output as a map for one record or a sequence of maps for n records.... and nil for no records..... i guess....

.... for doing transformations etc. you have all your basic but powerful clojure functions that make massaging maps and seqs a breeze..... also there is always a little bit of type conversion stuff... when you go from postgres to java.... from java to js.... but if you use luminus ... you will get those helpers out of the box.... so for example in my flashcard app i have a lot of timestamps.... ( ... when did you get the answer to some card right.... when did you leave a comment etc. etc.... )... and date types have always been tricky..... java.sql, java.util, joda etc..... but when doing this app i really did not have to worry about any of that stuff... luminus makes it all happens under to hood for you.... also.... using the great time libs for clj(script)....( https://github.com/dm3/clojure.java-time / https://github.com/andrewmcveigh/cljs-time ).... you can write your time related code... and it would work on the middle-ware as well as the front end....( ...with minor exceptions... )....(... luminus does provide the needed transit-adapter? ( not sure if its actually called that...) stuff also!!! )....

... in any case.... i went with this kiss principle based strategy... and i am quite pleased with it.... make of that whatever you will.... :-)

Answer 5

I recently moved an app from a 13 year old java/hibernate/MySQL design to Clojure/Datomic

I wrote a library that reads the MySQL table schema, creates an abstract table spec, then uses that spec to create the Datomic schema and migrates all the data over to Datomic. You're welcome to check it out for inspiration (I attempted to write it somewhat generally, but it's fairly tightly focused on my use case)
https://github.com/thosmos/mysql-to-datomic

The main point however is that I created my own abstract data spec to describe the tables, columns, and foreign keys, such that I had everything I needed to not only generate the Datomic schema, but also later auto-generated GraphQL and EQL APIs and front-end CRUD data entry forms with linked subforms and data validations, all from this abstract spec. I've attempted to pull the essence of this model-defining spec into a minimal library here:
https://github.com/thosmos/domain-spec

It again tightly serves my use case, but perhaps it will inspire you to realize that in Clojure you can do your own thing quite easily and powerfully without over-relying on libraries and dependencies.

Answer 6

some time has passed now and since the question you asked keeps coming up, i thought i could perhaps ask you, what you have decided on going with in the end. also, how did things turn out for you? what were the most important lessons learned? anything you would wanna do differently if you were to do it again?