How to developing multiple libraries/applications simultaneously?

Question

This feels like a rather basic project setup issue. I just wanted to learn from the gurus and get some ideas for the best way to develop libraries in tandem

I write a lot of little programs and scripts in Clojure and those naturally have lots of little helper functions. As you write your main script/application you end up tweaking the helpers, covering new corner cases you expose etc. But these helper functions end up being copied around haphazardly between scripts and apps. Things like functions that convert formats or make plots etc. Obviously this isn't ideal!

I'd like to split these independent helper functions off into their own mini libraries

One workflow that I've seen is the "monorepo" setup. This is the design used by thing/geom. However I think long term this has proven to be a bit problematic b/c Karsten Schmidt ended up having a whole ton of awesome features.. but they're all wrapped up and hidden away in the mono repo. He has a whole matrix library there, and renderers and things for working with 3D geometry etc. etc. But I'm guessing there is a discoverability issue and maybe people don't wanna have to pull in the whole kitchen sink to access one tool in some corner

Instead I'm trying to find a nice deps.edn way of doing this but I don't really get what the language designers' expected workflow is.

There is a way to work with local repos: https://clojure.org/guides/deps_and_cli#_using_local_libraries
And then there is a way to work with remote repos: https://clojure.org/guides/deps_and_cli#_using_git_libraries (or you can work with Maven repos)

But how do you do both at once..? I obviously need to tweak and add features to the libraries as I work on the application :) Should I have a local development deps.edn that points at local folders and then another deps.edn with remote repos to distribute? (If so, what's the best way to set that up with Cider?)

Or can I have one deps.edn that looks locally and then falls back to remote repos if it doesn't find anything?

Or maybe there is a better alternative I haven't considered?

Just wondering how people are handling this

(Also posted here: https://clojureverse.org/t/how-to-developing-multiple-libraries-applications-simultaneously/8660)

Answer 1

Let's say that you have two projects: app and lib
app use a few functions from lib
you spend most of the time developing app, but sometimes you need to change something in lib.

at app/deps.edn you can add a profile :dev-local with a :override-deps {lib/lib {:local/root "../lib"}}

Then you can just develop/reload all functions from lib and app without restart your repl. Just by saving a file.

A few editors, like cursive, "understand" this kind of setup and do cool things, like goto definition for you.

Comments

Thank you for that. Yeah - I think this is pretty good workable solution. Just need to push updates online and make sure your local libs are at the versions as the remotes

Answer 2

After brainstorming a bit...

Option 1: Have two deps.edn files. One for yourself, one for others. This isn't ideal b/c you'll realistically end up only testing your local one :)

Option 2: Have a local alias that triggers an override of the dependencies with the :override-deps key. https://clojure.org/guides/deps_and_cli#override_deps The issue would be similar to Option 1 where you're not forced to sync the remote and local versions since they don't have to be at the same hash

Option 3: Just only use a remote and push every little change there and update your SHA hashes all the time. This is a bit fiddly as you need to be pushing and updating hashes all the time. You also expose your dirty laundry by pushing half baked broken things

Thinking about the whole situation more, I actually don't really know how things can be improved on. The only minor quality of life change would be if you could specify a dependency and a SHA and then multiple fallback sources. So you'd have a local directory first and then your remote (github) one second.. That way local and remote have to be in sync at the same SHA. But in a sense that' just option 3 without an extra push

The only foolproof way would be an external daemon that keeps hashes updated automatically - but ... that's kinda gross :)