Welcome! Please see the About page for a little more info on how this works.

0 votes
in Tools by

DBFS (databricks file system) is not properly implementing POSIX permissions.
Specially the bash flag "-w" fails to detect a non-writable project directory properly.

As a consequence in my case , starting clojure on DBFS tries to write the .cpcache to current dir, which is failing as it is in fact read-only, and this cannot be overwritten.

see here for details and error messages:
https://clojurians.slack.com/archives/C03S1KBA2/p1726750856210489

Changing unix file permission fails as well on DBFS

by
I think the original proposal of introducing CLJ_PROJECT_CACHE would fix my current case.

1 Answer

0 votes
by

Sounds like a DataBricks issue?

by
yes, or we accept that "-w" is not a reliable way to detect "read-only" directories and we find a way that the Clojure script does not rely on it  without having a way to "enforce" an other directory.

Not sure, if there are other file system with similar issues.

DBFS does not supporrt any "file permissions", so it will never be solved in DBFS
by
The current logic would as well not allow to "change" the location of .cpcache, when the file system would be" full", for example. (full, but writable)
Or when for other reasons then "read-only" I don't like its default location.
by
Classpaths are often long and must be conveyed to the Java process. You can either pass on the command-line (but many systems have process line length limits), or you can write the classpath in a file and convey that way. The Clojure CLI does the latter, but this requires writing a file somewhere. There are multiple places the file can be written and logic that to detect the correct place to do so. If the system reports that it can be written but can't, or cannot be written, I don't know what the CLI can be expected to do about this (it seems problematic for downloading Maven libs, downloading git libs, prepping git libs, and many other features of the CLI/tools.deps).

Trying to write files to test whether they can be written does not work because we use the presence of the file to detect whether the cached file exists. Adding more environment variables is not desired - there are too many configuration options already and in many cases this logic exists in multiple locations (bash and tools.deps clojure) that must be kept in sync and all code that reads/writes the cache has to agree. All of these options are bad to varying degrees.
by
The one thing the CLI could do is to offer a way to set the place for project specific .cpcache which is **always** used if given, and not only as fallback when local dir is not writable. This would remove it's reliance on a working "-w" , which on some filesystems (at least on DBFS) is not working.
...