Tag metadata is not applied to method values

Question

Is it expected that this would work, with the new method values functionality?

(let [dir (jio/file "/Users/colin")]
  (.listFiles dir ^FileFilter File/.isDirectory))

This produces the error: More than one matching method found: listFiles. However, I’d have expected this to disambiguate using the FileFilter type tag.

The method value does seem to implement IObj:

(instance? IObj File/.isDirectory)
=> true

However, the metadata doesn't seem to be applied, at least for :tag:

(let [x ^FileFilter File/.isDirectory]
  (meta x))
=> nil

Comments

What version of Clojure did this happen with?

Answer 1

Another way to make this call is to use a qualified method with a param-tag to tell the compiler which overload you want:

(^[FileFilter] File/.listFiles dir File/.isDirectory)

It seems in this case, the File/.listFiles call is emitted directly (without being wrapped in an IFn thunk). The File/.isDirectory is wrapped in a thunk, but thanks to the ^[FileFilter] param-tag the compiler knows how to convert it.

Comments

Yes, this is what I ended up going with. This is for an automated refactoring from the reified interface to the fn version. This solution is not ambiguous, but it's a much more invasive change due to potentially having to add numerous imports as well as the actual change.

---

Currently (1.12.0-beta1), this will wrap the File/.isDirectory method in a thunk, then convert that thunk to the desired FileFilter functional interface.

We intend to support a more optimal conversion directly from File::isDirectory to FileFilter but we're still considering some of the fine details of how and when that happens (that will happen post 1.12).

Answer 2

This is one of those places where you need to use let with a type hint:

user=> (require '[clojure.java.io :as io])
nil
user=> (import '(java.io File FileFilter))
java.io.FileFilter
user=> (let [dir (io/file ".")
             ^FileFilter f File/.isDirectory]
         (.listFiles dir f))
#object["[Ljava.io.File;" 0x37c5fc56 "[Ljava.io.File;@37c5fc56"]

Although, given this:

user=> (instance? clojure.lang.IMeta File/.isDirectory)
true

It is a bit surprising that neither your original, nor this version seems to work:

user=> (let [dir (io/file ".")
             f ^FileFilter File/.isDirectory]
         (.listFiles dir f))
Syntax error (IllegalArgumentException) compiling . at (REPL:3:10).
More than one matching method found: listFiles
user=>

Comments

I see, so it only works if you type hint the local binding, not the method value itself. I'd be interested to know if this is considered a bug which will be fixed, I've added functionality to Cursive to translate reified forms to fns, and I'll have to take that case into account if this isn't going to be fixed.

---

Re "type hint the local binding, not the method value itself", this is intentional and not a bug. In the two places where FI conversion can happen there is effectively "assignment" with both a source expression ("right hand side") and the assigned target ("left hand side").

In a Java method call, the source is the argument type (the "type" of that expression can come from lots of places - a type hint on the argument, type flow of symbol or expression, or directly from the literal expression). The compiler does not actually "know" where the type of the source expression comes from, but importantly IT CAN BE WRONG. The target in this case is the parameter of the Java method you are invoking - that is either known, or it's reflective.

In a let binding, the source is the binding init expression, which similarly has a type that comes from ... somewhere, and also COULD BE WRONG. The target type can only come from the type hint on the binding symbol.

In general, Clojure developers type hint either of these mostly without discrimination and they usually have the same effect, because the binding (LHS) will take on the reported type (RHS) of the expression. But these are very different in how they are treated in the compiler and for implicit conversion, it matters a lot! So in the `let`, if you are requesting an explicit conversion to an FI type, it's important that you type hint the target specifically.

In a Java method invocation, the type hint there is tricky because it really has dual roles - in one sense you are stating the source expression type of the argument, and in another you may be using it in the Java invocation to select an overload choice and avoid reflection. In that case, it will importantly select the overload and thus act as the target type. If you want to separate those two roles, you can do that with param-tags, which only talk about target and overload selection independently from the source argument expression. That will lead you back to the answer from glchapman.

Answer 3

"^FileFilter File/.isDirectory" is attaching metadata to a symbol at read time

"(meta ^FileFilter File/.isDirectory)" is looking at metadata on the result of evaluating the symbol File/.isDirectory, not the symbol itself

"(instance? IObj File/.isDirectory)" is showing that the result of evaluating the symbol File/.isDirectory is something that implements IObj, not that File/.isDirectory itself implements IObj (but it does because it is a symbol)

Comments

That led to me try this -- which works:

user=> (require '[clojure.java.io :as io])
nil
user=> (import '(java.io File FileFilter))
java.io.FileFilter
user=> (let [dir (io/file ".")]
         (.listFiles dir (with-meta File/.isDirectory {:tag FileFilter})))
#object["[Ljava.io.File;" 0x512d4583 "[Ljava.io.File;@512d4583"]
user=>

---

So the problem in this case is that the creation of the adapter in the compiler isn't taking the metadata from the symbol and applying it to the adapter? It seems like it should, since this behaviour was very surprising to me, at least.

---

no, because the type hint metadata is only used by the compiler, so tagging the value at runtime cannot go back in time to direct the compiler about how to compile things. I am just pointing out that the instance? check of what File/.isDirectory evaluates to, and calling meta on what File/.isDirectory evaluates to doesn't actually convey anything

---

what Sean's example is actually showing is when you forcing to be a reflective call, by obscuring the type of the object that File/.isDirectory evaluates to by passing it through a function call, it works. in his example you can replace the call to with-meta with a call to identity instead and it will also work.

so this seems like a bug where the method selection process is different when reflecting to pick a method at a compile time vs. reflecting to pick a method at runtime

---

you can also do something like '(.listFiles dir ^java.io.FileFilter (identity File/.isDirectory))
' to work around the fact that the type hint on the generated method function is ignored

---

If I do:

(def ^String x "foo")

Then my understanding is that it is also true that the metadata is applied to the symbol x at read time. However, when the compiler then compiles the def form, it knows to transfer the metadata from the symbol to the var itself.

I haven't examined the compiler code in detail for the new cases, but my mental model for this case:

(.listFiles dir ^FileFilter File/.isDirectory)

is that the meta is applied to the symbol at read time, and then the compiler figures out that it will have to generate code to create a method adapter which the symbol will evaluate to. I would expect it to use the metadata to determine which adapter to create (either a FileFilter or a FilenameFilter), and then apply the metadata to that object, in the same way as the compiler does when compiling code to create a var. Then it can use the metadata on that adapter object to determine which variant of .listFiles should be compiled.

One thing I'm not sure of is whether the adapters are created per call site, or if they are shared. If they're shared, then I guess the adapters can never have call site specific metadata applied to them.

---

you are missing a step.

File/.isDirectory is seen as a method literal, so it is transformed into an IFn that invokes that method.

then (.listFiles dir x) is seen as a functional interface adaption so the an adaptor from IFn to the functional interface is generated.

the bug here is really when compiling (.listFiles dir File/.isDirectory) you should get a reflection warning and not an error (you'll get an error down the road because reflection at runtime will pick from one of the possible listFiles and then adapt the IFn from File/.isDirectory for that)

the fact that File/.isDirectory results in a pretty generic function wrapping the method call is also a problem for adding metadata, because using with-meta on fn objects is horrendous (adds a wrapper around it that calls the original using apply which is slow, and changes the identity of the function which is against the contract of with-meta)


it is the case that if you use a function literal to wrap the method instead of relying on clojure to do it

   (let [dir (io/file ".")]
    (.listFiles dir ^java.io.FileFilter #(.isDirectory %)))

that works, so the compiler should do something similar (it isn't about copying the metadata from the symbol to the runtime object, like for vars, that trick is there because the the copied metadata becomes part of the compile time environment for subsequent forms, which is not possible here). it may just be a matter of having the compiler copy the tag metadata out to make it available in the Expr tree the compiler builds.

---

Thanks, this has been really useful for clarifying my thinking around this. I think for my use case (automatic refactoring of the reified version to the method value version), the best thing to do is to update the enclosing interop call (.listfiles in this case) to use the new syntax with param-tags to disambiguate.

---

Created a ticket at https://clojure.atlassian.net/browse/CLJ-2867