PathRef fails for socket files (was: GenIdea/idea and Bloop/install both fail for VirtusLab/scala-cli)

[mtk]

i'm not sure if this is a bug in mill or in scala-cli's build.sc. but even if it is a scala-cli build.sc bug, the mill error messages don't provide any useful info.

a few weeks ago, both of these targets began to fail reliably. i verified this from a brand new empty linux ID with no ~/.config nor ~/.local directory. i did a "git clean -dfx" and removed any lingering java processes with the help of 'jps' before running either failing command.

i git bisect'ed the problem and found that the first bad commit was VirtusLab/scala-cli@e114105 and the last good commit was VirtusLab/scala-cli@ced5e0d.

i don't know enough mill to know whether this change is problematic or not but it doesn't seem like it would be.

i'm attaching a log from the invocation of each command.

fail.log

[lefou]

Looks like the integration.modulesPath target is throwing an exception. Can you run it from cli and verify it's succeeding? Could it be, that it tries, for some reasons, to return a PathRef of the project directory, and while calculating the content hash, it fails because it can't open the unix socket file?

I wonder whether PathRef should be more forgiving when it encounters a socket file?

[lefou]

Yeah, I can reproduce with:

import mill._

def socketfile = T{
  PathRef(T.workspace)
}

Output:

± mill socketfile
No mill version specified.
You should provide a version via '.mill-version' file or --mill-version option.
Retrieving latest mill version ...
Using mill version 0.10.4
Compiling /home/lefou/work/opensource/mill/reproduce-socketfile/build.sc
[1/1] socketfile 
1 targets failed
socketfile java.nio.file.FileSystemException: /home/lefou/work/opensource/mill/reproduce-socketfile/out/mill-worker-LthlJvSD+shQfuObr5wuSIlVd0o=-1/mill-ead6604ff439ef261d60771d013bedff-io: Kein passendes Gerät bzw. keine passende Adresse gefunden
    java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
    java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
    java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
    java.base/java.nio.file.Files.newByteChannel(Files.java:371)
    java.base/java.nio.file.Files.newByteChannel(Files.java:422)
    java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
    java.base/java.nio.file.Files.newInputStream(Files.java:156)
    os.Path.getInputStream(Path.scala:491)
    os.read$inputStream$.apply(ReadWriteOps.scala:229)
    mill.api.PathRef$.$anonfun$apply$2(PathRef.scala:48)
    mill.api.PathRef$.$anonfun$apply$2$adapted(PathRef.scala:38)
    scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
    scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
    scala.collection.AbstractIterable.foreach(Iterable.scala:926)
    scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:896)
    mill.api.PathRef$.apply(PathRef.scala:38)
    ammonite.$file.build.$anonfun$socketfile$2(build.sc:4)
    mill.define.Task$TraverseCtx.evaluate(Task.scala:380)

[lefou]

Unfortunately, I can't tell how to detect socket files. We already check for regular, existing, readable files and still get the exception. Unless someone comes up with a proper implementation to detect socket files, the only thing we can do is to gracefully ignore the content of unreadable files or avoid using the project directory in PathRefs.

[mtk]

not sure what you mean by can't check for socket files. that info isn't returned by a linux 'stat' system call?

[lefou]

not sure what you mean by can't check for socket files. that info isn't returned by a linux 'stat' system call?

Sure, stat has this information, but I think calling a system specific tool for each file we find is a bit expensive and platform specific. What I meant is, I don't know how to get this information with Java API.

[mtk]

does this help? https://www.tabnine.com/code/java/methods/java.nio.file.Files/getAttribute

…

On Mon, May 16, 2022 at 9:55 AM Tobias Roeser ***@***.***> wrote: not sure what you mean by can't check for socket files. that info isn't returned by a linux 'stat' system call? Sure, stat has this information, but I think calling a system specific tool for each file we find is a bit expensive and platform specific. What I meant is, I don't know how to get this information with Java API. — Reply to this email directly, view it on GitHub <#1875 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAYG2KDTR5YGJLU2S7C26LVKJHWDANCNFSM5V7TVAPQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[lefou]

does this help? https://www.tabnine.com/code/java/methods/java.nio.file.Files/getAttribute

I don't know yet. Do you mean a specific snippet? Beside some custom socket detection for docker, I couldn't spot any example for cross-platform socket detection.

[mtk]

i have no idea if windows has the equivalent functionality. wouldn't you rather work correctly on two of three?

…

Message ID: ***@***.***>

[mtk]

sorry, "two out of three"?

…

On Mon, May 16, 2022 at 6:56 PM Mark Kennedy ***@***.***> wrote: i have no idea if windows has the equivalent functionality. wouldn't you rather work correctly on two of three? > Message ID: ***@***.***> >

[lefou]

Sure, properly supporting some platforms is a nice intermediate step, yet the link you provided does not show how to do it. We want to detect arbitrary socket files, and the documentation of java.nio.file.Files.getAttribute and various implementations and attribute providers does not reveal how to detect a socket. I think we may need to fallback to just handle the exception and gracefully continue to digest the hash.

[lefou]

@mtk Can you test PR #1878? Does it fix your issues?

[mtk]

i look into this some more and you're absolutely right. no way to check it! interesting omission. thanks for the free lesson!