https://gradle.com/ logo
Join Slack
Powered by
Hey I have a simple question (famous last words......
# community-support
j
Hey I have a simple question (famous last words...). I think it is also one of these riddles that could be in a Gradle quiz. And I am hoping this is documented somewhere and I just didn't find it. How do I exclude empty directories from a 
FileTree
? Input is:
Copy code
src
└── a
    ├── b
    │   └── c
    │       └── x.tmp
    └── z.txt
Some task using 
matching
:
Copy code
val src = layout.projectDirectory.dir("src").asFileTree.matching {
    include("**/*.txt")
}
tasks.register<Sync>("sync") {
    from(src)
    into(layout.buildDirectory.dir("target"))
}
Then I get (I find it surprising that 
**/*.txt
matches the directories 🤷‍♀️ )
Copy code
build/target
└── a
    ├── b
    │   └── c
    └── z.txt
How do I get?
Copy code
build/target
└── a
    └── z.txt
(Wrong solutions in Thread)
👀 1
Copy code
.matching { exclude { it.isDirectory } }
...is wrong as it excludes everything. Also the files that are inside directories (in the example 
a/z.txt
)
Copy code
.filter { !it.isDirectory }
...is wrong too. Yes, it excludes the directory entries, but 
filter { }
in general throws away the relative path of the file tree elements (also TIL for me). So 
a/z.txt
becomes 
z.txt
.
Copy code
tasks.register<Sync>("sync") {
    includeEmptyDirs = false
    ...
}
Yes this can be done in the context of a 
Sync
/ 
Copy
task, but I want to use the 
FileTree
in another context. This still shows the issue though: it does not exclude the empty directories from the input tracking. If I rename one of the unimportant directories, the task will rerun although the output is the same.
v
iirc you cannot 😞 But I hope I remember wrongly.
😵 2
t
m
What is the snapshot value of a directory? Its listing?
j
Thank you @Thomas Broyer! That maybe sufficient to solve our use case. I didn't know about that or I forgot. (So hard to keep track of all these annotations and how the interact.)
What is the snapshot value of a directory?
I think it is only the (relative) path and the fact that it is there. As each file inside would be tracked separately. But I also do not know the details and that (empty) directories are tracked at all was also a surprise to me in this case.
m
TIL as well. What would a task do based on empty directories, that's be interesting...
What's even more interesting is that 
src.forEach {}
doesn't show the directories...
(but 
src.visit {}
does, which is probably what the Sync task is using)
j
I am wondering if I should create an issue about 
**/*.txt
matching directories that do not include 
*.txt
files. I think this is not an intuitive behavior. But I would think that this would be mentioned somewhere already... I mean it must be like this since 
gradle-0.1
. (Someone knows about an issue or forum post on this?)
m
I'd be curious to have the official explanation, this all sounds very suprising to me
j
I was in the same situation. Only that I was in a room with others (who rely on me being the "expert") who showed my this and I was like: How can that be? What did we do wrong in our task configurations? 😄 And it took quite some time for me to realize and accept that this is behavior that seems to exist since ever, but I did not knew about. 😭 And of course as always it has several layers: • The unexpected matching behavior of 
**
includes (which I never used much) • The realization that empty directories are tracked as part of a FileTree
🙃 1
FTR: 
@IgnoreEmptyDirectories
works in so far that changing something unrelated would no longer lead to the task being out-of-date. What would still happen is that the empty directories are created in the state they are in in the first run. But that's expected (they are ignored, not removed). But then you can combine it with 
includeEmptyDirs = false
to get it all.
m
galaxy brains
j
Copy code
val src = layout.projectDirectory.dir("src").asFileTree.matching {
    include("**/*.txt")
}
tasks.register<CustomSync>("sync") {
    inFiles.from(src)
    outDir = layout.buildDirectory.dir("target")
}

abstract class CustomSync : DefaultTask() {
    @get:Inject
    abstract val files: FileSystemOperations

    @get:InputFiles
    @get:IgnoreEmptyDirectories  // !!!
    abstract val inFiles: ConfigurableFileCollection

    @get:OutputDirectory
    abstract val outDir: DirectoryProperty

    @TaskAction
    fun syncOp() {
        files.sync {
            includeEmptyDirs = false // !!!
            from(inFiles)
            into(outDir)
        }
    }
}
v
Yay, I was wrong, it's just not trivial. 😄
Copy code
val src = layout.projectDirectory.dir("src").asFileTree.matching {
    include("**/*.txt")
    exclude { it.isDirectory && it.file.walk().none { it.isFile && (it.extension == "txt") } }
}
Or
Copy code
val src = layout.projectDirectory.dir("src").asFileTree.matching {
    include {
        (!it.isDirectory && (it.file.extension == "txt"))
                || (it.isDirectory && it.file.walk().any { it.isFile && (it.extension == "txt") })
    }
}
a
v
I'm no so sure that is even a valid issue @Adam. A 
FileTree
is not a tree of 
File
objects, but a tree of files. Whether the empty directories when using 
visit
/ 
Copy
/ 
Sync
should be there is questionable and might be a bug. But that if you filter out all files nothing is in the file tree is expected. Even with the 
src
from OP, if you use 
.forEach
or 
.files
, you only get the actual files. Only if you use 
visit
, you get the empty directories.
m
Petition to rename 
FileTree
to 
RegularFileTree
😄
🧠 1
v
Well, it long predates 
RegularFile
, that's probably why 😄
m
Yeaaaa
Anyone wants to open a JavaDoc PR there? I don't think this is super explicit right now
Copy code
A FileTree represents a hierarchy of files
That's not super obvious to me whether the files are "files and directories" or just "regular files"
➕ 1
v
Well, it is "files", not "{@code File}s" 😄
But yeah, docs can always be better
a
are you using a FileTree as a task input @Jendrik Johannes? If so, could you use convert the file tree to a FileCollection? It will only contain files (not dirs).
Copy code
val srcDir = layout.buildDirectory.asFile.get().resolve("tmp-src")
srcDir.resolve("a/b/c/tmp.x").apply {
  parentFile.mkdirs()
  writeText("tmp")
}
srcDir.resolve("a/file.txt").apply {
  parentFile.mkdirs()
  writeText("tmp")
}

val src = fileTree(srcDir).matching {
  include("**/*.txt")
}
tasks.register<Sync>("syncDemo") {
  from(src.files)
  into(layout.buildDirectory.dir("target"))
  doLast {
    println(destinationDir.walk().joinToString("\n"))
  }
}
m
A 
FileTree
is a 
FileCollecton
😅
a
true, it's confusing. FileTree#getFiles converts the contents though https://docs.gradle.org/current/kotlin-dsl/gradle/org.gradle.api.file/-file-tree/get-files.html
v
And 
src.files
does not make it a 
FileCollection
, but a 
Set<File>
and looses the relative paths, thus you would flatten all files in 
target/
j
Yes. You have to be very careful not to loose relative path information. That is also the issue I mentioned above with 
.filter { !it.isDirectory }
- it gives you a "FileCollection" that is not a "FileTree" and then also no longer contains the path information. And flattens everything.
➕ 1
m
Actually, 
FileCollection
is a lot more explicit:
Copy code
A {@code FileCollection} represents a collection of file system locations which you can query in certain ways
I understand this can be either files or directories
j
Thanks for crafting these solutions @Vampire - it is fascinating what you can do by using all kinds of APIs 😄 Unfortunately, I need a generic solution where the user can define arbitrary 
**
patterns and the part that excluded the empty dirs is inside my plugins.
v
.filter { !it.isDirectory }
=> 
.matching { include { it.isDirectory } }
But as then all directories are excluded, also the files are gone 😄
I understand this can be either files or directories
Exact
For example a source root directory
m
> a source root directory Yes but also I think a root directory shouldn't be used as Input to a task so it's debatable what the value of having directories in 
FileCollection
is.
But there are probably a lot of historical reasons
v
You Blog is still categorized as Gaming site 😞
😂 1
But yes, of course a directory needs to be a task input. How else would you give the directory with relative files to some task?
a
oh, is the relative path information based on the input type? I thought Gradle would recompute it on the fly, based on the provided files?
m
ou Blog is still categorized as Gaming site
I mean, Gradle can be fun
a directory needs to be a task input.
How else would you give the directory with relative files to some task?
I think we've had this discussion already. 
FileCollection
contains the normalized paths
You just give "files" to your task, not directories
"files + normalized paths" actually
v
Ah, that discussion, well, I think we agreed to not fully agree 😄
m
Ah yes, probably, I'm too lazy to look it up
v
oh, is the relative path information based on the input type?
Nah, if it is a 
FileTree
it has relative paths, if it is a 
FileCollection
it is just the plain files without relative path information. What is considered for Gradle input fingerprinting depends on more like 
PathSensitivity
and so on @Adam
m
if it is a 
FileTree
it has relative paths, if it is a 
FileCollection
it is just the plain files without relative path information.
But a 
FileTree
is a 
FileCollection
😅
Just some 
FileCollection
do not contain the relative path information I guess
v
... yes You can use a file tree as file collection and thereby discard the relative path information of course
a
Would this work? Manually create a FileCollection and mark it with Relative sensitivity, and add the regular files from the FileTree, and the base FileTree dir (so Gradle can relativise the regular files).
Copy code
// user configurable
@Internal
abstract val fileTree: FileTree

@InputFiles
@PathSensitivity(RELATIVE)
protected val inputFiles: FileCollection 
    get() = objects.fileCollection()
        .from(fileTree.dir)
        .from(fileTree.files)
v
iirc there the relative paths would just be the file names
For files in the root of the file collection, the file name is used as the normalized path. For directories in the root of the file collection, an empty string is used as normalized path. For files in directories in the root of the file collection, the normalized path is the relative path of the file to the root directory containing it.
So with your construct you would have the whole directory including all files in them with relative paths and additionally the filterd files from the filetree without relative path.
31 Views
Open in Slack
PreviousNext
Powered by