# godirwalk
`godirwalk` is a library for traversing a directory tree on a file
system.
[](https://godoc.org/github.com/karrick/godirwalk) [](https://dev.azure.com/microsoft0235/microsoft/_build/latest?definitionId=1&branchName=master)
In short, why did I create this library?
1. It's faster than `filepath.Walk`.
1. It's more correct on Windows than `filepath.Walk`.
1. It's more easy to use than `filepath.Walk`.
1. It's more flexible than `filepath.Walk`.
Depending on your specific circumstances, [you might no longer need a
library for file walking in
Go](https://engineering.kablamo.com.au/posts/2021/quick-comparison-between-go-file-walk-implementations).
## Usage Example
Additional examples are provided in the `examples/` subdirectory.
This library will normalize the provided top level directory name
based on the os-specific path separator by calling `filepath.Clean` on
its first argument. However it always provides the pathname created by
using the correct os-specific path separator when invoking the
provided callback function.
```Go
dirname := "some/directory/root"
err := godirwalk.Walk(dirname, &godirwalk.Options{
Callback: func(osPathname string, de *godirwalk.Dirent) error {
// Following string operation is not most performant way
// of doing this, but common enough to warrant a simple
// example here:
if strings.Contains(osPathname, ".git") {
return godirwalk.SkipThis
}
fmt.Printf("%s %s\n", de.ModeType(), osPathname)
return nil
},
Unsorted: true, // (optional) set true for faster yet non-deterministic enumeration (see godoc)
})
```
This library not only provides functions for traversing a file system
directory tree, but also for obtaining a list of immediate descendants
of a particular directory, typically much more quickly than using
`os.ReadDir` or `os.ReadDirnames`.
## Description
Here's why I use `godirwalk` in preference to `filepath.Walk`,
`os.ReadDir`, and `os.ReadDirnames`.
### It's faster than `filepath.Walk`
When compared against `filepath.Walk` in benchmarks, it has been
observed to run between five and ten times the speed on darwin, at
speeds comparable to the that of the unix `find` utility; and about
twice the speed on linux; and about four times the speed on Windows.
How does it obtain this performance boost? It does less work to give
you nearly the same output. This library calls the same `syscall`
functions to do the work, but it makes fewer calls, does not throw
away information that it might need, and creates less memory churn
along the way by reusing the same scratch buffer for reading from a
directory rather than reallocating a new buffer every time it reads
file system entry data from the operating system.
While traversing a file system directory tree, `filepath.Walk` obtains
the list of immediate descendants of a directory, and throws away the
node type information for the file system entry that is provided by
the operating system that comes with the node's name. Then,
immediately prior to invoking the callback function, `filepath.Walk`
invokes `os.Stat` for each node, and passes the returned `os.FileInfo`
information to the callback.
While the `os.FileInfo` information provided by `os.Stat` is extremely
helpful--and even includes the `os.FileMode` data--providing it
requires an additional system call for each node.
Because most callbacks only care about what the node type is, this
library does not throw the type information away, but rather provides
that information to the callback function in the form of a
`os.FileMode` value. Note that the provided `os.FileMode` value that
this library provides only has the node type information, and does not
have the permission bits, sticky bits, or other information from the
file's mode. If the callback does care about a particular node's
entire `os.FileInfo` data structure, the callback can easiy invoke
`os.Stat` when needed, and only when needed.
#### Benchmarks
##### macOS
```Bash
$ go test -bench=. -benchmem
goos: darwin
goarch: amd64
pkg: github.com/karrick/godirwalk
BenchmarkReadDirnamesStandardLibrary-12 50000 26250 ns/op 10360 B/op 16 allocs/op
BenchmarkReadDirnamesThisLibrary-12 50000 24372 ns/op 5064 B/op 20 allocs/op
BenchmarkFilepathWalk-12 1 1099524875 ns/op 228415912 B/op 416952 allocs/op
BenchmarkGodirwalk-12 2 526754589 ns/op 103110464 B/op 451442 allocs/op
BenchmarkGodirwalkUnsorted-12 3 509219296 ns/op 100751400 B/op 378800 allocs/op
BenchmarkFlameGraphFilepathWalk-12 1 7478618820 ns/op 2284138176 B/op 4169453 allocs/op
BenchmarkFlameGraphGodirwalk-12 1 4977264058 ns/op 1031105328 B/op 4514423 allocs/op
PASS
ok github.com/karrick/godirwalk 21.219s
```
##### Linux
```Bash
$ go test -bench=. -benchmem
goos: linux
goarch: amd64
pkg: github.com/karrick/godirwalk
BenchmarkReadDirnamesStandardLibrary-12 100000 15458 ns/op 10360 B/op 16 allocs/op
BenchmarkReadDirnamesThisLibrary-12 100000 14646 ns/op 5064 B/op 20 allocs/op
BenchmarkFilepathWalk-12 2 631034745 ns/op 228210216 B/op 416939 allocs/op
BenchmarkGodirwalk-12 3 358714883 ns/op 102988664 B/op 451437 allocs/op
BenchmarkGodirwalkUnsorted-12 3 355363915 ns/op 100629234 B/op 378796 allocs/op
BenchmarkFlameGraphFilepathWalk-12 1 6086913991 ns/op 2282104720 B/op 4169417 allocs/op
BenchmarkFlameGraphGodirwalk-12 1 3456398824 ns/op 1029886400 B/op 4514373 allocs/op
PASS
ok github.com/karrick/godirwalk 19.179s
```
### It's more correct on Windows than `filepath.Walk`
I did not previously care about this either, but humor me. We all love
how we can write once and run everywhere. It is essential for the
language's adoption, growth, and success, that the software we create
can run unmodified on all architectures and operating systems
supported by Go.
When the traversed file system has a logical loop caused by symbolic
links to directories, on unix `filepath.Walk` ignores symbolic links
and traverses the entire directory tree without error. On Windows
however, `filepath.Walk` will continue following directory symbolic
links, even though it is not supposed to, eventually causing
`filepath.Walk` to terminate early and return an error when the
pathname gets too long from concatenating endless loops of symbolic
links onto the pathname. This error comes from Windows, passes through
`filepath.Walk`, and to the upstream client running `filepath.Walk`.
The takeaway is that behavior is different based on which platform
`filepath.Walk` is running. While this is clearly not intentional,
until it is fixed in the standard library, it presents a compatibility
problem.
This library fixes the above problem such that it will never follow
logical file sytem loops on either unix or Windows. Furthermore, it
will only follow symbolic links when `FollowSymbolicLinks` is set to
true. Behavior on Windows and other operating systems is identical.
### It's more easy to use than `filepath.Walk`
While this library strives to mimic the behavior of the incredibly
well-written `filepath.Walk` standard library, there are places where
it deviates a bit in order to provide a more easy or intuitive caller
interface.
#### Callback interface does not send you an error to check
Since this library does not invoke `os.Stat` on every file system node
it encounters, there is no possible error event for the callback
function to filter on. The third argument in the `filepath.WalkFunc`
function signature to pass the error from `os.Stat` to the callback
function is no longer necessary, and thus eliminated from signature of
the callback function from this library.
Furthermore, this slight interface difference between
`filepath.WalkFunc` and this library's `WalkFunc` eliminates the
boilerplate code that callback handlers must write when they use
`filepath.Walk`. Rather than every callback function needing to check
the error value passed into it and branch accordingly, users of this
library do not even have an error value to check immediately upon
entry into the callback function. This is an improvement both in
runtime performance and code clarity.
#### Callback function is invoked with OS specific file system path separator
On every OS platform `filepath.Walk` invokes the callback function
with a solidus (`/`) delimited pathname. By contrast this library
invokes the callback with the os-specific pathname separator,
obviating a call to `filepath.Clean` in the callback function for each
node prior to actually using the provided pathname.
In other words, even on Windows, `filepath.Walk` will invoke the
callback with `some/path/to/foo.txt`, requiring well written clients
to perform pathname normalization for every file prior to working with
the specified file. This is a hidden boilerplate requirement to create
truly os agnostic callback functions. In truth, many clients developed
on unix and not tested on Windows neglect this subtlety, and will
result in software bugs when someone tries to run that software on
Windows.
This library invokes the callback function with `some\path\to\foo.txt`
for the same file when running on Windows, eliminating the need to
normalize the pathname by the client, and lessen the likelyhood that a
client will work on unix but not on Windows.
This enhancement eliminates necessity for some more boilerplate code
in callback functions while improving the runtime performance of this
library.
#### `godirwalk.SkipThis` is more intuitive to use than `filepath.SkipDir`
One arguably confusing aspect of the `filepath.WalkFunc` interface
that this library must emulate is how a caller tells the `Walk`
function to skip file system entries. With both `filepath.Walk` and
this library's `Walk`, when a callback function wants to skip a
directory and not descend into its children, it returns
`filepath.SkipDir`. If the callback function returns
`filepath.SkipDir` for a non-directory, `filepath.Walk` and this
library will stop processing any more entries in the current
directory. This is not necessarily what most developers want or
expect. If you want to simply skip a particular non-directory entry
but continue processing entries in the directory, the callback
function must return nil.
The implications of this interface design is when you want to walk a
file system hierarchy and skip an entry, you have to return a
different value based on what type of file system entry that node
is. To skip an entry, if the entry is a directory, you must return
`filepath.SkipDir`, and if entry is not a directory, you must return
`nil`. This is an unfortunate hurdle I have observed many developers
struggling with, simply because it is not an intuitive interface.
Here is an example callback function that adheres to
`filepath.WalkFunc` interface to have it skip any file system entry
whose full pathname includes a particular substring, `optSkip`. Note
that this library still supports identical behavior of `filepath.Walk`
when the callback function returns `filepath.SkipDir`.
```Go
func callback1(osPathname string, de *godirwalk.Dirent) error {
if optSkip != "" && strings.Contains(osPathname, optSkip) {
if b, err := de.IsDirOrSymlinkToDir(); b == true && err == nil {
return filepath.SkipDir
}
return nil
}
// Process file like normal...
return nil
}
```
This library attempts to eliminate some of that logic boilerplate
required in callback functions by providing a new token error value,
`SkipThis`, which a callback function may return to skip the current
file system entry regardless of what type of entry it is. If the
current entry is a directory, its children will not be enumerated,
exactly as if the callback had returned `filepath.SkipDir`. If the
current entry is a non-directory, the next file system entry in the
current directory will be enumerated, exactly as if the callback
returned `nil`. The following example callback function has identical
behavior as the previous, but has less boilerplate, and admittedly
logic that I find more simple to follow.
```Go
func callback2(osPathname string, de *godirwalk.Dirent) error {
if optSkip != "" && strings.Contains(osPathname, optSkip) {
return godirwalk.SkipThis
}
// Process file like normal...
return nil
}
```
### It's more flexible than `filepath.Walk`
#### Configurable Handling of Symbolic Links
The default behavior of this library is to ignore symbolic links to
directories when walking a directory tree, just like `filepath.Walk`
does. However, it does invoke the callback function with each node it
finds, including symbolic links. If a particular use case exists to
follow symbolic links when traversing a directory tree, this library
can be invoked in manner to do so, by setting the
`FollowSymbolicLinks` config parameter to `true`.
#### Configurable Sorting of Directory Children
The default behavior of this library is to always sort the immediate
descendants of a directory prior to visiting each node, just like
`filepath.Walk` does. This is usually the desired behavior. However,
this does come at slight performance and memory penalties required to
sort the names when a directory node has many entries. Additionally if
caller specifies `Unsorted` enumeration in the configuration
parameter, reading directories is lazily performed as the caller
consumes entries. If a particular use case exists that does not
require sorting the directory's immediate descendants prior to
visiting its nodes, this library will skip the sorting step when the
`Unsorted` parameter is set to `true`.
Here's an interesting read of the potential hazzards of traversing a
file system hierarchy in a non-deterministic order. If you know the
problem you are solving is not affected by the order files are
visited, then I encourage you to use `Unsorted`. Otherwise skip
setting this option.
[Researchers find bug in Python script may have affected hundreds of studies](https://arstechnica.com/information-technology/2019/10/chemists-discover-cross-platform-python-scripts-not-so-cross-platform/)
#### Configurable Post Children Callback
This library provides upstream code with the ability to specify a
callback function to be invoked for each directory after its children
are processed. This has been used to recursively delete empty
directories after traversing the file system in a more efficient
manner. See the `examples/clean-empties` directory for an example of
this usage.
#### Configurable Error Callback
This library provides upstream code with the ability to specify a
callback to be invoked for errors that the operating system returns,
allowing the upstream code to determine the next course of action to
take, whether to halt walking the hierarchy, as it would do were no
error callback provided, or skip the node that caused the error. See
the `examples/walk-fast` directory for an example of this usage.
Codebase Browser
kubernetes