Pluggable, extensible virtual file system for Go
vfs provides a unified interface for working with different storage backends (local OS, Azure, S3, GCS, SFTP, etc.) in a consistent way. Rather than handling each backend independently, you interact with a core set of interfaces that abstract backend details. This allows you to:
io.Reader/io.Writer interfaces).By focusing on FileSystem, Location, and File interfaces, you can build reusable flows for file operations without needing to deeply understand each backend's specific APIs. Users can add or swap backends as needed, providing flexibility for hybrid or evolving storage requirements.
go get -u github.com/c2fo/vfs/v7…This snippet shows the basic setup: an osFile is created from a URI and written to using standard libraries.
See the GoDoc for deeper technical details, backend-specific options, and additional APIs.
Specialized operations like copying between S3 locations or advanced configs (e.g. Azure credentials) are documented in each backend doc.
These backends are maintained in contrib/backend/ as separate Go modules:
See contrib/backend/README.md for guidelines on contributing new backends.
Q: Why am I seeing an empty file when using io.Copy on some backends if my source is empty?
A: An empty Reader often means the backend doesn't write a file until data is actually written. Use utils.TouchCopy if you need to ensure a zero-byte file is created.
Q: Will vfs v6 still be supported? A: Yes and no. We will continue to provide security patches and bug fixes for v6, but new features and enhancements will be added to v7.
Q: How long will v6 be supported? A: We will support v6 until the next major release, v8, is released. Then v7 will be supported until v9 is released.
Please review these changes and update your code accordingly to ensure compatibility with v7.
The project now uses the aws-sdk-go-v2 library instead of the deprecated, EOL aws-sdk-go. This update necessitated
these changes to the S3 backend:
s3fs.Client() function now returns an s3.Client which is a subset of AWS's sdk v2 functionality.
This change may require updates to your code if you were relying on client functionality not directly required by the
s3 vfs backend.Option.Retry field is now an aws.Retryer instead of a request.Retry. Ensure that your Option logic is
compatible with the new type.https to az. Update your code to use the new scheme.blob.core.windows.net to my-container-name, such that the full URI
is az://my-blob-name/path/to/file.txt rather than
https://my-storage-account-name.core.windows.net/my-container-name/path/to/file.txt.Options.Retry field, with the now deprecated vfs.Retry type, has been moved to gs.FileSystem as the new gs.Retyer
type. It is now set via gs.NewFileSystem function using functional option gs.WithRetryer. All previous vfs.Retry
usage has been replaced with gs.WithRetryer. Update your code to use the new gs.WithRetryer functional option.Some methods in the Location and FileSystem interfaces have been deprecated because they use terminology that doesn't apply to all backends. They will be removed in a future release. Update your code to use the new methods. See #235.
location.Volume() method which returns the authority as a string has been deprecated in favor of the location.Authority() method which
returns an authority.Authority struct. Update your code to use the Authority().String() method instead of Volume().location.ChangeDir() method ash been deprecated in favor of the existing location.NewLocation() method. Update your code to use the
NewLocation() method instead of ChangeDir().vfs.Options struct has been deprecated in favor of using backend-specific structs.filesystem.Retry() method has been deprecated in favor of using backend-specific functional options, such as gs.WithRetryer.Additionally, we have added functional option interface, FileSystemOption, to allow for more flexible configuration
of backends. This interface allows for more complex configuration options to be passed to the via the NewFileSystem function.
This will replace backend-specific chainable functions that require casting the filesystem to the backend type first. See #238.
With v6, sftp.Options struct changed to accept an array of Key Exchange algorithms rather than a string. To update,
change the syntax of the auth commands.
"keyExchanges":"diffie-hellman-group-a256"becomes
"keyExchanges":["diffie-hellman-group-a256"]git clone https://github.com/c2fo/vfs.git)git checkout -b feature/fooBar)git commit -am "Add some fooBar")git push origin feature/fooBar)Distributed under the MIT License. See the LICENSE.md file for details.
No open issues yet, or sync has not completed.