Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
G

gcsfs

> 编程语言
Open source

Pythonic file-system interface for Google Cloud Storage

394 stars0 likes1 views
WebsiteGitHub

About

Pythonic file-system interface for Google Cloud Storage

gcsfs

|Build Status| |Docs|

GCSFS is a Python library that provides a familiar, file-system-like interface to Google Cloud Storage (GCS). Built on top of fsspec, it allows you to interact with cloud buckets as if they were local directories, making it a favorite for data scientists and engineers.


Getting Started

Installation

Install via pip or conda:

bash
# Using pip
pip install gcsfs

# OR using conda
conda install -c conda-forge gcsfs

Basic Usage

python
import gcsfs

# Initialize the filesystem
fs = gcsfs.GCSFileSystem(project='my-google-project')

# List files in a bucket
files = fs.ls('my-bucket')

# Read a file directly into a string/bytes
with fs.open('my-bucket/data.txt', 'rb') as f:
    content = f.read()

Specialized Bucket Support

GCSFS now automatically supports advanced Google Cloud Storage features through its ExtendedFileSystem implementation.

1. Hierarchical Namespace (HNS)

Hierarchical Namespace (HNS) replaces the traditional "flat" GCS structure with true logical directories.

  • Atomic Renames: Moving or renaming a directory is an O(1) metadata operation. No more slow "copy-then-delete" for large folders.
  • High Performance: Offers up to 8x higher initial Queries Per Second (QPS) for read/write operations.
  • AI/ML Ready: Ideal for heavy checkpointing and managing millions of small files.

2. Rapid Buckets (Zonal Storage)

Rapid Buckets are zonal storage resources designed for ultra-low latency and maximum throughput.

  • Zonal Co-location: Place your data in the same zone as your GPU/TPU clusters to minimize network lag.
  • True Appends: Unlike standard GCS objects, you can append data to existing objects in Rapid buckets without a full rewrite.
  • Streaming I/O: Optimized for high-speed model loading and real-time logging.

Integration & Auth

GCSFS plays nicely with the rest of the Python data ecosystem.

Authentication Modes

  • Default: Uses your local gcloud credentials or environment service accounts.
  • Cloud: Explicitly use Google Metadata service (token='cloud').
  • Anonymous: Access public data without a login (token='anon').
  • Service Account: Pass the path to your JSON key file (token='path/to/key.json').

[!TIP] Note on Async: GCSFS is built on aiohttp. If you are building high-concurrency applications, you can use the asynchronous API by passing asynchronous=True to the GCSFileSystem constructor.


Support

Work on this repository is supported in part by:

"Anaconda, Inc. - Advancing AI through open source."

Issues· 111 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Pythonhacktoberfest

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 17, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言