让 Reddit 数据可供研究人员、管理员和其他人使用。通过大型数据文件、 API 或 Web 界面与数据进行交互。
Making Reddit data accessible to researchers, moderators and everyone else.
Interact with the data through large dumps, an API or web interface.
All download links are organized here. Once a new dump is available, it will also be added on the releases page.
Alternatively for downloading data of users or smaller subreddits, you can use this tool.
For information on how the data was collected and modified, see here.
Depending on your use case, you can try my (limited) API. For manual queries, you can use this tool.
First download one more dumps from the above links.
Generally I'd recommend to work with the compressed files instead of unpacking them. Unless of course you have seemingly infinite disk space.
With the helper scripts in this repository you can quickly get started.
(You need at least Python 3.10)
git clone --recursive https://github.com/ArthurHeitmann/arctic_shift.git
cd arctic_shift
zstandard librarypip install zstandard
Open scripts/processFiles.py in your editor. That script can process .zst, .zst_blocks, .jsonl/ndjson and .json files.
Enter the path to a file or folder in fileOrFolderPath (since it is a raw string, you don't have to escape
backslashes). If you enter a folder, all files in that folder will be processed.
Add your code to the processFile function.
Run the file and be (very) patient.
Removal requests and generic support requests can be submitted here. To check if your data is in the dataset, search for your username here.
Removal forms of other archives: Pushshift | PullPush | potentially archive.org.
If you have questions, you can DM me on discord (raiderbv if the link doesn't work) or email. Alternatively open an issue or pull request.
If you want to support my projects, you can donate through PayPal or buy me a coffee.
暂无开放 Issues，或尚未同步最近议题。