tiktok-scraper

TikTok Scraper & Downloader

Scrape and download useful information from TikTok.

No login or password are required

This is not an official API support and etc. This is just a scraper that is using TikTok Web API to scrape media and related meta information.

---

---

Content

• Features
• To Do
• Contribution
• Installation
• Usage
  • In Terminal
    • Terminal Examples
    • Manage Download History
    • Scrape and Download in Batch
    • Output File Example
  • Docker
    • Build
    • Run
  • Module
    • Methods
    • Options
    • Use with Promises
    • Use with Events
    • How to get/set session value
    • How to access/download video
    • Output Example
      • Video Feed Methods
      • getUserProfileInfo
      • getHashtagInfo
      • getVideoMeta
      • getMusicInfo

Important notes

• As of right now it is NOT possible to download video without the watermark

Features

• Download unlimited post metadata from the User, Hashtag, Trends, or Music-Id pages
• Save post metadata to the JSON/CSV files
• Download media with and without the watermark and save to the ZIP file
• Download single video without the watermark from the CLI
• Sign URL to make custom request to the TikTok API
• Extract metadata from the User, Hashtag and Single Video pages
• Save previous progress and download only new videos that weren't downloaded before. This feature only works from the CLI and only if download flag is on.
• View and manage previously downloaded posts history in the CLI
• Scrape and download user, hashtag, music feeds and single videos specified in the file in batch mode

To Do

• CLI: save progress to avoid downloading same videos
• Rewrite everything in TypeScript
• Improve proxy support
• Add tests
• Download video without the watermark
• Indicate in the output file(csv/json) if the video was downloaded or not
• Build and run from Docker
• CLI: Scrape and download in batch
• CLI: Load proxies from a file
• CLI: Optional ZIP
• Renew API
• Set WebHook URL (CLI)
• Add new method to collect music metadata
• Add Manual Pagination
• Improve documentation
• Download audio files
• Web interface

Contribution

• Don't forget about tests

yarn test

yarn build

Installation

tiktok-scraper requires Node.js v10+ to run.

Install from NPM

npm i -g tiktok-scraper

Install from YARN

yarn global add tiktok-scraper

USAGE

In Terminal

…

• Terminal Examples
• Manage Download History
• Scrape and Download in Batch

Output File Example

Docker

By using docker you won't be able to use --filepath and --historypath , but you can set volume(host path where all files will be saved) by using -v

Build

docker build . -t tiktok-scraper

Run

Example 1: All files including history file will be saved in the directory($pwd) where you running the docker from

docker run -v $(pwd):/usr/app/files tiktok-scraper user tiktok -d -n 5 -s

Example 2: All files including history file will be saved in /User/blah/downloads

docker run -v /User/blah/downloads:/usr/app/files tiktok-scraper user tiktok -d -n 5 -s

Module

Methods

…

Options

…

Don't forget to check the examples folder

Promise

…

Event

const TikTokScraper = require('tiktok-scraper');

const users = TikTokScraper.userEvent("tiktok", { number: 30 });
users.on('data', json => {
    //data in JSON format
});
users.on('done', () => {
    //completed
});
users.on('error', error => {
    //error message
});
users.scrape();

const hashtag = TikTokScraper.hashtagEvent("summer", { number: 250, proxy: 'socks5://1.1.1.1:90' });
hashtag.on('data', json => {
    //data in JSON format
});
hashtag.on('done', () => {
    //completed
});
hashtag.on('error', error => {
    //error message
});
hashtag.scrape();

Get Set Session

NOT REQUIRED

Very common problem is when tiktok is blacklisting your IP/PROXY and in such case you can try to set session and there will be higher chances for success

Get the session:

• Open https://www.tiktok.com/ in any browser
• Login in to your account
• Right click -> inspector -> networking
• Refresh page -> select any request that was made to the tiktok -> go to the Request Header sections -> Cookies
• Find in cookies sid_tt value. It usually looks like that: sid_tt=521kkadkasdaskdj4j213j12j312;
• sid_tt=521kkadkasdaskdj4j213j12j312; - this will be your authenticated session cookie value that should be used to scrape user/hashtag/music/trending feed

Set the session:

• CLI:

  • Set single session by using option --session. For example --session sid_tt=521kkadkasdaskdj4j213j12j312;
  • Set path to the file with the list of sessions by using option --session-file. For example --session-file /var/bob/sessionList.txt
    • Example content /var/bob/sessionList.txt:

    sid_tt=521kkadkasdaskdj4j213j12j312;
    sid_tt=521kkadkasdaskdj4j213j12j312;
    sid_tt=521kkadkasdaskdj4j213j12j312;
    sid_tt=521kkadkasdaskdj4j213j12j312;
    

• In the MODULE you can set session by setting the option value sessionList . For example sessionList:["sid_tt=521kkadkasdaskdj4j213j12j312;", "sid_tt=12312312312312;"]

Download Video

This part is related to the MODULE usage (NOT THE CLI)

The {videoUrl} value is binded to the cookie value {tt_webid_v2} that can contain any value

Method 1: default headers

When you extract videos from the user, hashtag, music, trending feed or single video then in response besides the video metadata you will receive headers object that will contain params that were used to extract the data. Here is the important part, in order to access/download video through {videoUrl} value you need to use same {headers} values.

    headers: {
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36",
        "referer": "https://www.tiktok.com/",
        "cookie": "tt_webid_v2=689854141086886123"
    },

Method 2: custom headers

You can pass your own headers with the {options}.

const headers = {
    "user-agent": "BOB",
    "referer": "https://www.tiktok.com/",
    "cookie": "tt_webid_v2=BOB"
}
getVideoMeta('WEB_VIDEO_URL', {headers})
user('WEB_VIDEO_URL', {headers})
hashtag('WEB_VIDEO_URL', {headers})
trend('WEB_VIDEO_URL', {headers})
music('WEB_VIDEO_URL', {headers})
// And after you can access video through {videoUrl} value by using same custom headers

Json Output Example

Video Feed

Example output for the methods: user, hashtag, trend, music, userEvent, hashtagEvent, musicEvent, trendEvent

…

getUserProfileInfo

{
    secUid: 'MS4wLjABAAAAv7iSuuXDJGDvJkmH_vz1qkDZYo1apxgzaxdBSeIuPiM',
    userId: '107955',
    isSecret: false,
    uniqueId: 'tiktok',
    nickName: 'TikTok',
    signature: 'Make Your Day',
    covers: ['COVER_URL'],
    coversMedium: ['COVER_URL'],
    following: 490,
    fans: 38040567,
    heart: '211522962',
    video: 93,
    verified: true,
    digg: 29,
}

getHashtagInfo

{
    challengeId: '4231',
    challengeName: 'love',
    text: '',
    covers: [],
    coversMedium: [],
    posts: 66904972,
    views: '194557706433',
    isCommerce: false,
    splitTitle: ''
}

getVideoMeta

…

getMusicInfo

…

---

License

---

MIT

Free Software