Scrapping

ScrapeCreators API

ScrapeCreators provides 100+ REST endpoints to scrape public data from 20+ social media platforms. One API key, one header, simple curl requests.

Quick start

Authentication

Every request needs a single header:

x-api-key: YOUR_API_KEY

Get your key at https://scrapecreators.com (100 free credits on signup, no card required).

Store the key in an environment variable so it stays out of scripts and doesn't end up in version control or chat history:

export SCRAPECREATORS_API_KEY="your-key-here"

Your first request

curl -s -G "https://api.scrapecreators.com/v1/tiktok/profile" \
  --data-urlencode "handle=khaby.lame" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" | jq .

That's the whole pattern. Every endpoint works the same way: GET request, query parameters, one auth header.

Base URL

https://api.scrapecreators.com

How credits work

Common parameters

These work across most endpoints:

| Parameter | Description | |-----------|-------------| | trim=true | Returns a trimmed, smaller response — keeps context window manageable and saves tokens when you only need key fields like names, stats, and IDs | | cursor | Pagination cursor returned in previous response — pass it to get the next page. Each page costs 1 credit, so only paginate if the user needs more results. Note: the v3 TikTok profile/videos endpoint uses max_cursor instead of cursor | | includeExtras=true | Returns additional fields (like counts, descriptions) on YouTube endpoints — without this, channel-videos only returns titles and IDs | | sort_by | Sort results — values depend on endpoint (e.g., popular, relevance, total_impressions, recency). Check the platform reference for valid values per endpoint, since passing an invalid value returns a 400 error |

Pagination

Many endpoints return paginated results. The pattern is:

1. Make the first request without a cursor 2. The response includes a cursor (or next_cursor or next_page_id) field 3. Pass that value as ?cursor=... in the next request 4. Repeat until cursor is null/empty

# First page
curl -s -G "https://api.scrapecreators.com/v3/tiktok/profile/videos" \
  --data-urlencode "handle=khaby.lame" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" | jq .
Next page (using max_cursor from previous response)
curl -s -G "https://api.scrapecreators.com/v3/tiktok/profile/videos" \
  --data-urlencode "handle=khaby.lame" \
  --data-urlencode "max_cursor=CURSOR_VALUE" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" | jq .

Some v2 endpoints require manual pagination — check the platform reference for specifics.

Supported platforms and endpoints

Each platform has its own reference file with full endpoint details. Read the relevant one based on the user's request.

| Platform | Reference file | Key endpoints | |----------|---------------|---------------| | TikTok | references/tiktok.md | Profile, videos, comments, search, trending, shop, songs, transcripts, live, followers/following, audience demographics | | Instagram | references/instagram.md | Profile, posts, reels, comments, search reels, transcripts, story highlights, embed | | YouTube | references/youtube.md | Video details, channel info, channel videos, shorts, search, transcripts | | Twitter/X | references/twitter.md | Profile, community, community tweets | | LinkedIn | references/linkedin.md | Person profile, company page, company posts, post details, ad library search, ad details | | Facebook | references/facebook.md | Posts, comments, reels, ad library, transcripts, group posts, profile | | Reddit | references/reddit.md | Profile, subreddit details/posts/search, post comments, search, ad library | | Other platforms | references/other-platforms.md | Pinterest, Threads, Bluesky, Snapchat, Twitch, Kick, Truth Social, Google (ads + search), link-in-bio platforms (Linktree, Komi, Pillar, Linkbio, Linkme), Amazon Shop |

Making requests: the pattern

Every call follows the same shape. Always use -G with --data-urlencode to safely pass query parameters — this prevents shell injection from user-provided values (handles, search queries, URLs) and properly encodes special characters:

curl -s -G "https://api.scrapecreators.com/v1/{platform}/{endpoint}" \
  --data-urlencode "param=value" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY"

Pipe through jq to format the JSON, or through jq '.some.field' to extract specific data. Using jq is important because raw API responses are often large nested JSON — extracting just the fields the user needs makes the output readable and keeps your context window clean.

Chaining requests

A common pattern is to fetch a profile first (to confirm the account exists and get IDs), then drill into their content:

# 1. Get profile
PROFILE=$(curl -s -G "https://api.scrapecreators.com/v1/tiktok/profile" \
  --data-urlencode "handle=creator_name" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY")
2. Get their recent posts
POSTS=$(curl -s -G "https://api.scrapecreators.com/v3/tiktok/profile/videos" \
  --data-urlencode "handle=creator_name" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY")
3. Get comments on a specific video (extract video ID from posts)
VIDEO_ID=$(echo "$POSTS" | jq -r '.aweme_list[0].aweme_id')
COMMENTS=$(curl -s -G "https://api.scrapecreators.com/v1/tiktok/video/comments" \
  --data-urlencode "video_id=$VIDEO_ID" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY")

Saving results

# Save raw JSON
curl -s -G "https://api.scrapecreators.com/v1/instagram/profile" \
  --data-urlencode "handle=natgeo" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" > natgeo_profile.json
Save as CSV (extract specific fields with jq)
curl -s -G "https://api.scrapecreators.com/v3/tiktok/profile/videos" \
  --data-urlencode "handle=creator" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" \
  | jq -r '.aweme_list[] | [.aweme_id, .desc, .statistics.play_count, .statistics.digg_count] | @csv' \
  > creator_posts.csv

Processing and analyzing scraped data

After fetching data, here are common analysis patterns:

Extract key metrics

# Get engagement stats from a profile
curl -s -G "https://api.scrapecreators.com/v1/tiktok/profile" \
  --data-urlencode "handle=creator" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" \
  | jq '{followers: .statsV2.followerCount, following: .statsV2.followingCount, likes: .statsV2.heartCount}'

Aggregate across posts

# Average engagement across recent posts
curl -s -G "https://api.scrapecreators.com/v3/tiktok/profile/videos" \
  --data-urlencode "handle=creator" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" \
  | jq '[.aweme_list[] | .statistics.digg_count] | (add / length)'

Get video transcripts for content analysis

# Fetch transcript of a TikTok video
curl -s -G "https://api.scrapecreators.com/v1/tiktok/video/transcript" \
  --data-urlencode "video_id=VIDEO_ID" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" | jq -r '.data.transcript'

Search and filter

# Find TikTok creators by niche with audience filters
curl -s -G "https://api.scrapecreators.com/v1/tiktok/creators/popular" \
  --data-urlencode "min_followers=100000" \
  --data-urlencode "audience_country=US" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" | jq '.data'

Important notes

Error handling

If a request fails: 1. Check the HTTP status code 2. The response body usually contains an error message 3. Common issues: invalid API key (401), endpoint not found (404), timeout on slow requests (502/503)

# Check status code
curl -s -G -o response.json -w "%{http_code}" \
  "https://api.scrapecreators.com/v1/tiktok/profile" \
  --data-urlencode "handle=creator" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY"

Choosing the right endpoint

When the user describes what they need, match it to the right platform and endpoint:

When in doubt, check the platform reference file for the full endpoint list.

Full API docs

For the complete interactive documentation with response schemas and playground: https://docs.scrapecreators.com

Authentication

Every request needs a single header:

x-api-key: YOUR_API_KEY

Get your key at https://scrapecreators.com (100 free credits on signup, no card required).

Store the key in an environment variable so it stays out of scripts and doesn't end up in version control or chat history:

export SCRAPECREATORS_API_KEY="your-key-here"

Your first request

curl -s -G "https://api.scrapecreators.com/v1/tiktok/profile" \
  --data-urlencode "handle=khaby.lame" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" | jq .

That's the whole pattern. Every endpoint works the same way: GET request, query parameters, one auth header.