Which Twitter API is best for AI or LLM training data?

Which Twitter API is best for AI or LLM training data?

For corpus building the decision turns on three things and features are not among them: cost per tweet, how far back and how deep you can reach, and what shape the records arrive in. A standard read here returns about 20 tweets for $0.0008, which is $0.04 per 1,000 and about $400 for ten million, delivered as plain JSON with the fields inline. The official platform sells the same volume through monthly rungs from $200, where a fixed post allowance rather than the sticker price sets your ceiling, so the effective per-tweet cost depends entirely on how completely you use the rung. The sections below work through each of those three, then the two collection decisions that are expensive to reverse, then the licensing question that no API can answer for you.

What is the cost per tweet for bulk collection?

Every other consideration is fixed cost. Cost per tweet is the one that multiplies by your corpus size, so it deserves to be settled first and settled precisely. A standard read returns roughly 20 tweets for $0.0008, which is $0.00004 a tweet, or $0.04 per 1,000. Ten million tweets is therefore about $400 of reads. On the official platform the equivalent arithmetic runs through a subscription rung: Basic opens at $200 a month and bundles a fixed post allowance, so your effective per tweet price depends entirely on whether you use the allowance fully. Under-use it and you have paid a premium for tweets you never collected. Exhaust it and you are pushed to the next rung regardless of how little of that rung you need. A metered rate card and an allowance behave differently at the margin, and for a one-off corpus build the difference is usually decisive, because a corpus build is exactly the spiky workload an allowance prices badly.

How much does a million tweet corpus cost?

About $40 in reads at $0.04 per 1,000, assuming full pages of roughly 20 tweets a call. Budget above that figure rather than at it, because partial pages, retries and the exploratory queries you run before settling on a final frame all add calls. A useful planning rule is to treat the arithmetic as a floor and add a margin for the collection you will throw away, which on a first build is often a third of it.

Can you get the full historical Twitter archive?

Corpus quality is mostly a question of what you can reach rather than how fast you can pull it. Three limits do most of the damage. The first is archive depth: a provider that only serves recent posts cannot build a historical set at any price, which rules it out for anything longitudinal. The second is per account depth, since a timeline read that stops after a few thousand posts will silently truncate prolific accounts and skew your sample toward quieter ones. The third is deleted and protected content, which is absent everywhere and should be, but which means your corpus is a record of what remained public at collection time rather than of what was posted. State that in your dataset card. A model trained on a corpus whose survivorship bias is undocumented will surprise somebody later, and the person it surprises is usually not the person who built it.

Does Twitter API data arrive ready to train on?

The unglamorous cost of corpus building is transformation, and it is routinely underestimated because it does not appear on an invoice. Responses arrive as plain JSON with the fields inline, so a row is usable as a training record with light flattening and no expansion syntax to learn. Where teams lose time is in the parts nobody budgets: normalising timestamps to a single timezone, resolving entity offsets so mentions and links can be masked, deciding whether to keep or strip quoted content, and choosing a canonical form for text that arrives with HTML entities. Settle those rules once, write them down beside the dataset, and apply them at ingestion. A corpus assembled under three different normalisation rules because the collection ran across three months is worse than a smaller corpus assembled under one, and the inconsistency is nearly impossible to detect after the fact.

How do you deduplicate a tweet corpus?

Deduplicate on post id at write time rather than in a later pass, because paginated collection overlaps at window edges by design and retries produce repeats. A corpus carrying duplicate rows over-weights whatever was duplicated, and what gets duplicated is rarely random: it is the most-retweeted and most-paginated material, which is to say the least representative. Doing it at ingestion also means the count you report is the count you have, rather than a number you have to caveat later. Two subtleties worth handling at the same time. A retweet and its original are different ids and both legitimate rows, so decide explicitly whether your corpus wants both. And identical text posted by different accounts is not a duplicate by id, so if you are deduplicating for content rather than for provenance you need a second pass on a text hash, which is a different decision with different consequences for what the corpus represents.

How do you sample tweets without biasing the corpus?

Decide the sampling frame explicitly and write it down, because a corpus assembled without one is shaped by whatever the search ranking happened to return. Collecting everything matching a keyword gives you a corpus shaped like that keyword's community, which is fine when it is intentional and misleading when it is not. If you want breadth, sample deliberately across accounts and across time rather than pulling the top results and stopping. If you want a community, say so and characterise it. The failure mode to avoid is the accidental frame: a collection that ran for three weeks and captured whatever was popular in those three weeks, described afterwards as a sample of the platform. Any result computed on that corpus is a result about those three weeks, and the person who reads it later will not know that unless the dataset card says so.

Should you collect a snapshot or collect continuously?

That follows from what the model is for. A snapshot is reproducible and easy to document, which suits a benchmark or a paper. A rolling collection tracks drift and suits a production model that must not go stale. If you choose rolling, version the corpus explicitly and keep the boundaries, because a training set that quietly grew between two runs makes any comparison between those runs meaningless.

What happens to deleted and protected posts in a corpus?

They are absent, everywhere, by construction, and that absence is a property of your dataset rather than a gap to apologise for. Deleted posts are gone from the source, and protected accounts are not public, so no provider can return either and any that claimed to would be the wrong provider to use. The consequence worth recording is survivorship: your corpus is a record of what remained public at the moment you collected it, not of what was posted. That skews in a specific and predictable direction, because posts get deleted for reasons that correlate with their content, so a corpus of surviving posts under-represents exactly the material somebody thought better of. State it in the dataset card. A model trained on an undocumented survivorship bias will surprise somebody eventually, and it will not be the person who built it.

What should a tweet dataset card record?

The collection window, the sampling frame, the deduplication rule, the normalisation rules, the endpoints used, and the fact that deleted and protected content is absent by construction. That last point is the one most often omitted and the one most likely to matter, because it means your corpus records what survived to collection time rather than what was posted. A reviewer who cannot tell what the corpus is a sample of cannot judge any result computed on it.

Is a scraper cheaper than an API for building a corpus?

Rarely, once you count the parts that do not appear on an invoice. A self-built scraper carries proxy costs, account churn, and an ongoing maintenance burden every time the site markup changes, and it tends to fail in the middle of long collections, which is exactly when failure is most expensive. For a one-off small corpus it can be cheaper. For anything sustained or anything you need to reproduce, metered access is usually both cheaper and far less eventful.

Are you allowed to train a model on tweets?

Whether you may train on this material is a legal question about your jurisdiction, your use, and the agreements you have accepted, and it is not settled by which API you call. It is genuinely unsettled in several jurisdictions and moving. Treat it the way you would treat any other data licensing question: get an answer from counsel before the corpus exists rather than after, keep provenance records showing what was collected and when, and keep the collection reproducible so you can demonstrate what you did. Anyone who tells you the answer is simple, in either direction, is selling something. What a data provider can honestly give you is reliable access and a clean record of what was retrieved. It cannot give you a licence to train, and you should be suspicious of any vendor that implies otherwise.

What a corpus build costs, by access model

Questions and answers

How many tweets does one call return?

About 20 on a full page, which is where the $0.04 per 1,000 figure comes from. Budget above the arithmetic rather than at it: partial pages, retries and the exploratory queries you run before settling on a final frame all add calls, and on a first build the collection you end up discarding is often a third of the total.

Can I resume an interrupted collection?

Yes, if you stored the cursor. Persist next_cursor alongside your results rather than holding it in memory, so an interrupted run resumes at the right page instead of restarting and re-paying for everything you already have. On a long collection this is the difference between a retry costing minutes and costing the whole budget again.

What format do the records arrive in?

Plain JSON with the fields inline, so a row is usable as a training record after light flattening and there is no expansion or field-selection syntax to learn first. The work that remains is normalisation rather than parsing: timestamps to one timezone, a canonical form for text that arrives with HTML entities, and a decision about entity offsets if you intend to mask mentions or links.

Does collecting at higher volume get cheaper per tweet?

No, the rate card is flat, which is the main structural difference from a subscription rung. Ten million tweets costs ten times what a million costs, with no threshold where the unit price drops and equally no cliff where a rung runs out and forces an upgrade you did not need. For a spiky one-off build that flatness is usually the deciding property rather than the headline rate.

Can I filter by language at collection time?

Yes, and doing it at collection is cheaper than doing it afterwards, because you are not paying to retrieve rows you intend to discard. Be aware that platform language detection is imperfect on short text, so a language-filtered collection is a good first pass rather than a clean partition, and a corpus that must be single-language still needs a verification step of its own.

Keep reading

• Twitter search API
• Twitter API pricing, full comparison
• How to get Twitter data as JSON
• How to download a full account tweet history