What the Twitter Firehose Actually Was, and What You Get Instead in 2026

TL;DR: The firehose was a specific product: every public post, in realtime, with no sampling, sold under enterprise contract and for years resold by Gnip, which the platform acquired in April 2014. It is not what anyone sells self serve today. What replaced it for almost everyone is per-call reads, where you ask a question and get a page back. On the measured yield of 8.19 posts per search call and the standard rate of 0.0008 dollars, one million matched posts costs about 97.68 dollars, and 29.72 percent of those calls return nothing at all. What you cannot buy at any price on that model is a completeness guarantee, and most people asking for the firehose are actually asking for one.

Somebody asks for "the firehose" roughly once a week, and about nine times out of ten they do not want it. They want to be sure they are not missing anything, which is a different request, and the difference is worth an article because it changes the budget by two orders of magnitude.

This post does three things. It says what the word meant when it was a product with a price and a contract. It records what happened to that product and when. And it prices the modern alternative using measurements from our own billing logs rather than from a rate card, because the rate card is the least interesting half of what a call costs.

Every first-party number below comes from the same place: the log that prices this API, which records one row per call with its endpoint, its status and the number of rows it returned. The window is 2026-06-22 to 2026-09-02 and it covers 5,155,699 calls. Where a figure is arithmetic on those rows rather than a direct reading, the table says so, and where a claim is a design judgement rather than a measurement, it is tagged as such rather than dressed up as data.

What was the Twitter firehose?

The firehose was a realtime stream carrying every public post on the platform, unsampled and unfiltered, delivered over a persistent connection to a customer who had signed an enterprise contract for it. That is the whole definition, and the two words that matter are every and unsampled.

What each historical product actually carried

Product	Share of the stream	Filtering	How it was sold	Status in 2026	Source
Firehose	100%	None, everything delivered	Enterprise contract	Not self serve	published
PowerTrack	100% before filtering	Your rules applied before delivery	Enterprise contract	Documented enterprise product	published
Decahose	10%, random	None beyond the sample	Enterprise contract	Documented enterprise product	published
Gardenhose	10%, historical name	None	Legacy access	Retired	unknown
Spritzer	1%, historical name	None	Legacy public sample	Retired	published

as of 2026-09-02

For most of its commercial life the firehose was not sold by the platform directly. It was resold by Gnip, a social data company that held full-stream access as one of a handful of authorised partners, until the platform acquired Gnip in April 2014 in a deal reported at around 134 million dollars. That acquisition is the reason the enterprise documentation still carries the Gnip name in its URL structure today, which is a small piece of archaeology that confuses everyone who trips over it.

Around the full stream sat a product family, and the family is where the confusion starts, because the members are frequently discussed as though they were price tiers of the same thing. They are not. They are different products with different completeness properties.

PowerTrack, documented inside the enterprise API family, is the filtered full stream. The platform applies your rules to the complete firehose before delivery, so you receive everything matching them and nothing else. It is complete with respect to your filter, which is a genuinely strong guarantee and the one that compliance and archival use cases actually need.

The Decahose is a ten percent random sample of the same realtime stream. The word random is doing the load-bearing work. A random tenth of a population supports inference about that population within known error bounds. Ten percent of the posts that happened to match your query supports nothing of the sort, because a query is a filter and a filter has no sampling properties at all.

Below those sat two older names that appear constantly in documentation written before 2015 and almost never in anything current. Gardenhose and Spritzer were sample tiers of the public streaming API, and both are retired. They are worth knowing only because a search for firehose pricing will surface a decade-old blog post that treats them as live products.

Why does the word firehose cause so much trouble?

Five different things get called a firehose, and a conversation where two people mean different ones can run for a long time before anybody notices. The confusion is not carelessness: each of the five is a real property of a real product, and the products that had them were sold together.

The first is the literal product described above. The second is any high-volume push feed, where the speaker means the delivery mechanism rather than the completeness. The third is unfiltered access, where the speaker means no query rather than no sampling. The fourth is historical completeness, where somebody wants the full archive rather than the live stream, which is an entirely different product problem. And the fifth is permission to scale, where the real request is the absence of a rate limit.

The real choice

A stream and a paged read are not two prices for one thing

	Enterprise stream	Per-call reads
What arrives	Everything matching your rules	The page your call asked for
Who filters	The platform, before delivery	You, in the query
Completeness claim	Contractual	None available
How you are billed	Negotiated, annual	Per call, no minimum
Fails by	Dropping a connection	Returning a short page
Right for	Archives and compliance	Named subjects and terms

Two questions separate all five, and they are independent of each other.

What share of the matching data does this carry? That is the completeness question, and the answers are all of it, a stated random fraction of it, or an unknown fraction of it. Most commercial offers are the third, described in language borrowed from the first.

How is it delivered? That is the transport question, and the answers are the server pushes to you or you ask the server. A stream can carry a one percent sample and still be a stream. A per-call read can walk a complete conversation and still be a poll.

Vendors answer the second question when asked the first with remarkable consistency, and usually without meaning to deceive, because inside a company that runs a streaming product the two feel like the same fact. Ask both, separately, in writing.

What actually happened to the Twitter firehose?

The access model that most developers and researchers used did not erode gradually. It closed on a specific pair of dates in 2023, first with the announcement that free access was ending and then, two months later, with the paid self-serve tiers that replaced it.

In February 2023 the platform announced that free access to the API was ending, with a paid basic tier to follow. The reaction from the developer community was immediate and is still readable:

https://www.reddit.com/r/programming/comments/10rw2ig/twitter_to_end_free_access_to_its_api_in_elon/

Two months later the replacement arrived as a set of self-serve tiers, announced by the platform's own developer account:

https://x.com/XDevelopers/status/1641222782594990080

That thread is the cleanest first-party marker of the transition, and the discussion it produced captures what the change meant for people who had built on the old model:

https://www.reddit.com/r/programming/comments/1265nzt/twitterdev_announces_new_twitter_api_tiers/

The academic consequence was covered widely at the time, including in Nature, because a large body of published social science had been built on access that no longer existed. The practical effect on research is not simply that data got more expensive. It is that data access moved from an application form to a budget line, and a budget line is not evenly distributed across institutions.

The self-serve model has moved again since. As of this writing the platform documents pay-per-use pricing with credits rather than the flat tier structure that replaced the free tier in 2023, with per-resource rates that vary by endpoint. Anything you read describing the current public API as Free, Basic, Pro and Enterprise is describing a model that has been superseded, which is a good reason to check the pricing page rather than a comparison article, including this one.

What did not change is the enterprise side. The filtered full stream and the sampled stream are still documented, still sold by contract, and still not something you can put on a card. That is the honest answer to "is the firehose still available": the products exist, the door is a sales conversation, and the vast majority of people who knock on it do not need what is behind it.

What did complete access actually mean?

The reason the firehose commanded an enterprise price was not bandwidth. It was the guarantee. A stream that carries every matching post lets you make a class of claim that no other access model supports: that what you have is what existed.

Take that away and four claims become unverifiable from outside, permanently.

You cannot verify that you received every matching post, because there is no external ground truth to compare against. The only thing that could tell you what you missed is the very system that decided what to send you.

You cannot distinguish a genuine absence from a miss. An empty page and a dropped page look identical at the client. Both are a short result set.

You cannot establish that a deleted post ever existed. Deletion removes the evidence rather than flagging it, so an archive built from public reads has a hole shaped exactly like the thing that was removed.

And you cannot tell whether an account was public at the moment you read it. Visibility changes leave no trace in the data you already hold, which is why a timeline you collected in June may be unreproducible in September with no error anywhere.

What you can claim

The difference between what you retrieved and what existed

Search calls returning zero: 29.72% · Reply reads returning zero: 62.19% · Ways to verify from outside: 0

☑ Walk cursors to exhaustion rather than stopping on a short page · ☑ Record the row count on every call · ☑ Keep a watermark per subject and read forward from it · ☑ State coverage as what you retrieved, never as what existed

This is the part where sampling gets misunderstood, and the misunderstanding is expensive in both directions.

A random sample is not a cheaper version of complete data. It is a different instrument that answers a different class of question well and another class badly. The classic result here is Morstatter and colleagues comparing the platform's sampled streaming output against the firehose, which found that agreement depends heavily on what you measure and on the size of the sample relative to the query. Aggregate rankings of common items held up reasonably. Network measures and rare signals did not.

The practical translation is one sentence. A sample describes the shape of something common and misleads about anything rare, and the burden is on the analysis to say which of the two it is doing.

There is a second, sharper failure that has nothing to do with sample size. Query results are not a sample. A query is a filter chosen by a human, and it selects on exactly the property that makes a post interesting to that human. Computing a population rate from query results is measuring the query, and the error is not reduced by collecting more of them. If your question is about the whole platform rather than about named subjects, the sampling frame has to be constructed deliberately, before the data is collected, and it has to be a property of the subjects rather than of the posts.

Our coverage honesty piece works through what a monitoring product can and cannot claim on public reads, and the companion reliability post measures the empty-page rates that make short results so easy to misread.

What do you actually get in 2026?

The replacement for the firehose, for almost everybody, is a paged read. You ask a question, the platform returns a page, and a cursor tells you whether there is more.

The path today

From a query to a stored record, with no completeness signal anywhere

That path has one property worth staring at: no step in it reports what you did not receive. The cursor tells you there is more. It does not tell you how much more, and nothing tells you what a page omitted. Every completeness property you want has to be constructed by you, from the outside, out of things you record as you go.

The construction is not difficult and it is worth stating explicitly because most pipelines skip it.

def walk(fetch, query, watermark=None, page_budget=200):
    """Page a read to exhaustion, honestly.

    Three things every version of this needs and most versions lack:
      - stop on the CURSOR, never on a short page
      - carry a budget, and say so loudly when it is spent
      - report whether the walk ended by exhaustion or by truncation
    """
    cursor, seen, pages, truncated = None, [], 0, False
    while True:
        page = fetch(query, cursor=cursor)
        seen.extend(page.items)
        pages += 1
        if watermark and any(i.id <= watermark for i in page.items):
            break                      # caught up with what we already have
        if not page.next_cursor:
            break                      # genuine exhaustion
        if pages >= page_budget:
            truncated = True           # NOT the same as exhaustion
            break
        cursor = page.next_cursor
    return seen, {"pages": pages, "truncated": truncated,
                  "returned": len(seen)}

The truncated flag is the load-bearing part. Without it, a run that hit its page budget and a run that genuinely exhausted the result set produce identical output, and every downstream consumer treats both as complete. With it, the consumer can decide what to do, and an analyst reading a chart six months later can tell whether a flat line was a quiet week or a spent budget.

The measurements in the rest of this post come from that same kind of logging applied to 5,155,699 calls. The subset with a recorded row count is 2,451,925 calls, which returned approximately 29.3 million records between them, a mean of 11.94 records per call across the whole mix. That mean is close to meaningless on its own, which is exactly the point of the next section.

What one call actually returns

Pricing is per call. Value is per record. The ratio between them is a property of the endpoint, and it varies by a factor of fifty across the catalogue, from a single-object lookup that returns exactly one row to a follower read that returned a measured mean of 49.80.

Measured rows returned per successful call, by read path

Read path	Calls measured	Mean rows	Median rows	Share returning zero	Source
user/followers	209,792	49.80	50	0.21%	measured
user/following	15,218	40.65	49	4.15%	measured
user/tweets	630,228	19.04	20	0.74%	measured
tweet/advanced_search	562,238	8.19	3	29.72%	measured
tweet/retweeters	74,815	7.38	0	52.36%	measured
tweet/replies	17,077	7.15	0	62.19%	measured
tweet/detail	883,520	1.00	1	0.00%	measured

n = 2,451,925 · as of 2026-09-02

A follower read returned a mean of 49.80 accounts, with a median of 50, which tells you the pages are almost always full. A user timeline read returned 19.04 posts against a median of 20, same story. Search is different in kind: a mean of 8.19 posts against a median of 3, with a maximum observed page of 20.

That gap between mean and median is the shape of search itself. Most queries match a handful of posts on any given page and a minority match a full one, so the average is pulled upward by the dense queries while the typical call returns very little. Anybody sizing a search workload against the mean will be roughly right in aggregate and wrong about every individual query, which is a recipe for a job that finishes on time in testing and overruns in production.

The zero-return rate is the other half of the same finding.

Nearly three in ten successful search calls returned no posts at all. On reply reads it was more than three in five, and on retweeter reads more than half. Every one of those was a 200 and every one was billed. A pipeline that measures "did the call succeed" rather than "did the call return anything" is blind to a third of its own search traffic.

Two operational rules follow.

Never stop a paging loop because a page came back short. On search, a short page is the normal case rather than the terminal one, and code that treats a page of three as the end of the result set will silently truncate the majority of queries it runs. Our pagination guide covers the cursor semantics in detail.

And record the row count separately from the status. The two answer different questions and only one of them is in the response code. The advanced search operators reference covers how much of the empty-page rate you can remove by making the query itself more precise, which is usually more effective than any amount of retry logic.

What a million posts actually costs

Because the firehose has no published price, the useful comparison is not against it. It is against what the alternative costs for the volume you actually need, and that number is a function of the endpoint you choose far more than of the rate you are quoted.

What a thousand posts costs, by the path that produced them

Read path	Rows per call	Calls per 1,000 rows	Cost per 1,000 rows	Cost per 1,000,000 rows	Source
user/tweets	19.04	53	$0.042	$42.02	derived
tweet/advanced_search	8.19	123	$0.098	$97.68	derived
tweet/replies	7.15	140	$0.112	$111.89	derived
tweet/detail	1.00	1,000	$0.800	$800.00	derived

as of 2026-09-02

Unit cost

Dollars per 1,000 posts, by read path

Point	Value ($)
Single post lookup	0.8 $
Reply reads	0.112 $
Advanced search	0.098 $
Timeline reads	0.042 $

Every one of those calls is billed at the same standard rate of 0.0008 dollars. The nineteen times spread in unit cost comes entirely from yield. A timeline read that returns 19.04 posts costs 4.2 cents per thousand posts. A search call that returns 8.19 costs 9.8 cents. A single post lookup, which returns exactly one object by definition, costs 80 cents per thousand.

That last row is the one that catches people, because a lookup feels like the cheapest possible call. It is the cheapest possible call and the most expensive possible way to acquire posts, and both statements are true at once. Anything you can reach through a paged read should be reached through a paged read, and the single-object lookup should be reserved for the cases where you have an identifier and need exactly that object.

Scaled to a million matched posts, the search path needs about 122,100 calls and bills about 97.68 dollars.

Worked example

One million matched posts, priced on per-call reads

Posts wanted: 1,000,000 · Search calls needed: 122,100 · Billed: $97.68

☑ 36,288 of those calls are expected to return nothing · ☑ The same million posts from timelines costs about $42.02 · ☑ The free signup credit of $0.50 covers about 5,120 posts this way · ☐ None of this proves you received every matching post

Roughly 36,288 of those calls are expected to return nothing, on the measured 29.72 percent zero-return rate. They are still billed, because a call that ran and found nothing consumed the same work as one that found twenty posts. That is not a quirk of one vendor's pricing model; it is what per-request billing means, and it is a strong argument for spending effort on query precision rather than on retry sophistication. Our cost of a million-tweet dataset works the same arithmetic from the dataset side, and the per-workload cost analysis does it per named job rather than per post.

Here is the sizing, written out, because a formula you can paste beats a table you have to interpolate.

STANDARD_RATE = 0.0008          # dollars per call

# Measured mean rows per successful call, twitterapis.com usage_logs,
# 2026-06-22 to 2026-09-02, n=2,451,925 calls with a recorded row count.
YIELD = {
    "user/followers":        49.80,
    "user/following":        40.65,
    "user/tweets":           19.04,
    "tweet/advanced_search":  8.19,
    "tweet/retweeters":       7.38,
    "tweet/replies":          7.15,
    "tweet/detail":           1.00,
}

def cost_for(rows_wanted: int, path: str) -> tuple[int, float]:
    """Calls needed and dollars billed for a target row count."""
    calls = rows_wanted / YIELD[path]
    return round(calls), calls * STANDARD_RATE

for path in ("user/tweets", "tweet/advanced_search", "tweet/detail"):
    calls, usd = cost_for(1_000_000, path)
    print(f"{path:24} {calls:>9,} calls  ${usd:,.2f}")

Run that and the ranking is obvious immediately, which is the point: the decision that moves your bill is which endpoint answers your question, not which vendor you buy it from. The pricing page carries the per-endpoint rates and the cost calculator will do this arithmetic against a stated volume if you would rather not.

One caveat on all of the above, stated plainly because it is the limit of what this data can support. These figures describe the cost of retrieving posts your query matched. They say nothing about what fraction of the posts that existed your query matched, and nothing in a per-call model can. Cost and coverage are separate axes here, and conflating them is the single most common error in vendor comparisons.

Which access model does your question actually need?

Almost every request for the firehose is a request for confidence, and confidence is available in several forms at very different prices. The model that fits follows from two things: whether your subjects are named, and whether a missed post costs you anything at all.

Which access model each common question actually needs

The question	Subjects	Needs completeness	Right model	Wrong model and why	Source
Track twelve competitor accounts	Named, few	No	Per-call timeline reads	A stream delivers millions of posts you discard	unknown
Measure share of voice for a category	Unnamed, many	No, a sample suffices	A designed sample	Query results are a filter, not a sample	unknown
Build a compliance archive	Named, all activity	Yes	Enterprise contract	Per-call reads cannot prove completeness	unknown
Alert within seconds of a mention	Named terms	No	Push events or a tight poll	A daily batch answers a different question	unknown
Reconstruct a deleted conversation	Named, historical	Impossible	Neither	No product returns what was removed	unknown

as of 2026-09-02

Work through the common cases and the pattern is consistent: the access model follows from whether your subjects are named and whether a miss costs you anything.

Tracking a set of named accounts. Competitor monitoring, portfolio tracking, executive listening. The subjects are named and there are usually fewer than a hundred of them. Per-call timeline reads are both cheaper and simpler, and a stream would deliver millions of posts you immediately discard. Our competitor analysis piece covers the build side of this in detail.

Measuring a rate across a category. Share of voice, sentiment distribution, adoption of a term. The subjects are unnamed and numerous, and the honest instrument is a designed sample rather than a complete corpus. This is the one case where a sampled stream is genuinely the right product, and also the case where most teams reach for query results instead and quietly compute a number that means nothing.

Building an archive for compliance. Every post from named accounts, retained, with a defensible claim that nothing is missing. This is the case the enterprise products exist for, and it is the one where per-call reads genuinely cannot substitute, because the claim you need is exactly the claim that cannot be constructed from the outside.

Alerting within seconds. A push mechanism or a tight poll, depending on how many subjects you have and how tight seconds needs to be. Our realtime mentions guide covers the polling design and its latency floor.

Reconstructing something that was deleted. No product does this. The correct answer is to say so early rather than to price it, and to record snapshots going forward if the requirement is real.

The platform has continued to add delivery surfaces since the tier change, including an event subscription product announced to paid developer tiers:

https://x.com/XDevelopers/status/1979341139833438693

A push surface changes the failure mode rather than removing it. Polling fails by missing what happened between two reads, which is a gap you can bound by shortening the interval. A subscription fails by dropping, which is a gap you cannot see at all unless you build a heartbeat and reconcile against a periodic read. Neither is better in the abstract. Choose the one whose failure you can actually detect, and then build the detection.

How to sample on purpose

If your question is about a population rather than about named subjects, you need a sample, and a sample you designed beats a sample you inherited from your query every time.

Five rules cover almost every case.

Sample the subjects, not the posts. A random tenth of accounts drawn from a defined frame is defensible. A random tenth of the pages your query returned is not, because the pages were already selected by the query. This is the single most common methodological error in social data analysis and it survives peer review regularly.

Fix the window before you look. A window that moves while you collect turns a sample into a convenience sample, and the direction of the bias depends on when your job happened to run. Decide the window, write it down, and treat a change to it as a new study rather than as a refresh.

Record the sampling rule alongside the data. A rate you cannot reconstruct is a rate you cannot defend. Store the frame, the selection rule, the seed and the window in the same place as the rows, because in six months the only person who remembers any of it will be nobody.

Measure your own miss rate. Pick a handful of subjects, walk them exhaustively, then run your normal pipeline against the same subjects and count the difference. That number is the closest thing to a coverage measurement available on public reads, and it is worth re-running whenever the pipeline changes.

Never infer a population rate from a query. Repeating for emphasis, because it is the one that costs the most. A query is a filter. Filters have no sampling properties.

Here is the frame construction as code, since the rule that gets skipped is usually the first one.

import hashlib

def in_sample(subject_id: str, rate: float, seed: str) -> bool:
    """Deterministic subject-level sampling.

    Hashing the subject id means the SAME subjects stay in the sample
    across runs, which is what makes a longitudinal comparison valid.
    Re-randomising per run gives you a different population every day
    and a trend line that measures your sampler.
    """
    h = hashlib.sha256(f"{seed}:{subject_id}".encode()).digest()
    bucket = int.from_bytes(h[:8], "big") / float(1 << 64)
    return bucket < rate

# The frame, the rate and the seed all get stored WITH the rows.
SAMPLE = {"frame": "accounts_active_2026_q3", "rate": 0.10,
          "seed": "2026-09-02-firehose-post", "window": "2026-09-01/2026-09-30"}

The seed matters more than it looks. Hashing the subject identifier with a fixed seed keeps the same subjects in the sample across every run, which is what makes a comparison between two months a comparison of the world rather than a comparison of two different samples. Re-drawing at random each run produces a trend line that measures your sampler.

When do you genuinely need enterprise access?

There is a real set of cases where an enterprise contract is genuinely the right instrument, and it is a good deal smaller than the number of people who ask for one. Four conditions cover almost all of them, and each is testable before any sales conversation starts.

You need it when a missed post is a compliance failure rather than an inconvenience. Regulated archival, legal hold, anything where an auditor will ask whether the record is complete and "we retrieved what our query matched" is not an acceptable answer.

You need it when the question is genuinely platform-wide and the accuracy requirement is tight enough that a sample will not do. That is rarer than it sounds, because most platform-wide questions have error bars wide enough that a designed sample answers them, and the ones that do not usually turn out to be about a named subset after all.

You need it when latency is measured in seconds across an unbounded subject set. A poll over a few hundred named subjects can be fast. A poll over an unbounded term space cannot.

And you need it when the volume is high enough that per-call economics stop being favourable. That crossover is arithmetic rather than opinion: take your expected monthly record count, divide by the yield of the endpoint that would serve it, multiply by the per-call rate, and compare. At the measured search yield, ten million posts a month is under a thousand dollars, which is well below any enterprise contract we have ever heard described. The crossover is a long way up.

Outside those four cases, the honest advice is that per-call reads are not a compromise. They are the correct instrument, and the enterprise product would deliver a great deal of data you would immediately throw away. Our choosing guide covers the vendor question, and the alternatives comparison covers the landscape as it currently stands.

Replacing a stream with polls, without breaking the analysis

Teams that had a stream and now have per-call reads usually port the pipeline in the wrong order. They keep the consumer, replace the transport, and discover months later that the numbers moved. The consumer is the part that has to change, because a stream and a poll deliver different guarantees and the code downstream was written against the stronger one.

Three differences drive everything else.

A stream pushes each item once, so deduplication is a defensive measure against reconnects. A poll re-reads a window, so deduplication is load-bearing: without it every item is counted as many times as your interval overlaps. That sounds obvious and it is still the most common porting bug, because the duplicate rate is low enough during testing to look like noise and high enough in production to move a monthly total by double digits.

A stream has an obvious failure: the connection drops. A poll has an invisible one: the interval was longer than the window the endpoint returns, so items fell between two reads and nothing anywhere records it. The stream failure pages somebody. The poll failure produces a slightly lower number.

And a stream arrives in event order, while a poll arrives in page order, which is usually reverse chronological. Any consumer that assumed monotonic arrival needs rewriting, and the assumption is often implicit rather than written down.

The shape that handles all three is small.

from dataclasses import dataclass, field

@dataclass
class Watermark:
    """Per-subject state. The whole difference between a poll that is
    correct and one that merely runs."""
    newest_seen_id: str | None = None
    last_read_at: float | None = None
    ids_in_flight: set[str] = field(default_factory=set)

def poll_once(fetch, subject, wm: Watermark, overlap_pages: int = 1):
    """Read forward from the watermark, with a deliberate overlap.

    The overlap is not paranoia. Page boundaries move as new items
    arrive, so a read that starts exactly at the watermark can skip an
    item that shifted across the boundary between two runs.
    """
    fresh, pages, cursor, caught_up = [], 0, None, False
    while True:
        page = fetch(subject, cursor=cursor)
        pages += 1
        for item in page.items:
            if wm.newest_seen_id and item.id <= wm.newest_seen_id:
                caught_up = True
            elif item.id not in wm.ids_in_flight:
                fresh.append(item)
        if caught_up and pages > overlap_pages:
            break
        if not page.next_cursor:
            break
        cursor = page.next_cursor
    if fresh:
        wm.newest_seen_id = max(i.id for i in fresh)
    wm.ids_in_flight = {i.id for i in fresh[-200:]}
    return fresh, {"pages": pages, "caught_up": caught_up}

The caught_up flag is the one to alert on. A poll that never catches up, run after run, is a poll whose interval is longer than the subject's posting rate, and the gap it is leaving does not appear anywhere else. A poll that catches up on the first page every time is a poll running more often than it needs to, which is a cost finding rather than a correctness one.

The interval itself should be derived rather than chosen. Take the subject's observed posting rate, take the page size the endpoint returns, and set the interval so that a full page never accumulates between two reads. For a user timeline returning 19.04 posts per call at the measured mean, a subject posting five times an hour can be polled every two hours with room to spare, and a subject posting fifty times an hour cannot be polled hourly without risk. One interval for every subject is the assumption that produces silent gaps on exactly the subjects that matter most.

What changed twice in the platform's own pricing

Anybody comparing access models in 2026 is reading a landscape that has moved twice in three years, and a surprising amount of published comparison content describes a model that no longer exists.

The first move was the end of free access, announced in February 2023 and executed over the following weeks. The second was the tier structure that replaced it in March 2023, with named self-serve levels. The third state, current as of this writing, is pay-per-use pricing with credits, where the unit is a resource or a request and the rate varies by endpoint rather than by named plan.

That progression matters for a practical reason beyond accuracy. A named-tier model and a per-use model have different failure shapes for a buyer. Under tiers, the risk is buying the wrong tier and either paying for headroom you do not use or hitting a ceiling mid-month. Under per-use, the risk is a job with no ceiling on it, because nothing structural stops the spend. The mitigations are different: tier shopping in the first case, and a hard per-run call budget in the second.

It also means any comparison article, including this one, has a shelf life. The durable content here is the method, which is to price your own workload against a measured yield rather than to compare headline rates. The specific numbers are true on the date at the top of this post and should be re-derived rather than inherited. Our rate card walkthrough carries the current per-endpoint pricing on this API, and the platform rate limit reference is the first-party source for the ceilings on theirs.

There is a second-order effect worth naming for anyone budgeting a research project rather than a product. Under an application-based access model, the cost of a study was measured in review time. Under a metered model it is measured in money, and money is allocated by institution rather than by merit. That does not make the data worse. It changes who can collect it, which is a different problem and one that a methods section cannot solve.

Writing a methods section that survives contact with a reviewer

If the output of your collection is a paper, a report, or anything a stranger will audit, the access model has to be in the write-up. This is the part that used to be a single sentence naming the API and can no longer be.

Six things belong in it, and all six are cheap to record at collection time and impossible to reconstruct afterwards.

The access model. Per-call reads, a sampled stream, an enterprise contract. Two studies that both say they used the platform's API can now mean entirely different things, and the reader cannot tell which without being told.

The exact query. Not a description of the query, the query. Operator syntax, filters, language restrictions, everything. A paraphrase is not reproducible, and the advanced search operators each change the result set in ways that a summary sentence hides.

The retrieval window. Both the window the query covered and the wall-clock period during which collection ran. Those are different and both matter, because engagement counts and account visibility both move while you collect.

The sampling rule, if any. Frame, rate, seed. If there was no sampling, say that explicitly rather than leaving it to inference, because the absence of a statement reads as the absence of a rule.

The stopping rule. Did the walk exhaust the cursor, or did it hit a budget? This is where the truncation flag earns its keep, and a study that cannot answer it has an unknown-sized hole in its corpus.

The yield and the empty rate. How many calls, how many records, how many calls returned nothing. Those three numbers let a reader sanity check the corpus size against the claim, and they cost nothing to record.

The sampling comparison literature is the standard reference for why the fourth and sixth of those matter, and the practical upshot is that a corpus described only by its size is a corpus nobody can evaluate. Two million posts collected by an exhaustive walk over named subjects and two million posts collected by a query that hit a page budget are not the same evidence, and they look identical in a table.

The full measured picture behind every yield figure in this post, in one place, so the arithmetic can be checked rather than taken on trust. Timeline reads: 630,228 calls measured, mean 19.04 posts, median 20, largest observed page 33, and 4,659 calls returning nothing. Advanced search: 562,238 calls, mean 8.19, median 3, largest page 20, and 167,107 calls returning nothing. Follower reads: 209,792 calls, mean 49.80, median 50, largest page 70, and 435 empty. Following reads: 15,218 calls, mean 40.65, median 49, and 631 empty. Retweeter reads: 74,815 calls, mean 7.38, median 0, largest page 93, and 39,174 empty. Reply reads: 17,077 calls, mean 7.15, median 0, largest page 41, and 10,620 empty. Single post lookups: 883,520 calls, exactly 1 object every time. Account lookups: 59,037 calls, also exactly 1. Together those eight paths are 2,451,925 of the 5,155,699 calls in the window, or 47.6 percent of it, and they are the subset that carries a recorded row count at all.

Three of those numbers deserve to be read together rather than separately. The largest observed search page is 20 while the mean is 8.19, which means the typical page is 41 percent full. The largest observed timeline page is 33 while the mean is 19.04, which is 58 percent. And a follower page averages 49.80 against a largest observed 70, or 71 percent. Page fill is not a constant, it is not published anywhere, and it moves your unit cost by a factor of about 1.7 across those three paths alone.

Do you need an archive or a stream?

Half the requests for a firehose turn out to be requests for history rather than for realtime, and the two have opposite cost shapes. Realtime wants a narrow filter run often. Archive wants a wide pull run once. A product that tries to serve both with one pipeline ends up paying realtime frequency on archive breadth, which is the most expensive quadrant available.

Separate them explicitly, because the separation changes the design at every level.

A backfill is a bounded job. It has a start date, an end date and a subject list, and when it finishes it is done. It should be budgeted as a one-off, it should run at whatever concurrency the rate ceiling allows, and it should write a completion record per subject so that a resumed run does not re-walk what it already has.

An increment is an unbounded job. It runs forever, it reads forward from a watermark, and its cost per run should be roughly constant regardless of how long the archive has been accumulating. If your incremental cost grows over time, something in the pipeline is re-walking history, and that is the single most common way a data bill quietly triples.

The two need separate budgets in code, not just in spirit, because they have completely different acceptable truncation behaviour. A backfill that truncates has a hole in a fixed historical window and you know exactly where it is. An increment that truncates has a hole that moves, and it will keep moving until somebody notices the flag.

def backfill(fetch, subject, since, until, page_budget=500):
    """Bounded historical pull. Runs once per subject, records how it
    ended, and writes a watermark the increment can read forward from."""
    rows, cursor, pages, truncated = [], None, 0, False
    while True:
        page = fetch(subject, cursor=cursor, since=since, until=until)
        rows.extend(page.items)
        pages += 1
        if not page.next_cursor:
            break
        if pages >= page_budget:
            truncated = True
            break
        cursor = page.next_cursor
    return rows, {
        "mode": "backfill",
        "subject": subject,
        "window": [since, until],
        "pages": pages,
        "rows": len(rows),
        "truncated": truncated,           # a KNOWN hole, in a KNOWN window
        "watermark": max((r.id for r in rows), default=None),
    }

Note the two things that record ends with. The truncation flag says whether this subject's history is complete as far as this job is concerned, and the watermark hands the increment a starting point. A backfill that returns rows without those two fields has done the expensive half of the work and thrown away the part that makes it reusable.

There is a third mode people forget, which is the repair. Subjects go private and come back. Accounts get renamed. A backfill that ran in June against a subject that was briefly protected has a hole nothing will ever fill unless somebody re-runs it. Schedule a periodic re-walk of a random subset of subjects, compare against what you already hold, and count the difference. That count is the closest thing to a coverage metric available on public reads, and it is worth more than any vendor's completeness claim because you measured it yourself. Our reliability measurements cover the failure side of the same instrumentation.

Estimating what you missed

Nobody can tell you your miss rate on a public read API, because the only system that knows what it withheld is the one that withheld it. You can estimate it, and an estimate with a stated method beats a completeness claim with no method behind it.

The technique is a recapture comparison, borrowed from ecology and simple enough to run in an afternoon. Take a sample of subjects. Walk each one exhaustively, with a generous page budget and no time pressure, and treat that as the reference. Then run your normal production pipeline against the same subjects over the same window. The difference between the two is your pipeline's miss rate on that subject class.

def miss_rate(reference: set[str], production: set[str]) -> dict:
    """Compare an exhaustive reference walk against the production pull.

    `reference` is not ground truth. It is a MORE thorough read, so the
    result is a lower bound on the miss rate, and it should be reported
    as one rather than as a coverage figure.
    """
    missed = reference - production
    extra = production - reference        # nonzero means the window moved
    return {
        "reference_rows": len(reference),
        "production_rows": len(production),
        "missed": len(missed),
        "miss_rate_lower_bound": len(missed) / max(len(reference), 1),
        "unexpected_extra": len(extra),
        "note": "reference is a thorough read, not ground truth",
    }

Two things about that function are deliberate and both are the honest part.

It returns a lower bound rather than a miss rate, because the reference walk is itself a public read and shares every blind spot the production pipeline has. Anything invisible to both is invisible to the comparison. Reporting it as a coverage figure would be claiming more than the method supports.

And it reports unexpected_extra rather than discarding it. Rows that production found and the reference did not mean the window moved between the two runs, which is a signal about your window definition rather than about coverage. A nonzero extra count on a supposedly fixed window is a bug in the harness, and finding it here is much cheaper than finding it in a chart.

Run this once per subject class rather than once overall. The miss rate on a low-volume account and on an account posting fifty times an hour are different numbers driven by different mechanisms, and averaging them produces a figure that describes neither.

A worked example: share of voice, done honestly

Share of voice is the single most requested platform-wide metric and the one most often computed wrongly, so it is worth walking end to end. The wrong version is fast, produces a number that moves, and measures the queries rather than the market it claims to describe.

The wrong version takes four brand terms, runs a search for each, counts the results, and divides. It produces a number, the number moves week to week, and somebody puts it on a slide. Every step of that is defensible in isolation and the result measures the queries rather than the market.

Three things are wrong with it.

The counts are not comparable across terms, because each query has its own match rate and its own empty-page rate. A term that appears inside other words matches more. A term with a common homograph matches things that are not about the brand at all. Nothing in the arithmetic corrects for either.

The counts are truncated at different points. A high-volume term hits your page budget and a low-volume one does not, so the ratio between them is partly a ratio of your own budgets. This is the failure that makes share of voice appear to move when nothing in the world changed.

And the denominator does not exist. Share of what? The set of posts your four queries matched is not a market, and no amount of care in the numerator fixes a denominator that was never defined.

The honest version changes the unit from posts to subjects and defines the frame first.

Build a subject frame: a defined set of accounts, constructed from a rule you can write down, such as accounts that posted in a named category during a named month. Sample from that frame deterministically, using the hash-and-seed approach above so that the same accounts stay in the sample across periods. Read each sampled account's timeline over a fixed window. Classify each post for brand mention using the same classifier for every brand. Then compute the share across the sample and report it with the frame, the rate, the seed and the window attached.

That version is more work and it produces a number that means something, moves for real reasons, and can be defended when somebody asks how it was calculated. It also costs less, because reading a few thousand sampled timelines at 19.04 posts per call is cheaper than exhaustively searching four high-volume terms, which is the pleasant surprise at the end of nearly every one of these redesigns.

The pattern generalises to most platform-wide questions. Whenever you find yourself wanting every post, ask what population the question is about, whether you can enumerate a frame for it, and whether a sample of that frame would answer it. If the answer to the last two is yes, you have just converted an enterprise data problem into a per-call one, and the conversion is usually available. Our use-case guide covers the shapes this takes across common projects.

What to ask a vendor, in writing

Procurement conversations about social data go wrong in a specific way: the buyer asks about completeness and the seller answers about delivery, and both parties leave the call believing they agreed on something.

Five questions, asked in writing, resolve it.

What share of matching items does this carry, and how is that share determined? The acceptable answers are all of them, a stated random fraction, or an unknown fraction. The third is a legitimate answer for a per-call product and an unacceptable one for anything sold as complete.

Is that share contractual or best effort? A number in marketing copy and a number in a contract are different products at different prices.

What does the interface report when it does not return everything? If the honest answer is nothing, that is fine and it should be written down, because it tells you exactly how much of the completeness work is yours.

What is the unit of billing, and what is the expected yield per unit? A per-call rate without a yield figure is not a price. The nineteen times spread measured on this API across four read paths is entirely a yield effect on a single flat rate.

What happens to items that were deleted, made private, or edited between two reads? Every one of those is a real, common event, and a vendor who has not thought about them has not run a production archive.

None of those five are adversarial and all five are answerable in a paragraph by anyone who runs the product. A vendor who answers all five plainly is telling you what you are buying. A vendor who answers the second question when you asked the first is not necessarily being dishonest, but you should ask again.

The six questions that settle it

Every version of this conversation resolves into the same short list of six questions, and it is worth running through them before any procurement discussion rather than during one, because the answers usually settle the access model without anybody having to ask a vendor anything.

Do you need every post, or a defensible estimate? Only one of those needs a stream, and most stated requirements for the first turn out to be the second once somebody asks what the number is for.

Is your subject set named, or is it the whole platform? Named subjects are cheap on per-call reads and get cheaper as you name them more precisely.

How stale can the answer be? Seconds is a subscription. Minutes is a poll. Hours is a batch, and a batch is an order of magnitude simpler than either.

Does a missed post cost you anything? If the honest answer is no, sample deliberately and write down that you did. Half the cost of complete data is paid by people who never needed it and were never asked.

Do you need history, or only what happens next? Backfill once, store a watermark, then read incrementally. A pipeline that re-walks history to discover new posts is paying the most expensive access pattern available to answer the cheapest question. Our history scraping guide covers the watermark pattern, and the date-range search guide covers bounding the backfill.

What would you do differently with ten times the data? If the answer is nothing, you have found your budget. This question ends more procurement conversations than any other on the list.

For a first-party walkthrough of what the current API surface offers, the platform's own developer channel is the place to start:

https://www.youtube.com/watch?v=VOWgSRfToAI

Five things people get wrong about the old model

Because the firehose stopped being purchasable before most people currently working with social data started, a lot of what circulates about it is folklore. Five items come up often enough to correct.

It was never free, and it was never cheap. The firehose was an enterprise product sold by contract for the whole of its commercial life. The free access that ended in 2023 was the public API, which was a different product with different guarantees. Conflating the two produces the widely repeated but wrong story that the platform took away a free complete feed, when what it took away was a free incomplete one.

Sampling was always part of the product family. The Decahose is not a post-2023 austerity measure. A sampled tier existed alongside the full stream for years, because a random sample is the correct instrument for a large class of questions and the customers who bought it knew that. Reading the sample tier as a downgrade misreads what it was for.

Access was already narrow before it narrowed. Full-stream resale ran through a small set of authorised partners rather than being generally available, which is why the acquisition of one of them in 2014 was a significant enough event to reshape the market. The number of organisations who ever held real firehose access was small, and it was small for the whole period people now describe as open.

The public streaming API was not a small firehose. It was a sampled stream with a documented but modest share, and the difference between it and the full stream is the entire subject of the sampling literature. Anybody who built on the public stream and now says they lost the firehose lost something real and it was not that.

Nothing about the old model solved deletion. A complete stream captures what was posted. It does not tell you what was later removed, and a record that a post existed is not the same as permission to keep serving it. Every archive built on any access model has had to solve retention and deletion compliance separately, and that has not changed.

The reason to be precise about this history is not pedantry. It is that a team planning around a remembered version of the past will price a project against a product that did not exist in the form they remember, and the correction usually arrives after the budget is approved.

What the yield numbers imply for a research budget

For anybody sizing a study rather than a product, the measured yields translate into a budget in about four lines of arithmetic, and it is worth doing before the grant application rather than after.

Start with the corpus you need, expressed in records rather than in posts, because a record is what you are billed for retrieving. Then pick the endpoint that will produce it, because that decision is worth nineteen times. Then divide by the measured yield of that endpoint and multiply by the per-call rate.

A study reading the full timelines of two thousand sampled accounts over a three-month window is a good worked example. If each account posted a hundred times in the window, that is 200,000 posts. At the measured user timeline yield of 19.04 posts per call, that is about 10,504 calls, and at the standard rate it bills about 8.40 dollars. The whole corpus costs less than a single article processing charge.

Change one decision and the number moves sharply. If the same 200,000 posts are collected by searching for terms rather than by reading timelines, the search yield of 8.19 applies instead, the call count rises to about 24,420, and the bill rises to about 19.54 dollars. Still small, and now more than double, entirely because of an endpoint choice that most methods sections do not record.

Scale it up and the same ratio holds. Two million posts by timeline reads is about 84.03 dollars. The same two million by search is about 195.36 dollars. Nothing about the rate changed. The yield did.

Two caveats belong beside those figures. Neither includes the hydration calls a study usually needs on top of the posts themselves, such as resolving author profiles, and those are per-subject rather than paged so they do not amortise. And none of it includes the cost of the empty calls, which on search is nearly three in ten and is already inside the measured mean rather than added on top. Our cost by workload analysis walks through the hydration fan-out in detail, and the dataset cost piece does the same arithmetic for a fixed corpus size.

The broader point for a research budget is that the collection cost is now small enough that it is rarely the binding constraint, and the binding constraint has moved to method. It is much easier to spend a thousand dollars collecting the wrong corpus than it is to spend a thousand dollars collecting the right one, and no access model protects against that.

The completeness conversation, one more time

Everything in this post converges on a single distinction that is easy to state and hard to keep hold of during a procurement conversation: there is what you retrieved, and there is what existed, and only the first of those is measurable on a public read API.

There is what you retrieved, and there is what existed. On a public per-call read API you can measure the first precisely and you can say nothing rigorous about the second. Every technique in this post is a way of making that gap visible and bounded rather than pretending it is not there: walk cursors to exhaustion, record row counts, keep watermarks, flag truncation, sample deliberately, and measure your own miss rate against a more thorough read.

None of that closes the gap. It documents it, which is the honest available option and is worth considerably more than a completeness claim nobody can check.

The firehose closed that gap by contract. That is what it was for, that is why it was expensive, and that is why the small set of use cases that genuinely need a contractual guarantee still go to an enterprise product. For everybody else, and it really is almost everybody, the question is not how to get complete data. It is how to describe honestly what you have, and how to make sure the thing you are missing is not the thing you are measuring.

The short version

The firehose was every public post in realtime, unsampled, sold by contract. It was resold by Gnip until the platform acquired Gnip in April 2014, and the enterprise family around it still exists as a filtered full stream and a ten percent random sample. What ended in 2023 was the free and low-cost access that most developers and researchers used, and what replaced it for self-serve buyers is per-request reads with per-request billing.

What that trade actually costs is measurable, and the measurement is not the rate card. At the standard read rate and a measured search yield of 8.19 posts per call, one million matched posts bills about 97.68 dollars, with roughly 36,288 of the 122,100 calls returning nothing. Timeline reads do the same million for about 42.02 dollars because they return 19.04 posts per call. The endpoint you choose moves the bill by nineteen times on identical pricing.

What that trade cannot buy at any price is a completeness guarantee. There is no external way to verify that you received everything, so the honest posture is to walk cursors to exhaustion, record row counts, keep watermarks, flag truncation, and describe your coverage as what you retrieved rather than as what existed. Anybody who tells you otherwise about a public read API is describing a product they do not have.

If your subjects are named, none of this is a compromise and you should stop looking for a stream. If your question is about a population, design a sample and say so in your methods. And if you genuinely need contractual completeness, that is a sales conversation about an enterprise product rather than an API key, and it is worth knowing that before you spend a quarter building around the assumption that a self-serve key will get you there.

You can measure the yields in this post against your own queries on the free signup credit, which is fifty cents and covers about 625 standard calls without a card. Point it at the queries you actually care about, log the row count on every call, and you will know your own numbers rather than ours. The getting started guide covers the first call, the endpoint reference lists what each one returns, the rate limit guide covers the ceilings, the sentiment walkthrough covers a full pipeline, the ideas piece covers what people build with it, the per-user cost analysis covers the seat-based view of the same arithmetic, and the rate card covers the pricing itself. When you are ready, sign up and the credit is already there.