How do you get tweets with Python?

The whole client in about fifteen lines

Import os and requests, read the key out of os.environ, and hang one header on a Session. The client is a base URL constant, a Session with Authorization set to a Bearer value, and a get function that joins the base to /twitter/ plus the docs path, passes your keyword arguments as params, sets timeout to 30 seconds, calls raise_for_status and returns res.json(). That is the entire thing. Every read endpoint in the catalog is reachable through it, because the only difference between one route and the next is the path string and the parameter names. Resolving a handle is get with user/info and username; a search is get with tweet/advanced_search and query.

Why a Session rather than bare requests.get

A Session holds the header, so authentication is configured in one place instead of being restated at every call site, and it keeps the underlying TCP connection alive across calls, which matters once a paging loop makes hundreds of them. It is also the object you mount a retry adapter on later, so starting with a Session avoids a rewrite the first time you need backoff. Reading the key from the environment rather than pasting it into the file deserves the same discipline: a key committed to a repository has to be rotated, and one pasted into a notebook cell tends to survive in the saved output long after you have forgotten about it.

What the helper sends on the wire

Each call becomes a GET to the base host, then /twitter/, then the path exactly as the docs spell it, with your keyword arguments serialized into the query string. The header is Authorization with a Bearer prefix and the key; x-api-key carrying that same value is accepted as an alternative if it suits your gateway better. There is no signing step, no token exchange and nothing to refresh, so the request a script sends on its first day is the request it sends a year later. timeout at 30 seconds stops a hung connection stalling a job silently, and raise_for_status turns a 4xx or 5xx into an exception rather than a dict you would parse by mistake.

Reading a profile and a page of search results

get('user/info', username='naval') returns a dict with a single key, user, so the id and follower count read as profile['user']['id'] and profile['user']['followers_count']. A search is get('tweet/advanced_search', query='from:naval min_faves:500'), which comes back with tweets, count and next_cursor, and you iterate page['tweets'] directly. The optional product parameter picks the ranking, one of Latest, Top, People, Photos or Videos, defaulting to Latest. A page holds roughly 20 results, and the endpoint notes say to page with the cursor rather than trying to raise the page size, so asking for a bigger response is not the way to fetch more. Operators inside the query are evaluated server-side exactly as they are on the advanced search page, so from:, since:, until:, min_faves: and lang: all behave the way you already expect them to.

A cursor loop as a generator

Wrapping paging in a generator keeps cursor handling in one place and lets the caller write an ordinary for loop. Start with cursor set to None, put it into params once it exists, call get, yield the page, then read page.get('next_cursor') and return when it is missing. Iterating that over user/followers and summing len(page['followers']) totals a follower list without the caller ever seeing a cursor. One caveat belongs inside the loop rather than in a comment: follower-graph routes return a non-null cursor even on the final page, so a stop condition that only checks for a missing cursor will spin there forever. Break on an empty collection too. Each page the generator yields is one billed call, so put a page limit around the loop while you are still testing it.

Retries that do not spend money

Mount an HTTPAdapter on the session carrying a urllib3 Retry with total set to 3, backoff_factor at 0.5, status_forcelist listing 500, 502, 503 and 504, and allowed_methods restricted to GET. That retries server errors and nothing else. Never retry a 4xx: a 401 is a rejected key and a 404 is an account that does not exist, and repeating either spends again without changing the answer. Because billing happens per call, a retry policy is a spending policy, which is the argument for keeping that forcelist to exactly those four codes rather than reaching for something broader when a job starts failing. backoff_factor at 0.5 makes the waits grow between attempts instead of hammering a struggling server at a fixed interval.

Handling a 429 without guessing

One ceiling covers every route on a key, 600 requests a minute with 20 in flight at once, and crossing either returns 429 carrying a Retry-After header that says how long to wait. Sleep for that value and resume rather than inventing a backoff of your own, since the header is the only figure reflecting the real window. Keep the 429 branch separate from the 5xx retry adapter, which does not include that code. The neighbouring statuses each mean something specific and none of them is transient: 401 says the key is wrong, missing or revoked, and 403 says the route is not enabled for that key. Both ceilings apply per key rather than per route, so spreading a job across several endpoints buys no extra headroom.

Writing pages to JSON or CSV

The decoded body is already a dict, so json.dump on a page writes it to a file with no reshaping and preserves every field the API returned. CSV is the lossy option: hand csv.DictWriter an explicit field list, because a tweet object carries nested keys such as author that a flat table cannot hold, and a media post carries an extended_entities block that certainly cannot. Accumulate pages in memory and write once at the end rather than reopening the file inside the loop, and keep every id column as text, since a spreadsheet will happily round a 19-digit id into a different number. created_at is a string in Twitter's own format rather than ISO, so whoever reads the CSV needs that parse rule alongside the file.

Sizing a bulk job

Twenty concurrent workers under a 600 per minute ceiling is the fastest safe shape, and a ThreadPoolExecutor sized to twenty sits exactly on the concurrency limit. Going wider produces rejections rather than throughput. On cost, the modelled figure is 20 records per call, which is $0.04 per 1,000 records at a full page, but 20 is the default page size requested upstream, a ceiling rather than a yield. Measured across 396,817 successful calls, timeline reads averaged 18.78 records and search averaged 7.62, with 29.5 percent of searches returning none. Size a search-heavy budget from the lower number, because a short page bills exactly what a full one bills. The $0.50 of signup credit covers 625 calls, which is roughly 12,500 records if every page comes back full. We bill $0.0008 a call, per our public pricing.

What each status means to a Python client

Questions and answers

Do I need a Python SDK or a pip package for this?

Only requests itself. There is no vendor package to add, because the client is a Session with one header plus a function that builds a URL and decodes JSON. The credential is static and nothing refreshes it, so attaching the header once means every later call is already authenticated. The client published on the Python page is complete rather than illustrative and runs exactly as printed.

Which header does the Python client send?

Authorization with a Bearer prefix and your key, read from os.environ so it never sits in the file. The API also accepts x-api-key carrying the same value if that fits your stack better. Either header works on every route, and there is no handshake, no callback URL and no app registration behind it, because your code is calling this API rather than calling X directly.

How do I move a script off Tweepy?

Replace the handful of Tweepy calls the script actually uses with the get helper. You cannot repoint a base URL and be finished, because Tweepy wraps the official API and both the endpoint names and the response shapes differ from what you get here. In practice it is a smaller job than the word migration implies, since most scripts touch two or three operations and the rest is your own code.

Should I use asyncio for this?

Only when you are fetching many independent resources at once. For a sequential job the network is the bottleneck rather than the client, so requests stays simpler and easier to debug. If you do want concurrency, httpx with an AsyncClient accepts the same header and the same URLs, so the helper converts almost line for line and the paging generator becomes an async generator with no logic change.

How many requests can a Python job make at once?

Twenty in flight, under a ceiling of 600 a minute on one key. A worker pool sized to twenty sits exactly at that limit and is usually the fastest safe configuration for a bulk read. Wider pools do not go faster, they generate rejections you then have to handle, and each of those is a round trip you waited on and got nothing back from.

How do I keep the key out of my source files?

Read it from the environment the way the published client does with os.environ, and for a scheduled job put it in whatever secret store your runner already provides rather than in the job definition. A key pasted into a notebook cell tends to survive in the saved output, which is one of the easier ways to leak one without ever noticing it happened.

Which errors should the retry adapter cover?

The four server codes in the status_forcelist and nothing beyond them: 500, 502, 503 and 504, with total set to 3 and allowed_methods limited to GET. A client error is a statement about your request, so repeating it changes nothing and bills again. The 401 branch is worth handling explicitly, exiting with a message that names the key variable rather than letting a stack trace bury the cause.

What should the 429 branch do?

Read the Retry-After header, sleep exactly that long, then resume from the same cursor you were on. Do not fold 429 into the adapter's forcelist, because urllib3 will then back off on its own schedule rather than the one the response asked for. If you hit it regularly the fix is upstream: lower the worker count toward twenty, or space the loop out.

How many tweets does one call actually return?

Twenty is the default page size and it is a ceiling rather than a promise. Measured across 396,817 successful calls, a timeline read averaged 18.78 records and a search averaged 7.62, with almost thirty percent of searches returning nothing at all. A short page costs what a full one costs, so a search-heavy job pays more per record than the headline per-thousand figure suggests.

Why does the helper set a timeout?

Because a request with no timeout can hang until something kills the process, and a scheduled job that hangs looks exactly like one that is merely slow. Thirty seconds is what the published client uses. Pair it with raise_for_status so a 4xx or 5xx becomes an exception at the call site instead of a dict your parsing code walks straight into and misreads.

Keep reading

• Python client
• Language clients
• Node client
• Twitter Search API
• How does Twitter API pagination work?
• The JSON shape you will be parsing