Python Friday #114: Debug Tweepy

Tweepy is very good at abstracting the Twitter API. But what can we do if Tweepy stops working from one day to another?

This post is part of my journey to learn Python. You can find the other parts of this series here. You find the code for this post in my PythonFriday repository on GitHub.

 

An annoying problem appears

A few weeks back I could use my Tweepy script and add the people I follow to a list. As I tried it with a new script I wrote for Python Friday, I run into these messages:

Rate limit exceeded. Sleeping for 901 seconds
Rate limit exceeded. Sleeping for 902 seconds
Rate limit exceeded. Sleeping for 901 seconds

Tweepy created the list, but it did not add any people. It fetched the friends and then it hit the rate limit. After 15 minutes it hit the rate limit again. And again.

The console had no other output, and I could not run the Fiddler proxy to see what requests where send to Twitter. How could we figure out what is going on?

 

Activate the logger

As explained in my Python Friday post #62, Python has a built-in logger. As it turns out, Tweepy uses the same logger, and we can activate it with this code:

If we now run a script with Tweepy, we get a verbose log of everything that happens in the background:

DEBUG:requests_oauthlib.oauth1_auth:Updated url: https://api.twitter.com/2/lists/1499844349756121093/members
DEBUG:requests_oauthlib.oauth1_auth:Updated headers: {‘User-Agent’: ‘Python/3.10.2 Requests/2.27.1 Tweepy/4.6.0’, ‘Accept-Encoding’: ‘gzip, deflate’, ‘Accept’: ‘*/*’, ‘Connection’: ‘keep-alive’, ‘Cookie’: ‘guest_id=*****, ‘Content-Length’: ’34’, ‘Content-Type’: ‘application/json’, ‘Authorization’: ‘OAuth oauth_nonce=”*****”, oauth_timestamp=”*****”, oauth_version=”1.0″, oauth_signature_method=”HMAC-SHA1″, oauth_consumer_key=”*****”, oauth_token=”******”, oauth_signature=”*******”‘}
DEBUG:requests_oauthlib.oauth1_auth:Updated body: b'{“user_id”: “******”}’
DEBUG:urllib3.connectionpool:https://api.twitter.com:443 “POST /2/lists/1499844349756121093/members HTTP/1.1” 429 94
DEBUG:tweepy.client:Received API response: 429 Too Many Requests
Headers: {‘date’: ‘Fri, 04 Mar 2022 20:28:54 UTC’, ‘server’: ‘tsa_f’, ‘api-version’: ‘2.36’, ‘content-type’: ‘application/json; charset=utf-8’, ‘cache-control’: ‘no-cache, no-store, max-age=0’, ‘content-length’: ’94’, ‘x-access-level’: ‘read-write’, ‘x-frame-options’: ‘SAMEORIGIN’, ‘content-encoding’: ‘gzip’, ‘x-xss-protection’: ‘0’, ‘x-rate-limit-limit’: ‘300’, ‘x-rate-limit-reset’: ‘1646426162’, ‘content-disposition’: ‘attachment; filename=json.json’, ‘x-content-type-options’: ‘nosniff’, ‘x-rate-limit-remaining’: ‘298’, ‘strict-transport-security’: ‘max-age=631138519’, ‘x-response-time’: ‘177’, ‘x-connection-hash’: ‘*****’}
Content: b'{“title”:”Too Many Requests”,”detail”:”Too Many Requests“,”type”:”about:blank”,“status”:429}’
WARNING:tweepy.client:Rate limit exceeded. Sleeping for 901 seconds.

I could see that Tweepy called the right Twitter API endpoint with the right data in the request. Everything looked good on my end, which led me to take a closer look at the Twitter API.

 

The site-wide limits of Twitter

After hours of digging through various blog posts and Stack Overflow answers, I found this hint:

Read operation API limits are reported in rate_limit_status (and in the rate limit header on the response from an API call); but the posting / update limits are not. This is because these values are adaptive and may vary at times of high traffic, or if the system suspects that something suspicious is going on.
From: https://twittercommunity.com/t/cant-add-members-to-a-list-code-104/25824/3

As suggested elsewhere, I tried to add a user to a list by hand on the Twitter website and I got this error:

Should you get a message with “You aren’t allowed to add members to this list” you may have hit the same adaptive rate limit. I had to wait 24 hours to add users to lists, others could do it after 12 hours. Maybe you are lucky and have not to wait a whole day. Nevertheless, it is Twitter who blocks you and not an error in Tweepy.

 

Helpful sites

These two web sites where a great help while I was debugging this strange problem:

• Tweeterid.com – converts a Twitter Id to the screen name and vis versa.
• UnixTimestamp.com – converts a UNIX timestamp to a human readable form.

 

Next

Now that we know what is going on in Tweepy, we are no longer lost if something unexpected happens. Next week, we will look at how to work with lists on Twitter.