One big benefit of asynchronous jobs is that they do not need to be processed right away. Let’s look how we can tell Celery to keep our jobs around for a while before starting the work.
This post is part of my journey to learn Python. You can find the other parts of this series here. You find the code for this post in my PythonFriday repository on GitHub.
Rate limit a task
Celery offers us a built-in way to throttle the task processing. We can limit our prepare() task to one run per minute with this code:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
import time from celery import Celery app = Celery('tasks', broker='amqp://guest@localhost//') app.control.rate_limit('celery_task.prepare', '1/m') @app.task def prepare(order_id): time.sleep(5) # simulate work print(f"order #{order_id} prepared") if __name__ == '__main__': app.worker_main() |
If we now restart our client, it will tell Celery to take it easy. We see in the Celery log that 4 jobs arrive at 19:32 and Celery processes them until 19:35 (4 jobs with approximately 3 seconds delay between them):
[19:32:32] New rate limit for tasks of type celery_task.prepare: 1/m.
[19:32:35] Task celery_task.prepare[d44f404b-…] received
[19:32:36] Task celery_task.prepare[29bc6e4d-…] received
[19:32:36] Task celery_task.prepare[97ddafea-…] received
[19:32:37] Task celery_task.prepare[36dbaee4-…] received
[19:32:40] order #15 prepared
[19:32:40] Task celery_task.prepare[d44f404b-…] succeeded in 5.0s: None
[19:33:37] order #16 prepared
[19:33:37] Task celery_task.prepare[29bc6e4d-…] succeeded in 5.015999999974156s: None
[19:34:37] order #17 prepared
[19:34:37] Task celery_task.prepare[97ddafea-…] succeeded in 5.015999999974156s: None
[19:35:37] order #18 prepared
[19:35:37] Task celery_task.prepare[36dbaee4-…] succeeded in 5.01500000001397s: None
While minutes are a good way to show the delay in action, Celery offers more options:
The rate limits can be specified in seconds, minutes or hours by appending “/s”, “/m” or “/h” to the value. Tasks will be evenly distributed over the specified time frame.
Rate limit multiple tasks / queues
Unfortunately, Celery does not help us when we need to limit multiple tasks with the same restriction. That would be helpful if we have multiple tasks that talk to the same (Twitter) endpoint.
If you google for a solution, you find various approaches. The best solution I found was from Aliev Magomed. Take a look if you have this problem.
Next
The more serious our work with Celery is, the more important will be to log what is going on. Next week we take a look on how to write log messages in a Celery task.