Python Friday #123: Running Celery on Windows

If we try to run Celery on Windows, we will run into a problem: Windows is not officially supported by Celery. However, that does not mean that it will not work. Let’s look what we need to do to run Celery on Windows.

This post is part of my journey to learn Python. You can find the other parts of this series here. You find the code for this post in my PythonFriday repository on GitHub.

 

Work not done

If we start Celery on Windows, as we would start it on Linux, we get a strange behaviour: Celery accepts our tasks but then it keeps them around and does not process them:

[…] Task celery_task.prepare[e36b94bb-1cd2-4109-aa7d-6099cc8f1db1] received
[…] Task celery_task.prepare[f097be78-1518-45db-aed5-d01913ab2517] received
[…] Task celery_task.prepare[27b857f7-a5a5-4dfa-9b90-2e8463cd4e1a] received
[…] Task celery_task.prepare[153a53d9-7f06-41f4-bbac-8a58542901ca] received

If we check RabbitMQ we see that our queue contains 4 jobs:

Timeout: 60.0 seconds …
Listing queues for vhost / …
name messages
celery 4

 

Changing the start command

We need to change the command that we use to start Celery and append the option -P threads:

With this command Celery starts and will not only accept our tasks, but it will process them:

[…] order #1 prepared
[…] Task celery_task.prepare[e36b94bb-1cd2-4109-aa7d-6099cc8f1db1] succeeded in 5.0310000000172295s: None
[…] order #2 prepared
[…] Task celery_task.prepare[f097be78-1518-45db-aed5-d01913ab2517] succeeded in 5.0310000000172295s: None
[…] order #3 prepared
[…] Task celery_task.prepare[27b857f7-a5a5-4dfa-9b90-2e8463cd4e1a] succeeded in 5.032000000006519s: None
[…] order #4 prepared
[…] Task celery_task.prepare[153a53d9-7f06-41f4-bbac-8a58542901ca] succeeded in 5.01600000000326s: None

And since our task get processed, the queue in RabbitMQ is empty:

Timeout: 60.0 seconds …
Listing queues for vhost / …
name messages
celery 0

 

Problem solved?

This little change should be all it takes with version 5.2.6 of Celery to run it on Windows.

Nevertheless, I suggest you use Windows only for development purposes and run the production on Linux. Should you notice a difference I suggest you create a dev container for Python.

 

Next

With a running Celery installation, we can now start exploring the interesting features. Next week we take a look on how we can slow down Celery.