The series on Tweepy showed my that I have a big gap in my Python knowledge: How can I separate work in an asynchronous way? In this post I look at RabbitMQ and how we can use it to get more flexibility in our Python applications.
This post is part of my journey to learn Python. You can find the other parts of this series here. You find the code for this post in my PythonFriday repository on GitHub.
Install RabbitMQ
When it comes to asynchronous processing of work, the term message-oriented middleware rings a bell. This approach allows us to split the work between multiple applications and decouple it in time (making it asynchronous). A message can be any kind of information that allows another application to do the work it should do. (That flexibility is annoying when you start with this approach, but it will be a great help as soon as you start building your own application.)
One popular open-source solution for this approach is RabbitMQ:
RabbitMQ is a messaging broker – an intermediary for messaging. It gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.
You have multiple options to install RabbitMQ and they are well described in the documentation. For my machine I used the Docker image that we can install and run with this command:
|
1 |
docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.9-management |
If everything works, we can connect to the Docker container image and check if there are jobs waiting for us:
|
1 |
rabbitmqctl list_queues |
Timeout: 60.0 seconds …
Listing queues for vhost / …
You should get the same output (== nothing) if you run the command in a freshly started container.
Install the Pika client
Our application needs a client to talk with RabbitMQ. I followed the great tutorial and installed Pika:
|
1 |
pip install pika --upgrade |
Send a message
The simplest working model for RabbitMQ is a queue. We have a producer that sends messages to a queue and we have a consumer that takes the messages and processes them. To send a message to RabbitMQ we can use this code from the documentation:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# Example from https://www.rabbitmq.com/tutorials/tutorial-one-python.html import pika connection = pika.BlockingConnection( pika.ConnectionParameters(host='localhost')) channel = connection.channel() channel.queue_declare(queue='hello') channel.basic_publish(exchange='', routing_key='hello', body='Hello World!') print(" [x] Sent 'Hello World!'") connection.close() |
Receive a message
The consumer needs to check the queue and then process the message. We can use this code with a callback that runs whenever a message comes in:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# Example from https://www.rabbitmq.com/tutorials/tutorial-one-python.html import pika, sys, os def main(): connection = pika.BlockingConnection( pika.ConnectionParameters(host='localhost')) channel = connection.channel() channel.queue_declare(queue='hello') def callback(ch, method, properties, body): print(" [x] Received %r" % body) channel.basic_consume(queue='hello', on_message_callback=callback, auto_ack=True) print(' [*] Waiting for messages. To exit press CTRL+C') channel.start_consuming() if __name__ == '__main__': try: main() except KeyboardInterrupt: print('Interrupted') try: sys.exit(0) except SystemExit: os._exit(0) |
Run the consumer and the producer
The great benefit of RabbitMQ is that our consumer does not need to run when the producer sends its message. This gives us a lot of flexibility and helps us to create a robust application.
We can run the producer with this command:
|
1 |
python .\rabbitmq_send.py |
[x] Sent ‘Hello World!’
If we now check our Docker container, we should see that one message is waiting in the hello queue of RabbitMQ:
|
1 |
rabbitmqctl list_queues |
Timeout: 60.0 seconds …
Listing queues for vhost / …
name messages
hello 1
If we now start the consumer, it will find the message and prints it out:
|
1 |
python .\rabbitmq_receive.py |
[*] Waiting for messages. To exit press CTRL+C
[x] Received b’Hello World!’
If we start the consumer before we send a message, it will wait until a message arrives. We can also stop the consumer, let the producer send more messages and RabbitMQ will keep the messages around until the consumer is ready to receive them.
Next
With this basic example of RabbitMQ we are able to explore the many additional features we could use. Before I try that, I will take a look at Celery, a tool that works on top of RabbitMQ and offers us some higher-level functionality.
3 thoughts on “Python Friday #121: RabbitMQ and Python”