RabbitMQ has been one of my interests of late, as I’ve identified it as part of our technology path at work. There are other very good resources that dive pretty deep in RabbitMQ and how to use it. The goal of this guide is to help you get on your feet quickly and easily. It assumes a couple of things:
- You already know about message queues and have some experience or knowledge on the subject.
- You know what AMQP is.
- You are already interested in RabbitMQ enough to try it out.
If you’re good on those things, let’s get started…
RabbitMQ is written in erlang. As such, you should have already downloaded and installed erlang as a first step.
Download RabbitMQ and install it, which is pretty easy. I like to setup RabbitMQ in an /opt/rabbitmq directory. To do that, I set some environment variables before compiling (bash assumed):
export TARGET_DIR=/opt/rabbitmq
export SBIN_DIR=/opt/rabbitmq/sbin
export MAN_DIR=/opt/rabbitmq/man
Then I compile and install with “make install.” Because I like to run as my own user or a service user, I’ll chown -R myuser /opt/rabbitmq as appropriate.
There are a few other things we need to do including make the log directory and the directory RabbitMQ will use to store its data:
mkdir /var/log/rabbitmq
chown myuser /var/log/rabbitmq
mkdir /var/lib/rabbitmq
chown myuser /var/lib/rabbitmq
Now as “myuser” we can “cd /opt/rabbitmq/sbin” and run “./rabbitmq-server” and what you should see is:
RabbitMQ 1.6.0 (AMQP 8-0)
Copyright (C) 2007-2009 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.
Licensed under the MPL. See http://www.rabbitmq.com/node : rabbit@binti
log : /var/log/rabbitmq/rabbit.log
sasl log : /var/log/rabbitmq/rabbit-sasl.log
database dir: /var/lib/rabbitmq/mnesia/rabbitstarting database …done
starting core processes …done
starting recovery …done
starting persister …done
starting guid generator …done
starting builtin applications …done
starting TCP listeners …done
If you have the hang of starting RabbitMQ and now want to run it in the background, instead do: “./rabbitmq-server -detached”.
Once we’ve gotten this far, we’ve got our broker up and running and now we’ll need some way to talk to it. For the purposes of this article, I’m going to talk about amqplib and Python. There are AMQP libraries for just about every relevant language at this point. RabbitMQ 1.6.0 implements the AMQP 0.8 standard. The easiest way to install amqplib is a simple “easy_install amqplib”.
But before we dive into code, there are a few key concepts we need to talk about:
Queues: You should get these already, one puts a message in a queue and a consumer app receives it somewhere else.
Exchanges: These are a little more tricky than queues. I like to think of them as namespaces. One of the keen things about RabbitMQ exchanges is that different exchanges will get a different erlang process which should help make better use your available hardware resources. There are three types of exchanges that we need to talk about:
Direct: a direct exchange means when you put a message in, it goes to one consumer and he’s all that will get that message routed through the exchange.
Fanout: a fanout exchange sends your message to every consumer that listening to a particular exchange / queue combination.
Topic Exchange: this type of exchange allows you to do neat things like listen to the same queue across exchanges on one consumer, multiple queues in one namespace in a consumer and other wildcard type trickery.
Bindings: In RabbitMQ you bind your exchanges and queues together in unique combinations which determine how messages are routed to what consumers.
Memory: As of RabbitMQ 1.6.0 all messages are kept in memory. If you have nothing consuming your messages and you send too many of them, you’ll run out of memory.
Monitoring: The main install has the app rabbitmq_ctl which you can use to inspect the various parts of RabbitMQ. This isn’t very good for remote monitoring or visualization. For that there’s a great project called Alice which is also erlang based.
Speed: There are two ways to get messages from RabbitMQ: basic_get and basic_consume.
basic_get is where your app, on a message by message basis, asks RabbitMQ for a message. This is the slower of the two methods and will not allow single consumer applications to scale to a very high transaction rate. Note that RabbitMQ will not register these connections as a consumer and you will not see them in list_queues or in Alice as such.
basic_consume is where your app registers itself with RabbitMQ as a consumer and RabbitMQ will send messages to you as fast as you’re able to consume them.
Durability: If you want to have the definitions of your queues and exchanges hang around if you have to restart RabbitMQ you need to define them as durable.
Auto-Delete: If you want your queues and exchanges to exist even when there are no consumers waiting for messages on them, you need to turn auto-delete off.
Persistence: If you do not tell RabbitMQ that you want it to hang on to your messages if it reboots, it will not do so. You must set the delivery mode of a message to “2” to tell it to persist it until it is consumed.
Auto-Ack: You can tell RabbitMQ to automatically acknowledge receipt of a message, or you can do it yourself. This is a boolean setting that you use when you’re consuming messages via basic_get or basic_consume.
Queue and Exchange definitions: By default, queues and exchanges do not exist until you connect a consumer to them. You can cheat and do this in your code that enqueues your messages.
Now that we have that out of the way, here’s some sample Consumer code:
#!/bin/env python
""" Sample Consumer Code """
import amqplib.client_0_8 as amqp
# This is the function that basic_consume will send messages to
def process_message( message ):
""" Callback function used by channel.basic_consume """
print 'Received: %s' % message.body
# Rabbit Server to connect to
host = '127.0.0.1'
port = 5672
# Exchange and queue information
exchange_name = 'test'
exchange_type = 'direct'
queue_name = 'messages'
routing_key = 'test.messages'
# Let's set this up by default, we'll use it later
process_messages = True
# Connect to Rabbit
connection= amqp.Connection( host ='%s:%s' % ( host, port ),
userid = 'guest',
password = 'guest',
ssl = False,
virtual_host = '/' )
# Create a channel to talk to Rabbit on
channel = connection.channel()
# Create our exchange
channel.exchange_declare( exchange = exchange_name,
type = exchange_type,
durable = True,
auto_delete = False )
# Create our Queue
channel.queue_declare( queue = queue_name ,
durable = True,
exclusive = False,
auto_delete = True )
# Bind to the Queue / Exchange
channel.queue_bind( queue = queue_name,
exchange = exchange_name,
routing_key = routing_key )
# Let AMQP know to send us messages
consumer_tag = channel.basic_consume( queue = queue_name,
no_ack = True,
callback = process_message )
# Loop while process_messages is True
while process_messages:
# Wait for a message
channel.wait()
# Close the channel
channel.close()
# Close our connection
connection.close()
# This might go somewhere like a signal handler
def cancel_processing():
""" Stop consuming messages from RabbitMQ """
global channel, consumer_tag, process_messages
# Do this so we exit our main loop
process_message = False
# Tell the channel you dont want to consume anymore
channel.basic_cancel( consumer_tag )
Note that a lot of what is in that example is commented code and whitespace for ease of reading, the actual implementation is pretty darn simple.
Now that we have a consumer going let’s send some messages in:
#!/bin/env python
import amqplib.client_0_8 as amqp
# Connect
connection = amqp.Connection( host = "localhost:5672",
userid = "guest",
password = "guest",
virtual_host = "/",
insist = False )
# Create our channel
channel = connection.channel()
""" We've already declared our queue, exchange and binding in our consumer so just send the messages """
for i in range(0, 10):
message = amqp.Message("Test message %i!" % i)
message.properties["delivery_mode"] = 2
channel.basic_publish( message,
exchange = "test",
routing_key = "test.messages")
That’s it! If we did this right, you’ve now setup RabbitMQ, sent some messages and consumed them on the other end of the pipe.
If I’ve kept you this long and you’re still interested, but still have questions, I highly recommend this article which goes much more in depth and has been a valuable guide for me.
If you’re into both python and RabbitMQ, you might want to check out my consumer framework “rejected.py,” it’s on GitHub.
I hope you enjoyed the first of my A.D.D. Guides. I’d be happy to answer any questions and would appreciate feedback so I may improve this and future articles to come.
