Plans for Pika 0.9.14 and 0.10

    Comments

    The current plans for pika 0.9.14 and the 0.10 branch are as follows:

    I will be releasing 0.9.14 as a bugfix only release in the near future. I’m just trying to scratch out a day or two to get 0.9.14 ready to go. Worst case scenario is I am planning on doing a Sprint at pyCon in a few weeks and 0.9.14 will come out then.

    0.10 is a dramatic cleanup of the internals that aims to support the existing API through a compatibility layer but clean up a lot of the code and structure while adding Python 3 support. 0.10 will change how BlockingConnection works to use a background thread for IO management with RabbitMQ allowing blocking behavior in use but appropriate behavior in talking with RabbitMQ.


    Race for Hope 2013

    Comments

    I'm running again this year in the Philadelphia Race for Hope 2013, a 5k fund raising event for the National Brain Tumor Society. My mother, Marie Head (nee Schluter), passed in April of 2010 from a Glioblastoma multiforme tumor, her second brain tumor, nearly 20 years after her first diagnosed tumor.

    Funds raised here helps to direct promising research, support families who are currently coping with the effects of a brain tumor diagnosis, and advocate for change.

    I have joined this movement to bring the issue of brain tumors to the forefront. Progress is being made, but there is so much more to be done. Please support my efforts by sponsoring my run


    RabbitMQ in Depth

    Comments

    My book, RabbitMQ in Depth is today's Manning deal of the day. If you're looking to pick up a copy, be sure to use today's coupon code "dotd0624au" to get it for 50% off. RabbitMQ in Depth is a practical guide to building and maintaining message-based systems. This book covers detailed architectural and operational use of RabbitMQ with an emphasis on not just how it works but why it works the way it does. You'll find examples and detailed explanations of everything from low-level communication to integration with third-party systems. You'll also find the insights you need to make core architectural choices and develop procedures for effective operational management.


    Adding and Removing nodes from Chef via Cobbler

    Comments

    At MeetMe, we use Cobbler to bootstrap new servers and Chef for configuration management. With a bit of hacking we have Cobbler adding nodes into Chef so they configure upon first boot. Using Cobbler's trigger system we have two triggers that currently maintain nodes in Chef. I'd like to expand this to remove clients as well and to do so upon a rename of a machine, not just removal.

    To add a node to chef, there's an install pre hook python script that I hacked up a little to make work with the latest version of Cobbler and Chef. The first thing I needed to do was setup knife to work with the cobbler user for the trigger to work. This is pretty straight forward, just make sure that the user cobbler runs as has a ~/.chef directory and a working knife.rb configuration file.

    The unpleasant thing I found with Cobbler is the triggers now sit within the Cobbler package installation in Python. The trigger files in the Gist need to live in the site-packages/cobbler/modules directory (/usr/lib/python2.6/site-packages/cobbler/modules/ on our CentOS6 cobbler box). With the triggers in place, using management classes in Cobbler we're now able to auto-configure machines upon boot which removes a few steps from our process.

    If you have any updates, fixes or suggestions on how to make this integration even better, please let me know!

    The trigger files: https://gist.github.com/gmr/5339326


    New blog system, lost comments

    Comments

    Posterous.com finally is shutting down (or has by now). I had moved from Tumblr to Posterous because I liked the Posterous view of blogging and content and thought if I was able to send posts via email that I'd blog more often. It turns out that how I add blog posts has nothing to do with how often I blog.

    Anyway, as a result of Posterous shutting down, I decided it was time to bring my blog back into my control. I went through a minor selection process, deciding that I did not want to use a traditional blogging system but instead I wanted to use one of the newer static blog generation tools. I settled on Nikola, a Python based system for doing so.

    The conversion from Posterous was fairly painless. It took a few hours to get my content back in (mostly) and working the way I wanted. First I converted the Posterous backup from WordPress format, importing it into Nikola using Nikola's built in tool for doing so.

    Unfortunately this left me with poorly formatted and unmaintainable HTML so I then ran html2rst on the HTML files, converting them to Nikola's native reStructured Text format. Then I had to clean up the conversion post by post. One of my posts on using sphinx autodoc did not convert well because it was documenting using rst in rst and I didn't have time to figure out exactly why it kept blowing up.

    Once I converted my posts, I found Nikola is pretty interesting, once I started mucking with the internals. It's basically a tool that wraps a bunch of plugins and a task dispatcher using doit. I ended up making my own theme and hacking in support for embedding the tag list on any of the major pages. I still have a fair amount of stuff I'd like to do, such as change the format of the archives and tags page. I ended up coding my own "sharing" implementation for blog posts, as I did not want to embed the javascript clients for Twitter, Facebook, Google+ etc into the page. If anyone can point me to a simple endpoint for Facebook like information for a URL, I'd appreciate it as I'd like to add that in, no pun intended.

    Unfortunately, because I was using Posterous and they had their own comment system and now I have Disqus running my comments, I have lost my comment history. I don't know what I am going to do about that. They were in the backup that Posterous provided me.


    redis_git_dirty?

    Comments

    Have you been wondering what the redis_git_dirty stat is in the Redis info command is?

    Me too, so I downloaded the source and started grepping. What I found was that it is a constant that is defined in the src/mkreleasehdr.sh file. The exact command run to gather the value is:

    GIT_DIRTY=git diff 2> /dev/null | wc -l

    What is it doing? It's counting the number of lines when running the git diff command against the redis source code.

    The purpose? It looks like it's a development flag to indicate how different the compiled version of redis is from what the last git commit was.

    Should you monitor it? Nope, it's a waste of bits. In theory, this should always be 0 in a production release.


    clihelper

    Comments

    I tend to write a fair amount of command-line applications in Python that more often than not are meant to run as daemons. I also tend to use the same patterns in doing so. At first I wrote daemonization code myself following the general pattern that can be found in many places. Then I discovered python-daemon, the reference implementation of PEP-3143.

    For logging, I would often use the same code, cut-and-paste from one application to another. After digging into the logging documentation for Python 2.7, I decided that DictConfig in logging was for me, but I needed support in 2.6. I also wanted something I could install via pip instead of copying the code from 2.7's logging package and including it in my code. Thus logging-config was born. *Edit: I have now removed logging-config and moved to logutils thanks to a comment by Vinay Sajip below.

    Today, I am releasing clihelper, a Python module that aims to make writing command-line applications and daemons in Python easier. It uses python-daemon and logging/logutils together with a YAML based configuration file to let one focus on writing the core application and not the details about how to deal with command-line option handling, configuration, logging and daemonization.

    Getting started with clihelper is meant to be very straightforward; simply extend the clihelper.Controller class:

    class MyApp(clihelper.Controller):
        def process(self):
            self._logger.info('Would be processing at the specified interval now')
    

    Next, in the main guard for the python module you are putting the MyApp class clihelper.setup method should be called, then call the clihelper.run method passing in the class that will be used as your application controller.:

    if __name__ == '__main__':
        clihelper.setup('MyApp', 'MyApp is just a demo', '0.0.1')
        clihelper.run(MyApp)
    

    Next, the configuration file should be created:

    Application:
        wake_interval: 60
    
    Daemon:
        user: myappuser
        group: myappgroup
        pidfile: /var/run/myapp.pid
    
    Logging:
        version: 1
        formatters: []
        verbose:
          format: '%(levelname) -10s %(asctime)s %(process)-6d %(processName) -15s %(name) -10s %(funcName) -20s: %(message)s'
          datefmt: '%Y-%m-%d %H:%M:%S'
        handlers:
          console:
            class: logging.StreamHandler
            formatter: verbose
            debug_only: True
        loggers:
          clihelper:
            handlers: [console]
            level: INFO
            propagate: true
          myapp:
            handlers: [console]
            level: DEBUG
            propagate: true
        disable_existing_loggers: true
    incremental: false
    

    Now invoke your application via the command line. Try passing --help to see the base level options.

    clihelper allows you to add your own command line options and does not need to be interval based. Instead, one can use a blocking IOLoop or other similar concepts. In the class that extends clihelper.Controller, redefine the clihelper.Controller.run method. In addtion, you'll likely want to extend the clihelper.Controller.cleanup method to tell the IOLoop to stop when the application has been signalled to stop. An example with the Tornado IO Loop may look something like:

    import clihelper
    from tornado import ioloop
    
    class Test(clihelper.Controller):
        def run(self):
            # Setup the socket and listen
            self.ioloop = ioloop.IOLoop.instance()
            try:
                self.ioloop.start()
            except KeyboardInterrupt:
                LOGGER.info('CTRL-C caught, shutting down')
                self.cleanup()
    

    There are a few other options, so if you're inclined to try it out, I suggest reading the documentation and the code. I hope that someone else will find this useful. If you have any suggestions or improvements, please do not hesitate to send them my way.


    Disabling default iTunes behavior with media keys in OSX

    Comments

    In a search for a non-hacky way to disable direct mappings to iTunes for the play/pause/next/previous buttons in the keyboard in OSX, I stumbled across the "Remote control daemon" (rcd). As it turns out, disabling the iTunes launch behavior for these keys is as easy as unloading rcd with the following command:

    launchctl unload -w /System/Library/LaunchAgents/com.apple.rcd.plist

    I've not run across any negative impact of doing so and now when I use Enqueue, the buttons work for just it.


    PyCon by Network Adapter Manufacturer

    Comments

    On Friday, Doug Hellmann of PyMOTW fame commented on Twitter that he'd be interested in the breakdown of laptop vendors for people attending PyCon, something I had been pondering that morning as well.

    In a attempt to come up with an approximation of this number, this morning I kicked off Kismet to listen on the network for approximately 10 minutes while it collected 2,371 MAC addresses. Taking the data it collected, I wrote a script that looked up the MAC addresses it found to look the manufacturer. I was surprised at inconsistency of the company names in the MAC address vendor database. I cleaned up the multiple versions of manufacturer strings and ended up with data I could then easily chart.

    The results confirmed the impression one gets just walking around and looking at what people are using:

    Network Adapters

    A pretty strong showing of Apple devices. It appears that the lower end numbers are either mobile devices on the network or network gear.


    The Future of Pika

    Comments

    I've been listening to a lot of the conversations at PyCon about asynchronous development. The basic sentiment I've picked up on is that Callback Passing Style (CPS) is not currently in favor in the Python community. This in addition to the popular use of the BlockingConnection in Pika has lead me to think about how to plan Pika's future enhancements. After some conversation with Tony, I believe I have an outline that should appeal to Python developers while keeping Pika asynchronous at its core and retaining CPS. I think that CPS is very powerful and believe it's still very important to Pika's future.

    After I release 0.9.5, I will start development on Pika 2.0 which will be an effort to create a more pythonic approach to using Pika while retaining the ability to use CSP and keeping it asynchronous at its core.

    The roadmap for changes to Pika 2.0:

    • Backwards incompatible change that drops Python 2.4, 2.5 support

    • Add Python 3 support

    • Remove existing connection adapter system

    • Implement new pattern for use, behavior based use focused on both Asynchronous callback passing style and "Pythonic" development.
      • Both behaviors available from the same API calling same classes and methods

      • Async:
        • Merge existing connections into one connection system with IOLoop override
        • Supporting internal IOLoop, tornado, twisted
      • Pythonic:
        • high-level blocking on synchronous AMQP commands
        • Generator for receiving messages from Basic.Publish
    • API notation closer to AMQP spec for implementation.

    • *.*Ok frames will only be passed back in CPS use.
      • Calling methods like queue.declare will return a success indicator and attributes returned in the Ok frame will be assigned to attributes of the class.
    • basic.consume and basic.get will return a single object with a materialized view of the Method, Header and Body frames.

    • Build in support for specific broker types and pure AMQP 0-9-1.

    Here's an example of what I expect Pika 2.0 to look like for non-CPS use. Note this is more of an idea of how it will work for someone using Pika than a spec or actual code.:

    from pika.rabbitmq import Connection
    from pika import Basic
    from pika import Channel
    from pika import Exchange
    from pika import Queue
    
    from sys import exit
    
    # All the attributes can be passed in via constructor or assigned
    connection = Connection()
    connection.host = 'localhost'
    connection.port = 5762
    connection.user = 'guest'
    connection.pass = 'guest'
    connection.vhost = '/'
    
    # Not much new here
    try:
        connection.open()
    except pika.ConnectException as e:
        print "Could not connect: %s" % e
        sys.exit(0)
    
    # Channel construction outside of connection context, instead pass
    # the Connection in
    channel = Channel()
    try:
        channel.open(connection)
    except pika.TimeoutException as e:
        print "Could not open a channel: %s" % e
    except pika.ConnectionClosedException as e:
        print "Could not open a channel, the connection is closed"
    
    # All the attributes can be passed in via constructor or assigned
    exchange = Exchange(channel)
    exchange.name = 'not_microsoft'
    exchange.type = 'fanout'
    exchange.durable = True
    exchange.declare()
    
    # All the attributes can be passed in via constructor or assigned
    queue = Queue(channel)
    queue.name = 'my_queue'
    queue.auto_delete = False
    queue.durable = True
    queue.passive = False
    
    # Declare the queue and expect a bool
    if not queue.declare():
        raise Exception("Could not declare my queue")
    
    # Print info about the queue that was mapped automatically when
    # Queue.DeclareOk was received
    print 'Queue "%s"' % queue.name
    print ' Depth     : ' % queue.message_count
    print ' Consumers : %i' % queue.consumer_count
    
    # Bind the queue
    queue.bind(exchange=exchange, routing_key='not_microsoft.my_queue')
    
    # Generator returning one type for a message
    for message in Basic.consume(my_channel, routing_key="myqueue"):
        print 'Delivery Tag   : %s' % message.delivery_tag
        print 'Channel        : %i' % message.channel
        print 'Body Size      : %i' % len(message.body)
        print 'Properties'
        print '  Content-Type : %s' % message.properties.content_type
        print '  Timestamp    : %s' % message.properties.timestamp
        print '  User Id      : %s' % message.properties.user_id
        print '  App Id       : %s' % message.properties.app_id
        print 'Body           : %s' % message.body
    

    I am looking for feedback on this direction. Do these changes and the example make sense to existing Pika and RabbitMQ uses Would you change anything about this direction What would you improve?


My Book