Horses are terrible people - Home

Starting Development on Golconde

2008-08-31T02:30:00Z

I’ve started prototyping Golconde, which is designed to be a loosely coupled data distribution system for PostgreSQL. The concept is to implement a trigger based tool using python and Stomp to send Inserts/Updates/Deletes to ActiveMQ which, as a message broker, will allow for the queries to be sent to one or many target systems.

The goal here is not to implement a replication system, as there are fine PostgreSQL based tools for replication. Golconde is a system that does not care about ACID compliance and implements a fast, low-overhead system where data is mostly correct. A mostly correct system could play a roll in non-mission critical service oriented applications.

Why create a system like Golconde? Replication systems generally carry a high-cost associated with transaction level consistency. In many cases, I have found that there are applications that need not worry about that level of consistency. By periodically doing full table copies to the remote servers, we re-synchronize the base data and then catch up on changes that happened after the full snapshot was taken.

Take the example of a Twitter post cloud generation and search tool. The importance of data consistency is not as high as application responsiveness. Due to the volume of data being aggregated in Twitter, the lack of information about a post or the removal of a post is not likely to effect the functionality of the application - at least not as much as if we were writing an accounting package.

If you’re interested in participating in the design and implementation, or just have advice or feedback, I’d love to have additional brains help form what I think may be a valuable scaling tool for some PostgreSQL based applications.

Edit: I've updated the article to reflect the proper spelling of the name, which I named directly after a Rene Magritte painting, but is indirectly named after an ancient Indian City. In the previous post, Depez had asked about a mailing list, I will put one up shortly.

A Vendor Combination to Avoid

2008-06-23T18:45:00Z

CDW + Sun Microsystems == Absolute incompetence.

Unofficial pgCon 2008 Group Photo

2008-05-25T02:36:00Z

Click for Larger Version

Here are some misc pictures I took @ pgCon.

Playr, PgSQL Scripts Released

2008-05-22T13:38:00Z

I've just created the 0.1 release tarballs for both Playr and our PgSQL Script bundle.

Playr is our log replay, PostgreSQL stress testing application and attempts to answer the question "How much headroom will our new server give us?" Our PgSQL Script bundle is just a hodgepodge collection of scripts which currently has two scripts we've developed for in-house use; concurrentReindex and parallelVacuum. Both "projects" are fairly new, use at your own risk.

Polaroid - Again

2008-05-18T05:23:00Z

Received the new FLM3732 replacement unit on Thursday in a box that's far too flimsy to ship a television in, though there was adequate foam packing. The LCD has a "dark streak" in it, about 1" wide and about 10" high on the right side of the LCD. It's most noticeable against solid backgrounds, such as when I'm watching golf in 1080i. Round two for support to get a good replacement? I think I'll send my blog to the CEO of Polaroid.

Reindexing Concurrently

2008-05-09T15:44:00Z

Index bloat can be a major pain in heavy OLTP databases. With Reindex being a blocking operation in Postgres, you can not reasonably reclaim index bloat in a large database by hand without going offline. Thus enter concurrentReindex.php. This PHP cli script will concurrently reindex all non-unique, non-system catalog indexes in a given database. I have only tested this against a Postgres 8.2 database, but it should work in theory against 8.2 or higher. Feedback, enhancements and patches welcome.

Another strike for Polaroid

2008-05-03T23:13:00Z

Called up today to inquire to the status of the replacement TV and found out that a supervisor put the replacement on hold as of April 21st due to needing secondary confirmation from the technician beyond what they put in the computer. That was the last status update. I received no call from them. I couldn't imagine running a business the way that they seem to. I think the next step is figuring out how to best get the word out to never buy from Polaroid and documenting all of the calls, exchanges, broken promises and lack of support I have received in fixing my flat screen TV that was under warranty when it went bad.

Follow-up on PDO and libpq Bug

2008-04-19T01:31:00Z

A while back i blogged about a bug with PDO and prepared statements. After digging a little it appears that as of libpq.so.4 (PostgreSQL 8.0 tree) native prepared statement support was added. If PDO detects you have libpq.so.4 at compile time it will use the libpq prepare instead of its own internal prepare. What then happens is random prepared statements fail with heavy use. Again the only way around this right now is to use libpq.so.3 from PostgreSQL 7.4. Now to figure out who to bug about it. I have a feeling the PDO folks will point to libpq and the libpq maintainers will point to PDO.

Zend Studio 6

2008-04-19T01:10:00Z

I'll admit, I was not happy with the jump from Zend Development Environment 5.5 to Zend Studio for Eclipse 6.0. I've tried other Eclipse based IDE implementations, most notably, Aptana. I found them to be slow, buggy and not very reliable. Eclipse is complicated and the interface designers clearly shouldn't be designing interfaces. I have to give Eclipse its due though, it is very flexible and robust.

I voiced my concern about Zend Studio 6 to my contacts at Zend, including the product manager for Zend Studio. To me it was too far of a departure from ZDE. From a business perspective I understand why they discontinued ZDE and moved to Zend Studio, money. Why pay programmers to maintain the core development environment when you can pay them to enhance an already popular and extendable Eclipse? That being said, as a developer I feel alienated and left out in the cold.

I decided about two months ago to give Zend Studio a fair shake. Not the load it up, poke around in it, find what I disliked and move on evaluation I initially did. What I found since doing so is I like some things about it, but the negative aspects far outweigh the positive.

Eclipse and Zend Studio on top of it is slow, unstable and prone to lockups. The remote development support, mounting remote filesystems over SSH or SFTP is unreliable and unusable. Remote projects and the various Zend Studio specific add-ons feel more like a kludge after-thought than a core part of the overall experience. The frequent lockups, instability and time it takes to work on remote filesystems, in comparison to ZDE 5.5 has made me switch back. Even the random errors in ZDE due to the Leopard upgrade are far less impacting than the issues with Zend Studio for Eclipse.

I do have to hand it to Zend Studio though, and it's more credit to Eclipse than to Zend; The XML editing in Eclipse is hands down better than ZDE.

Polaroid follow-up

2008-04-07T18:11:00Z

We finally had a Polaroid service tech show up today to repair the TV. Turns out the issue was with the LCD and so they're going to send us a new TV. I do appreciate the replacement (which we should receive in a few weeks), but the lack of follow up was ridiculous and the amount of time it is taking to get this taken care of is just plain unacceptable.

Polaroid Warranty Service

2008-03-21T20:12:00Z

Polaroid customer support for warranty service is a joke.

Nearly two months ago, which was two months before the warranty expired for my FLM3732 LCD television, the television stopped turning on. One would think it would be a simple process to get Polaroid to honor their warranty on their products, but thus far, it has not as easy as it should be.

I’m on my 15th call into the support and have faxed the receipt to them 6 times (with confirmation from the machine). They have yet to call me to inform me of any status and they do not have any info on the receipts that I have faxed, each time I call. They have verified the fax number is correct several times. To top it off, each time when you send a fax, they ask you to wait two business days before calling back. They won’t verify it has been received and they insist they must wait until the receipt has been entered into the system before doing anything.

It seems at this rate I’ll never have the unit repaired and should just ditch Polaroid and swear them off for good.

php 5.2.x, PDO, libpq bug

2008-02-03T00:46:00Z

I am working on writing up a test scenario to submit the bug report, but I thought I'd let people know of a PDO+libpq issue I've recently found. The problem manifests itself in prepared statements. Using libpq for PostgreSQL 8.0 or higher, under heavy use, PDO will somehow drop prepared statements before PostgreSQL has set them up, so in the logs, you will see lots of entries like the following:

ERROR:  prepared statement "pdo_pgsql_stmt_b6dc71e8" does not exist

The quick fix is to install libpq for PostgreSQL 7.4 and recompile PHP against that. I'll update more when I have additional information.

The Best Spam I've Ever Received

2007-12-20T17:01:00Z

Do I need to say any more?

The Order of The Illuminati under the jurisdiction of the all Seeing Eye, Master Nicholas Brenner has after series of secret deliberations selected you to be an optional beneficiary of our 2008 foundation laying grants and also an optional opening at the round table of the Order of The Illuminati.

These grants are issued every year around the world in accordance with the objectives of the Order of The Illuminati as stated by Thomas Paine in 1810 which is to ensure the continuous freedom of man and to enhance
mans living conditions.

We will also advice that these funds which amount to US$2,000,000.00(two million united states dollars) be used to better the lot of man through your own initiative and also we will go further to inform that the open slot to join the Order is optional, you can decline the offer.

In order to claim your grant, contact the Grand Lodge Office secretary David P. Roberts.

Email: illuminate@katamail.com

Dr Paul Winsdor,
Order of The Illuminati

The ultimate goal of the Order of The Illuminati has always been to replace the destructive forces of monarchism, nationalism, religion with the productive dynamics of business. Toward that goal, Neo-Tech/Zonpower adds

· free-market dynamics for societal decisions plus.

· the essentialness of working-class individuals.

For, such working-class individuals are disconnected from the elite class, which is intellectually indoctrinated into closed-circle visions of nature. Thus, non indoctrinated workers have retained their childhood capacities for fully integrated honesty -- for wide-scope viewings of nature. While vision-controlled elites, specialists, and philosophers -- including most Objectivists -- have lost their capacities for fully integrated honesty.

The main objective of the Order of The Illuminati is making good men better, making strong men powerful.

Order of The Illuminati.

PrivatePaste.com Updates

2007-12-10T02:21:00Z

After much neglect I finally sat down today and updated PrivatePaste.com. The space bug in the empty textarea box for pastes is now fixed. File Uploads are now enabled, secure, and can expire. I've added a copy view which also attempts to copy to the clipbboard. In addition I added a download as text link for content. I also did some long needed database maintenance and added some indexes which should speed the site up a bit. If you’ve used it in the past, you should find the new features useful. If you haven’t used it, stop by and check it out!

Graphing Last.fm

2007-11-29T21:51:00Z

I’ve been on last.fm for a few years, since it was called audioscrobbler. I’ve enjoyed seeing what musical listening trends I have and like the whole concept. I was pointed to LastGraph today, a service that goes and gets your last.fm data for a given range and makes a very cool graph of it. Here’s an example for December 2006-March 2007 for me:

Click for a larger view

Here’s a link to my entire Last.fm history, which I found interesting to check out. Do you have as last.fm account that you actively use? If so, drop your graph in the comments!