JRuby ETL Performance Boost

25 01 2008

I spent today looking for ways to get our JRuby ETL to handle more data. Today’s work was mostly centered around cutting back on reads from the database. I had a lot of luck taking advantage of data already available in the JMS message we receive. The numbers aren’t in, but I think this should make it fast enough. It also looks like upgrading to a newer JRuby release will be fairly painless. That should get us some JIT, too. Sweet.

While I was poking through the code, I realized that it could use some work. It’s not thread safe, for one thing. That’s not a problem in production now, but I’d like to be able to run multiple processes one day and process more messages.

So I had the idea. The current tool uses TestCase, but I thought that I might start writing rspecs based on a) what the ETL does now, and b) what I’d like it to do in version 2.0. Once the specs are solid, I’ll start a rewrite. It’ll be interesting to see what comes of it.

As an aside, I popped into the local RUG. Coincidently, they were talking about rspec. Nothing earth shattering, but it seemed like a good group (if a little heavy with M$ Mono users).

Advertisements

Actions

Information

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




%d bloggers like this: