Well Crap! A Tech Debt Post

Well crap! Several weeks ago I was laughing with my co-workers about people who have nothing better to do then argue about the what ‘tech debt’ is. So now I’m posting about what ‘tech debt’ is.

So, I’m an asshole.

Uncle Bob had tweet about messes. This tweet.

I and others debated with him on twitter.

Things quieted down. Then DocOnDev posted this.

I was going to leave a long and rambling comment on Doc’s post. When I finished, I realized that it was long enough and rambling enough to be a blog post. So here we are.

Since arguing with Uncle Bob over that tweet, I’ve had time to think about this, so I’m going to indulge in some rambling commentary:

Writing software is a human endeavour. There will be messes. We may find them distasteful, but they are normal. And you won’t eliminate them by fiat.

If you have more then one committer on a project, you are at risk for accidental code duplication; new developers to the team will make naive mistakes; someone is going to try to sneak that multi-purpose Util class in there somewhere (arrrgghh); and a lot of code just isn’t going to express its intent.

When we go to refactor this code, will we treat it differently because its “distasteful”? Nope. We will test, and we will refactor, until the code is clean, just like any other debt. We need to account for these messes when we are considering how much of our budget goes towards paying back debt.

That being said, if you’re intentionally taking on debt by leaving your code a mess, and excusing it by saying “I’m delivering value faster”, then you’re a fool. How many opportunities will your project miss out on because developers will be cleaning up your mess? After all, this is debt. If you don’t pay for now, you’ll pay for it later, with interest.

Tech Debt isn’t supposed to be about taking a crap in your code base and calling it value. I think that may be what Doc and Uncle Bob are getting at.e

Just What Are You Testing? Write Tests Intently

Unit tests need to be robust and reliable. If your tests frequently raise false positives, or worse, fail to report actual errors, then they are not providing the level of comfort they should.

Effective test code, like production code, expresses its intent. Test code that is too general can be brittle, or worse, outright fail.

Here’s a simplified example, based on some code I’ve been working on in TriSano.

class Loinc < ActiveRecord::Base
  validates_presence_of :loinc_code
end

What we have here is a simple ActiveRecord class, named Loinc. Its only validation ensures that a loinc code is present. If we were to write a spec for this validation, it might look like this:

  it "should not be valid if loinc code is blank" do
    Loinc.create(:loinc_code => nil).should_not be_valid
  end

This test is actually very expressive. In fact, the code reads almost the same as the description. “A new loinc (with no value for loinc code) should not be valid.”

What we’ll start to see, however, is that this code is not actually expressing the proper intent of the test. Let’s make a change:

class Loinc < ActiveRecord::Base
  validates_presence_of :loinc_code
  validates_presence_of :scale_id
end

Now we’ve added a second validation, on the scale_id field. Our spec still passes, so everything’s hunky-dory, yes? Well, no.

The intent of our test code is to verify that a blank loinc code makes the instance invalid. The code actually tests that a blank loinc code *or* a blank scale id makes the instance invalid.

Pragmatically, this means that we haven’t properly isolated this test.

Nothing is broken yet, but, when merging in the commit that includes the scale id change, let’s assume the developer accidentally merges away the loinc code validation (hey, it happens). So we have:

class Loinc < ActiveRecord::Base
  validates_presence_of :scale_id
end

When we run our test, it still passes! That wasn’t what we intended at all.

To fix the test, consider what behavior we are expecting. Since this is a Rails app, we are expecting that, If a user tries to create or update a Loinc instance with a blank loinc code, they will receive an error message. That should be the intent of our test.

In code, our spec might look like this:

  it "should produce an error if loinc code is blank" do
    Loinc.create.errors.on(:loinc_code).should == "can't be blank"
  end

Our test is still expressive (well, maybe, a little less expressive), but now it is expressing our programs actual intent, and the validation tests for the loinc code field are isolated from other fields’ validations. If we run our test now, we receive the failure we’d expect.

Links

• TriSano on GitHub
• Practices of an Agile Developer: Practice 25 – Program Intently and Expressively

How Fetchmail (and Mutt) Saved Me From Email Mediocrity.

I’ve always had an ambivalent relationship with email. I used pine in college, but since then, it’s been mostly windows (and windows flavored) email clients.

At the start of my career, it was Outlook. Outlook could read email, but it’s primary feature seemed to be this calendaring thing. Largely it seemed to schedule meetings for me at the busiest times of day, and then canceled them at the last minute. It was easily thwarted by scheduling all day events or just ignoring it altogether.

When Thunderbird stabilized (it’s name; Minotaur? Pbbbbttt!) I decided to switch to that. It also read mail, but had two important features over Outlook. A mediocre feed reader, and not-being-Outlook.

I’ve been using Thunderbird ever since then, though I’ve occasionally tried other clients. Evolution was pretty good. It read mail and integrated with Gnome. It also had this crash-to-desktop feature that I could live without, so I went back to Thunderbird.

At the end of the day, these were all just windows-y applications. You claw at the screen with your mouse all day, dragging little sprite mails into little sprite folders. Configuration is a clicky-clicky exercise of navigating annoying menu tress to overcrowded little dialogs. Setting up one mail box, complete with filtering and signature wiz-bang, is just annoying. Setting up two or three? Even more so.

Why is it so annoying? Why can’t I grab the email from all my accounts and sort it into one set of folders? Why isn’t there a simple text file configuration for all this?

With those thoughts in mind, I started looking for the next great mail client. But in one of the first blogs I read, I stumbled across the magic words: ‘Fetchmail’ and ‘Maildirs’. *HEAD SLAP*

I feel the need to take a short Yegge here to defend my nerd cred.

Please understand that I spent most of my early career on Windows machines. My first programming job was at a VB6 shop.

Windows and Windows applications seem to have a general approach to users that says “Here, let me do that for you, before you break something.” Like how I take the TV remote from my mother when she’s about to “switch it over to DVD”.

Windows says “here, let me install that for you”. Or “here, let me scour the network for printers so all you need to do is pick one from a list” Or, everyone’s favorite, “you seem to be having trouble with that letter. Why don’t we turn it over to this anthropomorphic, animated paper clip. We think he can do it without spilling coffee on the keyboard.”

The point here is not to turn this into a “Windows makes you stupid” rant. The point is, that after years of that learned helplessness, I still sometimes have trouble getting in tune with the Tao of Linux. But I’ve got it now. I still get to be in the nerd club, right? We cool, right?

End Yegge.

So where was I? Oh yeah, ‘Fetchmail’ *SLAP*

Fetchmail is super easy to install and setup. I wish someone had told me about this before.

*Note: All my steps are Ubuntu flavored*

$ sudo apt-get install fetchmail fetchmailconf

I installed fetchmailconf. It was kind of crappy, but it helped me get the global settings configured. After that, I just edited the run control file myself.

So here’s what my ~/.fetchmailrc looks like (sorta)

# Configuration by fetchmailconf
set postmaster "my_username"
set bouncemail
set no spambounce
set properties ""

# don't keep on server because it's my hotmail account and who cares
poll pop3.live.com with proto POP3
       user 'user@hotmail.com' there with password 'secret' is 'my_username' here options ssl

# keep, just because (see, it's IMAP)
poll mail.01.com with proto IMAP
       user 'user@work.com' there with password 'secret' is 'my_username' here options keep ssl

# no keep. gmail is configured to archive after pop access.
poll pop.gmail.com with proto POP3
       user 'user@gmail.com' there with password 'secret' is 'my_username' here options ssl

mda "/usr/bin/maildrop"

This configures fetchmail to pull from three different accounts and deliver all the mail to me. The line that starts with ‘mda’ tells fetchmail not to bother going to port 25, just hand it off to a program called maildrop.

Oh, yeah, I also installed maildrop

$ sudo apt-get install maildrop

Maildrop is what actually sorts the mail. It uses a file called ~/.mailfilter to decide where to deliver the mail. This file contains the the regular expressions that decide what mail folder to deliver a message to.

DEFAULT="$HOME/Maildir/"
TRISANO="$DEFAULT/.Trisano"
WORK="$DEFAULT/.Work"
PLAY="$DEFAULT/.Play"

logfile "$HOME/maildrop.log"

if (/^(To|Cc|Bcc):.*user@work/)
{
  to $WORK
}

if (/^X-Apparently-To:.*trisano-dev/)
{
  to $TRISANO
}

if (/^(To|Cc|Bcc|Delivered-To):(.*user@hot|.*user@gmail)/)
{
  to $PLAY
}

to $DEFAULT

This is an example of a ~/.mailfilter file based on my first attempt. It sets up a default folder for mail. In this case, I’m using the Maildirs format, which I prefer over a single spoolfile. This file also specifies a few subfolders and uses simple regular expressions to decide which folder(s) to deliver a message to.

The regular expressions, by the way, are applied against the entire message, including the headers. So the expression /^X-Apparently-To:.*trisano-dev/, looks for a line starting with the header X-Apparently-To (an awesome header) and then the word trisano-dev (a google groups mailing list, in this case) anywhere else on the line. If it finds a match, it drops that message in the $TRISANO folder.

To make this work, I needed to create the maildirs folders. A utility installs with maildrop that takes care of that. I ran maildirmake once for each folder.

$ maildirmake ~/Maildir
$ maildirmake ~/Maildir/.Trisano
$ maildirmake ~/Maildir/.Play
$ maildirmake ~/Maildir/.Work

While I was getting set up, I ran fetchmail from the command line. Some of the early runs took a while because I had a lot of email laying around on servers.

Once everything worked, I set up cron to run fetchmail every few minutes. I actually use gnome-schedule for that, which is handy, because I don’t speak fluent cron.

$ sudo apt-get install gnome-schedule

Well, I have nicely filtered messages going into my new maildirs. Problem is, Thunderbird can’t read maildirs. But that’s OK, because I was tired of Thunderbird anyway (isn’t that how all this got started?).

I was looking for a nice emacs mail client to use long term but, by happy coincidence, eggyknap was chatting up Mutt in IRC one day, so I decided to give it a try.

$ sudo apt-get install mutt

A simple Mutt configuration looks like this (in ~/.muttrc)

set mbox_type=Maildir

set realname='Ryan L. Bell'

set folder="~/Maildir"
set mask="!^\\.[^.]"
set mbox="~/Maildir"
set record="+.Sent"
set postponed="+.Drafts"
set spoolfile="~/Maildir"
set signature="~/.sig"
set sort=threads

This tells Mutt I’m using Maildirs and that my ~/Maildir folder should be opened by default. It also tells Mutt what folders to use for Sent messages and saving email Drafts. These folders will be in Maildir format, but Mutt handles creating them, so Bob’s your uncle.

The last two lines tell Mutt which signature file to use for outgoing messages, and tells Mutt to sort using threads, by default.

I didn’t think I could face a sendmail configuration, so for simplicity’s sake, I just have Mutt talking directly to my smtp server. One day I may try using Lamson for this. We’ll see.

Again, from the ~/.muttrc file:

#
# Outgoing
#
set smtp_url="smtp://user@smtp.someserver.com:587"
set smtp_pass="secret"

Next, I set up Mutt to know about all my maildirs. This was not strictly necessary, but it makes navigating between folders in Mutt much more pleasant.

#
# Mailboxes
#
mailboxes "=.Play"
mailboxes "=.Work"
mailboxes "=.Trisano"
mailboxes "=.Sent"
mailboxes "=.Drafts"

The last things I added to my mutt configuration were these folder hooks.

#
# Folder hooks
#
folder-hook .*Play    set from="user@home.com"
folder-hook .*Play    set signature="~/.sig"

folder-hook .*Work    set from="user@work.com"
folder-hook .*Work    set signature="~/.sig-work"

folder-hook .*Trisano set from="user@work.com"
folder-hook .*Trisano set signature="~/.sig-work"

These hooks set my Reply-To header and signature file based on what folder I am in.

This barely scratches the surface of what you can configure Mutt to do. I’ve included a link to the Mutt configuration page in the links section.

Now I have exactly the email configuration I desired. I can control my mail filtering with regular expressions and a simple text file. I can grab mail from all my accounts and sort it into a simple set of folders. With Mutt, I have plenty of hooks to change the behavior of the client to suit my needs. I can even modify the message headers when a message is sent.

So what will I do with all this new found power and control? Who knows? But it won’t be boring.

Whew! Ran a little long this time. I guess I’ve been storing up some blogging action points. As always, corrections, additions, and enhancements are always welcome in the comments.

Links

• Maildirs Man Page
• Fetchmail manual
• Configuring Mutt

Run IE in VirtualBox

I tweeted this earlier today, but decided to blog it too. Here is an excellent screen cast on running IE (well, Windows too) in VirtualBox. Since the vhd’s are from Windows, and for the express purpose of cross browser testing, I’m assuming everything is above board.

A couple of notes: The screen cast is for OS X, but everything worked fine on Ubuntu. I just did a couple of things differently. I used wine to extract the exe’s, rather then 7zip. I also received a funky error from VirtualBox when I tried to run the image, but the error description told me exactly how to fix it.

The last thing I’ll note is that many comenters had problems w/ VirtualBox 2.1.0, so stick to 2.0.6.

Run Skype on Ubuntu 8.04

I’ve had Skype working on Ubuntu for a while, but I thought I’d blog it so it would be easier to find if I had to do it again.

To get Skype, I just downloaded the Ubuntu bundle from Skype’s site. Or you can go and checkout their other linux downloads.

For the record, I was not having trouble making calls. Once I was on a call, dialing numbers produced funny echo sounds on the line. This made it impossible to participate in conference calls because I couldn’t navigate phone trees.

Here’s how I fixed it:

Get the pulseaudio-utils.

sudo apt-get install pulseaudio-utils

Then, run skype through padsp.

padsp skype

That is all.

Rcov Update For JRuby 1.1.4

Just a quick post. An update that makes rcov compatible with JRuby 1.1.4 has been released. Here’s the announcement on the user’s list. Again, I’d like to commend Jay on his work on this. I really have not had the time to work on this at all, and Jay is the only reason anything has been released at all. Thanks Jay!

Rcov for JRuby Gem Available

It’s been a long time coming, but, thanks to Jay, who’s really kicked some ass to make this happen, the long suffering Rcov port to JRuby finally is bearing fruit. The first gem is now available.

There are a few known issues. Performance isn’t that great. And there is some slight variation between what Rcov reports in JRuby vs. MRI. But Jay’s been using it in his build environment, and so should you. So go ahead and grab the gem and help us make it great!

A Short Overview of My First OSCON

Here’s a quick overview of my first OSCON.

Portland

Portland seems like a nice city. The weather was very reasonable for July. Cool at night. Warm during the day. It did not rain.

The MAX is a nice train system. It makes everything in downtown Portland pretty easy to get to. There’s even a fareless zone.

Portland’s brand of pan handler seems to be chatty young idealists with Save the Children binders. They are much harder to get rid of by tossing a quarter on the ground and ducking into the Rite Aid. I pushed mine in front of a bus.

Tutorials

Because of plane issues and company obligations, I could only attend one tutorial. But it was Randal Schwartz talking about Seaside. If I could only get to one tutorial, that seemed like a pretty good one to make. Unfortunately, it wasn’t that great. Half of the tutorial ended up being intro to smalltalk. The format was live demo, which makes me feel like I’m the third wheel in a pair programming exercise.

When you’re talking about Seaside, and you don’t get to continuations until the last seven minutes of the talk, then all you’ve presented is how to use smalltalk to generate HTML.

I might just be a little bitter because I was looking forward to seeing the Magritte and the Glorp stuff.

Keynotes

Danese Cooper and Nat Torkington were both very good. One of the MySQL guys said MS is irrelevant. Moving along.

Breakout Sessions

Laika tests for interoperability between EHR’s. It’s part of the CCHIT certification process for 2008 (suckers!). I plan to have more to say about this in another blog post.

Hadoop and EC2 are cool and make a nice pairing. Furthermore, this session attracted more women then all of the other sessions I attended combined. Advice to lonely, single nerds: work on your Hadoop.

Subversion worst practices.

memcached is a cool, simple way to scale your webapps. However, it’s very difficult to talk about with reggae music blaring through the connector wall.

Sam Ruby is the most patient presenter I have ever seen. Perhaps that comes with having to deal with Ruby programmers. To be fair, Sam said he’d like it informal. It was.

Meta Programming Ruby was a good talk, but there wasn’t much new there. Except proof that, given enough Ruby programmers with too little to do, someone will eventually try to recreate Tapestry. The presenter was a cool guy though.

Other

rosscooperman pointed out that the random arrangement of motion sensitive soap dispensers in the restrooms would consistently produce a soapy wrist, unless one was diligent.

update – I was a tad more snarky then I had intended in the original post. I think the post as it is now constituted contains the correct amount of snarkiness.

Time’s Making Changes

What could be so stupendous as to merit sharing a title with The Best of Tesla?

The paperwork is finally done and everything is official, so I guess I can talk about it. I’ve got a new job with Collaborative Software Initiative. I start the first full week in July. I’m really excited. This is going to be a big change for me in a lot of ways.

First of all, since the company’s in Portland and I’m not, I’ll be working from home. People like to talk about the ups and downs of working from home: how you can never get away from work that way; how it’s hard to keep your personal life and your work separate; how you’re going to miss human contact. I think those people are jealous or nuts. When I’ve gotten to work from home in the past, it has ruled. I really think I’m going to eat it up.

The client is in Utah (well, the client is Utah), so I will be traveling a scosh, which I’ve never done. I’m looking forward to getting to see new places, but this is the aspect of the job that has me the most anxious. Since my daughter’s birth, I don’t think I’ve ever been away from her longer then twenty-four hours.

The product is a JRuby on Rails application, and it’s being developed in a lean/agile manner. I’ve been looking for a chance to work in an environment that was Ruby, or Smalltalk, and was agile. I’m not as fond of Java (the language) as I used to be, and my current company just doesn’t seem to be ready for agile development (if this post is any indicator).

The product is open source (or will be soon). This represents a huge change for me. A few months ago, I don’t think I appreciated how important this would be to me. Back then, I was trying to set up a talk about some work I had done in our software using JRuby for eRubyCon. I thought it would be a good talk and fit right into the eRubyCon concept. I was eventually shut down because there was concern amongst the powers-that-be that I would be exposing too much of our architecture and we’d be giving up a competitive advantage.  That was when I realized that I’m an open source guy, and I need to be making my living writing open source.

That is lot of change for one guy with a family to feed to make. I feel like I should be nervous. I’m not. I’m excited and I can’t wait to get started.

Run SLIME and CLISP on Windows

Friends! Let me free you from your complicated lives. From the complicated
part, not the lives part. -Bender

I managed to get SLIME and CLISP to play nice on Windows. As is often
the case when trying to get something interesting working on Windows,
I couldn’t find a single comprehensive source that explained
everything. I’ll document what I did here, and maybe that will save
someone else some time.

Oh yeah, and I did know about Lisp in a Box, but that felt like
cheating.

I already had cygwin installed. I did a full install (I had some
time on my hands). So I already had CLISP installed, too.

You’re going to need emacs to run SLIME. I use the Windows emacs
build, rather cygwin’s build. The cygwin build of emacs 22 is pretty
flaky on my machine.

When you go to download SLIME, make sure you grab the cvs tarball
and not the 2.0 tarball. The 2.0 release is not compatible with the
current CLISP release. The SLIME readme contains the magic incantation
to load SLIME into emacs. This goes in your .emacs file:

(add-to-list 'load-path "~/.site-lisp/slime/")  ; your SLIME directory
(setq inferior-lisp-program "/bin/clisp.exe -K full") ; your Lisp system
(require 'slime)
(slime-setup)

Change the paths so that they work for your system. Make sure that you
leave the “-K full” on the clisp executable.

There’s a path error that occurs if you run SLIME now. This is caused
by the difference between cygwin paths and windows paths. Note that I
don’t use any of that cygpath crap in my .emacs file. I also didn’t
want to write the elisp functions that would make Swank (it’s a SLIME
thing) work. I liked this suggestion from CLiki better:

mkdir /c
touch /c/NOT_MOUNTED
mount C:\\ /c

Start up emacs and run M-x SLIME. You should get the CL-USER> prompt
if everything worked.

I think that’s everything I did. If I missed anything, drop a note in
the comments.