Generating .ldif Files using Groovy

I recently needed to do a bulk update of an openLDAP directory to add a password for a really big batch of test users.  I wrote a little Groovy script to do it.  Thought it might be helpful.  First things first, generate a password for the test users using the slappasswd command.  Using the defaults will give a you password with SSHA encryption.  Now we need an .ldif file that describes the change to make.  For more information on ldapmodify commands, head here.  The script here will loop through a group of sequentially id'd users in a couple of ous and assign them our nicely hashed password:

        File newFile = new File('modify-script.ldif')
        String username = ''
        String toAdd = ""

        def stuCount = 80000
        def admCount = 2000
        def teachCount = 18000

        StringBuilder sb = new StringBuilder()

        def ous = [ 'ou1', 'ou2' ]

        for ( ouname in ous ) {

            for ( i in 1..stuCount ) {
                username = "user_${epname}_${i}"

                toAdd = """
dn: uid=${username},ou=people,ou=${ouname},dc=yourdc,dc=com
changetype: modify
add: userPassword
userPassword: {SSHA}zW7Q/yQQ8IKZiX8ANJIGugi0deNebN1o
                    
"""
                sb.append( toAdd )
            }

            sb.append( "\n\n" )

            newFile << sb.toString()
        }

This will produce a file (modify-script.ldif) with a bunch of entries like these:

dn: uid=user_ou1_1,ou=people,ou=ou1,dc=yourdc,dc=com
changetype: modify
add: userPassword
userPassword: {SSHA}zW7Q/yQQ8IKZiX8ANJIGugi0deNebN1o
                    

dn: uid=user_ou1_2,ou=people,ou=ou1,dc=yourdc,dc=com
changetype: modify
add: userPassword
userPassword: {SSHA}zW7Q/yQQ8IKZiX8ANJIGugi0deNebN1o

...

Now you can simply run the ldapmodify command to update the users:

ldapmodify -x -D "cn=admin,dc=yourdc,dc=com" -w yourpass -f modify-script.ldif

I've been able to use derivatives of this script for a few different tasks.  Hope it helps someone else.

UberConf, Days 2+3

After a great day one, we were looking forward to another great, long day, and we weren't disappointed.

Scala for Java Programmers - Day 2 started with a fun Scala workshop by Venkat Subramaniam that spanned two sessions.  This was extremely hands-on, and very worthwhile.  I learned enough about a language I have had interest in for a while to get started on my own, and got a nice little dose of Computer Science and compiler fun too.  My only complaint here was that we spent too much time using closures and Scala syntax craziness to write code in one, perl-ish, unreadable line.  We should have used the time writing 'terse' obfuscated code playing more with Scala parallel programming.  Actors are nifty, and it's amazing how clearly you can express complicated multi-threaded program flow.  If I had access to this goodness, I can think of one project I did at Blackboard that could have been written in half the time, in 1/5 the LOC, and in a much more readable and intuitive way.  Hindsight is 20/20, I suppose, but man that made me burn a bit.

Emergent Design - After lunch, we attended Neal Ford's talk on emergent design.  This was a good talk, mostly because Neal Ford is a good speaker, but I wouldn't say this was anything new.  No big design up front, test first, don't solve problems that don't yet exist, as this leads to heaps of code debt.

Stability Antipatterns - This was a workshop by Michael Nygard about common antipatterns you see as it relates to stability - this was more on the operations and integration level than at the code level, and had a fair amount of good tips for things to remember - mostly how to keep your application from dying because there is an external resource that you don't control that is failing.  Some of the horror stories he shared were hilarious, and some you could totally see happening to you (if they hadn't already).  Good session.  Good reminder of things to consider that often fall by the wayside.

On to Day 3, which was kind of disappointing.

Implementing Domain-Driven Design - I was excited about this one because DDD is one of those things you always hear about, but never really get.  After the session, I still didn't really get what the big deal was - drive the design of your application by the real-world domain.  Seems pretty simple to me.  What wasn't simple was the 100 slides of UML class diagrams the presenter used to demonstrate DDD.  Yikes, not that awesome.

SOLR - A Case Study - This was an interesting enough presentation, as Solr is something that we are evaluating.  I learned enough to know that we didn't really need to do this right away, but not much more.  It wasn't really a case study.  What I would have liked was either, Solr real-world example, or Migrating from Lucene to Solr, but alas we didn't really get either.  It was still a good presentation though, just not what I was looking for.

Android Mobile Development - Gah!  Fail!  This could have been a great way to close things out, but the presenter instead did the presentation completely out of order, gave nobody any opportunities to code, and was just generally all over the place.  I was at least able to use the time to make my way through some of the Android tutorials that Google provides.  Man what a bummer.

Uberconf Impressions, Day 1

Last week a group of us from eCollege attended the Uberconf presentation here in the Denver area.  I figured I'd post a few impressions:

First off - the venue - the Westin in Westminster, CO.  Nice place.  Really awesome food whether you are a vegetarian or a carnivore.  Free drinks all day.  Decent coffee, although the cups were oddly small.  One note to the Westin - people operate better at temperatures higher than 58 degrees.  It was FREAKING COLD in there.

Now on the conference organization.  This was one of the best flows of any conference I've attended.  There weren't too many people working the conference, and it still went well.  Easy registration process, good marketing materials (4GB USB stick - not bad), and two t-shirts I would be super embarrassed about wearing.  Easy to figure out what sessions were coming up and where they were going to be.  One minor nit - it's a developer conference, which means laptops - next time, get more power strips in the rooms, at least for the workshops.

And, finally the important stuff - the sessions (day 1):

Patterns of Modular Architecture - This was not quite what I had hoped.  I was looking for patterns, and instead got a fairly basic explanation of how to make your dependencies fairly one-way and isolated.  Neat I guess?  A decent refresher, but definitely nothing earth-shattering, and the most interesting part was OSGI, which the presenter covered for literally two minutes at the end of the presentation.  It did look promising, though.

Common Antipatterns and how to Avoid Them - This was well presented, but it was still a little light.  Good reminders on some common sense things NOT to do, and some reasonably interesting stuff out there - that this presentation was really hitting home with some of the attendees scared me - some of these were things that just shouldn't be happening now, and they obviously still are!

Architect for Scale - This was a good session - lots of 'things you should be doing when you get to this size/load'.  There were a fair amount of good notes about what was not enough, what was too much, and how to move up without huge disruptions

Automated Deployment with Maven - This was a huge win.  It was a discussion about how to use Maven plugins to streamline and automate the numerous steps that go into releasing a module of code.  With maven, it involves updating poms, branching, integrating, labeling, etc, and many manual steps.  With some of the tips in the presentation, I was able to make a releasable unit this morning in about 10 minutes with the help of some maven doc.  Why were we not doing this!?!?  Oh well - awesome.  Worth the price of the registration itself.

Architecture - Non-functional Requirements - This was more of a 'what goes into architecture' talk, than something specifically about non-functionals, but very interesting nonetheless.  As someone who is headed down that path, it validated much of what I think architecture is about.  There was some great back and forth in this presentation from the presenter of this session and the presenter of the earlier Antipattern session.

PowerMock - Mocking Statics - Supplemental Documentation

I have recently suffered through a bit of pain working with PowerMock.  It seemed extremely promising, as it sits on top of EasyMock, which we use for mock objects in our unit tests (it also works with Mockito, apparently), and promises to give you the tools to mock those pesky static calls that kill testability.  I had a few issues getting started, and they centered around jUnit 3.

PowerMock with jUnit 3

All the examples and documentation seem to be based on jUnit 4. This page has great examples on how to run your static mocking tests when running on jUnit 4, but how to do it on jUnit 3??  An extremely simple example of how to do this in jUnit 4 is here:

@RunWith(PowerMockRunner.class)
public class AwesomeTest {

  @PrepareForTest(StaticClass.class)
  public void testDoExecute() throws Exception {
    ...
    PowerMock.mockStatic(StaticClass.class);

    expect( StaticClass.doStuff( param ) ).andReturn( "yo!" );
    replay( StaticClass.class );

    //test some stuff!
  }
}

To accomplish the same thing in jUnit 3, without the annotation support, you can do this:

public class MyPowerMockSuite extends PowerMockSuite {

  public static TestSuite suite() throws Exception {
  
    return new PowerMockSuite(AwesomeTest.class);
  }
 
  public static void main(String[] args) throws Exception {
  
    junit.textui.TestRunner.run( suite() );
  }  
}

public class AwesomeTest {

  @PrepareForTest(StaticClass.class)
  public void testDoExecute() throws Exception {
    ...
    PowerMock.mockStatic(StaticClass.class);

    expect( StaticClass.doStuff( param ) ).andReturn( "yo!" );
    replay( StaticClass.class );

    //test some stuff!
  }
}

PowerMock and Multiple Classes

To mock methods from multiple classes, you will need to amend the '@PrepareForTest' annotation, just passing a string array like so:

public class MyPowerMockSuite extends PowerMockSuite {

  public static TestSuite suite() throws Exception {
  
    return new PowerMockSuite(AwesomeTest.class);
  }
 
  public static void main(String[] args) throws Exception {
  
    junit.textui.TestRunner.run( suite() );
  }  
}

public class AwesomeTest {

  @PrepareForTest({StaticClass.class,StaticClassier.class})
  public void testDoExecute() throws Exception {
    ...
    PowerMock.mockStatic(StaticClass.class);

    expect( StaticClass.doStuff( param ) ).andReturn( "yo!" );
    replay( StaticClass.class );

    PowerMock.mockStatic(StaticClassier.class);

    expect( StaticClassier.doStuff( param ) ).andReturn( "yo!" );
    replay( StaticClassier.class );

    //test some stuff!
  }
}

Gotchas

PowerMock 1.3.7 only works with EasyMock 2.5.2, which we have had some issues with.  I had to move back to PowerMock 1.2.5 to use EasyMock 2.4.  I didn't find a lot of issues with using this version, though we had some odd calls that were required where it told me not to stub a return when the method returned something, or where I had to stub out calls that didn't seem like they should be required.  With a simple static method that didn't really call many other static methods, it seems to work like a charm.

When Stupid Attacks (.bat file edition)

I'd just like to point out that today I spent about 3 hours trying to write a Windows batch file to do some string replacement. This script is only for developers and development machines. The same script took me seriously less than 5 minutes for Linux. Why so hard on Windows? Because their scripting environment sucks.

But, I stuck it out. I wrote a long and scary script that tried to take into account empty lines, commented lines, and lines that actually stores the properties that I wanted to write. I got something that worked 'mostly', and thought about spending even more time on it when I finally stopped being stupid and decided that I should be using the Windows port of sed. It was one of those times that I just wasn't thinking clearly in my haste to write some code. Just a friendly reminder - if it's hard and feels like a hack, it's probably the wrong thing to do, and there's probably something smarter to do - sometimes you just have to take a step back to do it.

New Homepage is Up

Check out the new and improved kirkandjena.com.


It's no longer one sentence and a link on a white background! I promise that the site will grow and improve as time passes. I have the basic template done, but there is definitely more to add, like bios, resumes, a feed of pictures from our gallery site, etc. I also want to add an 'apps' area that I can use to show off any sort of nerdiness I do in the evenings. This is a start, and not a bad one - it only took me a couple hours last night to do it. Yay for websites!

Struts 2 Error Reporting Tips

As I wrote before, Struts 2 isn't the best framework for error handling or notification. I think for the most part the entire framework is an incredible improvement over Struts 1, which had so many silly moving parts that just made it a genuine pain in the ass to work with. Struts 2 is simple, and I take well to the Action-As-Bean pattern used here. Moreover, it's what we use, so I had to learn to deal with the oddness of it. Since I wrote that post, I have come up with/implemented a couple handy tricks:

1) Display the Error Details on the Error Page

Obviously I didn't invent this, but we weren't doing it yet, so here's what I did:

You will likely create a global error mapping. In the action that you map to, add this code that will trap the error details:

Throwable error = (Throwable) request.getAttribute("javax.servlet.error.exception");

request.setAttribute( "errorCause", error.getCause().toString() );
request.setAttribute( "errorMessage", error.getMessage() );
request.setAttribute(  "errorStackTrace", error.getStackTrace() );

Now you can insert a little line into your error page that shows the error, instead of having struts swallow it up forever.

<%= if ( request.getAttribute( "errorMessage" ) != null { %>

-- print out the data --

<% } %>

2) Debug Better

If you get the source code for Struts 2, you can add it to your Eclipse project, and insert a breakpoint in the following piece of code in com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept():

public String intercept(ActionInvocation invocation) throws Exception {
    String result;

    try {
        result = invocation.invoke();
    } catch (Exception e) {

        /* BREAKPOINT HERE */

        if (logEnabled) {
            handleLogging(e);
        }
        List exceptionMappings = invocation.getProxy().getConfig().getExceptionMappings();
        String mappedResult = this.findResultFromExceptions(exceptionMappings, e);
        if (mappedResult != null) {
            result = mappedResult;
            publishException(invocation, new ExceptionHolder(e));
        } else {
            throw e;
        }
    }

    return result;
}

This way you can see if something weird happened inside struts that didn't get reported in the logs. Boom, you can see if there is some mysterious internal error getting thrown, rather than trying to divine it using rain dances and crossed fingers. These two things have really improved my experience with Struts. If I were a better person, I would upgrade to see if it's fixed, and if not, I'd probably add some decent logging.

Bonus Tip

If your application starts up, but you are getting invalid results from your struts action, like the "no result defined for action" message, make sure your source control tool didn't merge incorrectly and add a double mapping for the action. We ran across this recently, where an old mapping was included, and was picked up first. Perhaps there should be a way to tell Struts2 to error out and fail if it finds the same mapping twice in a file? Just something to watch for.

Happy Strutting!

Consuming RSS Feeds with Groovy/Grails

Building off of my recent post about RSS Feed parsing using the ROME Library, I had an idea for a fun application to build using Grails. The first part of this application, which I'll share when I am done, is the part where we read the feed. Now, the last time I did this in Java, it was pretty easy to do, but man, this is just a little bit better:

    def readFeed( url )
  {
      def xmlFeed = new XmlParser().parse(url);

      def feedList = []
    
      (0..< xmlFeed.channel.item.size()).each {

         def item = xmlFeed.channel.item.get(it);
         RSSFeed feed = new RSSFeed( item.title.text(), item.link.text(),
             item.description.text(), item.pubDate.text() )
         feedList << feed
      }

      feedList
  }

Yep, that's it. One line to pull back the feeds. The iterator, and one line to create my RSSFeed object. Then add the feed to a list, and return the list to your controller. In 25 minutes, I have a feed reader application that's basically functional, taking a feed as an input, and returning a page that displays the post title linked to the post, and the posts contents and date. All of life should be this easy.

Why Grails is Sweet

Grails is sweet because you do things that are relatively complex, like upload a photo, resize it, save it, and persist the metadata using this code in a controller class, and nothing more:

        def photo = new Photo(params)
Member m = authenticatedMember()
def myFile = request.getFile( "file" )
def imageTool = new ImageTool()
photo.path = "";
photo.member = m

if(myFile && photo.save())
{
String imagepath = grailsApplication.config.imagePath +
File.separatorChar + "${photo.id}.jpg"
myFile.transferTo(new File(imagepath))

imageTool.load(imagepath)
imageTool.thumbnail(640)

String fixedImagePath = grailsApplication.config.imagePath + File.separatorChar + "${photo.id}-fixed.jpg"
imageTool.writeResult(fixedImagePath, "JPEG")
imageTool.square()
imageTool.swapSource()
photo.path = fixedImagePath
}

Oh by the way, when you define a domain class with a byte array to store a file, then generate the edit/create view, it automatically does all the multipart form submission stuff in the .gsp file. It NEVER gets this easy with Java/JSP. Hmm. Awesome. This is made possible by the innate goodness of grails and a plugin called ImageTools. It makes me dread going back to work and using Java sometimes. Sigh. It's amazing how much you can get done in a short time with this framework, and the more I have a-ha moments with it, the more I think I won't be going back to just Java when I make the switch full time, especially in the knowledge that whether it's Groovy/Grails or JRuby/Rails, I can still fall back on familiar Java libraries. Yessir, this is a good time to be a nerd...

jQuery Goodness

This week I did a lot of client-side development.  Usually, this would be reason to cry and/or drink heavily, but this week, it was one of the more pleasant parts of my week.  I thought that I'd go ahead and share a one of the goodies that I experienced.

Thickbox

Thickbox is an awesome library that pops up a page that you defined as a dialog that can be modal if you specify that option.  It's pretty peppy, and looks sharp.  Thickbox uses decoration of links on a page (set up by specifying a class of 'thickbox').  Anchor tags are set up to register an onclick event that pops up a page that you specified, be it a static page, or the result of a server-side call.  Anything can be displayed in a thickbox.  It can be information, or a form that you submit back to the server.  It's been pretty simple for us, but I did run across an issue today.  I am loading some pre-rendered html into another div that displays as a floating popup on the page.  One of the links there needs to call an overlay that is loaded via thickbox.  The button was defined as an input, which should work, but all the documentation refers to anchor tags, so I figured that we could get around it by doing this:

<script type="text/javascript">

function showOverlay {

jQuery( '#floatingDiv' ).hide();
jQuery( '#hiddenTbLink' ).click();

</script>
<input type="button" id="id" onclick="javascript:showOverlay();" value="showIt">
<a id="hiddenTbLink" style="display: none;" class="thickbox" href="http://www.blogger.com/path/to/action"></a>

This hides the floating div and invokes the click action on the anchor tag that contains the thickbox element.  This should work, right?  Wrong.  The html is dynamically inserted into the DOM after the document.ready code is called.  That is where you perform the thickbox initialize call.  So, what we had to do is add call to tb_init() in the code that loads the floating div, thus doing the jQuery DOM manipulation and allowing the thickbox-enabled link.  Much better now.  I sat back with glee, realizing that in a few hours, I had enabled this floating div and figured out how, from that floating div, I could make a call to load a beautiful thickbox element.  It was all too easy.

Field Guide to Setting Up Your Home Webserver

Well, it's taken a while, but I finally have http://kirkandjena.com up and serving pages from my server at home. This has been much more difficult than it should have been and the saga features four different operating systems, two DNS servers, multiple calls to my cable Internet provider, and more than a few headaches. I think that there is very little thorough documentation out there about setting up a website from your house, so let me take a stab at it.

First the background:

I have a Dell SC430, an old piece of crap small business server that is pretty stripped down. I had OpenSUSE 10.2 installed on it initially, which at first was fine - it seemed to agree with my system in a pretty plug-and-play fashion. I was pleased enough with the UI, and it performed great considering it really wasn't a very high-horsepower setup, but then bad stuff happened, and I moved onto Ubuntu 8.04. Now, I have already described my admiration for Ubuntu 8.0.4. It seems to come with most everything that you need on your server. My biggest problem with Ubuntu wasn't really a problem with Ubuntu I guess - the resolution on my 22-in monitor wouldn't be recognized, no matter what I tried. Apparently, this has to do with the bobo graphics card (Radeon ES1000) that is delivered with the Dell SC430. Anyway, long story short, the screen was awful to work with, and I had heard great things about OpenSUSE 11.0.

I gave it a shot. The screen resolution was recognized right away. That was good - it was very fast, and had a great look to it. The rub was, every time I tried to setup the network, I would lose my network connection altogether. Even when I hit finish after making no changes, it would hose my connection completely, and I would have to uninstall and reinstall my network driver. Not that great. The package manager is much better on SUSE 11 though. So, I had an epiphany. My loyal readers will remember that RCN screwed me out of running a media center PC at home, because they are bad, bad corporate citizens. I thought, why don't I make this server just a server that I only connect to using ssh, and use the idle media center as my computer at my desk. Eureka! I downloaded Ubuntu Server Edition 8.10, since Ubuntu did what I wanted it to. Within mere minutes, I had a server installation loaded up, no problem.

Now that I have woven this tale of woe, it's time to actually talk about what I meant to talk about: how to actually set yourself up to have a domain name and a webserver.

Step 1) Get your domain name. I use godaddy.com. It's a terrible website, but it is pretty cheap and simple enough.

Step 2) Get a static IP - this is more or less expensive based on whether or not you have an evil ISP. Despite RCN's lame decision to force cable boxes on the masses, they have REALLY cheap business internet and static IP costs. I pity the fool who uses Verizon DSL. It was MUCH more expensive through them.

Step 3) Set up your router - the steps I am describing are for a netgear wireless router. I assume it's pretty much the same no matter what router you have as long as it's not a complete piece of junk, and was built recently:

a) Set a static IP address for your server - this can be achieved by mapping an IP address to your server's mac address. This means that every time your server receives an IP address, it can be one of your choosing.
b) Setup port forwarding - now that you have it established that your server is listening on port xxx.xxx.xxx.xxx, you can setup port forwarding. The obvious port would be 80, the default http port. You can point 80 over so that all http requests to your ip address are routed to the web server on the server.

Step 4) Set up your entry in a Domain Name Server (DNS). You need to tell the Internet that your domain name is associated to your statically assigned IP address. This is accomplished by adding an entry to a DNS somewhere. There are a bunch that you have have to pay for, but thankfully, there are a few that are free, and seem to work pretty well. I personally used everydns.net. It's simple to say "associate this name with this IP address". Then go back to your hosting account, and set the nameservers on the domain name to point to the nameservers you just setup. Now the web knows that 195.12.4.1 points to myawesomesite.com. This takes a while to propagate across the web, so give it time. Once it gets to your ISP though, you will be able to type your domain name in and have it resolve to your IP.

This is the setup that takes place outside of your server. Now you can move on to the good bits, setting up your content, your webserver, and any operating-system level configuration required to serve pages to the internet.

Step 1) If you don't have it already, get apache. There are about a billion tutorials on installing apache, so I won't bother you with the details there. Once you have it installed and running, you should be able to type http://localhost and see the stock apache page that says you have it installed. Now you are ready to do your thing.

Step 2) Decide how you want to host your content. Do you want to host just one directory worth of stuff, or do you want to have a slightly more complicated setup. I want to use kirkandjena.com for more than just one directory. I would like to have a homepage that can link to as many things as can think of to host. Here's how I did that:

a) set up my web root at /my/path - all you have to do is have an index.html, or index.php - whatever start page is defined in your apache configuration - just search for 'DirectoryIndex'. Add your homepage there.

b) setup as many directories of other content that you want. These directories will be referred to by subdirectories. kirkandjena.com/photos is an example of this. These will host the content of your 'applications'.

c) now setup your virtual host. There are quite a few examples on how to do this on the web. The most clear documentation I have found is here. The idea is that you want to tell apache that if it gets a request with this path and this URL, on this port, here is what you want it to do. In this case, I may want to host multiple sites, but currently just kirkandjena.com. So I define one virtual host, saying if you get this request, here's the pages you should serve:

 
<Directory /photo/app/directory>
 
Options FollowSymLinks
 
AllowOverride Limit Options FileInfo
 
</Directory>
 
<VirtualHost kirkandjena.com>
 
DocumentRoot /my/content/path
 
ServerName kirkandjena.com
 
Alias /photos "/photo/app/directory"
 
<directory "/photo/app/directory">
 
Options Indexes
 
AllowOverride None
 
Order allow,deny
 
Allow from all
 
</directory>
 
</VirtualHost>

Step 3) Setup your /etc/host file. This file will tell your machine that when things get requested on certain IPs, do this:

127.0.0.1 localhost
127.0.1.1 ubuntu-server
xxx.xxx.xxx.xxx kirkandjena.com

The IP address for your site comes from the settings above (whatever IP address you assigned to your server). Restart apache for good measure after updating this file.


So that's it - now restart apache, and if your directories are in the right place, you should be all set to start serving your pages to the world. Hope this helps, and feel free to chime in with some pointers or corrections.

This Just In:

From the "don't do this on your site" file:

If you are going to have a form that requires a captcha to submit, and you choose to display virtually illegible captchas (see below), then don't zap the form contents into the ether when your user inevitably gets the captcha wrong. (Talking to you, mybloglog). I wrote this nice long post to support to let them know there was an issue I didn't really care about, but thought they might, and now it's just lost in the webs.

Boo.

Mac Tricks I Should Have Been Using a Long Time Ago

As I mentioned when I first got my Macbook, I think that sudo sucks. What I failed to do was make my life any easier by developing some shortcuts.

I just finally got around to adding a bunch of aliases to my /etc/profile file:

alias tomcatDebug='sudo /tomcat-dir/bin/catalina.sh jpda start'
alias tomcatRun='sudo /tomcat-dir/bin/catalina.sh run'
alias tomcatStart='sudo /tomcat-dir/bin/catalina.sh start'
alias tomcat='sudo /tomcat-dir/bin/catalina.sh'
alias apacheStop='sudo /apache-dir/bin/apachectl stop'
alias apacheStart='sudo /apache-dir/bin/apachectl start'

I am going to add about a million of these, because I constantly do way too much switching directories and typing. Any self-respecting nerd probably has stuff like this, but for the rest of us dummies, hopefully this will help.

PandaStream, or Why The Internets Are So Cool

Tonight, as if I didn't do enough nerdy stuff during the day at my job, I decided I was going to try out PandaStream. As I noted, we had a bit of trouble with our video provider, and my colleague Bob, had stumbled across this open source video package. We can own the entire process, without having to requisition the machinery to support video upload, encoding, and content serving. This is a good thing, as long as it works. Well, I can say it works. Before I go on to describe how it all comes together, let me just describe the surreal realizations I came to...

I am sitting on my couch, wirelessly connected to the internet, creating an application instance in the cloud on Amazon EC2, then deploying a git project to the server running Ruby/Merb, configured to interface with Amazon S3 and Amazon SimpleDB. In 2 hrs, on my couch, I am able to set all this up, using the great set of instructions here. 5 years ago, wireless internet was just getting solid. 5 years ago, I am pretty sure that if Merb had actually been invented, that very few people outside of its creator knew what the hell it was, and Ruby was still a pretty obscure scripting language invented by a Japanese guy. Amazon's web service offerings in cloud computing were but a twinkle in Werner Vogels' eye. Open source applications were gaining steam, but certainly not as trusted or ubiquitous as they are today. I think that this is all pretty remarkable, and it's a testament to many creative and brilliant folks that I am able to sit around on my couch and do these amazing things and have them feel pretty routine. Thanks, smart and creative people. Us mere mortals are in your debt.

So anyway, I had an EC2 account already setup, and bringing S3 and SimpleDB online were a snap (once I realized I needed SimpleDB). Getting the security certs and stuff took about 5 minutes to resolve, thanks to Amazon's great AWS Documentation. I kicked off the panda instance without issue, hopped into the instance via ssh, and jumped into the merb console. This didn't go smoothly as I was presented with a security error that just said I couldn't be authenticated using my credentials. To test my credentials, I grabbed a java library for AWS. I used this library to test my creds with S3, and it worked. Only then did I go and look closer at the error and realize it was related to simpleDB, which I didn't have an account for. Creating that made it work much better. Once I was able to hop into the merb console, there were about 10 commands to run to set up, add security, and configure some default video encoding. This took maybe 5 minutes. From there, I logged into the instance and uploaded a video. Then I took some embed code and stuck it in a test html file. It just worked.

Anyone with any experience with pandastream, we are really curious to see if this is production tested, or if anyone knows anything about it that would make us NOT want to use it.

Next episode, integration via REST APIs. Stay tuned.

What's Awesome? Fuser.

So I was poking around in the blogosphere and I saw something about Fuser and its latest release. It's a browser app that aggregates all your mail and your myspace and/or facebook account into one webpage, with tabbed view like Yahoo mail.


I need something like this, because I use 5 different computers when you count the dual-boot laptop. All my stuff is scattered in different sent mail folders in Live Mail on Vista, Evolution on Ubuntu, and Thunderbird on my Mac at work. I would rather it be in one place, and with Fuser I get that. It's got a pretty good UI, and seems to work relatively quickly. Setting up your accounts (I have hotmail, yahoo mail, gmail, and an IMAP mail account) is quick and painless, and the initial sync really isn't too painful.

All the basic features are there, except for one thing: Search. (Edit - Emily from Fuser has enlightened me - it's tucked away in the top right, and isn't that peppy, but certainly better than nothing!)

I did have a problem using this site on Firefox 3.0.3 on Mac Leopard, but it turns out it was just something weird with the latest Java update. I found a great post, describing my symptoms, and once I followed the instructions, the site seemed to work again there.

The only thing that would be really great would be to make this site work in Fluid for the Mac, but currently the browser certification doesn't allow to load this page in Fluid. I plan to try tonight in Prism to see if that works instead.

What's That, Amazon?

Now, before I start writing, I have to confess - I love Amazon's recommendation system. I frequently head there whenever I want to find a new book, a new album, or perhaps a new band to check out. 95% of the time when I go there, I think to myself - wow, that's pretty good - I really did enjoy that. They have done a good job harnessing the 'wisdom of crowds' in a way that produces strong output that keeps me coming back. That's why I found this one so funny:


Note the reasoning behind the recommendation of the Opera album:

Recommended because you rated Mother's Milk and more.

I clicked the "fix this button", and it showed me that I got there because I rated Mother's Milk by Red Hot Chili Peppers and Blue Train by John Coltrane. I can't fix that I liked those, so I have to sit idly by and hope that saying I am not interested will result in Amazon clearing it's head of the thought that I want to listen to the great tenors of our time sing opera to slap bass played by Flea and jazz sax by Coltrane. So strange!

Software Estimation Is Hard

So as part of my new role at Sportsvite, I am taking over the leadership of development on partner projects. These are usually slightly smaller projects that are either directly on our site, or somehow interfacing with our site. While they are smaller, they are a lot MORE of them. Like, currently we have six projects ongoing, and I have to estimate all of them.

I don't know how other software developers feel about estimating projects, but for me, it's always the most difficult part of my job. If I developed software like I estimated it, I would be in serious trouble - it would go something like this:

"Here is my code. It may or not work, depending on A, B, and C, and it could be right, but probably not, and you could probably take the expected response time and multiply it by 1.5 to get a more accurate number."

That probably wouldn't fly, right? But for some reason, in software estimation seems to work like that. I think that's kind of crazy, but there hasn't been any better methodology put out there, really? Has there?

From wikipedia on sofware estimation:

The ability to accurately estimate the time and/or cost taken for a project to come in to its successful conclusion is a serious problem for software engineers. The use of a repeatable, clearly defined and well understood software development process has, in recent years, shown itself to be the most effective method of gaining useful historical data that can be used for statistical estimation. In particular, the act of sampling more frequently, coupled with the loosening of constraints between parts of a project, has allowed more accurate estimation and more rapid development times.

So I am not alone! This is a problem for other people, which makes me feel better. There are methodologies:

Function Point Analysis
Proxy-Based Estimating
Evidence-Based Scheduling

The problem is that, especially in a small shop, we

a) don't have comparable projects that can be used to provide historical data, especially for partner projects that are often more one-off than core product development. I can say with some confidence how long it takes to add a new Struts 2 action and mapping - that part is easy, but what about the elements on the page? What about the logic?
b) don't have the infrastructure in place to really record time spent on any one task, and often find that if we did, it would be inaccurate because we wear so many hats, each hat being taken on and off randomly throughout the day/week.
c) can't afford just yet to spend a lot of time making exhaustive estimates, because that's time that won't be spent in construction and testing.

I wish there was a silver bullet that you could shoot at this problem, but it seems like there isn't. Like anything else, there has to be a point where you make something a priority to really solve it, but it's become such a joke in the industry it seems, that people aren't trying too-too hard to really get good estimates.

Ah well, if anyone has an idea, or a good experience as a successfull estimator at a small software outfit, please feel free to leave a comment describing how you did it well.

Obscure Things Every Java Developer Should Know Depending On Who You Talk To

In my career, I have found that the best way to learn something is to give a presentation on it. In lieu of that, perhaps writing about it would be a good way to make it sink in, and hopefully someone else can stumble across it and benefit from it as well. Here are some of those things:

hashCode() v. equals()

The following comes from this awesome site javapractices.com:

All objects have both identity (the object's location in memory) and state (the object's data). The == operator always compares identity. The default implementation of equals compares identity as well. Fun facts:

• if a class overrides equals, it must override hashCode
• when they are both overridden, equals and hashCode must use the same set of fields
• if two objects are equal, then their hashCode values must be equal as well

From an interesting blog post on hashmaps and hashcode():

When an object is inserted into a HashMap, the position at which the Value object is inserted in its internal Entry-array depends on the hashCode of the Key object passed in. The hashCode generated by the Key object is not used directly, but is processed by the hash(k) method.

Immutable Objects

What is an immutable object?

Immutable objects are simply objects whose state (the object's data) cannot change after construction.

Why are they so good? They:

• are simple to construct, test, and use
• are automatically thread-safe and have no synchronization issues
• do not need a copy constructor
• do not need an implementation of clone
• allow hashCode to use lazy initialization, and to cache its return value
• do not need to be copied defensively when used as a field
• make good Map keys and Set elements (these objects must not change state while in the collection)
  

Wanna learn this stuff visually, without paying tuition??

CHECK THIS OUT - Cal Berkely Data Structures Lectures (kinda awesome)

This is one of those "man the internets are sweet" type things. I will use this to learn all the nasty bits that I don't really want to but might have to.

Nerdy Things I've Learned Today

1) LAMP should be easier. Today, on a new server that had mysql, apache, and php already installed, I had to just through flaming hoops of package dependency hell to recompile php from source using the correct modules to allow for use of mysqli. Seems kind of wacky to me. Rebuild? To use a library? Boooooo. Anyway, thank goodness for google. Anyone who wants an incredibly concise guide to building a LAMP environment on Linux, check this one out - it helped me a lot.

2) The scp command will attempt to eat all your upload bandwidth. It is a greedy command. You can 'throttle' the bandwidth consumption using the -l flag (the number is kb/s):

scp -l 50 large-local-file root@awesome-remote-server:/great-directory

This is the best way I have found to teach scp to share.

3) You need good internet connection to work as a developer if you are serious about it. Ugh.

Tomcat: Manager Loading Issues

Here's another 'good samaritan', 'hopefully this'll help someone down the road' post. I recently put a clean Tomcat 6.0.18 instance on my machine, and deployed my app. Everything worked fine with the app, but when I tried to access the manager application, I wasn't prompted for the login credentials like expected. I figured it might have been an issue with my (customized) tomcat-users.xml file, but it wasn't - when I checked out the manager log file, I saw a bunch of references to this error:

javax.xml.parsers.FactoryConfigurationError: Provider org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found

I was just getting a 401 page with the message "authorization is required" or something to that effect. I did some digging, and found a bunch of halfway answers that included reinstalling Tomcat - this isn't really acceptable, so I figured I'd look in tomcat/lib, and sure enough there was no xml jar there, so I copied in xerces-impl-xxxx.jar in there, restarted, and could access manager no problem - not sure why this happened, but if you ever see that, make sure you have a xerces parser in your classpath...