Grails Image Processing with the ImageTools Plugin

I recently had a requirement on my side project to add some simple image processing.  I had previously used the ImageTools plugin, but that seems to have falled into a bit of disrepair.  I had a multistep process to get this going, and I figured I'd share it.

According to the git repository, the last checkin on version 1.0.4 was way back in January 2010.  Grabbing the plugin using the standard grails install-plugin as documented on the plugin documentation page just doesn't cut it.  When attempting to use the documentation, you get a message that tells you

unable to resolve class ImageTool @ line xxx, column xxx. def imageTool = new ImageTool();

This is troubling, as that is the only class included in the plugin.  Fortunately I found this comment on a closed (???) issue on github:

So, to solve this I had to run a git clone on this source tree and then follow the instruction at the link below to create my own version of this plugin with correct package.

http://grails.org/doc/latest/guide/12.%20Plug-ins.html

So it would seem that you need to grab the source, then create a plugin.  This is not as easy as you might think.  First of all, you might not have git installed.  Follow the instructions here to 'git git'.  Next, you need to grab the code.  From the command prompt in the directory you want the code, run the following command:

git clone http://github.com/ricardojmendez/grails-imagetools.git

Now you have the code.  Even though it's a plugin, you have to tell grails it's a plugin, by running the grails create-plugin command.  Now you have a zip file that you can install in your project, using the standard grails install-plugin command.

From there you can follow the instructions.  The import statement is recognized, and we can get finally get down to actually writing the controller code:

import org.grails.plugins.imagetools.*

...

//the controller method we submit to
def myProfileSave = {

...

uploadPhoto( file, user )

...

}

def uploadPhoto( upfile, user ) {

 if(upfile.empty) {
  flash.message = 'File cannot be empty'
  redirect( action: 'myProfile' )
 }

 def imageTool = new ImageTool()

 imageTool.load( upfile.getBytes() )

 // Crops it to a square
 imageTool.square()
 
        //make the new squared image the image to operate on 
 imageTool.swapSource()
  
 // Reduces the image size
 imageTool.thumbnail( 125 )

 def fileBase = grailsApplication.config.imageLocation
 def fullPath = "${fileBase}/user/photo"
        File fullPFile = new File( fullPath )

 if ( !fullPFile.exists() ) {
  fullPFile.mkdirs()
 }
  
 def filePath = "${fullPath}/user_${user.id}_125.jpg"

 File toMake = new File( filePath )

 if ( !toMake.exists() ) {
  toMake.createNewFile()
 }
  
 // Saves the result
 imageTool.writeResult( filePath, "JPEG" )
}

Once you get the plugin working, it's a breeze, as you can see!  It's a great, simple wrapper around the nasty and complicated JAI library.

Some Tips:

1) You'll probably want to increase your heap size if you are running with a default or small heap.  This requires a bit of memory.
2) Remember to specify a multi-part form.  A good explanation of this can be found in the grails documentation.
3) Profit!

UberConf, Days 2+3

After a great day one, we were looking forward to another great, long day, and we weren't disappointed.

Scala for Java Programmers - Day 2 started with a fun Scala workshop by Venkat Subramaniam that spanned two sessions.  This was extremely hands-on, and very worthwhile.  I learned enough about a language I have had interest in for a while to get started on my own, and got a nice little dose of Computer Science and compiler fun too.  My only complaint here was that we spent too much time using closures and Scala syntax craziness to write code in one, perl-ish, unreadable line.  We should have used the time writing 'terse' obfuscated code playing more with Scala parallel programming.  Actors are nifty, and it's amazing how clearly you can express complicated multi-threaded program flow.  If I had access to this goodness, I can think of one project I did at Blackboard that could have been written in half the time, in 1/5 the LOC, and in a much more readable and intuitive way.  Hindsight is 20/20, I suppose, but man that made me burn a bit.

Emergent Design - After lunch, we attended Neal Ford's talk on emergent design.  This was a good talk, mostly because Neal Ford is a good speaker, but I wouldn't say this was anything new.  No big design up front, test first, don't solve problems that don't yet exist, as this leads to heaps of code debt.

Stability Antipatterns - This was a workshop by Michael Nygard about common antipatterns you see as it relates to stability - this was more on the operations and integration level than at the code level, and had a fair amount of good tips for things to remember - mostly how to keep your application from dying because there is an external resource that you don't control that is failing.  Some of the horror stories he shared were hilarious, and some you could totally see happening to you (if they hadn't already).  Good session.  Good reminder of things to consider that often fall by the wayside.

On to Day 3, which was kind of disappointing.

Implementing Domain-Driven Design - I was excited about this one because DDD is one of those things you always hear about, but never really get.  After the session, I still didn't really get what the big deal was - drive the design of your application by the real-world domain.  Seems pretty simple to me.  What wasn't simple was the 100 slides of UML class diagrams the presenter used to demonstrate DDD.  Yikes, not that awesome.

SOLR - A Case Study - This was an interesting enough presentation, as Solr is something that we are evaluating.  I learned enough to know that we didn't really need to do this right away, but not much more.  It wasn't really a case study.  What I would have liked was either, Solr real-world example, or Migrating from Lucene to Solr, but alas we didn't really get either.  It was still a good presentation though, just not what I was looking for.

Android Mobile Development - Gah!  Fail!  This could have been a great way to close things out, but the presenter instead did the presentation completely out of order, gave nobody any opportunities to code, and was just generally all over the place.  I was at least able to use the time to make my way through some of the Android tutorials that Google provides.  Man what a bummer.

Uberconf Impressions, Day 1

Last week a group of us from eCollege attended the Uberconf presentation here in the Denver area.  I figured I'd post a few impressions:

First off - the venue - the Westin in Westminster, CO.  Nice place.  Really awesome food whether you are a vegetarian or a carnivore.  Free drinks all day.  Decent coffee, although the cups were oddly small.  One note to the Westin - people operate better at temperatures higher than 58 degrees.  It was FREAKING COLD in there.

Now on the conference organization.  This was one of the best flows of any conference I've attended.  There weren't too many people working the conference, and it still went well.  Easy registration process, good marketing materials (4GB USB stick - not bad), and two t-shirts I would be super embarrassed about wearing.  Easy to figure out what sessions were coming up and where they were going to be.  One minor nit - it's a developer conference, which means laptops - next time, get more power strips in the rooms, at least for the workshops.

And, finally the important stuff - the sessions (day 1):

Patterns of Modular Architecture - This was not quite what I had hoped.  I was looking for patterns, and instead got a fairly basic explanation of how to make your dependencies fairly one-way and isolated.  Neat I guess?  A decent refresher, but definitely nothing earth-shattering, and the most interesting part was OSGI, which the presenter covered for literally two minutes at the end of the presentation.  It did look promising, though.

Common Antipatterns and how to Avoid Them - This was well presented, but it was still a little light.  Good reminders on some common sense things NOT to do, and some reasonably interesting stuff out there - that this presentation was really hitting home with some of the attendees scared me - some of these were things that just shouldn't be happening now, and they obviously still are!

Architect for Scale - This was a good session - lots of 'things you should be doing when you get to this size/load'.  There were a fair amount of good notes about what was not enough, what was too much, and how to move up without huge disruptions

Automated Deployment with Maven - This was a huge win.  It was a discussion about how to use Maven plugins to streamline and automate the numerous steps that go into releasing a module of code.  With maven, it involves updating poms, branching, integrating, labeling, etc, and many manual steps.  With some of the tips in the presentation, I was able to make a releasable unit this morning in about 10 minutes with the help of some maven doc.  Why were we not doing this!?!?  Oh well - awesome.  Worth the price of the registration itself.

Architecture - Non-functional Requirements - This was more of a 'what goes into architecture' talk, than something specifically about non-functionals, but very interesting nonetheless.  As someone who is headed down that path, it validated much of what I think architecture is about.  There was some great back and forth in this presentation from the presenter of this session and the presenter of the earlier Antipattern session.

Grails Wizardry: Mocking all your web services with Grails

Background

At work, we are using EC2 to run capacity tests on our environment, which would normally be pretty trivial, but we are a big SAAS shop with tons and tons of endpoints internally.  Without setting up EVERYTHING in the cloud, we can't really run our tests...or can we?  We would love to use Amazon VPC for EC2, but it requires a fair amount of intervention from your network team.  That will be a great alternative down the road - being able to point back into the internal network to access machines rather than deploying them in the cloud.  So, plan b - Using grails, I was able to mock every single .NET service that our application talks to in about three hours.  Here are the details:

We use SOAP services that return some XML, I simply used Fiddler to see what responses are sent back over the wire for each service, and I was ready to rock.  There are two main pieces to the puzzle:

MarkupBuilders


Generating XML is almost too easy in Groovy.  Using the instructions here, I was able to write some code that spits out some xml based on info passed in.  For instance this sample Grails controller will take a login and company and return an XML document with a user id and company id:

import groovy.xml.MarkupBuilder

class UserController {

  /**
  * /MyService/users/getuser?login=myuser&company=mycompany
  */
  def returnUser = {

        def user = params.login
        def comp = params.company

        def writer = new StringWriter()
        def xml = getXMLBuilder( writer )

        xml.'user'('xmlns:xsi': getNsXsi(), 'xmlns:xsd': getXsd(), 'xmlns': getXmlNs() ) {
          userId( "${comp }@${user}" )
          compId( "${comp}@compId" )
        }

        render( text: writer.toString(), contentType:"text/xml", encoding:"UTF-8" )
  }

  def getXMLBuilder( StringWriter writer ) {

        writer.append(  "" )

        return new MarkupBuilder( writer )
  }

  def getNsXsi() {
        return "http://www.w3.org/2001/XMLSchema-instance"
  }

  def getXsd() {
        return "http://www.w3.org/2001/XMLSchema"
  }

  def getXmlNs() {
        return "http://yourcompany.com/UserStuff/2009/11/01"
  }
}

Now that you have defined your controller, you need to make it so your application can make its normal request to the url it has configured but still get to your controller.  That's where the URLMappings.groovy class comes in:

UrlMappings

The UrlMappings file already exists by default when creating a project.  This file is built to map external urls to your controllers - you can use it to create nice shiny RESTful URLs, or just to do some utilitarian mappings for backwards compatibility, etc.  Obviously not every project's requirements map to the grails conventions on how urls map to controllers.  To add access to your controller from a different url, you just need to add the following snippet - this will allow Grails to map your request from something like

/MyService/users/getuser?login=me&company=acme

to

/user?login=me&company=acme 

    "/MyService/users/getuser" {
          controller = "user"
          action = "returnUser"
    }

More information can be found at this link.

At this point, you have a service running that will return the valid XML for your service, and will respond to a call at a URL that doesn't actually match.  Now you can rinse and repeat for each service that you need to mock out.  Happy Grailing!

PowerMock - Mocking Statics - Supplemental Documentation

I have recently suffered through a bit of pain working with PowerMock.  It seemed extremely promising, as it sits on top of EasyMock, which we use for mock objects in our unit tests (it also works with Mockito, apparently), and promises to give you the tools to mock those pesky static calls that kill testability.  I had a few issues getting started, and they centered around jUnit 3.

PowerMock with jUnit 3

All the examples and documentation seem to be based on jUnit 4. This page has great examples on how to run your static mocking tests when running on jUnit 4, but how to do it on jUnit 3??  An extremely simple example of how to do this in jUnit 4 is here:

@RunWith(PowerMockRunner.class)
public class AwesomeTest {

  @PrepareForTest(StaticClass.class)
  public void testDoExecute() throws Exception {
    ...
    PowerMock.mockStatic(StaticClass.class);

    expect( StaticClass.doStuff( param ) ).andReturn( "yo!" );
    replay( StaticClass.class );

    //test some stuff!
  }
}

To accomplish the same thing in jUnit 3, without the annotation support, you can do this:

public class MyPowerMockSuite extends PowerMockSuite {

  public static TestSuite suite() throws Exception {
  
    return new PowerMockSuite(AwesomeTest.class);
  }
 
  public static void main(String[] args) throws Exception {
  
    junit.textui.TestRunner.run( suite() );
  }  
}

public class AwesomeTest {

  @PrepareForTest(StaticClass.class)
  public void testDoExecute() throws Exception {
    ...
    PowerMock.mockStatic(StaticClass.class);

    expect( StaticClass.doStuff( param ) ).andReturn( "yo!" );
    replay( StaticClass.class );

    //test some stuff!
  }
}

PowerMock and Multiple Classes

To mock methods from multiple classes, you will need to amend the '@PrepareForTest' annotation, just passing a string array like so:

public class MyPowerMockSuite extends PowerMockSuite {

  public static TestSuite suite() throws Exception {
  
    return new PowerMockSuite(AwesomeTest.class);
  }
 
  public static void main(String[] args) throws Exception {
  
    junit.textui.TestRunner.run( suite() );
  }  
}

public class AwesomeTest {

  @PrepareForTest({StaticClass.class,StaticClassier.class})
  public void testDoExecute() throws Exception {
    ...
    PowerMock.mockStatic(StaticClass.class);

    expect( StaticClass.doStuff( param ) ).andReturn( "yo!" );
    replay( StaticClass.class );

    PowerMock.mockStatic(StaticClassier.class);

    expect( StaticClassier.doStuff( param ) ).andReturn( "yo!" );
    replay( StaticClassier.class );

    //test some stuff!
  }
}

Gotchas

PowerMock 1.3.7 only works with EasyMock 2.5.2, which we have had some issues with.  I had to move back to PowerMock 1.2.5 to use EasyMock 2.4.  I didn't find a lot of issues with using this version, though we had some odd calls that were required where it told me not to stub a return when the method returned something, or where I had to stub out calls that didn't seem like they should be required.  With a simple static method that didn't really call many other static methods, it seems to work like a charm.

On Intiators, Blockers, Supporters, and Observers

In reading Sway: The Irresistible Pull of Irrational Behavior, I found myself fascinated with the chapter on the four personality types -



Initiators, who have ideas and drive innovation, and are the general optimists.
Blockers, who are the doom-and-gloom, are you sure that's a good idea types.
Supporters, who pick the side of the initiators or blockers.
Observers, who simply provide comments.


I have done a lot of thinking about what the right balance of personalities is for a team.  In a perfect world you want a bunch of initiators, because they can just think up all kinds of good ideas and knock them out, moving the team and the organization forward.  This generally doesn't end up happening this way.  I can speak from experience when I say that having a manager who is an initiator and an employee who is an initiator is not necessarily a good mix.  This can lead to something of a whirlwind of work that might all be interesting, but isn't necessarily good for the organization.   


You also don't want a lot of blockers and nothing else, because you will sit around spending endless hours justifying why something has to get done, even when it seems relatively clear why it's a good idea.  This is the other end of the spectrum.  Instead of doing too much and having some of it not be the right stuff, you end up doing not enough of anything.  People get fired because of this.


A perfect balance in my opinion is to have a team made up of an initiator, a blocker, a supporter, and to have the manager be an observer.  The initiator gets people fired up, introduces ideas and enthusiasm, and generally dives in with gusto.  The blocker is there to make sure the initiator doesn't get carried away, questioning whether something is the right thing to do, or whether it's necessary.  The blocker makes the initiator better, because in order to do something new, the initiator knows they will have to pick their battles well, and to justify their enthusiasm with a business case that can sway the supporter and the observer.  In my opinion, a manager should be the observer, but can play a bit of a supporter role as well.  A manager who is a blocker won't get very far, and a manager who is an initiator can be seen as a little too gung-ho or overzealous by their employees.  As an observer, the manager can interject when they see fit, shaping the way that the work evolves without forcing their will on anyone.  This makes for happy employees.


It's just interesting because my office now sort of fits this dynamic, and the give and take is really great.  Coming from other places where the balance was definitely lacking, I think it's a great place to really grow.  Blockers can become more open to new ideas, and initiators can learn to harness their enthusiasm and be more thorough.

When Stupid Attacks (.bat file edition)

I'd just like to point out that today I spent about 3 hours trying to write a Windows batch file to do some string replacement. This script is only for developers and development machines. The same script took me seriously less than 5 minutes for Linux. Why so hard on Windows? Because their scripting environment sucks.

But, I stuck it out. I wrote a long and scary script that tried to take into account empty lines, commented lines, and lines that actually stores the properties that I wanted to write. I got something that worked 'mostly', and thought about spending even more time on it when I finally stopped being stupid and decided that I should be using the Windows port of sed. It was one of those times that I just wasn't thinking clearly in my haste to write some code. Just a friendly reminder - if it's hard and feels like a hack, it's probably the wrong thing to do, and there's probably something smarter to do - sometimes you just have to take a step back to do it.

Facebook Connect Users Beware!!!

We just ran across a bug that must have been introduced very recently, where all users with new Facebook accounts were unable to access our application when 'connect'-ing their user name to Facebook. It was a weird issue, because I was able to connect my account with no issue. I finally noticed that the Facebook profile ID we were storing was the same across all users who were running up against this issue.

Truncation issue!

We had defined it as a 12-length INT, which really doesn't matter - the biggest value an INT can store in MySQL is 2147483647. Anyhow, updating the table to store the value as a BIGINT fixed the problem, but I can't image we're the only ones to run across this mess, and the FB Connect documentation is weak enough that I have never seen anything about suggested storage. This must have happened in just the last few days, so hopefully others will come across this and make the change proactively if needed.

Every Now and Then I Get Lightbulbs

I have a good idea (I think). I have run an idea I had this morning by two people whose opinion I trust, and both of them seem to indicate that my idea is good, or at least good in principle, which is good enough for me. I am pretty jazzed about it - I think it will be a good learning opportunity, a fantastic social sciences experiment, and potentially something that could do some good, so I am hoping I can get it right, and that it drives some interest!

More about said idea when my thoughts crystallize a little more, and hopefully some samples as we go...also, things have settled down a bit, so hopefully a lot more blogging in general over the next few weeks.

Like OMG! Google App Engine Supports Java! SHUT UP!!

Ever since last week's announcement that Google App Engine would officially support Java, there have been no less than 2,002,555,1111 blog posts and articles written on the subject. Let this one be the one that says

WHO CARES

Seriously. Who cares? Has ever an announcement caused so much excitement? This is hardly the most exciting thing to happen in the world of software. It's someone announcing that their platform that isn't THAT exciting to begin with supports a hacked version of your language...color me less than exciting. It's neat. It's neat like the Macbook Air is neat. Nobody needs it, and certainly I'd think that nobody needs to drop everything to create a tutorial for how every single framework in the whole world interacts with Google App Engine.

It just seems like there are way better things to talk about. I guess it's shiny thing syndrome, but I just don't see the big deal. Google would probably counter by saying 'hey look at all the people excited about it - there must be something there!'. I say, once the shine comes off of it, a new toy will likely replace the excitement around this.

Tip: MySQL and Timestamps as Integers

So, say you managed a legacy project, and on that project, they chose to store timestamps for creation dates in the database. This is quite normal. Now pretend that the timestamp was stored as a TIMESTAMP. Still quite simple. You want to get something after a certain date, just add WHERE CREATED_ON >= '2009-02-01' to get everything February and after. Now imagine that somebody who wrote this application chose instead to store this data as an INTEGER.

This becomes a little more annoying, but not undoable - I had to do a bit of digging in the mysql manual, but here's the query:

select id, first_name, last_name, created_on, as create_date from members where FROM_UNIXTIME( created_on ) >= '2009-02-01';

Hope this helps others out there who suffer from badly designed database maintenance syndrome.

Struts 2 Error Reporting Tips

As I wrote before, Struts 2 isn't the best framework for error handling or notification. I think for the most part the entire framework is an incredible improvement over Struts 1, which had so many silly moving parts that just made it a genuine pain in the ass to work with. Struts 2 is simple, and I take well to the Action-As-Bean pattern used here. Moreover, it's what we use, so I had to learn to deal with the oddness of it. Since I wrote that post, I have come up with/implemented a couple handy tricks:

1) Display the Error Details on the Error Page

Obviously I didn't invent this, but we weren't doing it yet, so here's what I did:

You will likely create a global error mapping. In the action that you map to, add this code that will trap the error details:

Throwable error = (Throwable) request.getAttribute("javax.servlet.error.exception");

request.setAttribute( "errorCause", error.getCause().toString() );
request.setAttribute( "errorMessage", error.getMessage() );
request.setAttribute(  "errorStackTrace", error.getStackTrace() );

Now you can insert a little line into your error page that shows the error, instead of having struts swallow it up forever.

<%= if ( request.getAttribute( "errorMessage" ) != null { %>

-- print out the data --

<% } %>

2) Debug Better

If you get the source code for Struts 2, you can add it to your Eclipse project, and insert a breakpoint in the following piece of code in com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept():

public String intercept(ActionInvocation invocation) throws Exception {
    String result;

    try {
        result = invocation.invoke();
    } catch (Exception e) {

        /* BREAKPOINT HERE */

        if (logEnabled) {
            handleLogging(e);
        }
        List exceptionMappings = invocation.getProxy().getConfig().getExceptionMappings();
        String mappedResult = this.findResultFromExceptions(exceptionMappings, e);
        if (mappedResult != null) {
            result = mappedResult;
            publishException(invocation, new ExceptionHolder(e));
        } else {
            throw e;
        }
    }

    return result;
}

This way you can see if something weird happened inside struts that didn't get reported in the logs. Boom, you can see if there is some mysterious internal error getting thrown, rather than trying to divine it using rain dances and crossed fingers. These two things have really improved my experience with Struts. If I were a better person, I would upgrade to see if it's fixed, and if not, I'd probably add some decent logging.

Bonus Tip

If your application starts up, but you are getting invalid results from your struts action, like the "no result defined for action" message, make sure your source control tool didn't merge incorrectly and add a double mapping for the action. We ran across this recently, where an old mapping was included, and was picked up first. Perhaps there should be a way to tell Struts2 to error out and fail if it finds the same mapping twice in a file? Just something to watch for.

Happy Strutting!

Consuming RSS Feeds with Groovy/Grails

Building off of my recent post about RSS Feed parsing using the ROME Library, I had an idea for a fun application to build using Grails. The first part of this application, which I'll share when I am done, is the part where we read the feed. Now, the last time I did this in Java, it was pretty easy to do, but man, this is just a little bit better:

    def readFeed( url )
  {
      def xmlFeed = new XmlParser().parse(url);

      def feedList = []
    
      (0..< xmlFeed.channel.item.size()).each {

         def item = xmlFeed.channel.item.get(it);
         RSSFeed feed = new RSSFeed( item.title.text(), item.link.text(),
             item.description.text(), item.pubDate.text() )
         feedList << feed
      }

      feedList
  }

Yep, that's it. One line to pull back the feeds. The iterator, and one line to create my RSSFeed object. Then add the feed to a list, and return the list to your controller. In 25 minutes, I have a feed reader application that's basically functional, taking a feed as an input, and returning a page that displays the post title linked to the post, and the posts contents and date. All of life should be this easy.

Garbage Collection, Tomcat, Hibernate, and You

java.lang.OutOfMemoryError: PermGen space

If you are using Sun's Hotspot JVM, maybe you've seen this in your log file, and you don't know what to do. Maybe this will help.

Ever since we upgraded at work from Hibernate 2.1 to 3.2, we have been fighting memory management issues. Some of those can probably be traced back to increased traffic, some probably to odd legacy code that can be refactored pretty easily, and much of it relates to Hibernate and cgLib. There are endless blog posts about this, so if you want to read about the way cgLib, Hibernate, and the Perm region of the JVM heap interact, google away.

To break it down, here's how to know you may run across a problem:

/bin/jps

This will give you a listing of running JVM instances on your machine. Tomcat presents as:

xxxx Bootstrap

this is the JVM pid that you can use to check out memory utilization. Now run the garbage collection util to see what's going on in your heap:

jstat -gcutil

You will be presented with the following:

S0	S1	E	O	P	YGC	YGCT	FGC	FGCT	GCT
0.00	31.58	60.13	57.76	99.66	2647	5.192	6	2.114	7.306

The measures above give you space utilization in percentage of the regions of the JVM memory space. For a detailed explanation of this, check out the official Sun paper on the topic. For our purposes, we will do a quick overview:

S0 and S1 are both survivor spaces.
E is Eden space.
O is Old space.
P is Perm space.

When an object is created, it lives in Eden space. If it makes it past a garbage collection while still containing an active reference to it, it will move on to Survivor space. From there, if it's still actively referenced, it moves on to Old space. Garbage Collections of Eden and Survivor spaces are not the big garbage collections - the are the Young Garbage Collections (YGC). Once something gets into the Old space, it can only be removed by a Full Garbage Collection (FGC). You can see from the above table that a YGC is not expensive - 2647 of them were performed in a total of 5.192 seconds, while just six full collections take 2.114 seconds. Finally, there is the Perm space. This is where the JVM structures and class objects are put. The classloader sticks stuff here to help you so it doesn't have to constantly load and reload these structures. The problem is that if you have a lot of these things, it's going to just keep sticking things in there until it's full.

There are a couple of ways to combat this issue:

1. Maybe you haven't increased the default size of the Perm space. You can do that by adding this JVM flag to your startup script: -XX:MaxPermSize=m. To give some perspective, the default is 32m.
   
2. Look at your classpath. Are you loading a LOT of libraries in? Do you need them all? Perhaps you can remove some of those libraries.

If you still have problems with Perm space after making the above changes, try these JVM arguments and see if you have any more issues.

-XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled

Whereas we were constantly hovering around 99.88% perm space utilization prior to these changes, now we are usually back down around 60%, so an extraordinary event in the system won't trigger an outofmemory situation. Hope this helps.

Why Grails is Sweet

Grails is sweet because you do things that are relatively complex, like upload a photo, resize it, save it, and persist the metadata using this code in a controller class, and nothing more:

        def photo = new Photo(params)
Member m = authenticatedMember()
def myFile = request.getFile( "file" )
def imageTool = new ImageTool()
photo.path = "";
photo.member = m

if(myFile && photo.save())
{
String imagepath = grailsApplication.config.imagePath +
File.separatorChar + "${photo.id}.jpg"
myFile.transferTo(new File(imagepath))

imageTool.load(imagepath)
imageTool.thumbnail(640)

String fixedImagePath = grailsApplication.config.imagePath + File.separatorChar + "${photo.id}-fixed.jpg"
imageTool.writeResult(fixedImagePath, "JPEG")
imageTool.square()
imageTool.swapSource()
photo.path = fixedImagePath
}

Oh by the way, when you define a domain class with a byte array to store a file, then generate the edit/create view, it automatically does all the multipart form submission stuff in the .gsp file. It NEVER gets this easy with Java/JSP. Hmm. Awesome. This is made possible by the innate goodness of grails and a plugin called ImageTools. It makes me dread going back to work and using Java sometimes. Sigh. It's amazing how much you can get done in a short time with this framework, and the more I have a-ha moments with it, the more I think I won't be going back to just Java when I make the switch full time, especially in the knowledge that whether it's Groovy/Grails or JRuby/Rails, I can still fall back on familiar Java libraries. Yessir, this is a good time to be a nerd...

jQuery Goodness

This week I did a lot of client-side development.  Usually, this would be reason to cry and/or drink heavily, but this week, it was one of the more pleasant parts of my week.  I thought that I'd go ahead and share a one of the goodies that I experienced.

Thickbox

Thickbox is an awesome library that pops up a page that you defined as a dialog that can be modal if you specify that option.  It's pretty peppy, and looks sharp.  Thickbox uses decoration of links on a page (set up by specifying a class of 'thickbox').  Anchor tags are set up to register an onclick event that pops up a page that you specified, be it a static page, or the result of a server-side call.  Anything can be displayed in a thickbox.  It can be information, or a form that you submit back to the server.  It's been pretty simple for us, but I did run across an issue today.  I am loading some pre-rendered html into another div that displays as a floating popup on the page.  One of the links there needs to call an overlay that is loaded via thickbox.  The button was defined as an input, which should work, but all the documentation refers to anchor tags, so I figured that we could get around it by doing this:

<script type="text/javascript">

function showOverlay {

jQuery( '#floatingDiv' ).hide();
jQuery( '#hiddenTbLink' ).click();

</script>
<input type="button" id="id" onclick="javascript:showOverlay();" value="showIt">
<a id="hiddenTbLink" style="display: none;" class="thickbox" href="http://www.blogger.com/path/to/action"></a>

This hides the floating div and invokes the click action on the anchor tag that contains the thickbox element.  This should work, right?  Wrong.  The html is dynamically inserted into the DOM after the document.ready code is called.  That is where you perform the thickbox initialize call.  So, what we had to do is add call to tb_init() in the code that loads the floating div, thus doing the jQuery DOM manipulation and allowing the thickbox-enabled link.  Much better now.  I sat back with glee, realizing that in a few hours, I had enabled this floating div and figured out how, from that floating div, I could make a call to load a beautiful thickbox element.  It was all too easy.

Reading Feeds using the ROME API

A recent task had me doing something pretty simple - reading some RSS feeds for display on pages inside our application.  I know that it's easy enough to just write some custom code that parses an RSS feed - after all it's just XML, right?  I didn't want to do that, so I did some digging, and found two real options - ROME, and Commons FeedParser.

It quickly became obvious that ROME was the correct choice, and that's when the fun started. ROME is a Sun project that seemed to provide the most flexibility as far as reading different syndication formats, and clarity of API docs.  Using this tutorial from the site, getting the feed and parsing it was really really easy.  Just pass a feed url to the feed reader, and get some content items back.  

Loop through them and display the correct content.  

URL feedUrl = new URL( feed );

SyndFeedInput input = new SyndFeedInput();
SyndFeed feed = input.build( new XmlReader( feedUrl ) ;

List feedEntries = feed.getEntries();

There are some subtleties, however, that seemed to merit a post, as I didn't really find any clear explanations for these things in one place.

Issue 1:  Content Encoding

We are parsing a feed from a wordpress blog, and it seems that some posters always post content that has the weird characters that signify a content encoding issue.  The weird diamonds with question marks in them (or just empty boxes in Opera) that are inserted where there is a sort of 'half-space' on the actual blog.  I determined that the blog was using UTF-8 (this seems to be the default encoding for a WordPress instance.  After much searching, I came across this post, which seemed to contain about a million suggestions for how to handle the error.  What worked for some didn't seem to work for others, and certainly didn't work for me! I tried to read the url as a stream, and to no avail.  

Instead, I settled on setting the character encoding type on the HttpServletResponse object, which seems to take care of things.  Seems a little weird to me, but that's okay as long as it works and I don't have to write custom parsers.  After updating things, here's how my code looked:

  URL feedUrl = new URL( feed );
  String respEncoding = "";
  if ( encoding == null )
  {
    encoding = "UTF-8";
    respEncoding = "UTF8";
  }
  else
  {
    respEncoding = encoding.replaceAll( "-", "" );
  }
  
  XmlReader.setDefaultEncoding( "UTF-8" );   
  
  SyndFeedInput input = new SyndFeedInput();
  SyndFeed feed = input.build( new XmlReader( feedUrl.openStream(), true ) ) ;

  List feedEntries = feed.getEntries();
        
  response.setCharacterEncoding( respEncoding );

Issue 2: Where's My Content?

The first blog I tested was easy to parse, once I got that list of feeds.  I just needed to to display the description field on the SyndFeedEntry object, and it gave me a nicely formatted (accounting for html inside the post) abridged posting.  Then I tried to display a blog that was hosted by the blogger platform (the very blog you are reading now).  Would you believe that the description property was unset.  No content. Now I am left having to get the raw content  straight out of the raw content feed.  I didn't really want an if-else for this in the display, and the way the SyndEntry was made, it wasn't really super simple to subclass it, so I went ahead and created my own class that took that SyndEntryImpl object and decorated with a few simple convenience methods:

 /**
  * Helper method to return the contents whether they come from the description field (ie wordpress is kind and does this) or raw content
  * @return
  */
 public String getAbridgedContents()  {
   //first try the description
   if ( myEntry.getDescription() != null && myEntry.getDescription().getValue() != null )
   {
     return myEntry.getDescription().getValue();
   }
  
   //if that's not working, use the raw contents
   StringBuilder sb = new StringBuilder();
   SyndContent sc = null;
  
   for ( int i = 0; i < myEntry.getContents().size(); i++ )
   {
     sc = (SyndContent) myEntry.getContents().get( i );
     sb.append( sc.getValue() );
   }
  
   String ret = sb.substring( 0, 255 ) + " [...]";
  
   return ret;
 }

 public String getDateString()
 {
   SimpleDateFormat dateFormat = new SimpleDateFormat("EEE, MMM d" );
  
   return dateFormat.format( myEntry.getPublishedDate() );
 }

 public String getCatName()
 {
   if ( myEntry.getCategories() != null && myEntry.getCategories().size() > 0 )
   {
     SyndCategory sc = (SyndCategory) myEntry.getCategories().get( 0 );
    
     return sc.getName();
   }
   else
   {
     return null;
   }
 }

 public String getCatUri()
 {
   if ( myEntry.getCategories() != null && myEntry.getCategories().size() > 0 )
   {
     SyndCategory sc = (SyndCategory) myEntry.getCategories().get( 0 );
    
     return sc.getTaxonomyUri();
   }
   else
   {
     return null;
   }
 }

Hopefully this will help people get their feedreader working quickly.

Developers and Writers

When I worked at Blackboard, one of the things that people frequently groused about (myself included) was the requirement that we write a certain amount of posts on the internal blog (we used Confluence by Atlassian, which had some blog/journaling-type functionality.  It was hard enough finishing the never-ending assignments that often ran concurrently in groups of five or six, so how could we possibly jump out of the IDE once or twice a day and write something about all this work?  Isn't that just a waste of time?!!?  

Looking back, the answer is unequivocally "No".  

Anyone, including the lowliest junior foot soldier developer, should be able to elucidate their work in plain language, so that someone who is not technical can read and understand it.  This is an important skill that has to be constantly developed.  It's another thing that must be treated as part of a complete developer arsenal, just like learning programming languages, libraries, important protocols, database servers, and operating systems.   Next time you are asked to draft a design document or blog about what you are working on, don't look at it as a black hole of timewaste.  Look at it as a way to get better at your job.  Any dummy can code - it's the people that can explain what and why they are doing it and make sense that are actually valuable.

Often writing something down and then reading it can help you realize that what you have done is a monstrocity, or unnecessary, or doesn't actually fulfill the requirement.  Even more often, if you don't see something after reading your own work, someone else who reads it might have a 'light bulb moment' themselves. Either way, you have a) learned something new, b) made your product better, c) gotten some valuable writing practice.  

If none of the above has inspired you, how about the fact that you get to do something other than completely nerding out for a little while?  Isn't that reason enough?

Go write, developers!  Life isn't all bits and bytes.

Fun and Games with Struts

I had a requirement for a project at work to insert some preprocessing into our Struts actions that will check to see if something fancy has to happen to decorate the page differently based on a branded association. I thought to myself, "hmm, that will be easy".  I was mostly right.  I have been working mostly with Struts 2 lately, which provides us with a handy dandy prepare() method to override, that is always called before the execute() method in a Struts 2 class that extends ActionSupport.  This is simple:

public class AwesomeActionSupport implements Preparable
{ 
  private Integer integerToSet;    
  private Boolean someBool;

  public void prepare() throws Exception    
  {          
    if( someBool )           
    {                
      integerToSet = new Integer( 1 );          
    }           
    else           
    {
      integerToSet = 1;          
    }   
  }

  public String execute()   
  {         
    System.out.println( integerToSet );

    return SUCCESS;   
  }

  //GETTERS/SETTERS FOR PRIVATE VARS ABOVE

}

As you can see it's quite simple when using Struts 2.  Sadly, most of the functionality in our application is still running Struts 1 for now.  This makes something easy much less simple.  There isn't really a built-in mechanism to make a pre-execution call.  Here's what I came up with:

public abstract class AwesomeAction extends Action
{
  public ActionForward execute(ActionMapping mapping, ActionForm form,                               
                               HttpServletRequest request, HttpServletResponse response) 
    throws Exception   
  {    
    //do prep work    
    request.setAttribute( "whatever the action needs", theValue );    
    return perform( mapping, form, request, response );      
  }

  /**   
    * A stub for performing the execute method, to be implemented by each    
    * individual struts action, after the execute pre-processing is performed   
    *    
    * @param mapping   
    * @param form   
    * @param request   
    * @param response   
    * @return   
    * @throws Exception   
    */  
  public abstract ActionForward perform( ActionMapping mapping, ActionForm form,                                         
                                         HttpServletRequest request, HttpServletResponse response) 
    throws Exception;
}

So now when you write your actual action class, instead of extending the Action class, you extend this action class that does the preprocessing, then delegates the real work to the perform method that you have to define.  Seems to work great, with the only caveat being you can't use a Dispatch action with multiple methods.  If you really want to do that, then upgrade your action/application to Struts 2!

Get Agile!

Well here at Sportsvite, we are working on getting agile. We've had a new team member join us (Jeff), and with him he brought a bunch of ideas about software development and process. He is a pretty strong devotee to the Scrum process (to which I am quite open), to Spring (which I think is in many cases overrated, but maybe I am just being stupid), and to the New England Patriots, which I suppose we can forgive.

As a small team that is spread out across multiple offices with different work schedules, we are truly hitting some edge cases when it comes to Scrum/Agile implementation. As we navigate the process, we can talk more about how it's going, but here are the issues that I see up front:

1) Multiple Responsibilities

We have one platform, but many streams of work. How can we reconcile sprints to the timelines that exist based on external client demands? While the core product timeline may say that we plan on delivering some software on March 20, but a client says March 6, how do we address this - theoretically we will have a bunch of half-done stuff going out live. Normally, this can be addressed by using source control branches, etc, but what if there are platform dependencies? Just a matter of careful planning, but there is the definite potential for trouble here.

2) Verification

On a small team without dedicated QA, how can you really verify that things work? So part of this problem is not having a dedicated quality team, but you can obviously make do. The issue here is that in a perfect world you have QA folks working in lockstep with the developers and UI designers writing test cases and readying for the verification step - in our case if the developers have to do this, they aren't developing nearly as much, and then they have to stop dead in their tracks to test, rather than getting folks testing while the developer can finish up more tasks. Not a shortcoming of agile, just another hurdle in general.

The idea that we came up with is a two week sprint of work, with a one week verification sprint. This sort of violates the spirit, to me, as a sprint should theoretically result in 'finished' work, but it's an adaptation that seemed to be the best compromise available in our situation (comments and suggestions always welcome)!

3) Heavy UI Requirements

Part of the big idea with agile is to coordinate effort and get everyone on the same page that things are ready to deploy in a timely and high-quality fashion. Here, and at many places where you develop a world-facing site, you have a heavy reliance on UI designers. Here we get templates and we apply them once the server side work is done, turning HTML into JSP. This is okay, but now we want to make it all cohesive. How can we do this, when the UI needs to be done ahead of the developers? We can have the UI team working ahead of the engineers, then catch up with them towards the end of the development sprint in order to do any cleanup and browser testing fixes. This is okay, but one of the best benefits I see in this is that the sprint planning session should take place before the UI work starts, so we aren't hurrying through last minute requirements on the UI, and I am not sure if this is going to be the case if the UI folks are already making templates.

4) Geographical Issues

We have developers in Washington, and non-technical staff and UI designers in New York city. Traditional scrum is very hands on and paper/white board based. We obviously can't all be in the same room, can't all physically watch people manipulate the task list, and whatnot, but we do have technology to do it. We just need a way to manage the 'state' of our project. To that end we can use tools, but we haven't found one that is all the way there and reasonably priced. We are using Scrumy for now to see how it goes - it's pretty nice, but there are definitely shortcomings.

I am hoping that people out there who are agile pros have run across things like this. Obviously the positives outweigh the negatives. We will be able to plan out our work better, get a better idea of how long things take, to prioritize more effectively, and hopefully to deliver better software on tighter timelines. We just can't do the vanilla agile implementation.