Quartz - Job Processing Done Right

On our site, we have some background jobs that run periodically - we used JBoss to manage them, and it usually works. Sometimes they don't. The problem there is that once the job scheduler stops doing its thing, you can't really do anything but rerun the jobs periodically until you have a chance to bounce the server, which you can't do until night time (and obviously that isn't even particularly desirable). Well, today was one of those days. JBoss stopped running its scheduler, with nary a log message or stack trace. Just stopped. Yesterday it worked. Today, not so much. That sucks. We decided that it was a good time to untether ourself from JBoss and move to Quartz. In about a half of an afternoon, I had downloaded the code, perused the documentation, rewritten our scheduled processes to use Quartz, and tested and committed it. You can't often say that when you undertake something like this (believe me - we are currently in the testing stage of a long-overdue Hibernate 2-3 upgrade that hasn't been fun or easy), so I thought in honor of Quartz's simplicity and ease of use, I'd write a post, including how we got it working.

What is Quartz?

From their site:

"Quartz is a full-featured, open source job scheduling system that can be integrated with, or used along side virtually any J2EE or J2SE application"

How Do I Use Quartz?

Quartz is simple to use. To get started, download the distribution from their site, stick it in your classpath, and you are ready to go. First things first - you need to get your properties file set up. A simple configuration need only include this (or less if you want to fall back to defaults):

#===============================================================
# Configure Main Scheduler Properties
#===============================================================

org.quartz.scheduler.instanceName = QuartzScheduler
org.quartz.scheduler.instanceId = AUTO

#===============================================================
# Configure ThreadPool
#===============================================================

org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 5
org.quartz.threadPool.threadPriority = 5

#===============================================================
# Configure JobStore
#===============================================================

org.quartz.jobStore.misfireThreshold = 60000
org.quartz.jobStore.class = org.quartz.simpl.RAMJobStore

Once you have this going, you need to setup your jobs file. Most of the examples that came with Quartz show how to declare jobs in Java code. This is fine and all, but what if you don't want to have to recompile to change a quick setting? No good. I want to use XML. If you don't agree, feel free to follow one of the many examples they provide. To use XML, there are a couple things to keep in mind. First off - Quartz doesn't want to read these jobs from XML by default - you have to tell it to. In order to do so, you just need to add the following to quartz.properties:

org.quartz.plugin.jobInitializer.class = org.quartz.plugins.xml.JobInitializationPlugin
org.quartz.plugin.jobInitializer.fileName = my-jobs.xml
org.quartz.plugin.jobInitializer.overWriteExistingJobs = false
org.quartz.plugin.jobInitializer.failOnFileNotFound = true

Then just create a file my-jobs.xml:

<?xml version="1.0" encoding="UTF-8"?>
<quartz xmlns="http://www.opensymphony.com/quartz/JobSchedulingData"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
overwrite-existing-jobs="true">

<job>
<job-detail>
<name>AwesomeProcessor</name>
<group>awesome-group</group>
<description>Run the awesome job</description>
<job-class>
com.sportsvite.AwesomeProcessor
</job-class>
<job-data-map allows-transient-data="false">
<entry>
<key>superbad</key>
<value>true</value>
</entry>
</job-data-map>
</job-detail>

<trigger>
<cron>
<name>awesomeTrigger</name>
<group>awesome-trigger-group</group>
<job-name>awesomeProcessor</job-name>
<job-group>awesome-group</job-group>
<!-- trigger every 30 min -->
<cron-expression>0 0/30 * * * ?</cron-expression>
</cron>
</trigger>
</job>
</quartz>

Pretty simple, right? That's all you have to do on the configuration side.

NOTE: When you deploy the code, make sure that these two files end up in WEB-INF/classes so they are picked up in the classpath.

Now you need to kick off the Scheduler somehow. Most of the exercises show you how to do it in the code, but we don't want to rely on that, right? If you are running in a servlet container, they give you a simple servlet, the QuartzInitializerServlet. It is included in the jar file, and all you have to do is add it to the web.xml:

<servlet>
<servlet-name>QuartzInitializer</servlet-name>
<display-name>Quartz Initializer Servlet</display-name>
<servlet-class>org.quartz.ee.servlet.QuartzInitializerServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>

So we have told the Quartz scheduler to kick off a job called AwesomeProcessor, so we need to code the class. This is simple - you just need a class that implements the Quartz interface Job, and has the following method:

public void execute( JobExecutionContext ctxt )
{
String s = (String) ctxt.getString( "superbad" );
doFoo( s );
}

Note the use of the JobExecutionContext - you can use that to pass a bunch of arguments in a Map-like structure. It corresponds to the job-data element in the my-job.xml file.

Now when you fire up your server, the Quartz scheduler will activate and you'll see a message like this in standard out:

18:27:45,722 INFO [JobSchedulingDataProcessor] Scheduling 1 parsed job.
18:27:45,723 INFO [JobSchedulingDataProcessor] Adding job: awesome-group.AwesomeProcessor
18:27:45,735 INFO [JobSchedulingDataProcessor] 1 scheduled job.
18:27:45,735 INFO [QuartzScheduler] Scheduler QuartzScheduler_$_NON_CLUSTERED started.
18:27:45,736 INFO [Engine] StandardContext[]QuartzInitializer: Scheduler has been started...
18:27:45,736 INFO [Engine] StandardContext[]QuartzInitializer: Storing the Quartz Scheduler Factory in the servlet context at key: org.quartz.impl.StdSchedulerFactory.KEY

That's it! It's going to work now. The best thing about Quartz is that if you have bad XML or some other issue, it gives you a meaningful error message that you can easily remedy. Not all open source products can say the same.

Kudos to Quartz. It made my programming day pretty pleasant. Hope this helps others get up to speed. For a bunch of examples and other helpful information, check out the Quartz Wiki.

Tigers! Books! Travel!

Tigers Win!

1-7. In the standings, it isn't much of a difference, but man does it look better than 0-7. Seriously. WAY BETTER. Bonderman got a win for the first time since roughly 2004, and the bats woke up a bit. Clete Thomas continues to impress. Happy day(s) are here again, for now at least.

In other news, the Nats stink.

In other news, I finished reading America's Three Regimes, by Morton Keller. This was a good read, interesting and informative. Keller describes the evolution of the political system and parties in particular through the life of the American Nation. Reading it makes me even more upset about the two-party system and it's inability to serve national interests, especially at the end of the book, when Keller talked about the rise of special interests in the political arena. Sigh. I would definitely recommend this book - a shade under 300 pages, and while packed with information, doesn't read like a textbook.

I have moved on to the Wisdom of Crowds. So far so good. I'll write more about it shortly, as it's gotten me thinking a bit. As I read more of these sort of sociology meets technology meets economics books, it's interesting to contrast the ideas and concepts across each of them. Surely they overlap and become sort of repetitive, but each book does have its own interesting and usually valid message that it conveys.

I am also STILL reading Midnight in Sicily, albeit slowly. I have read too much of it to turn back now, and with some weekend travel looming (more later), hopefully I can finally finish it. I am also continuing with my historical fiction. The latest book that Amazon recommended me is The Last Kabbalist of Lisbon, by Richard Zimler. It's a well-written and eye-opening account of the plight of the Jews in Portugal around the time of the Inquisition. I am not sure why I keep ending up reading historical fiction about converted and secret Jews (David Liss books as well), but they are all fascinating, and satisfy my craving for history while still being light and easy reads.

The next couple weeks are going to be busy - here at Sportsvite, we are getting ready to do the first release that will include my work. That's exciting, but pushing code into production is always stressful. The weekends are for travel. This weekend, we are off to Massachusetts to celebrate the first birthday of Jena's nephew William. That ought to be a blast! He's a joy to be around, and it will be great to see Jena's family as well. Next weekend, it's off to Philly to catch a baseball game and hang out with college friends. Then the following week, it's off to the bustling metropolis of Crawfordsville, IN, to celebrate my grandmother's 90th birthday! So exciting! She's still amazingly full of life, and it will be great to see them and the rest of my family. Lots coming up - hopefully I'll have some good pictures to post.

MacBookPro.Sweetsauce.Awesome.Computer.com

So. It happened. I got a MacBook Pro. Here at Sportsvite, the entire engineering team (yes all three of us) has Macbook Pros. Out of sheer computer envy, I kept watching my compadres on their fancy computers and thinking to myself "Man, I sure would be more productive if I had one of those." I think that in the back of my mind I knew that was BS, and I just wanted to have the same computer as them. I am here to say that I was right, and wrong.

What's Awesome

Old Computer (Core 2 Duo 1.66Mhz, Vista Home Premium) JBoss Startup: 90s
New Computer( Core 2 Duo 2.2Mhz, OS X Leopard) JBoss Startup: 16s
Old Computer Screen: 14in, 1200x800
New Computer Screen: 15in, 1440x900
Old Computer Startup: 2-3 min
New Computer Startup: 30-60 sec

The incredibly crisp seamless performance.

This whole .dmg file, where you open it and drop it into the Applications directory and it just works? Not bad!

The giant crisp display.

The smooth, quiet keyboard.

The dock. Nice.

The non-ridiculous power cord, and the magnetic connection to the computer that it uses.

Garageband. I have no clue how it works, but it will be a part of my guitar practice sessions for a while, until I do.

Adium. The little green duck is hilarious, and it seems be a really awesome simple chat tool.

What's Not So Awesome

Well for starters, where the hell is the right mouse button? I don't know, but I wish I could find it. Don't tell me to hit apple either every time I click. That's LAME.COM. But, I can look past this - I am sure all those devoted Mac snobs out there must look at it as a sign of their greater purpose and ability. I think it's kinda lame, but I am okay with it. Fortunately, there are USB ports, into which you can plug nice mice with all the buttons. Problem solved!

Installing Apache http server. This one was a tough one. This really sucked. A lot. Like on the suckiness scale, this one had to be about a 30 out of 10. Fortunately, my friend Bob showed me DarwinPorts, which seems to be awesome. They have all these ready made packages, and you just type things like port install apache2. True to its name, it does just that! Huzzah! By the way, I also hate mod_jk2, but oh well. It all works now.

Sudo. The idea of sudo sucks. I will have to get around to do setting it up so everything is done as root like I do on my linux box at home. Sudo is lame. Sudo just means you have to type a four-letter word before you do something catastrophically stupid. I don't like Sudo.

The apparently lack of the start menu like in Windows. I don't really like that either. I know I can use finder to get to Applications, but it's just not the same as being able to just get there from my screen. Again, likely something I'll figure out.

The fact that JDK6 isn't ready yet. Come on guys! Good thing I am not doing performance engineering anymore, since all those sweet tools like jmap, jhat, jstack, and jconsole were greatly improved in JDK6.

No pgUp or pgDn by themselves. Also no home or end. This is one place where my Toshiba really shines - it's got a great and friendly keyboard.

Replacing 'control' with this 'Apple' key, but not having it in the same place as 'Control' - that sucks. maybe there is some way to reroute the keyboard. I can see this causing so many problems.

More to come later

Social InterWeb 2.0

DISCLAIMER: This stuff seems really cool and profound to me, but I may be talking about things that are pretty inane and problems that have already been solved, so forgive me if this post is really stupid.

Well, it's almost here. In a little over a week, I will be fully immersed in the social web. As someone who isn't probably the hippest dude when it comes to the new Internet, I have some boning up to do on how people use the web, what makes a site both 'cool' and (more importantly in my humble opinion) 'useful'. I see (younger) people today who route almost all their interaction through some form of social web component. I usually think in terms of just emailing my buddy, or *gasp*, calling them to see what they are up to next weekend. For that matter, my buddy was someone who I had known for a while, or at the very least, had actually met. Now there is whole dynamic whereby you can have 'friends' who you have no interaction with outside of 'poking' or posts on a wall, or a comment in a myspace page. This is all weird, but this is what I need to start to grasp.

See, I am the guy who doesn't answer his cell phone half the time that it rings. It's a tool of my convenience, not a direct line to me wherever I may be. I have IM, but I don't have a big friend list. I have a myspace page, but it has the same 51 friends that it had after about 2 wks of having the page. I have a facebook page that I put up to do development, and something like 9 people found it and did friend requests. I have one post on my 'wall' (thanks Zahra!).

I do participate in the social web - I provide recommendations, feedback, and ratings about things that I have done, places I have visited, books I have read, music I have listened to. I get recommendations for things to buy from others who participate. I think it's fascinating to examine the science of the social web - to understand how social networks actually work. I have been thumbing through Programming Collective Intelligence, by Toby Segaran, which gives a primer on the algorithms behind so many of the Web 2.0 sites. Fascinating to think about how the people are connected, and how someone can take these connections and find a way to monetize them. Things like the Small World Phenomenon are pretty incredible. I wonder (and I am sure this happens a lot more than I know), but I wonder if there is some sort of mathematical formula you can use to see:

a) How much is a given profile worth? (social networking potential)
b) How can you say what features have more social reach (how many degrees of connection do they span and how quickly)
c) How vital is a given member to a group (structural cohesion)

Are companies like facebook and myspace already utilizing these kinds of models to apply value to their users? Is this possible? Does it work? Such cool stuff.

As someone who comes from a business background, and who is interested in sociology, I think that my new job will provide me with an avenue to explore the intersection of technology, marketing, sociology, and sales. It makes what is an exciting move that much more fascinating to me, and I can't wait to dive in headfirst. I'll be letting you know how it goes, and hopefully writing a lot more about this as I start to figure more of it out.

Sail On...

Well, it's finally come. I am moving on from Blackboard. It might seem like a crazy move, considering I just got promoted, and have a team under me, and a high profile position doing some pretty cool work, but it's true. I am sailing on. For reasons that I don't really need to discuss, I am just ready to go. I have learned a lot, and grown my skill set and confidence level enough that I am ready to move.

I am headed to a startup called Sportsvite. It's a social networking site (yes another social networking site), but one that actually serves a purpose - to connect recreational athletes with leagues, games, and each other. I will be part of a small team doing all kinds of Java web application development. I am extremely excited to get going, and I already have a lot of ideas about how to make the product even cooler than it already is. I think there is a lot of potential there, and it will be refreshing to get into a smaller more dynamic place where process and 'best practices' don't overrule common sense and client needs.

So the turnover at Blackboard will pretty much envelop the remainder of the month. Hopefully once March comes around, things will start settling down as I get into the new job. In other news, I went skiing this weekend. I did pretty well for my third time out, despite the rough conditions. Slopes alternated between closed, slushy, and icy. Not ideal, but still a lot of fun, and always great to get away. While there, I managed to knock through The Second Horseman by Kyle Mills. It was good like all his books. Nothing earth-shattering. Now, I have moved on to a few books. I am still working on Music and the Brain, which is really fascinating, but not light reading by any stretch. I am also tackling 'The Career Programmer', by Christopher Duncan, about how to work as a programmer and not go crazy. I have to say that after finishing part one of three on the train this week, I can't say I have experienced much of what the author describes. He comes off as so cynical that it's hard to take anything he says seriously, even though there is valuable wisdom sprinkled thoughout. I am also reading Midnight in Sicily by Peter Robb, but it's off to such a slow start, it may not make the cut. I will provide an update in the next installment.