Java 7 – Fork/Join

While we lamented how feature-poor Java 7 turned out to be, one thing that made it that turns out to be a boon to high-performance concurrent development is the new Fork/Join framework. This framework is targeted at multi-processor systems (really almost all hardware today) in situations where a batch of work can broken into smaller recursive calls. But more than just that, it also uses a work-stealing algorithm where threads with no work can steal available work from other threads that are busy. What makes that so useful is that you can try to break down your work into fairly small, roughly equal pieces, but some pieces can take longer than others and you’ll still get good use of your processing resources.

The Fork/Join framework builds on the ExecutorService we discussed in our Concurrency series and the implementation is ForkJoinPool that can execute the ForkJoinTask which is usually implemented as a child of either RecursiveTask(returns data) or RecursiveAction(no returned data).

As is our custom, let’s use a sample task and some applicable code. Let’s assume you want to write a class that will calculate the size on disk of a given directory and all its children. And in our case we’ll just make the assumption we’re running some wicked fast SSDs so we can actually benefit from concurrent scans.

Here’s our code:

import java.util.concurrent.*;
import java.io.*;
import java.util.*;

public class DirectorySizer extends RecursiveTask<Long> {
  
  private List<File> mFiles;
  private boolean mAllFiles = true;
  
  public DirectorySizer(List<File> files) {
    mFiles = files;
    for (File file : files) {
      if (file.isDirectory()) {
        mAllFiles = false;
      }
    }
  }
  
  protected Long compute() {
    if (mFiles.size() <=4 && mAllFiles) {
      return computeLocal();
    } else {
      return forkAndJoin();
    }  
  }
  
  private Long computeLocal() {
    long length = 0;
    for (File file : mFiles) {
	  length += file.length();
    }
    return length;
  }
  
  private Long forkAndJoin() {
    List<File> dirsAndFiles = new ArrayList();
	for (File file : mFiles) {
      if (file.isFile()) {
        dirsAndFiles.add(file);
      } else {
        dirsAndFiles.addAll(Arrays.asList(file.listFiles()));
      }
    }
    int rightSize = dirsAndFiles.size() / 2;
    int leftSize = dirsAndFiles.size() - rightSize;
    List<File> leftList = dirsAndFiles.subList(0, leftSize);
    List<File> rightList= dirsAndFiles.subList(leftSize, leftSize+rightSize);
    DirectorySizer d1 = new DirectorySizer(leftList);
    d1.fork();
    DirectorySizer d2 = new DirectorySizer(rightList);
    return d2.compute() + d1.join();
  }
  
  public static void main(String[] args) throws Exception {
    List<File> files = Arrays.asList(new File(args[0]).listFiles());
    DirectorySizer sizer = new DirectorySizer(files);
    ForkJoinPool pool = new ForkJoinPool();
    Long size = pool.invoke(sizer);
    System.out.println(args[0] + " is " + size + " bytes ");
  }
}

Let's break down the usage of the Fork/Join framework. By extending RecursiveTask, all we really need to do is implement the compute method calculating the size when the amount of work is small enough to suit our needs and using a fork/join when the chunk of work is still too large. In our main method we get all the benefit of the framework simply by creating a new ForkJoinPool and passing our top-level instance to the invoke method.

While it's true the potential uses of this framework are limited and require a fairly narrow problem scope, it's nice to see continued advancement in Java in the concurrency space to build on all the goodies we got in Java 5. If you have any concurrency problems you'd like us to tackle in our blog, drop us line. We enjoy talking shop with fellow developers.

Java Concurrency Part 6 – CountDownLatch

Some concurrency utilities in Java naturally get more attention than others just because they serve general purpose problems instead of more specific ones. Most of us encounter things like executor services and concurrent collections fairly often. Other utilities are less common, so sometimes they may escape us, but it’s good to keep them in mind. CountDownLatch is one of those tools.

CountDownLatch – a more general wait/notify mechanism

Java developers of all sorts should be familiar with the wait/notify approach to blocking until a condition is reached. Here is a little sample of how it works:

	
public void testWaitNotify() throws Exception {
   final Object mutex = new Object();
   Thread t = new Thread() {
      public void run() {
      // we must acquire the lock before waiting to be notified
      synchronized(mutex) {
         System.out.println("Going to wait " +
                            "(lock held by " + Thread.currentThread().getName() + ")");
            try {
               mutex.wait(); // this will release the lock to be notified (optional timeout can be supplied)
            } catch (InterruptedException e) {
               e.printStackTrace();
            } 
            
            System.out.println("Done waiting " +
                               "(lock held by " + Thread.currentThread().getName() + ")");
         }
      }
   };

   t.start(); // start her up and let her wait()

   // not normally how we do things, but good enough for demonstration purposes
   Thread.sleep(1000);

   // we acquire the lock released by wait(), and notify()
   synchronized (mutex) {
      System.out.println("Going to notify " +
                         "(lock held by " + Thread.currentThread().getName() + ")");
      mutex.notify();
      System.out.println("Done notify " +
                         "(lock held by " + Thread.currentThread().getName() + ")");
   }

}

Output

Going to wait (lock held by Thread-0)
Going to notify (lock held by main)
Done notify (lock held by main)
Done waiting (lock held by Thread-0)

A CountDownLatch can actually be used similar to a wait/notify with only one notify – that is, as long as you don’t want wait() to stall if notify() is called before you have acquired the lock and invoked wait(). It is actually more forgiving because of this, and in some cases, it’s just what you want. Here’s a sample:

public void testWaitNotify() throws Exception {
   final CountDownLatch latch = new CountDownLatch(1); // just one time
   Thread t = new Thread() {
      public void run() {
         // no lock to acquire!
         System.out.println("Going to count down...");
         latch.countDown();
      }
   };
	
   t.start(); // start her up and let her wait()
   System.out.println("Going to await...");
   latch.await();
   System.out.println("Done waiting!");
}

As you can see, it is simpler than wait/notify, and requires less code. It also allows us to invoke the condition that ultimately releases the block before we call wait(). This can mean safer code.

A Real World Example

So we know we can use it as a simpler wait/notify mechanism, but you probably saw the constructor argument that we used above. In the constructor, you specify the number of times the latch needs to be counted down before unlocking. What possible uses are there for this? Well, it can be used to make a process wait until a certain number of actions have been taken.

For example, if you have an asynchronous process that you can hook into via listeners or something similar, you can create unit tests that verify a certain number of invocations were made. This lets us long only as long as we need to in the normal case (or some limit before we bail and assume failure).

I recently ran into a case where I had to verify that JMS messages were being pulled off the queue and handled correctly. This was naturally asynchronous and outside of my control, and mocks weren’t an option since it was a fully assembled application with Spring context, etc. To test this, I made minor changes to the consuming services to allow listeners to be invoked when the messages were processed. I was then able to temporarily add a listener, which used a CountDownLatch to keep my tests as close to synchronous as possible.

Here’s an example that shows the concept:

public void testSomeProcessing() throws Exception {
   // should be called twice
   final CountDownLatch testLatch = new CountDownLatch(2);
   ExecutorService executor = Executors.newFixedThreadPool(1);
   AsyncProcessor processor = new AsyncProcessor(new Observer() {
      // this observer would be the analogue for a listener in your async process
      public void update(Observable o, Object arg) {
         System.out.println("Counting down...");
         testLatch.countDown();
      }
   });
	
   //submit two tasks to be process 
   // (in my real world example, these were JMS messages)
   executor.submit(processor);
   executor.submit(processor);
	
   System.out.println("Submitted tasks. Time to wait...");
   long time = System.currentTimeMillis();
   testLatch.await(5000, TimeUnit.MILLISECONDS); // bail after a reasonable time
   long totalTime = System.currentTimeMillis() - time;
	
   System.out.println("I awaited for " + totalTime + 
                      "ms. Did latch count down? " + (testLatch.getCount() == 0));
	
   executor.shutdown();
}

// just a process that takes a random amount of time 
// (up to 2 seconds) and calls its listener
public class AsyncProcessor implements Callable<Object> {
      private Observer listener; 
      private AsyncProcessor(Observer listener) {
      this.listener = listener;
   }

   public Object call() throws Exception {
      // some processing here which can take all kinds of time...
      int sleepTime = new Random().nextInt(2000);
      System.out.println("Sleeping for " + sleepTime + "ms");
      Thread.sleep(sleepTime);
      listener.update(null, null); // not standard usage, but good for a demo
      return null;
   }
}

Output

Submitted tasks. Time to wait...
Sleeping for 739ms
Counting down...
Sleeping for 1742ms
Counting down...
I awaited for 2481ms. Did latch count down? true

Conclusion

That’s about it for the CountDownLatch. It is not a complicated subject and it has limited usages, but it’s good to see examples and know it’s their in your tool chest when you hit a problem like I did. In the future, I’ll definitely keep it in mind for a simpler wait/notify, if nothing else. If you have questions or comments about this post or others in the series, just leave a message.

Testing in Grails – Common base class for Unit/Integration/Functional Tests

Groovy and even more so, Grails, demands automated testing. In fact, extensive coverage is a must to have confidence that we are not regressing or introducing runtime resolution issues with a single errant keystroke. Grails expects you to write tests and each grails project has the familiar test/unit, test/integration and test/functional source folders. Of course, we want to make it as easy as possible to write our tests. You already use a common base class for your grails tests, GrailsUnitTestCase. And it is a well-established practice in environments using testing frameworks such as jUnit to extend this approach to your own needs.

For example, it’s nice to be able to verify failure cases as we discussed in our last post. Perhaps you’re using a helper similar to the one previously described (shown below) in a base unit test class and want to extend its use to integration tests.

def expectThrown(Class expectedThrowable = Throwable, Closure closure) {
    try {
        closure()
    } catch(Throwable t) {
        if (!expectedThrowable.isInstance(t)) {
            throw t
        }
        return t
    }
    throw new AssertionError("Expected Throwable $expectedThrowable not thrown")
}

Well, if you have the base class in your test/unit source folder, it’s not available to test/integration and test/functional. You can place the base class src/groovy, but then this becomes a production class and is included in your .war file.

The current “solution” actually does start there. Place your base class in src/groovy. Then we want to add an exclusion in BuildConfig.groovy to ensure that this class – and any others you want excluded – are not packaged in the war.

BuildConfig.groovy

grails.war.resources = { stagingDir ->
    delete(file:"${stagingDir}/WEB-INF/classes/com/carfey/test/CarfeyGrailsTestCase.class")
    delete(dir:"${stagingDir}/WEB-INF/classes/com/carfey/test")
}

Tested against Grails 1.3.7.

Testing Failure Cases in jUnit in Java & Groovy

Unit testing expected failure cases is just as important as testing expected positive behaviour. In fact, you could argue that ensuring all failures are happening as expected is more important since it’s one of your best defenses against bad data. If you search for examples of how to test for failures (using old style jUnit 3), you’ll likely come across something like this.

public void testException() { 
    try { 
         Object o = getList().get(0)
         fail("Should raise an ArrayIndexOutOfBoundsException"); 
    } catch (ArrayIndexOutOfBoundsException e) { 
    } 
}

or perhaps

public void testException() { 
    try { 
         Object o = getList().get(0)
    } catch (ArrayIndexOutOfBoundsException e) {
        return;
    } 
    fail("Should raise an ArrayIndexOutOfBoundsException"); 
}

While both of these approaches are ok, they aren’t quite as precise as a good unit test could be. First, you want to be very specific with your verification. Let’s say down the road someone starts throwing an ArrayIndexExceedsSizeException that is a child of ArrayIndexOutOfBoundsException. Your test case will continue to pass. This isn’t ideal since you’ll want to update your test cases to verify all variations of runtime behaviour. And also, neither of these approaches gives you a decent way of making additional assertions against the exception (checking for messages, error codes, etc.)

This takes a bit more code, but the following will always be accurate and will start failing when these types of changes happen and gives you a clear approach for additional assertions.

public void testException() { 
    ArrayIndexOutOfBoundsException actual = null;
    try { 
         Object o = getList().get(0)
    } catch (ArrayIndexOutOfBoundsException e) {
        actual = e;
    } 
    assertExceptionThrown(ArrayIndexOutOfBoundsException.class, actual);    
    assertEquals(actual....
}

protected static void assertExceptionThrown(Class expected, Exception actual) {
	if (actual == null || !actual.getClass().equals(expected)) {
		fail("Exception thrown not of type " + expected.getName() + " but was " + (actual == null ? "null" : actual.getClass().getName()));
	}
}

For groovy tests, you can use the above, or you can use the beauty of closures.

void testException() {
    def ex = expectThrown(ArrayIndexOutOfBoundsException) {
        Object o = getList().get(0)
    }
    assert ex.message....
}

def expectThrown(Class expectedThrowable = Throwable, Closure closure) {
    try {
        closure()
    } catch(Throwable t) {
        if (!expectedThrowable.isInstance(t)) {
            throw t
        }
        return t
    }
    throw new AssertionError("Expected Throwable $expectedThrowable not thrown")
}

You can put these and other exception assertion helpers in a base class that extends your jUnit test class (TestCase or GroovyTestCase). In our next post, we’ll cover how to get around some Grails issues sharing test base classes amongst unit, integration and functional tests.

Eclipse and Memory Analyzer (MAT)

In times past, when it came to tracking down sporadic memory problems in a complex Java application, it required using a commercial product such as JProbe or a lot of painful and inefficient attempts to recreate the issue. Even if the problem were easy to recreate, unless the problem was blatantly obvious, your application might need to be enhanced in an iterative fashion to collect sufficient information to begin to diagnose the problem. And don’t make the rookie mistake of assuming the class/method/line that threw the OOM is your problem area. That just happens to be the code that couldn’t obtain the necessary memory.

Fortunately, now we have a decent free tool that integrates with the Eclipse known as Memory Analyzer, or MAT. Installation can be done via the standard update sites – in Eclipse, choose the Help menu, Install New Software, Select All Available Sites and type memory analyzer and follow the installation instructions. Charts feature isn’t required if all you want to do is track down your problem areas. And I’ll assume you either have a heap dump generated on an OutOfMemory or have connected to a live vm to extract one. If you don’t have one and the vm is running, you can use MAT to extract one. Just look in File menu, Acquire Heap Dump option.

When you first open your heap dump, MAT will ask if you’d like to run one of their canned reports.

This likely isn’t worth your while. I haven’t found them to provide any meaningful insight. Perhaps if your application is very simple, the problem is obvious or in the case of tuning, the Component Report, these reports might be worth a shot. But likely your best starting point is to take a look at the dominator tree.

Either use the icon, the Dominator Tree link under the Biggest Objects by Retained Size section, or select it as an Action.

First of all, the default Dominator Tree view shows single objects. If nothing jumps out at you immediately, you can take one or both of the following actions.

  • Select the Group Result By… icon and group by class
  • Right click the header row and select Columns -> Edit Filter -> Class Name or start typing in the Regex row if available

This allows you to see # of Objects of each type and also filter them by your code, or perhaps other package name patterns that you suspect may be causing problems. Once you’ve found an entry that you’d like to dive into, you have several ways of getting additional information. If you expand the class selection, you’ll be able to see what references the objects are holding and their sizes – the retained objects. And MAT is smart enough that the filters you may have applied do not extend to these retained objects.

If you right click the class entry and select “Show Retained Set”, you will see all the retained objects of everything ultimately contained by that object, a very convenient view that can help you quickly pinpoint a problem class.

Perhaps you can’t understand why there are so many instances of a particular object type in your heap dump, feeling that they should be short-lived or limited in quantity. The “Path to GC Roots” (available when selecting a single object instance) and “Merge Shortest Path to GC Roots” (for multiple instances) give you a view of the objects in question that allows you to see which objects are maintaining references to the object(s) in question.

With these tools, views and filters, I’ve not found it necessary to use some of the other features of MAT but by all means, take a look at OQL (SQL style ability to query the heap details), Histograms and even specialized Java Collections usage details.

Once you have a class you’d like to investigate at the code level, assuming you have the source code in your workspace, right-click the class from any of these views and select “Open Source File”.

I’ve been able to successfully use MAT to pinpoint memory problems in applications that I had very little exposure to. It provides a concrete memory overview that doesn’t show bias when long-time project developers may be inclined to do so. If you happen to have a decent understanding of your application, even better. That knowledge will help you put context around some of the data MAT shows you. What’s your experience with MAT been? Please share your comments below.

Problems with ORMs

Object Relational Mapping tools like Hibernate have helped developers make huge productivity gains in dealing with relational databases in the past several years. ORMs free developers to focus on application logic and avoid writing a lot of boilerplate SQL for simple tasks like inserts or queries.

However, the well-documented problems with object-relational impedance mismatch inevitably cause headaches for developers. Relational databases are a specialized technology built on sound concepts, but they don’t necessarily line up to the object-oriented world. There are a few approaches and styles to using ORMs which have various pros and cons.

One of the fundamental choices to using an ORM is deciding whether the you will generate your ORM mappings from the database schema or the other way around, generating your database schema off of some ORM definition (possibly an XML configuration file, annotations or something like XDoclet).

The former approach of generating your ORM layer from your database schema means you have to deal with the database in its own language and terms, whether you deal with DDL specific to the database or have some abstraction layer, but nevertheless you are forced to treat the database as what it is. Unfortunately, it means you need expertise in the technology and it may take more work than allowing your schema to be generated. However, this forces developers to understand and deal with the RDBMS properly – it is dangerous and harmful to treat a DBMS as a simple datastore. Developers need to consider the impact of keys, indexes, etc. when designing applications, and shielding them from the realities of relational databases can be dangerous, and in my experience this always turns out badly. A related issue is the use of POJOs, which end up being manipulated by the ORM framework. While this sounds nice in theory, in practice you can hit various kinds of issues and it can be tempting to mix application logic with what should really amount to data access objects. Developers and architects like to praise the separation of concerns by using Spring and other frameworks, and there’s no real reason why the same concept shouldn’t be applied here. One other minor issue is the need to maintain the POJOs and mapping definition, but this is usually not too much work.

The second approach of generating your ORM mappings and code from your schema is my preferred approach. In my experience of using both approaches, having beans generated off of your schema allows your beans to be intelligently designed and be only as complicated as required, while making available fetch by PK, by index, etc. all for free. It also becomes easier to manage things like lazy collections and referenced objects since it is all managed within the persistent class itself. This approach also avoids the need for writing boilerplate POJOs and forces you to treat your data access objects separately from your domain objects and business logic. In my experience with generating data access beans off your schema, the beans end up being richer, more usable, perform better, and once you have your infrastructure in place, maintenance costs are lower. One may think that you end up needing additional data-wrapper classes, but in practice the need for separate bean classes is independent of what is going on in your data access layer. One issue here is the availability of frameworks to do this generation work for you – in the past, I have worked with custom-built solutions that worked well and paid off, but required initial up-front work. On smaller projects there may not be enough pay-off to be worth investing in such an effort. At the same time, there are ORMs out there that take this approach and generate persistent entity classes, such as jooq, but I have to try them.

Hibernate is the most popular ORM in the Java world and while it is a step up from dealing with writing copious amounts of SQL, it has its problems. Generally the approach is to define your mappings and POJOs and then let Hibernate manage your SQL generation. The problem with this is your defined schema is often less than ideal and things like proper indexing end up overlooked. Hibernate also forces you to end up using their transaction and query mechanisms, though the extent you choose to use their proprietary stuff is up to you. This isn’t necessarily a bad thing in all cases, but personally I have a distaste for the HQL language which is frequently used, since it introduces a familiar-yet-different language to developers which others will later have to maintain and try to figure out. There are also issues with query optimization that can crop up, and having done significant work on performance tuning in the past, access to actual queries for tuning is a must for me. I also believe that trying to implement inheritance in persistence classes is just a bad idea – it’s not worth trying to force a concept onto a technology which naturally does not accommodate it. Hibernate tempts developers to implement inheritance in the database through support for table-per-hierarchy and table-per-class mechanisms, but it is a mistake in my mind since you end up with a poor data model and problems later managing and extending the hierarchy. I also do not like to populate what should be a clean relational model – you can’t pretend relational DBs are object-oriented datastores.

If you take one thing away from this post, it should be to not ignore the actual technologies you are using. Treat an RDBMS for what it is, and learn to use it. Do the same for object-oriented systems. By all means, try to make your life easier by using ORMs to avoid writing boilerplate code and unnecessary SQL, but don’t think you can avoid dealing with some kind of translation or code to deal with the natural mismatch that occurs. Do not expect a framework or tool to solve the problem for you. Developers are paid to think and discerning the best road to take, so we shouldn’t be afraid to deal with problems as they come up and solve them intelligently. As with many things, the 80-20 rule seems to apply here. Use ORMs to take care of 80% of the work for you, and be prepared to write SQL and some persistence code for the other 20%. Don’t expect too much or you will end up with several types of problems – strange bugs, performance issues, a poorly designed object-oriented model, etc.

I’d love to hear your experience and thoughts on ORMs in any language and the issues you’ve faced, along with how you dealt with them. This is one of those topics where experience is incredibly valuable, so please share your thoughts.

Groovy for Scheduling

While we’ve already written a post on our scripting support in Obsidian, this post will focus specifically on our Groovy support, what we use it for and what it makes possible for you.

Given that Groovy is built for the JVM and almost any java code will work as is in a groovy script/class, supporting Groovy in Obsidian makes all sorts of things not only possible, but downright simple. Internally, Obsidian uses Groovy scripts to support a few different dynamic/configurable jobs. First, we have SpringBeanJob. This job is intended to serve one of two purposes. First, maybe you’re already using TimerTask in conjunction with Spring for scheduling. But you’re switched over to Obsidian for a more feature rich scheduling experience. You want to get up and and running quickly? Use SpringBeanJob and get your job running on Obsidian without a build/deployment cycle.

All you need to do is configure this job with the code needed to get an instance of your bean. And you indicate the name of the method(s) to call. Optionally, you can indicate the classes you need to import if you don’t wish to fully qualify your code. Here’s a sample of what the Groovy script will look like.

import com.carfey.myproject.TimerTaskBean
import com.carfey.myproject.SpringContext

def springBeanObject = SpringContext.get(TimerTaskBean.class)

springBeanObject.fireFirstMethod() //maybe run() used to call two separate methods
springBeanObject.fireSecondMethod()

That’s basically how it will work using our standard GroovyJob. Now, let’s consider the second possible use. Maybe you have a wired bean that you want to run on a schedule. To use it with TimerTask or even Quartz/Spring integration, you’d have to go through a build/deploy cycle, if only to modify Spring config. Now look back at the Groovy script above. With Obsidian, it’s the exact same approach as in our last case, with the beauty of no build or deploy. Use SpringBeanJob, get your bean handle, invoke your method(s).

We also use GroovyJob internally to provide a ReflectiveJob which is similar in nature to our SpringBeanJob, but provides support for arguments in Constructor and method calls.

But really, while our free utility jobs are easy to use and can serve some useful purposes, there’s no need to restrict yourself to these specific cases. We have created the GroovyJob so you can do pretty much anything you’d like.

Think about all the cases where dynamic access to an environment might prove useful. Maybe you need to get some data from the database to assist in troubleshooting, or you’d like to throw a quick job out to deal with some emergency condition, or just keep a failing job running by working around a failure condition. Or maybe during a testing cycle, a job is failing and you can’t figure out why. Imagine being able to take the job code, add in some extra logging lines, extract some contextual data from the db, disable the existing job and schedule a GroovyJob with your new script code. And as you get feedback, adjust and rerun. You’re probably starting to see how powerful the GroovyJob can be.

For security purposes, we don’t expose any of our script jobs in our live online demo, but download Obsidian and try them out.

The scripting jobs were available in our initial Obsidian release and these new freely provided utility jobs will be released in the upcoming release, Obsidian 1.3. Keep watching this space for a release announcement.

Java 7 is Pathetic

Java 7 is finally nearing general release, but I have to say that I’m fairly unimpressed by the features being delivered considering Java 6 was released 4 1/2 years ago. It’s already been delayed for years, and what is there to show for it? The time between Java 6 and 7 is almost a third of Java’s entire life up to this point (Java was released in 1996). And nothing major is coming, despite the advancements made in .NET, and the general productivity improvements that have been made in more current languages.

Java is hardly a cutting edge language, and Sun was generally quite conservative with language changes, but the language’s progression seems to have stalled in a major way. The most useful features that were proposed for Java 7 like lambda support, and language support for collections has been deferred to at least Java 8, and even then I doubt it will be delivered when promised (late 2012) considering how long they have taken to deliver a mediocre update in Java 7.

So what did make it in? Generally, it appears like housekeeping items like updating the XML stack, JDBC versions, etc. made it in, in addition to some minor syntax improvements like improved numeric literal support (how long could could this honestly take), automatic resource management, exception multi-catch and diamond-syntax for generic instantiation.

Fortunately, InvokeDynamic support is making it in which keeps the release at least somewhat significant, since this is a boon to developers of dynamic languages on the JVM. The JVM is turning into a very nice platform for various dynamic languages, and it’s nice to see them deliver something language developers have been clamoring for during the last several years.

Other than that, the most useful things coming in the release are probably strings in switch (really though, big deal!) and some concurrency and collections updates. It looks to me like we’ll be waiting a couple more years for any improvements that actually help developer productivity, like closures/lambdas. I’m not sure why Oracle hasn’t been able to get their act together with this release, but it’s unfortunate considering Java will continue to be one of the few go-to languages for business applications for quite a while yet.

For details on everything that is included in Java 7 you can head to OpenJDK. You can check out a preview version of JDK 7.

Scheduler Monitoring Done Right

One of the most important features of Obsidian is the ability to be notified of application events and also quickly locate and correct any issues that arise. While products like Quartz and cron4j give us the basic scheduling, we have to develop our own solutions for monitoring and notification of events. Unfortunately, in many corporate environments, developers are very limited in how much time they can work on infrastructure pieces and things like error handling and monitoring end up getting far too little focus. When you are developing in-house or revenue-generating software, managers just want the product out the door, and aren’t worried about maintenance implications until far too late.

With these realities in mind, we’ve built in 3 levels of functionality to facilitate monitoring. Based on decades of software and operations experience, we’ve identified the primary areas where developers and support teams spend their time, and the ways that we could reduce costs so that using our scheduler would be a huge gain for our clients.

Level One – Logging Console

The first problem we identified is many applications rely on text logs generated by frameworks like log4j to troubleshoot and locate issues. The problem is, these logs can be very verbose, especially over time, and if not correctly configured, the information you need to troubleshoot an issue just may not be there. It is also a major pain to find what you are looking for within these logs. Add in issues with access control for production environments and delays from change requests, and you have a major headache on your hands.

To alleviate this problem, we developed a dashboard console within our scheduler web application. This dashboard has records of dozens of types of application events. This dashboard can be made accessible to read-only users, and can even be used by support teams with little software experience to see issues when they arise. Events can be filtered by type, time, severity and the contents of the message. Any time an event worth logging occurs, it is put into the dashboard history and can then be navigated from the dashboard console.

The real beauty of this approach is that the information is available at your fingertips and can easily be located again and again with a few simple clicks. All events are logged so there is no chance of missing information because it is at too fine a level, and you can easily configure our default job to clean old dashboard information. It also avoids the issue of granting production access to support teams, or having to contact administrators to provide logs.

Obsidian Log Dashboard

Level Two – Notifications

Another big issue of system monitoring and maintenance is knowing immediately when things go wrong, or when unexpected things happen. We knew we needed some kind of notification support, but we had to carefully consider the best way to provide this to our users so they have the flexibility and simplicity they require to maintain their applications inexpensively.

Since we had such a clean and detailed system to log application events, we leveraged this to provide event notifications. Every event may be tied to a specific entity – this may be a scheduled job, a job configuration, etc. We also have severity for every event which range from Trace to Fatal (inspired by logging levels).

The natural way to approach notification was to allow users to subscribe to a certain severity level for events of a certain type. Not only can they be alerted when errors occur, but they can also subscribe to informational events or warnings. Users can also target a specific entity to limit when they are notified – for example, they may wish to know when any job fails, or when a specific job fails. We outlined every type of subscribable event and exposed a clean and simple interface within our administration console so that users could quickly change their event subscriptions, and even temporarily disable them.

We include email notification support out of the box currently and this also gives virtually all mobile users access to SMS notifications as well – see Wikipedia’s list of SMS email gateways to see the address to use for your carrier.

Obsidian Notifications

Level Three – Full and Complete Application Logging

While we log just about any event of any relevance within Obsidian, we also know that organizations have current standards and infrastructure for logging in place. In response to this, we use log4j to log all events logged to the dashboard, plus additional diagnostic information. Since we use the well-established log4j framework, you have access to all its appender types including text appenders, network appenders and JDBC appenders. In this way, developers can have customize Obsidian’s logging to fit their needs. We still believe our notification and event logging approach is tough to beat, but we want to make life as easy for our customers as possible. This approach also makes it easy for our clients to embed Obsidian within their applications while keeping logging consistent and simple.

If you have any questions about our experience or would like to suggest a feature for Obsidian’s monitoring and logging, leave a comment.

Obsidian Scheduler 1.2 Released!

We’ve added more to Obsidian Scheduler. Version 1.2 is now available here.

Features added in this release include

  • Description annotation on job class enables inline help in admin web app
  • Scheduled one-time runs
  • New Job Status supporting any unscheduled events (ad hoc & chaning)
  • Bundled Winstone support for quick start
  • Option to wait indefinitely for job completions on graceful shutdown
  • Licence key release on graceful shutdown

As always, our live online demo has been updated with this latest release. Fully functional and completely open for your use at http://demo.carfey.com.