Java Enterprise Software Versus What it Should Be

A lot of developers end up in the Java “enterprise” world at some point in their careers. I know the term alone conjures up all kinds of reactions, and rightly so. Often environments where lots of interesting technical challenges exist end up being those that nobody wants to work on because they are brittle, difficult to work on and just no fun. Often the problems that crop up in big projects are due to management, but I’ve seen developers make lots of bad decisions that lead to awful pieces of software, all in the name of “enterprise”.

What is Enterprise?

You could argue that the term can mean just about anything, and that’s true, but for the sake of this article I’m going to define it in a way that I think lines up with the common usage. The average enterprise project has these attributes:

  • typically in a large corporate environment
  • multiple layers of management/direction involved
  • preference for solutions from large vendors like Red Hat, IBM or Microsoft
  • preference for well-known, established (though sometimes deficient) products and standards
  • concerns about scaling and performance

Now that I’ve defined what type of project we are talking about, let’s see what they usually end up looking like.

The Typical Enterprise Java Project

Most of us have seen the hallmarks of enterprise projects. It would help if we had an example, so let’s pretend it’s an e-commerce platform with some B2B capabilities. Here’s what it might look like:

  • EJB3 plus JPA and JSF – these fit a “standard” and everyone use them, so it’s safe choice.
  • SOAP – it’s standard and defines how things like security should work, so there’s less to worry about.
  • JMS Message-Driven Beans – it fits into the platform and offers reliability and load-balancing.
  • Quartz for job scheduling – a “safe” choice (better the enemy you know than the devil you don’t).
  • Deployed on JBoss – it has the backing of a large company and paid support channels.

Now, the problem with a project like this isn’t necessarily the individual pieces of technology selected. I definitely have issues with some of the ones in my example, but the real issue is how the choices are made and what the motivations are for using certain technologies.

The stack of software above is notoriously more difficult to manage and work with compared to other choices. Development will take longer to get off the ground, changes will be more difficult to make as requirements evolve, and the project will ultimately end up more complicated than other possible solutions.

Enterprise Decision-Making

The goals that enterprise project usually fixate on when making choices are:

  • Low-risk technology – make a “safe” choice that won’t result in blow-back, even if it is known to have serious drawbacks.
  • Standards obsession – worrying more about having well-defined specifications like EJB3 or SOAP than offering the simplest solution that does the job effectively.
  • Need for paid support with an SLA, often without concern to the quality or timeliness of responses.
  • Designing out of fear of unknown future requirements.

These goals aren’t bad ones, with the exception of the last one, but they tend to overshadow the real goals of every software project. The primary objective of all software projects is to deliver a project that:

  • is on-time;
  • meets requirements;
  • is reliable;
  • performs well; and
  • is easy to maintain and extend.

These should be what decision-makers focus on in software projects, whether small or large. It’s obvious that sometimes special organizational needs factor into the choices that are made, but fundamentally good choices generally apply in all types of organizations.

So what if we were to reimagine our project with those goals in mind instead?

A Reimagined Enterprise Project

First, a little disclaimer: there are many ways one can go in any project, and I won’t claim that the technologies below are better that the ones mentioned previously. Tools need to be evaluated according to your needs, and everyone’s are different.

What I will try to do is demonstrate an example technology stack along with the reasoning for each choice. This will show how well designed systems can be built which survive in the enterprise world without succumbing to the bad choices that are often made.

Here’s the suggested stack:

  • Spring MVC using Thymeleaf – stable history, lots of development resources, quick development and flexibility. Don’t be afraid of using platforms or libraries, but try to avoid too much “buy-in” to their stack that you might regret.
  • Simple database layer using jOOQ for persistence where useful. This lets us manage performance in a more fine-grained way, while still getting easy database interaction and avoiding ORM pitfalls.
  • REST using Jackson JSON processor – REST and JSON are both popular because they are easy-to-use and understand, cheap to develop, use simple standards and are familiar to developers. Lock-in isn’t much of a problem either – we could easily switch to another JSON processor without much difficulty, unlike SOAP. This can be easily secured with SSL and basic authentication.
  • JMS messaging using JSON-encoded messages on ActiveMQ – loose coupling, reliability and load-balancing, without the nastiness of being stuck with Message-Driven Beans.
  • Obsidian Scheduler – simple-to-use, offers excellent monitoring and reduces burden on developers. Once again, the goal is to simplify and reduce costs where possible.
  • Deployed on Tomcat – no proprietary features used. This helps us follow standards, avoid upgrade issues and keep things working in the future. Who needs support with SLAs when things aren’t inexplicably breaking all the time?

I think the stack and corresponding explanations above help give an idea to what an enterprise project can be if it’s approached from the right angle. The idea is to show that even enterprise projects can be simple and be nimbly built – bloated frameworks and platforms aren’t a necessary part, and rarely offer any significant tangible benefit.

In Closing

Recent trends in development to technologies like REST have been very encouraging and inroads are being made into the enterprise world. Development groups are realizing that simplicity leads to reliability and cost-effective solutions long as the underlying technology choices meets the performance, security, etc. needs of the project.

The software world moves quickly, and is showing promising signs that it’s moving in the right direction. I can only hope that one day the memories of bloated enterprise platforms fade into obscurity.

Using Obsidian Maintenance Jobs

When we began development on Obsidian Scheduler, one of its key objectives was transparency. Quoting my own post,

transparent means letting us know what is going on. We want to have available to us information such as when was the job scheduled to run, when did it start, when did it complete, which host ran it (in pool scenarios). We want to know if it failed, what the problem was. We want the option to be notified either being emailed, paged, and/or messaged when all or specific jobs fails. We want detailed information available to us should we wish to investigate problems in our executing jobs. And we want all of this without having to write code, create our own interface, parse log files or do extra configuration. The scheduler should just work that way.

We’re very happy that we achieved that goal and Obsidian gives us the transparency we sought.

As you likely know, Obsidian stores all this information in a database. This allows you to retain as much of this history as you desire or are required by your business group’s policy. In theory, you could retain all of this history indefinitely in the live database ensuring this information would be available at any time simply by accessing the Obsidian UI. In practice, this is often not required. Many organizations simply maintain an indefinite archive of database backups and can retrieve and restore these to an environment they would use specifically to perform any desired historical investigations.

Obsidian is bundled with two maintenance jobs that can be used to keep the Obsidian database trim and holding only the necessary recent history. These maintenance jobs take advantage of Obsidian’s built-in ability for parametrization allowing you to choose the desired retention period. These jobs are not scheduled by default, so let’s take a look at how we can configure them to run.


This job cleans up the execution history of your jobs, retaining the most recent history. When scheduling a new job in Obsidian, you will find the Job Class com.carfey.ops.job.maint.JobHistoryCleanupJob in the selection list. Once chosen, you will see that it supports a single required parameter – maxAgeDays. Assuming you want job execution history retained for the most recent 90 days, set this value to 90 days. Here’s a screenshot of what it would look like.


This job cleans up the audit and information logs that track all activity in Obsidian. As described in our wiki, these logs contain everything from scheduler system activity such as spawning and execution to user activity such as changing a schedule, its configuration or adding new jobs. These logs are classified by severity level and the Job Class com.carfey.ops.job.maint.LogCleanupJob is configured using the same required parameter maxAgeDays, but has an additional required parameter – level. It defaults to ALL and allows multiple values, but you can also retain this log data for different retention periods by severity by scheduling and configuring this job for each classification. Notice these samples:

That’s all there is to it! In both cases, the absence of a configured maintenance job or in the case of the LogCleanupJob, configured for a given severity level, means that data will be retained indefinitely. Of course, you can always decide later on to configure such a job and at that time Obsidian will take care of it for you.

Is there something else you’d like to see Obsidian do? Drop us a line or leave a comment below. We listen carefully to all our customers’ feature requests and give priority to customer’s needs in our product roadmap. We also appreciate hearing how Obsidian is helping you.

Configuring Clustering in Quartz and Obsidian Schedulers

Job scheduling is used on many software projects to enable both internal jobs and third-party integration. Clustering can provide a huge boost to reliability by providing fail-over and load-sharing. I believe that clustering should be implemented for reliability on just about all software projects, so I’ve decided to outline how to go about doing that in two popular cluster-enabled Java job schedulers. This post is going to cover how to set up clustering for Quartz and Obsidian. It will explain what work is required to configure each, and help you watch for some common pitfalls. This guide will assume you have both schedulers running in the base configuration already.

Both Quartz and Obsidian have their strong points, and this post won’t debate which is better, but it will provide the information you need to cluster either one.


Quartz is the most popular open-source job scheduling option, and it allows for you to cluster scheduler instances via the JDBCJobStore. Though it’s also possible to do clustering with Terracotta without a persistent job store, I recommend against it for most projects since job execution history is very useful to ongoing operations and troubleshooting, and database-clustering performs adequately in all but extreme cases.


Obsidian is a commercial job scheduler which provides free individual instances, and a single clustering licence free for a year. It provides a similar type of clustering as Quartz, and it also provides a full UI, a REST API, downtime recovery, any many other advanced configuration options.

Configurating Obsidian for Clustering

I’ll cover Obsidian first simply because there’s little to do in comparison to Quartz. Since running Obsidian always uses a database, if you have an instance running, there will be no database configuration to update. If you haven’t set it up yet, check out the Getting Started guide – the installation package comes with an interactive Ant tool to build the properties file for you.

So here’s the thing: to cluster Obsidian, simply start up additional instances. Obsidian doesn’t have a non-clustered mode, and all instances automatically handle adjusting the load-sharing algorithm when new members join the cluster (or drop out). Your system clocks should be roughly in sync, but if they differ by a few seconds, it’s not a problem since Obsidian gracefully handles this. Still, you may synchronize your server times if desired.

Note to those using the free version: right after download, you can start two instances and they will automatically join the same cluster. If you do not have adequate licences, new members will not be able to join the cluster.

There’s no need to deploy different properties files or anything like that. As an example, if you use Amazon’s EC2 service, you could use the same image for multiple nodes in the cluster and everything would work correctly. Each cluster member will automatically assign itself a unique instance name based on its local host name and a unique suffix if required.

However, we recommend you assign each cluster member an explicit name which will help with troubleshooting if there are issues on a specific host. To do so, simply set the Java system property schedulerDesignation to the host name of your choice. For example, if starting an instance using the standalone scheduler using java directly, simply add the value -DschedulerDesignation=myHostName to the end of the command.

That’s it! Clustering Obsidian is literally as easy as copying an installation and starting it on another server or virtual machine. As a bonus, unlike other schedulers, in Obsidian you can set a job to only run on a specific host, even with clustering enabled.

In summary, the steps are:

  1. Grab a copy of your Obsidian installation (WAR, standalone package or embedded application bundle).
  2. Start it up! At this step you can provide the schedulerDesignation you wish to use.

Configurating Quartz for Clustering

For Quartz, we’re only going to cover configuring database clustering since we believe it is the right choice to most projects.

Note before you get started: If you have a non-clustered version of Quartz running, you must shut it down before starting any cluster-enabled instances.

Another note about timing: Since Quartz uses very aggressive timing, you must ensure your different instances have precisely synchronized times. Even a second difference will cause your load-sharing to be effectively disabled and all jobs will run on a single host.

Setting up clustering for Quartz can be a bit overwhelming since it exposes so many properties, and it’s hard to figure out which must be added to your properties file to get up and running, but I will try to simplify the process as much as possible.

Quartz is configured via the file. Here’s a sample file that outlines the bare minimum properties you will have to configure for clustering. If you want to see full details on any portion of the config, see the Quartz configuration reference. Note that the properties file should be identical on all hosts, with the one exception being the org.quartz.scheduler.instanceId which is used to identify different hosts.

# Basic Quartz configuration to provide an adequate pool of threads for execution

org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 25

# Datasource for JDBCJobStore
org.quartz.dataSource.myDS.driver =com.mysql.jdbc.Driver
org.quartz.dataSource.myDS.URL = jdbc:mysql://localhost:3306/scheduler
org.quartz.dataSource.myDS.user = myUser
org.quartz.dataSource.myDS.password = myPassword

# JDBCJobStore
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.dataSource = myDS

# Turn on clustering
org.quartz.jobStore.isClustered = true

org.quartz.scheduler.instanceName = ClusteredScheduler
# If instanceIf is set to AUTO, if will auto-generate an id automatically.
# I recommend giving explicit names to each clustered host for easy identification.
org.quartz.scheduler.instanceId = Host1

Note about JDBC settings: For whatever reason, you have to configure both the JDBC driver class and the job store’s “driver delegate” class. These will have to be set to the appropriate value for your database platform.

As I mentioned, these are the bare minimum properties you will have to configure to get clustering running. The one big pain with this configuration is that the instanceId which identifies hosts resides in the same properties file as all the other properties which should all remain the same. Keeping different versions in sync can problematic, and isn’t required for Obsidian. You can use the “AUTO” setting to avoid having to set explicit instance IDs, but as with Obsidian, we recommend you give explicit names as it can help you locate where issues are happening more quickly if you know up front which host is which.

So the main steps to enabling clustering are:

  1. Prepare properties for each cluster member. Ensure only the org.quartz.scheduler.instanceId varies in the properties file.
  2. Turn off all running instances by shutting down the application which is running it, or disabling just the Quartz process (see here).
  3. Start your application, or start the Quartz process (see here).
  4. Ensure that you never start a non-clustered instance again!


I hope this helps you get off the ground quickly with your new or existing project. Clustering is a great feature in any scheduler, and I feel it provides a lot of value that software and operations teams might be missing out on.

Null and 3-Dimensional Ordering Helpers in Java

When dealing with data sets retrieved from a database, if we want them ordered, we usually will want to order them right in the SQL, rather than order them after retrieval. Our database will typically be more efficient due to available processing power, potential use of available indexes and overall algorithm efficiency in modern RDBMSes. You also have great flexibility to have complex ordering criteria when you use these ORDER BY clauses. For example, assume you had a query that retrieved employee information including salary and relative steps (position) from the top position. You could easily have a first-level ordering where salaries are grouped into categories (<= $50 000, $50 001 to $100 000, > $100 001), and the next level ordered by relative position. If you could assume that salaries were appropriate for all employees, this might give you a nice idea of where there is too much or too little management in the company – a very rough approach I wouldn’t recommend, this is just a sample usage.

You get some free behaviour from your database when it comes to ordering, whether you realize it or not. When dealing with NULLs, the database has to make a decision how to order them. Every database I’ve ever worked with and likely all relational databases have a default behaviour. In ascending order, MySQL and SQL Server put NULL ahead of real values, they are “less than” a non-NULL value. Oracle and Postgres put NULL after real values, they are “greater than” non-NULL values. Oracle and Postgres nicely give you the NULLS FIRST and NULLS LAST instructions so you can actually override the defaults. Even in MySQL and SQLServer, you can find ways to override the defaults using functions in your order by clause. In MySQL I use IFNULL. In SQL Server, you could use ISNULL. These both give you the option of replacing null with a particular value. Just replace an appropriate value for the type you are sorting.

All sorting supported in these types of queries is two-dimensional. You pick columns and the rows are ordered by those. When you need to sort by additional dimensions of the data, you’re probably getting into areas that are addressed in other related technologies such as data warehousing and OLAP cubes. If that is appropriate and available for your case, by all means use those powerful features.

In many cases though, we either don’t have access to those technologies or we need our operations to be on current data. For example, let’s say you are working on an investment system, investor’s accounts, trades, positions, etc. are all maintained. You need to write a query to help extract trade activity for a given time frame. Our data comes back as a two-dimensional datasets even though we have more dimensions. Our query will return data on account(s) and the trade(s) per account. We need our results to be ordered by those accounts whose effected trades have the highest value. But we need to maintain the trades with their accounts. Simply ordering our query by the value of the effected trade would likely break the rows of the same account apart.

We have a choice, we can either order in the database and control our reconstruction of returning data to maintain the state and order of reconstructed objects or we can sort after the fact. In most cases, we probably don’t want to write new code each time we come across this problem that deals specifically with reconstituting the data from that query into our object model’s representation. Hopefully our ORM will help or we have some preexisting, functional and well-tested code that we can reuse to do so.

Another option is to sort in our code. We actually get lots of flexibility by doing this. Perhaps we have some financial functions that are written in our application that we can now use. We also don’t have to do all the sorting ourselves as we can take advantage of JDK features for Comparator and Collection sorting.

First, let’s deal with our null ordering problem. Let’s say our Trade object has some free public constant Comparators. These allow us to use a collection of Trades along with java.util.Collections.sort(List<Trade>, Comparator<Trade>). Trade.TRADE_VALUE_NULL_FIRST is the one we want to use. This Comparator is nothing more than a passthrough to a global Null Comparator helper.

private static final Comparator<Trade> TRADE_VALUE_NULL_FIRST = new Comparator<Trade>(){
  public int compare(Trade o1, Trade o2) {

... ComparatorUtil ...

public static NullFirstComparator<BigDecimal> DECIMAL_NULL_FIRST_COMPARATOR = 
  new NullFirstComparator<BigDecimal>();
public static NullLastComparator<BigDecimal> DECIMAL_NULL_LAST_COMPARATOR = 
  new NullLastComparator<BigDecimal>();
public static NullLastComparator<String> STRING_NULL_LAST_COMPARATOR = 
  new NullLastComparator<String>();

public static class NullFirstComparator<T extends Comparable<T>> implements Comparator<T> {
  public int compare(T o1, T o2) {
    if (o1 == null && o2 == null) {
	return 0;
    } else if (o1 == null) {
	return -1;
    } else if (o2 == null) {
	return 1;
    } else {
	return o1.compareTo(o2);
public static class NullLastComparator<T extends Comparable<T>> implements Comparator<T> {
  public int compare(T o1, T o2) {
    if (o1 == null && o2 == null) {
	return 0;
    } else if (o1 == null) {
	return 1;
    } else if (o2 == null) {
	return -1;
    } else {
	return o1.compareTo(o2);

Now we have a simple, reusable solution we can use with any class and any nullable value in JDK sorting. Now we expose any ordering constants appropriate for business usage in our class. Now let’s deal with the more complex issue of hierarchical value ordering. We don’t want to write new code everytime we have to do something like this. So let’s just extend our idea of ordering helpers to hiearchical entities.

public interface Parent<C> {
  public List<C> getChildren();
public class ParentChildPropertiesComparator<P extends Parent<C>, C> implements Comparator<P> {
  private List<Comparator<C>> mChildComparators;
  public ParentChildPropertiesComparator(List<Comparator<C>> childComparators) {
    mChildComparators = Collections.unmodifiableList(childComparators);
  public List<Comparator<C>> getChildComparators() {
    return mChildComparators;
  public int compare(P o1, P o2) {
    int compareTo = 0;
    for (int i=0; i < mChildComparators.size() && compareTo == 0; i++) {
	Comparator<C> cc = mChildComparators.get(i);
	List<C> children1 = o1.getChildren();
	List<C> children2 = o2.getChildren();
	Collections.sort(children1, cc);
	Collections.sort(children2, cc);
	int max = Math.min(children1.size(), children2.size());
	for (int j=0; j < max && compareTo == 0; j++) {
	  compareTo =, children2.get(j));
    return compareTo;

This is a little more complex, but still simple enough to easily grasp and reuse. We have the idea of a parent. This is not an OO relationship. This is a relationship of composition or aggregation. A parent can exist anywhere in the hierarchy, meaning a parent can also be a child. But in our sample, we have a simple parent/child relationship - Account/Trade. This new class, ParentChildPropertiesComparator supports the idea of taking in a List of ordered Comparators on the children entities but sorting on the parents. In our scenario, we are only sorting on one child value, but it could easily be several just as you could sort more than 2 levels of data.

We are assuming in our case that Account already implements the Parent interface for accounts. If not, you can always use the Adapter Design Pattern. Our Account/Trade sorting would now look like this.

List<Account> accounts = fetchPreviousMonthsTradeActivityByAccount();
List<Comparator<Trade>> comparators = Arrays.asList(Trade.TRADE_VALUE_NULL_FIRST);
ParentChildPropertiesComparator<Account, Trade> childComparator = 
  new ParentChildPropertiesComparator<Account, Trade>(comparators);
Collections.sort(accounts, childComparator);

Really! That's it. Our annoying problem of sorting accounts by those with highest trade values where some of those trade values could be null is solved in just a few lines of code. Our accounts are now sorted as desired. I came up with this approach and it is used successfully as a part of a query builder for a large-volume financial reconciliation system. Introduction of new sortable types and values requires only minimal additions. Take this approach for a whirl and see how incredibly powerful it is for dealing with sorting requirements across complex hierarchies of data. And drop us a line if you need help in implementation or have any comments.

Problems with ORMs Part 2 – Queries

In my previous post on Object-Relational Mapping tools (ORMs), I discussed various issues that I’ve faced dealing with the common ORMs out there today, including Hibernate. This included issues related to generating a schema from POJOs, real-world performance and maintenance problems that crop up. Essentially, the conclusion is that ORMs get you most of the way there, but a balanced approach is needed, and sometimes you just want to avoid using your ORM’s toolset, so you should be able to bypass it when desired.

One huge flaw in modern ORMs that I see though is that they really want to help you solve all your SQL problems. What do I mean by this why would I say this is a fault? Well, I believe that Hibernate et al just try too hard and end up providing features that actually hurt developers more than they help. The main thing I have in mind when I say this is query support. Actual support for complex queries that are easily maintained is seriously lacking in ORMs and not because they’ve omitted things — it’s just because the tools they provide don’t use SQL, which was designed from the ground up for exactly this purpose.

Experiences in Hibernate

It’s been my experience that when you use features like HQL, frequently you’re thinking about saving yourself a few minutes up front, and there’s nothing wrong with this in itself, but it can cause serious problems. It’s my experience that frequently you end up wanting or needing to replace HQL with something more flexible, either because of a bug fix or enhancement, and this is where the trouble starts.

I consider myself an experienced developer and I pride myself on (usually) not breaking things — to me, that is one of the hallmarks of good developers. When you’re faced with ripping out a piece of code and replacing it wholesale, such as replacing HQL with SQL, you’re basically replacing code that has had a history that includes bug fixes, enhancements and performance tweaks. You are now responsible for duplicating every change to this code that’s ever been made and it’s quite possible you don’t understand the full scope of the changes or the niggling problems that were corrected in the past.

Note that this also applies to all the other query methods that Hibernate provides, including the Query API, and through extension, query support within the JPA. The issue is that you don’t want a solution that is brittle or limited that it has to be fully replaced later. This means that if you need to revert to SQL to get things done, there’s a good chance you should just do that in the first place. This same concept applies to all areas of software development.

So what do we aim for if the basic querying support in ORMs like Hibernate isn’t good enough?

Criteria for a Solid ORM

Bsaically, my personal requirements for an ORM come down to the following:

  • Schema first – generate your model from a database, not the other way around. If you have a platform-agnostic way of specifying DDL for the database, great, but it’s not a deal-breaker. Generating a database from some other domain-specific language or format helps nobody and results in a poorly designed schema.
  • SQL only – if you want to help me avoid writing code, then generate/expose key-based, etc. lookups for me. Don’t ask me to use your query API or some new query language. SQL was invented for queries, so let me use the right tool.
  • Give me easy ways to populate my domain objects from queries. This gives me 99% of what I’ll ever need, while giving me flexibility.
  • Allow me to populate arbitrary Java beans with query results – don’t tie me into your registry of known types.
  • Don’t force me into using a typical transaction container like the one Hibernate or Spring provides – they are a disaster and I’ve never see a practical use for them that made any sense. Let me handle where connections/transactions are acquired and released in my application – typically this only happens in a few places with clear semantics anyway. This can be some abstracted version of JDBC, but let me control it.
  • No clever/magic behaviour in my domain objects – when working with Hibernate, I spend a good time solving the same old proxy and lazy-loading issues. They never end and can’t be solved once-and-for-all which indicates a serious design issue.

Though these points seem completely reasonable to me, I’ve not encountered any ORMs that really meet my expectations, so at Carfey we’ve rolled our own little ORM, and I have to say that weekend projects and just general development with what we have is far easier and faster than Hibernate or the other ORMs I’ve used. What does it provide?

A Simple Utilitarian ORM

  • Java domain classes are generated from a DB schema. There’s no platform-agnostic DDL yet, but it’s on our TODO list. Beans include support for child collections, FK references, but it’s all lazy and optional – the beans support it, but if you don’t use them, there’s no impact. Use IDs directly if you want, or domain objects themselves. Persistence handles persisting dirty objects only, and saves are only done when requested – no magic flush behaviour.
  • Generated domain classes are for persistence only! Stick your business logic, etc. elsewhere.
  • SQL is used for all lookups, including primary key fetches and foreign key relationships. If you need to enhance a lookup, just steal the generated SQL and build on it. Methods and SQL is generated automatically from any indexed column so they are all provided for you automatically and are typesafe. This also provides a warning to the developer – if a lookup is not available in your domain class, it likely will perform poorly since no index exists.
  • Any domain class can be populated from a custom query in a typesafe manner – it’s flexible but easy to use.
  • Improved classes hide the standard JDBC types such as Connnection and Statement for ease of use, but we don’t force any transaction semantics on you, and you can always fall back to things like direct result set handling.
  • Some basic required features like a connection pool, database metadata, and soon, database slave failover.

We at Carfey don’t believe we’ve created some incredible new ORM that surpasses every other effort out there, and there are many features we’d have to add if this was a public project, but what we have works for us, and I think we have the correct approach. And at the very least, hopefully our experience can help you choose how you use your preferred ORM wisely and not spend too much time serving the tool instead of delivering software.

As a final note, if you have experience with ORMs that meet my list of requirements above and you’ve had good experiences with it, I’ve love to hear about it and would consider it for future Carfey projects.

Evolving Document Structures with Morphia and MongoDB

In my previous post on Morphia, I went through some typical usages and mentioned some caveats and workarounds for known problems. I showed how easy it is to work with Morphia and how cleanly it interacts with the Java world.

To follow up on that post, I’m going to discuss how to deal with some real life needs: handling changing schemas and customizing your mapping to handle things like read-only fields and replacing simple fields with complex objects.

Changing Schemas

As nearly anyone who has worked with databases in the development world knows, schemas are always evolving. Fields get deprecated or outright dropped, tables become obsolete, new fields are added, and so on.

While a lot of this pain is avoided by using a schemaless datastore like MongoDB, sometimes we still do need special handling for changes, and in the case of Morphia, we essentially have defined a schema, so we do have to find ways to deal with this. The nice part about it is that Morphia makes it very clean and easier than you’ll see in just about in any ORM.

Deprecating Fields

One good example is a deprecated field that has been replaced by another field. Let’s imagine you have a bug tracking system with documents that look something like this:

  desc: "IE Rendering broken on intranet site",
  componentName: "INTRANET",
  dateCreated: ISODate("2011-09-06T20:52:50.258Z")

Here is the Morphia definition:

class Issue {
  @Id private long id;
  private String desc;
  private String componentName;

  private Date dateCreated = new Date();

Now imagine at some point we decide to do away with the component field and make it a more generic free text field where users can enter multiple components, versions, or other helpful information. We don’t want to just stick that in the component field, as that would lead to confusion.

Thankfully, we have a something in the Morphia toolkit that is made exactly for this – The @AlsoLoad annotation. This annotation allows us to populate a POJO field with one of multiple possible sources. We simply update our Morphia mapping to indicate an old field name, and we can easily remove references to the old field without breaking anything. This keeps our code and documents clean.

class Issue {
  @Id private long id;
  private String desc;
  @AlsoLoad("componentName") // handle old componentName field
  private String affects;

  private Date dateCreated = new Date();

So here we’ve defined automatic translation of our old field without any need to update documents or write special logic within our POJO class to handle documents differently depending on when they were created.

One important note: in this example, if both the affects field and the old componentName field exist, Morphia will throw an exception, so don’t try using this for anything other than deprecating fields, or perhaps populating a single field with two mutually exclusive properties.

Supporting Read-Only for Deprecated Fields

Another possibility is that you just have to support an old field in document that the application no longer writes. This is a very simple one: use the @NotSaved annotation. When you use this on a field, the data will be loaded but not written by Morphia.

In our previous example, we could just as easily have decided to just support display for the old field but not treat populate it into the affects field, so let’s alter our Morphia POJO a bit to show how @NotSaved is used.

class Issue {
  @Id private long id;
  private String desc;
  private String affects;
  @NotSaved("componentName") // load old componentName field for display only
  private String componentName
  private Date dateCreated = new Date();

Replacing a Field with An Embedded Object

Now what if our componentName field had actually changed to a complex component object which has a name, version and build number? This is a bit trickier since we want to replace one field with multiple. We can’t attempt to load the field from multiple sources since they have different structures. Of course, we can use an embedded object to store the complex component information, but how can we make our code work seamlessly either way without having to update our documents?

In this case, the simplest approach would be to use a combination of three annotations. First we would mark the old field with the @NotSaved annotation, introduce a new embedded Component object using the @Embedded annotation, and finally take advantage one more annotation that Morphia provides – @PostLoad. This one lets us have a method that is executed after the POJO is populated from MongoDB.

Here’s the example:

class Issue {
  @Id private long id;
  private String desc;
  private String affects;
  @NotSaved("componentName") // load old componentName to convert to component
  private String componentName
  @Embedded // our new complex Component
  private Component component;
  private Date dateCreated = new Date();
  // getters and setters ...
  protected void handleComponent() {
      if (component == null && componentName != null) {
        component = new Component(componentName, null, null);

class Component {
  private String componentName;
  private Long version;
  private Long buildNumber;
  public Component(String componentName, Long version, Long buildNumber) {
    // ...
  // getters and setters ...

In this case, we could remove the getter and setter for the componentName field, so that our mapped object only exposes the new and improved interface.


By using the powerful tools that Morphia gives us through its annotation support, we can meet these goals:

  1. Let our document structure adapt with the application and stay clean.
  2. Seamlessly handle changing structure in our Java code without error-prone code.
  3. Expose only the new schema while supporting the old (truly obsolete the old code and fields.

Hopefully this helps a few of you out with adapting to evolving documents, or at least to become more familiar with the abilities some of these Morphia annotations give you.

Problems with ORMs

Object Relational Mapping tools like Hibernate have helped developers make huge productivity gains in dealing with relational databases in the past several years. ORMs free developers to focus on application logic and avoid writing a lot of boilerplate SQL for simple tasks like inserts or queries.

However, the well-documented problems with object-relational impedance mismatch inevitably cause headaches for developers. Relational databases are a specialized technology built on sound concepts, but they don’t necessarily line up to the object-oriented world. There are a few approaches and styles to using ORMs which have various pros and cons.

One of the fundamental choices to using an ORM is deciding whether the you will generate your ORM mappings from the database schema or the other way around, generating your database schema off of some ORM definition (possibly an XML configuration file, annotations or something like XDoclet).

The former approach of generating your ORM layer from your database schema means you have to deal with the database in its own language and terms, whether you deal with DDL specific to the database or have some abstraction layer, but nevertheless you are forced to treat the database as what it is. Unfortunately, it means you need expertise in the technology and it may take more work than allowing your schema to be generated. However, this forces developers to understand and deal with the RDBMS properly – it is dangerous and harmful to treat a DBMS as a simple datastore. Developers need to consider the impact of keys, indexes, etc. when designing applications, and shielding them from the realities of relational databases can be dangerous, and in my experience this always turns out badly. A related issue is the use of POJOs, which end up being manipulated by the ORM framework. While this sounds nice in theory, in practice you can hit various kinds of issues and it can be tempting to mix application logic with what should really amount to data access objects. Developers and architects like to praise the separation of concerns by using Spring and other frameworks, and there’s no real reason why the same concept shouldn’t be applied here. One other minor issue is the need to maintain the POJOs and mapping definition, but this is usually not too much work.

The second approach of generating your ORM mappings and code from your schema is my preferred approach. In my experience of using both approaches, having beans generated off of your schema allows your beans to be intelligently designed and be only as complicated as required, while making available fetch by PK, by index, etc. all for free. It also becomes easier to manage things like lazy collections and referenced objects since it is all managed within the persistent class itself. This approach also avoids the need for writing boilerplate POJOs and forces you to treat your data access objects separately from your domain objects and business logic. In my experience with generating data access beans off your schema, the beans end up being richer, more usable, perform better, and once you have your infrastructure in place, maintenance costs are lower. One may think that you end up needing additional data-wrapper classes, but in practice the need for separate bean classes is independent of what is going on in your data access layer. One issue here is the availability of frameworks to do this generation work for you – in the past, I have worked with custom-built solutions that worked well and paid off, but required initial up-front work. On smaller projects there may not be enough pay-off to be worth investing in such an effort. At the same time, there are ORMs out there that take this approach and generate persistent entity classes, such as jooq, but I have to try them.

Hibernate is the most popular ORM in the Java world and while it is a step up from dealing with writing copious amounts of SQL, it has its problems. Generally the approach is to define your mappings and POJOs and then let Hibernate manage your SQL generation. The problem with this is your defined schema is often less than ideal and things like proper indexing end up overlooked. Hibernate also forces you to end up using their transaction and query mechanisms, though the extent you choose to use their proprietary stuff is up to you. This isn’t necessarily a bad thing in all cases, but personally I have a distaste for the HQL language which is frequently used, since it introduces a familiar-yet-different language to developers which others will later have to maintain and try to figure out. There are also issues with query optimization that can crop up, and having done significant work on performance tuning in the past, access to actual queries for tuning is a must for me. I also believe that trying to implement inheritance in persistence classes is just a bad idea – it’s not worth trying to force a concept onto a technology which naturally does not accommodate it. Hibernate tempts developers to implement inheritance in the database through support for table-per-hierarchy and table-per-class mechanisms, but it is a mistake in my mind since you end up with a poor data model and problems later managing and extending the hierarchy. I also do not like to populate what should be a clean relational model – you can’t pretend relational DBs are object-oriented datastores.

If you take one thing away from this post, it should be to not ignore the actual technologies you are using. Treat an RDBMS for what it is, and learn to use it. Do the same for object-oriented systems. By all means, try to make your life easier by using ORMs to avoid writing boilerplate code and unnecessary SQL, but don’t think you can avoid dealing with some kind of translation or code to deal with the natural mismatch that occurs. Do not expect a framework or tool to solve the problem for you. Developers are paid to think and discerning the best road to take, so we shouldn’t be afraid to deal with problems as they come up and solve them intelligently. As with many things, the 80-20 rule seems to apply here. Use ORMs to take care of 80% of the work for you, and be prepared to write SQL and some persistence code for the other 20%. Don’t expect too much or you will end up with several types of problems – strange bugs, performance issues, a poorly designed object-oriented model, etc.

I’d love to hear your experience and thoughts on ORMs in any language and the issues you’ve faced, along with how you dealt with them. This is one of those topics where experience is incredibly valuable, so please share your thoughts.

Using MongoDB with Morphia

In the past few years, NoSQL databases like CouchDB, Cassandra and MongoDB have gained some popularity for applications that don’t require the semantics and overhead of running a traditional RDBMS. I won’t get into the design decisions to go into choosing a NoSQL database as others have done a good enough job already, but I will relate my experience with MongoDB and some tricks on using it effectively in Java.

I recently have had a chance to work with MongoDB (as in humongoous), which is a document-oriented database written in C++. It is ideal for storing documents which may vary in structure, and it uses a format similar to JSON, which means it supports similar data types and structures as JSON. It provides a rich yet simple query language and still allows us to index key fields for fast retrieval. Documents are stored in collections which effectively limit the scope of a query, but there is really no limitation on the types of heterogeneous data that you can store in a collection. The MongoDB site has decent docs if you need to learn the basics of MongoDB.

MongoDB in Java
The Mongo Java driver basically exposes all documents as key-value pairs exposed as map, and lists of values. This means that if we have to store or retrieve documents in Java, we will have to do some mapping of our POJOs to that map interface. Below is an example of the type of code we would normally have to write to save a document to MongoDB from Java:

BasicDBObject doc = new BasicDBObject();

doc.put("user", "carfey");

BasicDBObject post1 = new BasicDBObject();
post1.put("subject", "spam & eggs");
post1.put("message", "first!");

BasicDBObject post2 = new BasicDBObject();
post2.put("subject", "sorry about the spam");

doc.put("posts", Arrays.asList(post1, post2));


This is fine for some use cases, but for others, it would be better to have a library to do the grunt work for us.

Enter Morphia
Morphia is a Java library which acts sort of like an ORM for MongoDB – it allows us to seamlessly map Java objects to the MongoDB datastore. It uses annotations to indicate which collection a class is stored in, and even supports polymorphic collections. One of the nicest features is that it can be used to automatically index your collections based on your collection- or property-level annotations. This greatly simplifies deployment and rolling out changes.

I mentioned polymorphic storage of multiple types in the same collection. This can help us map varying document structures and acts somewhat like a discriminator in something like Hibernate.

Here’s an example of how to define entities which will support polymorphic storage and querying. The Return class is a child of Order and references the same collection-name. Morphia will automatically handle the polymorphism when querying or storing data. You would pretty much do the same thing for annotating collections that aren’t polymorphic, but you wouldn’t have multiple classes using the same collection name.

Note: This isn’t really an example of the type of data I would recommend storing in MongoDB since it is more suited to a traditional RDBMS, but it demonstrates the principles nicely.

@Entity("orders") // store in the orders collection
@Indexes({ @Index("-createdDate, cancelled") }) // multi-column index
public class Order {
    @Id private ObjectId id; // always required

    private String orderId;
    @Embedded // let's us embed a complex object
    private Person person;
    private List<Item> items;
    private Date createdDate;
    private boolean cancelled;
    // .. getters and setters aren't strictly required
    // for mapping, but they would be here

@Entity("orders") // note the same collection name
public class Return extends Order {
    // maintain legacy name but name it nicely in mongodb
    @Property("rmaNumber") private String rma;
    private Date approvedDate;
    private Date returnDate;

Now, below I will demonstrate how to query those polymorphic instances. Note that we don’t have to do anything special when storing the data. MongoDB stores a className attribute along with the document so it can support polymorphic fetches and queries. Following the example above, I can query for all order types by doing the following. Note that we need to disable validation to use the discriminator class name as a query filter.

// ds is a Datastore instance
Query<Order> q = ds.createQuery(Order.class).filter("createdDate >=", date);
List<Order> ordersAndReturns = q.asList();

// and returns only
Query<Return> rq = ds.createQuery(Return.class)
    .filter("createdDate >=", cutOffDate)
    .filter("className", Order.class.getName());
List<Return> returnsOnly = rq.asList();

If I only want to query plain orders, I would have to use a className filter as follows. This allows us to effectively disable the polymorphic behaviour and limit results to a single target type.

Query<Order> q = ds.createQuery(Order.class)
    .filter("createdDate >=", cutOffDate)
    .filter("className", Order.class.getName());

List<Order> ordersOnly = q.asList();

Morphia currently uses the className attribute to filter results, but at some point in the future is likely to use a discriminator column, in which case you may have to filter on that value instead.

Note: At some point during startup of your application, you need to register your mapped classes so they can be used by Morphia. See here for full details. A quick example is below.

Morphia m = ...
Datastore ds = ...;
ds.ensureIndexes(); //creates all defined with @Indexed
ds.ensureCaps(); //creates all collections for @Entity([email protected](...))

Problems with Varying Structure in Documents

One of the nice features of document-oriented storage in MongoDB is that it allows you to store documents with different structure in the same collection, but still perform structured queries and index values to get good performance.

Morphia unfortunately doesn’t really like this as it is meant to map all stored attributes to known POJO fields. There are currently two ways I’ve found that let us deal with this.

The first is disabling validation on queries. This will mean that values which exist in the datastore but can’t be mapped to our POJOs will be dropped rather than blowing up:

// drop unmapped fields quietly
Query<Order> q = ds.createQuery(Order.class).disableValidation(); 

The other option is to store all unstructured content under a single bucket element using a Map. This could contain any basic types supported by the MongoDB driver including Lists and Maps, but no complex objects unless you have registered converters with Morphia (e.g. morphia.getMapper().getConverters().addConverter(new MyCustomTypeConverter()) .

public class Order {
    // .. our base attributes here
    private Map<String, Object> attributes; // bucket for everything else (

Note that Morphia may complain on startup that it can’t validate the field (since the generics declaration is not strict), but as of the current release version (0.99), it will work with no problem and store any attributes normally and retrieve them as maps and lists of values.

Note: When it populates a loosely-typed map from a retrieved document, it will use the basic MongoDB Java driver types BasicDBObject and BasicDBList. These implement Map and List respectively, so they will work pretty much as you expect, except that they will not be equals() to any input maps or lists you may have stored, even if the structure and content appear to be equal. If you want to avoid this, you can use the @PostLoad annotation to annotate a method which can perform normalization to JDK maps and lists after the document is loaded. I personally did this to ensure we always see a consistent view of MongoDB documents whether they are pulled from a collection or not yet persisted.

If you have questions or don’t understand anything I’ve gone over, just leave a comment and I’ll be glad to help.

Also, check out our post on handling changing document structures in Morphia.

When to use Pessimistic Locking

There are cases where we need to use a pessimistic locking strategy. While optimistic updates are an absolute minimum, we deploy a pessimistic locking strategy into a carefully thought out design. We use pessimistic locking strategies in two primary cases:

  • As a semaphore to ensure only a single process executes a certain block of code at a time
  • As a semaphore for the data itself

Let’s first make sure we agree on the term semaphore. In this context, we’re talking about a binary semaphore, or a token. The token is either available or is held by something. The token is required to be able to proceed with a certain task. A request is made to obtain the token and it is returned if available. Options when it’s not available are to wait for it, blocking until it becomes available or to fail.

Using a semaphore to ensure that certain blocks in your code are only run by a single process at a given time is a great strategy to facilitate load balancing and failover safely. We can now group related changes under token, requiring any other process endeavouring to do the same thing to wait for our lock to release. When the process has terminated either successfully or abnormally, the token is then returned by a commit or rollback.

When used as a semaphore for the data itself, it allows us to ensure the details of the data are consistent for the duration of our block of code. Changes will be prevented thereby ensuring that the actions we take are based on a consistent state for the duration of the code. For example, we may want to make a collection of changes on some foreign key related data based on the parent and need to ensure that the parent data is in an unchanged state. Pessimistically locking the row allows us to guarantee just that. And again, by relying on our existing infrastructure for managing the transaction, either successful completion or failure resulting in a commit/rollback releases our lock.

As mentioned, pessimistic locking does have the potential to cause performance problems. Most database implementations either allow you to not wait at all (thanks Oracle!), or at the very least wait for a maximum amount of time for the lock to be acquired. Knowing your database implementation and its ins and outs is crucial if you’ll be using this approach.

Our new scheduler has been tested with hundreds of concurrent jobs running across dozens of instances without any collisions or deadlocks or any degradation in performance. Drop us a line if you’d like to learn more how we were able to do so.

Optimistic Updates and Transaction Isolation

While we at Carfey Software prefer to run our transactions using READ COMMITTED isolation level, we have taken steps to ensure that our optimistic updates using a version column will work even with READ UNCOMMITTED. How does READ UNCOMMITTED impact optimistic updates? You might start to see that it is the dirty reads, or reads of data changes that won’t necessarily be committed, that cause an issue.

To ensure a safe, accurate optimistic update strategy, you must ensure that whatever value you use for a version column will never again be used on that row – using a GUID is overkill but would do the trick, we use a permanently incrementing value by table. This is because a dirty read of a version column could grab an uncommitted row that is eventually rolled back and a differing transaction modifying and committing the row resulting in a different version represented by the same version number. Let’s visualize the problem.

We’re running READ UNCOMMITTED. Our starting state is modified by a first transaction and we’re using a common strategy of incrementing the version number on the update. Then a second transaction comes along and gets a dirty read.

Afterwards, the first transaction is rolled back. And before transaction 2 does anything, a third transaction reads, writes an increase of 5 to the score and then commits, incrementing the version to 1. When our second transaction comes in for a write and wants to increment the score by 3, it verifies that the version is 1, it incorrectly assumes it can safely make its write and actually only increases the score by 1.

You might think that with dirty reads, safe optimistic updates aren’t possible. But all you need to do is make sure that a version is never used again for a row of data. In our above scenario, if we don’t reuse the version 1, the optimistic write would fail to validate and force a reread. Be aware that if you’re running READ UNCOMMITTED, many ORMs’ optimistic update strategies don’t protect against the above scenario.

As we said, we use optimistic updates as a minimum. Our next post will review cases where we do use pessimistic blocking.