Null and 3-Dimensional Ordering Helpers in Java

When dealing with data sets retrieved from a database, if we want them ordered, we usually will want to order them right in the SQL, rather than order them after retrieval. Our database will typically be more efficient due to available processing power, potential use of available indexes and overall algorithm efficiency in modern RDBMSes. You also have great flexibility to have complex ordering criteria when you use these ORDER BY clauses. For example, assume you had a query that retrieved employee information including salary and relative steps (position) from the top position. You could easily have a first-level ordering where salaries are grouped into categories (<= $50 000, $50 001 to $100 000, > $100 001), and the next level ordered by relative position. If you could assume that salaries were appropriate for all employees, this might give you a nice idea of where there is too much or too little management in the company – a very rough approach I wouldn’t recommend, this is just a sample usage.

You get some free behaviour from your database when it comes to ordering, whether you realize it or not. When dealing with NULLs, the database has to make a decision how to order them. Every database I’ve ever worked with and likely all relational databases have a default behaviour. In ascending order, MySQL and SQL Server put NULL ahead of real values, they are “less than” a non-NULL value. Oracle and Postgres put NULL after real values, they are “greater than” non-NULL values. Oracle and Postgres nicely give you the NULLS FIRST and NULLS LAST instructions so you can actually override the defaults. Even in MySQL and SQLServer, you can find ways to override the defaults using functions in your order by clause. In MySQL I use IFNULL. In SQL Server, you could use ISNULL. These both give you the option of replacing null with a particular value. Just replace an appropriate value for the type you are sorting.

All sorting supported in these types of queries is two-dimensional. You pick columns and the rows are ordered by those. When you need to sort by additional dimensions of the data, you’re probably getting into areas that are addressed in other related technologies such as data warehousing and OLAP cubes. If that is appropriate and available for your case, by all means use those powerful features.

In many cases though, we either don’t have access to those technologies or we need our operations to be on current data. For example, let’s say you are working on an investment system, investor’s accounts, trades, positions, etc. are all maintained. You need to write a query to help extract trade activity for a given time frame. Our data comes back as a two-dimensional datasets even though we have more dimensions. Our query will return data on account(s) and the trade(s) per account. We need our results to be ordered by those accounts whose effected trades have the highest value. But we need to maintain the trades with their accounts. Simply ordering our query by the value of the effected trade would likely break the rows of the same account apart.

We have a choice, we can either order in the database and control our reconstruction of returning data to maintain the state and order of reconstructed objects or we can sort after the fact. In most cases, we probably don’t want to write new code each time we come across this problem that deals specifically with reconstituting the data from that query into our object model’s representation. Hopefully our ORM will help or we have some preexisting, functional and well-tested code that we can reuse to do so.

Another option is to sort in our code. We actually get lots of flexibility by doing this. Perhaps we have some financial functions that are written in our application that we can now use. We also don’t have to do all the sorting ourselves as we can take advantage of JDK features for Comparator and Collection sorting.

First, let’s deal with our null ordering problem. Let’s say our Trade object has some free public constant Comparators. These allow us to use a collection of Trades along with java.util.Collections.sort(List<Trade>, Comparator<Trade>). Trade.TRADE_VALUE_NULL_FIRST is the one we want to use. This Comparator is nothing more than a passthrough to a global Null Comparator helper.

private static final Comparator<Trade> TRADE_VALUE_NULL_FIRST = new Comparator<Trade>(){
  public int compare(Trade o1, Trade o2) {
    return ComparatorUtil.DECIMAL_NULL_FIRST_COMPARATOR.compare(
        o1.getTradeValue(), 
        o2.getTradeValue());
}};

... ComparatorUtil ...

public static NullFirstComparator<BigDecimal> DECIMAL_NULL_FIRST_COMPARATOR = 
  new NullFirstComparator<BigDecimal>();
public static NullLastComparator<BigDecimal> DECIMAL_NULL_LAST_COMPARATOR = 
  new NullLastComparator<BigDecimal>();
...snip...
public static NullLastComparator<String> STRING_NULL_LAST_COMPARATOR = 
  new NullLastComparator<String>();

public static class NullFirstComparator<T extends Comparable<T>> implements Comparator<T> {
  public int compare(T o1, T o2) {
    if (o1 == null && o2 == null) {
	return 0;
    } else if (o1 == null) {
	return -1;
    } else if (o2 == null) {
	return 1;
    } else {
	return o1.compareTo(o2);
    }
  }
}
public static class NullLastComparator<T extends Comparable<T>> implements Comparator<T> {
  public int compare(T o1, T o2) {
    if (o1 == null && o2 == null) {
	return 0;
    } else if (o1 == null) {
	return 1;
    } else if (o2 == null) {
	return -1;
    } else {
	return o1.compareTo(o2);
    }
  }
}

Now we have a simple, reusable solution we can use with any class and any nullable value in JDK sorting. Now we expose any ordering constants appropriate for business usage in our class. Now let’s deal with the more complex issue of hierarchical value ordering. We don’t want to write new code everytime we have to do something like this. So let’s just extend our idea of ordering helpers to hiearchical entities.

public interface Parent<C> {
  public List<C> getChildren();
}
public class ParentChildPropertiesComparator<P extends Parent<C>, C> implements Comparator<P> {
  private List<Comparator<C>> mChildComparators;
  public ParentChildPropertiesComparator(List<Comparator<C>> childComparators) {
    mChildComparators = Collections.unmodifiableList(childComparators);
  }
  public List<Comparator<C>> getChildComparators() {
    return mChildComparators;
  }
  public int compare(P o1, P o2) {
    int compareTo = 0;
    for (int i=0; i < mChildComparators.size() && compareTo == 0; i++) {
	Comparator<C> cc = mChildComparators.get(i);
	List<C> children1 = o1.getChildren();
	List<C> children2 = o2.getChildren();
	Collections.sort(children1, cc);
	Collections.sort(children2, cc);
	int max = Math.min(children1.size(), children2.size());
	for (int j=0; j < max && compareTo == 0; j++) {
	  compareTo = cc.compare(children1.get(j), children2.get(j));
	}
    }
    return compareTo;
  }
}

This is a little more complex, but still simple enough to easily grasp and reuse. We have the idea of a parent. This is not an OO relationship. This is a relationship of composition or aggregation. A parent can exist anywhere in the hierarchy, meaning a parent can also be a child. But in our sample, we have a simple parent/child relationship - Account/Trade. This new class, ParentChildPropertiesComparator supports the idea of taking in a List of ordered Comparators on the children entities but sorting on the parents. In our scenario, we are only sorting on one child value, but it could easily be several just as you could sort more than 2 levels of data.

We are assuming in our case that Account already implements the Parent interface for accounts. If not, you can always use the Adapter Design Pattern. Our Account/Trade sorting would now look like this.

List<Account> accounts = fetchPreviousMonthsTradeActivityByAccount();
List<Comparator<Trade>> comparators = Arrays.asList(Trade.TRADE_VALUE_NULL_FIRST);
ParentChildPropertiesComparator<Account, Trade> childComparator = 
  new ParentChildPropertiesComparator<Account, Trade>(comparators);
Collections.sort(accounts, childComparator);

Really! That's it. Our annoying problem of sorting accounts by those with highest trade values where some of those trade values could be null is solved in just a few lines of code. Our accounts are now sorted as desired. I came up with this approach and it is used successfully as a part of a query builder for a large-volume financial reconciliation system. Introduction of new sortable types and values requires only minimal additions. Take this approach for a whirl and see how incredibly powerful it is for dealing with sorting requirements across complex hierarchies of data. And drop us a line if you need help in implementation or have any comments.

Computing Common and Unique Elements In Multiple Collections – Java

This week, we’ll take a break from higher level problems and technology posts to deal with just a little code problem that a lot of us have probably faced. It’s nothing fancy or too hard, but it may save one of you 15 minutes someday, and occasionally it’s nice to get back to basics.

So let’s get down to it. On occasion, you’ll find you need to determine which elements in one collection exist in another, which are common, and/or which don’t exist in another collection. Apache Commons Collections has some some utility methods in CollectionUtils that are useful, notably intersection(), but this post goes a bit beyond that into calculating unique elements in collection of collections, and it’s always nice to get down to the details. We’ll also make the solution more generic by supporting any number of collections to operate against, rather than just two collections as CollectionUtils does. Plus there’s the fact that not all of us choose to or are able to include libraries just to get a couple useful utility methods.

When dealing with just two collections, it’s not a difficult problem, but not all developers are familiar with all the methods that java.util.Collection defines, so here is some sample code. They key is using the retainAll and removeAll methods together to build up the three sets – common, present in collection A only, and present in B only.

Set<String> a = new HashSet<String>();
a.add("a");
a.add("a2");
a.add("common");

Set<String> b = new HashSet<String>();
b.add("b");
b.add("b2");
b.add("common");

Set<String> inAOnly = new HashSet<String>(a);
inAOnly.removeAll(b);
System.out.println("A Only: " + inAOnly );

Set<String> inBOnly = new HashSet<String>(b);
inBOnly .removeAll(a);
System.out.println("B Only: " + inBOnly );

Set<String> common = new HashSet<String>(a);
common.retainAll(b);
System.out.println("Common: " + common);

Output:

A Only: [a, a2]
B Only: [b, b2]
Common: [common1]

Handling Three or More Collections

The problem is a bit more tricky when dealing with more than two collections, but it can be solved in a generic way fairly simply, as shown below:

Computing Common Elements
Computing common elements is easy, and this code will perform consistently even with a large number of collections.

   
public static void main(String[] args) {
   List<String> a = Arrays.asList("a", "b", "c");
   List<String> b = Arrays.asList("a", "b", "c", "d");   
   List<String> c = Arrays.asList("d", "e", "f", "g");
    
   List<List<String>> lists = new ArrayList<List<String>>();
   lists.add(a);
   System.out.println("Common in A: " + getCommonElements(lists));
   
   lists.add(b);
   System.out.println("Common in A & B: " + getCommonElements(lists));
   
   lists.add(c);
   System.out.println("Common in A & B & C: " + getCommonElements(lists));
   
   lists.remove(a);
   System.out.println("Common in B & C: " + getCommonElements(lists));
}
    
    
public static <T> Set<T> getCommonElements(Collection<? extends Collection<T>> collections) {

    Set<T> common = new LinkedHashSet<T>();
    if (!collections.isEmpty()) {
       Iterator<? extends Collection<T>> iterator = collections.iterator();
       common.addAll(iterator.next());
       while (iterator.hasNext()) {
          common.retainAll(iterator.next());
       }
    }
    return common;
}

Output:

Common in A: [a, b, c]
Common in A & B: [a, b, c]
Common in A & B & C: []
Common in B & C: [d]

Computing Unique Elements
Computing unique elements is just about as straightforward as computing common elements. Note that this code’s performance will degrade as you add a large number of collections, though in most practical cases it won’t matter. I presume there are ways this could be optimized, but since I haven’t had the problem, I’ve not bothered tryin. As Knuth famously said, “We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil”.

   
public static void main(String[] args) {
   List<String> a = Arrays.asList("a", "b", "c");
   List<String> b = Arrays.asList("a", "b", "c", "d");   
   List<String> c = Arrays.asList("d", "e", "f", "g");
    
   List<List<String>> lists = new ArrayList<List<String>>();
   lists.add(a);
   System.out.println("Unique in A: " + getUniqueElements(lists));
   
   lists.add(b);
   System.out.println("Unique in A & B: " + getUniqueElements(lists));
   
   lists.add(c);
   System.out.println("Unique in A & B & C: " + getUniqueElements(lists));
   
   lists.remove(a);
   System.out.println("Unique in B & C: " + getUniqueElements(lists));
}
    
    
public static <T> List<Set<T>> getUniqueElements(Collection<? extends Collection<T>> collections) {
    
    List<Set<T>> allUniqueSets = new ArrayList<Set<T>>();
    for (Collection<T> collection : collections) {
       Set<T> unique = new LinkedHashSet<T>(collection);
       allUniqueSets.add(unique);
       for (Collection<T> otherCollection : collections) {
          if (collection != otherCollection) {
             unique.removeAll(otherCollection);
          }
       }
   }
       
    return allUniqueSets;
}

Output:

Unique in A: [[a, b, c]]
Unique in A & B: [[], [d]]
Unique in A & B & C: [[], [], [e, f, g]]
Unique in B & C: [[a, b, c], [e, f, g]]

That’s all there is to it. Feel free to use this code for whatever you like, and if you have any improvements or additions to suggest, leave a comment. Developers all benefit when we share knowledge and experience.

Files and Directories in the JDK

In Java, java.io.File is one of the more frequently used low-level API objects. It also happens to be lacking in some basic functionality that we’ve all needed at some point, doesn’t provide different representations/API for files and directories and doesn’t throw fine-grained exceptions to differentiate between different types of error conditions (i.e., file already exists, directory not empty, invalid path, etc.)

At Carfey, we’ve been using our own File and Directory classes for years and have been enhancing their utility over time by adding new functionality as the need arises – File.moveToFile(File newFile), File.writeFromStream(InputStream is), Directory.listFilesRecursively(), Directory.emptyDirectory, etc.

Probably the only real pain you’d encounter using an alternate representation of java.io.File would be in interacting with other libraries that would need access to java.io.File objects. That is easily accommodated by exposing a getJavaFile() method on each class.

Free for Use
Attached you can find our classes and test code available under the MIT licence. It does depend on our open-source Date library which we spoke about here. If you’d rather skip the extra library, you can drop the getLastModifiedDate method from the File class.

Here are some of the highlights.

First, we use different classes to represent File and Directory. Rather than having to invoke isFile() and isDirectory(), we have the type to guide us. Our Directory class has some public constants, such as PATH_SEPARATOR which is equivalent to java.io.File.pathSeparator but uses standard constant naming convention. Also TEMP_DIR, which provides easy access to the temporary directory loaded using the System property java.io.tmpdir. We get typed Exceptions such as DirectoryNotCreatedException, InvalidDirectoryException, DirectoryNotDeletedException with descriptive messages when attempting to construct a Directory reference or physically create/delete a Directory but encountering problems. We can listFiles() or even listFilesRecursively() on a Directory. Want to completely empty a directory of files and subdirectories? Use emptyDirectory().

Our File class similarly gives us typed Exceptions (InvalidFileException, FileIOException, FileNotDeletedException, FileNotDeletedException) with descriptive messages on construction, create or delete of physical Files but encountering problems. We have very convenient move and write methods: moveToFile(File moveFile), writeFromStream(InputStream is), write(String contents). More convenience methods for getting a FileInputStream or FileOutputStream for the given File. Even a getDirectory() that will return one of our Directory objects representing the containing Directory.

Also in the bundle is our own IOUtil class that provides some necessary functions for our File and Directory classes. You happen to get some of its own helpful functionality including

public static final byte[] getBytes(InputStream is) 

public static long copyStream(InputStream src, OutputStream dest, int bufferSize, boolean flushEachRead, StreamListener... listeners)

public static InputStream streamFromReader(Reader reader)

public static String streamAsString(InputStream is, String encoding)

public static <T extends Serializable> T cloneThroughSerialize(T t)

public static Object deserialize(InputStream is)

public static void serializeToOutputStream(Serializable ser, OutputStream os)

If you’re an Eclipse user, this brings up an interesting issue with resolving imports. If you add Carfey File and Directory to your project, you might want to avoid the annoyance of having to select the type of File you want to use. Eclipse allows you to customize how imports are done. If you’ve never done this before, check it out. You can use with other common names such as List, Util, StringUtil, etc.

Want to avoid this?

Go here

Add the restrictions

Now when you “Organize Imports” with Ctrl+Shift+O, you won’t be prompted to choose which File you want.

Happy coding and if you find any bugs or have questions, leave a comment here.

Easy Deep Cloning of Serializable and Non-Serializable Objects in Java

Frequently developers rely on 3d party libraries to avoid reinventing the wheel, particularly in the Java world, with projects like Apache and Spring so prevalent. When dealing with these frameworks, we often have little or no control of the behaviour of their classes.
This can sometimes lead to problems. For instance, if you want to deep clone an object that doesn’t provide a suitable clone method, what are your options, short of writing a bunch of code?

Clone through Serialization
The simplest approach is to clone by taking advantage of an object being Serializable. Apache Commons provides a method to do this, but for completeness, code to do it yourself is below also.

@SuppressWarnings("unchecked")
public static  T cloneThroughSerialize(T t) throws Exception {
   ByteArrayOutputStream bos = new ByteArrayOutputStream();
   serializeToOutputStream(t, bos);
   byte[] bytes = bos.toByteArray();
   ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(bytes));
   return (T)ois.readObject();
}

private static void serializeToOutputStream(Serializable ser, OutputStream os) 
                                                          throws IOException {
   ObjectOutputStream oos = null;
   try {
      oos = new ObjectOutputStream(os);
      oos.writeObject(ser);
      oos.flush();
   } finally {
      oos.close();
   }
}

// using our custom method
Object cloned = cloneThroughSerialize (someObject);

// or with Apache Commons
cloned = org.apache.commons.lang. SerializationUtils.clone(someObject);

But what if the class we want to clone isn’t Serializable and we have no control over the source code or can’t make it Serializable?

Option 1 – Java Deep Cloning Library
There’s a nice little library which can deep clone virtually any Java Object – cloning. It takes advantage of Java’s excellent reflection capabilities to provide optimized deep-cloned versions of objects.

Cloner cloner=new Cloner();
Object cloned = cloner.deepClone(someObject);

As you can see, it’s very simple and effective, and requires minimal code. It has some more advanced abilities beyond this simple example, which you can check out here.

Option 2 – JSON Cloning
What about if we are not able to introduce a new library to our codebase? Some of us deal with approval processes to introduce new libraries, and it may not be worth it for a simple use case.

Well, as long as we have some way to serialize and restore an object, we can make a deep copy. JSON is commonly used, so it’s a good candidate,since most of us use one JSON library or another.

Most JSON libraries in Java have the ability to effectively serialize any POJO without any configuration or mapping required. This means that if you have a JSON library and cannot or will not introduce more libraries to provide deep cloning, you can leverage an existing JSON library to get the same effect. Note this method will be slower than others, but for the vast majority of applications, this won’t cause any performance problems.

Below is an example using the GSON library.

@SuppressWarnings("unchecked")
public static  T cloneThroughJson(T t) {
   Gson gson = new Gson();
   String json = gson.toJson(t);
   return (T) gson.fromJson(json, t.getClass());
}
// ...
Object cloned = cloneThroughJson(someObject);

Note that this is likely only to work if the copied object has a default no-argument constructor. In the case of GSON, you can use an instance creator to get around this. Other frameworks have similar concepts, so you can use that if you hit an issue with an unmodifiable class having not having the default constructor.

Conclusion
One thing I do recommend is that for any classes you need to clone, you should add some unit tests to ensure everything behaves as expected. This can prevent changes to the classes (e.g. upgrading library versions) from breaking your application without your knowledge, especially if you have a continuous integration environment set up.

I’ve outlined a couple methods to clone an object outside of normal cases without any custom code. If you’ve used any other methods to get the same result, please share.

Ignoring Self-Signed Certificates in Java

A problem that I’ve hit a few times in my career is that we sometimes want to allow self-signed certificates for development or testing purposes. A quick Google search shows the trouble that countless Java developers have run into over the years. Depending on the exact certificate issue, you may get an error like one of the following, though I’m almost positive there are other manifestations:

java.security.cert.CertificateException: Untrusted Server Certificate Chain


javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Getting around this often requires modifying JDK trust store files which can be painful and often you end up running into trouble. On top of that, every developer on your team will have to do the same thing, and every new environment you’ll have the same issues recurring.

Fortunately, there is a way to deal with the problem in a generic way that won’t put any burden on your developers. We’re going to focus on vanilla HttpURLConnection type connections since it is the most general and should still help you understand the direction to take with other libraries. If you are using Apache HttpClient, see here.

Warning: Know what you are doing!

Be aware of what using this code means: it means you don’t care at all about host verification and are using SSL just to encrypt communications. You are not preventing man-in-the-middle attacks or verifying you are connected to the host you think you are. This generally comes down to a few valid cases:

  1. You are operating in a locked down LAN environment. You are not susceptible to having your requests intercepted by an attacker (or if you are, you have bigger issues).
  2. You are in a test or development environment where securing communication isn’t important.

If this matches your needs, then go ahead and proceed. Otherwise, maybe think twice about what you are trying to accomplish.

Solution: Modifying Trust Managers

Now that we’re past that disclaimer, we can solve the actual problem at hand. Java allows us to control the objects responsible for verifying a host and certificate for a HttpsURLConnection. This can be done globally but I’m sure those of you with experience will cringe at the thought of making such a sweeping change. Luckily we can also do it on a per-request basis, and since examples of this are hard to find on the web, I’ve provided the code below. This approach is nice since you don’t need to mess with swapping out SSLSocketFactory implementations globally.

Feel free to grab it and use it in your project.

package com.mycompany.http;

import java.net.*;
import javax.net.ssl.*;
import java.security.*;
import java.security.cert.*;

public class TrustModifier {
   private static final TrustingHostnameVerifier 
      TRUSTING_HOSTNAME_VERIFIER = new TrustingHostnameVerifier();
   private static SSLSocketFactory factory;

   /** Call this with any HttpURLConnection, and it will 
    modify the trust settings if it is an HTTPS connection. */
   public static void relaxHostChecking(HttpURLConnection conn) 
       throws KeyManagementException, NoSuchAlgorithmException, KeyStoreException {

      if (conn instanceof HttpsURLConnection) {
         HttpsURLConnection httpsConnection = (HttpsURLConnection) conn;
         SSLSocketFactory factory = prepFactory(httpsConnection);
         httpsConnection.setSSLSocketFactory(factory);
         httpsConnection.setHostnameVerifier(TRUSTING_HOSTNAME_VERIFIER);
      }
   }

   static synchronized SSLSocketFactory 
            prepFactory(HttpsURLConnection httpsConnection) 
            throws NoSuchAlgorithmException, KeyStoreException, KeyManagementException {

      if (factory == null) {
         SSLContext ctx = SSLContext.getInstance("TLS");
         ctx.init(null, new TrustManager[]{ new AlwaysTrustManager() }, null);
         factory = ctx.getSocketFactory();
      }
      return factory;
   }
   
   private static final class TrustingHostnameVerifier implements HostnameVerifier {
      public boolean verify(String hostname, SSLSession session) {
         return true;
      }
   }

   private static class AlwaysTrustManager implements X509TrustManager {
      public void checkClientTrusted(X509Certificate[] arg0, String arg1) throws CertificateException { }
      public void checkServerTrusted(X509Certificate[] arg0, String arg1) throws CertificateException { }
      public X509Certificate[] getAcceptedIssuers() { return null; }      
   }
   
}

Usage

To use the above code, just call the relaxHostChecking() method before you open the stream:

URL someUrl = ... // may be HTTPS or HTTP
HttpURLConnection connection = (HttpURLConnection) someUrl.openConnection();
TrustModifier.relaxHostChecking(connection); // here's where the magic happens

// Now do your work! 
// This connection will now live happily with expired or self-signed certificates
connection.setDoOutput(true);
OutputStream out = connection.getOutputStream();
...

There you have it, a complete example of a localized approach to supporting self-signed certificates. This does not affect the rest of your application which will continue to have strict hosting checking semantics. This example could be extended to use a configuration setting to determine whether relaxed host checking should be used, and I recommend you do so if using this code is primarily a way to facilitate development with self-signed certificates.

If you have any questions about the example or have questions about doing this in a specific HTTP library, leave a post and I’ll do my best to help.

Java 7 – Project Coin Feature Overview

We discussed previously everything that didn’t make it into Java 7 and then reviewed the useful Fork/Join Framework that did make it in. Today’s post will take us through each of the Project Coin features – a collection of small language enhancements that aren’t groundbreaking, but are nonetheless useful for any developer able to use JDK 7.

I’ve come up with a bank account class that showcases the basics of Project Coin features. Take a look…

public class ProjectCoinBanker {
  
  private static final Integer ONE_MILLION = 1_000_000;
  private static final String RICH_MSG = "You need more than $%,d to be considered rich.";

  public static void main(String[] args) throws Exception {
	System.out.println(String.format(RICH_MSG, ONE_MILLION));

	String requestType = args[0];
	String accountId = args[1];
	switch (requestType) {
		case "displayBalance": 
			printBalance(accountId); 
			break;
		case "lastActivityDate" : 
			printLastActivityDate(accountId); 
			break;
		case "amIRich" : 
			amIRich(accountId); 
			break;
		case "lastTransactions" : 
			printLastTransactions(accountId, Integer.parseInt(args[2])); 
			break;
		case "averageDailyBalance" : 
			printAverageDailyBalance(accountId); 
			break;
		default: break;
	}
  }
  
  private static void printAverageDailyBalance(String accountId) {
        String sql = String.format(AVERAGE_DAILY_BALANCE_QUERY, accountId);
	try (
	      PreparedStatement s = _conn.prepareStatement(sql);
              ResultSet rs = s.executeQuery();
             ) {
	        while (rs.next()) {
		  //print the average daily balance results
                }
	     } catch (SQLException e) {
		// handle exception, but no need for finally to close resources
                for (Throwable t : e.getSuppressed()) {
		   System.out.println("Suppressed exception message is " + t.getMessage());
		}
	     }
  }
  
  private static void printLastTransactions(String accountId, int numberOfTransactions) {
	List<Transaction> transactions = new ArrayList<>();
	... handle fetching/printing transactions
  }
  
  private static void printBalance(String accountId) {
	try {
		BigDecimal balance = getBalance(accountId);
		//print balance
	} catch (AccountFrozenException | ComplianceViolationException | AccountClosedException e) {
	    System.err.println("Please see your local branch for help with your account.");
	}
  }
  
  private static void amIRich(String accountId) {
	try {
		BigDecimal balance = getBalance(accountId);
		//find out if the account holder is rich
	} catch (AccountFrozenException | ComplianceViolationException | AccountClosedException e) {
	    System.out.println("Please see your local branch for help with your account.");
	}
  }
  
  private static BigDecimal getBalance(String acccountId) 
      throws AccountFrozenException, AccountClosedException, ComplianceViolationException {
      ... getBalance functionality
  }
 
}

Briefly, our ProjectCoinBanker class demonstrates basic usage of the following Project Coin features.

  • Underscores in numeric literals
  • Strings in switch
  • Multi-catch
  • Type inference for typed object creation
  • try with resources and suppressed exceptions

First of all, underscores in numeric literals are pretty self-explanatory. Our example, private static final Integer ONE_MILLION = 1_000_000; shows that the benefit is visual. Developers can quickly look at the code to verify that values are as expected. Underscores can be used in places other than natural groupings locations, being ignored wherever they are placed. Underscores in numeric literals cannot begin or terminate a numeric literal, otherwise, you’re free to place them where you’d like. While not demonstrated here, binary literal support has also been added. In the same way that hex literals are prefixed by 0x or 0X, binary literals would be prefixed by 0b or 0B.

Strings in switch are also self-explanatory. The switch statement now also accepts String. In our example, we switch on String argument passed to the main method to determine what request was made. On a side note, this is purely a compiler implementation with an indication that JVM support for switching on String may be added at a later date.

Type inference is another easy-to-understand improvement. Now instead of our old code List<Transaction> transactions = new ArrayList<Transaction>(); we can simply do List<Transaction> transactions = new ArrayList<>(); since the type can be inferred. Probably won’t find anyone who would argue that it shouldn’t have been so since the introduction of generics, but it’s now here.

Multi-catch will turn out to be very nice for the conciseness of exception handling code. Too many times when wanting to actually do something based on the exception type thrown, until now we were forced to have multiple catch blocks all doing essentially the same thing. The new syntax is very clean and logical. Our example, catch (AccountFrozenException | ComplianceViolationException | AccountClosedException e) shows how easily it can be done.

Finally, the last demonstration of a Project Coin feature is the try with resources syntax and support for retrieving suppressed exceptions. A new interface has been introduced, AutoCloseable that has been applied to all the expected suspects including Input/OutputStreams, Readers/Writers, Channels, Sockets, Selectors and java.sql resources Statement, ResultSet and Connection. In my opinion, the syntax is not as natural as the multi-catch change was, not that I have an alternative in mind.

    try (
        PreparedStatement s = _conn.prepareStatement(sql);
        ResultSet rs = s.executeQuery();
        ) {
	    while (rs.next()) {
	      //print the average daily balance results
            }
	} catch (SQLException e) {
	    //handle exception, but no need for finally to close resources
            for (Throwable t : e.getSuppressed()) {
	        System.out.println("Suppressed exception message is " + t.getMessage());
	    }
	}

First we see that we can include multiple resources in try with resources – very nice. We can even reference previously declared resources in the same block as we did with our PreparedStatement. We still handle our exception, but we don’t need to have a finally block just to close the resources. Notice too that there is a new method getSuppressed() on Throwable. This allows us to access any Exceptions that were thrown in trying to “autoclose” the declared resources. There can be at most one suppressed exception per resource declared. Note: if the resource initialization throws an exception, it would be handled in your declared catch block.

That’s it. Nothing earth-shattering but some simple enhancements that we can all begin using without too much trouble. Project Coin also includes a feature regarding varargs and compiler warnings. Essentially, it boils down to a new annotation (@SafeVarargs) that can be applied at the method declaration to allow developers to remove @SuppressWarnings("varargs") from their consuming code. This has been applied on all the key suspects within the JDK, but the same annotation is available to you in any of your genericized varags methods.

The Project Coin feature set as it is described online is inconsistent at best. Hopefully this will give you a solid summary of what you can use from the proposal in JDK 7.

Testing GWT Apps with Selenium or WebDriver

Good functional testing is one of the most difficult tasks for web application developers and their teams. It is a challenge to develop tests that are cheap to maintain and yet provide good test coverage, which helps reduce QA costs and increase quality.

Both Selenium and WebDriver (which is essentially now the successor to Selenium) provide a good way to functionally test web applications in multiple target environments without manual work. In the past, web UIs were built using the page navigation to allow users to submit forms, etc. These days, more and more web applications use Ajax and therefore act and look a lot more like desktop applications. However, this poses problems for testing – Selenium and WebDriver are designed to work with user interations resulting in page navigation and don’t play well with AJAX apps out of the box.

GWT-based applications in particular have this problem, but there are some ways I’ve found to develop useful and effective tests. GWT also poses other issues in regards to simulating user input and locating DOM elements, and I discuss those below. Note that my code examples use Groovy to make them concise, but they can be pretty easily converted to Java code.

Problem 1: Handling Asynchronous Changes

One issue that developers face pretty quickly when testing applications based on GWT is detecting and waiting for a response to user interaction. For example, a user may click a button which results in an AJAX call which would either succeed and close a window or, alternatively, show an error message. What we need is a way to block until we see the expected changes, with a timeout so we can fail if we don’t see the expected changes.

Solution: Use WebDriverWait

The easiest way to do this is by taking advantage of the WebDriverWait (or Selenium’s Wait). This allows you to wait on a condition and proceed when it evaluates to true. Below I use Groovy code for the conciseness of using closures, but the same can be done in Java, though with a bit more code due to the need for anonymous classes.

def waitForCondition(Closure closure) { 
    int timeout = 20
    WebDriverWait w = new WebDriverWait(driver, timeout) 
    w.until({
        closure() // wait until this closure evaluates to true
    } as ExpectedCondition)
}
	   
def waitForElement(By finder) {
    waitForCondition {
        driver.findElements(finder).size() > 0;
    }
}

def waitForElementRemoval(By finder) {
    waitForCondition {
        driver.findElements(finder).size() == 0;
    }
}

// now some sample test code 

submitButton.click() // submit a form

// wait for the expected error summary to show up
waitForElement(By.xpath("//div[@class='error-summary']"))
// maybe some more verification here to check the expected errors

// ... correct error and resubmit

submitButton.click() 
waitForElementRemoval(By.xpath("//div[@class='error-summary']"))
waitForElementRemoval(By.id("windowId"))

As you can see from the example, your code can focus on the actual test logic while handling the asynchronous nature of GWT applications seamlessly.

Problem 2: Locating Elements when you have little control over DOM

In web applications that use templating (JSPs, Velocity, JSF, etc.), you have good control and easy visibility into the DOM structure that your pages will have. With GWT, this isn’t always the case. Often, you’re dealing with nested elements that you can’t control at a fine level.

With WebDriver and Selenium, you can target elements using a few methods, but the most useful are by DOM element ID and XPath. How can we leverage these to get maintainable tests that don’t break with minor layout changes?

Solution: Use XPath combined with IDs to limit scope

In my experience, to develop functional GWT tests in WebDriver, you should use somewhat loose XPath as your primary means of locating elements, and supplement it by scoping these calls by DOM ID, where applicable.

In particular, use IDs at top level elements like windows or tabs that are unique in your application and won’t exist more than once in a page. These can help scope your XPath expressions, which can look for window or form titles, field labels, etc.

Here are some examples to get you going. Note that we use // and * in our XPath to keep our expressions flexible so that layout changes do not break our tests unless they are major.

By byUserName = By.xpath("//*[@id='userTab']//*[text()='User Name']/..//input")
WebElement userNameField = webDriver.findElement(byUserName)
userNameField.sendKeys("my new user")

// maybe a user click and then wait for the window to disappear
By submitLocator = By.xpath("//*[@id='userTab']//input[@type='submit']")
WebElement submit = webDriver.findElement(submitLocator)
submit.click()

// use our helper method from Problem 1
waitForElementRemoval By.id("userTab")

Problem 3: Normal element interaction methods don’t work!

GWT and derivatives (Vaadin, GXT, etc.) often are doing some magic behind the scenes as far as managing the state of the DOM goes. To the developer, this means you’re not always dealing with plain <input> or <select>, etc. elements. Simply setting the value of the field through normal means may not work, and using WebDriver or Selenium’s click methods may not work.

WebDriver has improved in this regard, but issues still persist.

Solution: Unfortunately, just some workarounds

The main problems you’re likely to encounter relate to typing into fields and clicking elements.

Here are some variants that I have found necessary in the past to get around clicks not working as expected. Try them if you are hitting issues. The examples are in Selenium, but they can be adapted to the corresponding calls in WebDriver if you require them. You may also use the Selenium adapter for WebDriver (WebDriverBackedSelenium) if you want to use the examples directly.

Click Issues

Sometimes elements won’t respond to a click() call in Selenium or WebDriver. In these cases, you usually have to simulate events in the browser. This was true more of Selenium before 2.0 than WebDriver.

// Selenium's click sometimes has to be simulated with events.
def fullMouseClick(String locator) {
    selenium.mouseOver locator
    selenium.mouseDown locator
    selenium.mouseUp locator
}

// In some cases you need only mouseDown, as mouseUp may be
// handled the same as mouseDown.
// For example, this could result in a table row being selected, then deselected.
def mouseOverAndDown(String locator) {
    selenium.mouseOver locator
    selenium.mouseDown locator
}

Typing Issues

These are the roundabout methods of typing I have been able to use successfully in the past when GWT doesn’t recognize typed input.

	
// fires only key events (works for most GWT inputs)
// Useful if WebDriver sendKeys() or Selenium type() aren't cooperating.
def typeWithEvents(String locator, String text) {
    def keyEvents = ["keydown", "keypress", "keyup"]
    typeWithEvents(locator, text, keyEvents)
}

// fires key events, plus blur and focus for really picky cases
def typeWithFullEvents(String locator, String text) { 
    def fullEvents = ["keydown", "keypress", "keyup", "blur", "focus"]
    typeWithEvents(locator, text, fullEvents)
}


// use this directly to customize which events are fired
def typeWithEvents(String locator, String text, def events) {
    text.eachWithIndex { ch, i ->
        selenium.type locator, text.substring(0, i+1)
        events.each{ event ->
            selenium.fireEvent locator, event
        }
    }
}

Note that the exact method that works will have to be figured out by trial-and-error and in some cases, you may get different behaviour in different browsers, so if you run your functional tests against different environments, you’ll have to ensure your method works for all of them.

Conclusion

Hopefully some of you find these tips useful. There are similar tips out there but I wanted to compile a good set of examples and workarounds so that others in similar situations don’t hit dead-ends or waste time on problems that require lots of guessing and time.

If you have any other useful tips or workarounds, please share by leaving a comment. Maybe you’ll save someone having to work late or on a weekend!

Swapping out Spring Bean Configuration at Runtime

Most Java developers these days deal with Spring on a regular basis and there are lots of us out there that have become familiar with its abilities as well as its limitations.

I recently came across a problem that I hadn’t hit before: introducing the ability to rewire a bean’s internals based on configuration introduced at runtime. This is valuable for simple configuration changes or perhaps swapping out something like a Strategy or Factory class, rather than rebuilding of a complex part of the application context.

I was able to find some notes about how to do this, but I thought that some might find my notes and code samples useful, especially since I can confirm this technique works on versions of Spring back to 1.2.6. Unfortunately, not all of us are lucky enough to be on the latest and greatest of every library.

Scope of the Problem

The approach I’m going to outline is meant primarily to target changes to a single bean, though this code could easily be extended to change multiple beans. It could be invoked through JMX or some other UI exposed to administrators.

One thing it does not cover is rewiring a singleton all across an application – this could conceivably be done via some reflection and inspection of the current application context, but is likely to be unsafe in most applications unless they have some way of temporarily shutting down or blocking all processing for a period while the changes are made all over the application.

The Code

Here’s the sample code. It will take a list of Strings which contains bean definitions, and wire them into a new temporary Spring context. You’ll see a parent context can be provided, which is useful in case your new bean definitions need to refer to beans already configured in the application.

public static <T> Map<String, T> extractBeans(Class<T> beanType, 
   List<String> contextXmls, ApplicationContext parentContext) throws Exception {

   List<String> paths = new ArrayList<String>();
   try {
      for (String xml : contextXmls) {
         File file = File.createTempFile("spring", "xml");
         // ... write the file using a utility method
         FileUtils.writeStringToFile(file, xml, "UTF-8");
         paths.add(file.getAbsolutePath());
      }

      String[] pathArray = paths.toArray(new String[0]);
      return buildContextAndGetBeans(beanType, pathArray, parentContext);

   } finally {
      // ... clean up temp files immediately if desired
   }
}

private static <T> Map<String, T> buildContextAndGetBeans(Class<T> beanType, 
               String[] paths, ApplicationContext parentContext) throws Exception {

   FileSystemXmlApplicationContext context = 
      new FileSystemXmlApplicationContext(paths, false, parentContext) {
         @Override  // suppress refresh events bubbling to parent context
         public void publishEvent(ApplicationEvent event) { }

         @Override 
         protected Resource getResourceByPath(String path) {
            return new FileSystemResource(path); // support absolute paths
         }
      };

   try {
      // avoid classloader errors in some environments      
      context.setClassLoader(beanType.getClassLoader());
      context.refresh(); // parse and load context
      Map<String, T> beanMap = context.getBeansOfType(beanType);

      return beanMap;
   } finally {
      try {
         context.close();
      } catch (Exception e) {
         // ... log this
      }
   }
}

If you look at buildContextAndGetBeans(), you’ll see it does the bulk of the work by building up a Spring context with the supplied XML bean definition files. It then returns a map of the constructed beans of the type requested.

Note: Since the temporary Spring context is destroyed, ensure your beans do not have lifecycle methods that cause them to be put into an invalid state when stopped or destroyed.

Here’s an example of a Spring context that might be used to rewire a component. Imagine we have an e-commerce system that does fraud checks, but various strategies for checking for fraud. We may wish to swap these from our service class without having to stop and reconfigure the application, since we lose business when we do so. Perhaps we are finding a specific abuse of the system that would be better dealt with by changing the strategy used to locate fraudulent orders.

Here’s a sample XML definition that could be used to rewire our FraudService.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
   <bean id="fraudStrategy" class="com.example.SomeFraudStategory">
      <!-- example of a bean defined in the parent application context that we can reference -->
      <property name="fraudRuleFactory" ref="fraudRuleFactory"/>
   </bean>
</beans>

And here is the code you could use to rewire your bean with a reference to the defined fraudStrategy, assuming you have it in a utility class called SpringUtils:

public class FraudService implements ApplicationContextAware {

   private ApplicationContext context;
   // volatile for thread safety (in Java 1.5 and up only)
   private volatile FraudStrategy fraudStrategy;
   
   @Override // get a handle on the the parent context 
   public void setApplicationContext(ApplicationContext context) {
      this.context = context;
   }

   public void swapFraudStategy(String xmlDefinition) throws Exception {
      List<Sting> definitions = Arrays.asList(xmlDefinition);
      Map<String, FraudStrategy> beans = 
         SpringUtils.extractBeans(FraudStrategy.class, definitions, context);
      if (beans.size() != 1) {
         throw new RuntimeException("Invalid number of beans: " + beans .size());
      }
      this.fraudStrategy = beans.values().iterator().next();
   }
   
}

And there you have it! This example could be extended a fair bit to meet your needs, but I think it shows the fundamentals of how to create a Spring context on the fly, and use its beans to reconfigure your application without any need for downtime.

Evolving Document Structures with Morphia and MongoDB

In my previous post on Morphia, I went through some typical usages and mentioned some caveats and workarounds for known problems. I showed how easy it is to work with Morphia and how cleanly it interacts with the Java world.

To follow up on that post, I’m going to discuss how to deal with some real life needs: handling changing schemas and customizing your mapping to handle things like read-only fields and replacing simple fields with complex objects.

Changing Schemas

As nearly anyone who has worked with databases in the development world knows, schemas are always evolving. Fields get deprecated or outright dropped, tables become obsolete, new fields are added, and so on.

While a lot of this pain is avoided by using a schemaless datastore like MongoDB, sometimes we still do need special handling for changes, and in the case of Morphia, we essentially have defined a schema, so we do have to find ways to deal with this. The nice part about it is that Morphia makes it very clean and easier than you’ll see in just about in any ORM.

Deprecating Fields

One good example is a deprecated field that has been replaced by another field. Let’s imagine you have a bug tracking system with documents that look something like this:

{
  _id:1,
  desc: "IE Rendering broken on intranet site",
  componentName: "INTRANET",
  dateCreated: ISODate("2011-09-06T20:52:50.258Z")
}

Here is the Morphia definition:

@Entity("issues")
class Issue {
  @Id private long id;
  private String desc;
  private String componentName;

  private Date dateCreated = new Date();
}

Now imagine at some point we decide to do away with the component field and make it a more generic free text field where users can enter multiple components, versions, or other helpful information. We don’t want to just stick that in the component field, as that would lead to confusion.

Thankfully, we have a something in the Morphia toolkit that is made exactly for this – The @AlsoLoad annotation. This annotation allows us to populate a POJO field with one of multiple possible sources. We simply update our Morphia mapping to indicate an old field name, and we can easily remove references to the old field without breaking anything. This keeps our code and documents clean.

@Entity("issues")
class Issue {
  @Id private long id;
  private String desc;
  
  @AlsoLoad("componentName") // handle old componentName field
  private String affects;

  private Date dateCreated = new Date();
}

So here we’ve defined automatic translation of our old field without any need to update documents or write special logic within our POJO class to handle documents differently depending on when they were created.

One important note: in this example, if both the affects field and the old componentName field exist, Morphia will throw an exception, so don’t try using this for anything other than deprecating fields, or perhaps populating a single field with two mutually exclusive properties.

Supporting Read-Only for Deprecated Fields

Another possibility is that you just have to support an old field in document that the application no longer writes. This is a very simple one: use the @NotSaved annotation. When you use this on a field, the data will be loaded but not written by Morphia.

In our previous example, we could just as easily have decided to just support display for the old field but not treat populate it into the affects field, so let’s alter our Morphia POJO a bit to show how @NotSaved is used.

@Entity("issues")
class Issue {
  @Id private long id;
  private String desc;
 
  private String affects;
  
  @NotSaved("componentName") // load old componentName field for display only
  private String componentName
  
  private Date dateCreated = new Date();
}

Replacing a Field with An Embedded Object

Now what if our componentName field had actually changed to a complex component object which has a name, version and build number? This is a bit trickier since we want to replace one field with multiple. We can’t attempt to load the field from multiple sources since they have different structures. Of course, we can use an embedded object to store the complex component information, but how can we make our code work seamlessly either way without having to update our documents?

In this case, the simplest approach would be to use a combination of three annotations. First we would mark the old field with the @NotSaved annotation, introduce a new embedded Component object using the @Embedded annotation, and finally take advantage one more annotation that Morphia provides – @PostLoad. This one lets us have a method that is executed after the POJO is populated from MongoDB.

Here’s the example:

@Entity("issues")
class Issue {
  @Id private long id;
  private String desc;
 
  private String affects;
  
  @NotSaved("componentName") // load old componentName to convert to component
  private String componentName
  
  @Embedded // our new complex Component
  private Component component;
  
  private Date dateCreated = new Date();
  // getters and setters ...
  
  @PostLoad
  protected void handleComponent() {
      if (component == null && componentName != null) {
        component = new Component(componentName, null, null);
      }
  }
}

class Component {
  private String componentName;
  private Long version;
  private Long buildNumber;
	
  public Component(String componentName, Long version, Long buildNumber) {
    // ...
  }
  
  // getters and setters ...
}

In this case, we could remove the getter and setter for the componentName field, so that our mapped object only exposes the new and improved interface.

Conclusion

By using the powerful tools that Morphia gives us through its annotation support, we can meet these goals:

  1. Let our document structure adapt with the application and stay clean.
  2. Seamlessly handle changing structure in our Java code without error-prone code.
  3. Expose only the new schema while supporting the old (truly obsolete the old code and fields.

Hopefully this helps a few of you out with adapting to evolving documents, or at least to become more familiar with the abilities some of these Morphia annotations give you.

Java 7 – Fork/Join

While we lamented how feature-poor Java 7 turned out to be, one thing that made it that turns out to be a boon to high-performance concurrent development is the new Fork/Join framework. This framework is targeted at multi-processor systems (really almost all hardware today) in situations where a batch of work can broken into smaller recursive calls. But more than just that, it also uses a work-stealing algorithm where threads with no work can steal available work from other threads that are busy. What makes that so useful is that you can try to break down your work into fairly small, roughly equal pieces, but some pieces can take longer than others and you’ll still get good use of your processing resources.

The Fork/Join framework builds on the ExecutorService we discussed in our Concurrency series and the implementation is ForkJoinPool that can execute the ForkJoinTask which is usually implemented as a child of either RecursiveTask(returns data) or RecursiveAction(no returned data).

As is our custom, let’s use a sample task and some applicable code. Let’s assume you want to write a class that will calculate the size on disk of a given directory and all its children. And in our case we’ll just make the assumption we’re running some wicked fast SSDs so we can actually benefit from concurrent scans.

Here’s our code:

import java.util.concurrent.*;
import java.io.*;
import java.util.*;

public class DirectorySizer extends RecursiveTask<Long> {
  
  private List<File> mFiles;
  private boolean mAllFiles = true;
  
  public DirectorySizer(List<File> files) {
    mFiles = files;
    for (File file : files) {
      if (file.isDirectory()) {
        mAllFiles = false;
      }
    }
  }
  
  protected Long compute() {
    if (mFiles.size() <=4 && mAllFiles) {
      return computeLocal();
    } else {
      return forkAndJoin();
    }  
  }
  
  private Long computeLocal() {
    long length = 0;
    for (File file : mFiles) {
	  length += file.length();
    }
    return length;
  }
  
  private Long forkAndJoin() {
    List<File> dirsAndFiles = new ArrayList();
	for (File file : mFiles) {
      if (file.isFile()) {
        dirsAndFiles.add(file);
      } else {
        dirsAndFiles.addAll(Arrays.asList(file.listFiles()));
      }
    }
    int rightSize = dirsAndFiles.size() / 2;
    int leftSize = dirsAndFiles.size() - rightSize;
    List<File> leftList = dirsAndFiles.subList(0, leftSize);
    List<File> rightList= dirsAndFiles.subList(leftSize, leftSize+rightSize);
    DirectorySizer d1 = new DirectorySizer(leftList);
    d1.fork();
    DirectorySizer d2 = new DirectorySizer(rightList);
    return d2.compute() + d1.join();
  }
  
  public static void main(String[] args) throws Exception {
    List<File> files = Arrays.asList(new File(args[0]).listFiles());
    DirectorySizer sizer = new DirectorySizer(files);
    ForkJoinPool pool = new ForkJoinPool();
    Long size = pool.invoke(sizer);
    System.out.println(args[0] + " is " + size + " bytes ");
  }
}

Let's break down the usage of the Fork/Join framework. By extending RecursiveTask, all we really need to do is implement the compute method calculating the size when the amount of work is small enough to suit our needs and using a fork/join when the chunk of work is still too large. In our main method we get all the benefit of the framework simply by creating a new ForkJoinPool and passing our top-level instance to the invoke method.

While it's true the potential uses of this framework are limited and require a fairly narrow problem scope, it's nice to see continued advancement in Java in the concurrency space to build on all the goodies we got in Java 5. If you have any concurrency problems you'd like us to tackle in our blog, drop us line. We enjoy talking shop with fellow developers.