XMLVM is Actually Pretty Fast

DISCLAIMER: The Benchmark results in this post turned out to be flawed due to GCC/LLVM optimizations unduly favouring XMLVM. Please see the next post where I discuss the corrected benchmark and associated results.

In a previous post, I described some of my experiments in creating an Avian port for CodeNameOne. As mentioned in the post, one of my motivations was to see if I could improve on the performance of the current default iOS port which uses XMLVM to convert the Java code into C code (and ultimately compiled with Xcode). I had a hunch that binaries produced via the XMLVM translation would be slower than an AOT compiled Java implementation because of the way it works (It converts VM byte code operations into C stack operations in the C language – which would presumably produce much less succinct code).

After some tedious conversion of XMLVM runtime library calls to JNI calls, I was able to successfully build my Avian iOS port for CodeNameOne. Next I created a simple application (based on the CodenameOne Tabbed Application template) that runs the Towers of Hanoi problem, and built two versions of the app:

  • One using the CodenameOne build server (which uses XMLVM in its build process)
  • – Another one using my Avian, Proguard, and Xcode.

The Benchmark Results

The results were surprising.

I ran both apps on my iPhone 4s which is on iOS 5. The app is set to solve the tower of Hanoi problem for n=30 moving from pole 1 to pole 3. The time required to complete the problem was pretty consistent. On average, the XMLVM app would complete the problem in 35 seconds, and the Avian app would complete it in 42 seconds.

NOTE: After fixing the benchmarks so that GCC/LLVM *had* to run all of the code, it turned out that Avian was actually a bit faster.

This is quite the *opposite* of what I expected. While I’m a little disappointed in the results, I am also encouraged. This means that the performance of apps developed in CodeNameOne for iOS (using their build server) is actually quite good. I can be confident that I am building on a solid foundation.

I posted these results on both the CodenameOne forum and the Avian forum and a few explanations for the outcome were proposed. One explanation, that makes sense is that the XMLVM build benefits from optimizations of GCC and LLVM such as method inlining, loop unrolling, autovectorization, code motion, and intelligent register allocation. Avian’s AOT compiler, being simpler, and far less mature doesn’t implement any of these optimizations so the resulting binary is actually working at a slight disadvantage.

Joel Dice (creator of Avian) did note that a simple benchmark like Towers of Hanoi is not really helpful for determining real-world performance. A full comparison on real-world tasks would be more informative.

UPDATE: The Benchmark results in this post turned out to be flawed due to GCC/LLVM optimizations unduly favouring XMLVM. Please see the next post where I discuss the corrected benchmark and associated results.

The Executable Size

Another important factor when producing a mobile app, is the actual executable size. Why install a 100MB application when you can get the same app in under a meg. I used Proguard on the Avian build to trim down the code size (so we don’t need to include the entire JRE. XMLVM, also uses a number of optimizations to ensure that dead code isn’t included in the final executable.

My test app came out at 3.5MB for the XMLVM build, and 6.0MB for the Avian build. So XMLVM seems to have won here again.

What Now For the Avian Port

Now that I have established that the current XMLVM implementation of the iOS port is quite fast indeed, there is little need to continue to develop an Avian port. This exercise was academic in nature, and I’m satisfied that I have achieved my goals, which were:

  1. Can it be done? (i.e. Create an Avian port for CodenameOne). The answer was YES!
  2. To learn about the CodenameOne architecture. I learned a lot.
  3. To learn about the Avian architecture. I learned a lot here too.
  4. Find out whether the performance would be dramatically improved by using an AOT java compiler instead of XMLVM -> C -> Xcode. The answer, for now, was NO. XMLVM is pretty fast as it is.

Despite that fact that my experiment didn’t yield any performance improvements, there still might be some advantages to developing apps on the Avian stack. Off the top of my head, these include:

  1. A more familiar Java environment. JNI instead of XMLVM’s runtime when interacting with the native environment.
  2. JDK7 support. I’m not actually sure what version of Java I could use with CodenameOne’s XMLVM implementation. It is at least Java 5, but I don’t think it is at Java 7. I notice that they use Apache Harmony for the iOS port.
  3. Perhaps better debugging and exception handling. Shai Almog noted that XMLVM doesn’t support ClassCastException and doesn’t check cast validity. Its stack traces are also a bit cryptic when you get an exception.

Up Next For Avian/iOS

I am keen to try my next experiment with Avian and iOS: Combining my Java-Objective-C bridge with Avian to produce a Java solution that is similar to MonoTouch. I.e. Where you write the code in Java but still use the Apple development tools for the UI. This wouldn’t be a cross-platform solution like CodenameOne produces, but it would be useful, I think.

Up Next for CodenameOne

Time to develop some real-world apps!


comments powered by Disqus