In my last post, I discussed the results of a benchmark comparison between XVMLVM and Avian both solving the Towers of Hanoi problem on an iPhone 4s. Originally it looked like XMLVM ran slightly faster, solving the problem in 35 seconds (vs 42 seconds for the Avian version).
The code that I used for the benchmark are as follows:
Codename One controller contains the actual benchmark timing code:
However, today I decided to try writing the same code directly in C using CodenameOne’s native interfaces. This way I would have a baseline with which to compare the performance – so that we know just how much performance we are giving up by using Java instead of C to write an iPhone App.
The native code for this benchmark is as follows:
And it was used in a CodenameOne controller to actually perform the benchmark timing as follows:
I was initially frustrated to find that the C version of my benchmark ran in 0 milliseconds (compared to about 35,000 ms for my Java version compiled with XMLVM/GCC and 42,000ms for my Java version compiled with Avian). Something had to be wrong. I checked to make sure that it was actually calling my function. It was indeed running the function. I tried increasing n to ridiculously large values (3000000), but it still ran in 0ms.
As it turns out the GCC/LLVM compiler was outsmarting my benchmark. Because the benchmark function didn’t return any values to to the caller, and didn’t affect any state outside of its local variables, it decided that the entire function could be replaced with No Op. This is sort of like a compiler’s version of “if a tree falls in the woods and nobody sees it…”. As far as GCC is concerned, the tree didn’t fall.
To solve this problem, I added a static variable that is incremented each time the function is called. This way the compiler wouldn’t be able to eliminate the code as dead. The revised C code looks like:
After modifying the benchmark to update a static variable (so that the compiler could no longer cheat), I was able to finally get a benchmark result. However, this discovery cast a shadow over my previous benchmark results, so I decided to modify the Java version to update a static variable (so that no cheating could take place).
The modified Java version looks like:
This time around, I found that Avian was marginally faster than XMLVM (about 15% faster).
The results of the revised benchmark are as follows:
| Hand-coded C version (GCC/LLVM) | 31.7 seconds |
|---|---|
| XMLVM version | 65.8 seconds |
| Avian version | 57.1 seconds |
Changing to Use Object Methods Instead of Static Method Calls
After posting the above results in the XMLVM forum, Arno (the creator of XMLVM) suggested I try changing the TowerOfHanoi.move() method to not be static. I made the change as follows:
And the the actual benchmark timing code changed to:
Clearly, Arno had some insider knowledge about how the method dispatching works because this produced wildly different results for both XMLVM and Avian:
| XMLVM version | 40.2 seconds |
|---|---|
| Avian version | 77.4 seconds |
So XMLVM, in this test, narrows the gap between native C function calls, and its own method dispatching performance. (Less than 33% difference, which is negligible in the scheme of things). Avian clearly doesn’t handle object methods as quickly as it does static methods, and this really brings to a forefront Joel Dice’s comment that little benchmarks like this don’t really mean much. Applications do much more than perform recursive functions with simple integer arithmetic. The true performance of an application comes down to many factors including graphics, networking, memory allocation, floating point calculations, etc…
These experiments, for me, at least reaffirm that Java is a viable platform for developing for mobile. It should *not* suffer from the same performance problems that current HTML5 solutions do.
A Victory Lap for Java
The benchmarks above make it look like Java is at a disadvantage when compared to purely native apps written in Objective-C (even if in some tests XMLVM appears to have almost caught up). But these benchmarks are comparing to C function calls, which are very fast. What if we change the benchmark to use Objective-C messages. This, is not really fair, since it is well-known that Objective-C message passing is much slower than typical function and method dispatching. Nonetheless, let’s make try, if for no other reason than to know just *how* slow it is.
I changed the Native code as follows, so that it is using message calls for all recursive calls.
Not surprisingly, this benchmark ran much more slowly than the previous version:
| Objective-C Message Calls | 154.5 seconds |
|---|
Thoughts in Summary
AOT-compiled Java (either by Avian or via XMLVM) produces performance that is comparable to native C code. It may actually perform better, but these benchmarks are very narrow in scope. The relative performance of XMLVM and Avian seem to vary widely depending on which operations are being performed. XMLVM seems to handle object methods very well, whereas Avian seemed to have slightly quicker dispatching of static methods.
In any case, I don’t have any reservations about the performance of CodenameOne in relation to native apps. Whether it uses XMLVM, Avian, or some other solution for deploying Java to iOS, it should be able to provide sufficient performance for any sort of application.
See Also
- CodenameOne + Avian = Java on iOS – My blog post discussing my experiment to create an Avian port for CodenameOne to run on iOS
- XMLVM is Actually Pretty Fast – My blog post discussing my initial benchmark results comparing Avian to XMLVM
- Codename One Avian Project on Github