Entitled OSS Users and the Xamarin RoboVM acquisition

RoboVM has been acquired by Xamarin, it was announced; and it would no longer be open source.


It only took five minutes for the forum posts and reddit threads to start up condemning the move as some sort of robbery. The RoboVM was accused of luring unsuspecting users into its community on the promise of open source, only to pull a switcheroo and sell out to big business. Some users were demanding the RoboVM team continue to share their work for free, because … that would only be fair.

RoboVM, on the other hand, explained that they had been open source for a few years and had received little to no contributions from the community, so there wasn’t much incentive to continue with that approach. My personal experience with managing open source projects is consistent with theirs. I released the first version of Xataface in 2005. In that time it has hundreds of thousands of downloads, and is still used in many enterprises as the back-bone of their web information systems (I don’t have an exact count since most apps built with Xataface are internal). In that time, I can count the number of community contributions on my fingers and toes. I’m thankful to all of the users who did contribute. But let’s be real, the case for open sourcing a project because the community will contribute is not compelling.

Shut up and Fork it!

No, really. The source (albeit a couple of months out of date) is still on GitHub and it is licensed under the GPL. That repository represents countless hours of high-quality work by incredibly skilled individuals. That is one hell of a contribution to the open source community. Let them move on; And if you want your open source RoboVM, you can build on this fantastic source base.

Personally, I think it is highly likely that the last open source version of RoboVM will continue to circulate for a long time to come. At least in its core as an AOT java VM, it should be maintainable by people on the outside because most of the heavy lifting is already done there. It is the value-added components like the iOS API bindings, and tool support, that will be difficult for the community to maintain going forward. These things are evolving too fast for volunteers to keep up with.

Dependent Tools

If you are an iOS developer who just uses RoboVM to build iOS apps in Java, then the move to close the source probably won’t affect you – except that your costs may be going up some. I wonder more about the impact that this has on other developer tools that have made RoboVM an integral part of their tool chain. I’m thinking about companies like Gluon that provides JavaFX support for iOS and Android. They use RoboVM for their iOS builds. DukeScript, which allows you to write Java apps with an HTML5 UI and deploy to iOS (and other platforms), also uses RoboVM for its iOS builds. How will they respond.

I had argued as recently as 6 months ago that we (at Codename One) should incorporate RoboVM into our toolchain rather than maintain our own Java VM. But we ultimately decided that there was too much risk in that approach because “what if RoboVM closes down, or goes closed source”. 20/20 hindsight shows that we made the right choice and our new iOS VM is now quite mature, performant, and robust. But most importantly we are not dependent upon other external factors for maintaining it.

What Open Source VMs are Left for iOS?

RoboVM wasn’t the only open source VM for iOS. It was just the most active, and provided the best and most comprehensive bindings to the iOS native APIs. But there are alternative VMs that the open source community may turn to for their supply chain. For example:

  1. Codename One (proper) – (Full disclosure, I work for Codename One)… Codename One is open source and provides a full cross-platform Java solution for write once run anywhere.
  2. Codename One’s VM – Codename One has developed its own Java VM for iOS that works as a cross-compiler from Java to C. This is open source and is a good option for Java tools that need a path to iOS.
  3. Avian – Avian is an AOT Java compiler that can be used to compile java directly to iOS binaries. It is written in C++, and has a very permissive license.
  4. XMLVM. This project has been discontinued. But I mention it for completeness in case people want to revive it.
  5. OpenJDK for iOS. It has been approved for an iOS port of the Open JDK to be developed. This may also present a long-term option, but this is still only in the planning stage.
  6. J2ObjC – A transpiler that converts Java source code into Objective-C source code.
  7. JUniversal – Java source transpiler to C# and C++ that includes a runtime library to help with portability.

If a Drive Dies Once, It’s a Dud

Purchased an iMac from Best Buy in 2011. A few months later it wouldn’t boot up due to a problem with the hard drive. Took it to the Mac store, and their solution was to reformat the drive. After the format, their diagnostic tests said that the drive was “A OK”.

Rewind 15 years. In another lifetime I was a computer tech support agent for a major computer company. We would regularly receive calls from people whose hard drives had crashed. Our standard procedure was to first try and reformat their drive and let them go. This might buy them a day, a week, or even a few years. But inevitably they would be calling tech support again with a dead drive. It was predictable.

So now that I was faced with this situation on my own computer, my instincts jumped to the forefront and screamed at me that “this is a bad drive… they need to replace it”. I urged them to replace it, but they refused, saying that there was nothing wrong with the drive. If I wanted a different drive, I would have to buy it. Warranty wouldn’t cover it.

I pushed just shy of making a scene.

I should have made a scene.

I have now suffered through 5 years of a computer that has always seemed to be a little slower than it should be. And the drive has died. Of course, outside the warranty.

Now, I’m purchasing a new drive and going through a few days of lowered productivity because I didn’t make enough of a scene in the Apple store that day.

If the drive dies in the first year for any reason, it’s a dud. Don’t fall for the “reformat and it’s fine trick”. It’s not fine.

iOS Certificate Wizard

I don’t know about you, but I cringe every time I need to deploy apps to iOS. There is always something with the certificates or provisioning profiles that requires tinkering. No more, I say!

This is a sneak peak of a new certificate wizard I developed for Codename One. It reduces the problem of generating certificates and provisioning profile down to a few clicks. Watch for this to be included in the Codename One plugin as soon as next week.

JavaFX WebView HTML5 API Support


I played with the JavaFX web view recently. The idea was to embed an HTML5 application that I had built into a plugin that we could embed inside the a Java-based IDE like NetBeans, Eclipse, or IntelliJ. The application ran fine inside the major browsers so, in theory, it should run fine inside the JavaFX web view, since it is based on WebKit. Of course, I expected a few hiccups along the way — it wouldn’t just work. Unfortunately, rather than hiccups , I hit road blocks.

My initial attempts to load the app resulted in a blank screen with no error messages reported. The first troubleshooting step, of course, was to attach error listeners to the web view. I found the API to a little bit opaque on exactly how best to log errors, but it wasn’t hard to find forum threads like this StackOverflow post that provided tips.

Unfortunately, after installing logging to just about every part of the web view, I still ended up with a blank white screen with no errors.

I was determined to get this working so I installed Firebug Lite, as described in this thread. This gave me the ability to probe the DOM at runtime more easily to see “where things were at”. I poked around for quite a while but didn’t get very far.

My next thought was that I must have been relying on some APIs that weren’t supported in the version of WebKit that shipped with JavaFX. At this point, I wasn’t even sure which Javascript engine was being used. Were they somehow using Nashorn for the Javascript Engine? Or were they using V8 or something else? It turns out that the WebView just uses the default Javascript Engine that shipped with WebKit (Javascript Core). Since the webview was proving so difficult to debug, I decided to download the same version of WebKit that was being used in JavaFX. I retrieved the version from “User Agent” string – it was 537.44.

Unfortunately, the app worked fine in that version of WebKit.

Fast forward 3 or 4 hours after banging my head against FireBug, WebKit and JavaFX, I decided to see exactly which APIs were supported in the WebView to see if something was missing.

I don’t know why I didn’t try this at the beginning. Next time I will do this at the beginning.

This information isn’t published anywhere that I could find so I loaded up html5test.com – one of those web sites that tells you information about your browser. And the result was this.

Check out that link if you want to see exactly which APIs are supported. Some highlights:

  • Score: 318 out 555
  • No file or file system APIs.
  • No IndexedDB or WebSQL.
  • No WebGL
  • No request.getAnimationFrame

I still don’t exactly know what the problem was with my app. It is a pretty complex app that uses a lot of APIs. Likely it was making a callback that never returns… or something like that.

I ultimately opted for a different solution that still met the requirements.

The nice thing about Javascript is that it is so dynamic, so missing APIs can be overcome with polyfills. JavaFX’s ability to talk back and forth between Java and Javascript offers even more flexibility. Perhaps, if I get the time and motivation, I’ll start a project for implementing polyfills for the JavaFX web view to bring in some of these missing APIs.

My WWDC 2015 “Hey that’s cool!” list

The 2015 WWDC keynote included a lot of “meh”, but a few “hey that’s cool”s. Here is my “hey that’s cool” list.

1. Search API

Apple announced a new search API that iOS app developers can use to allow users to search within their application from.

We now have an API for search. So now when a user performs a search, we can find content behind the apps they have installed on the device, and pull those up in results. And when they tap, they’re deep linked directly into the application. We even provide a convenient back link so they can get right back to their search results. We think these kinds of intelligence features really make a huge difference in your experience in iOS.

Android has provided a similar API for a while now, so this feature is really just playing a little catch-up. But what this means for Codename One is that it will now be possible to design a cross-platform API for this.

2. Swift Open Source

When Swift was first announced (and not open sourced), many pundits were cynical of Apple’s motives – particularly suspecting that Apple was looking to create yet another “lock-in” mechanism for developers. While this is still probably true, their decision to open source it at least some good faith. Their announcement from the Swift blog

Here is what we can tell you so far: 1. Swift source code will be released under an OSI-approved permissive license. 2. Contributions from the community will be accepted — and encouraged. 3. At launch we intend to contribute ports for OS X, iOS, and Linux. 4. Source code will include the Swift compiler and standard library.

We think it would be amazing for Swift to be on all your favorite platforms.

With Swift’s apparent success, one must wonder how long Objective-C will remain “supported”. Apple historically has had no qualms about pulling the rug out from developers when they want to make a shift, and Objective-C certainly is showing its age – so don’t be surprised if, some time in the next couple of years – we get a new WWDC bombshell that Objective-C is being scrapped.

Codename One currently compiles everything down to plain old C code so we would be largely unaffected by such a change – although there would be some native portions that use Objective-C that would need updating. Luckily, if you’re a Codename One user, you don’t need to concern yourself with these details because you are working in Java.

3. iPad Split Screen

This is the coolest advance of them all. Being able to have two apps open at once makes the iPad much more functional – one step closer to replacing the laptop for a lot of users. As a power user, I have often felt that the iPad is not living up to its potential as a computing device – and usually ends up working like just a large iPhone. Small advances like split screen open the doors for app developers to make innovative apps that empower the user to do things that weren’t previously possible.

One small bit of irony is that many apps that were developed using Apple’s development toolkit will need have modifications made to support the different sizing that would occur when an app is displayed in split screen. Codename One apps should “just work”, since it is designed from the ground up, with layout managers, to work across many different devices and screen dimensions.

4. Metal

Metal was announced last year as a more efficient graphics layer than OpenGL for iOS. Now it is available on Mac OS X. Apple describes Metal thusly:

Metal provides the lowest-overhead access to the GPU, enabling you to maximize the graphics and compute potential of your apps on iOS and OS X. With a streamlined API, precompiled shaders, and support for efficient multi-threading, Metal can take your game or app to the next level of performance and capability.

In the key note they cited performance improvements by switching to Metal between 30% and 700%. Codename One is currently using OpenGL for all of our graphics rendering but we have had our eye on Metal for some time. It is uncertain how much we can stand to gain by a switch to Metal. My conservative estimate would be closer to 30% than 700% – but who knows.

5. Apple Pay

Apparently Apple Pay is catching on. In the keynote they dazzle us with lots of numbers (e.g. 250 banks have signed on, etc…) that make it look like they really have a chance of taking over. They have successfully chiseled out a large chunk of the music and software pie. If they can get a piece of every retail transaction … I shudder to think of how big they could become. Welcome our new overlords.

Writing Synchronous Wrappers for Asynchronous Methods in TeaVM

In my last post, I shared a little bit about the work I’ve been doing porting Codename One into Javascript using TeaVM. In that post I mentioned that TeaVM supports multithreading in the browser. In my opinion, this is a major advancement as it finally allows us to write code synchronously in the browser. I.e. NO MORE CALLBACK HELL!

This is great, however, all of the existing Javascript APIs work asynchronously so if we want to work with them synchronously, we need to write synchronous wrappers for them. I expect that, over time, we’ll develop a standard library of such wrappers, but for now, we may need to do some wrapping on our own.

Example: Synchronous Wrapper for XHR Requests

The following method wraps XMLHTTPRequest to allow us to load the contents of a URL as a byte array. A similar mechanism could be used to fetch the contents as text, or a blob.

public Uint8Array getArrayBuffer(String url){
    Window window = (Window)JS.getGlobal();
    final XMLHttpRequest req = window.createXMLHttpRequest();
    final Object lock = new Object();
    final boolean[] complete = new boolean[1];
    req.open("get", url, true);
    req.setOnReadyStateChange(new ReadyStateChangeHandler() {

        public void stateChanged() {
            if ( req.getReadyState() == XMLHttpRequest.DONE ){

                new Thread() {
                    public void run() {

    while (!complete[0]){
            try {
            } catch (InterruptedException ex) {
                Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);

    if (req.getResponse() == null  ){
        System.out.println("Failed to load resource "+url);
        System.out.println("Status code was "+req.getStatus());
        return null;

    return window.createUint8Array(
            (ArrayBuffer)req.getResponse()), req.getResponseType());


See a neater version of this code as a Gist

A couple of things to notice:

  1. Inside stateChanged(), we spawn a new Thread because threading primitives like synchronized and notifyAll() won’t work directly running in a native Javascript callback (at the time of this writing).
  2. The loop with lock.wait() doesn’t block the main Javascript thread. It uses callbacks in the background to pause execution of this “thread” until the lock.notifyAll() method is called.

Now we can use this method directly in our Java code to load data from a remote server synchronously without blocking the UI thread, and without introducing callback hell into our code.

Porting a Java Project to the Web with TeaVM

I am always experimenting with new technologies just to see what new cool things are possible. As a Java developer, I’m usually toying with JVM-related technologies. The latest new and cool technology that I have been experimenting with is TeaVM: A JVM for the browser. After using it for about 2 months now, I am confident in saying that TeaVM is awesome!

TeaVM vs GWT

Java in the browser is nothing new. GWT has offered Java to Javascript cross-compilation for years. So how is TeaVM any different or better than GWT?

  1. TeaVM operates on Java byte code rather than source code. This means you can convert precompiled .class files or even full .jar files to Javascript.
  2. TeaVM supports (or can potentially support) alternative JVM languages like Scala, JRuby, Kotlin, etc, since it operates on .class files (which all JVM languages ultimately compile to). GWT is limited to just Java.
  3. Multithreading. This is groundbreaking. TeaVM actually supports threading primitives like synchronized, Thread.sleep(), Object.wait/notify, etc.. This is a very new feature of TeaVM. Many have tried to come up with a threading solution in the browser, but TeaVM is the first (in history?) to have successfully done it. See live demo of multithreading

What I’m Using it For

I’m currently working on porting Codename One into Javascript. Codename One is a Write-once-run-anywhere solution for mobile application development. Up until now, it supported deploying apps to iOS, Android, Windows Phone, J2ME, BlackBerry, and the desktop (Windows/Mac)… but notably, it did not support deployment to the web. For the past year, I have been having on and off discussions with the other Codename One developers about the possibility of a Javascript port, but I was advised that it was impossible because Javascript didn’t support multi-threading. Attempts had been made before using GWT but they did not prove successful. Codename One is a heavy user of threading primitives since it maintains its own event-dispatch thread (very similar to Swing), and this would lock up the browser if allowed to run in the Javascript single-threaded environment.

A little over a year ago I came across TeaVM for the first time, and I asked Alexey (its author) whether there was any support for threading. He said “no” at the time, but that he had an idea of how he could implement green threads. At that time I wasn’t working for Codename One and had limited time to spend on projects like this, so I just left it there.

Fast forward to a couple of months ago. I was given the green light to go ahead and attempt a Javascript port. At that time still, it was basically me saying “I think I can do it”, but the rest of the team was skeptical, having failed before with GWT. I contacted Alexey again and told him what I was planning to do.

Aside: I was a little naive at this point to think I could do the port with my current knowledge and tools. Knowing what I know now, there is no way I could have succeeded without the the help of Alexey and TeaVM.

At this point I thought I could take the code that was generated by TeaVM and modify it to use Stratifiedjs to support continuations/pseudo threads. He replied that he was aware of this project but that it wasn’t a good fit because he had a better way.

The full thread that shows the development of Async code generation (i.e. multithreaded support) can be see here.

Within days, he had completed a prototype of his new “CPS style” code generator that supported asynchronous code. He implemented Thread.sleep() and Thread.yield() and posted a test case. Based on these implementations, I added initial support for Object.wait(), Object.notify() and a few other threading primitives. Alexey, has since refactored and improved these implementations, such that probably none of my original contributions are recognizable. The full scope and scale of this project, frankly were a little over my head.

I have now been working on the Javascript port for Codename One for a couple of months. There have been quite a few bug reports, but Alexey has been quick to fix them all. I would say that TeaVM’s multithreading implementation is pretty stable at this point. And it is getting more stable daily.

The Poker Demo

Here is a small teaser of the result of the new Codename One port that is built using TeaVM: the Poker demo. The application itself was not modified from the version deployed to iOS, Android, etc… It just required a port layer to implement the “native” components of the Codename One framework, and the rest was left up to TeaVM to work its magic.

Run the demo yourself here


I have been very impressed by the performance that I have been able to achieve with TeaVM. The generated code is very efficient resulting (give or take) in one line of Javascript code corresponding with one line of Java code.

You might think that adding threading support and blocking synchronous method support would lock up the browser, but you would be dead wrong. TeaVM’s transformation to continuations allows it to provide support for synchronous code without ever blocking the javascript thread. This is because, under the hood, these are implemented using callbacks.


Personally I prefer to use Netbeans as my IDE so I’m not benefiting from the debugging support that Alexey created for Eclipse. Therefore, I do my debugging directly in chrome. This actually isn’t as bad as it sounds. Stack traces in chrome are very readable and map directly to the corresponding Java stack trace (i.e. if an error occurs in javascript it is fairly easy to track it back to the original Java code that caused it).

TeaVM Stacktrace


Another nice thing about TeaVM is that it pays attention to things like executable size. It uses static analysis to strip out dead code prior to conversion so that your app is as small as possible. It also includes an option to minify the code, which further reduces the code size.

More to Come

Personally I think that Java needs to be in the browser, since it is essentially the “operating system of the future”. I also think that the browser can benefit greatly from Java as applications get more complex and difficult to maintain. Currently TeaVM provides the best route between Java and the browser, so I expect it to catch on over the following months and years.

Do you currently have a project that depends on GWT? Or do you have a Java project that you would like to offer on the web? You may want to consider porting it to TeaVM because of all of its nice features (i.e. support for other JVM languages, and multithreading). I have had a great time porting Codename One to TeaVM. I’m happy to answer questions or offer tips if you are interested in porting your project over as well. In addition, I think you’ll find that Alexey is very responsive to bug reports, feedback and questions.

For more information about TeaVM, check out its Github repository.

Hacking the OSCON 2014 Schedule with Codename One & Mirah

I gave a tutorial on Codename One at OSCON this year. Part way through the conference, I learned that OSCON had published its schedule as a public JSON feed and were encouraging developers to create their own schedule apps out of it. I was too busy at the time – but I wish I had known about this before hand as it would have made for a good subject for the tutorial. Alas, five months too late, here is a rough stab at this challenge, for the purpose of showing off a Mirah macro I just developed to make JSON Parsing easier in Codename One.

The State of JSON Parsing In Codename One

Codename One apps are Java apps, so they usually consist of a well-structured data model of Java classes and their instantiated objects. JSON (Javascript Object Notation) data is not strongly typed. It encapsulates a generic tree of maps, lists, and primitive data types. Codename One can easily parse JSON strings to a corresponding but generic data structure consisting only of java.util.Maps, java.util.Lists, Strings, and Doubles:

   JSONParser parser = new JSONParser();
   Map data = parser.parseJSON(jsonString);

This is a start, but it is not satisfactory. For example, our schedule application will have classes like Schedule, Event, Speaker, and Venue. We will definitely want to convert this Map into more specific Java types to help achieve a better level of abstraction in our app. But how?

Currently We need to Manually copy the contents of the generic data structure into the associate java types : Tedious and Error-prone

Why can’t we use a library like Jackson?

Libraries that automate the mapping of JSON data to Java types (like Jackson does) all require reflection. Codename One doesn’t support reflection because it needs to know what code is being used before it is deployed as a means of optimizing the performance and size of the app. If it were changed to support reflection, it would mean that the entire Java class library would have to be included in the app because the compiler wouldn’t know until runtime which classes were in use.

So using a library to handle this is out.

Mirah Macros to the Rescue

If you’ve been following any of my recent development, you know that I have been getting pretty deep into Mirah lately. Why Mirah?

It provides Ruby-like syntax, yet compiles directly to .class files with no runtime dependencies. This makes it ideal for Codename One development. You get all of the performance of Java but the productivity of Ruby.

In order to be able to use Mirah in my own projects, I developed a plugin for NetBeans that allows you to include Mirah source inside existing Java projects and work interchangeably with Java source. This capability is handy since some things are still better done in Java. It also allows me to sprinkle little bits of Mirah into my projects as I see fit.


One of the most powerful features of Mirah is its support for macros. A Mirah macro is a bit of code that is executed at compile time and is able to modify the AST. This allows you to do cool things like add properties and methods to classes, or even generate new classes entirely. This ability is also very powerful for Codename One development as it gives us the ability to perform compile-time reflection.

The data_mapper Macro

I created a macro named data_mapper that generates a class that knows how to convert between JSON and objects of a particular class. It works as follows:

   data_mapper MyClass:MyClassMapper

This generates a class named MyClassMapper that knows how to convert between JSON data and objects of MyClass. All generated “mapper” classes are subclasses of DataMapper.

The OSCON Schedule

The OSCON schema includes 4 types:

  1. Schedule : The umbrella structure.
  2. Event : Represents a single event on the schedule.
  3. Speaker : Represents a speaker at OSCON.
  4. Venue : A venue or room.

In my OSCON application, I created 4 corresponding classes. A simplified version of the Schedule class is:

    public class Schedule {
        private List<Event> events;
        private List<Speaker> speakers;
        private List<Venue> venues;

        // getters & setters, etc...

And the Event class is roughly:

public class Event {
    private String serial;
    private String name;
    private String eventType;
    private String venueSerial;
    private String description;
    private String websiteUrl;
    private Date timeStart;
    private Date timeStop;

    private List<String> speakers;
    private List<String> categories;

    // getters & setters, etc...

Parsing the JSON schedule into my class types, first, involved using the data_mapper macros to generate mappers for my classes:

data_mapper Event:EventMapper
data_mapper Venue:VenueMapper
data_mapper Speaker:SpeakerMapper
data_mapper Schedule:ScheduleMapper

The following is the sum total of the code that is used to actually load the JSON feed and parse it into my Java types:

ScheduleMapper scheduleMapper = new ScheduleMapper();
        new EventMapper(),
        new VenueMapper(),
        new SpeakerMapper()
), new DataMapper.Decorator() {

    public void decorate(DataMapper mapper) {
Schedule schedule = scheduleMapper.readJSONFromURL(

Let’s walk through this example, because there are a couple of implementation details included in this example because it is real-world, and thus not completely trivial.

  1. If we were only mapping a JSON feed that contained a single Java type, we could just instantiate the DataMapper object and parse the feed. However, since there are multiple types in the feed, and the mappers need to know to use each other (e.g. The EventMapper needs to know that when it encounters a Speaker, that it should use the SpeakerMapper etc..), we need to set up a shared context. That is what the DataMapper.createContext() call does. It binds all of the mappers together so that they work as one.
  2. The properties in the JSON feed use snake_case, whereas our Java classes use camelCase for their properties respectively. We use the setReadKeyConversions() method on all mappers to specify that it should convert these automatically.
  3. The readJSONFromURL() method includes a third parameter that specifies the sub-key that is treated as the root of the data structure. Without this it would try to map the top-level JSON object to the Schedule object which would fail. (The schedule is contained in a sub-object with key Schedule).

There are other settings that you can play with such as date formats, but luckily the rest of the default settings work fine with this JSON feed.

At this point, the schedule object is completely populated with the OSCON schedule data, so it can be used to build our app.

Painless, right?


Screen Shot 2014-12-05 at 6.40.35 PM

Screen Shot 2014-12-05 at 6.40.57 PM

Screen Shot 2014-12-05 at 6.40.47 PM

Download the App Source Code

This app is just a work in progress, but it may form the foundation for your own schedule app if you choose. You can download the source from GitHub here.

Build Instructions

Requires Netbeans 7.4 or higher with the following plugins installed:

  1. Codename One
  2. Netbeans Mirah Plugin
  3. Netbeans CN1ML Plugin


  1. Clone the cn1-mirah-json-macros Gitub Project clone https://github.com/shannah/cn1-mirah-json-macros.git
  2. Open the OSCONSchedule project that is a subdirectory of the main project directory, in NetBeans.
  3. Build the project. Modify it at will


Apache 2.0


I created a short screencast of this tutorial so you can see the data_mapper macro in action.


  1. Codename One Homepage
  2. DataMapper API Docs
  3. Codename One Reflection Utilities Download – Contains the libraries necessary to do the JSON parsing in a Codename One app.
  4. Mirah Netbeans Module
  5. OSCON App Source
  6. OSCON DIY Schedule JSON Feed

Using SOUNDEX and MySQL Full-Text Search for Fuzzy Matching

I recently received a question from a Xataface user about how to support searches for misspelled names. Out of the box, Xataface provides some fairly advanced search support that allows users to search on particular columns, across all columns in a single table, or across all tables (from a specified subset of the tables in the database). For the single-table searches, Xataface uses combinations of LIKE and = comparisons depending on the type of query and field type. E.g. If a user enters a search in the last_name column of the people table for “Williams”, it will perform an SQL query like the following, under the hood:

SELECT * FROM `people` where `last_name` LIKE 'Williams'

Note: This isn’t exactly how Xataface generates a query like this. This is simplified for readability.

This works great, but what if we wanted to be able to find results of misspellings of “Williams”. E.g. If the user entered a search for ‘Wlliams’, we still want the same result set to be found.

Soundex is a phonetic algorithm for indexing names by sound, as pronounced in English. It happens to provide a very simple way to search for misspellings. Both PHP and MySQL include a SOUNDEX hashing function that will take string input and produce the SOUNDEX code for that input. The SOUNDEX code for strings that are misspelled are often the same. E.g.

SOUNDEX('Williams') === 'W452'


SOUNDEX('Wlliams') === 'W452'

This means that we can allow users to search for misspellings on the last_name field by modifying the SQL query as follows:

SELECT * FROM `people` where SOUNDEX(`last_name`) = SOUNDEX('Wlliams')

This should match a superset of the records that were matched in the original query. You might verbalize this query as “Find all people with last names that sound like Williams.

Matching Records that Contain A Word

The above example works great if we are searching on a column that contains a single word and we are matching against the whole word, but it falls short in cases where we need to find values that contain a search term. E.g. Suppose the people table also has a bio field that contains a short profile of the person. Now suppose we want to find all people whose bio contains the word “cardiac”. Xataface might generate an SQL query like:

SELECT * FROM `people` where `bio` like '%cardiac%'

Now if we want to find misspellings, we can’t simply do:

SELECT * FROM `people` where SOUNDEX(`bio`) like SOUNDEX('cardiac')

MySQL doesn’t provide a simple way to perform string splitting and mapping inside a WHERE clause so we would need to do some work prior to the search to allow us to search for misspellings in the bio field. One possible solution create an index on the bio field that includes soundex codes for each word. E.g. we could add a field named bio_soundex to the people table. Then every time the bio is changed we could update the bio_soundex. In Xataface we could perform this in the beforeSave trigger:

function beforeSave(Dataface_Record $record){
   if ( $record->valueChanged('bio') ){
      $words = preg_split(
      $soundexWords = array();
      foreach ( $words as $word){
         $soundexWords[] = soundex($word);
      $record->setValue('bio_soundex', implode(' ', $soundexWords));

This example makes use of Xataface specific hooks and classes, but the concept of splitting the field content into individual words, producing the soundex code for each word, then saving the concatenation of these codes (separated by spaces) into another field can easily be done in any system.

Now, we are able to perform a query like this:

SELECT * FROM `people` WHERE `bio_soundex` LIKE CONCAT('%',SOUNDEX('crdiac'),'%')

And this will return all records where the bio contains the word “cardiac” (and many misspellings thereof).

Currently Xataface doesn’t automatically perform this type of soundex conversion for single-table searches. This example shows how you can do it yourself. However, as I describe next, Xataface does perform automatic soundex indexing for multi-table searches.

Fuzzy Matching in Multi-Table Search

Xataface has supported a multi-table search option for several years now. It works by maintaining a central index table with an entry for each record that can be searched. This index stores record details such as its table, id, URL, title, description, and searchable text which has a MySQL full-text search index that is used for the actual searching. Using MySQL’s full-text indexing feature allows Xataface to perform relevance sorting. This index is automatically updated each time a record is saved, and Xataface provides administrators with tools to manually clear and regenerate the index.

If a user enters a search for “Cardiac” in the multi-table search box, Xataface will generate an SQL query similar to:

  match(searchable_text) AGAINST ('Cardiac') as `relevance`
FROM dataface__index
WHERE `lang`='en' AND 
match(searchable_text) against ('Cardiac')
ORDER BY `relevance` DESC

In order to modify the index to support misspelled searches, I used a strategy similar to the one described above (for the bio field). At the time a record is indexed, I parse the content of the searchable_text into words, then produce a string of concatenated soundex codes corresponding to those words, and append them to searchable text field.

The actual code is as follows (From here:

$searchable_text = preg_replace('/<[^>]*?>/', ' ', $searchable_text);
$words = preg_split('/((^\p{P}+)|(\p{P}*\s+\p{P}*)|(\p{P}+$))/', $searchable_text, -1, PREG_SPLIT_NO_EMPTY);
$soundexAddons = array();
foreach ( $words as $word ){
    $soundexAddons[] = soundex(trim($word));

$searchable_text .= '[/////]: '.implode(' ', $soundexAddons);
$searchable_text = strip_tags($searchable_text);

$sql = "
    replace into dataface__index 
if ( !@xf_db_query($sql, df_db()) ){
    if ( !xf_db_query($sql, df_db()) ){
        trigger_error(xf_db_error(df_db()), E_USER_ERROR);

At the time of the search, I perform a similar transformation to the search phrase to append the soundex codes of all words in the query to the end of the query. E.g. If the user searches for “Crdiac”, xataface will actually search for “Crdiac C632″ (C632 is the soundex code for Cardiac).

This snippet shows how this is accomplished (From here:

$words = explode(' ', $query['-search']);
$soundexAddons = array();
foreach ( $words as $word){
    $soundexAddons[] = soundex($word);
$orig_search = $query['-search'];
$query['-search'] .= ' '.implode(' ', $soundexAddons);

So the resulting SQL query will be something like:

  match(searchable_text) AGAINST ('Cardiac C632') as `relevance`
FROM dataface__index
WHERE `lang`='en' AND 
match(searchable_text) against ('Cardiac C632')
ORDER BY `relevance` DESC

More Information

CN1ML: Using HTML to Design Codename One User Interfaces

post-header In Codename One, you traditionally have two choices for designing user interfaces:

  1. Use the GUI builder (i.e. WYSIWYG)
  2. Code the UI manually

Although the GUI builder is great, I often find myself coding the UI by hand because it gives me the most control – and it lets me get directly to the code. Of course, you can mix and match GUI builder GUIs with custom code, and I often do. E.g. I’ll set up each Form’s structure in the GUI builder and populate the details in code. Sometimes shifting back and forth between a WYSIWYG context and Java source code can be trying, though.. For some reason, I don’t like shifting gears. When I’m working in code, I like to dive deep in code and look at nothing else but code. When I’m doing desktop WYSIWYG designs, I want to stay in that environment. But I don’t alternate between them terribly well.

Anyhoo… so that’s why I sometimes find myself coding the entire UI in Java.

This has an ENORMOUS down-side, though. For simple UIs with under 5 components, it is manageable. But when you start to get into a complex UI with multiple levels of nested containers, things become something of a mess. And when you go back to modify the code later, it can be difficult to tell the forest from the trees.

A Codename One UI form is very similar to the HTML document model. It defines a tree structure of user interface components. Expressed in HTML, tree-based document models are quite clean and easy to navigate. It is easy to tell which tags are parents of other tags. Not so with Java code. So I thought:

Wouldn’t it be great if I could express my user interface in HTML?

So I set off to do just that.


Before starting on this mission, I had a few requirements in my head:

  1. The HTML should compile down to Java source code so that it can be compiled just like all of the rest of the classes in Codename One. I.e. The HTML should not be parsed, processed, or converted at runtime. It should be handled completely at compile-time. The reason for this is that we want the build server to be able to optimize the app executables and strip out unneeded classes to keep the app file-size down. It knows how to do this for Java source code, but not with extra HTML files that I might provide to it.
  2. I should be able to access any parts of the Component hierarchy from Java. Complex UIs require much more than just a simple template structure. You need to be able to attach event listeners, and possibly decorate the components dynamically using Java in ways that just can’t be expressed declaratively. Hence I wanted to make it easy to “access” parts of the generated UI objects.
  3. I should be able to work with these files seamlessly inside the IDE. I didn’t want to add an extra step when dealing with these HTML UI interfaces. I’d probably forget how to do it when it came time to return to the project.
  4. No Performance Penalty. I want the UIs to be just as fast as a hand-coded UI.

And so, based on these requirements, I came up with CN1MLCodename One Markup Language. Which is basically just HTML with a couple of special attributes to help specify how the resulting Codename One UI should look.

How it works

I created a Netbeans plugin for CN1ML so that you can add CN1ML templates directly into your Codename One projects. When you save the CN1ML template, it automatically generates a corresponding Java file with the same name, but different extension. E.g. If I have a CN1ML file at com/myapp/MyForm.cn1ml, it will automatically generate a java file at com/myapp/MyForm.java (which is a Java class with fully qualified class name “com.myapp.MyForm”.

You can then instantiate the form in Java my calling the com.myapp.MyForm constructor – which takes a single “Map” argument that can be used to pass parameters to the template. E.g.

    MyForm form = new MyForm(new HashMap());

    // Get the root container from the form 
    Container root = form.getRoot();

What it looks like

Here is a sample Contact Form in CN1ML

The resulting Java source code is long and ugly, but you can check it out here

And here is a screenshot of what this form looks like in an app:

Screen Shot 2014-09-24 at 5.19.05 PM

More about CN1ML

If you’re interested in this project, you can read more about it on the GitHub page.

How I Built It

I built the CN1ML parser/compiler in Mirah. This was one of my first Mirah projects since releasing the Mirah Netbeans Plugin last month. So far so good. I can safely say that Mirah is making me more productive than if I had coded this in Java. There are still some rough edges that I’m shaving off, but nothing major. And in places where Java would be a better choice, I can still write those portions in Java in the same project seamlessly.

In fact it was my experience developing the Mirah Netbeans module that gave me some of the inspiration for this CN1ML module. I looked at the problem and knew it could be achieved quite easily using the NetBeans API.


I created this screen cast to demonstrate how the CN1ML plugin works.


  1. CN1ML Netbeans Plugin Homepage
  2. CN1ML Tag and Attribute Reference
  3. Some Samples with Screenshots

Ramblings about Xataface, Java, and other software development issues