I gave a tutorial on Codename One at OSCON this year. Part way through the conference, I learned that OSCON had published its schedule as a public JSON feed and were encouraging developers to create their own schedule apps out of it. I was too busy at the time – but I wish I had known about this before hand as it would have made for a good subject for the tutorial. Alas, five months too late, here is a rough stab at this challenge, for the purpose of showing off a Mirah macro I just developed to make JSON Parsing easier in Codename One.

The State of JSON Parsing In Codename One

Codename One apps are Java apps, so they usually consist of a well-structured data model of Java classes and their instantiated objects. JSON (Javascript Object Notation) data is not strongly typed. It encapsulates a generic tree of maps, lists, and primitive data types. Codename One can easily parse JSON strings to a corresponding but generic data structure consisting only of java.util.Maps, java.util.Lists, Strings, and Doubles:

   JSONParser parser = new JSONParser();
   Map data = parser.parseJSON(jsonString);

This is a start, but it is not satisfactory. For example, our schedule application will have classes like Schedule, Event, Speaker, and Venue. We will definitely want to convert this Map into more specific Java types to help achieve a better level of abstraction in our app. But how?

Currently We need to Manually copy the contents of the generic data structure into the associate java types : Tedious and Error-prone

Why can’t we use a library like Jackson?

Libraries that automate the mapping of JSON data to Java types (like Jackson does) all require reflection. Codename One doesn’t support reflection because it needs to know what code is being used before it is deployed as a means of optimizing the performance and size of the app. If it were changed to support reflection, it would mean that the entire Java class library would have to be included in the app because the compiler wouldn’t know until runtime which classes were in use.

So using a library to handle this is out.

Mirah Macros to the Rescue

If you’ve been following any of my recent development, you know that I have been getting pretty deep into Mirah lately. Why Mirah?

It provides Ruby-like syntax, yet compiles directly to .class files with no runtime dependencies. This makes it ideal for Codename One development. You get all of the performance of Java but the productivity of Ruby.

In order to be able to use Mirah in my own projects, I developed a plugin for NetBeans that allows you to include Mirah source inside existing Java projects and work interchangeably with Java source. This capability is handy since some things are still better done in Java. It also allows me to sprinkle little bits of Mirah into my projects as I see fit.

Macros

One of the most powerful features of Mirah is its support for macros. A Mirah macro is a bit of code that is executed at compile time and is able to modify the AST. This allows you to do cool things like add properties and methods to classes, or even generate new classes entirely. This ability is also very powerful for Codename One development as it gives us the ability to perform compile-time reflection.

The data_mapper Macro

I created a macro named data_mapper that generates a class that knows how to convert between JSON and objects of a particular class. It works as follows:

   data_mapper MyClass:MyClassMapper

This generates a class named MyClassMapper that knows how to convert between JSON data and objects of MyClass. All generated “mapper” classes are subclasses of DataMapper.

The OSCON Schedule

The OSCON schema includes 4 types:

1. Schedule : The umbrella structure.
2. Event : Represents a single event on the schedule.
3. Speaker : Represents a speaker at OSCON.
4. Venue : A venue or room.

In my OSCON application, I created 4 corresponding classes. A simplified version of the Schedule class is:

    public class Schedule {
        private List<Event> events;
        private List<Speaker> speakers;
        private List<Venue> venues;

        // getters & setters, etc...

And the Event class is roughly:

public class Event {
    private String serial;
    private String name;
    private String eventType;
    private String venueSerial;
    private String description;
    private String websiteUrl;
    private Date timeStart;
    private Date timeStop;

    private List<String> speakers;
    private List<String> categories;

    // getters & setters, etc...

Parsing the JSON schedule into my class types, first, involved using the data_mapper macros to generate mappers for my classes:

data_mapper Event:EventMapper
data_mapper Venue:VenueMapper
data_mapper Speaker:SpeakerMapper
data_mapper Schedule:ScheduleMapper

The following is the sum total of the code that is used to actually load the JSON feed and parse it into my Java types:

ScheduleMapper scheduleMapper = new ScheduleMapper();
DataMapper.createContext(Arrays.asList(
        scheduleMapper,
        new EventMapper(),
        new VenueMapper(),
        new SpeakerMapper()
), new DataMapper.Decorator() {

    public void decorate(DataMapper mapper) {
        mapper.setReadKeyConversions(Arrays.asList(DataMapper.CONVERSION_CAMEL_TO_SNAKE));
    }
});
Schedule schedule = scheduleMapper.readJSONFromURL(
        "http://www.oreilly.com/pub/sc/osconfeed", 
        Schedule.class, 
        "/Schedule"
);

Let’s walk through this example, because there are a couple of implementation details included in this example because it is real-world, and thus not completely trivial.

1. If we were only mapping a JSON feed that contained a single Java type, we could just instantiate the DataMapper object and parse the feed. However, since there are multiple types in the feed, and the mappers need to know to use each other (e.g. The EventMapper needs to know that when it encounters a Speaker, that it should use the SpeakerMapper etc..), we need to set up a shared context. That is what the DataMapper.createContext() call does. It binds all of the mappers together so that they work as one.
2. The properties in the JSON feed use snake_case, whereas our Java classes use camelCase for their properties respectively. We use the setReadKeyConversions() method on all mappers to specify that it should convert these automatically.
3. The readJSONFromURL() method includes a third parameter that specifies the sub-key that is treated as the root of the data structure. Without this it would try to map the top-level JSON object to the Schedule object which would fail. (The schedule is contained in a sub-object with key Schedule).

There are other settings that you can play with such as date formats, but luckily the rest of the default settings work fine with this JSON feed.

At this point, the schedule object is completely populated with the OSCON schedule data, so it can be used to build our app.

Painless, right?

Screenshots

Download the App Source Code

This app is just a work in progress, but it may form the foundation for your own schedule app if you choose. You can download the source from GitHub here.

Build Instructions

Requires Netbeans 7.4 or higher with the following plugins installed:

1. Codename One
2. Netbeans Mirah Plugin
3. Netbeans CN1ML Plugin

Steps:

1. Clone the cn1-mirah-json-macros Gitub Project clone https://github.com/shannah/cn1-mirah-json-macros.git
2. Open the OSCONSchedule project that is a subdirectory of the main project directory, in NetBeans.
3. Build the project. Modify it at will

License

Apache 2.0

Screencast

I created a short screencast of this tutorial so you can see the data_mapper macro in action.

Links

1. Codename One Homepage
2. DataMapper API Docs
3. Codename One Reflection Utilities Download – Contains the libraries necessary to do the JSON parsing in a Codename One app.
4. Mirah Netbeans Module
5. OSCON App Source
6. OSCON DIY Schedule JSON Feed

In Codename One, you traditionally have two choices for designing user interfaces:

1. Use the GUI builder (i.e. WYSIWYG)
2. Code the UI manually

Although the GUI builder is great, I often find myself coding the UI by hand because it gives me the most control – and it lets me get directly to the code. Of course, you can mix and match GUI builder GUIs with custom code, and I often do. E.g. I’ll set up each Form’s structure in the GUI builder and populate the details in code. Sometimes shifting back and forth between a WYSIWYG context and Java source code can be trying, though.. For some reason, I don’t like shifting gears. When I’m working in code, I like to dive deep in code and look at nothing else but code. When I’m doing desktop WYSIWYG designs, I want to stay in that environment. But I don’t alternate between them terribly well.

Anyhoo… so that’s why I sometimes find myself coding the entire UI in Java.

This has an ENORMOUS down-side, though. For simple UIs with under 5 components, it is manageable. But when you start to get into a complex UI with multiple levels of nested containers, things become something of a mess. And when you go back to modify the code later, it can be difficult to tell the forest from the trees.

A Codename One UI form is very similar to the HTML document model. It defines a tree structure of user interface components. Expressed in HTML, tree-based document models are quite clean and easy to navigate. It is easy to tell which tags are parents of other tags. Not so with Java code. So I thought:

Wouldn’t it be great if I could express my user interface in HTML?

So I set off to do just that.

Requirements:

Before starting on this mission, I had a few requirements in my head:

1. The HTML should compile down to Java source code so that it can be compiled just like all of the rest of the classes in Codename One. I.e. The HTML should not be parsed, processed, or converted at runtime. It should be handled completely at compile-time. The reason for this is that we want the build server to be able to optimize the app executables and strip out unneeded classes to keep the app file-size down. It knows how to do this for Java source code, but not with extra HTML files that I might provide to it.
2. I should be able to access any parts of the Component hierarchy from Java. Complex UIs require much more than just a simple template structure. You need to be able to attach event listeners, and possibly decorate the components dynamically using Java in ways that just can’t be expressed declaratively. Hence I wanted to make it easy to “access” parts of the generated UI objects.
3. I should be able to work with these files seamlessly inside the IDE. I didn’t want to add an extra step when dealing with these HTML UI interfaces. I’d probably forget how to do it when it came time to return to the project.
4. No Performance Penalty. I want the UIs to be just as fast as a hand-coded UI.

And so, based on these requirements, I came up with CN1ML – Codename One Markup Language. Which is basically just HTML with a couple of special attributes to help specify how the resulting Codename One UI should look.

How it works

I created a Netbeans plugin for CN1ML so that you can add CN1ML templates directly into your Codename One projects. When you save the CN1ML template, it automatically generates a corresponding Java file with the same name, but different extension. E.g. If I have a CN1ML file at com/myapp/MyForm.cn1ml, it will automatically generate a java file at com/myapp/MyForm.java (which is a Java class with fully qualified class name “com.myapp.MyForm”.

You can then instantiate the form in Java my calling the com.myapp.MyForm constructor – which takes a single “Map” argument that can be used to pass parameters to the template. E.g.

    MyForm form = new MyForm(new HashMap());

    // Get the root container from the form 
    Container root = form.getRoot();

What it looks like

Here is a sample Contact Form in CN1ML

The resulting Java source code is long and ugly, but you can check it out here

And here is a screenshot of what this form looks like in an app:

More about CN1ML

If you’re interested in this project, you can read more about it on the GitHub page.

How I Built It

I built the CN1ML parser/compiler in Mirah. This was one of my first Mirah projects since releasing the Mirah Netbeans Plugin last month. So far so good. I can safely say that Mirah is making me more productive than if I had coded this in Java. There are still some rough edges that I’m shaving off, but nothing major. And in places where Java would be a better choice, I can still write those portions in Java in the same project seamlessly.

In fact it was my experience developing the Mirah Netbeans module that gave me some of the inspiration for this CN1ML module. I looked at the problem and knew it could be achieved quite easily using the NetBeans API.

Screencast

I created this screen cast to demonstrate how the CN1ML plugin works.

Resources

1. CN1ML Netbeans Plugin Homepage
2. CN1ML Tag and Attribute Reference
3. Some Samples with Screenshots

One afternoon, a few months ago, I was working on an iOS app in Codename One, and I began to wonder if there was some non-drastic way to remove some of the boiler plate from the coding process. I love Java. Its static-typing and verbosity enable me to build large, complex, fast, robust applications without losing my mind. However, these strengths can feel like weaknesses when you just want to get down to business and code up an algorithm.

I wasn’t looking to replace the whole Java eco-system – I chose most of my current development tools (e.g. Codename One) because they use Java – so I wasn’t imagining abandoning this. I still wanted to be able to use NetBeans, and Codename One, and all of my Java libraries. I just wanted to be able to occasionally skip some of the ceremony. I was imagining a sort of Javascript-like macro language with some type hints that could be automatically expanded to the full Java source by some automated preprocessor.

I was aware of all the popular JVM languages (e.g. Scala, Groovy, Kotlin, JRuby, Jython, etc..) and I do use them for some projects, but none of these are appropriate for mobile development with Codename One (which was the primary platform I wanted to target). The problem with these is that they all require runtime libraries that would have to be included in the final app. This substantially increases the app size, and, with mobile development, I need to keep the app size as small as possible. Some of these languages are statically compiled, and therefore, compile-time tools (e.g. Proguard) can be used to strip out portions of the runtime that aren’t used, but porting their runtimes into Codename One would have still been difficult, since the CN1 class library is only a small subset of the JavaSE libraries (for the purpose of keeping apps small, and maintaining compatibility with legacy devices e.g. BlackBerry and J2ME).

So, what I was looking for was a language that:

1. Compiled to JVM bytecode.
2. Did not add any runtime dependencies.

Mirah: The Silver Bullet

It turns out the Mirah was exactly what I was looking for. Mirah was originally developed in 2009 by Charles Nutter under the name “Duby”. This video of his presentation at Ruby Conf 2009 gives a good preview the language. It is heavily inspired by Ruby, and uses aggressive compile-time type inference to be able to remove most of the verbosity and boiler plate that is customary in a static language. I’m not going to get into the specifics of the language in this post. You can read about Mirah and its syntax on the Mirah website. I do want to mention, though, that it is awesome and that I intend to start shifting as much development to Mirah as possible going forward.

IDE Support

As it stood the day I discovered it, Mirah had potential as a language for Codename One development, but it still would not have been practical to start developing apps with it because there was no IDE support. Most/all of the Mirah devs seemed to be Ruby devs who migrated to Mirah for the performance. And Ruby devs don’t use IDEs like Java devs do.

I was not going to give up my IDE… Not for any language

If I were to switch to using a text editor, any productivity gains that I realized due to the streamlined language syntax would have been offset by the loss of IDE nicities like code completion, unit test generation, code navigation, and GUI builder integration. Even if there were a Mirah IDE or support for Mirah projects in an existing IDE, I was not willing to sacrifice all of the tools that are provided in the core Java Codename One projects.

I needed to be able to use Mirah inside my existing Java projects, interchangeably with Java.

In order to accomplish this, I decided to create a Mirah NetBeans plugin.

The Mirah NetBeans Plugin

I just released an update for the Mirah Netbeans Plugin today. It is now at a point where you can reasonably incorporate Mirah code into a Codename One application. There are lots of features, big and small, but the utility of the plugin boils down to two things:

1. It can build Codename One applications that include .mirah source files. (It also supports other project types, but I’ll focus on CN1 here).
2. It allows you to edit .mirah source files with all of the things you expect in an IDE (method completion, type hints, incremental compiling/error reporting).

So, if you have, this plugin installed in your NetBeans instance, you should be able to just add Mirah source files inside your existing Codename One projects. Two-way dependencies are supported. E.g. Your .java files can use classes defined in .mirah source, and vice-versa. This way you can choose to implement some parts of your App in Java, and other parts in Mirah. This is important because some parts need to be written in Java still (E.g. If you are writing a GUI builder app, the GUI Editor will be adding methods to the StateMachine class, which needs to be Java.

Download the Mirah Netbeans Plugin and start writing Mirah code in your Codename One apps today.

Proof of Concept: Poker Demo

As a proof of concept, I decided to port the Poker Demo that Shai wrote into Mirah.

• View Java Source
• View Mirah Source

You can clone the entire project and build it yourself using Netbeans with the Codename One and Mirah plugins installed.

Read more about this project along with some detailed code comparisons on the Poker Demo readme page.

The following screencast demonstrates the use of the NetBeans Mirah plugin to develop a Codename One application using Mirah.