CN1ML: Using HTML to Design Codename One User Interfaces

post-header In Codename One, you traditionally have two choices for designing user interfaces:

  1. Use the GUI builder (i.e. WYSIWYG)
  2. Code the UI manually

Although the GUI builder is great, I often find myself coding the UI by hand because it gives me the most control – and it lets me get directly to the code. Of course, you can mix and match GUI builder GUIs with custom code, and I often do. E.g. I’ll set up each Form’s structure in the GUI builder and populate the details in code. Sometimes shifting back and forth between a WYSIWYG context and Java source code can be trying, though.. For some reason, I don’t like shifting gears. When I’m working in code, I like to dive deep in code and look at nothing else but code. When I’m doing desktop WYSIWYG designs, I want to stay in that environment. But I don’t alternate between them terribly well.

Anyhoo… so that’s why I sometimes find myself coding the entire UI in Java.

This has an ENORMOUS down-side, though. For simple UIs with under 5 components, it is manageable. But when you start to get into a complex UI with multiple levels of nested containers, things become something of a mess. And when you go back to modify the code later, it can be difficult to tell the forest from the trees.

A Codename One UI form is very similar to the HTML document model. It defines a tree structure of user interface components. Expressed in HTML, tree-based document models are quite clean and easy to navigate. It is easy to tell which tags are parents of other tags. Not so with Java code. So I thought:

Wouldn’t it be great if I could express my user interface in HTML?

So I set off to do just that.

Requirements:

Before starting on this mission, I had a few requirements in my head:

  1. The HTML should compile down to Java source code so that it can be compiled just like all of the rest of the classes in Codename One. I.e. The HTML should not be parsed, processed, or converted at runtime. It should be handled completely at compile-time. The reason for this is that we want the build server to be able to optimize the app executables and strip out unneeded classes to keep the app file-size down. It knows how to do this for Java source code, but not with extra HTML files that I might provide to it.
  2. I should be able to access any parts of the Component hierarchy from Java. Complex UIs require much more than just a simple template structure. You need to be able to attach event listeners, and possibly decorate the components dynamically using Java in ways that just can’t be expressed declaratively. Hence I wanted to make it easy to “access” parts of the generated UI objects.
  3. I should be able to work with these files seamlessly inside the IDE. I didn’t want to add an extra step when dealing with these HTML UI interfaces. I’d probably forget how to do it when it came time to return to the project.
  4. No Performance Penalty. I want the UIs to be just as fast as a hand-coded UI.

And so, based on these requirements, I came up with CN1MLCodename One Markup Language. Which is basically just HTML with a couple of special attributes to help specify how the resulting Codename One UI should look.

How it works

I created a Netbeans plugin for CN1ML so that you can add CN1ML templates directly into your Codename One projects. When you save the CN1ML template, it automatically generates a corresponding Java file with the same name, but different extension. E.g. If I have a CN1ML file at com/myapp/MyForm.cn1ml, it will automatically generate a java file at com/myapp/MyForm.java (which is a Java class with fully qualified class name “com.myapp.MyForm”.

You can then instantiate the form in Java my calling the com.myapp.MyForm constructor – which takes a single “Map” argument that can be used to pass parameters to the template. E.g.

    MyForm form = new MyForm(new HashMap());

    // Get the root container from the form 
    Container root = form.getRoot();

What it looks like

Here is a sample Contact Form in CN1ML

The resulting Java source code is long and ugly, but you can check it out here

And here is a screenshot of what this form looks like in an app:

Screen Shot 2014-09-24 at 5.19.05 PM

More about CN1ML

If you’re interested in this project, you can read more about it on the GitHub page.

How I Built It

I built the CN1ML parser/compiler in Mirah. This was one of my first Mirah projects since releasing the Mirah Netbeans Plugin last month. So far so good. I can safely say that Mirah is making me more productive than if I had coded this in Java. There are still some rough edges that I’m shaving off, but nothing major. And in places where Java would be a better choice, I can still write those portions in Java in the same project seamlessly.

In fact it was my experience developing the Mirah Netbeans module that gave me some of the inspiration for this CN1ML module. I looked at the problem and knew it could be achieved quite easily using the NetBeans API.

Screencast

I created this screen cast to demonstrate how the CN1ML plugin works.

Resources

  1. CN1ML Netbeans Plugin Homepage
  2. CN1ML Tag and Attribute Reference
  3. Some Samples with Screenshots

Mirah for Codename One App Development

cn1plusmirah

One afternoon, a few months ago, I was working on an iOS app in Codename One, and I began to wonder if there was some non-drastic way to remove some of the boiler plate from the coding process. I love Java. Its static-typing and verbosity enable me to build large, complex, fast, robust applications without losing my mind. However, these strengths can feel like weaknesses when you just want to get down to business and code up an algorithm.

I wasn’t looking to replace the whole Java eco-system – I chose most of my current development tools (e.g. Codename One) because they use Java – so I wasn’t imagining abandoning this. I still wanted to be able to use NetBeans, and Codename One, and all of my Java libraries. I just wanted to be able to occasionally skip some of the ceremony. I was imagining a sort of Javascript-like macro language with some type hints that could be automatically expanded to the full Java source by some automated preprocessor.

I was aware of all the popular JVM languages (e.g. Scala, Groovy, Kotlin, JRuby, Jython, etc..) and I do use them for some projects, but none of these are appropriate for mobile development with Codename One (which was the primary platform I wanted to target). The problem with these is that they all require runtime libraries that would have to be included in the final app. This substantially increases the app size, and, with mobile development, I need to keep the app size as small as possible. Some of these languages are statically compiled, and therefore, compile-time tools (e.g. Proguard) can be used to strip out portions of the runtime that aren’t used, but porting their runtimes into Codename One would have still been difficult, since the CN1 class library is only a small subset of the JavaSE libraries (for the purpose of keeping apps small, and maintaining compatibility with legacy devices e.g. BlackBerry and J2ME).

So, what I was looking for was a language that:

  1. Compiled to JVM bytecode.
  2. Did not add any runtime dependencies.

Mirah: The Silver Bullet

It turns out the Mirah was exactly what I was looking for. Mirah was originally developed in 2009 by Charles Nutter under the name “Duby”. This video of his presentation at Ruby Conf 2009 gives a good preview the language. It is heavily inspired by Ruby, and uses aggressive compile-time type inference to be able to remove most of the verbosity and boiler plate that is customary in a static language. I’m not going to get into the specifics of the language in this post. You can read about Mirah and its syntax on the Mirah website. I do want to mention, though, that it is awesome and that I intend to start shifting as much development to Mirah as possible going forward.

IDE Support

As it stood the day I discovered it, Mirah had potential as a language for Codename One development, but it still would not have been practical to start developing apps with it because there was no IDE support. Most/all of the Mirah devs seemed to be Ruby devs who migrated to Mirah for the performance. And Ruby devs don’t use IDEs like Java devs do.

I was not going to give up my IDE… Not for any language

If I were to switch to using a text editor, any productivity gains that I realized due to the streamlined language syntax would have been offset by the loss of IDE nicities like code completion, unit test generation, code navigation, and GUI builder integration. Even if there were a Mirah IDE or support for Mirah projects in an existing IDE, I was not willing to sacrifice all of the tools that are provided in the core Java Codename One projects.

I needed to be able to use Mirah inside my existing Java projects, interchangeably with Java.

In order to accomplish this, I decided to create a Mirah NetBeans plugin.

The Mirah NetBeans Plugin

I just released an update for the Mirah Netbeans Plugin today. It is now at a point where you can reasonably incorporate Mirah code into a Codename One application. There are lots of features, big and small, but the utility of the plugin boils down to two things:

  1. It can build Codename One applications that include .mirah source files. (It also supports other project types, but I’ll focus on CN1 here).
  2. It allows you to edit .mirah source files with all of the things you expect in an IDE (method completion, type hints, incremental compiling/error reporting).

So, if you have, this plugin installed in your NetBeans instance, you should be able to just add Mirah source files inside your existing Codename One projects. Two-way dependencies are supported. E.g. Your .java files can use classes defined in .mirah source, and vice-versa. This way you can choose to implement some parts of your App in Java, and other parts in Mirah. This is important because some parts need to be written in Java still (E.g. If you are writing a GUI builder app, the GUI Editor will be adding methods to the StateMachine class, which needs to be Java.

Download the Mirah Netbeans Plugin and start writing Mirah code in your Codename One apps today.

Proof of Concept: Poker Demo

Poker Demo Screenshot

As a proof of concept, I decided to port the Poker Demo that Shai wrote into Mirah.

You can clone the entire project and build it yourself using Netbeans with the Codename One and Mirah plugins installed.

Read more about this project along with some detailed code comparisons on the Poker Demo readme page.

The following screencast demonstrates the use of the NetBeans Mirah plugin to develop a Codename One application using Mirah.

OSCON 2014 Reflections

I’m sitting in Portland International Airport waiting for the chariot that will return me to Vancouver, so I thought I’d pass the time by reflecting on my experience at OSCON. I am not generally the kind of guy that gets the fullest experience out of a conference. I attend the talks, maybe meet a few people, and return to my hotel room to watch some Netflix. But even a social caterpillar like me has fun at these things. I thoroughly enjoyed all of the talks that I attended. I learned a lot from the tutorials, and I found the keynotes engaging.

Three talks particularly stood out to me, for various reasons:

Using D3 for Data Visualizations

The first tutorial that I attended (on Sunday), was led by Josh Marinacci on the topic of HTML5 Canvas, SVG, and D3 for data visualization. It was based on his onine book HTML Canvas Deep Dive. I found the teaching style well structured and engaging. I picked up a few tips on style that I adopted for my tutorial which I gave the following day.

It is amazing how far Javascript has come. D3 has brought data visualization to the point where every researcher (i.e. people who are producing data) should at least try to learn it. I have recommended it to my wife, who has only taken one programming course in her life and does not program on a regular basis. She will be my guinea pig to see if it’s easy enough for non-devs to pick up.

Here are the results of three exercises in the tutorial:

  1. Agricultural Productivity By State
  2. Bar Chart Using Canvas
    • A bar chart drawn with HTML5 Canvas
  3. Pie Chart
    • A Pie Chart drawn using SVG

OpenUI5

I attended a talk by some of the OpenUI5 team. OpenUI5 is an HTML5 framework developed by SAP for cross-platform/mobile development. It provides a large number of widgets and development tools that you can use for developing a cross-platform app. And the only dependency it has is a single Javascript file.

The things that I like about it are:

  1. Light-weight. Single JS include.
  2. Really nice looking controls and layouts.
  3. Apache License
  4. Backed by a big company (so it has a better chance of survival than some of the other little promising HTML UI kits out there).

More about this talk

AsciiDoc and AsciiDoctor

I attended a talk by Dan Allen on JRuby where he demonstrated some cool aspects of the language, compared its performance with MRI (the canonical Ruby) and shows some tips on making Ruby and Java work nicely together. Dan is one of the developers behind AsciiDoc which, until OSCON, I hadn’t been aware of. Asciidoc looks like an excellent tool for developing documentation and writing books. I have experimented with lots of solutions over the past several years in this space, including (but not limited to) Doxygen, TeX, DocBook, JSDoc, PHPDocumentor, restructured text, and, more recently, Markdown.

I will definitely be giving Asciidoc a go as it appears to provide the simplicity of Markdown with the power of DocBook. The fact that it is a format that is supported by O’Reilly for their authors, lends weight to its viability for arbitrary documentation projects.

Mirah

OK, there weren’t any talks on Mirah per-se, but the JRuby talk that I mentioned above reminded me of my unfinished netbeans module for Mirah. I ended up spending most of my evening hours of OSCON getting the Mirah module ready for release.

I fell in love with Mirah at first sight. It deserves a lot more attention than it is getting. Hopefully the Netbeans module will convince a Java developer or two to take a look. At the very least, it will enable me to start writing code in Mirah that I would otherwise write in Java. And nobody will be the wiser :)

My plans for Mirah center around Codename One. It is uniquely positioned to provide an alternate language for developing Codename One applications. I plan to use its macro ability to provide a sort of DSL specifically for removing the ceremony and boiler plate (inherent in Java) surrounding all of the common functions. I think I can improve productivity on CN1 apps by at least a factor of 2, and perhaps even more.

I’ll be posting more on that later.

Some Keynotes that You Should Watch

Andrew Sorensen : The Concert Programmer

This was really amazing to watch. This guy uses a special programming language to compose and sequence music. He codes up a pretty cool song right in front of your eyes.

Simon Wardley : Introduction to Value Chain Mapping

Simon demonstrates a really cool method to visually analyze and depict the value-chain in a company. I’m not really a management guy, but this technique looks like it could be applied to quite a few things. Watch it. You’ll learn something.

My Own Talk

Oh yeah. I led a tutorial on Codename One also. I’ll talk more about that in a separate post.

Why I don’t recommend using GoDaddy for Hosting

This has come up a few times now on the Xataface forum. Users write into the forum because their Xataface application has stopped working — or at least everything except the homepage has stopped working. They report errors like “Connection reset” in their browser, or simply a blank white page. Their PHP error log is empty, and their access log also is devoid of any evidence that any HTTP requests have even occurred.

The cause:

GoDaddy’s mod_security settings. In particular, their rule that any HTTP request where the query string begins with a hyphen (“-”) will be blocked. This causes lots of problems for Xataface because it uses a convention that many of the query parameters must begin with a hyphen.

Here is a simple test to see what I mean (if you use GoDaddy).

Create a simple PHP script named hello.php with the following contents:

<?php
echo "hello world";

Then copy it to your GoDaddy server and try to access this script in your web browser at

http://yourdomain.com/hello.php

It should display “hello world” as you expect. Now try adding a query string to the URL that begins with a hyphen:
e.g.

http://yourdomain.com/hello.php?-foo

This request will fail. Either your browser will hang or you will just receive a connection reset error.

From experience, if you try to contact GoDaddy support to explain the problem, they will suggest that it is a problem with your code and will deny that they are blocking requests. I have heard reports from 3 or 4 people who have gone through this process. Ultimately they end up having to find different hosting.

There are lots of good, low cost, shared hosting providers out there that provide good service without having to deal with intrusive filtering like this. Personally, I’ve had pretty good luck with Dream host, but there are countless out there that should work fine. I do understand the value in trying to block malicious HTTP requests, but there is nothing malicious about a hyphen.

GoDaddy may not be the only host with this restriction. I encourage you to try the hello world litmus test from this article on your host before getting to comfortable with them. If they are blocking requests like that, and are unwilling to remove the filter, you don’t want to waste your time with them.

Shape up, GoDaddy!

Make the World A Better Place by Making A Codename One Library

Codename One recently opened up a plugin repository on their website where you can download libraries to extend the functionality of Codename One. Right now it is really just a static webpage because the list of available modules is small. Let’s change that! I cannot think of an easier platform to develop modules for. There is a Netbeans project type that does all of the packaging work for you.

All you have to do is write your code and hit build. Then upload it to GitHub (or wherever you want to host it) and let the CN1 folks know that your module exists.

What Kinds of Modules Should I Build?

UI Components

The Codename One UI Component model is both simple and flexible. It is modelled loosely after Swing (e.g. Base “Component” class, a “Container” class which is a component that can contain other components, Label, Button, TextField, etc..) but they ditched a lot of the chaff and bloat to come up with a solution that light, simple and portable.

Creating a custom component is as simple as subclassing the Container class (or Component) and laying out some child components inside it. Once you have your custom class, you can package it up in as a cn1lib, and upload it to github. Don’t forget to write some javadocs and examples for it so people know how to use it.

I see requests in the mailing list all the time asking whether Codename One has a component that does X. The answer is often “yes”, but if the answer is no, I generally don’t hear anything else – even though the asker has likely gone off and implemented it themselves for their application. I saw a recent request for a “Color Chooser” component. Such a component would be relatively easy to create using the CN1 API. Create a dialog that gives a few slider controls to select RGB, provide some callbacks and public methods to be able to obtain the selected color, and you have a reusable color chooser component that the whole community can enjoy.

Data Processing

If you find yourself solving a problem that involves converting data from one format to another (e.g. extracting or compressing a ZIP file, MP3 Encoding, etc..) why not wrap it in a cn1lib so that others can use and improve your library.

One of the main strengths of Java was that you could almost always find a 3rd party library to do what you need. Because Java follows strict standards for name-spacing, documentation, and packaging (jar files) you could find, download, and learn how to integrate modules into your application without having to spend days looking through code to see how it works.

Having a healthy ecosystem of 3rd-party libraries is preferable to having one large monolithic core because it allows us to streamline our applications to our particular purposes and not be weighed down by features we don’t need. Many processing tools don’t belong in the CN1 core because they are too niche.

If you are developing a method to do something useful, think for a moment whether this method might be useful to someone else. You might have to make some changes to your design to decouple your library from the rest of your app – but this is a good idea anyways.

Native Functionality

Codename One is a cross-platform tool for mobile development. However it also allows you to tap into native functionality if needed by writing native interfaces (sort of like JNI, but easier to user IMO). Does your application target a platform on which you want to make use of some native libraries? Then you’ll have to write a native interface to access them. If you’re going to write a native interface anyways, why not write it in such a way that it can be reused in other projects.

The iOS, Android, and Windows Phone SDKs have tons of goodies. Usually I try to find a cross-platform solution for a requested feature, but in some cases your app will be better for making use of native APIs. If we all do our share, and publish native interfaces in cn1libs as we write them, we collectively make the Codename One ecosystem richer and more capable. And all of our apps will get better.

HTML5/Javascript Components

The big 3 platforms (iOS, Android, and Windows Phone) all support HTML5 inside the BrowserComponent in Codename One. This opens up a lot of possibilities for integrating HTML5 components into your codename one apps. There are countless Javascript/CSS libraries and components that can easily be ported to Codename One. You can even use the Codename One Javascript bridge to communicate between Java and Javascript.

A word of caution with including HTML inside your Codename One apps. HTML components are much more difficult to debug than Java components because you don’t get the nice stack trace. Generally, I find it best to debug as much HTML/Javascript as I can separately inside Chrome, then integrate it into CN1 with as few Java-Javascript dependencies as possible.

Port Existing Java Libraries

This type of library can be, arguably, the most useful for the CN1 ecosystem. You can’t just use a regular Jar with Codename One because it only supports a subset of the libraries in JavaSE. The Codename One class library includes basically the CLDC 1.1 less the javax.microedition.* classes. It also includes a number of other classes that have been added as needed (e.g. the Java Collection classes) in addition to a number of its own packages under the com.codename1.* namespace. You can check out the Codename One javadocs for a full list of supported classes.

For porting existing Java libraries to Codename One, I generally try to find a J2ME library that does what I want, since J2ME is closest to CLDC. I then go through all of the source files and look at the import sections (at the top of each source file) to see if any classes outside of the java.io, java.lang, and java.util packages are used. If I find any dependencies outside of these packages, I take a closer look to see how easy it would be to remove the dependency. Sometimes I can find another class in the Codename One SDK that will do something similar, or I might just disable some of the functionality of the library if it isn’t needed.

After removing all of the illegal dependencies, I create a new Codename One Library project in Netbeans, and copy the library’s source files into the project. Building this project will force compliance with the Codename One libraries. If the project builds successfully, then you will have a library that is compatible with all devices on Codename One.

Performance: J2ME libraries will generally use the Thread-safe collection classes (e.g. Hashtable and Vector) instead of their faster cousins (HashMap and ArrayList). Since Codename One supports the more modern collection classes, and these perform much better, I generally go through libraries that I have ported from J2ME and change all references of Vector and Hashtable to ArrayList and HashMap respectively. I also replace StringBuffer instances to StringBuilder for the same reason.

JavaSE Libraries: JavaSE libraries can be trickier to port because they often rely on many classes that fall outside of the Codename One SDK. If they make heavy use of AWT and Swing then you might have difficulty finding Codename One equivalents to maintain the functionality. You may have to take some time to understand how the library works so that you can figure out how best to substitute out the illegal class references. The recent release of the CN1Pisces library means that you should now be able to port over most Java2D code into Codename One by changing calls from Java2D into the equivalent Pisces or Codename One Graphics calls.

A Word about the cn1lib File Format

If you are a Java developer, you might be wondering why the CN1 guys didn’t just use jars to distribute libraries. There are two reasons:

  1. Libraries may need to package both java and native code together in a standard way so that they can include native implementations. The cn1format supports this, while jar does not have a standard way of doing this.

  2. Codename One supports a slightly different API than standard Java so, for any given jar file, it is unlikely that CN1 will work with it without some modifications. Having its own format helps to ensure that all code in a cn1lib file is actually supported in Codename One. In this way it serves as a sort of compliance test.

Summary

There are only so many hours in the day to develop cool things, so I would prefer to develop components that work on as many devices as possible. Codename One has provided us with a platform for building cross-platform mobile apps that doesn’t have the same performance sacrifices as other cross-platform solutions. The foundation is strong. The user interface is built on a high-performance, light-weight stack that takes advances of hardware accelerated graphics. All code is compiled to native code so it should run just as fast as native, and sometimes even faster.

This is a foundation that we can build on. So let’s take advantage of it.

Git Hub, Codename One Javascript Components , PDF CJK Fonts

I haven’t posted in a while, mostly because I haven’t had time. I just wanted to post a short update on my development activities over the past while. One big thing to note is that I have finally adopted Git as my primary source version control system. I had been using SVN for years, but once I started playing around with Git it was clear that I had to make the move. Git is much faster for checking out and committing. It also makes branching and merging much easier than with SVN. The most compelling feature of Git, though, is Git Hub. It provides a much easier way to share my code with the world, and it makes it a simple matter for others to contribute.

My Github page

I have created repositories for Xataface and many of its modules on GitHub. I have also moved SWeTE over.

PDF CJK Fonts

I recently developed a library for PDFClown to support Chinese, Korean, and Japanese fonts. This is a feature that is nowhere to be found in the world of open source PDF editors (iText is excluded from this list because of its AGPL license). Check out the source for this module on the github project page.

Codename One Javascript Components

I have also created a few Codename One wrappers for Javascript components. So far I have created a charting module, and a rich text editor module. Both use my Javascript bridge for the Java-Javascript communication. I created these, partly, as a proof of concept. Codename One sits at a unique junction between Java, Native, and Javascript. The ideal situation is to have as much as possible implemented in pure Java so that it is available across all platforms (JME, BlackBerry, Windows Phone, Android, and iOS), but the fact is that there are thousands of mature, well-supported Javascript libraries that can nicely complement the codename one toolbox. Javascript components should be compatible with the big 3: WinPhone, Android, and iOS.. Any platform that supports a native browser component.

One issue that I have faced is that the Codename One build server currently doesn’t retain package structure for resources on iOS and WinPhone. All resources are just flattened into the topmost directory of the application bundle. This is a pretty big problem for Javascript components since most will consist of a directory of resources including javascript, CSS, HTML, and image files, and they depend on their relative package structure.

The problem has been reported in issue 809. There are two possible current workarounds:

  1. Use offline builds for iOS. These support package structures for resources.
  2. Try to embed everything into a single HTML file, and then embed this file as a String in a Java class. I used this strategy in the Charts module – everything is embedded into a package private Resources class.

CFIB Study on Municipal Spending Incorrect and Misleading

In the last few days, CKNW has been touting a new study by the CFIB (Canadian Federation of Independent Business) claiming that spending in the four largest cities in Canada is growing out of control. Specifically (in Vancouver) they claim that spending from 2000 to 2011 grew by 50% while population only increased by 15%.

Since this is only comparing 3 metrics (spending, inflation, and population growth), it should be easy to check out their numbers. Based on the City of Vancouver financial statements and the reported inflation in Canada, I find that their numbers on spending are both misleading and WAY off.

We have reports from 2000 to 2012, but to align with their study, I’ll talk about the changes from 2000 to 2011 only. Accounting for inflation, there has been an increase in spending of 40% from 2000 to 2011 (736 million in 2000; 1.03 billion in 2011 (in 2000 dollars)). This is lower than the 50% increase claimed in the study.

Further, the data shows that spending in 2011 and 2012 is actually lower than it was in 2008 (adjusted for inflation). If they want to complain about spending being out of control, they should look to the years 2000 to 2008 when spending increased by 55% adjusted for inflation. Since 2008, spending has decreased by 8 percent!

Of course, that would make this study 5 years too late!

I have created a spreadsheet with my calculations.

Sources

  1. Vancouver City Financial Reports
  2. Canada Historical Inflation
  3. CFIB Study

Edits

Initially I had stated that spending was up 24% from 2000 to 2011. Actually it is up 40%. That doesn’t change the fact, however, that spending is down since 2008.

Object Persistence in Codename One

One very useful feature of Codename One is its Storage class, which provides a cross-platform key-value store that can be used to store simple data (e.g. Strings, Integers, Doubles), large binary data (e.g. byte arrays of encoded movies or images), and custom data types (i.e. your own objects). Storage is not shared between applications so it is like your own persistent Hashtable that allows you to store anything you might need for your application.

The API is simple. It provides methods to read objects, write objects, delete objects, check for object existence, and listing objects that are currently stored in storage. All lookups are based on key-value lookups.

A simple example:

Storage s = Storage.getInstance();

// Save the "Hello World" string to storage
s.writeObject("mystring", "Hello World");

// Read my "Hello World" string back from storage
String hello = (String)s.readObject("mystring");

// Delete my string from storage
s.deleteStorageFile("mystring");

Just as we stored a String, we could have stored a Vector of Strings, or a Hashtable of key-value pairs, or a tree of Vectors and nested Hashtables. The only caveat is that the Vectors and Hashtables can only contain objects that can be externalized.

What Can Be Externalized?

I don’t have a definitive list of what can be externalized in Codename One, but in general, you can externalize:

  • Primitive types (e.g. int, float, long, double, byte, etc..)
  • Arrays of primitive types (e.g. int[], float[], long[], double[], etc..)
  • Strings
  • Vectors
  • Hashtables
  • Objects implementing the Externalizable interface.

Hence, if you want to save your own custom objects in Storage, you need to implement the Externalizable interface. It is worth noting that you can’t simply implement the Serializable interface as you do in regular java. You need to implement Codename One’s externlizable interface that explicitly defines how to read and write the objects to/from a DataOutputStream/DataInputStream. This is due to the fact that Codename One doesn’t support reflection. In addition to implementing the Externalizable interface, you also need to register your class with CodenameOne (via the Util.register() method) so that it knows which class to use when deserializing your objects.

Saving Custom Types to Storage

As mentioned above, any object that you want to persist to Storage must implement the Externalizable interface. If you try to save objects that don’t implement this interface it will raise an exception. If you, subsequently try to read an object that hasn’t been registered with Codename One via the Util.register() method, then Storage.readObject() will simply fail silently and return null. This will occur, if any object in the graph that you are trying to read is not registered.

The Externalizable interface requires 4 methods:

  1. getVersion() – This should return the version of your object. This will be used to record the version of the object when it is written to storage. This value will be passed to your internalize() method when you read the object so that you can handle old serialization structures properly when you modify your class.
  2. getObjectId() – This should return a unique String ID for the class (not the object as the method name seems to indicate). This should match the id that is registered with Util.register() so that it knows which class to instantiate when loading objects from a DataInputStream. But you could use anything here, as long as you use the same ID in the Util.register() method.
  3. externalize() – This method should write your object to a DataOutputStream.
  4. internalize() – This method should read your object from a DataInputStream.

Your class should also include a public constructor that takes no arguments.

Example Class:

Let’s look at a simple example class for a user profile.

class Profile {
    public String firstName, lastName;
    public int age;
    public List<String> emails = new Vector<String>();
}

Note that I’m making all of the members public for simplicity and to reduce code in this example. Normally you would probably make the members private and implement setter/getter methods to access them.

Now, let’s implement the Externalizable interface on this class:

class Profile implements Externalizable {
    public String firstName, lastName;
    public int age;
    public List<String> emails = new Vector<String>();
    
    
    @Override
    public int getVersion() {
        return 1;
    }

    @Override
    public void externalize(DataOutputStream out) throws IOException {
        
        Util.writeUTF(firstName, out);
        Util.writeUTF(lastName, out);
        out.writeInt(age);
        Util.writeObject(emails, out);
    }

    @Override
    public void internalize(int version, DataInputStream in) throws IOException {
        firstName = Util.readUTF(in);
        lastName = Util.readUTF(in);
        age = in.readInt();
        emails = (List<String>)Util.readObject(in);
    }

    @Override
    public String getObjectId() {
        return "Profile";
    }

}

I want to comment on a few things here that are important:

  1. The order in which we read members from the DataInputStream in the internalize() method must be exactly the same as the order in which we write them in the externalize() method.
  2. We use Util.writeUTF() to write Strings instead of the DataOutputStream’s writeUTF() method, because it handles null values. I.e. if you try to pass a null string to the DataOutputStream’s writeUTF() method, it will throw a NullPointerException.
  3. We use Util.readObject() Util.writeObject() for writing objects (like the Vector containing email addresses). DataInputStream/DataOutputStream don’t provide equivalents.

Reading and Writing Profiles

Now let’s test out our class:

Profile steve = new Profile();
steve.firstName = "Steve";
steve.lastName = "Hannah"

Storage s = Storage.getInstance();
s.writeObject("steve", steve);

Profile newSteve = (Profile)s.readObject("steve");

System.out.println("Profile first name : "+newSteve.firstName);

If you try to run this example you’ll get a NullPointerException when you try to access the firstName property of newSteve. If you retrace your steps, you’ll find that the object was written OK (you can use the Storage.listEntries() method to see what keys are stored in storage). It’s just that the line:

Profile newSteve = (Profile)s.readObject("steve");

returns null. This is because we forgot to register our Profile class with Util, so it didn’t know which class to use for deserialization. If we add the line:

Util.register("Profile", Profile.class);

at any point before we try to read the object from storage, then it will work as expected. This is a big gotcha.

Tip: If you are getting null values out of storage, you should make sure that you have registered classes for ALL objects that are being read, including nested objects.

Where to Place Registration Code?

I’m still sorting out where the best place is to store the code that registers a class with Util. Here are a few options:

  1. Explicitly register all classes that you are using inside your application’s controller (e.g. in the start() method). If your application is self contained, this may be the simplest way. However, if your app may be using classes from external libraries, you may find it difficult to identify all of the possible classes that you may need to retrieve from storage.
  2. Inside a static block for the class that implements the Externalizable interface. e.g.

    public class Profile implements Externalizable {
        static {
            Util.register("Profile", Profile.class);
        }
       …
    }
    

    This will work, if you have referenced the class from somewhere inside your code before you unserialize the object. However, if your object is nested and its class is not referenced directly from code, then this static block may not be run before the object is deserialized (which will result in readObject() returning null).

    For example, we might load a Vector of Profiles like this:

    Vector v = Storage.getInstance().readObject("profiles");
    

    If we don’t explicitly reference Profile in our code, here, then this will fail (silently) because the Profile class will not be registered yet. However, if we first reference the Profile class, it will work. E.g.

    Profile p = new Profile();
    Vector v = Storage.getInstance().readObject("profiles");
    

  3. Other ideas? You can place the registration code anywhere you like. You just have to be aware of when/if your registration will be run vs when your objects are likely to be read from Storage.

Trouble Shooting

During my experiments with Storage, I ran across a few "gotchas" that you should watch out for:

  1. Object keys cannot be "paths". I.e., don’t include the "/" character in the key for write/readObject or you may get some unexpected results. E.g.
    Don’t do:

    byte[] b = new byte[]{'a','b','c'};
    Storage.getInstance().writeObject("bytes/foobar", b);
    

    Or you will get an error like:

    java.io.FileNotFoundException: /Users/shannah/.cn1/bytes/foobar (Not a directory)
    

    I’m not sure if this is a bug, but it’s something to watch out for. You can use any other character you want. Just don’t use a slash!

  2. Make sure ALL objects in your hierarchy that you are saving implement the Externalizable interface (or are supported natively by CN1 eg. Vector, Hashtable, etc..)
  3. Make sure you have run Util.register() for ALL classes that will be read from storage before trying to read them from storage. If you do not do this, Storage.readObject() will fail silently, returning null.
  4. Use Util.writeUTF()/Util.readUTF() when writing/reading Strings in your externalize() method. Don’t use out.writeUTF() because this will throw a NullPointerException for null strings.
  5. Use Util.writeObject()/Util.readObject() for writing/reading objects inside your externalize()/internalize() methods. There is no equivalent in DataOutputStream/DataInputStream.

Two Months with the Nexus 7

I have now had the Nexus 7 for a little under two months so I thought I’d post a mini-review of the product.

I purchased the Nexus 7 because I needed a newer Android test device for my development and I had heard good things about it.

The Good

- It is very light. Compared to my first generation iPad, it is a joy to hold while I read the web in bed.

- Easy and intuitive setup. The interface of Android 4.2 seems to be much improved over my previous android device (2.2). Installing apps and switching between apps is fast and efficient.

- Development is easy. Compared to my iPhone where I had to jump through an endless series of hoops, just to get my own apps running on my device, the Nexus 7 was as simple as plug it in and go.

- The price. At $199 you can’t complain.

The Bad

- Battery life. My first generation iPad can sit on the table unused for weeks, and still have a full battery charge. With the Nexus 7, I can put it in my bag with a full charge, and it will be empty the next time I want to use it 3 days later.

- Scrolling. Compared to my first gen iPad (which is 3-year-old technology), the scrolling is jerky and sluggish in most apps. It seems that Android is still playing catch-up on the usability scale.

- Soft keyboard. Once again, comparing to my iPhone and 1st gen iPad, the keyboard is more difficult to use on the Nexus 7. The autocorrect is frequently wrong, and it doesn’t seem to get as good at guessing which key I intended to type.

The Case For Light-Weight UI Toolkits

This post is motivated by a recent Reddit thread where someone posted an announcement about the 1.0 release of Codename One, a toolkit for building cross-platform mobile applications. I was quite surprised by the stream of negativity in the responses from developers who had never tried the framework, but who mistakenly assumed that they knew everything about it because they have tried other cross-platform toolkits in the past.

Before I discuss the thread itself, I just want to take a minute to talk about light-weight UIs and why they are important.

Light-Weight UI vs Heavy-Weight UI

When people refer to a light-weight user interface toolkit, they are generally referring to a toolkit where all of the widgets and components are drawn using graphics primitives using the toolkit itself. A heavy-weight user interface, on the other hand, is one where the components from the underlying platform are just placed in the user interface, and aren’t drawn manually. Swing, JavaFX, HTML5, QT, and Codename One are examples of light-weight UI toolkits. AWT, SWT, and Appcelerator Titanium are examples of heavy-weight UI toolkits.

Why Are Light-Weight UI Toolkits Important for Cross-Platform Development?

When you are developing for multiple platforms, then a heavy-weight UI will be one that addresses the lowest common denominator. If every platform has a “Text box” widget, then you can safely add a text box to your UI, and the correct widget will be shown on the current platform. But if one of the platforms doesn’t have a text box widget, you need to either say that “text boxes aren’t supported on platform X”, or you have to come up with an alternative for that platform.

With a light-weight UI, every platform can have a text box because you don’t depend on the underlying platform for the actual widget. This opens quite a bit of flexibility.

It is kind of like the difference between creating art work with stickers vs painting the art directly onto canvas. Imagine you are teaching an art class via a webcast and you have 5 different art students each with their own toolkit. These toolkits consist of a set of stickers and stamps that they have brought from their own collection, and a paint set. For the “heavy-weight” portion of the class, you would be instructing the students to place stickers onto the canvas. E.g. You might say “Now place a sticker of a dog in the top right corner”. But what if some of the students don’t have a sticker of a dog? Then you might say, “If you don’t have a dog, just use a cat. And if you don’t have a cat, then just leave the space blank”.

At the end of the lesson, the art work produced by each student would be radically different. The stickers would be different, might be different sizes, and some canvases might be completely blank if the student didn’t have the appropriate sticker.

The alternative “light-weight” lesson would just involve some paint brushes and paints. Imagine that students were able to replicate your painting perfectly. So if you paint a dog in the top corner, they will be able to paint an identical dog in the top corner of their canvas.

At the end of the lesson, then, every canvas will look more-or-less identical. At some point, you might actually tell the students to draw something different in a section that reflects their local culture. This would be fine also, and the variation would be reflected in the finished product.

This is basically the difference between a heavy-weight and a light-weight UI toolkit.

The mobile development space is getting very fragmented and form factors vary greatly. If any space needs a good platform for lightweight UI development, it is the mobile space.

Back to the Reddit Post

Responses were full of hate for Java, light-weight user interfaces, and the concept of cross-platform toolkits in general. One pervasive, yet misinformed, line of reasoning the came through in some of the frequent posters’ comments was as follows:

  1. Cross platform toolkits result in “lowest common denominator” applications that don’t take advantage of the features of any particular platform.

  2. Lightweight UIs won’t look or behave like a native on any platform so it is much better to use a heavyweight UI (i.e. wrap native components) so that the app at least looks and behaves natively.

  3. Lightweight UIs are slow! There are a million HTML5 solutions out there, but they are all laggy. It is better to just use a heavyweight UI.

  4. Cross-platform toolkits sound good at the start, but, inevitably you will hit a road block that will prevent you from achieving your goal, and have to resort to building native applications for each target platform, wasting, perhaps, a year or more of the time that you spent trying to use the cross-platform framework.

This line of reasoning appears to make sense if you don’t dig into some of the embedded assumptions. For example, #1 and #4 only hold true for heavy-weight toolkits, and #2 and #3 simply aren’t true. Let me address all of these points individually.

Do Cross-Platform Toolkits Result in a Lowest Common Denominator App?

If you develop a lightweight UI, then there is no reason why the resulting application should be the lowest common denominator of all platforms. This is because, in a lightweight UI, you are not dependent on the native widgets. You can embed native components in cases where it makes sense, but you are not limited by this. Any component in a lightweight UI can be used across all platforms, and configured to behave differently on each platform if it makes sense. Components can be created on top of a lightweight UI that don’t even exist in any of the underlying platforms. This adds a tremendous amount of flexibility and offers enormous potential.

Take, for example, the JFXtras project, which is built upon JavaFX, a light-weight UI framework for the desktop. It is a collection of components and widgets that are all light-weight (so they are truly cross platform) and they look and feel fantastic. If you wanted to develop this set of widgets using a heavyweight toolkit it would be 5 times the work, and would be impossible to maintain.

Are lightweight UIs doomed to look “un-native”?

While lightweight UIs give you the flexibility to create an app that doesn’t look native, you can come very close to a native look, if that is what you are trying to achieve. Light-weight UIs that are well designed enable you to develop themes that look and behave just like the native platform. The Swing has been doing this for years on the desktop and, while you may think that you can spot a Swing application a mile away, I am willing to bet that you have probably used many Swing applications without knowing it. Of course you can spot the ones that don’t look native – where the author didn’t place a priority on following native UI guidelines. But if you ran across one that did follow the guidelines, you would be none the wiser.

In my opinion, the Codename One folks have done a fantastic job of developing native looking themes for all of the main mobile platforms. I showed an app with the iOS theme to quite a number of savvy users and none of them could tell, at all, that it wasn’t actually using native widgets. All of the buttons look the same, and it behaved the same. This is a testament to the design acumen of their team.

And if you don’t like the look and feel, that is no problem. It is light-weight. You can override the look and behaviour of any component to conform to your preferences, if you like. Apply these changes across all platforms or only specific ones.

So, lightweight UIs certainly are not doomed to look un-native.

Are Lightweight UIs Slow?

Well, they can be, but so can heavyweight UIs. Performance depends on many factors, and if you have tried any of the HTML5 toolkits out there for building mobile apps you have probably noticed that these apps are indeed sluggish. So you might be tempted to apply the logic that since HTML5 apps are slow, and HTML5 is a lightweight UI toolkit, that all lightweight UI toolkits are slow. This is certainly incorrect. HTML5 requires very complex layout rules, and it relies on Javascript as its language that is quite a bit slower than a native executable would be. This is very hard for low-powered mobile devices to handle in a performant way. Ultimately as device performance improves (maybe in a couple years), the HTML5 performance problems will dissipate.

Codename One uses a different strategy that is much more similar to Swing than to HTML. It uses 2D drawing technology of the host platform to build its components (OpenGL on iOS, etc..) which is very fast and flexible. This allows for the creation of complex user interfaces without barely any performance hit.

So, no, lightweight UIs don’t have to be slow, and in Codename One’s case, it is not slow.

Are You Destined to Hit a Wall If You Use a Cross-Platform Framework?

I have been burned. You have probably been burned. I think everyone has been burned once or twice when they start a project with a toolkit, then sometime later, they hit a wall and the toolkit just can’t be used to take the next step. Is this inevitable with a Cross-Platform framework?

The answer is, maybe. But if you choose your framework carefully, they you shouldn’t have a problem. Some key items to look for would include:

  1. Is it open source? If it isn’t open source, then the chances are you will get stuck at some point and have to abandon it.

  2. Can you access native APIs if necessary? If you can’t access native APIs, then you will likely have to abandon it at some point.

  3. Is it built on a strong, robust, and fast foundation? If it isn’t, then you’ll likely have to abandon it at some point.

If the framework hits all three of these points, then you should be OK. If you need to access a new API on a particular platform, you can just write a native plugin for that. If you run into a bug or a limitation, you can fix it yourself, as long as it is open source. And as long as the core of the framework is strong and fast, you can build your own component libraries as time goes on to extend the platform, without worrying about breaking the platform.

Most HTML5/Javascript frameworks will fail on #3. HTML5 and Javascript just aren’t robust. There are many commercial cross-platform frameworks out there also. I would be careful with those.

In the few months that I have been working with Codename One, I have found the platform to be open and robust. If I find something that it doesn’t do, it is usually quite easy for me to add it myself. The fact that they allow you to write native plugins, wrap native components when necessary (to add to the UI), and develop my own libraries and components that run on top of it, give me confidence that there is no “wall” that I could hit in the future that would be a show-stopper.

Ramblings about Xataface, Java, and other software development issues