Category Archives: Xataface

New Mobile Theme for Xataface

During my two week “vacation” from Codename One, I’ve been madly working on a new project using Xataface. For this project, I really needed the mobile interface to be smooth, so I decided to finally make Xataface’s core theme responsive. Along the way, I also made numerous improvements to the flow of the UI especially in relation to sorting and filtering results. Before I go into detail about the new features, here are some screenshots of my app, which uses this new mobile theme.

The new login screen is much cleaner and mobile friendly.
The new login screen is much cleaner and mobile friendly.
The mobile registration form - fields generated based on the fields in the users table.
The mobile registration form – fields generated based on the fields in the users table.
The list view for the "News Feed" table.
The list view for the “News Feed” table.

Screen Shot 2020-08-30 at 7.21.34 AM

Let me unpack that above screenshot of the list view to highlight the various aspects you can see here.

  1. The “tables” menu is rendered along the bottom of the screen as tabs.Table tabs
  2. Sorting and filtering buttons are rendered at the top of the list. Sort and filter buttons When you scroll down the page, these buttons are converted into floating buttons: Floating sort and filter buttons
  3. Notice the Floating Action Button in the lower right for adding new records. Floating Action Button. By default this shows the “New” and “Delete” actions, but you can add your own actions to this menu using the “table_actions_menu” category.
  4. Notice that action icons below each row. These are rendered from the “list_row_actions” category. List row actions


Clicking on the “Sort” button displays a sheet with the various options available for sorting.The sort dialog. You can select which columns should be sortable in the fields.ini file using the new sortable directive.


Clicking on the “Filter” button displays a sheet with the various options available for filtering.

Screen Shot 2020-08-30 at 7.40.27 AM

This filter dialog is “live”. The button at the bottom that says “Show 977 Results” will dynamically update as you enter your query so that you can see how many results there will be.

Optional Search Header

On some tables you may want the header to be a “search” field. This can be achieved using the the new fieilds.inii “search_field_header” directive, as demonstrated in this table:

Search Header

New/Edit Record Forms

Forms are Xataface’s bread and butter, so they need to be very mobile friendly. I’ve completely revamped the stylesheet to be responsive so that forms are a pleasure to use on the smaller displays.

New Record Form

I’ve also added a new feature to help reduce clutter on forms. You can now make field groups “hidden” by default. Hidden field groups are collapsed into buttons that are rendered at the bottom of the form:

Fieldgroups menu

The user can display a hidden field group by clicking on the corresponding icon. E.g. In the example form shown above, the user might want to edit the “narration”. They can do so by clicking on the “Narration” icon at the bottom of the form, which will reveal the narration-related fields.

Narration field group

More to Come

This is just a quick post to share some of the work. There are tons of new features that I didn’t cover here. I’ll be blogging more about them soon.

I’ve been slowly assembling a “definitive” guide for Xataface. You can see the current version (in progress) at

After I’ve ported all of the existing documentation into this manual, I’ll be using it as the basis for a new website. There is lots of new stuff in the pipe for Xataface, so stay tuned.

Entitled OSS Users and the Xamarin RoboVM acquisition

RoboVM has been acquired by Xamarin, it was announced; and it would no longer be open source.


It only took five minutes for the forum posts and reddit threads to start up condemning the move as some sort of robbery. The RoboVM was accused of luring unsuspecting users into its community on the promise of open source, only to pull a switcheroo and sell out to big business. Some users were demanding the RoboVM team continue to share their work for free, because … that would only be fair.

RoboVM, on the other hand, explained that they had been open source for a few years and had received little to no contributions from the community, so there wasn’t much incentive to continue with that approach. My personal experience with managing open source projects is consistent with theirs. I released the first version of Xataface in 2005. In that time it has hundreds of thousands of downloads, and is still used in many enterprises as the back-bone of their web information systems (I don’t have an exact count since most apps built with Xataface are internal). In that time, I can count the number of community contributions on my fingers and toes. I’m thankful to all of the users who did contribute. But let’s be real, the case for open sourcing a project because the community will contribute is not compelling.

Shut up and Fork it!

No, really. The source (albeit a couple of months out of date) is still on GitHub and it is licensed under the GPL. That repository represents countless hours of high-quality work by incredibly skilled individuals. That is one hell of a contribution to the open source community. Let them move on; And if you want your open source RoboVM, you can build on this fantastic source base.

Personally, I think it is highly likely that the last open source version of RoboVM will continue to circulate for a long time to come. At least in its core as an AOT java VM, it should be maintainable by people on the outside because most of the heavy lifting is already done there. It is the value-added components like the iOS API bindings, and tool support, that will be difficult for the community to maintain going forward. These things are evolving too fast for volunteers to keep up with.

Dependent Tools

If you are an iOS developer who just uses RoboVM to build iOS apps in Java, then the move to close the source probably won’t affect you – except that your costs may be going up some. I wonder more about the impact that this has on other developer tools that have made RoboVM an integral part of their tool chain. I’m thinking about companies like Gluon that provides JavaFX support for iOS and Android. They use RoboVM for their iOS builds. DukeScript, which allows you to write Java apps with an HTML5 UI and deploy to iOS (and other platforms), also uses RoboVM for its iOS builds. How will they respond.

I had argued as recently as 6 months ago that we (at Codename One) should incorporate RoboVM into our toolchain rather than maintain our own Java VM. But we ultimately decided that there was too much risk in that approach because “what if RoboVM closes down, or goes closed source”. 20/20 hindsight shows that we made the right choice and our new iOS VM is now quite mature, performant, and robust. But most importantly we are not dependent upon other external factors for maintaining it.

What Open Source VMs are Left for iOS?

RoboVM wasn’t the only open source VM for iOS. It was just the most active, and provided the best and most comprehensive bindings to the iOS native APIs. But there are alternative VMs that the open source community may turn to for their supply chain. For example:

  1. Codename One (proper) – (Full disclosure, I work for Codename One)… Codename One is open source and provides a full cross-platform Java solution for write once run anywhere.
  2. Codename One’s VM – Codename One has developed its own Java VM for iOS that works as a cross-compiler from Java to C. This is open source and is a good option for Java tools that need a path to iOS.
  3. Avian – Avian is an AOT Java compiler that can be used to compile java directly to iOS binaries. It is written in C++, and has a very permissive license.
  4. XMLVM. This project has been discontinued. But I mention it for completeness in case people want to revive it.
  5. OpenJDK for iOS. It has been approved for an iOS port of the Open JDK to be developed. This may also present a long-term option, but this is still only in the planning stage.
  6. J2ObjC – A transpiler that converts Java source code into Objective-C source code.
  7. JUniversal – Java source transpiler to C# and C++ that includes a runtime library to help with portability.

Using SOUNDEX and MySQL Full-Text Search for Fuzzy Matching

I recently received a question from a Xataface user about how to support searches for misspelled names. Out of the box, Xataface provides some fairly advanced search support that allows users to search on particular columns, across all columns in a single table, or across all tables (from a specified subset of the tables in the database). For the single-table searches, Xataface uses combinations of LIKE and = comparisons depending on the type of query and field type. E.g. If a user enters a search in the last_name column of the people table for “Williams”, it will perform an SQL query like the following, under the hood:

SELECT * FROM `people` where `last_name` LIKE 'Williams'

Note: This isn’t exactly how Xataface generates a query like this. This is simplified for readability.

This works great, but what if we wanted to be able to find results of misspellings of “Williams”. E.g. If the user entered a search for ‘Wlliams’, we still want the same result set to be found.

Soundex is a phonetic algorithm for indexing names by sound, as pronounced in English. It happens to provide a very simple way to search for misspellings. Both PHP and MySQL include a SOUNDEX hashing function that will take string input and produce the SOUNDEX code for that input. The SOUNDEX code for strings that are misspelled are often the same. E.g.

SOUNDEX('Williams') === 'W452'


SOUNDEX('Wlliams') === 'W452'

This means that we can allow users to search for misspellings on the last_name field by modifying the SQL query as follows:

SELECT * FROM `people` where SOUNDEX(`last_name`) = SOUNDEX('Wlliams')

This should match a superset of the records that were matched in the original query. You might verbalize this query as “Find all people with last names that sound like Williams.

Matching Records that Contain A Word

The above example works great if we are searching on a column that contains a single word and we are matching against the whole word, but it falls short in cases where we need to find values that contain a search term. E.g. Suppose the people table also has a bio field that contains a short profile of the person. Now suppose we want to find all people whose bio contains the word “cardiac”. Xataface might generate an SQL query like:

SELECT * FROM `people` where `bio` like '%cardiac%'

Now if we want to find misspellings, we can’t simply do:

SELECT * FROM `people` where SOUNDEX(`bio`) like SOUNDEX('cardiac')

MySQL doesn’t provide a simple way to perform string splitting and mapping inside a WHERE clause so we would need to do some work prior to the search to allow us to search for misspellings in the bio field. One possible solution create an index on the bio field that includes soundex codes for each word. E.g. we could add a field named bio_soundex to the people table. Then every time the bio is changed we could update the bio_soundex. In Xataface we could perform this in the beforeSave trigger:

function beforeSave(Dataface_Record $record){
   if ( $record->valueChanged('bio') ){
      $words = preg_split(
      $soundexWords = array();
      foreach ( $words as $word){
         $soundexWords[] = soundex($word);
      $record->setValue('bio_soundex', implode(' ', $soundexWords));

This example makes use of Xataface specific hooks and classes, but the concept of splitting the field content into individual words, producing the soundex code for each word, then saving the concatenation of these codes (separated by spaces) into another field can easily be done in any system.

Now, we are able to perform a query like this:

SELECT * FROM `people` WHERE `bio_soundex` LIKE CONCAT('%',SOUNDEX('crdiac'),'%')

And this will return all records where the bio contains the word “cardiac” (and many misspellings thereof).

Currently Xataface doesn’t automatically perform this type of soundex conversion for single-table searches. This example shows how you can do it yourself. However, as I describe next, Xataface does perform automatic soundex indexing for multi-table searches.

Fuzzy Matching in Multi-Table Search

Xataface has supported a multi-table search option for several years now. It works by maintaining a central index table with an entry for each record that can be searched. This index stores record details such as its table, id, URL, title, description, and searchable text which has a MySQL full-text search index that is used for the actual searching. Using MySQL’s full-text indexing feature allows Xataface to perform relevance sorting. This index is automatically updated each time a record is saved, and Xataface provides administrators with tools to manually clear and regenerate the index.

If a user enters a search for “Cardiac” in the multi-table search box, Xataface will generate an SQL query similar to:

  match(searchable_text) AGAINST ('Cardiac') as `relevance`
FROM dataface__index
WHERE `lang`='en' AND 
match(searchable_text) against ('Cardiac')
ORDER BY `relevance` DESC

In order to modify the index to support misspelled searches, I used a strategy similar to the one described above (for the bio field). At the time a record is indexed, I parse the content of the searchable_text into words, then produce a string of concatenated soundex codes corresponding to those words, and append them to searchable text field.

The actual code is as follows (From here:

$searchable_text = preg_replace('/<[^>]*?>/', ' ', $searchable_text);
$words = preg_split('/((^\p{P}+)|(\p{P}*\s+\p{P}*)|(\p{P}+$))/', $searchable_text, -1, PREG_SPLIT_NO_EMPTY);
$soundexAddons = array();
foreach ( $words as $word ){
    $soundexAddons[] = soundex(trim($word));

$searchable_text .= '[/////]: '.implode(' ', $soundexAddons);
$searchable_text = strip_tags($searchable_text);

$sql = "
    replace into dataface__index 
if ( !@xf_db_query($sql, df_db()) ){
    if ( !xf_db_query($sql, df_db()) ){
        trigger_error(xf_db_error(df_db()), E_USER_ERROR);

At the time of the search, I perform a similar transformation to the search phrase to append the soundex codes of all words in the query to the end of the query. E.g. If the user searches for “Crdiac”, xataface will actually search for “Crdiac C632” (C632 is the soundex code for Cardiac).

This snippet shows how this is accomplished (From here:

$words = explode(' ', $query['-search']);
$soundexAddons = array();
foreach ( $words as $word){
    $soundexAddons[] = soundex($word);
$orig_search = $query['-search'];
$query['-search'] .= ' '.implode(' ', $soundexAddons);

So the resulting SQL query will be something like:

  match(searchable_text) AGAINST ('Cardiac C632') as `relevance`
FROM dataface__index
WHERE `lang`='en' AND 
match(searchable_text) against ('Cardiac C632')
ORDER BY `relevance` DESC

More Information

Why I don’t recommend using GoDaddy for Hosting

This has come up a few times now on the Xataface forum. Users write into the forum because their Xataface application has stopped working — or at least everything except the homepage has stopped working. They report errors like “Connection reset” in their browser, or simply a blank white page. Their PHP error log is empty, and their access log also is devoid of any evidence that any HTTP requests have even occurred.

The cause:

GoDaddy’s mod_security settings. In particular, their rule that any HTTP request where the query string begins with a hyphen (“-“) will be blocked. This causes lots of problems for Xataface because it uses a convention that many of the query parameters must begin with a hyphen.

Here is a simple test to see what I mean (if you use GoDaddy).

Create a simple PHP script named hello.php with the following contents:

echo "hello world";

Then copy it to your GoDaddy server and try to access this script in your web browser at

It should display “hello world” as you expect. Now try adding a query string to the URL that begins with a hyphen:

This request will fail. Either your browser will hang or you will just receive a connection reset error.

From experience, if you try to contact GoDaddy support to explain the problem, they will suggest that it is a problem with your code and will deny that they are blocking requests. I have heard reports from 3 or 4 people who have gone through this process. Ultimately they end up having to find different hosting.

There are lots of good, low cost, shared hosting providers out there that provide good service without having to deal with intrusive filtering like this. Personally, I’ve had pretty good luck with Dream host, but there are countless out there that should work fine. I do understand the value in trying to block malicious HTTP requests, but there is nothing malicious about a hyphen.

GoDaddy may not be the only host with this restriction. I encourage you to try the hello world litmus test from this article on your host before getting to comfortable with them. If they are blocking requests like that, and are unwilling to remove the filter, you don’t want to waste your time with them.

Shape up, GoDaddy!

Git Hub, Codename One Javascript Components , PDF CJK Fonts

I haven’t posted in a while, mostly because I haven’t had time. I just wanted to post a short update on my development activities over the past while. One big thing to note is that I have finally adopted Git as my primary source version control system. I had been using SVN for years, but once I started playing around with Git it was clear that I had to make the move. Git is much faster for checking out and committing. It also makes branching and merging much easier than with SVN. The most compelling feature of Git, though, is Git Hub. It provides a much easier way to share my code with the world, and it makes it a simple matter for others to contribute.

My Github page

I have created repositories for Xataface and many of its modules on GitHub. I have also moved SWeTE over.


I recently developed a library for PDFClown to support Chinese, Korean, and Japanese fonts. This is a feature that is nowhere to be found in the world of open source PDF editors (iText is excluded from this list because of its AGPL license). Check out the source for this module on the github project page.

Codename One Javascript Components

I have also created a few Codename One wrappers for Javascript components. So far I have created a charting module, and a rich text editor module. Both use my Javascript bridge for the Java-Javascript communication. I created these, partly, as a proof of concept. Codename One sits at a unique junction between Java, Native, and Javascript. The ideal situation is to have as much as possible implemented in pure Java so that it is available across all platforms (JME, BlackBerry, Windows Phone, Android, and iOS), but the fact is that there are thousands of mature, well-supported Javascript libraries that can nicely complement the codename one toolbox. Javascript components should be compatible with the big 3: WinPhone, Android, and iOS.. Any platform that supports a native browser component.

One issue that I have faced is that the Codename One build server currently doesn’t retain package structure for resources on iOS and WinPhone. All resources are just flattened into the topmost directory of the application bundle. This is a pretty big problem for Javascript components since most will consist of a directory of resources including javascript, CSS, HTML, and image files, and they depend on their relative package structure.

The problem has been reported in issue 809. There are two possible current workarounds:

  1. Use offline builds for iOS. These support package structures for resources.
  2. Try to embed everything into a single HTML file, and then embed this file as a String in a Java class. I used this strategy in the Charts module – everything is embedded into a package private Resources class.

Scrolling iFrame on iOS

I just finished adding scrolling support to the RecordDialog class in Xataface for the iPad. The RecordDialog basically allows you to open an edit form or new record form in an iframe using a Javascript call. This is useful if you want to be able to pop up a form without the user having to navigate away from the page. It uses an iFrame for legacy reasons, until the forms API can be updated to work 100% through AJAX. This component has worked well for a long time on the desktop, and it works okay on the iPad and iPhone if the form fits inside the iframe without having to scroll. The trouble is that you can’t scroll an iframe on iOS, so if the form is too long, it just gets cut off and the user can’t see the bottom of the form.

After some Google searching I found a few strategies for overcoming this issue. The proposed solutions can be categorized into 3 groups, and all of the solutions have one common element: start by making the iframe unscrollable using the scrolling=”no” HTML attribute. From there, you can either:

  1. Set the iframe height to be as tall as the document body, and wrap the iframe in a scrollable div tag.
  2. Wrap the contents of the iframe in a scrollable div tag. (This requires sizing the DIV’s height to be the same as the iframe’s parent’s inner height).
  3. Use javascript touchStart and touchMove events to do scrolling with Javascript inside the iframe.

After experimenting with options #1 and #3, I settled on solution #2: wrapping the iframe’s contents because it seems to work best for all occasions. Of course this solution won’t work if you don’t own the contents of the iframe.

Option #1 (wrapping the iframe in a scrollable div) is problematic because, by changing the size of the iframe to be taller than the screen, potentially, it screws up calculations in the body of the iframe that rely on window height. I found that dialogs that are created to be displayed in the middle of the page end up displaying way down the page because the window is deemed to be effectively the size of the iframe, which has been artificially resized to be way too big.

Option #3 didn’t look as appealing because of the amount of custom javascript handling. I just have the feeling that I would have been starting down a long road of browser incompatibility glitches.


  • A good blog post on some of these strategies
  • A stack overflow conversation on the topic.
  • JavaOne: A Look Back

    I am now home from my Bay area adventure at Java One. This was the first conference I’ve attended since 2005, and is by far the largest conference I’ve ever been a part of. I was blown away by the scale of this one. The downtown core was plastered with Oracle and Java signs. Buses dipped in oracle insignias roamed the streets constantly, and you couldn’t walk a block without seeing at least 5 people brandishing Oracle swag. (I quite liked the back pack that they provided).

    In addition to the sessions (which I’ll get to next), there was a steady stream of entertainment events for the benefit of conference attendees. Union square (and a few other locations) were blocked off for music performances; and then there was the “fan appreciation” event featuring Pearl Jam and Kings of Leon. Doesn’t sound like nerd conference, does it?

    The Sessions

    My reason for attending was the sessions. From Sunday to Thursday my schedule was jam packed with talks about topics ranging from JavaFX to MongoDB and a smattering of Java ME, iOS, and Android development. For any given one hour slot, I had to choose between 2 or 3 talks that I really wanted to see, and another 8 or 9 that would have been interesting. Being able to hear about new technologies directly from the people who are behind it is awesome. It imparts a kind of understanding that cannot be achieved through a book or reading blog posts.

    This conference has sparked a fire in me to seek out more conferences like this, similar to the way that a my first exposure to Disneyland led me to look for other parks that offer the same experience. In fact, the way a child reacts to his first experience at Disneyland is an accurate way to describe my reaction to the conference schedule.

    The Topics

    Currently I have a number of desktop applications that have been developed in Swing and have been deployed as applications on OS X. Therefore, I was most interested in learning about Swing’s successor, JavaFX, and how to deploy applications on OS X. The conference was heavy on JavaFX (yay!) but a little light on OS X. There were two talks on OS X deployment, but sadly I could only attend the first one (which way a fantastic talk, by Scott Kovatch). Nonetheless, I have left the conference with stockpiles of new ammunition for building applications with JavaFX. In fact, I’m really exciting about using JavaFX and Scene Builder (the GUI editor for JavaFX) to start building some business applications at work.

    The demos provided by some of the community served as proof of concept to me that JavaFX is something special. E.g. Gerrit Grunwald’s (@hansolo_) set of gauges in the JFXtras project show what can be achieved with a little time and artistic flair.

    On the OS X front, it looks like we’re all set to go now with the Mac App store. The javafxpackager tool provides the ability to easily deploy applications as native bundles. This is a new direction for Java (up until now, they have been recommending Web Start distribution), and I think it is a good one.

    Another topic that I am keenly interested in is mobile development (iOS and Android). Last year’s java one included a demonstration of JavaFX running on an iPad, but then nothing more was heard. This year they announced that JavaFX will be available for Java7 SE Embedded, and they had a conference scheduler application running on an small, embedded device in a kiosk at various stations in the conference. But there were no announcements about JavaFX for iPad or Android. The announcement of its availability on embedded devices is a step in the right direction, though, as this is a prerequisite for running on the major embedded platforms.

    I did attend one talk (actually 2 talks on the same topic) on Java for iOS and Android. The CodenameOne project (that I’ve blogged about before) is a java library and framework for building cross-platform mobile apps (i.e. deployable on Android, iOS, Blackberry, and J2ME). This looks very promising.

    When I had reached my saturation point on JavaFX, I wandered across to the Parc55 building for a couple of Java Enterprise Edition talks. I attended a talk on the new Project Avatar, which provides a thin-server approach to web applications (i.e. most view logic and layout is handled on the client in javascript, and the server just sends over the data). This project looks quite interesting, both in concept and execution. The concept coincides with much of the more recent development I’ve been doing with Xataface.

    The last talk of the conference that I attended was on NoSQL and scaling strategies. There, Nike’s director of engineering discussed some of their architectures for handling heavy traffic with NoSQL databases and Data grids.

    My head is full. My notebook is full. Now to put all of this new knowledge into practice.


    I have been battling with some performance issues on one of my Xataface applications of late and I think I just found the cause of periodic slow-downs:


    The output of this command includes useful information about all of the tables in the database including such things as creation time, update time, average row length, and number of rows. In Xataface I primarily use this command to:

    1. Find the modification time of tables so that I can perform smarter caching operations.
    2. Determine if a view exists.

    When working exclusively on MyISAM tables, this command is very fast as all of the information returned is cached all the time. However, when we start to throw InnoDB tables (and possibly views .. haven’t looked into it yet) into the mix this command becomes quite slow because much of the data returned needs to be calculated (e.g. the number of rows). I was facing an issue where this command could take upwards of 10 seconds to return when the application hasn’t been used in a while. It would also periodically hang even when the application was in frequent use. Presumably this is because MySQL does some caching of the values in this command, but the cache doesn’t last long.

    In addition, InnoDB doesn’t keep track of modification times so despite the fact that it is performing calculations in this command, it still returns NULL for table update times. Which renders the function altogether useless for InnoDB tables.

    Xataface has long had a back-up strategy for keeping modification times in InnoDB tables. It keeps its own table of modification times. This table is updated whenever a record is updated from within Xataface. It doesn’t work for updates performed outside of the application. In most cases this is good enough. Even with this back-up solution, the primary method of retrieving table modification times was still the “SHOW TABLE STATUS” mysql command.

    Solution #1: Use the Information Schema

    My first attempt to rectify the involved a direct query of the INFORMATION_SCHEMA. You can obtain the modification times of all tables in the database with the following query:

    select TABLE_NAME as Name, UPDATE_TIME as Update_time from information_schema.tables where TABLE_SCHEMA='my_database_name'

    Unfortunately, it turns out that this query is also quite slow (though a bit faster than show table status). My initial tests showed that on a database with about 60 tables it would take about 0.2 seconds to return. Far to slow for an operation that doesn’t contribute directly to the building of the page. What I need is something that returns in less than 0.01 seconds so that it is effectively negligible.

    Solution #2: Use SHOW TABLES

    SHOW TABLES simply returns a list of the tables in the current database. It doesn’t include any stats or information about those tables other than the table names. It is also very fast (generally returns in 0.00 seconds …. i.e. too small to matter). This is enough information to build my own modification times or check for the existence of a table/view.


    Survey Builder 0.1 Released

    I have just released Survey Builder on Sourceforge. Survey Builder is an application that allows you to design and host surveys using pure HTML. It is the first application built with the new Xataface 2.0 look and feel.

    Check out the sourceforge project at
    Check out the documentation at
    Check out a sample survey at