Category Archives: Javascript

Adobe CQ5 Developer Training

I just spent the past week in a developer training course for Adobe Communiqué 5.4 – a content management system on steroids. I thought I’d jot down some of my thoughts while they’re fresh in my mind.

CQ5 is a Java based CMS that is built around the JCR-283 (Java Content Repository) spec which essentially defines a sophisticated object database that is indexed by Lucene for easy searching and cross-referencing of objects. CQ5’s JCR implementation is called CRX, but there is also an open source reference implementation named Apache Jackrabbit if you have an allergy to commercial software.

It is not entirely correct to call the JCR an object database as it isn’t used to store Java objects directly – but the fact that it defines a tree of nodes and that all content is stored and accessed in a hierarchical fashion makes its use very similar to that of an object database. As such, it is natural to draw comparisons with Zope and its object database, the ZODB.


Zope, a python-based application framework, is radically different than the traditional relationship database model of web application development. The ability to store Python objects directly in the database and have them indexed solved many development problems, but it also created a few problems that would make maintenance of an ever-changing web application more difficult. Namely:

  1. When you make changes to a class, it can break all of the existing objects of that class in the database (you need to run a migration).
  2. If you try to load an object whose class definition can’t be found the system barfs.

This problem of class versions, managing upgrades of content types etc.. , was the single biggest problem with devleoping on Zope – and while I’m sure that there are best practices to work around this problem, I believe that the JCR solution of of storing content nodes but not actual objects is a much cleaner way of handling content.

The JCR stores a tree of content nodes, each of which have properties and their own child nodes. These structures translate well to different formats like XML (so you can dump entire branches of the repository as XML) and JSON – not so with a pure object database like the ZODB whose structures can be far more complex and include dependencies to classes. Data in the JCR can always be browsed independent of the component libraries which may be loaded into the system. You can browse the repository using WebDAV, the web-based content explorer that is built into CRX (the JCR implementation that is packaged with CQ5), or using CRXDE (the Eclipse-based development environment that is freely available to developers).

You can still define custom node types for your repository but this would merely dictate the name of the node type and perhaps which properties are required.

So, at first glance, this seems like a very stable base upon which to build web applications.

The Stack

The CQ5 stack looks like this:

  • WCM – The web content management layer consisting of a bunch of flashy UI components built using the ExtJS javascript library. (this part is proprietary).
  • Sling – HTTP server that makes it easy to read and write from the repository using HTTP requests. Very slick (this part is open source).
  • CRX – The content repository itself. Handles all permissions, storage, replication, etc… This part is proprietary. It performs the same function as Apache Jackrabbit, but includes a number of enterprise level improvements including a more powerful security model (I am told).

Author & Publish Deployment Instances

The recommended deployment is to have separate author and publish environments each running their own stack, and use the built-in replication feature to propagate authors’ changes to the publish instance whenever a piece of content is activated. This functionality, luckily, has been streamlined to hide most of the complexity. Workflow is built-in to allow you to activate each piece of content individually. Activation automatically triggers replication to the publish instance(s). This model seems to be very well suited to websites with few authors and many public viewers. It is scalable also, as you can add as many publish instances as you want to share the load.

This standard flow control (replicating changes from the author instance to the publish instances) leads me to wonder about cases where you do want the public to be able to interact with your site (e.g. through comments). We didn’t get into this scenario very much in the training, but, as I understand it, any content posted to the publish instance will go into an “outbox” for that instance that will be replicated to the author instances and await approval. They will then be re-replicated back to the publish instances once approved.

Security Model

The security model is quite different than that of most systems. Rather than having security attached to content types (because there are no content types) like with a relational database, or defining a large set of permissions corresponding to each possible action in the system as Zope does, security is 100% attached to the nodes themselves. Each node in the JCR includes an ACL (access control list) which maps only a small set of permissions to each user. There are only a few possible permissions that can be assigned or denied on each node. Basically it boils down to permission to read, write, delete, create new, set permissions, and get permissions on a node level. If there are no permissions assigned to a user on a particular node, then it will use permissions from the node’s parent.

One implication of this security model is that you must pay attention to the content hierarchy when developing applications. You cannot treat this like a relational database!

This is important. I suspect that many developers coming from a relational database background will be tempted to try merge the best of both worlds and try to create pseudo-content types in the system. After-all, all properties in the JCR are indexed, so you could easily just add a property called ‘contentType’ to your nodes to identify them as a particular content type, then build functionality that allows users to add instances of this content type. You could then create view templates that aggregate these content types to treat them as a table. You could do this, but you must be aware that you don’t have the same level of control that you have in a relational database system over what a user can do with your content types.

If you are querying the repository solely based on a property on a node – and not based on the path, then you may be surprised by the results that you obtain. At the very least, the JCR security model, despite appearing to be simple, is actually far more difficult to implement than its relational cousin – when trying to imitate the functionality of a relational database. You cannot control what properties are added to every node in the repository so querying based on property values may produce undesirable results. Instead you have to fully embrace the hierarchical model of data step very carefully when you try to import concepts from other paradigms as they could cause you to inadvertently introduce holes.

Custom Content Types (Sort of)

While CQ doesn’t have custom content types, it does allow you to map content nodes to a set of rendering scripts which produces something very much likc a content type. By setting the “sling:resourceType” property on a node to the path to a “component” that you develop, you can dictate where CQ looks for scripts that are used to render the node when requests are made. Components can be either “page” components, which represent an entire page, or regular components, which are included inside a page.

You can register page components to show up in the list of types of pages that can be added by authors when they add a new page to the system. Similarly you can register your regular components to show up in the “sidekick” (i.e. component palette) for authors when they are editing a page, so that it can be dragged onto a page. You can define which types of components are allowed to be parents or children of other components, and you can define which parts of site are allowed to have a particular component types added.

The Component Hierarchy

You can also define a “resourceSuperType” for components to allow them to inherit from other components in the system. This is handy for code reuse as there are hundreds or thousands of existing components that can be overridden or extended. We ran through several exercises creating and extending components. I’m satisfied that this process is not difficult and quite powerful.

Component Dialogs

A component without a dialog is really a lame duck. Users (especially authors) need to be able to interact with your components. E.g. if you create a photo album component, you need to allow your user to add photos to it. Adding dialogs is not difficult but I suspect that the development process is slated for improvements and more automation for future releases. The dialog forms are created entirely by creating appropriately named subtrees under your component’s node. E.g. you would create a child node of a particular type named “dialog”, which contains a child node named “items”, which contains a subnode named “tabs”, etc… 6 or 7 layers deep.

Each tab, each widget, each panel, is represented by a node in the repository. This is clever but somewhat tedious. It is like building a UI using only the UI hierarchy tree in the left panel of the IDE without the visual editor. I suspect that future versions will probably include a proper WYSIWYG UI editor for developing these dialogs but for now this manual system will have to do.

Despite the tediousness of the process, in the scheme of things it is still quite efficient. In only a few minutes you can produce a multi-tab, multi-field UI with rich widgets that allows your users to add and edit a myriad of content types on your site.

TestDisk a Nifty Utility for fixing drives with bad boot sectors

Just ran into an interesting problem with an external hard drive that was being used as a time machine backup for laptop. Someone tried to connect this drive to their windows machine and it evidently screwed up the boot bits so not only would windows not recognize it, Macs wouldn’t recognize the disk either.

Tried running it through Disk Utility but received a message saying “Disk cannot be repaired.”

So I loaded up TestDisk and took it for a spin. Here is a photo gallery outlining the steps that I took.

Adventures with KeepAlive Slowing down PHP

Disclaimer: To my regular readers (if there are any) this post is entirely technical in nature so if you’re not looking for the solution to this particular problem (e.g. from Google) you’re probably not interested in this post.

I run a few sites that use PHP as a reverse proxy for other sites. The PHP script loads a specific webpage, then sub-requests for resources like stylesheets, images, and javascripts, are also processed by the same PHP script. For most of these other resources the script simply passes the content through unchanged.


When a page is loaded that includes a large amount of supporting resources like CSS and javascript files, it seems to take a long time to load some of the CSS and javascript files. Some of the scripts were taking upwards of 15 seconds to load when loaded as part of the page (although if loaded on their own through the PHP script, they load instantly).

Short Term Solution

The investigation of the problem determined that the problem lies in the KeepAlive setting in Apache. Turning Keep Alive off fixes the issue and yields excellent load times consistently for all resources. But WHY??

Long Term Solution

Keep Alive is generally a good thing and it is supposed to boost performance. So why is it a bad thing in this context? (I really don’t know the answer yet). I would like to find a PHP solution to this problem that will work with KeepAlive, but I can only speculate at this point as to why this might be happening.

Possible reasons that I’ve considered:

  1. I need to do something more than “exit” at the end of the PHP script to declare that I’m done with the request – as Apache could be keeping the connection open for longer than necessary and preventing subsequent requests from happening.
  2. Perhaps there is some setting that I am unaware of in PHP that limits the number of simultaneous requests it can handle from the same client… (I find this unlikely – at least at the level that we’re talking about here).


I found that simply adding the ‘Connection:close’ header in the PHP script resolves this issue.
But i’m still not satisfied. This shouldn’t be necessary IMHO. If you set the Connection-Length properly then flush that content to the browser, there’s no reason why PHP should be making the browser wait around indefinitely.

So maybe we’re missing a few performance points here – but at least it’s not hanging like it was.

Why do you need an app for that?

I have recently stopped using the Facebook and Mail apps for my iPad and iPhone in favour of the HTML equivalents offered through Safari. The reasons: Mobile-optimized web applications are now as good, or even better in many cases, as their native app equivalents. In the case of the Mail app, I have found that the gmail mobile version has a superior search feature and is much faster in loading my messages. For facebook, I just found the app limited and more buggy than the HTML equivalent.

In general, I’ve come to the conclusion that if you’re going to build a native app for something, you’d better have a good reason – and good reasons are becoming fewer as HTML browsers improve their support for HTML5. The only valid reason at this point is a requirement for significant client-side processing – e.g. a 3-D game. But as WebGL matures even this will be quite possible in HTML.

So here are some reasons for developing your mobile applications in HTML5:

  1. Increased productivity in most cases over development of native apps.
  2. Real standards-based, cross-platform support so you can write it once and have it work on all major platforms (Android, iPhone, Blackberry, etc…).
  3. True freedom of deployment. You are not locked into Apple’s (or the next would-be gatekeeper’s) store or subject to their sovereign choice of what can and cannot be installed.

The remaining reasons why you may still need to develop a native app:

  1. You want to use a platform specific feature that isn’t available through the web api. E.g. the accelerometer, or the contacts list, or push notifications, GPS, etc..
  2. You need more client-side processing power than you can harness through the web.

The compelling reasons to want to develop a native app:

  1. To get it into the app store and maybe make some money.
  2. Toolkits (e.g. XCode/Interface builder) are developed and promoted specifically for making apps target a specific platform. This can make it seem easier to break into the market since there are lots of learning resources on how to use these tools.

The biggest challenge right now facing HTML mobile developers is that the tools are less promoted and more scattered. This is because the major stakeholders (e.g. Apple) have a significant interest in getting you to develop your app natively so that it will be exclusively available for their platform, and they will get a cut of any revenue through their store model. If you look, however, there are tools out there for building slick HTML mobile apps that look and feel very much like native apps. Some examples include:

  1. Sencha Touch
  2. jqTouch

And there are more where those came from. If you still want to develop a native app and you would prefer to work with open technologies like HTML and Javascript you may want to look at Appcelerator which provides tools to develop applications in Javascript, HTML, and CSS that can be compiled into native apps all the major smart phones (e.g. Android, iOS).

iPhone Web App Development: JQTouch vs Dashcode

I’ve been playing around with iPhone and iPad web app development for the past week or so (I’m working up to building a native app, as there are benefits to playing both sides of the field here). Given my familiarity with jQuery, I felt inclined to try out the jQTouch framework to start with. After some successes and failures, I then decided to take Apple’s Dashcode development environment for a spin. Here are my initial impressions:

My adventures began by modifying the demo jQTouch application to do what I wanted. I was impressed by the capabilities of this framework out of the box. It looks good on both the iPhone and the iPad – and feels just like a native app. The user can add the page to their “home screen” to load it up just like a native app also. My first stumbling block came when I faced the iPhone’s “no multi-tasking” limitation. If the user spent some time browsing to a particular place in my app, then left the app for any reason (e.g. to take a call or check their email), then they would be sent back to the start screen of the app and lose where they were altogether. I tried a number of workarounds the involved things like local storage, session storage, cookies, sessions and the like, but all were a little klunky. I realize that this problem will be fixed in iPhone 4 with the advent of multi-tasking, but many iPhone’s don’t qualify for this upgrade so it was important to me to at least be able to retain state.

Rather than hack jQTouch to pieces to try to implement this functionality (believe me, I tried), I decided to see what else was on the web app development landscape. That led me to Dashcode.

So I started the project more-or-less from scratch using Dashcode. I was impressed most by the Dashcode interface – how it allows you to drag and drop GUI widgets, or code snippets. It really is a model to follow if you want to develop an IDE. With Dashcode, you are almost completely insulated from the HTML markup. Instead you work with the visual WYSIWYG pallettes or with the javascript API to update the model. Rendering is done using MVC, and the performance of the resulting app is noticeably (on my older iPhone 3G) snappier than with jQTouch..

Unfortunately, I ended up hitting the same wall as with jQTouch: saving and restoring application state is a bit of a bitch. It involves more than just recording “where you are” in the app, because of the way the “back” buttons are set up, you have to recreate the entire environment, history and all. This is more than I really wanted to do right now.

So at this point I had 2 nearly complete, and almost identical apps. One using jQTouch. The other using Dashcode.

But now that I had climbed the learning curve in both of these frameworks, I knew I could do better. So I started, more-or-less, from scratch with a new project using jQTouch that will incorporate all that I had learned. I’ll be releasing the app in a few weeks.

Here is a brief comparison of these two ways of building iPhone/iPad apps:

  1. Dashcode produces an app that is a little snappier and a little more native (on the iPhone).
  2. jQTouch seemed to allow me more flexibility since I have direct control over the markup. In addition the development cycle seemed to be a bit faster using jQTouch. (However I’m sure you could improve the dev time on Dashcode if you used jQuery and other productivity libraries on the javascript side of things… I had been trying to use its API directly).
  3. Dashcode doesn’t seem to scale up to the iPad larger screen as easily as jQTouch. The extra flexibility in jQTouch with markup etc, makes me feel better equipped to customize the interface for the device.
  4. Dashcode is a fantastic IDE.

iPad: Initial impressions

Just picked up an iPad today and am very impressed – not only by its potential for changing the world, but for its immediate utility. Here are just a few things that I notice after playing with it for 4 or 5 hours:

  1. When reading, scrolling down a page is quite hard on the eyes (you get kind of car sick doing that too much). I notice this particularly because of the extensive use of side “sweeping” in iPad applications. E.g. the Comic book apps allow you to read a comic one page at a time, and then just flip the page to see the next cells. This is very easy on the eyes. The Financial Times app also makes use of side scrolling to read its news paper – less scrolling down – more sweeping across. This makes for a much more pleasent reading experience. Look to see much more of this style in the future as more devices like the iPad hit the market.

  2. It looks like there’s nothing for sale yet in iBooks (probably a delay because we’re in Canada). However its extensive selection of free titles (generally classics that have passed their copyright expiration dates) makes for lots of choice still. I started to read “The Rise and Fall of the Confederate Government” by Jefferson Davis (the president of the Southern Confederacy during the American Civil War) this afternoon to complement the a book I’m currently reading about Abraham Lincoln and his cabinet. The iBooks experience is almost as good as reading a real book. It makes it very easy to flip pages, change font size, and skip to any part of the book. I appreciate it’s little status message ever present at the bottom right corner indicating how many pages are left in the current chapter.

  3. The Kindle App is also quite good, and currently provides a much larger selection of books to purchase. (iBooks appeared to have zero books available for sale). The only problem is that they make it damn hard to purchase any books. The Kindle app has a button to “Shop for books from the store”. But this just takes you to the amazon web site which is not optimized in any way for mobile devices or the iPad. After logging in, entering credit card info, and finally finishing the purchase, it downloaded a book on iPad software development into Kindle. They really need to build the purchase mechanism into the iPad to make it competitive with iBooks (which allows you to simply click on a book and purchase it on the spot – and be reading it in seconds).

  4. Comics! The iPad is a savior for the fledgling comic industry. Reading comics on this thing feels very natural. I downloaded both the Marvel Comics app and the Transformers Comics app, which allow you to read comics in a rich, full-screen, environment. They offer some free comics and have others for sale. I purchased one issue of Transformers for $1.99 and it was as simple as one click and I’m reading it.

  5. News papers are now multi-media. I downloaded the free Financial Times app which is a UK news paper. Reading it was much like reading a news paper, giving you the same ability to read the articles in columns and scan to the right to see other stories. They have embedded videos associated with some articles, and tapping on the video displays the high quality video right inside the window (as expected, but still really cool). I think we’re just hitting the tip of the iceberg in our exploration of how different media can be melded together to enhance the user experience.

  6. HTML5 Support – After reading about iPad’s rich support for HTML 5 it occurs to me that Apple is doing the world a favour by taking a hard stand against browser extensions like Flash and Java (sad to say). The market share of the iPhone OS (estimated at over 318 million units between iphone, ipod, and ipad) is so big that Apple is able to single-handedly force us to endure a little pain for the greater good in the long run. HTML has been moving at a snails pace since it’s introduction and this has been a sort of chicken-and-egg problem, whereby developers don’t use any cutting edge features because of lack of browser support, and the standard hasn’t progressed very fast because there hasn’t been the demand for cutting-edge features from the development side. Apple is changing that with the powerful HTML 5 support in Safari for iPhone and iPad. Finally HTML/CSS/Javascript is at the point where you can create a full desktop application using open web standards. In addition this model for networked applications is much cleaner and safer than previous strategies (such as Java and Flash applets) because they truly run inside a sandbox where they cannot harm your computer.

The Apple war against browser extensions (Flash and Java applets) reminds me in some ways of the American civil war and the related end to slavery. Just as slavery was entrenched into the culture of the south to the point where they could not willingly let the practice go without experiencing major economic hardships, we, as computer users, have become dependent on proprietary browser plugins like flash to make up for features that ought to have been built into the browser to begin with. Just as it took a bloody war (or some southerners might say a Northern dictator) to force the issue on slavery, it takes a dictator from the South to force us into the future of computing for our own good, when we lacked the will to make the leap on our own.

I’m very excited to start developing HTML 5 applications, now that there is a reliable and stable platform to target: the iPad!

Replacing Scriptaculous/Prototype with jQuery

I have used Scriptaculous in the past to sprinkle little bits of UI magic into Xataface. Specifically, I have used it to add collapsible sections, sortable sections (via drag-and-drop), and sortable tables (also via drag and drop). These worked great! The Scriptaculous library was a bit bulky and it made the initial page load time a little bit longer, but the result was worth it.

Unfortunately I have started to run into problems with Scriptaculous interfering with other scripts on the page. Scriptaculous is built on the Prototype.js library which adds a number of handy methods and attributes to the built-in javascript types, like objects, arrays, DOM Elements, and strings. As a proof of concept, this is great as it shows off the dynamic features of the javascript programming language. However this can cause problems with scripts that count on the results of the default behavior of these built-in types.

For example, I have made use of Kevin van Zonneveld’s php.js library which provides pure javascript implementations of familiar PHP functions. One such function is count() which is supposed to return the number of elements in a PHP array. In Javascript, this function can either take objects or arrays as a parameter in order to provide the closest possible behavior to its PHP counterpart. Essentially, all this function does is count the number of elements in the array (or object) and return the result as an integer. Unfortunately, after including the Prototype.js library, all objects now have a number of default properties and methods whether you want them or not because they are added to Object.prototype. This effectively breaks the count() function and I can’t see a viable way to work around the problem other than removing Prototype.js from the mix.

Why does prototype.js break the count() function?

Take the following example:

var o = {0 : 'a', 1: 'b', 2: 'c'};
count(o); // should return 3 but with Prototype.js installed it returns 25

This returns the wrong result because Prototype.js adds a number of methods and properties to all objects in the system, so the count() function must count these also.

jQuery to the Rescue

Luckily there is another library that does everything that I have been using Scriptaculous/Prototype.js for: jQuery. It is leaner and less intrusive. It doesn’t change any of the underlying types and it still provides the drag-and-drop sorting of sections, and collapsing/expanding of sections. And in most cases it provided a cleaner, faster solution than was required with Scriptaculous.

Clickable URLs with javascript

A client wants all email addresses and urls to be clickable in an application. I would like to develop a javascript solution for this.

I found this article that shows some example code for doing this on a particular piece of text, however, fixing the links on an entire page is a little bit more involved as it would require us to only convert the URLs that are static text – and leave the URLs, already inside an <a> tag, or appearing as tag attributes, alone.

My Solution

 * A function that makes all static urls and email addresses in given DOM 
 * element's subtree to be converted to links.
 * @param el DOM Element that we are checking
 * This will recursively call itself until the entire subtree has been walked.
 * @author Steve Hannah .
 * Some parts were adapted from
function makeLinksClickable(el){

    //If no element is provided, then we will traverse the entire document.
    if ( el == null ) el = document.body;
    if ( el.nodeType == 3 /* text node */ ){
        // We only want to replace urls in text nodes
        var s = el.nodeValue;
        if ( s.replace(/^\s+|\s+$/g, '').length == 0) return;
            // If there is no text in this text node, we will just return and do
            // nothing.
        var hlink = /\b(ht|f)tp:\/\/([^ \,\;\:\!\)\(\"\'\< \>\f\n\r\t\v])+/g;
            // regex for URLs
        var mlink = /\b([a-zA-Z0-9_\.\-])+\@(([a-zA-Z0-9\-])+\.)+([a-zA-Z0-9]{2,4})+\b/g;
            // regex for e-mail addresses
            // Copied from
        var frag = document.createElement('span');
            // Because we are having difficulty putting just the A tag into
            // the document after conversion, we create a wrapper <span> tag
            // that will hold the resulting <a> tag.  Then we will 
            // insert the <span> tag into the document.

        // Replace URLs in this text node.
        var new_s = s.replace(hlink, 
                            //s = $0.substring(1,$0.length); 
                                // remove trailing dots, if any
                            s = $0;
                            while (s.length>0 && s.charAt(s.length-1)=='.') 
                                // add hlink
                            return " " +; 
                     ) ;
        // Replace Email addresses in this text node.
        new_s = new_s.replace(mlink,
                            //s = $0.substring(1,$0.length); 
                                // remove trailing dots, if any
                            s = $0;
                            while (s.length>0 && s.charAt(s.length-1)=='.') 
                                // add hlink
                            return " " +'mailto:'+s); 
                     ) ;
        if ( new_s != s ){
            // We only want to perform the switch if a change was made
            frag.innerHTML = new_s;
            var parent = el.parentNode;
            parent.replaceChild(frag, el);
    } else {
        // This is not a text node, so we will loop through all of its child
        // nodes and recursively change the urls and email addresses in them.
        for ( var i=0, len=el.childNodes.length; i<len ; i++ ){
            if ( !el.childNodes[i] ) continue;
                // If for some reason this node is not set we skip it
            if ( el.childNodes[i].tagName == 'A' ) continue;
                // We don't want to change anything that is already inside
                // an <a> tag
            if ( el.childNodes[i].tagName == 'SCRIPT' ) continue;
                // We leave scripts alone because we don't want to screw up
                // any javascripts.
                // Now we recursively call ourselves.


// Now we register this function to be called as soon as the document is finished
// loading.

But Safari has trouble with string.replace() callbacks:

Apparently Safari has some trouble with the string.replace() method when it comes to passing a function as the second parameter, as we have done in our solution above. See This post for information on how to beat this problem. Then we have a fully-workable, cross-browser solution for converting static URLs and email addresses to links using javascript.

dhtmlXGrid, open source but painfully restricted

I have been attempting to incorporate dhtmlxGrid into Dataface to spice up the editing of related data on web forms. Since Dataface is open source under GPL, any library that I choose to include must also be open source and compatible with GPL. dhtmlxGrid standard edition fits this criterion so I decided to take it for a spin.

Unfortunately the standard edition appears to be no more than a trial version of the software as it does not contain any ability to extract data from the grid (that I can see). Without this ability, it is impossible to actually save, submit, or send data from the grid to a database or a form. What good is a grid like this if it cannot be saved? All of the features that would enable saving data are reserved for the professional edition which has quite a restrictive commercial license which is, safe to say, not compatible with GPL.

I do however give them props for releasing the standard edition under GPL. They could have just released a trial version with no distribution rights at all. As it is, at least I can dig into the code and add the necessary functionality myself.

I am working with the Firefox Javascript debugger to try to see where the data is hidden. This will be a little time consuming, but the end result will be worth it, I think.