I’ve been putting a lot of work into Xataface getting it ready for release. There are still some details to take care of, but I think it’s safe to say that this release will be a drastic improvement over any of the previous releases.

Javascript is a wonderful language, but I always felt as if it wasn’t terribly scalable. Not so much in the performance sense, but in the component sense. When working on large projects (like Xataface) Javascript can seem a little too “scripty” to build components in. I’m always fighting between the two conflicting goals of library size and library functionality.

If you build a useful library in Javascript it is usually either comprehensive and large, or small and single-purpose. It is possible to build the library in pieces and just include the parts that you want, but then you end up with a cumbersome set of script includes in your HTML page every time you want to use your library’s functionality. This is error-prone as it is easy to forget which scripts you need to load for any particular function to work.

In server-side languages you don’t have this problem. When you write a script in C, PHP, Python, Java (or any other language), you can just include your dependencies at the top of your file using an import, include, or require statement (syntax depends on the language). Then using your library is as simple as including one file – and all of that file’s dependencies will automatically be imported as well.

To solve this problem, I developed a Javascript compiler as part of Xataface. Some features of this compiler include:

• Compiler directives to process dependencies.
• Javascript Compression – to minimize the footprint.
• Duplicate Dependency tracking (i.e. If multiple scripts all require the same script as a dependency, then the dependency will only be included once).
• Compiler directives to include CSS files.
• Compiler directives to import files as strings.
• Configurable source paths. (i.e. Just like with other languages, you can specify multiple paths where source files will be located).

This doesn’t sound like a lot, but these features really improve the Javascript development experience. It allows me to develop reusable components that can be shared and imported into many different projects without having to deal with the complexities of including script tags in an HTML page, or making sure that the proper CSS files are included. Once you have built a component, it is done. All you need to do in order to use it in your application, is include it using the //require directive.

Let’s look at an example…

All of these features are provided by the Dataface_JavscriptTool class in Xataface. So you begin by obtaining an instance of this, and adding the source path where you want it to look for Javascripts.

import('Dataface/JavascriptTool.php');
$jt = Dataface_JavascriptTool::getInstance();
$jt->addPath('/path/to/javascripts', 'http://www.example.com/url/to/javascripts');

At this point the compiler knows to look in the /path/to/javascripts directory for scripts. But you haven’t told it to include any particular script. That’s our next step:

$jt->import('myscript.js');

If you were write this code inside the context of a Xataface application, your myscript.js file would run because Xataface handles the rest of the rendering at the end of the body of any template extending the Dataface_Main_Template.html template (which is pretty much all templates).

Note that by default the following directories are included in your javascript paths:
– DATAFACE_SITE_PATH/js
– DATAFACE_PATH/modules/XataJax/js
– DATAFACE_PATH/js

So you are able to use any script in those directories without needing to call the addPath() method of the JavscriptTool.

A look at the Javascript

myscript.js:

alert('hello world');

This just pops up an alert. Let’s get to some best practices right away though. The general structure of a script usually goes as follows:

//require <myotherscript.js>
(function(){
    alert('hello world');
})();

Note the //require statement on the first line. This tells the compiler to include the script myotherscript.js in its place. The compiler will look first in the /path/to/javascripts directory, then in DATAFACE_SITE_URL/js, then in DATAFACE_URL/modules/XataJax/js, then in DATAFACE_URL/js to see if it can find a script by that name. If it cannot find one it will throw an exception with a stacktrace. If it finds it, it will include the contents of the script in place of the require statement.

Example: Requiring jQuery

Xataface comes with lots of libraries included in its source paths. jQuery is one that is used in almost every script:

//require <jquery.packed.js>
(function(){
    var $ = jQuery;
    alert('Hello world');
   
})();

Note that on the first line, we make a reference $ to jQuery. This is because in Xataface jQuery.noConflict() is set so that it doesn’t conflict with other libraries that want to use the $ variable. By declaring it inside our wrapping function(){}, we are able to still use the $ shorthand without polluting the global namespace.

Using jQuery UI

The following example is meant to show why we need to be able to bind CSS with your javascript libraries. We are going to use jQuery UI’s dialog to display our ‘Hello world’ message, and this requires a CSS file in order to look proper:

//require <jquery.packed.js>
//require <jquery-ui.min.js>
//require-css <jquery-ui/jquery-ui.css>
(function(){
    var $ = jQuery;
    var dlg = $('<div>').append('Hello World').appendTo('body');
    dlg.dialog();
})();

These examples are just the tip of the iceberg of possibilities that a Javascript compiler brings. More cool examples will be forthcoming as Xataface 2.0’s release nears.

I have read many obituaries for the Java applet. Once upon a time, in the late 1990’s the applet was king of the internet and hope for the future of rich web-based applications. Its decline and (almost) demise can be attributed to a number of factors, not the least of which is the fact that HTML and Javascript have improved to the point where most of the the functions that used to require an applet can now be performed directly in the web browser without requiring any third-party plugins. Nowadays, you scarcely run across any websites that use applets, and when you do, they generally stick out like a sore thumb. They are generally slow to load, and they almost always pop up with a security dialog asking you to approve their execution based on a signed certificate.

Signed Applets – Full Access or No Access

This clumsy security model is the other reason for the growing irrelevance of applets. An applet can only run in two different modes:

1. Run inside the “sandbox”. The applet cannot access any system information from the client computer, cannot access the file system, and cannot make any network connections to any hosts other than where the applet was loaded from.
2. Signed Applet – Full access. The applet runs with full permissions of the user. E.g. it can connect to any host, access any client system information, and access the hard drive. This level of access is very risky for the user especially if the applet is just running inside some random webpage that they are trying to open. The applet has the ability to delete the user’s hard drive or copy files from their hard drive without the user even knowing.

Because applet developers are so limited with the sandbox model, most developers end up distributing their applets as signed applets so that they can be run on clients’ computers with full access privileges. This works out great if the client fully trusts the applet developer – but that isn’t the typical client/developer relationship. Most users just click “okay” whenever these types of security dialogs pop up, so it becomes very easy for malicious developers to create trojans for unsuspecting users who browse to their web pages.

Since the proliferation of signed applets across the internet would result in a very dangerous browsing experience for the average internet user, we can be thankful that the web has opted for the applet-less direction that instead makes use of HTML and Javascript – which work inside a secure sandbox. Signed applets are now generally reserved for B2B applications and in-house apps where the client computer has a trusting relationship with the developer.

Unsigned Applets Are Still Safe

All that said, unsigned applets that run inside the applet sandbox are safe for a client to run since they don’t provide any access to the client’s computer. Unsigned applets’ fade from prominence is the natural result of improvements to Javascript and HTML. Javascript/HTML/CSS now have the tools required to create complex user interfaces with rich client interaction. It is no longer necessary to embed a Swing-based user interface inside the browser to give the user a rich experience. Further, it is inconvenient to work with a hybrid of technologies so the developers will often steer clear of applets entirely – opting for pure javascript solutions. For many years, applets were still necessary to accomplish things like drag-and-drop from the desktop (though that required a signed applet), or some more complex user interface widgets, but HTML 5 has introduced most of the tools to do all of these things inside Javascript natively. So the applet has been even further crowded out.

So What Are Applets Useful For Now?

The applet’s relevance may have been eroded down to a few grains of sand, but it turns out that some of those grains still shine like diamonds. Javascript can do many things well, but there are still a few items that Java can bring to the party. For example:

1. Javascript doesn’t handle processor intensive operations well. It runs in a single thread so any intensive operations will make the browser hang and possibly even crash. Java, on the other hand, is multi-threaded so it can perform parallel processing in the background and pass results back to the browser in way that doesn’t interfere with the user experience.
2. Java provides a rich, almost limitless set of libraries. – Hence you can find a library to do just about anything you need. The provides an alternative to making server-side requests for processing using AJAX. Instead requests can be passed to a “silent” applet that runs on the client computer – saving both network usage and server usage.

The “Silent” Applet

Observing the strengths of Java vs Javascript yields an interesting model for making use of applets in a web page. I call this the “silent” applet because it is designed to be completely transparent to both the Javascript developer, and the user. A silent applet is one that is loaded silently in the web page and does not manifest itself visually in the web page. Rather is runs invisibly along side the web page as a daemon waiting for requests from the web page which it processes and returns to the web page. This is similar to the AJAX model where the client sends background requests to the server, and the server returns a response which the client then processes. With this model, the request is not sent to the server at all. This saves both network and server resources since much more processing can take place on the client.

The silent applet is useful in situations where a significant amount of processing is required so that performing the “calculation” inside javascript is either impossible or would be disruptive to the user experience. This might include rendering a PDF page as an image, generating a complex chart, performing an image, video, or sound transformation, etc….

Example Silent Applet: Web PDF Renderer

I am developing a PDF reporting module for Xataface. I want the UI to allow the user to arrange database fields and content over top of an existing PDF that they upload as a means of creating report templates that can then be rendered as full reports later on. For this I need to be able to display the PDF as a background image in the user’s work space. I also need the user to be able to navigate through the PDF to different pages and to zoom in and out. The current state of Javascript and HTML is such that the rendering of PDFs needs to be relegated to a different system. Either I need to render the PDF pages as images on the server-side, or I need to find a way to do it in the client.

Rendering the PDFs on the server side would require me to install some server-side extensions or applications that can handle PDFs. There are many, but my goal is to keep the server requirements minimal so that it will work inside a standard LAMP install. So my preference is to find a way to render the PDF on the client side. There are various flash and Java applets already that display a PDF inside the browser, however I’m concerned with mixing flash or applets into the UI of my editor as it will likely result in painful, or intermittent conflicts down the road as the technologies choose not to place nicely together. So I decided to create a silent applet for rendering the PDF.

The Web PDF Renderer applet runs as a silent daemon in the background. There is a thin javascript API that can be used to ask the applet to render pages of a PDF at various sizes. When the processing is done, the applet calls a javascript callback to update an <img> tag in the user interface. This allows me to keep the entire user interface in standard HTML and manipulated via the DOM.

Using the Library

One example use of the library is to render the PDF to an HTML <img> tag. The following example creates a new PDFPage wrapper (which creates an img tag), then appends the image to the body of the document, and finally renders the PDF page. The render() method sends a signal to the applet to render the PDF page. When the applet is complete, it sends the data back the PDFPage object which updates the image source with the new data.

        // Short reference to PDFPage constructor
        var PDFPage = xataface.modules.pdfreports.PDFPage;
        
        // Create a new page  (first page of document)
        var page = new PDFPage({
            width: 800,
            url: 'test.pdf',
            page: 0     
        });
        
        // Append page’s <img < tag to the document body
        $(‘body’).append(page.el); 
        
        // Render the page  (done asynchronously)
        page.render();
    

Inside the Box: The implementation

We need to set up a mechanism for passing messages back and forth from Javascript to the applet and back. For javascript to java communication it is as easy as calling public methods defined on the applet directly. These are exposed and callable in Javascript. For the reverse, we make use JSObject which is available standard as part of the Java plugin. There is quite a bit of information on Java-Javascript communication on Oracle’s website.

We created a global queue that is used to pass the messages:

PDFPage.queue = [];

Note that is is just an empty array at this point.

Calling the render() method will pass a message to the applet – it essentially works as follows (this has been simplified for the example, but gets the point of strategy across):

function PDFPage_render(){
    
    PDFPage.queue.push(this);
    startDispatch();
        
}

So render a page involves just two things:
1. We push the PDFPage object onto the message queue so that the applet can access it.
2. We call startDispatch(). This function essentially tells the applet that the queue has been updated so it can start processing.

The startDispatch() function up-close:

function startDispatch(){
        var applet = $('applet[name="'+PDFPage.appletID+'"]').get(0);
        if ( !applet ){
             var attributes = {
                name: PDFPage.appletID,
                code:       "com.xataface.modules.pdfreports.PDFRendererApplet",
               codebase: PDFPage.codebase,
                archive:    "WebPDFRenderer.jar, commons-codec-1.5.jar, commons-logging-1.1.1.jar, icepdf-core.jar",
                width:      1,
                height:     1
            };
            var parameters = {
                startDispatch:"PDFPage.startDispatch()",      
                queue:"PDFPage.queue"
            }; 
            var version = "1.5"; 
            deployJava.runApplet(attributes, parameters, version);
            
            
        } else {
        
            try {
                applet.startDispatch();
            } catch (e){
                setTimeout(function(){
                    startDispatch();
                }, 1000);
            }
        }
    
    }

At its core this is just a wrapper around the applet’s startDispatch() method – but it needs to handle some edge cases to load the applet the first time it is called. If the applet hasn’t yet been added to the dom, it adds it and tries to call itself again. If the applet is there but not loaded yet, it waits 1 second and tries again.

The deployJava.run() method is from the standard Java deployment code available here. Notice that we pass 2 parameters to the applet when we load it:

1. We pass the javascript path to the queue so that the applet knows where to find its message queue.
2. We pass the startDispatch() method call so that the applet is able to call its own startDispatch() method through javascript.

Inside the applet:

The start() method just loads the parameters that we passed, then calls the startDispatch() method for the first time:

public void start(){
        
        queue = (JSObject)JSObject.getWindow(this).eval(this.getParameter("queue"));
        JSObject.getWindow(this).eval(this.getParameter("startDispatch"));
    }

The guts can be found in the startDispath() method:

public synchronized void startDispatch(){
        
        if ( running ) return;
        running = true;
        
        
        Thread dispatcher = new Thread(new Runnable(){

            public void run() {
                while (!stopDispatcher){
                    try {
                        JSObject next = null;
                        try {
                            next  = (JSObject)queue.call("shift", null);
                        } catch (Exception ex){
                            break;
                        }
                        
                        if ( next == null ) break;
                        
                        String pdfURL = (String)next.getMember("url");
                        if ( pdfURL == null ) break;
                        
                        
                        PDFRenderer renderer = new PDFRenderer();
                        URL baseURL = PDFRendererApplet.this.getDocumentBase();
                        String baseURLStr = baseURL.toString();
                        int queryPos = baseURLStr.indexOf("?");
                        if ( queryPos >= 0 ){
                            baseURLStr = baseURLStr.substring(0, queryPos);
                        }
                        
                        if ( !baseURLStr.endsWith("/") ){
                            int lastSlashPos = baseURLStr.lastIndexOf("/");
                            baseURLStr = baseURLStr.substring(0, lastSlashPos+1);
                        }
                        if ( pdfURL.indexOf(":") < 0 ){
                            if ( pdfURL.indexOf("/") == 0 ){
                                pdfURL = baseURL.getProtocol()+"://"+baseURL.getHost()+
                                        (baseURL.getPort()>0?(":"+baseURL.getPort()):"")+pdfURL;
                            } else {
                                pdfURL = baseURLStr+pdfURL;
                            }
                        }
                        
                        System.out.println("PDF URL is "+pdfURL);
                        renderer.setPDFURL(new URL(pdfURL));
                        
                        String req = null;
                        try {
                            req =  (String)next.getMember("request");
                        } catch ( Exception ex){
                            
                        }
                        
                        if ( "numPages".equals(req) ){
                            next.call("update", new Object[]{renderer.getNumPages()});
                        } else {

                            int page = ((Number)next.getMember("page")).intValue();
                            int width = ((Number)next.getMember("width")).intValue();


                            renderer.setWidth(width);
                            renderer.setPage(page);
                            BufferedImage img = renderer.getResult();
                            ByteArrayOutputStream os = new ByteArrayOutputStream();
                            ImageIO.write(img, "png", os);
                            os.flush();

                            String encodedImage = new Base64().encodeToString(os.toByteArray());
                            os.close();

                            next.call("update", new Object[]{encodedImage});
                            
                        }
                        //System.out.println(encodedImage);
                        
                        
                        
                        
                    } catch (Exception ex) {
                        //Logger.getLogger(PDFRendererApplet.class.getName()).log(Level.SEVERE, null, ex);
                        ex.printStackTrace(System.out);
                    } 
                    
                    
                    
                    
                    
                }
                running = false;
            }
            
        });
        dispatcher.start();

See full applet source code

Edit: In newer versions of Java there are additional security restrictions on network requests when the method call is initiated from Javascript. Therefore is is necessary to wrap the startDispatch() method in an AccessController.doPrivileged() call (Solution found here). To accommodate this I renamed the startDispatch() method as int_startDispatch() and I created a new startDispatch() method as follows:

public synchronized void startDispatch(){
        AccessController.doPrivileged(
                
                new PrivilegedAction(){
                    public String run(){
                        int_startDispatch();
                        return "";
                    }
                }
            
            
        );
        
    }

This ensures that we won’t run into security issues when loading the PDFs.

Basically this spawns a thread the runs a loop. In each iteration, a message is loaded from the queue. It gets the PDF’s URL, and other information about what is being requested. When it is done, it calls the update() method of the original message which is a javascript method. It this method is responsible for adding the image back to the DOM.

See a demo of the Web PDF Renderer