Why Pyjamas Isn’t a Good Framework for Web Apps

Earlier this week, I stated that Pyjamas no longer seems like a viable solution for Grafpad (or many other web-apps for that matter). In this post, I will explain the flaws with Pyjamas that ultimately made me decide to switch away from it. I’m aware that Pyjamas project is currently getting an overhaul, and I hope that these flaws get addressed in the upcoming Pyjamas releases. Before I go any further into bashing Pyjamas, I want to mention that I’ve been using Pyjamas for several years, writing over 20,000 lines of Python code that runs inside the browser (as well as several Pyjamas wrappers for extending its functionality). I appreciate the problem Pyjamas is trying to solve, and I definitely think it’s a useful tool. Perhaps one day Pyjamas will be good enough for the browser, unfortuantely it has a lot of issues to solve before that’s the case.

Experienced JavaScript developers might already be familiar with many of the points I will bring up. To summarize Pyjamas’ flaws in one sentence, it basically assumes that JavaScript is still a joke of a language it was several years ago and tries to apply outdated solutions that don’t scale well. Today’s JavaScript, however, can run circles around its predecessor, both in terms of performance and functionality. Many innovative design patterns have also been posted for keeping JavaScript code clean and object-oriented. In some ways, JavaScript has even surpassed Python in terms of design, which still lacks proper private variables, for example. So what are some of the big offenders in Pyjamas?

Browser Detection instead of Feature Detection

Many of you are probably familiar with Pyjamas’ compilation scheme. If not, it basically creates multiple versions of the JavaScript code, one for each major browser (IE, Firefox, Safari/Chrome, Opera) and serves the appropriate one depending on your user-agent string. A quick Google search will reveal thousands of pages explaining the problems with this technique (called browser detection), so there is really no point for me to go into much detail here. The first problem with browser detection is that we assume that the user will be using one of the browsers we’re detecting (sorry Konqueror). The second problem is that we’re assuming the user is using one of the versions of this browser that still has the same issues/functionality. I’ve already posted about the changes I had to make in Pyjamas to make it use IE9 properly, which has full canvas support, yet Pyjamas still treats it like IE6 (ironically, IE9 actually behaves more like WebKit than IE6). The third problem is that many browsers spoof the user-agent string, pretending to be a different browser (for various reasons). These browsers may support features that the spoofed browser doesn’t support and vice versa, forcing us to use an unnecessary work-around for a feature that the browser supports natively (just like IE9 being forced to use VML instead of canvas), or preventing the feature from working altogether (imagine if Chrome, with no VML support, spoofed IE6 user-agent string).

Bloat and Boilerplate Hell

If you’ve peeked at Grafpad’s JavaScript, you probably saw 80,000 lines of code in a 3.5MB file. But did you know that the pre-compiled version of Grafpad front-end is only about 8,000 lines of code? We have 10 times the needed code just to pretend like we’re still using Python. What’s worse, most of that code is only there to support obscure Python functionality most of us are never going to use in a web-app anyway. Pyjamas has become the most complete Python framework for the browser, unfortunately it has also become the most bloated one, with most other frameworks (such as py2js) only needing to generate 1.5 lines of JavaScript for each line of Python code. You can see the 80/20 principle at work here, where 20% of Python’s features account for 80% of Pyjamas’ boilerplate. In my opinion, it would make a lot more sense to only support the commonly used features of Python, allowing the user to rewrite the bits that don’t work well for JavaScript. After all, the most tedious things to port between languages are the algorithms, not the object structure.

Debugging

In theory, Pyjamas is much easier to debug than JavaScript. Unlike JavaScript, which either throws vague errors or worse yet, silently fails a block of code and continues execution like nothing happened, Pyjamas throws Pythonic exceptions, which most of the time do a very good job pinpointing the exact line that caused the problem… at least when you run your program through Pyjamas Desktop. The problem is, Pyjamas Desktop has been broken for almost 3 years now, requiring you to either use a 3-year old Linux distribution (last known version to have support for python-hulahop) or rely on WebKit or MSXML implementations, neither of which supports canvas.

Alternatively, you can debug your code directly in the browser. Pyjamas sports a good set of Python exceptions emulated in JavaScript through clever use of try/catch blocks. Unfortunately, this alternative not only lacks proper stack trace, but also the original code (and compiling your Pyjamas app with any of the debug modes doesn’t solve this, regardless of what various outdated posts on the mailing list claim). Needless to say, the errors raised by Pyjamas in the browser are not very useful. If you made an IndexError on line 50 of your code by referencing “object.array[5]“, for example, expect Pyjamas to throw some weird error (that’s right, chances are it won’t even realize it’s an IndexError – or at least won’t report it well, the except blocks seem to work correctly) on line 30,000 of your compiled JavaScript, which will reference $p['getattr'](object, 'array').__getitem__(5), among a bunch of other boilerplate which could have caused an error as a result of an earlier error in your code or a Pyjamas bug. Even when debugging using Pyjamas Desktop, the browser errors can occasionally be inconsistent with normal Python (usually due to a bug in Pyjamas), and it’s a pain to troubleshoot these. And there is really not much Pyjamas can do to remedy this, in my opinion.

Adding additional assertions to catch every possible case to throw Pythonic errors is a fool’s errand no different than trying to parse HTML using regular expressions. Python’s ability to throw relevant assertions stems from its fundamental design. It’s very strict about using non-existing/undefined variables and comparison of irrelevant types. JavaScript, on the other hand, is very lazy/permissive about these, much like Perl. Python is proactive about its assertions, Pyjamas tries to be reactive. It’s unrealistic to forsee every special case that could arise and account for it with an assertion the same way Python would. Even if you manage to do so, you will have added even more code to Pyjamas’ already large chunk of boilerplate (not to mention potential for new Pyjamas bugs). One option is to compile these assertions away when the debug flag isn’t set, but even then you would be doing the exercise of examining all possible errors that Python could throw in each case, plugging in more “reactive” logic to make JavaScript work the same way. Instead, we should make the framework easy to debug in the environment it’s meant to be in. Since we can’t make JavaScript behave like Python, and we can’t do compile-time debugging like we would with C++ or Java, we should make the output easy to understand, so that we can map it back to the original code.

Python is not Java, DOM is not a Desktop

This brings me to my next point. GWT (the original inspiration for Pyjamas) might be more bloated than Pyjamas, but there is something it can do that Pyjamas can’t: compile-time error catching. If it wasn’t for Python being a dynamically-typed language, a lot of my rant in the previous section about debugging would be irrelevant. Additionally, I don’t feel that Pyjamas is approaching the problem from the right angle. Python has the advantage of being much more similar to JavaScript than Java ever will, and a lot of Pyjamas’ wrapper logic wouldn’t even be necessary if Pyjamas didn’t try to pretend to be GWT (in addition to pretending to be Python). GWT was designed to make web development similar to Desktop GUI development, since that’s the background many Java developers come from. What other purpose is there to fake MouseListener and KeyboardListener in an environment that wasn’t designed to need either (KeyboardListener, by the way, is another source of grief for Pyjamas – it’s what makes the keyboard pop-up all over the place on mobile devices, it also attaches a fake input element to the current element, pretending like they’re the same element, adding even more boilerplate and wrappers to the code)? What other purpose is there to build the entire DOM dynamically (which, by the way, is also extremely inefficient)? The browser page was not designed to function the same way your Desktop calculator app does. Anyone who has taken a few minutes to learn how the DOM works probably agrees that it’s actually superior to the old-fashioned Desktop way of writing the GUI. I’m lazy (otherwise I wouldn’t have written my front-end in Python), so when a new technology comes along that clearly makes my life easier, why ignore it?

If it wasn’t for trying to fake a Desktop GUI, Pyjamas wouldn’t need all these wrappers. Most other Python-faking frameworks allow one to invoke JavaScript logic as if it was a regular Python object/function. Pyjamas, on the other hand, requires one to first write a wrapper for Pyjamas Desktop using Python, then for the browser using some limbo version of Python/JavaScript hybrid (where you can’t even access elements of array using standard indexing), and finally rewrite a separate version of your limbo code for each non-compliant browser (definitely IE, and possibly some others). This wrapping might have been necessary in Java, but should not be needed for Python at all, and could have been prevented with better design. But wait, there IS an alternative! You can put raw JavaScript in your code using JS() method and passing it one giant string of JavaScript code. Unfortunately, that chunk of the code will get completely ignored in Pyjamas Desktop (which you’re using to debug your entire app, since the browser debugger is no help at all), and to actually reference anything from this chunk of code in the browser, you will need to reference these variables the same way: “a = JS(‘a’)” (again, don’t expect “a” to get set in Pyjamas Desktop). Oh, and don’t try to modify any of the DOM elements created by Pyjamas from anything other than Pyjamas, you will run into object state sync issues. Pyjamas wraps each DOM element in a Python object, which then stores the element’s state as a set of variables, and assumes it doesn’t change without Pyjamas’ permission. Pyjamas plays well with other JavaScript frameworks… as long as they don’t touch any portion of the DOM Pyjamas uses.

JavaScript has its Strengths

JavaScript might not be the cleanest language, and I still much prefer Python to it. But I must give it credit where credit is due. First of all, it integrates the DOM into itself really well. I can take any DOM element, assign a function to onMouseDown event as if it was a regular JavaScript object, and all of a sudden I got an element that reacts to my mouse clicks. No need for complicated ClickHandlers.

Pyjamas has a lot of abstraction layers, both to hide JavaScript inconsistencies, and make it easier to build widgets. However, native JavaScript libraries, like jQuery, do a much better job at both. Yes, jQuery doesn’t scale well for larger projects, but there are libraries that do, like MooTools (which, by the way, was inspired by Python). But realistically, if you create a simple wrapper for generating classes (or loot one from John Resig’s blog – the same guy who wrote jQuery), even jQuery becomes good enough for creating large projects. Pyjamas, on the other hand, adds so much abstraction, that sometimes I need hacks just to manipulate the DOM. If you look at the DOM of a typical Pyjamas app, you will notice layers of unnecessary elements: images wrapped in divs, wrapped in more divs, placed inside some table that resides inside yet another table. When I try to render my app on a tablet, it often crashes due to the DOM bloat.

Pyjamas also assumes that JavaScript is slow, which was true when the project first started. As a result, it duplicates parts of its boilerplate code to avoid an extra function call (while adding excessive function calls and abstractions in other places). Ironically, JavaScript engines have come a long way, and a lot of Pyjamas’ optimizations are no longer relevant (such as using object["property"] instead of object.property). In fact, a quick paint app I wrote in Pyjamas actually runs faster in Chrome than Pyjamas Desktop. That same app runs faster still when written in pure JavaScript. It’s especially noticeable when using the paint-bucket tool, which works by pixel-scanning and takes a couple seconds in Pyjamas yet almost instantaneous in JavaScript.

Summary

While Pyjamas is the most complete Python emulation in a browser, it has become a very bloated and brittle framework. It doesn’t embrace any portion of JavaScript, nor the DOM, trying to hide them away like some sort of deformed beast. By pretending to be pure Python, it not only puts unrealistic expecations on itself, but also fails to make use of good parts of JavaScript. Instead, Pyjamas embraces a solution designed for a statically-typed language, favoring a GUI structure that should have died a decade ago.

So What’s The Alternative?

I did mention that I am porting Grafpad away from Pyjamas. However, I’m not crazy enough to rewrite the entire project in pure JavaScript. Rewriting all the code in a language with different quirks and troubleshooting differences like division rounding and modulo signage is not my idea of fun. I also still prefer to keep my front-end code interchangeable with the back-end (more or less), which has already provided multiple advantages, such as moving the proprietary clipping and recognition algorithms to the back-end in just a few hours of work. I happen to have another ace up my sleeve. In the next post, I will review multiple alternatives for Pyjamas and explain the solution I’ve chosen.

Touch Gestures in Pyjamas

Before I continue, I wanted to apologize for the lack of updates. To pay the bills, I got a new job in October, which took away a lot of my free time. You also may be familiar with the current state of Pyjamas from the mailing list. Before I go further, I wanted to mention that I have realized Pyjamas is not the best API for Grafpad in November of last year, and have been working on alternative solutions (I will write up a detailed post explaining this soon). However, for those still using Pyjamas, I wanted to provide my solution for handling touch events on mobile devices.

Some of you probably noticed that Grafpad has had support for iPad/Android devices for almost a year now despite Pyjamas still lacking support for touch gestures. Some of you may have even seen the communication between Luke and me where I was trying to figure out how to write a proper touch event wrapper for Pyjamas. Alas, it has been almost a year, and the wrapper is nowhere to be seen, yet Grafpad got support for touch gestures within a few weeks of that conversation. So what happened?

To summarize what happened, after a few weeks of beating my head against the wall, I gave up on writing the wrapper. I have updated a dozen or so different Pyjamas modules, all the pieces seemed to be there. I’ve made sure that all places responsible for triggering mouse-event logic also triggered touch events, I even made sure my logic was consistent with GWT way of triggering touch events. Pyjamas wasn’t throwing any errors, but my touch handlers just weren’t firing. I could’ve spent more time to debug this correctly, but my main priority was Grafpad, and I couldn’t afford to spend more time on this.

In my frustration, I came up with a hack based on another set of blogs I saw (I no longer have the links to them, so if you recognize them from the source code, please reply to this post and I will add the link). The solution is actually quite simple, and required only some understanding of how the browsers already handle touch events.

Since most websites aren’t designed for mobile devices, mobile browsers have adopted some work-arounds, faking some mouse events to achieve sane behavior on most websites. For many websites these fake events work fine, for many web-apps they do not. In particular, when the user clicks an element on a website, the mobile browser first sends onMouseOver event (that’s right, not onMouseDown). The mobile browser then checks for a change in the DOM, if no change occurs, the browser then sends onMouseDown event. This is actually pretty smart, and in most cases works pretty well. Unfortunately for Grafpad, this triggers a tooltip and requires a second press to actually activate the option being clicked on. The next annoyance is due to how onMouseMove works. Unlike PCs/Macs, mobile devices do not send onMouseMove events continuously while the mouse/finger is pressed down. Instead, they send a single onMouseMove event when the mouse is released (simultaneously with onMouseUp). My guess is that this is done to conserve CPU resources, since continuous onMouseMove events can get pretty resource intensive. As a side-effect, Grafpad would refuse to draw the actual Shape the user would doodle, and instead produce only a straight line from the point of touch-start to the end. The last problem is the keyboard listener. Grafpad relies on it for keyboard shortcuts, having it active at all times. Mobile devices, unfortunately, assume that the website expects keyboard input form the user, activating a keyboard that takes up half of already scarce real-estate. So how did Grafpad deal with these issues?

Apparently, mobile browsers will only try to fake real mouse events if touch gestures aren’t already handled by the website itself. Fortunately, touch gestures are very easy to detect in pure JavaScript, when you don’t have to rely on additional wrappers. To summarize my solution in one sentence, I added some JavaScript code to detect real touch events, and fake mouse events at the same coordinates. This solved all issues except the keyboard. For the keyboard, I modified the keyboard handler CSS display style to none, which seemed to have worked on the iPad. Unfortunately, this didn’t do the trick on Android devices. While I still haven’t addressed those, an easy hack (I mean solution) would be to detect the browser/device from the user-agent string (or better yet, as soon as we notice a touch occur) and disable keyboard handler entirely (we know the user isn’t going to use keyboard shortcuts on a tablet anyway).

Now to the actual solution, here is the code that I added for faking mouse events:

var TOUCHDEVICE = false;
var LASTTOUCH = 0;

function touchHandler(event)
{
    TOUCHDEVICE = true;
    var touches = event.changedTouches,
        first = touches[0],
        type = "",
        ctrl = false;
         switch(event.type)
    {
        case "touchstart":
            var d = new Date()
            var current = d.getTime();
            if (current-LASTTOUCH <= 500){
                ctrl=true;
                event.preventDefault(); //prevent zoom-in lens on double-click
            }
            else {
                LASTTOUCH = current;
            }
            type = "mousedown";
            break;
        case "touchmove":
            type="mousemove";
            event.preventDefault(); //prevent screen dragging
            break;      
        case "touchend":
            type="mouseup";
            break;
        default: return;
    }

    var simulatedEvent = document.createEvent("MouseEvent");
    simulatedEvent.initMouseEvent(type, true, true, window, 1,
                              first.screenX, first.screenY,
                              first.clientX, first.clientY, ctrl,
                              false, false, false, 0/*left*/, null);

    first.target.dispatchEvent(simulatedEvent);
}

function init()
{
    document.addEventListener("touchstart", touchHandler, true);
    document.addEventListener("touchmove", touchHandler, true);
    document.addEventListener("touchend", touchHandler, true);
    document.addEventListener("touchcancel", touchHandler, true);  
}

The above code does several things to make touch events behave the way a user would expect. First of all, once init() gets called, we map every touch event to our function, preventing default behavior from the mobile browser. The actual touchHandler function then decides how to handle each touch event. There are a few things to note in the code. First, the TOUCHDEVICE boolean. This variable was never actually used, but was initially added for the purpose of disabling the keyboard handler on Android devices. The idea is that it gets triggered the first time a touch event occurs. On a PC/Mac such event will never occur, thus the user will not be affected by this functionality. On a mobile device, this event will occur as soon as the user performs the first touch, allowing us to disable the keyboard handler before it has any effect.

The next item of interest is the LASTTOUCH variable. This variable stores the time of the last touch event. As you can see from the code, it’s used to simulate the right-click event (ctrl+click) in case two taps are detected within 0.5 seconds of eachother.

The above logic should be sufficient for single-touch events. Multi-touch events are somewhat more complicated, and I will not cover them in this post (although you should be able to rewrite the above logic to handle those too, if needed – complete documentation for those is available on HTML5Rocks website). One last thing to add, to make the above code work, make sure to add the following attribute to your body tag:

<body ... onLoad="init();" >