Python to Javsacript: Compilers vs. Translators

One thing Alex and I always say about RapydScript is that it is really JavaScript with a Pythonic syntax. Following that, people often ask me what that means for them – they want to know how this affects how they develop code, and how it is different than something like Pyjamas/Pyjs. I want to answer that here to for anyone that has wondered what “RapydScript is Pythonic JavaScript” means, and how compilers like Pyjs are different from translators like RapydScript, and why I (full disclosure) prefer translators.

An Example

A clear example of the difference is with division. Say your source looks like:

a = 5
b = 0
c = a / b

The translator will output JavaScript like:

var a = 5;
var b = 0;
var c = a / b;

The translator process is very simple to understand – it’s pretty much just changing the syntax, but this leads to some gotcha’s. When this code runs, c will be set to Infinity, a JavaScript constant, while the original Pythonic source would have raised an Exception.

A compiler, on the other hand, attempts to mimic Python exactly so it may have an output more similar to:

var py_int = function(val){
    this.val = val;
};
py_int.prototype.div = function(denom){
    if (denom.val == 0) {
        throw ZeroDivisionError;
    }
    return py_int(self.val / denom.val);
};
var a = py_int(5);
var b = py_int(0);
var c = a.div(b);

The variables here will all be objects that include methods for all the operations. The division doesn’t directly divide 2 numbers, it runs the divide method in the objects. So when this code runs it will throw a ZeroDivisionError exception just like Python does.

So what are the tradeoffs?

Writing using a compiler is nice because you get to think like a Python developer, which can abstract away some things like cross browser support. It also means that, in many cases, code can be moved between the frontend and backend with no changes. So it’s easy to have Python code compile to Javascript. But if you’re doing something that’s JavaScript specific, like getting HTML elements, taking in keyboard inputs, etc, the compiler you’re using will have to have a working and documented API for accessing these functions.

The real drawback, though, is with the output code is slower, significantly heavier, and, with the compilers I’ve used, unreadable. There are several issues I have with this, but it really boils down unreadable code leads to usless tracebacks when running code in a browser, and apps don’t run (or run well) on mobile devices.

Translators take a very different approach and don’t try to run just like Python. The idea here is to do 80% of the work for 20% of the cost. Your code may look like Python but it will run like JavaScript, as with the division example. There’s a lot of overlap between the two languages, but they’re not exactly the same, so the main drawback here is that you might see some unexpected, but predictable, behavior. This is very easy though if you know JavaScript, and if not, it’s easy to learn the differences.

There are some nice benefits to using a translator. Your input code and output code will look very similar. They will be roughly the same size and it will be easy to map a line with an error in the JavaScript output back to the Pythonic input for easy debugging. The output will be on the order of kB instead of MB and will run faster.

So those are the main differences between compilers and translators like RapydScript. RapydScript is really JavaScript behind the scenes so it will behave differently than Python, which lets it run a lot more efficiently.

Which is right for you?

The choice of which to use comes down to a few things:

  • First, something I have not mentioned, libraries. In general, JavaScript libraries work better with translators and Python libraries work better with compilers. The one caveat is compilers can only translate Pure Python, so if you’re using something like numpy, which uses C, there’s no easy answer for you.
  • Second, if you don’t know any JavaScript, you will have a tougher time with a translator. Speaking from experience though, JavaScript is not very different from Python, and I encourage you to try a translator because you’ll save time debugging, and your app will be more maintainable in the long term.
  • Lastly is performance requirements. If performance is important, you will want to use a translator over a compiler.

I think it makes sense for a beginner who may writing a simple internal app that won’t have any performance requirements to use a compiler. But if you’ll be writing many apps, or even a complex one, I would go with a translator. Having picked up the differences between JavaScript and Python, I now exclusively use RapydScript, a translator, even if I don’t need the performance. It’s easier to debug in the browser where it actually runs, which saves me time.

Pyjamas and Web2py

UPDATE: Pyjamas has since been renamed Pyjs and is under new leadership. Everything is still backwards compatible.

At this point, if you’ve been following along the posts, you should know how to create a simple web2py application. In this post I’m going to describe how to write a page that can connect to a backend written in web2py. This is another step on the path of having an app on GAE, in fact, the code we write here will get deployed on GAE. This code will also run on any system, including your own computer, and avoids the lock-in some people experience when developing for GAE.

There are 2 ways that I use for communicating with a web server are RESTful JSON calls, and JSON-RPC calls. Instead of covering both I plan on just showing how to use JSON-RPC. I suggest when you design your app first search online for comparisons between REST & JSON-RPC to see the trade-offs.

Before we get into the code, I also have a slight curve-ball. I originally wrote this code using Pyjs, but I’ve switched to using Rapydscript for all my development. In a future post I’ll show how to connect Rapydscript to GAE, which I plan to link here. I suggest reading Alex’s earlier post https://blogs-pyjeon.rhcloud.com/?p=301 for a good summary of various Python to JS compilers and their trade-offs.

web2py Services

Time to code!

First you need to setup your web2py app to use web2py’s services module. This lets your application work as a web service so it can respond to calls from clients, including support for JSON-RPC calls. Inside of models/db.py add the following lines:

from gluon.tools import Service
service = Service()

In the controller, add a call function that returns the service. Then all you have to do is add a decorator to each functions you want to act as services receiving jsonrpc calls and web2py handles the rest. In controllers/default.py I have added the following:

@service.jsonrpc
def myfun(data_from_JSON):
    return data_from_JSON.upper()

def call():
    return service()

The example function here, myfun, will make everything uppercase. Also worth nothing is data_from_JSON is already decoded data from the JSON request.

To access this service use the URI ‘/cyborg/default/call/jsonrpc’. For more information on services check out http://web2py.com/books/default/chapter/29/10#Remote-procedure-calls.

Pyjs Clients

I’ve had this code floating around my computer for a couple years now. I’ve made several minor changes, some just because I wanted slightly lighter code, and some because of changes in Pyjs, but it was originally based on code by Amund Tveit on his blog (http://amundblog.blogspot.com/2008/12/ajax-with-python-combining-pyjs-and.html). This is a simple page with a text area for sending text to a JSON-RPC service.

from pyjamas.ui.RootPanel import RootPanel
from pyjamas.ui.TextArea import TextArea
from pyjamas.ui.Label import Label
from pyjamas.ui.Button import Button
from pyjamas.ui.VerticalPanel import VerticalPanel
from pyjamas.JSONService import JSONProxy


class JSONExample:
    def onModuleLoad(self):
        self.rpc_service = JSONProxy('/cyborg/default/call/jsonrpc', ['myfun'])

        self.text_area = TextArea()
        self.text_area.setText(r"Hello World")
        self.text_area.setCharacterWidth(80)
        self.text_area.setVisibleLines(4)

        button = Button('Send to Server', self)

        self.status = Label()

        panel = VerticalPanel()
        panel.add(self.text_area)
        panel.add(button)
        panel.add(self.status)

        RootPanel().add(panel)


    def onClick(self, sender):
        print('sending to server')
        self.status.setText('Waiting for response...')

        textarea_txt = self.text_area.getText()
        if self.rpc_service.myfun(textarea_txt, self) < 0:
            self.status.setText('Server Error')


    def onRemoteResponse(self, response, request_info):
        print('good response')
        self.status.setText(response)


    def onRemoteError(self, code, message, request_info):
        print('error')
        print(code)
        print(message)
        self.status.setText("Server Error or Invalid Response: ERROR  - " + str(message))


if __name__ == '__main__':
    app = JSONExample()
    app.onModuleLoad()

To make the call, you need an instance of the JSONProxy class, so I have the line

self.rpc_service = JSONProxy('/cyborg/default/call/jsonrpc', ['myfun'])

Then call your function using that instance, along with passing in an object/variable (which Pyjs encodes) to send to the service, and a reference to an instance of a class with onRemoteResponse and onRemoteError methods.

self.rpc_service.myfun(data, response_class)

In the example code above, response_class is self, so the JSON response comes through onRemoteResponse.

Putting it Together

Believe it or not, this was the trickiest part for me when I first got this working. Figuring out the correct URIs to use, and having code listening for the call was tough to debug. Luckily for anyone following along, the example code here already has everything setup correctly :).

We can start out just by making sure everything ties together. Go to your web2py folder then applications\cyborg\static. This is the static directory for the cyborg app. Now take all the output from Pyjs and put it there. I called my Pyjs file JSONExample.py, which generated JSONExample.html, so I access this using http://127.0.0.1:8000/cyborg/static/JSONExample.html. The ‘Send to Server’ button on the page should work. If it doesn’t, make sure you followed every step exactly. There might have also been changes/bugs in Pyjs or web2py – it’s not likely, but it has happened.

I personally wanted my main app to call the JSON-RPC service, not a file in the static director. I am going to have the index load the JSON page. First thing to do is cleanup the index function in default.py. I just want index to return the view, so index now returns an empty dictionary:

def index():
    return {}

I then replace the contents of applications\cyborg\views\default\index (the view for index() in the default controller) with the contents of the main Pyjs output file, JSONExample.html in my case.

The compiled html/js files are still in the static dir though. Remember, users are accessing the files in the views directory by accessing controllers. So if the other Pyjs files were in the views, you would still need to have a controller function to access them. There is no clean, simple way to put all your Pyjs cache files in the views directory. Instead we need to modify the new index.html to point to the files in the static dir. So the index.html code gets modified once for the module:

<meta name="pygwt:module" content="/cyborg/static/JSONExample">

and, in several places, for bootstrap.js:

<script language="javascript" src="/cyborg/static/bootstrap.js"></script>

Now you can visit http://127.0.0.1:8000/cyborg/ and send JSON-RPC calls!

One thing to note is that with the default routes calling /cyborg/, /cyborg/default, and /cyborg/default/index all load the same view/controller. If your JSONProxy class uses a relative link it might only work when visiting 1 of these URIs. That is why, in the Pyjs code, I refer to URI starting from /. When I was learning how to do this, I set everything using relative URI’s, like ../static/JSONExample, and ../default/call/jsonrpc, and that made everything difficult, so I stay away from that.

Pyjamas Alternatives for Web Development

In my last post I mentioned that I’m switching away from Pyjamas. In this one, I will explain the alternatives I looked into, as well as cons and pros of each one. While doing my research (which started almost half a year ago), I’ve stumbled upon multiple Python-inspired alternatives to JavaScript, many of which have been abandoned, and only few of which have reached the stage where they’re suitable for large projects. Out of those I have settled on a solution that works for me, but there is no perfect framework/language, your choice could be different than mine. Let’s look at a few frameworks and decide how usable each one is for writing a full application.

Pyjamas

While I might have issues with how Pyjamas approaches web development, I have to admit that it has gotten quite far, and it’s quite possible to write a usable application with it.

Verdict: usable – if overhead is not a problem

Skulpt

Skulpt does not process your Python code at all until execution time, this is a unique approach that I haven’t seen any other framework take. It’s essentially like a browser-based eval() for your Python. This is a neat concept, since it can be used to extend another framework to support Python-like eval() (something that even Pyjamas doesn’t do), allowing your script to write and execute code on the fly. While I’m not a big fan of eval() function, this would definitely be neat for a framework that aims to achieve complete Python compatibility. Similarly, one of Skulpt’s major disadvantages is having to include the entire “compiler” as a script in your page, if this project ever grows to have the same functionality as Pyjamas, the overhead of loading the extra logic could be enormous. The second disadvantage is speed, Python interpreted and executed in the browser on the fly will never be as fast as even boilerplate-based Pyjamas, let alone pure JavaScript. The final problem is that you can’t obfuscate/compress your code at all, since its interpreter needs to make sense of your Python code (including the whitespace), this might be a non-issue to open-source projects.

Verdict: not usable

pyxc-js

Like Skulpt, this compiler is unsuitable for any large-scale web development, but it has an elegant import mechanism, which is something many other frameworks/compilers lack. Just like Skulpt can be used to implement eval(), this compiler can be used to implement proper import in your compiler (I know Pyjamas has proper import, but it’s heavily integrated into the rest of the framework). This project has been abandoned and the author does not respond to emails. Aside from its import mechanism (which is based on a directed graph the compiler builds at compile time, eliminating any unused and repeated imports), this project doesn’t have much to offer. The documentation is lacking, it took me a while (and some tweaks) to be able to compile anything, the included test cases don’t work, and the generated code is rather messy (it often spits out multiple semi-colons at the end of the line and blank lines get semicolons too, and if I remember correctly there are occasional bugs in how your code gets compiled). On the plus side, it offers an internal compression mechanism that can shorten your code further (although I’m not sure how much I trust it, given the quality of the non-compressed code it generates).

Verdict: not usable

PyCow

This was the first framework aside from Pyjamas that I felt could support large projects. PyCow compiles Python code into MooTools-based JavaScript. More importantly, its source code is cleanly laid out and the original developer is very responsive and helpful (despite having abandoned the project). It relies on templates for many of its conversions rather than hard-coding them in the source or implementing alternative names for JavaScript functions. For example, list.append() gets converted to list.push() at compile time using a template specified in a hash table. This is a great idea, since it makes the compiler code cleaner and the final output stays closer to real JavaScript, introducing less overhead (and less need for large stdlib). The disadvantage of this approach is that we have no good way of knowing whether a certain method is part of our language and needs to be replaced by JavaScript equivalent, or if it belonds to another API and should be left alone. Here is an example where replacing the append() would break our code:

container = $('#element'); // jQuery used to create a container of elements
...
container.append(item);    // replace this append() and jQuery will complain

PyCow is also MooTools’-based, and I’d prefer a language that doesn’t tie me to a certain framework. I investigated rewriting it such that it generates classes using pure JavaScript, but later realized it would not be worth the effort. One other minor pet-peeve is that PyCow is written for Python 2.5, which happened to be around the same time as Python developers were rewriting their AST parser. As a result, PyCow uses a transitional AST package and needs to be rewritten to support the later one that came after 2.6 (2.5 is still closer to the current AST, however, than the AST parser Pyjamas uses form compiler module, which has been deprecated as of Python 2.6). This framework also doesn’t support any importing of modules, but I managed to fix that in my local version by ripping out the import mechanism from pyxc-js. Unfortunately, I’ve moved on since then, abandoning my PyCow enhancements (let me know if you’re interested in continuing from where I left off, however, I can send you the source code).

Verdict: usable

py2js (Variation 1)

There are at least 3 different frameworks by this name, and they’re only vaguely related to each-other. The first one has been abandoned since 2009, but it impressed me for several reasons. First of all, it takes PyCow’s templating one step further, allowing you to map virtually any function/method using more complex filtering. For instance, %0 represents the class that the method is being called on, %1 reprensents the first argument, %* represents all of them. You could use this to add very powerful templating. For example, if you want to output a list of elements as a comma-delimited string in Python, you could use join() as follows: ', '.join(list). In JavaScript, however, join() is a method of array/list (which actually makes more sense to me) rather than a string, requiring you to do list.join(', ') instead. But say you wanted to auto-convert your Python joins to JavaScript, all you would have to do is add the following rule to your template hash:

'%0.join(%1)' : '%1.join(%0)'

I was able to extend this variation of py2js to support multiple Python features with JavaScript alternatives this way. As mentioned earlier, however, this method is susceptible to renaming functions whose names you have no control over.

This compiler relies on templates heavily, introducing another advantage most other compilers lack. Once the code has been converted to AST, it can be manipulated almost entirely using templates (with little need to write additional code). With some tweaks, it can become a universal source-to-source compiler (well, universal is a bit strong of a word here, statically typed languages are a whole other animal). Imagine being able to convert Python to Perl, Ruby, or any other dynamically typed language just by generating a map of logic equivalents. Something like this, for example, could allow you to convert the syntax of a Python function to Perl (block would then further get interpreted by similar templates for loops, conditionals, etc.):

### Input Template
def <method>(%*):
    <block>

### Output Template
sub <method>{
    my (%*) = @_;
    <block>;
}

Admittedly, the more fancy you get with the language (i.e. using eval()), the more likely you’re to run into problems with the template breaking. This compiler was inspired by py2py, another project designed to clean up Python code. In theory, any input languages can be supported as well, as long as you write your own AST parser that converts the code to Pythonic AST tree (or a subset of). This is a fun project in itself, and I might continue it when I have more time. As a compiler, however, this project is not yet suitable for prime-time, lacking a lot of functionality, including classes.

Verdict: not usable

py2js (Variation 2)

The only similarity between this py2js and the previous is the name. They share nothing else (well… they do now, keep reading). Variation 2 just happens to be another project by the same name as variation 1. In fact, even the compile mechanism for this py2js is very different from other compilers mentioned before. The compiler works by “running” your code at compile time. You write your function/class in Python, and then add @JavaScript decorator to it if you want it to be included in the compiled output. You then “run” your program and it dumps JavaScript to STDOUT. While it’s a bit awkward to add @JavaScript decorator to every single function, this offers something most other compilers (including Pyjamas) lack. Since it actually runs your code, you get the benefit of partial “compile-time” error catching, a feature typically only seen in statically-typed languages. You can be sure, for example, that all syntax errors will be caught, as well as undeclared variables/methods. The disadvantage is that this compiler is a bit too strict, all your code has to be pure Python and all your variables (or stubs for them) have to exist at compile time. So, for example, if you want to use something from jQuery, you better have a jQuery stub written in Python (it doesn’t need to mimic jQuery functionality, however, only to have the method names defined). This project has also been abandoned several years ago in a premature stage, which is where the 3rd variation comes in.

Verdict: not usable

py2js (Variation 3) (now Pyjaco)

The 3rd variation started off as a fork of the 2nd variation, after 3 developers decided to take over the original, abandoned, project. Over time, this variation morphed into a powerful compiler, whose functionality is only outdone by Pyjamas itself. It added support for Pythonic assertions, tuples, classes, static methods, getattr()/setattr(), *args/**kwargs, better integration with JavaScript (allowing you to call JavaScript logic by wrapping it inside JS() or automatically handling the conversion at compile time if you use one of predefined globals (document, window, etc.)) as well as many other features you would expect from Python. It has its own standard library with most Python stdlib implemented.

It, however, also falls short in a few areas. Like Pyjamas, it tries to build Pythonic error handling on top of JavaScript. Admittedly, this task might be much easier here, due to partial “compile-time” error catching, which Pyjamas lacks. It also doesn’t properly translate Python’s math methods to JavaScript, a task that shouldn’t be too hard, however.

Additionally, while it’s convenient to have document and window be treated like special classes, this can add confusion when coding. In some cases your strings and numbers will get converted to JavaScript equivalents automatically, while in the others you have to do so manually (when invoking a native JavaScript method and passing it a string, for example). A py2js string is a wrapper around native JavaScript string with additional Python functionality, same applies to many other objects. This is a good thing, since it allows support for Python’s methods without having to overwrite native objects (and I believe Pyjamas takes the same approach for its primitive objects (arrays/strings/etc.). This is a minor pet-peeve, however, and would not be a problem most of the time.

Another annoyance that I already mentioned earlier, is having to explicitly define which functions you want compiled via @JavaScript tag + str() call on the function or class (but this is just a matter of personal preference).

The biggest problem with this third variation, however, is copyright issues (which the developers are trying to resolve now – they could already have been resolved). In its quest to acquire similar functionality as Pyjamas, this compiler has “borrowed” a lot of code from other projects (including the first py2js I mentioned). This py2js project itself carries MIT license, not all of the projects it borrows from, however, are compatible with that license. Additionally, some of these projects were not given proper mention in the copyright notice due to an oversight by one of the developers. As a result, the developers had a falling out, and now forked a couple alternative versions of this project, one of which is still maintained and aims to rewrite the code in question and/or get proper authorization to use it.

Additionally, if this project joins forces with the new Pyjamas, I believe both projects will benefit a lot. The code generated by py2js is significantly smaller (Pyjamas averages 1:10 ratio for non-compiled:compiled code, while py2js is closer to 1:2 – not counting stdlib) and more readable than Pyjamas, while retaining most of Pyjamas’ features. Pyjamas, on the other hand, already has its own implementation for most of py2js code that’s subject to copyright issues, which could be used to solve the biggest problem with py2js. Finally, py2js’ compile-time error catching makes it easier to build Pythonic exception handling on top of JavaScript. And to remedy the annoyance of @JavaScript decorators, the AST parser can be used to append them automatically to temporary version of the code. This can also be used as an opportunity to update Pyjamas to use the latest AST parser implementation, which it will need anyway if it ever wants to be compatible with Python 3 (which is missing the deprecated AST implementation).

Verdict: usable

UPDATE: I’ve been informed that this project now became Pyjaco, and the copyright is no longer an issue. So for those who want to stay closer to Python, this is a very solid alternative. Christian, the project leader, also informed me that not all of the details I mentioned are accurate. Also, apparently the developers seem to have misinterpreted the original post as me claiming that RapydScript (my own compiler) is the best for everything. That was not my intent, and I tried to avoid this issue by mentioning in the first paragraph of the original article that my choice is based on my own projects and the flexibility they need (mostly Grafpad), even stating that “your choice could be different than mine”. I hope they don’t hold a grudge at me because of this misunderstanding.

PyvaScript

PyvaScript is the opposite extreme of Pyjamas. It’s perhaps the closest to pure JavaScript out of all other Python-like languages. It was one of the first alternatives I looked into, and admittedly I originally decided it would be unsuitable for large projects. There are numerous disadvantages compared to other Python frameworks. Its stdlib is puny, and most of the Python methods you would want aren’t available. It not only lacks import logic, but even classes aren’t implemented. It can’t handle negative indexes for arrays. Its compiler isn’t even based on Python AST (it uses Ometa), as a result it’s often oblivious to errors other compilers would catch, and at other times it chokes on issues that other compilers have no problems with.

Its advantages, however, provide the perfect fit for the niche that Pyjamas left open. It’s the most JavaScript-compatible solution, supporting almost all JavaScript functionality out of the box. I can manipulate other JavaScript and DOM without the need of any wrappers or special syntax. Need to use jQuery? Just do it the same way you would in JavaScript: $(element).method(), the compiler won’t complain about the dollar sign. PyvaScript also supports anonymous functions, something regular Python lacks (except 1-liner lambdas), which makes it easier to write your code like JavaScript (I didn’t see the advantage to this at first, but believe me, when writing web apps, it really helps – especially when using other JavaScript logic as a tutorial).

Since PyvaScript adds no bells or whistles to your code, or requires any sort of wrappers, your code behaves a lot more like native JavaScript, allowing you to use closures more effectively and to bind functions to objects dynamically without the language complaining. At first this seemed like a bad idea, but after playing with it, I realized that with proper use of this, I can actually write cleaner code than with Python itself. Most importantly, PyvaScript is extremely light-weight, adding almost no overhead to the generated code and does not force a framework (MooTools, jQuery, etc.) on the user. I also realized, that PyvaScript’s lazy compilation has its own advantages (which I will explain later).

Verdict: usable – but doesn’t alleviate much pain from plain JS

CoffeeScript

Sharing a structure similar to that of Python, it deserves a mention as well. If you ignore its ugly syntax, and poor choice of variable scoping (preferring globals over locals), you will see that it has all the same advantages as PyvaScript, which makes it a good candidate for Pyjamas replacement as well. It has similar feel to Python (although it feels closer to Ruby), introduces list comprehensions, and even adds classes (something PyvaScript does not). It also adds namespaces, preventing variables in different modules from interfering. If I invert the scoping design, remove all the junk variables like on/off (synonyms for true/false), and modify the syntax to use Python tokens, this will be my ideal language for web development, but that’s a project for later.

Verdict: usable

And the winner is…

RapydScript, which I didn’t even mention yet. RapydScript (best described as PyvaScript++) is my wrapper around PyvaScript that provides the best of both worlds (for me at least). I’ve made a few enhancements (abusing PyvaScript’s lazy compilation to allow me to auto-generate broken Python code that becomes proper JavaScript once compiled by PyvaScript) that achive almost everything I would want, and just about all of CoffeeScript’s functionality. Some of the new RapydScript features include support for Pythonic classes including inheritance (single inheritance only, but you can bind functions from any class to fake multiple inheritance), Pythonic optional function arguments, anonymous functions now supported in dictionaries/object literals (something PyvaScript chokes on), beefed up stdlib (also optimized already implemented methods from PyvaScript’s stdlib), support for multi-line comments/doc-strings, preliminary (compile-time) checking for syntax errors and issues that PyvaScript chokes on (because of minor bugs in PyvaScript), module importing (currently everything gets dumped into a single namespace, but compiler does warn about conflicting names), checking proper use of Math functions, automatic insertion of ‘new’ keyword when creating an object (not sure why CoffeeScript doesn’t already do the same).

To me, the main advantages of RapydScript over PyvaScript are ability to break down my modules into separate files (like I would in Python), easier time to build large projects due to proper class implementation (class declaration is done the same way as in native Python), Pythonic declaration of optional arguments to a function (I’m not a big an of JavaScript’s solution for optional arguments), and support for anonymous functions as hash values (which allows me to build object literals the same way as in JavaScript). As for other projects, the main advantages of RapydScript are seamless integration with the DOM and other JavaScript libraries/modules (just treat them like regular Python objects), ability to use both Python and JavaScript best practices as well as rely on JavaScript tutorials (one of the biggest problems for projects in their alpha stage is lack of documentation, for RapydScript you really don’t need any), and lack of bloat (RapydScript gets me as close to Python as I need without forcing any boilerplate on me). As a nice bonus, I think I can add suppport for advanced compilation mode of Google’s Closure compiler with only minor tweaks.

If you want to give RapydScript a try, you can find it in the following repository: https://bitbucket.org/pyjeon/rapydscript. Keep in mind, this is still early alpha version and some of the functionality might change. Chances are, I will eventually rewrite this, using CoffeeScript as base, but the functionality should stay very similar (after all, I am rewriting Grafpad in this, and I don’t want to rewrite it a third time). In my next post, I will dive into more detail about RapydScript, for those interested in using it and needing a tutorial.

Why Pyjamas Isn’t a Good Framework for Web Apps

Earlier this week, I stated that Pyjamas no longer seems like a viable solution for Grafpad (or many other web-apps for that matter). In this post, I will explain the flaws with Pyjamas that ultimately made me decide to switch away from it. I’m aware that Pyjamas project is currently getting an overhaul, and I hope that these flaws get addressed in the upcoming Pyjamas releases. Before I go any further into bashing Pyjamas, I want to mention that I’ve been using Pyjamas for several years, writing over 20,000 lines of Python code that runs inside the browser (as well as several Pyjamas wrappers for extending its functionality). I appreciate the problem Pyjamas is trying to solve, and I definitely think it’s a useful tool. Perhaps one day Pyjamas will be good enough for the browser, unfortuantely it has a lot of issues to solve before that’s the case.

Experienced JavaScript developers might already be familiar with many of the points I will bring up. To summarize Pyjamas’ flaws in one sentence, it basically assumes that JavaScript is still a joke of a language it was several years ago and tries to apply outdated solutions that don’t scale well. Today’s JavaScript, however, can run circles around its predecessor, both in terms of performance and functionality. Many innovative design patterns have also been posted for keeping JavaScript code clean and object-oriented. In some ways, JavaScript has even surpassed Python in terms of design, which still lacks proper private variables, for example. So what are some of the big offenders in Pyjamas?

Browser Detection instead of Feature Detection

Many of you are probably familiar with Pyjamas’ compilation scheme. If not, it basically creates multiple versions of the JavaScript code, one for each major browser (IE, Firefox, Safari/Chrome, Opera) and serves the appropriate one depending on your user-agent string. A quick Google search will reveal thousands of pages explaining the problems with this technique (called browser detection), so there is really no point for me to go into much detail here. The first problem with browser detection is that we assume that the user will be using one of the browsers we’re detecting (sorry Konqueror). The second problem is that we’re assuming the user is using one of the versions of this browser that still has the same issues/functionality. I’ve already posted about the changes I had to make in Pyjamas to make it use IE9 properly, which has full canvas support, yet Pyjamas still treats it like IE6 (ironically, IE9 actually behaves more like WebKit than IE6). The third problem is that many browsers spoof the user-agent string, pretending to be a different browser (for various reasons). These browsers may support features that the spoofed browser doesn’t support and vice versa, forcing us to use an unnecessary work-around for a feature that the browser supports natively (just like IE9 being forced to use VML instead of canvas), or preventing the feature from working altogether (imagine if Chrome, with no VML support, spoofed IE6 user-agent string).

Bloat and Boilerplate Hell

If you’ve peeked at Grafpad’s JavaScript, you probably saw 80,000 lines of code in a 3.5MB file. But did you know that the pre-compiled version of Grafpad front-end is only about 8,000 lines of code? We have 10 times the needed code just to pretend like we’re still using Python. What’s worse, most of that code is only there to support obscure Python functionality most of us are never going to use in a web-app anyway. Pyjamas has become the most complete Python framework for the browser, unfortunately it has also become the most bloated one, with most other frameworks (such as py2js) only needing to generate 1.5 lines of JavaScript for each line of Python code. You can see the 80/20 principle at work here, where 20% of Python’s features account for 80% of Pyjamas’ boilerplate. In my opinion, it would make a lot more sense to only support the commonly used features of Python, allowing the user to rewrite the bits that don’t work well for JavaScript. After all, the most tedious things to port between languages are the algorithms, not the object structure.

Debugging

In theory, Pyjamas is much easier to debug than JavaScript. Unlike JavaScript, which either throws vague errors or worse yet, silently fails a block of code and continues execution like nothing happened, Pyjamas throws Pythonic exceptions, which most of the time do a very good job pinpointing the exact line that caused the problem… at least when you run your program through Pyjamas Desktop. The problem is, Pyjamas Desktop has been broken for almost 3 years now, requiring you to either use a 3-year old Linux distribution (last known version to have support for python-hulahop) or rely on WebKit or MSXML implementations, neither of which supports canvas.

Alternatively, you can debug your code directly in the browser. Pyjamas sports a good set of Python exceptions emulated in JavaScript through clever use of try/catch blocks. Unfortunately, this alternative not only lacks proper stack trace, but also the original code (and compiling your Pyjamas app with any of the debug modes doesn’t solve this, regardless of what various outdated posts on the mailing list claim). Needless to say, the errors raised by Pyjamas in the browser are not very useful. If you made an IndexError on line 50 of your code by referencing “object.array[5]“, for example, expect Pyjamas to throw some weird error (that’s right, chances are it won’t even realize it’s an IndexError – or at least won’t report it well, the except blocks seem to work correctly) on line 30,000 of your compiled JavaScript, which will reference $p['getattr'](object, 'array').__getitem__(5), among a bunch of other boilerplate which could have caused an error as a result of an earlier error in your code or a Pyjamas bug. Even when debugging using Pyjamas Desktop, the browser errors can occasionally be inconsistent with normal Python (usually due to a bug in Pyjamas), and it’s a pain to troubleshoot these. And there is really not much Pyjamas can do to remedy this, in my opinion.

Adding additional assertions to catch every possible case to throw Pythonic errors is a fool’s errand no different than trying to parse HTML using regular expressions. Python’s ability to throw relevant assertions stems from its fundamental design. It’s very strict about using non-existing/undefined variables and comparison of irrelevant types. JavaScript, on the other hand, is very lazy/permissive about these, much like Perl. Python is proactive about its assertions, Pyjamas tries to be reactive. It’s unrealistic to forsee every special case that could arise and account for it with an assertion the same way Python would. Even if you manage to do so, you will have added even more code to Pyjamas’ already large chunk of boilerplate (not to mention potential for new Pyjamas bugs). One option is to compile these assertions away when the debug flag isn’t set, but even then you would be doing the exercise of examining all possible errors that Python could throw in each case, plugging in more “reactive” logic to make JavaScript work the same way. Instead, we should make the framework easy to debug in the environment it’s meant to be in. Since we can’t make JavaScript behave like Python, and we can’t do compile-time debugging like we would with C++ or Java, we should make the output easy to understand, so that we can map it back to the original code.

Python is not Java, DOM is not a Desktop

This brings me to my next point. GWT (the original inspiration for Pyjamas) might be more bloated than Pyjamas, but there is something it can do that Pyjamas can’t: compile-time error catching. If it wasn’t for Python being a dynamically-typed language, a lot of my rant in the previous section about debugging would be irrelevant. Additionally, I don’t feel that Pyjamas is approaching the problem from the right angle. Python has the advantage of being much more similar to JavaScript than Java ever will, and a lot of Pyjamas’ wrapper logic wouldn’t even be necessary if Pyjamas didn’t try to pretend to be GWT (in addition to pretending to be Python). GWT was designed to make web development similar to Desktop GUI development, since that’s the background many Java developers come from. What other purpose is there to fake MouseListener and KeyboardListener in an environment that wasn’t designed to need either (KeyboardListener, by the way, is another source of grief for Pyjamas – it’s what makes the keyboard pop-up all over the place on mobile devices, it also attaches a fake input element to the current element, pretending like they’re the same element, adding even more boilerplate and wrappers to the code)? What other purpose is there to build the entire DOM dynamically (which, by the way, is also extremely inefficient)? The browser page was not designed to function the same way your Desktop calculator app does. Anyone who has taken a few minutes to learn how the DOM works probably agrees that it’s actually superior to the old-fashioned Desktop way of writing the GUI. I’m lazy (otherwise I wouldn’t have written my front-end in Python), so when a new technology comes along that clearly makes my life easier, why ignore it?

If it wasn’t for trying to fake a Desktop GUI, Pyjamas wouldn’t need all these wrappers. Most other Python-faking frameworks allow one to invoke JavaScript logic as if it was a regular Python object/function. Pyjamas, on the other hand, requires one to first write a wrapper for Pyjamas Desktop using Python, then for the browser using some limbo version of Python/JavaScript hybrid (where you can’t even access elements of array using standard indexing), and finally rewrite a separate version of your limbo code for each non-compliant browser (definitely IE, and possibly some others). This wrapping might have been necessary in Java, but should not be needed for Python at all, and could have been prevented with better design. But wait, there IS an alternative! You can put raw JavaScript in your code using JS() method and passing it one giant string of JavaScript code. Unfortunately, that chunk of the code will get completely ignored in Pyjamas Desktop (which you’re using to debug your entire app, since the browser debugger is no help at all), and to actually reference anything from this chunk of code in the browser, you will need to reference these variables the same way: “a = JS(‘a’)” (again, don’t expect “a” to get set in Pyjamas Desktop). Oh, and don’t try to modify any of the DOM elements created by Pyjamas from anything other than Pyjamas, you will run into object state sync issues. Pyjamas wraps each DOM element in a Python object, which then stores the element’s state as a set of variables, and assumes it doesn’t change without Pyjamas’ permission. Pyjamas plays well with other JavaScript frameworks… as long as they don’t touch any portion of the DOM Pyjamas uses.

JavaScript has its Strengths

JavaScript might not be the cleanest language, and I still much prefer Python to it. But I must give it credit where credit is due. First of all, it integrates the DOM into itself really well. I can take any DOM element, assign a function to onMouseDown event as if it was a regular JavaScript object, and all of a sudden I got an element that reacts to my mouse clicks. No need for complicated ClickHandlers.

Pyjamas has a lot of abstraction layers, both to hide JavaScript inconsistencies, and make it easier to build widgets. However, native JavaScript libraries, like jQuery, do a much better job at both. Yes, jQuery doesn’t scale well for larger projects, but there are libraries that do, like MooTools (which, by the way, was inspired by Python). But realistically, if you create a simple wrapper for generating classes (or loot one from John Resig’s blog – the same guy who wrote jQuery), even jQuery becomes good enough for creating large projects. Pyjamas, on the other hand, adds so much abstraction, that sometimes I need hacks just to manipulate the DOM. If you look at the DOM of a typical Pyjamas app, you will notice layers of unnecessary elements: images wrapped in divs, wrapped in more divs, placed inside some table that resides inside yet another table. When I try to render my app on a tablet, it often crashes due to the DOM bloat.

Pyjamas also assumes that JavaScript is slow, which was true when the project first started. As a result, it duplicates parts of its boilerplate code to avoid an extra function call (while adding excessive function calls and abstractions in other places). Ironically, JavaScript engines have come a long way, and a lot of Pyjamas’ optimizations are no longer relevant (such as using object["property"] instead of object.property). In fact, a quick paint app I wrote in Pyjamas actually runs faster in Chrome than Pyjamas Desktop. That same app runs faster still when written in pure JavaScript. It’s especially noticeable when using the paint-bucket tool, which works by pixel-scanning and takes a couple seconds in Pyjamas yet almost instantaneous in JavaScript.

Summary

While Pyjamas is the most complete Python emulation in a browser, it has become a very bloated and brittle framework. It doesn’t embrace any portion of JavaScript, nor the DOM, trying to hide them away like some sort of deformed beast. By pretending to be pure Python, it not only puts unrealistic expecations on itself, but also fails to make use of good parts of JavaScript. Instead, Pyjamas embraces a solution designed for a statically-typed language, favoring a GUI structure that should have died a decade ago.

So What’s The Alternative?

I did mention that I am porting Grafpad away from Pyjamas. However, I’m not crazy enough to rewrite the entire project in pure JavaScript. Rewriting all the code in a language with different quirks and troubleshooting differences like division rounding and modulo signage is not my idea of fun. I also still prefer to keep my front-end code interchangeable with the back-end (more or less), which has already provided multiple advantages, such as moving the proprietary clipping and recognition algorithms to the back-end in just a few hours of work. I happen to have another ace up my sleeve. In the next post, I will review multiple alternatives for Pyjamas and explain the solution I’ve chosen.

Touch Gestures in Pyjamas

Before I continue, I wanted to apologize for the lack of updates. To pay the bills, I got a new job in October, which took away a lot of my free time. You also may be familiar with the current state of Pyjamas from the mailing list. Before I go further, I wanted to mention that I have realized Pyjamas is not the best API for Grafpad in November of last year, and have been working on alternative solutions (I will write up a detailed post explaining this soon). However, for those still using Pyjamas, I wanted to provide my solution for handling touch events on mobile devices.

Some of you probably noticed that Grafpad has had support for iPad/Android devices for almost a year now despite Pyjamas still lacking support for touch gestures. Some of you may have even seen the communication between Luke and me where I was trying to figure out how to write a proper touch event wrapper for Pyjamas. Alas, it has been almost a year, and the wrapper is nowhere to be seen, yet Grafpad got support for touch gestures within a few weeks of that conversation. So what happened?

To summarize what happened, after a few weeks of beating my head against the wall, I gave up on writing the wrapper. I have updated a dozen or so different Pyjamas modules, all the pieces seemed to be there. I’ve made sure that all places responsible for triggering mouse-event logic also triggered touch events, I even made sure my logic was consistent with GWT way of triggering touch events. Pyjamas wasn’t throwing any errors, but my touch handlers just weren’t firing. I could’ve spent more time to debug this correctly, but my main priority was Grafpad, and I couldn’t afford to spend more time on this.

In my frustration, I came up with a hack based on another set of blogs I saw (I no longer have the links to them, so if you recognize them from the source code, please reply to this post and I will add the link). The solution is actually quite simple, and required only some understanding of how the browsers already handle touch events.

Since most websites aren’t designed for mobile devices, mobile browsers have adopted some work-arounds, faking some mouse events to achieve sane behavior on most websites. For many websites these fake events work fine, for many web-apps they do not. In particular, when the user clicks an element on a website, the mobile browser first sends onMouseOver event (that’s right, not onMouseDown). The mobile browser then checks for a change in the DOM, if no change occurs, the browser then sends onMouseDown event. This is actually pretty smart, and in most cases works pretty well. Unfortunately for Grafpad, this triggers a tooltip and requires a second press to actually activate the option being clicked on. The next annoyance is due to how onMouseMove works. Unlike PCs/Macs, mobile devices do not send onMouseMove events continuously while the mouse/finger is pressed down. Instead, they send a single onMouseMove event when the mouse is released (simultaneously with onMouseUp). My guess is that this is done to conserve CPU resources, since continuous onMouseMove events can get pretty resource intensive. As a side-effect, Grafpad would refuse to draw the actual Shape the user would doodle, and instead produce only a straight line from the point of touch-start to the end. The last problem is the keyboard listener. Grafpad relies on it for keyboard shortcuts, having it active at all times. Mobile devices, unfortunately, assume that the website expects keyboard input form the user, activating a keyboard that takes up half of already scarce real-estate. So how did Grafpad deal with these issues?

Apparently, mobile browsers will only try to fake real mouse events if touch gestures aren’t already handled by the website itself. Fortunately, touch gestures are very easy to detect in pure JavaScript, when you don’t have to rely on additional wrappers. To summarize my solution in one sentence, I added some JavaScript code to detect real touch events, and fake mouse events at the same coordinates. This solved all issues except the keyboard. For the keyboard, I modified the keyboard handler CSS display style to none, which seemed to have worked on the iPad. Unfortunately, this didn’t do the trick on Android devices. While I still haven’t addressed those, an easy hack (I mean solution) would be to detect the browser/device from the user-agent string (or better yet, as soon as we notice a touch occur) and disable keyboard handler entirely (we know the user isn’t going to use keyboard shortcuts on a tablet anyway).

Now to the actual solution, here is the code that I added for faking mouse events:

var TOUCHDEVICE = false;
var LASTTOUCH = 0;

function touchHandler(event)
{
    TOUCHDEVICE = true;
    var touches = event.changedTouches,
        first = touches[0],
        type = "",
        ctrl = false;
         switch(event.type)
    {
        case "touchstart":
            var d = new Date()
            var current = d.getTime();
            if (current-LASTTOUCH <= 500){
                ctrl=true;
                event.preventDefault(); //prevent zoom-in lens on double-click
            }
            else {
                LASTTOUCH = current;
            }
            type = "mousedown";
            break;
        case "touchmove":
            type="mousemove";
            event.preventDefault(); //prevent screen dragging
            break;      
        case "touchend":
            type="mouseup";
            break;
        default: return;
    }

    var simulatedEvent = document.createEvent("MouseEvent");
    simulatedEvent.initMouseEvent(type, true, true, window, 1,
                              first.screenX, first.screenY,
                              first.clientX, first.clientY, ctrl,
                              false, false, false, 0/*left*/, null);

    first.target.dispatchEvent(simulatedEvent);
}

function init()
{
    document.addEventListener("touchstart", touchHandler, true);
    document.addEventListener("touchmove", touchHandler, true);
    document.addEventListener("touchend", touchHandler, true);
    document.addEventListener("touchcancel", touchHandler, true);  
}

The above code does several things to make touch events behave the way a user would expect. First of all, once init() gets called, we map every touch event to our function, preventing default behavior from the mobile browser. The actual touchHandler function then decides how to handle each touch event. There are a few things to note in the code. First, the TOUCHDEVICE boolean. This variable was never actually used, but was initially added for the purpose of disabling the keyboard handler on Android devices. The idea is that it gets triggered the first time a touch event occurs. On a PC/Mac such event will never occur, thus the user will not be affected by this functionality. On a mobile device, this event will occur as soon as the user performs the first touch, allowing us to disable the keyboard handler before it has any effect.

The next item of interest is the LASTTOUCH variable. This variable stores the time of the last touch event. As you can see from the code, it’s used to simulate the right-click event (ctrl+click) in case two taps are detected within 0.5 seconds of eachother.

The above logic should be sufficient for single-touch events. Multi-touch events are somewhat more complicated, and I will not cover them in this post (although you should be able to rewrite the above logic to handle those too, if needed – complete documentation for those is available on HTML5Rocks website). One last thing to add, to make the above code work, make sure to add the following attribute to your body tag:

<body ... onLoad="init();" >

Strategy pattern in Pyjamas

I was recently trying to implement a Strategy pattern inside my Pyjamas web app. It’s basically a design pattern that allows the programmer to use multiple algorithms interchangeably without relying on inheritance, and it’s meant for cases when inheritance would add too much complexity.

For example, imagine that we’re implementing a class for a video game character, which has an attack() method. This method would probably vary depending on the weapon the character is holding. If the character is bare-handed he will probably punch the opponent. If the character is holding a sword, he’ll swing the sword at the opponent instead. If the character is holding a spear, he’ll perform a throwing motion. In this case, inheritance is clearly a poor choice, we’re not going to morph the character into a new object each time he switches a weapon. One other way to implement this is by using if/elif/else statements, but this solution isn’t much better, since we’ll end up with a behemoth method spanning several hundred lines of code as soon as we have a decent arsenal of weapons. We could also implement if/elif/else setup that calls other methods, but even then we’re adding unnecessary overhead. The cleanest solution is a Strategy pattern, which essentially maps the correct method at run-time.

Unfortunately, Pyjamas doesn’t seem to support either the types module or get() method for binding a function/method to a class method. Without it, Strategy pattern seems impossible to implement in Python.

For example, using types module, a clean way to implement this pattern in Python would be as follows:

import types

def strategyA(possible_self):
...

instance = OrigObject()
instance.strategy = types.MethodType(strategyA, instance)
instance.strategy()


This runs fine in Pyjamas Desktop, but throws an import error in the browser since types module does not exist in current Pyjamas implementation:

ImportError: No module named types, types in context None


There is an internalast fork of Pyjamas that does implement the types module, but does not implement the MethodType function within it. Essentially, the only difference would be the type of error you get.

There is an alternative way to implement the Strategy pattern. Every function/method in Python has descriptor methods that allow you to customize when you use an attribute. One of such methods is get() which allows you to rebind the method as follows:

instance.strategy = strategyA.__get__(self, OrigObject)


Unfortunately this solution also doesn’t work, throwing a type error in Pyjamas (since descriptors aren’t yet implemented in Pyjamas):

TypeError: Object function ... has no method '__get__'



The main problem is that Pyjamas will not let you bind an unbound method to an instance in Pyjamas for use as a bound method later. Which brings us to a somewhat hackerish (is that a word?) workaround. The idea is to define all alternative methods inside the class itself, and then swap between them, as if using variables. For example, if we have two alternative algorithms, strategyA and strategyDefault, we can switch back and forth between them as follows or each instance of the class independently:

self.strategyDefault, self.strategyA = self.strategyA, self.strategyDefault

If you need more than 2 methods, you can still use the same approach, but instead of swapping 2 methods, set the active one to point to one of the alternatives, keeping a backup of the original in another variable.

I must admit that even despite being the language where “there should only be one obvious way to do it” Python allows the programmer great flexibility in terms of implementation. And that flexibility is one of Pyjamas’ best features, in that even if Pyjamas drops the ball, Python tends to have plenty of firepower to pick up the slack.

Adding True Internet Explorer 9 support to Pyjamas

About two months ago, Rich Newpol added Internet Explorer 9 support to pyjamas. Before then, pyjamas tried to use old mozilla (pre-3.5 version) format whenever it detected IE9, which would result in an epic fail. One small problem, however, is that Rich’s solution was not good enough for the app I’m developing. The solution was to tell IE9 to render the website the same way previous versions of IE would render it. Since my app relies heavily on HTML5 canvas element, this means I’d be stuck with old crappy VML. Instead I have started examining pyjamas to add true IE9 support. In the end, it turned out simpler than I thought. This post talks about how I did it, as well as the quirks I had to address.

Setting up ie9 directory

To maintain at least partial support in older IE browsers, I decided to create a new __ie9__ directory inside pyjamas library. My first goal was to get pyjamas to compile a separate cache file for IE9 (*.ie9.cache.html). To do so, I had to modify home.nocache.html file inside of boilerplate to detect ie9:

...
20 else if (ua.indexOf('msie7.0') != -1) {
21 return 'ie6';
22 }
23 else if (ua.indexOf('msie 8.0') != -1) {
24 return 'ie6';
25 }
26 else if (ua.indexOf('msie 9.0') != -1) {
27 return 'ie9';
28 }
...
50 window["prop$user.agent"] = function() {
51 var v = window["provider$user.agent"]();
52 switch (v) {
53 case "ie9":
54 case "ie6":
55 case "mozilla":
56 case "oldmoz":
57 case "opera":
58 case "safari":
59 return v;
60 default:
61 parent.__pygwt_onBadProperty("%(app_name)s", "user.agent",
["ie9", "ie6", "mozilla", "oldmoz", "opera", "safari"], v);
62 throw null;
63 }
64 };
...

This tells the website to load *.ie9.cache.html if the browser identifies itself as ‘msie 9.0′. The next task is to modify pyjamas so it actually generates *.ie9.cache.html file for us. The script responsible for this is pyjs/src/pyjs/browser.py. Here are the changes I made to it:

...
21 AVAILABLE_PLATFORMS = ('IE6', 'IE9', 'Opera', 'OldMoz', 'Safari', 'Mozilla')
...
45 class BrowserLinker(linker.BaseLinker):
46
47 # parents are specified in most-specific last
48 platform_parents = {
49 'mozilla':['browser'],
50 'ie6':['browser'],
51 'ie9':['browser'],
52 'safari':['browser'],
53 'oldmoz':['browser'],
54 'opera':['browser'],
55 }
...

With these files modified, pyjamas should now generate *.ie9.cache.html file for us, that the *.nocache.html file will load if you visit the page in IE9. The problem is that there is nothing in our ie9 folder yet.

Populating ie9 directory

The obvious starting point would probably be to copy the contents of __ie6__ directory into __ie9__ and start tweaking it from there. However, after playing with that approach for a while, I started realizing that __ie9__ is so different from previous versions that the best approach is to actually leave __ie9__ blank and start troubleshooting from there. It turns out that if you leave __ie9__ directory empty, the compiled *.ie9.cache.html file is actually somewhat usable. After some testing, I found that the only methods that did not work as expected were DOM.eventGetButton(), DOM.getAbsoluteLeft(), DOM.getAbsoluteTop() (it’s possible I missed some, but my very complex web app seems to be working fine in IE9 now). After writing my own implementations of those, I noticed that DOM.py from __mozilla__ implements all 3 of those methods similar to my personal implementation, and all 3 implementations fix IE9 as well. This is not too much of a surprise, since Opera 11 now seems to run just fine with mozilla cache file as well, as I reported on pyjamas mailing list here. Perhaps __mozilla__ should be renamed to __default__ or __compliant__? Anyway, if you’ve been following along, chances are that even with all these changes you still might have issues with your web-app layout. The next section is designed to help you troubleshoot those.

Layout and appearance issues

First of all, IE9 will automatically throw you into ‘Quirks Mode’ if you don’t define a correct DOCTYPE. Quirks mode is designed to help with rendering non-W3C-compliant webpages. Its purpose is to render pages that were designed for older browsers and are no longer compliant with today’s standards. You can find more about it on Wikipedia. Actually, all browsers have a quirks mode, and they will all throw you into it if you don’t define a valid DOCTYPE. And if you look through pyjamas examples, you’ll probably notice a consistent lack of DOCTYPE. That’s right, most of pyjamas examples actually run in quirks mode when opened inside a browser (and not just in IE). This is usually not a big deal, unless you happen to be running IE9.

It turns out IE9 does not support event handler when in quirks mode. Unfortunately, event handler is one of the most important building blocks for pyjamas, as it handles just about all user interaction. So to make your web app compatible with IE9, you will have to make it W3C-compliant, this is something that will affect all browsers, not just IE9. Most of your changes will be in your app’s main html file itself and your css stylesheet, although it’s possible that you will need some slight code changes as well (in particular if you used DOM.setStyleAttribute() method).

First of all, we need to define a valid DOCTYPE. To do so, open up your app’s main html file (should be located in your app’s public directory) and add this line at the very top (before tag):

<!DOCTYPE html>

This tells it to render your app using the latest html standard (which would be HTML5). If your app looks and works great, congratulations, you know your html and can probably stop reading now. If you’re someone like me, however, you’re probably staring at a deformed version of your app while scratching your head. First of all, let’s address the issues that are common across all browsers. To do so, read my previous post about making your app HTML5-compliant. Once done, your app should appear correctly in all browsers (except maybe some glitchtes in IE9). Here are the last few fixes for the annoyances with IE9.

If you use a text area element (pyjamas’ TextArea() class), IE9 seems to override the width and height you set in the stylesheet unless you use ‘!important’ tag. It also seemed to add a scrollbar for me, so I had to set ‘overflow’ property to auto as well. I also used this oportunity to prevent Chrome from making my text area resizable (that little corner in bottom-right corner that the user can drag) by setting ‘resize’ to none:

.gwt-TextArea {
    overflow: auto;
    resize: none;
    width: 300px !important; /* important tag needed for IE9 */
    height: 100px !important; /* important tag needed for IE9 */
}

The second annoyance seems to be that unlike other browsers, IE9 (when operating in strict mode) does not assume that a single value for a field should be applied to both dimensions. This means that to center a background image, for example (as I needed to do with my loading icon), you’d need to specify “center” twice in CSS stylesheet. That should be pretty much it, I have to admit that for the most part IE9 is actually more standards compliant than Safari or Chrome. There might be other quirks, which I haven’t ran into with my app. Feel free to mention them to me so I can update this post.

Making Your Pyjamas App HTML5-compliant

I recently decided to update my pyjamas app so that it’s HTML5-compliant. I did this because I recently customized my pyjamas version to work in Internet Explorer 9 (I needed true IE9 support, not the backwards compatibility mode current version of pyjamas provides). Unfortunately, IE9 seems to be very anal about it’s rendering modes, it will either render the page in strict HTML5, or revert to half-broken ‘Quirks Mode’ that doesn’t even support event handler. It seemed easier to tweak my app to be compatible with HTML5 (besides I’d have to do so eventually anyway) than to try to write a special version of event handler for IE just so that I can run my app in a mode that’s not even meant to be used by modern websites anyway.

First of all, I had to tell the browser to render the page in HTML5 mode. To do so, I opened the main .html file for the app (in project’s public directory) and added the DOCTYPE at the very top (before <html> tag):

<!DOCTYPE html>

This tells the browser to render the page using the latest html standard (which happens to be HTML5). Second, I noticed that my fancy “loading” gif no longer appeared in the center of the page (not vertically at least). Apparently, setting my pyjamas frame to 100% height was no longer good enough. In HTML5, I now have to set both, <body> and <html> tags to take up 100% height:

html, body {
    height:100%;
}

The next issue with HTML5 that I noticed is that all my images now seemed to have a small footer below them (a few pixels of space below the image). This is probably not a big deal for most, but when you use images as icons, or need to fit exactly within a certain container, it can definitely mess up your layout. The reason for that is that HTML5 modified default display style for images to ‘inline’, which treats them the same way as text, giving them a ‘line-height’ property. There are two ways to fix the images back to normal. One way is to define ‘line-height’ for the image as 0:

.gwt-Image {
    line-height: 0;
}

This was my initial solution, but there is a problem with it. There is a bug in Firefox that overrides line-height, even if you specify ‘!important’ after it. The other alternative is to change image back to the way it was rendered before HTML5:

.gwt-Image {
    display: block;
}

This solution works in all browsers. Additionally, I noticed that the the text I placed inside tables using pyjamas’ Label() class rendered with too much spacing, as if it put every cell inside

tags. Turns out the work around for that was the same as for the images (even though the problems differ). To render the tables correctly, I modified the above code as follows:

.gwt-Table > tbody, .gwt-Image {
    display: block; /* HTML5 effects panel alignment fix, and footer removal fix for images */
}

Note that for the above fixes to work, you must not be overriding the .gwt-Image that pyjamas assigns to images, and should add .gwt-Table style to tables:

table.setStyleName('gwt-Table')

Finally, I noticed that my fonts were no longer consistent and that tables whose dimensions were defined via CSS were rendering incorrectly as well. The reason for that is that HTML5 no longer assumes that unitless numbers correspond to pixels. Which means that for all fonts, you now need to define “pt”, “px”, or “em” after the number, unless the number is zero. So something like

.text {
    font-size: 10;
}

should be changed to

.text {
    font-size: 10px;
}

The same change also applies to tables, divs, etc. If you have something as follows in your CSS:

.some-box {
    width: 200;
    height: 150;
}

you should change it to

.some-box {
    width: 200px;
    height: 150px;
}

I realize that some web designer is probably reading this now laughing at me, since I should have known to define the units to begin with. However, coming from a programming background, I see CSS more as a necessary evil, and usually stop tweaking it as soon as the page renders as expected (and up until HTML5, it rendered as expected without ‘px’ defined). Along the same lines, if you use Widget.setSize(), Widget.setWidth(), or Widget.setHeight() methods in pyjamas, make sure that you pass in the units as well (Widget.setSize('100px', '50px')) or use Widget.setPixelSize() instead. That’s all you need to do to make your pyjamas app HTML5-compliant.

To summarize:

  1. Add <!DOCTYPE html> tag at the top of the main .html file
  2. Set html and body to 100% height in CSS
  3. set images and table’s tbody to display: block
  4. Add units to every number in CSS (unless the number is 0)

There are a few IE9-specific quirks that this will not solve, but I’ll discuss them in my next post.

Pyjamas Applications on Google App Engine

The next topic I want to cover is writing an app on top of GAE. This one took me a while to originally figure out. I had absolutely no framework experience before, and no idea how GAE worked. Initially, I was very confused at first so I want to discuss how GAE works and what I learned. This post assumes you have the GAE SDK installed and you’ve created an application using the Google App Engine dashboard.

Let’s start with the most important file for your web app – app.yaml. Anytime someone visits any URL under your domain, http://yourapp.appspot.com, their request will go through the app.yaml file to figure out what to load. In proper terms, the app.yaml file forwards the URL path to a “request handler”. The app.yaml file only complicates things for web app with static files, but it becomes very powerful when you start building more complex apps that use services like JSONRPC.

My first GAE app used the helloworld example from Pyjamas. I’ll post the app.yaml file I used and explain what the different sections mean, and then I’ll explain how to package everything to upload it.

application: myapp
version: 1
runtime: python
api_version: 1
handlers:
- url: /
  static_files: output/Hello.html
  upload: output/Hello.html

- url: /*
  static_dir: output

The first 4 lines are standard. The only unique thing you’ll want is your application name. This application should already be created.

The rest of the file is a list of handlers. Each handler starts with a URL pattern. When a user visits a URL on your domain, GAE uses the first handler with a URL patterns that matches the client request. In the app.yaml file above the URL pattern for the first handler is /, which will catch requests for http://yourapp.appspot.com/. The line after the URL pattern tells us we have a static file handler. This static file handler will load output/Hello.html – the main HTML file for the helloworld app. This means that when a user visits http://yourapp.appspot.com/, they load output/Hello.html.

The 2nd handler is for catching calls to http://yourapp.appspot.com/*. This handler is for a static directory, not just a single file. This handler is here because Pyjamas compiles into multiple javascript/cache files which Hello.html loads. Files like Hello.nocache.html need to be accessible from URLs like http://yourapp.appspot.com/Hello.nocache.html in order for the app to work.

There’s one last type of handler: script handlers. I personally use the script handlers to add support for JSONRPC calls, which I will demonstrate in a future post. For the full details on how to configure your app.yaml file visit the GAE documentation at http://code.google.com/appengine/docs/python/config/appconfig.html .

Finally, with the app.yaml file complete, all you need to do is put everything into 1 nice package. I put the app.yaml file at the same level as the output directory (to be clear, 1 level above all the output files). Add this existing project to the Google App Engine Launcher, Deploy it, and you’re good to go!

Installing Pyjamas on Windows XP

Now for the good stuff! I followed a wiki post and got an old Pyjamas version working. The old Pyjamas version was missing some features, so I figured out a way to build the latest version from scratch. For documentation purposes, here are the steps I followed on Windows XP:

  1. Download and run a Python 2.6 installer (here’s a link to the latest: http://www.python.org/download/releases/2.6.6/)
  2. Install comtypes 0.6.1. (comtypes-0.6.1.win32.exe in http://sourceforge.net/projects/comtypes/files/comtypes/0.6.1/)
  3. Permanently update the path for Python (and Pyjamas while you’re at it)
    1. Go to System Properties through Control Panel
    2. Open the Advanced tab, then click on the Environmental Variables button
    3. Add the following to your “PATH” system variable: c:\python26;c:\Pyjamas\pyjs\bin; (the 2nd dir will exist after you install pyjamas)
  4. Install Git for Windows (If you have Git skip this step)
    1. Install msygit from http://code.google.com/p/msysgit/downloads/detail?name=Git-1.7.3.1-preview20101002.exe&can=2&q= B
    2. Install TortisGit from http://code.google.com/p/tortoisegit/downloads/detail?name=TortoiseGit-1.5.8.0-32bit.msi&can=2&q=
  5. Git the latest Pyjamas code.
    1. Create a directory C:\Pyjamas\
    2. Open an explorer window, and navigate to the C:\ drive.
    3. Right click on the Pyjamas directory and select Git Clone.
    4. Enter the URL: https://github.com/pyjs/pyjs.git git://pyjs.org/git/pyjamas.git
  6. Open a new command line (this has to be done after updating the path):
    1. > cd C:\Pyjamas\pyjs C:\Pyjamas\pyjamas
    2. > python bootstrap.py Note: The bootstrap won’t print anything to the screen, but it will create the bin directory (which was added to your path in step 3)
  7. Now test an application:
    1. > cd examples\helloworld
    2. > python __main__.py
    3. Use Firefox or IE and open output\Hello.html Note: Chome won’t load AJAX pages off your local machine for security reasons. You’ll have to upload it somewhere to see it in chrome. For more info see http://code.google.com/p/chromium/issues/detail?id=40787
  8. If you find anything different, update the wiki! Pyjamas Wiki

I ran through steps 1-4 once about 6 months ago. Since then I periodically rerun steps 5 and 6 to get the latest Pyjamas code.

One last note. I wouldn’t recommend pyjamas desktop on Windows if you’re doing Canvas apps. It will be good when a webkit pyjamas desktop version is out, but if you want pyjamas desktop, I’d go with Ubuntu 9.10. Because I have an Ubuntu box, I’ve never put in the time into figuring out how to get pyjamas desktop working on Windows with the latest Pyjamas code, but I do know others that are using it.