Pyjamas Alternatives for Web Development

In my last post I mentioned that I’m switching away from Pyjamas. In this one, I will explain the alternatives I looked into, as well as cons and pros of each one. While doing my research (which started almost half a year ago), I’ve stumbled upon multiple Python-inspired alternatives to JavaScript, many of which have been abandoned, and only few of which have reached the stage where they’re suitable for large projects. Out of those I have settled on a solution that works for me, but there is no perfect framework/language, your choice could be different than mine. Let’s look at a few frameworks and decide how usable each one is for writing a full application.

Pyjamas

While I might have issues with how Pyjamas approaches web development, I have to admit that it has gotten quite far, and it’s quite possible to write a usable application with it.

Verdict: usable – if overhead is not a problem

Skulpt

Skulpt does not process your Python code at all until execution time, this is a unique approach that I haven’t seen any other framework take. It’s essentially like a browser-based eval() for your Python. This is a neat concept, since it can be used to extend another framework to support Python-like eval() (something that even Pyjamas doesn’t do), allowing your script to write and execute code on the fly. While I’m not a big fan of eval() function, this would definitely be neat for a framework that aims to achieve complete Python compatibility. Similarly, one of Skulpt’s major disadvantages is having to include the entire “compiler” as a script in your page, if this project ever grows to have the same functionality as Pyjamas, the overhead of loading the extra logic could be enormous. The second disadvantage is speed, Python interpreted and executed in the browser on the fly will never be as fast as even boilerplate-based Pyjamas, let alone pure JavaScript. The final problem is that you can’t obfuscate/compress your code at all, since its interpreter needs to make sense of your Python code (including the whitespace), this might be a non-issue to open-source projects.

Verdict: not usable

pyxc-js

Like Skulpt, this compiler is unsuitable for any large-scale web development, but it has an elegant import mechanism, which is something many other frameworks/compilers lack. Just like Skulpt can be used to implement eval(), this compiler can be used to implement proper import in your compiler (I know Pyjamas has proper import, but it’s heavily integrated into the rest of the framework). This project has been abandoned and the author does not respond to emails. Aside from its import mechanism (which is based on a directed graph the compiler builds at compile time, eliminating any unused and repeated imports), this project doesn’t have much to offer. The documentation is lacking, it took me a while (and some tweaks) to be able to compile anything, the included test cases don’t work, and the generated code is rather messy (it often spits out multiple semi-colons at the end of the line and blank lines get semicolons too, and if I remember correctly there are occasional bugs in how your code gets compiled). On the plus side, it offers an internal compression mechanism that can shorten your code further (although I’m not sure how much I trust it, given the quality of the non-compressed code it generates).

Verdict: not usable

PyCow

This was the first framework aside from Pyjamas that I felt could support large projects. PyCow compiles Python code into MooTools-based JavaScript. More importantly, its source code is cleanly laid out and the original developer is very responsive and helpful (despite having abandoned the project). It relies on templates for many of its conversions rather than hard-coding them in the source or implementing alternative names for JavaScript functions. For example, list.append() gets converted to list.push() at compile time using a template specified in a hash table. This is a great idea, since it makes the compiler code cleaner and the final output stays closer to real JavaScript, introducing less overhead (and less need for large stdlib). The disadvantage of this approach is that we have no good way of knowing whether a certain method is part of our language and needs to be replaced by JavaScript equivalent, or if it belonds to another API and should be left alone. Here is an example where replacing the append() would break our code:

container = $('#element'); // jQuery used to create a container of elements
...
container.append(item);    // replace this append() and jQuery will complain

PyCow is also MooTools’-based, and I’d prefer a language that doesn’t tie me to a certain framework. I investigated rewriting it such that it generates classes using pure JavaScript, but later realized it would not be worth the effort. One other minor pet-peeve is that PyCow is written for Python 2.5, which happened to be around the same time as Python developers were rewriting their AST parser. As a result, PyCow uses a transitional AST package and needs to be rewritten to support the later one that came after 2.6 (2.5 is still closer to the current AST, however, than the AST parser Pyjamas uses form compiler module, which has been deprecated as of Python 2.6). This framework also doesn’t support any importing of modules, but I managed to fix that in my local version by ripping out the import mechanism from pyxc-js. Unfortunately, I’ve moved on since then, abandoning my PyCow enhancements (let me know if you’re interested in continuing from where I left off, however, I can send you the source code).

Verdict: usable

py2js (Variation 1)

There are at least 3 different frameworks by this name, and they’re only vaguely related to each-other. The first one has been abandoned since 2009, but it impressed me for several reasons. First of all, it takes PyCow’s templating one step further, allowing you to map virtually any function/method using more complex filtering. For instance, %0 represents the class that the method is being called on, %1 reprensents the first argument, %* represents all of them. You could use this to add very powerful templating. For example, if you want to output a list of elements as a comma-delimited string in Python, you could use join() as follows: ', '.join(list). In JavaScript, however, join() is a method of array/list (which actually makes more sense to me) rather than a string, requiring you to do list.join(', ') instead. But say you wanted to auto-convert your Python joins to JavaScript, all you would have to do is add the following rule to your template hash:

'%0.join(%1)' : '%1.join(%0)'

I was able to extend this variation of py2js to support multiple Python features with JavaScript alternatives this way. As mentioned earlier, however, this method is susceptible to renaming functions whose names you have no control over.

This compiler relies on templates heavily, introducing another advantage most other compilers lack. Once the code has been converted to AST, it can be manipulated almost entirely using templates (with little need to write additional code). With some tweaks, it can become a universal source-to-source compiler (well, universal is a bit strong of a word here, statically typed languages are a whole other animal). Imagine being able to convert Python to Perl, Ruby, or any other dynamically typed language just by generating a map of logic equivalents. Something like this, for example, could allow you to convert the syntax of a Python function to Perl (block would then further get interpreted by similar templates for loops, conditionals, etc.):

### Input Template
def <method>(%*):
    <block>

### Output Template
sub <method>{
    my (%*) = @_;
    <block>;
}

Admittedly, the more fancy you get with the language (i.e. using eval()), the more likely you’re to run into problems with the template breaking. This compiler was inspired by py2py, another project designed to clean up Python code. In theory, any input languages can be supported as well, as long as you write your own AST parser that converts the code to Pythonic AST tree (or a subset of). This is a fun project in itself, and I might continue it when I have more time. As a compiler, however, this project is not yet suitable for prime-time, lacking a lot of functionality, including classes.

Verdict: not usable

py2js (Variation 2)

The only similarity between this py2js and the previous is the name. They share nothing else (well… they do now, keep reading). Variation 2 just happens to be another project by the same name as variation 1. In fact, even the compile mechanism for this py2js is very different from other compilers mentioned before. The compiler works by “running” your code at compile time. You write your function/class in Python, and then add @JavaScript decorator to it if you want it to be included in the compiled output. You then “run” your program and it dumps JavaScript to STDOUT. While it’s a bit awkward to add @JavaScript decorator to every single function, this offers something most other compilers (including Pyjamas) lack. Since it actually runs your code, you get the benefit of partial “compile-time” error catching, a feature typically only seen in statically-typed languages. You can be sure, for example, that all syntax errors will be caught, as well as undeclared variables/methods. The disadvantage is that this compiler is a bit too strict, all your code has to be pure Python and all your variables (or stubs for them) have to exist at compile time. So, for example, if you want to use something from jQuery, you better have a jQuery stub written in Python (it doesn’t need to mimic jQuery functionality, however, only to have the method names defined). This project has also been abandoned several years ago in a premature stage, which is where the 3rd variation comes in.

Verdict: not usable

py2js (Variation 3) (now Pyjaco)

The 3rd variation started off as a fork of the 2nd variation, after 3 developers decided to take over the original, abandoned, project. Over time, this variation morphed into a powerful compiler, whose functionality is only outdone by Pyjamas itself. It added support for Pythonic assertions, tuples, classes, static methods, getattr()/setattr(), *args/**kwargs, better integration with JavaScript (allowing you to call JavaScript logic by wrapping it inside JS() or automatically handling the conversion at compile time if you use one of predefined globals (document, window, etc.)) as well as many other features you would expect from Python. It has its own standard library with most Python stdlib implemented.

It, however, also falls short in a few areas. Like Pyjamas, it tries to build Pythonic error handling on top of JavaScript. Admittedly, this task might be much easier here, due to partial “compile-time” error catching, which Pyjamas lacks. It also doesn’t properly translate Python’s math methods to JavaScript, a task that shouldn’t be too hard, however.

Additionally, while it’s convenient to have document and window be treated like special classes, this can add confusion when coding. In some cases your strings and numbers will get converted to JavaScript equivalents automatically, while in the others you have to do so manually (when invoking a native JavaScript method and passing it a string, for example). A py2js string is a wrapper around native JavaScript string with additional Python functionality, same applies to many other objects. This is a good thing, since it allows support for Python’s methods without having to overwrite native objects (and I believe Pyjamas takes the same approach for its primitive objects (arrays/strings/etc.). This is a minor pet-peeve, however, and would not be a problem most of the time.

Another annoyance that I already mentioned earlier, is having to explicitly define which functions you want compiled via @JavaScript tag + str() call on the function or class (but this is just a matter of personal preference).

The biggest problem with this third variation, however, is copyright issues (which the developers are trying to resolve now – they could already have been resolved). In its quest to acquire similar functionality as Pyjamas, this compiler has “borrowed” a lot of code from other projects (including the first py2js I mentioned). This py2js project itself carries MIT license, not all of the projects it borrows from, however, are compatible with that license. Additionally, some of these projects were not given proper mention in the copyright notice due to an oversight by one of the developers. As a result, the developers had a falling out, and now forked a couple alternative versions of this project, one of which is still maintained and aims to rewrite the code in question and/or get proper authorization to use it.

Additionally, if this project joins forces with the new Pyjamas, I believe both projects will benefit a lot. The code generated by py2js is significantly smaller (Pyjamas averages 1:10 ratio for non-compiled:compiled code, while py2js is closer to 1:2 – not counting stdlib) and more readable than Pyjamas, while retaining most of Pyjamas’ features. Pyjamas, on the other hand, already has its own implementation for most of py2js code that’s subject to copyright issues, which could be used to solve the biggest problem with py2js. Finally, py2js’ compile-time error catching makes it easier to build Pythonic exception handling on top of JavaScript. And to remedy the annoyance of @JavaScript decorators, the AST parser can be used to append them automatically to temporary version of the code. This can also be used as an opportunity to update Pyjamas to use the latest AST parser implementation, which it will need anyway if it ever wants to be compatible with Python 3 (which is missing the deprecated AST implementation).

Verdict: usable

UPDATE: I’ve been informed that this project now became Pyjaco, and the copyright is no longer an issue. So for those who want to stay closer to Python, this is a very solid alternative. Christian, the project leader, also informed me that not all of the details I mentioned are accurate. Also, apparently the developers seem to have misinterpreted the original post as me claiming that RapydScript (my own compiler) is the best for everything. That was not my intent, and I tried to avoid this issue by mentioning in the first paragraph of the original article that my choice is based on my own projects and the flexibility they need (mostly Grafpad), even stating that “your choice could be different than mine”. I hope they don’t hold a grudge at me because of this misunderstanding.

PyvaScript

PyvaScript is the opposite extreme of Pyjamas. It’s perhaps the closest to pure JavaScript out of all other Python-like languages. It was one of the first alternatives I looked into, and admittedly I originally decided it would be unsuitable for large projects. There are numerous disadvantages compared to other Python frameworks. Its stdlib is puny, and most of the Python methods you would want aren’t available. It not only lacks import logic, but even classes aren’t implemented. It can’t handle negative indexes for arrays. Its compiler isn’t even based on Python AST (it uses Ometa), as a result it’s often oblivious to errors other compilers would catch, and at other times it chokes on issues that other compilers have no problems with.

Its advantages, however, provide the perfect fit for the niche that Pyjamas left open. It’s the most JavaScript-compatible solution, supporting almost all JavaScript functionality out of the box. I can manipulate other JavaScript and DOM without the need of any wrappers or special syntax. Need to use jQuery? Just do it the same way you would in JavaScript: $(element).method(), the compiler won’t complain about the dollar sign. PyvaScript also supports anonymous functions, something regular Python lacks (except 1-liner lambdas), which makes it easier to write your code like JavaScript (I didn’t see the advantage to this at first, but believe me, when writing web apps, it really helps – especially when using other JavaScript logic as a tutorial).

Since PyvaScript adds no bells or whistles to your code, or requires any sort of wrappers, your code behaves a lot more like native JavaScript, allowing you to use closures more effectively and to bind functions to objects dynamically without the language complaining. At first this seemed like a bad idea, but after playing with it, I realized that with proper use of this, I can actually write cleaner code than with Python itself. Most importantly, PyvaScript is extremely light-weight, adding almost no overhead to the generated code and does not force a framework (MooTools, jQuery, etc.) on the user. I also realized, that PyvaScript’s lazy compilation has its own advantages (which I will explain later).

Verdict: usable – but doesn’t alleviate much pain from plain JS

CoffeeScript

Sharing a structure similar to that of Python, it deserves a mention as well. If you ignore its ugly syntax, and poor choice of variable scoping (preferring globals over locals), you will see that it has all the same advantages as PyvaScript, which makes it a good candidate for Pyjamas replacement as well. It has similar feel to Python (although it feels closer to Ruby), introduces list comprehensions, and even adds classes (something PyvaScript does not). It also adds namespaces, preventing variables in different modules from interfering. If I invert the scoping design, remove all the junk variables like on/off (synonyms for true/false), and modify the syntax to use Python tokens, this will be my ideal language for web development, but that’s a project for later.

Verdict: usable

And the winner is…

RapydScript, which I didn’t even mention yet. RapydScript (best described as PyvaScript++) is my wrapper around PyvaScript that provides the best of both worlds (for me at least). I’ve made a few enhancements (abusing PyvaScript’s lazy compilation to allow me to auto-generate broken Python code that becomes proper JavaScript once compiled by PyvaScript) that achive almost everything I would want, and just about all of CoffeeScript’s functionality. Some of the new RapydScript features include support for Pythonic classes including inheritance (single inheritance only, but you can bind functions from any class to fake multiple inheritance), Pythonic optional function arguments, anonymous functions now supported in dictionaries/object literals (something PyvaScript chokes on), beefed up stdlib (also optimized already implemented methods from PyvaScript’s stdlib), support for multi-line comments/doc-strings, preliminary (compile-time) checking for syntax errors and issues that PyvaScript chokes on (because of minor bugs in PyvaScript), module importing (currently everything gets dumped into a single namespace, but compiler does warn about conflicting names), checking proper use of Math functions, automatic insertion of ‘new’ keyword when creating an object (not sure why CoffeeScript doesn’t already do the same).

To me, the main advantages of RapydScript over PyvaScript are ability to break down my modules into separate files (like I would in Python), easier time to build large projects due to proper class implementation (class declaration is done the same way as in native Python), Pythonic declaration of optional arguments to a function (I’m not a big an of JavaScript’s solution for optional arguments), and support for anonymous functions as hash values (which allows me to build object literals the same way as in JavaScript). As for other projects, the main advantages of RapydScript are seamless integration with the DOM and other JavaScript libraries/modules (just treat them like regular Python objects), ability to use both Python and JavaScript best practices as well as rely on JavaScript tutorials (one of the biggest problems for projects in their alpha stage is lack of documentation, for RapydScript you really don’t need any), and lack of bloat (RapydScript gets me as close to Python as I need without forcing any boilerplate on me). As a nice bonus, I think I can add suppport for advanced compilation mode of Google’s Closure compiler with only minor tweaks.

If you want to give RapydScript a try, you can find it in the following repository: https://bitbucket.org/pyjeon/rapydscript. Keep in mind, this is still early alpha version and some of the functionality might change. Chances are, I will eventually rewrite this, using CoffeeScript as base, but the functionality should stay very similar (after all, I am rewriting Grafpad in this, and I don’t want to rewrite it a third time). In my next post, I will dive into more detail about RapydScript, for those interested in using it and needing a tutorial.