Equality Operator

Matrix-agent-Smith-clones One thing that always bothered me, and many others, is how useless the == operator is in JavaScript. It seems to combine the worst of both worlds from equality and identity:

  • When comparing primitive types, it acts as a lax equality operator
  • When comparing objects, it acts as an identity operator

I can’t think of any use case for which such behavior is desired. In fact, reversing this behavior would produce a much more useful operator since the concept of “equals” tends to relax as data gets more complex because some dimensions are simply irrelevant. Do I care that two objects were not created at the same exact time? Do I care that their IDs differ? Do I care if the measured length is 4.00000000001 inches rather than 4? When it comes to equality comparison, JavaScript fails miserably. This is actually one of Douglas Crockford’s pet peeves about the language, which he explains in more detail in his book: JavaScript: The Good Parts.

Fortunately, there is also a more consistent identity operator (===), often erroneously called “strict equality” by people who don’t understand the difference between identity and equality. Unfortunately, as I just mentioned in previous paragraph, there is no concept of equality operator in the language at all. For those unfamiliar with these concepts, identity answers the question of “Does A and B refer to the same object?” while equality answers the question of “Is B a clone of A?“.

Naturally, both of these questions come up in programming a lot, so it’s a shame that only one can be easily answered in JavaScript. You probably already stumbled into this problem if you ever tried something like this:

new Date('1/1/2000') == new Date('1/1/2000')

or this:

{"foo": 1} == {"foo": 1}

The problem is not unique to JavaScript, many other languages lack equality operator as well. Typically, however, those languages have an alternative mechanism for handling this case, such as operator overloading. Indeed, there are cases when operator overloading is actually superior – such as when objects contain meta data that one wishes to omit from comparison (i.e. exact creation time, unique id, etc.). Unfortunately, JavaScript doesn’t allow for operator overloading either, and herein lies the problem. While one can easily roll a proper equality function (and many libraries such as underscore already include one) you would still have to decide whether it’s worth using on a case-by-case basis.

We’re not in FORTRAN age anymore, where developers had to tweak each operation. Today we’re spoiled, we often can get away with simply telling the compiler what to do rather than how, and enjoy optimal performance 90% of the time with negligible overhead. One such example is the sort function. When was the last time you had to wonder if you should use merge or insertion sort? You simply use the built-in sort and assume that unless there is something very special about your data, you’re better off moving on to other things rather than attempting to optimize the last 10% out of the algorithm. In most modern languages equality operator falls in the same category. Sure, you’d be able to shave off a few microseconds by replacing that == (equality) with is (identity) for certain comparison operations in Python, but is it worth the extra brain cycles? Most of the time the answer is “No”.

Why then do we have to be explicitly aware of this in JavaScript? Moreover, why can’t RapydScript compile == into deep equality? First, I should mention that, like some other libraries/languages for JavaScript, RapydScript already has a deep equality comparison via the inbuilt eq function. Yes, eq(a, b) (previously called deep_eq) has been supported for years. The problem is that (up until recently) I wasn’t able to make the decision for you of whether you want eq or ==. The issue boils down to the fact that if I decide to compile == to equality across the board for the developer, I effectively introduce enormous overhead (about 700% according to jsperf) on the user in cases where he/she expected identity comparison instead (which is about 90% of the time, since primitives are a lot more common). There has been a lot of discussion about this, which you can follow in issues 93 and 94 on my github page.

As you probably already guessed from previous paragraph, I’m now able to bridge that gap. Effective last month (that’s right, I snuck a change in without anyone noticing), RapydScript is the first JavaScript-based transcompiler to support high-performance deep equality. How performant is this operation, you may ask? Well, take a look for yourself:

Screen Shot 2015-12-05 at 2.59.38 PM

According to jsperf, the overhead is negligible (that’s right, the measurement noise is greater than any visible difference – as you can see from the deep-equality version outperforming identity). So what changed? Why am I suddenly able to blow the doors off of typical deep equality? Well, nothing new happened in JavaScript world. What changed is my approach. I decided to think about the problem creatively and had a sudden eureka moment (which I’m sure other compilers will copy in the future). Unfortunately you’ve probably already looked at the code from JsPerf above, ruining my surprise. But in case you haven’t, here is the pattern I came up with:

A === B || typeof A === "object" && eq(A, B)

How does it work? As you probably learned in your introductory programming class (assuming it was a legitimate C/Java class rather than an online tutorial about JavaScript), binary operators have short-circuit ability in just about all languages. This means that if left-hand side is truthy for || (or) operator, or falsy for && (and), the rest of the line is ignored (it’s as if it’s not there). That means that if A and B are primitives that are equal, the above will be equivalent to a simple A === B comparison. That handles the positive case, but negative is a bit trickier. I was struggling with it for a while (yes, the above equation makes it seem simpler than it really is – everything seems easy in hindsight), until I found a performant operation that works in both, browser and node (my first attempt was to check if A.constructor exists, which doesn’t work on all platforms). Fortunately, the very same “feature” that makes typeof useless in most cases (the fact that every non-primitive is an object) becomes the saving grace of this operation. As you can see from the JsPerf test, the overhead for this operation is negligible as well.

The best part is that unlike native JavaScript, where the developer would have to type that out by hand (because hiding it in a function call introduces the 700% overhead we’re trying to avoid), RapydScript can unroll == operator into that magic automatically. You’re probably wondering, then, how is it that this change has been in RapydScript for over a month if the == still compiles to ===? Well, for safety I’ve hidden this operator behind an import. If you wish all your == operators to compile to proper equality test shown above, add the following line at the top of your file:

from danger_zone import equality

That’s it. Now all your == will compile to the magic shown above, and != will compile to its inverse (thanks to DeMorgan’s Law). The above import will also tweak implementation of indexOf and lastIndexOf to perform deep equality as well, which in turn makes tests like if a in arr be based on deep equality (delivering a consistent experience across the board). As before, the identity operator is still there as well, via is and is not, which compile to === and !==, respectively. In the future, I also want to add an optimizer to the compiler, which will strip away the remainder of the line at compile time if one of the operands is a constant.

Modifying Native JavaScript Objects

Random Number Generator There are still debates in JavaScript community about modifying native JavaScript objects (Array, Number, String, Object). Some developers believe it’s evil, while other encourage it. If you’ve looked at RapydScript’s stdlib, you probably already know my stance on it – I’m actually in favor of such modifications when they make sense. I might not have as much experience in JavaScript as the gurus, but I have enough experience in language design to form sane opinions about which practices are evil and which are not.

Most arguments against modifying the native objects hold no water. They give examples where the developer overrides a native method to work differently from original implementation. That is indeed evil, assuming that calling this method with the same arguments as original JavaScript implementation no longer produces the same result. For example, if I decided to rewrite String.prototype.replace such that it worked globally, like in most other languages, instead of on a first occurrence:

String.prototype.replace = function(orig, sub) {
    return this.split(orig).join(sub);
}

I could break (or make them behave differently than they should) other libraries/widgets on the same page that assume that replace() will only replace the first occurrence. I completely agree that one should never override existing methods to do something else. It’s important that the basic subset of the JavaScript language works exactly as others would expect it to. If you can’t guarantee that the foundation of your house stays level, you can’t guarantee that your house will not collapse.

There are some gray area cases, where the developer could extend the functionality of existing method, such that it still works as expected given original arguments, but does something else when more arguments are given:

Array.prototype.pop = function(index) {
    if (!arguments.length) {
        index = this.length - 1;
    }
    return this.splice(index, 1)[0];
}

In this case, myArray.pop() will still work exactly as the user would expect. However, if called with myArray.pop(0), the function will behave like myArray.shift(), or like splice() method if called with an index in the middle. The only real disadvantage here is that by overriding the native method, the developer has made the function slower than the original (native methods tend to work faster). Claiming that this is bad because a function could break another logic that mistakenly called it with an argument (which it ignored before) on the other hand is not a legitimate argument. The bug is in the function making the bad call, not in the overriding logic. This is why I like Python, it will complain about unexpected arguments right away instead of ignoring them so they could become a bigger problem later.

Appending your own methods, on the other hand, is the most benign way to modify a native object. This is when we implement something like this, for example:

Array.prototype.copy = function() {
    return this.slice();
}

When doing, so, however, it’s important to be aware if other libraries are appending a method with the same name but different functionality, then you might want to pick a different name (this is usually not a problem with libraries/APIs since you’re unlikely to need more than one library for native object manipulation). So far, the most popular argument I’ve seen against this type of native object modification is potential name collision if future version of JavaScript decides to add a method by the same name. This argument is moot. You’re not developing your app for a hypothetical language that will exist 10 years from now, and when the time comes you’ll easily be able to rename the offending function, since you know all references to it are your own. The entire EcmaScript specification is easily available, and unless you plan to drop support for all browsers older than 6 months, the JavaScript implementation you’ll be using as basis will probably lag behind that by a few months to a few years, giving you more than enough time to handle naming collisions.

You might notice, however, that if you start appending methods to the native Object object, any jQuery code running on the same page will break. The problem is not with overwriting native methods, but with poor assumptions made by jQuery developers who wrote buggy code. The problem is that jQuery developers are part of the group that believe that appending to native JavaScript objects is evil, and instead of properly testing that they’re iterating through object’s own properties, they make the assumption that no one else will append anything to Object when they scan through it using “for key in Object” type logic. To avoid the same mistake as jQuery made, make sure to iterate only through your own keys:

for (var key in obj) {
    if obj.hasOwnProperty(key) {
        ...
    }
}

Alternatively, if you only care about the latest browsers, you can use Object.getOwnPropertyNames() instead. It was a poor design on JavaScript’s part to default to iterating through every property of the object, but that can’t be changed now. jQuery developers have been confronted about their assumption before, they claim the main reason was performance. Independent tests showed about 5% performance hit from adding this check, so I don’t buy the performance claim, especially since jQuery already makes multiple performance sacrifices in the name of usability (show/hide logic having safety checks to figure out the object’s current state, for example). In my opinion, John Resig’s stance on this is no different than failing to do a “division by zero” check (in Python, because JavaScript will automatically return infinity) and then claiming that it’s for performance reasons and anyone who passes arguments that eventually result in a division by zero is the one at fault.

I haven’t checked if this is fixed in jQuery 2.0. Without support for older browsers there is no reason not to use Object.getOwnPropertyNames(). In the meantime, try not to overwrite Object (other native objects are fine) if you’re using jQuery anywhere at all (if you’re not, it’s not a problem). I should also mention that if you don’t plan to support older browsers (such as IE8), a better practice would be to use defineProperty method rather than appending to the prototype, since it will omit the method from getting picked up when iterating through the object, making the jQuery bug a non-issue.

As far as my own stance, it’s fine to append to the functionality of a native object, but never to remove. Modifying existing property to work differently than before, given the same function signature, counts as removal, and is evil as well. I have removed the dependence on overwriting native methods in RapydScript’s stdlib2, but that was mainly to clean up the standard library and make it compatible with jQuery, not because I’m against modifying native objects.

RapydScript is Self-Hosting

While the RapydScript community is small, our members are passionate about the project, and I’m grateful for that. Salvatore, for example, has put together a number of demos showcasing RapydScript integration with WebGL, NodeBox, GlowScript, and a number of other JavaScript projects. Charles made a Chip’s Challenge clone. There are also examples of RapydScript integration with D3, and a Paint app that uses HTML5 canvas and jQuery. When we started a little over a year ago, members would ask us “can RapydScript work with ___?”. Now instead of just saying “yes” we can often point them to a demo.

To take things one step further, I decided to port the compiler itself (originally in JavaScript) to RapydScript as well. This effort took a few weeks, and a lot of tweaking to a Decompiler project that was put together a while ago. The decompilation wasn’t without issues, and I had to manually tweak some code. Along the way, I also discovered a few minor bugs in RapydScript, which I have fixed in the process. Overall, however, I was pleasantly surprised by how easy it was to port the code. The ported version is not written in Pythonic style that RapydScript tries to encourage because the code-generation was mostly automated, yet it already looks more legible than the JavaScript version it was ported from. I’m very proud of this, since it’s an important step for a compiler to achieve the self-hosting status, and I’m not yet aware of any other Python-to-JS compiler that can say the same (they’re all written in Python, but the subset of Python that the languages use isn’t enough to support the compiler itself).

I hope that this will make future enhancements and tweaks to RapydScript easier, both for myself and other members of the community. As a bonus, I’ve also added kwargs implementation to RapydScript. It works similar to Python, but not completely like it (for performance reasons), I suggest you read the manual on it before use to avoid surprises.

Scoping in JavaScript

I do a lot of programming in Perl, not because I like it, but because the company I work for uses it as its main language. In fact, I hate Perl (it tries to be overly implicit), but I do like how it handles scoping. Any variable declared inside a block (anything surrounded by {} brackets) will be local to that inner-most block. This means that variables declared inside loops, conditionals or even stand-alone {} will not be seen from the outside.

JavaScript will not localize variables like this. Any var declaration will simply be scoped to the inner-most function. That doesn’t mean, however, that you can’t use (function() {})(); same way you would use the brackets in Perl. In fact, you’re probably already familiar with wrapping chunks of code in the above pattern. Many developers do it to prevent leaking variables into global scope, including Facebook’s Like button:

(function(d, s, id) {
  var js, fjs = d.getElementsByTagName(s)[0];
  if (d.getElementById(id)) return;
  js = d.createElement(s); js.id = id;
  js.src = "//connect.facebook.net/en_US/all.js#xfbml=1";
  fjs.parentNode.insertBefore(js, fjs);
}(document, 'script', 'facebook-jssdk'));

Similar pattern, however, can also be applied anywhere else in your code. Consider a page, for example, with multiple elements sharing similar element ID structure, varying only by the index used within the ID. We then want to iterate through these elements, giving them all a click-handler. The following jQuery-based code seems like it will do the job:

for (var i=0; i<n; i++) {
    var cachedi = i;
    $('#' + i + '-element').click(function() {
        $('#' + cachedi + '-popup').show();
    })
}

At first glance, this code looks fine. We made sure to cache the index so that cachedi gets the value of i at the time of the function creation rather than using i directly, which would use i at the time of function call (after the loop terminates and i is set to n). However, running the above code we still get all elements attempting to trigger the popup with [n]-popup ID. The problem is that our declaration of cachedi gets moved outside the for loop and the same instance of the variable gets used in every single closure we generate inside the loop. There is an easy work-around, however:

for (var i=0; i<n; i++) {
    (function() {
        var cachedi = i;
        $('#' + i + '-element').click(function() {
            $('#' + cachedi + '-popup').show();
        })
    })();
}

Now our code works as expected. This is a handy trick for anyone wishing saner scoping in JavaScript. In fact, I’d prefer that RapydScript would scope things this way too, but that would contradict Python’s loop scoping. An alternative to this trick (and probably a more orthodox solution in JavaScript) would be to move the cachedi declaration inside the function making use of this closure.

Python to Javsacript: Compilers vs. Translators

One thing Alex and I always say about RapydScript is that it is really JavaScript with a Pythonic syntax. Following that, people often ask me what that means for them – they want to know how this affects how they develop code, and how it is different than something like Pyjamas/Pyjs. I want to answer that here to for anyone that has wondered what “RapydScript is Pythonic JavaScript” means, and how compilers like Pyjs are different from translators like RapydScript, and why I (full disclosure) prefer translators.

An Example

A clear example of the difference is with division. Say your source looks like:

a = 5
b = 0
c = a / b

The translator will output JavaScript like:

var a = 5;
var b = 0;
var c = a / b;

The translator process is very simple to understand – it’s pretty much just changing the syntax, but this leads to some gotcha’s. When this code runs, c will be set to Infinity, a JavaScript constant, while the original Pythonic source would have raised an Exception.

A compiler, on the other hand, attempts to mimic Python exactly so it may have an output more similar to:

var py_int = function(val){
    this.val = val;
};
py_int.prototype.div = function(denom){
    if (denom.val == 0) {
        throw ZeroDivisionError;
    }
    return py_int(self.val / denom.val);
};
var a = py_int(5);
var b = py_int(0);
var c = a.div(b);

The variables here will all be objects that include methods for all the operations. The division doesn’t directly divide 2 numbers, it runs the divide method in the objects. So when this code runs it will throw a ZeroDivisionError exception just like Python does.

So what are the tradeoffs?

Writing using a compiler is nice because you get to think like a Python developer, which can abstract away some things like cross browser support. It also means that, in many cases, code can be moved between the frontend and backend with no changes. So it’s easy to have Python code compile to Javascript. But if you’re doing something that’s JavaScript specific, like getting HTML elements, taking in keyboard inputs, etc, the compiler you’re using will have to have a working and documented API for accessing these functions.

The real drawback, though, is with the output code is slower, significantly heavier, and, with the compilers I’ve used, unreadable. There are several issues I have with this, but it really boils down unreadable code leads to usless tracebacks when running code in a browser, and apps don’t run (or run well) on mobile devices.

Translators take a very different approach and don’t try to run just like Python. The idea here is to do 80% of the work for 20% of the cost. Your code may look like Python but it will run like JavaScript, as with the division example. There’s a lot of overlap between the two languages, but they’re not exactly the same, so the main drawback here is that you might see some unexpected, but predictable, behavior. This is very easy though if you know JavaScript, and if not, it’s easy to learn the differences.

There are some nice benefits to using a translator. Your input code and output code will look very similar. They will be roughly the same size and it will be easy to map a line with an error in the JavaScript output back to the Pythonic input for easy debugging. The output will be on the order of kB instead of MB and will run faster.

So those are the main differences between compilers and translators like RapydScript. RapydScript is really JavaScript behind the scenes so it will behave differently than Python, which lets it run a lot more efficiently.

Which is right for you?

The choice of which to use comes down to a few things:

  • First, something I have not mentioned, libraries. In general, JavaScript libraries work better with translators and Python libraries work better with compilers. The one caveat is compilers can only translate Pure Python, so if you’re using something like numpy, which uses C, there’s no easy answer for you.
  • Second, if you don’t know any JavaScript, you will have a tougher time with a translator. Speaking from experience though, JavaScript is not very different from Python, and I encourage you to try a translator because you’ll save time debugging, and your app will be more maintainable in the long term.
  • Lastly is performance requirements. If performance is important, you will want to use a translator over a compiler.

I think it makes sense for a beginner who may writing a simple internal app that won’t have any performance requirements to use a compiler. But if you’ll be writing many apps, or even a complex one, I would go with a translator. Having picked up the differences between JavaScript and Python, I now exclusively use RapydScript, a translator, even if I don’t need the performance. It’s easier to debug in the browser where it actually runs, which saves me time.

Pyjamas Alternatives for Web Development

In my last post I mentioned that I’m switching away from Pyjamas. In this one, I will explain the alternatives I looked into, as well as cons and pros of each one. While doing my research (which started almost half a year ago), I’ve stumbled upon multiple Python-inspired alternatives to JavaScript, many of which have been abandoned, and only few of which have reached the stage where they’re suitable for large projects. Out of those I have settled on a solution that works for me, but there is no perfect framework/language, your choice could be different than mine. Let’s look at a few frameworks and decide how usable each one is for writing a full application.

Pyjamas

While I might have issues with how Pyjamas approaches web development, I have to admit that it has gotten quite far, and it’s quite possible to write a usable application with it.

Verdict: usable – if overhead is not a problem

Skulpt

Skulpt does not process your Python code at all until execution time, this is a unique approach that I haven’t seen any other framework take. It’s essentially like a browser-based eval() for your Python. This is a neat concept, since it can be used to extend another framework to support Python-like eval() (something that even Pyjamas doesn’t do), allowing your script to write and execute code on the fly. While I’m not a big fan of eval() function, this would definitely be neat for a framework that aims to achieve complete Python compatibility. Similarly, one of Skulpt’s major disadvantages is having to include the entire “compiler” as a script in your page, if this project ever grows to have the same functionality as Pyjamas, the overhead of loading the extra logic could be enormous. The second disadvantage is speed, Python interpreted and executed in the browser on the fly will never be as fast as even boilerplate-based Pyjamas, let alone pure JavaScript. The final problem is that you can’t obfuscate/compress your code at all, since its interpreter needs to make sense of your Python code (including the whitespace), this might be a non-issue to open-source projects.

Verdict: not usable

pyxc-js

Like Skulpt, this compiler is unsuitable for any large-scale web development, but it has an elegant import mechanism, which is something many other frameworks/compilers lack. Just like Skulpt can be used to implement eval(), this compiler can be used to implement proper import in your compiler (I know Pyjamas has proper import, but it’s heavily integrated into the rest of the framework). This project has been abandoned and the author does not respond to emails. Aside from its import mechanism (which is based on a directed graph the compiler builds at compile time, eliminating any unused and repeated imports), this project doesn’t have much to offer. The documentation is lacking, it took me a while (and some tweaks) to be able to compile anything, the included test cases don’t work, and the generated code is rather messy (it often spits out multiple semi-colons at the end of the line and blank lines get semicolons too, and if I remember correctly there are occasional bugs in how your code gets compiled). On the plus side, it offers an internal compression mechanism that can shorten your code further (although I’m not sure how much I trust it, given the quality of the non-compressed code it generates).

Verdict: not usable

PyCow

This was the first framework aside from Pyjamas that I felt could support large projects. PyCow compiles Python code into MooTools-based JavaScript. More importantly, its source code is cleanly laid out and the original developer is very responsive and helpful (despite having abandoned the project). It relies on templates for many of its conversions rather than hard-coding them in the source or implementing alternative names for JavaScript functions. For example, list.append() gets converted to list.push() at compile time using a template specified in a hash table. This is a great idea, since it makes the compiler code cleaner and the final output stays closer to real JavaScript, introducing less overhead (and less need for large stdlib). The disadvantage of this approach is that we have no good way of knowing whether a certain method is part of our language and needs to be replaced by JavaScript equivalent, or if it belonds to another API and should be left alone. Here is an example where replacing the append() would break our code:

container = $('#element'); // jQuery used to create a container of elements
...
container.append(item);    // replace this append() and jQuery will complain

PyCow is also MooTools’-based, and I’d prefer a language that doesn’t tie me to a certain framework. I investigated rewriting it such that it generates classes using pure JavaScript, but later realized it would not be worth the effort. One other minor pet-peeve is that PyCow is written for Python 2.5, which happened to be around the same time as Python developers were rewriting their AST parser. As a result, PyCow uses a transitional AST package and needs to be rewritten to support the later one that came after 2.6 (2.5 is still closer to the current AST, however, than the AST parser Pyjamas uses form compiler module, which has been deprecated as of Python 2.6). This framework also doesn’t support any importing of modules, but I managed to fix that in my local version by ripping out the import mechanism from pyxc-js. Unfortunately, I’ve moved on since then, abandoning my PyCow enhancements (let me know if you’re interested in continuing from where I left off, however, I can send you the source code).

Verdict: usable

py2js (Variation 1)

There are at least 3 different frameworks by this name, and they’re only vaguely related to each-other. The first one has been abandoned since 2009, but it impressed me for several reasons. First of all, it takes PyCow’s templating one step further, allowing you to map virtually any function/method using more complex filtering. For instance, %0 represents the class that the method is being called on, %1 reprensents the first argument, %* represents all of them. You could use this to add very powerful templating. For example, if you want to output a list of elements as a comma-delimited string in Python, you could use join() as follows: ', '.join(list). In JavaScript, however, join() is a method of array/list (which actually makes more sense to me) rather than a string, requiring you to do list.join(', ') instead. But say you wanted to auto-convert your Python joins to JavaScript, all you would have to do is add the following rule to your template hash:

'%0.join(%1)' : '%1.join(%0)'

I was able to extend this variation of py2js to support multiple Python features with JavaScript alternatives this way. As mentioned earlier, however, this method is susceptible to renaming functions whose names you have no control over.

This compiler relies on templates heavily, introducing another advantage most other compilers lack. Once the code has been converted to AST, it can be manipulated almost entirely using templates (with little need to write additional code). With some tweaks, it can become a universal source-to-source compiler (well, universal is a bit strong of a word here, statically typed languages are a whole other animal). Imagine being able to convert Python to Perl, Ruby, or any other dynamically typed language just by generating a map of logic equivalents. Something like this, for example, could allow you to convert the syntax of a Python function to Perl (block would then further get interpreted by similar templates for loops, conditionals, etc.):

### Input Template
def <method>(%*):
    <block>

### Output Template
sub <method>{
    my (%*) = @_;
    <block>;
}

Admittedly, the more fancy you get with the language (i.e. using eval()), the more likely you’re to run into problems with the template breaking. This compiler was inspired by py2py, another project designed to clean up Python code. In theory, any input languages can be supported as well, as long as you write your own AST parser that converts the code to Pythonic AST tree (or a subset of). This is a fun project in itself, and I might continue it when I have more time. As a compiler, however, this project is not yet suitable for prime-time, lacking a lot of functionality, including classes.

Verdict: not usable

py2js (Variation 2)

The only similarity between this py2js and the previous is the name. They share nothing else (well… they do now, keep reading). Variation 2 just happens to be another project by the same name as variation 1. In fact, even the compile mechanism for this py2js is very different from other compilers mentioned before. The compiler works by “running” your code at compile time. You write your function/class in Python, and then add @JavaScript decorator to it if you want it to be included in the compiled output. You then “run” your program and it dumps JavaScript to STDOUT. While it’s a bit awkward to add @JavaScript decorator to every single function, this offers something most other compilers (including Pyjamas) lack. Since it actually runs your code, you get the benefit of partial “compile-time” error catching, a feature typically only seen in statically-typed languages. You can be sure, for example, that all syntax errors will be caught, as well as undeclared variables/methods. The disadvantage is that this compiler is a bit too strict, all your code has to be pure Python and all your variables (or stubs for them) have to exist at compile time. So, for example, if you want to use something from jQuery, you better have a jQuery stub written in Python (it doesn’t need to mimic jQuery functionality, however, only to have the method names defined). This project has also been abandoned several years ago in a premature stage, which is where the 3rd variation comes in.

Verdict: not usable

py2js (Variation 3) (now Pyjaco)

The 3rd variation started off as a fork of the 2nd variation, after 3 developers decided to take over the original, abandoned, project. Over time, this variation morphed into a powerful compiler, whose functionality is only outdone by Pyjamas itself. It added support for Pythonic assertions, tuples, classes, static methods, getattr()/setattr(), *args/**kwargs, better integration with JavaScript (allowing you to call JavaScript logic by wrapping it inside JS() or automatically handling the conversion at compile time if you use one of predefined globals (document, window, etc.)) as well as many other features you would expect from Python. It has its own standard library with most Python stdlib implemented.

It, however, also falls short in a few areas. Like Pyjamas, it tries to build Pythonic error handling on top of JavaScript. Admittedly, this task might be much easier here, due to partial “compile-time” error catching, which Pyjamas lacks. It also doesn’t properly translate Python’s math methods to JavaScript, a task that shouldn’t be too hard, however.

Additionally, while it’s convenient to have document and window be treated like special classes, this can add confusion when coding. In some cases your strings and numbers will get converted to JavaScript equivalents automatically, while in the others you have to do so manually (when invoking a native JavaScript method and passing it a string, for example). A py2js string is a wrapper around native JavaScript string with additional Python functionality, same applies to many other objects. This is a good thing, since it allows support for Python’s methods without having to overwrite native objects (and I believe Pyjamas takes the same approach for its primitive objects (arrays/strings/etc.). This is a minor pet-peeve, however, and would not be a problem most of the time.

Another annoyance that I already mentioned earlier, is having to explicitly define which functions you want compiled via @JavaScript tag + str() call on the function or class (but this is just a matter of personal preference).

The biggest problem with this third variation, however, is copyright issues (which the developers are trying to resolve now – they could already have been resolved). In its quest to acquire similar functionality as Pyjamas, this compiler has “borrowed” a lot of code from other projects (including the first py2js I mentioned). This py2js project itself carries MIT license, not all of the projects it borrows from, however, are compatible with that license. Additionally, some of these projects were not given proper mention in the copyright notice due to an oversight by one of the developers. As a result, the developers had a falling out, and now forked a couple alternative versions of this project, one of which is still maintained and aims to rewrite the code in question and/or get proper authorization to use it.

Additionally, if this project joins forces with the new Pyjamas, I believe both projects will benefit a lot. The code generated by py2js is significantly smaller (Pyjamas averages 1:10 ratio for non-compiled:compiled code, while py2js is closer to 1:2 – not counting stdlib) and more readable than Pyjamas, while retaining most of Pyjamas’ features. Pyjamas, on the other hand, already has its own implementation for most of py2js code that’s subject to copyright issues, which could be used to solve the biggest problem with py2js. Finally, py2js’ compile-time error catching makes it easier to build Pythonic exception handling on top of JavaScript. And to remedy the annoyance of @JavaScript decorators, the AST parser can be used to append them automatically to temporary version of the code. This can also be used as an opportunity to update Pyjamas to use the latest AST parser implementation, which it will need anyway if it ever wants to be compatible with Python 3 (which is missing the deprecated AST implementation).

Verdict: usable

UPDATE: I’ve been informed that this project now became Pyjaco, and the copyright is no longer an issue. So for those who want to stay closer to Python, this is a very solid alternative. Christian, the project leader, also informed me that not all of the details I mentioned are accurate. Also, apparently the developers seem to have misinterpreted the original post as me claiming that RapydScript (my own compiler) is the best for everything. That was not my intent, and I tried to avoid this issue by mentioning in the first paragraph of the original article that my choice is based on my own projects and the flexibility they need (mostly Grafpad), even stating that “your choice could be different than mine”. I hope they don’t hold a grudge at me because of this misunderstanding.

PyvaScript

PyvaScript is the opposite extreme of Pyjamas. It’s perhaps the closest to pure JavaScript out of all other Python-like languages. It was one of the first alternatives I looked into, and admittedly I originally decided it would be unsuitable for large projects. There are numerous disadvantages compared to other Python frameworks. Its stdlib is puny, and most of the Python methods you would want aren’t available. It not only lacks import logic, but even classes aren’t implemented. It can’t handle negative indexes for arrays. Its compiler isn’t even based on Python AST (it uses Ometa), as a result it’s often oblivious to errors other compilers would catch, and at other times it chokes on issues that other compilers have no problems with.

Its advantages, however, provide the perfect fit for the niche that Pyjamas left open. It’s the most JavaScript-compatible solution, supporting almost all JavaScript functionality out of the box. I can manipulate other JavaScript and DOM without the need of any wrappers or special syntax. Need to use jQuery? Just do it the same way you would in JavaScript: $(element).method(), the compiler won’t complain about the dollar sign. PyvaScript also supports anonymous functions, something regular Python lacks (except 1-liner lambdas), which makes it easier to write your code like JavaScript (I didn’t see the advantage to this at first, but believe me, when writing web apps, it really helps – especially when using other JavaScript logic as a tutorial).

Since PyvaScript adds no bells or whistles to your code, or requires any sort of wrappers, your code behaves a lot more like native JavaScript, allowing you to use closures more effectively and to bind functions to objects dynamically without the language complaining. At first this seemed like a bad idea, but after playing with it, I realized that with proper use of this, I can actually write cleaner code than with Python itself. Most importantly, PyvaScript is extremely light-weight, adding almost no overhead to the generated code and does not force a framework (MooTools, jQuery, etc.) on the user. I also realized, that PyvaScript’s lazy compilation has its own advantages (which I will explain later).

Verdict: usable – but doesn’t alleviate much pain from plain JS

CoffeeScript

Sharing a structure similar to that of Python, it deserves a mention as well. If you ignore its ugly syntax, and poor choice of variable scoping (preferring globals over locals), you will see that it has all the same advantages as PyvaScript, which makes it a good candidate for Pyjamas replacement as well. It has similar feel to Python (although it feels closer to Ruby), introduces list comprehensions, and even adds classes (something PyvaScript does not). It also adds namespaces, preventing variables in different modules from interfering. If I invert the scoping design, remove all the junk variables like on/off (synonyms for true/false), and modify the syntax to use Python tokens, this will be my ideal language for web development, but that’s a project for later.

Verdict: usable

And the winner is…

RapydScript, which I didn’t even mention yet. RapydScript (best described as PyvaScript++) is my wrapper around PyvaScript that provides the best of both worlds (for me at least). I’ve made a few enhancements (abusing PyvaScript’s lazy compilation to allow me to auto-generate broken Python code that becomes proper JavaScript once compiled by PyvaScript) that achive almost everything I would want, and just about all of CoffeeScript’s functionality. Some of the new RapydScript features include support for Pythonic classes including inheritance (single inheritance only, but you can bind functions from any class to fake multiple inheritance), Pythonic optional function arguments, anonymous functions now supported in dictionaries/object literals (something PyvaScript chokes on), beefed up stdlib (also optimized already implemented methods from PyvaScript’s stdlib), support for multi-line comments/doc-strings, preliminary (compile-time) checking for syntax errors and issues that PyvaScript chokes on (because of minor bugs in PyvaScript), module importing (currently everything gets dumped into a single namespace, but compiler does warn about conflicting names), checking proper use of Math functions, automatic insertion of ‘new’ keyword when creating an object (not sure why CoffeeScript doesn’t already do the same).

To me, the main advantages of RapydScript over PyvaScript are ability to break down my modules into separate files (like I would in Python), easier time to build large projects due to proper class implementation (class declaration is done the same way as in native Python), Pythonic declaration of optional arguments to a function (I’m not a big an of JavaScript’s solution for optional arguments), and support for anonymous functions as hash values (which allows me to build object literals the same way as in JavaScript). As for other projects, the main advantages of RapydScript are seamless integration with the DOM and other JavaScript libraries/modules (just treat them like regular Python objects), ability to use both Python and JavaScript best practices as well as rely on JavaScript tutorials (one of the biggest problems for projects in their alpha stage is lack of documentation, for RapydScript you really don’t need any), and lack of bloat (RapydScript gets me as close to Python as I need without forcing any boilerplate on me). As a nice bonus, I think I can add suppport for advanced compilation mode of Google’s Closure compiler with only minor tweaks.

If you want to give RapydScript a try, you can find it in the following repository: https://bitbucket.org/pyjeon/rapydscript. Keep in mind, this is still early alpha version and some of the functionality might change. Chances are, I will eventually rewrite this, using CoffeeScript as base, but the functionality should stay very similar (after all, I am rewriting Grafpad in this, and I don’t want to rewrite it a third time). In my next post, I will dive into more detail about RapydScript, for those interested in using it and needing a tutorial.