Equality Operator

Matrix-agent-Smith-clones One thing that always bothered me, and many others, is how useless the == operator is in JavaScript. It seems to combine the worst of both worlds from equality and identity:

  • When comparing primitive types, it acts as a lax equality operator
  • When comparing objects, it acts as an identity operator

I can’t think of any use case for which such behavior is desired. In fact, reversing this behavior would produce a much more useful operator since the concept of “equals” tends to relax as data gets more complex because some dimensions are simply irrelevant. Do I care that two objects were not created at the same exact time? Do I care that their IDs differ? Do I care if the measured length is 4.00000000001 inches rather than 4? When it comes to equality comparison, JavaScript fails miserably. This is actually one of Douglas Crockford’s pet peeves about the language, which he explains in more detail in his book: JavaScript: The Good Parts.

Fortunately, there is also a more consistent identity operator (===), often erroneously called “strict equality” by people who don’t understand the difference between identity and equality. Unfortunately, as I just mentioned in previous paragraph, there is no concept of equality operator in the language at all. For those unfamiliar with these concepts, identity answers the question of “Does A and B refer to the same object?” while equality answers the question of “Is B a clone of A?“.

Naturally, both of these questions come up in programming a lot, so it’s a shame that only one can be easily answered in JavaScript. You probably already stumbled into this problem if you ever tried something like this:

new Date('1/1/2000') == new Date('1/1/2000')

or this:

{"foo": 1} == {"foo": 1}

The problem is not unique to JavaScript, many other languages lack equality operator as well. Typically, however, those languages have an alternative mechanism for handling this case, such as operator overloading. Indeed, there are cases when operator overloading is actually superior – such as when objects contain meta data that one wishes to omit from comparison (i.e. exact creation time, unique id, etc.). Unfortunately, JavaScript doesn’t allow for operator overloading either, and herein lies the problem. While one can easily roll a proper equality function (and many libraries such as underscore already include one) you would still have to decide whether it’s worth using on a case-by-case basis.

We’re not in FORTRAN age anymore, where developers had to tweak each operation. Today we’re spoiled, we often can get away with simply telling the compiler what to do rather than how, and enjoy optimal performance 90% of the time with negligible overhead. One such example is the sort function. When was the last time you had to wonder if you should use merge or insertion sort? You simply use the built-in sort and assume that unless there is something very special about your data, you’re better off moving on to other things rather than attempting to optimize the last 10% out of the algorithm. In most modern languages equality operator falls in the same category. Sure, you’d be able to shave off a few microseconds by replacing that == (equality) with is (identity) for certain comparison operations in Python, but is it worth the extra brain cycles? Most of the time the answer is “No”.

Why then do we have to be explicitly aware of this in JavaScript? Moreover, why can’t RapydScript compile == into deep equality? First, I should mention that, like some other libraries/languages for JavaScript, RapydScript already has a deep equality comparison via the inbuilt eq function. Yes, eq(a, b) (previously called deep_eq) has been supported for years. The problem is that (up until recently) I wasn’t able to make the decision for you of whether you want eq or ==. The issue boils down to the fact that if I decide to compile == to equality across the board for the developer, I effectively introduce enormous overhead (about 700% according to jsperf) on the user in cases where he/she expected identity comparison instead (which is about 90% of the time, since primitives are a lot more common). There has been a lot of discussion about this, which you can follow in issues 93 and 94 on my github page.

As you probably already guessed from previous paragraph, I’m now able to bridge that gap. Effective last month (that’s right, I snuck a change in without anyone noticing), RapydScript is the first JavaScript-based transcompiler to support high-performance deep equality. How performant is this operation, you may ask? Well, take a look for yourself:

Screen Shot 2015-12-05 at 2.59.38 PM

According to jsperf, the overhead is negligible (that’s right, the measurement noise is greater than any visible difference – as you can see from the deep-equality version outperforming identity). So what changed? Why am I suddenly able to blow the doors off of typical deep equality? Well, nothing new happened in JavaScript world. What changed is my approach. I decided to think about the problem creatively and had a sudden eureka moment (which I’m sure other compilers will copy in the future). Unfortunately you’ve probably already looked at the code from JsPerf above, ruining my surprise. But in case you haven’t, here is the pattern I came up with:

A === B || typeof A === "object" && eq(A, B)

How does it work? As you probably learned in your introductory programming class (assuming it was a legitimate C/Java class rather than an online tutorial about JavaScript), binary operators have short-circuit ability in just about all languages. This means that if left-hand side is truthy for || (or) operator, or falsy for && (and), the rest of the line is ignored (it’s as if it’s not there). That means that if A and B are primitives that are equal, the above will be equivalent to a simple A === B comparison. That handles the positive case, but negative is a bit trickier. I was struggling with it for a while (yes, the above equation makes it seem simpler than it really is – everything seems easy in hindsight), until I found a performant operation that works in both, browser and node (my first attempt was to check if A.constructor exists, which doesn’t work on all platforms). Fortunately, the very same “feature” that makes typeof useless in most cases (the fact that every non-primitive is an object) becomes the saving grace of this operation. As you can see from the JsPerf test, the overhead for this operation is negligible as well.

The best part is that unlike native JavaScript, where the developer would have to type that out by hand (because hiding it in a function call introduces the 700% overhead we’re trying to avoid), RapydScript can unroll == operator into that magic automatically. You’re probably wondering, then, how is it that this change has been in RapydScript for over a month if the == still compiles to ===? Well, for safety I’ve hidden this operator behind an import. If you wish all your == operators to compile to proper equality test shown above, add the following line at the top of your file:

from danger_zone import equality

That’s it. Now all your == will compile to the magic shown above, and != will compile to its inverse (thanks to DeMorgan’s Law). The above import will also tweak implementation of indexOf and lastIndexOf to perform deep equality as well, which in turn makes tests like if a in arr be based on deep equality (delivering a consistent experience across the board). As before, the identity operator is still there as well, via is and is not, which compile to === and !==, respectively. In the future, I also want to add an optimizer to the compiler, which will strip away the remainder of the line at compile time if one of the operands is a constant.

Using Angular Rapydly

RapydScript with AngularJS I’ve been asked a few times if RapydScript works with Angular. The answer is the same as with any other JavaScript framework/library/etc. – of course! RapydScript takes the same approach to compilation/abstraction as CoffeeScript, so there really isn’t anything JavaScript can do that RapydScript can’t. Salvatore has already proven this with multiple demos using frameworks I haven’t even thought of. I urge any developer with similar questions to just try it out, it’s easier (and more fun) to spend an hour building an app than hesitating about it for weeks.

With that said, I decided to put together this tutorial by cloning the 5 examples shown here via RapydScript. I suggest you read the original article before this post. I should also mention that I myself have not used AngularJS before, and decided to learn it through these examples as well. I was a bit disappointed to see that a big chunk of AngularJS is handled magically from their html templates rather than JavaScript itself. This means that there is very little actual JavaScript (and hence RapydScript) used. As far as html itself, you could either write it directly, or use RapydML or Jade (both will have similar syntax). For the purposes of this post, I will leave css code alone, although you can easily convert it to sass as well. Let’s get started.

Setup (Only Needed for RapydML)

Looking at the first example, we immediately notice Django-like variables in the html code. That’s a sign that we may want to use template engines in RapydML. So let’s slap one together. Glancing through the examples we see that most AngularJS variables follow this format:

{{var}}

Doesn’t look like they’re doing anything fancy, so let’s create a very simple template engine in a new pyml file:

angular = TemplateEngine('{%s}')
angular.js = create('{%s}')

I should also mention that you can use AngularJS from RapydML directly, without creating a template engine, I chose to do so for clarity. But nothing prevents you from writing this:

div:
    'This is an angular var: {{myvar}}'

Example 1 (Navigation Menu)

This one doesn’t require any JavaScript. We’ll just need to slap together a RapydML template, leveraging the 2-line template engine we just wrote:

import angular

div(id="main", ng-app):
    nav(class="angular.js(active)", ng-click="\$event.preventDefault()"):
        for $name in [home, projects, services, contact]:
            a(href='#', class="$name", ng-click="active='$name'"):
                python.str.title('$name')

    p(ng-hide='active'):
        'Please click a menu item'
    p(ng-show='active'):
        'You choose'
        b:
            "angular.js(active)"

That’s it, compile it and plug into AngularJS. You will need the most recent version of RapydML, older versions did not handle template engine calls within strings, the new one handles them in double-quoted strings and ignores them in single-quoted strings.

Example 2 (Inline Editor)

As before, we start with the RapydML code:

import angular

div(id='main', ng-app, ng-controller='InlineEditorController', ng-click='hideTooltip()'):
    div(class='tooltip', ng-click='\$event.stopPropagation()', ng-show='showtooltip'):
        input(type='text', ng-model='value')
    p(ng-click='toggleTooltip(\$event)'):
        "angular.js(value)"

Next, let’s add some RapydScript:

def InlineEditorController($scope):
    $scope.showtooltip = False
    $scope.value = 'Edit me.'

    $scope.hideTooltip = def():
        $scope.showtooltip = False

    $scope.toggleTooltip = def(e):
        e.stopPropagation()
        $scope.showtooltip = not $scope.showtooltip

Once again, after compiling the first block of code via RapydML and second via RapydScript, we’ll have a fully-working example.

Example 3 (Order Form)

In this example, we see a new way of hooking into AngularJS. Note the {active: service.active} class tag. We could either reference is in the code verbatim (since it is quoted), or add a new method to the angular template engine for consistency:

angular.active = create('active: %s')

We can now continue with the 3rd example, first the RapydML code:

import angular

form(ng-app, ng-controller='OrderFormController'):
    h1:
        'Services'
    ul:
        li(ng-repeat='service in services', ng-click='toggleActive(service)', ng-class="angular.active(service.active)")
        "angular.js(service.name)"
        span:
            "angular.js(service.price | currency)"

    div(class='total'):
        'Total: '
        span:
            "angular.js(total() | currency)"

Finally, let’s complement it with RapydScript:

def OrderFormController($scope):
    $scope.services = [
        {
            name: 'Web Development',
            price: 300,
            active: True
        }, {
            name: 'Design',
            price: 400,
            active: False
        }, {
            name: 'Integration',
            price: 250,
            active: False
        }, {
            name: 'Training',
            price: 220,
            active: False
        }
    ]

    $scope.toggleActive = def(s):
        s.active = not s.active

    $scope.total = def():
        total = 0
        angular.forEach($scope,services, def(s):
            nonlocal total
            if s.active:
                total += s.price
        )
        return total

As usual, looks similar to the JavaScript version. RapydScript did prevent us from shooting ourselves in the foot by accidentally declaring total as global, something you’d need to be careful about in native JavaScript version.

Example 4 (Instant Search)

For this example, I’ll actually show alternative RapydML code, that doesn’t use our template engine:

div(ng-app='instantSearch', ng-controller='InstantSearchController'):
    div(class='bar'):
        input(type='text', ng-model='searchString', placeholder='Enter your search terms')

    ul:
        li(ng-repeat='i in items | searchFor:searchString')
            a(href='{{i.url}}'):
                img(ng-src='{{i.image}}')
            p:
                '{{i.title}}'

That was even easier than the template engine version, I think I’ll stick to this for my next example as well. Now the RapydScript portion:

app = angular.module('instantSearch', [])
app.filter('searchFor', def():
    return def(arr, searchString):
        if not searchString:
            return arr

        result = []
        searchString = searchString.toLowerCase()
        angular.forEach(arr, def(item):
            if item.title.toLowerCase().indexOf(searchString) != -1:
                result.push(item)
        )
        return result
)

def InstantSearchController($scope):
    $scope.items = [
        {
            url: 'http://tutorialzine.com/2013/07/50-must-have-plugins-for-extending-twitter-bootstrap/',
            title: '50 Must-have plugins for extending Twitter Bootstrap',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/07/featured_4-100x100.jpg'
        }, {
            url: 'http://tutorialzine.com/2013/08/simple-registration-system-php-mysql/',
            title: 'Making a Super Simple Registration System With PHP and MySQL',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/08/simple_registration_system-100x100.jpg'
        }, {
            url: 'http://tutorialzine.com/2013/08/slideout-footer-css/',
            title: 'Create a slide-out footer with this neat z-index trick',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/08/slide-out-footer-100x100.jpg'
        }, {
            url: 'http://tutorialzine.com/2013/06/digital-clock/',
            title: 'How to Make a Digital Clock with jQuery and CSS3',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/06/digital_clock-100x100.jpg'
        }, {
            url: 'http://tutorialzine.com/2013/05/diagonal-fade-gallery/',
            title: 'Smooth Diagonal Fade Gallery with CSS3 Transitions',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/05/featured-100x100.jpg'
        }, {
            url: 'http://tutorialzine.com/2013/05/mini-ajax-file-upload-form/',
            title: 'Mini AJAX File Upload Form',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/05/ajax-file-upload-form-100x100.jpg'
        }, {
            url: 'http://tutorialzine.com/2013/04/services-chooser-backbone-js/',
            title: 'Your First Backbone.js App – Service Chooser',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/04/service_chooser_form-100x100.jpg'
        }
    ]

On to the last example.

Example 5 (Switchable Grid)

First the RapydML code, as usual:

def layout($type, $imgtype):
    ul(ng-show="layout == '$type'", class="$type"):
        li(ng-repeat='p in pics'):
            a(href='{{p.link}}', target='_blank'):
                img(ng-src="{{p.images.$imgtype.url}}")

div(ng-app='switchableGrid', ng-controller='SwitchableGridController'):
    div(class='bar'):
        for $icon in [list, grid]:
            a(class='$icon-icon', ng-class="{active: layout == '$icon'}", ng-click="layout = '$icon'")
    layout(list, low_resolution)
    layout(grid, thumbnail)
            p:
                '{{p.caption.text}}'

Notice that I chose to be a little more DRY and create an extra function, you don’t have to do that if you believe it hurts readability. Next, the RapydScript code:

app = angular.module('switchableGrid', ['ngResource'])
app.factory('instagram', def($resource):
    return {
        fetchPopular: def(callback):
            api = $resource('https://api.instagram.com/v1/media/popular?client_id=:client_id&callback=JSON_CALLBACK', {
                client_id: '642176ece1e7445e99244cec26f4de1f'
            }, {
                fetch: { method: 'JSONP' }
            })

            api.fetch(def(response):
                callback(response.data)
            )
    }
)

def SwitchableGridController($scope, instagram):
    $scope.layout = 'grid'
    $scope.pics = []
    instagram.fetchPopular(def(data):
        $scope.pics = data
    )

Conclusion

While the RapydScript code above might be a bit cleaner than the original JavaScript code, it’s hard to argue that significant value was gained from porting it. The main benefit of RapydScript comes into play when building larger-scale applications that leverage classes and use numerous variables. Such code increases the chances of shooting yourself in the foot or getting confused when dealing with native JavaScript. Indeed, if you look at some of the jQuery plugins, you’ll see that the code itself seems unmaintainable. After a few layers of anonymous functions it’s hard to figure out what’s being passed around where. AngularJS keeps the code short by handling a lot of the logic for you on the background, RapydScript allows you to keep your own logic readable as it grows. No matter how awesome a framework is, eventually you’ll be writing a lot of your own logic as well.

RapydScript is Self-Hosting

While the RapydScript community is small, our members are passionate about the project, and I’m grateful for that. Salvatore, for example, has put together a number of demos showcasing RapydScript integration with WebGL, NodeBox, GlowScript, and a number of other JavaScript projects. Charles made a Chip’s Challenge clone. There are also examples of RapydScript integration with D3, and a Paint app that uses HTML5 canvas and jQuery. When we started a little over a year ago, members would ask us “can RapydScript work with ___?”. Now instead of just saying “yes” we can often point them to a demo.

To take things one step further, I decided to port the compiler itself (originally in JavaScript) to RapydScript as well. This effort took a few weeks, and a lot of tweaking to a Decompiler project that was put together a while ago. The decompilation wasn’t without issues, and I had to manually tweak some code. Along the way, I also discovered a few minor bugs in RapydScript, which I have fixed in the process. Overall, however, I was pleasantly surprised by how easy it was to port the code. The ported version is not written in Pythonic style that RapydScript tries to encourage because the code-generation was mostly automated, yet it already looks more legible than the JavaScript version it was ported from. I’m very proud of this, since it’s an important step for a compiler to achieve the self-hosting status, and I’m not yet aware of any other Python-to-JS compiler that can say the same (they’re all written in Python, but the subset of Python that the languages use isn’t enough to support the compiler itself).

I hope that this will make future enhancements and tweaks to RapydScript easier, both for myself and other members of the community. As a bonus, I’ve also added kwargs implementation to RapydScript. It works similar to Python, but not completely like it (for performance reasons), I suggest you read the manual on it before use to avoid surprises.

Python to Javsacript: Compilers vs. Translators

One thing Alex and I always say about RapydScript is that it is really JavaScript with a Pythonic syntax. Following that, people often ask me what that means for them – they want to know how this affects how they develop code, and how it is different than something like Pyjamas/Pyjs. I want to answer that here to for anyone that has wondered what “RapydScript is Pythonic JavaScript” means, and how compilers like Pyjs are different from translators like RapydScript, and why I (full disclosure) prefer translators.

An Example

A clear example of the difference is with division. Say your source looks like:

a = 5
b = 0
c = a / b

The translator will output JavaScript like:

var a = 5;
var b = 0;
var c = a / b;

The translator process is very simple to understand – it’s pretty much just changing the syntax, but this leads to some gotcha’s. When this code runs, c will be set to Infinity, a JavaScript constant, while the original Pythonic source would have raised an Exception.

A compiler, on the other hand, attempts to mimic Python exactly so it may have an output more similar to:

var py_int = function(val){
    this.val = val;
};
py_int.prototype.div = function(denom){
    if (denom.val == 0) {
        throw ZeroDivisionError;
    }
    return py_int(self.val / denom.val);
};
var a = py_int(5);
var b = py_int(0);
var c = a.div(b);

The variables here will all be objects that include methods for all the operations. The division doesn’t directly divide 2 numbers, it runs the divide method in the objects. So when this code runs it will throw a ZeroDivisionError exception just like Python does.

So what are the tradeoffs?

Writing using a compiler is nice because you get to think like a Python developer, which can abstract away some things like cross browser support. It also means that, in many cases, code can be moved between the frontend and backend with no changes. So it’s easy to have Python code compile to Javascript. But if you’re doing something that’s JavaScript specific, like getting HTML elements, taking in keyboard inputs, etc, the compiler you’re using will have to have a working and documented API for accessing these functions.

The real drawback, though, is with the output code is slower, significantly heavier, and, with the compilers I’ve used, unreadable. There are several issues I have with this, but it really boils down unreadable code leads to usless tracebacks when running code in a browser, and apps don’t run (or run well) on mobile devices.

Translators take a very different approach and don’t try to run just like Python. The idea here is to do 80% of the work for 20% of the cost. Your code may look like Python but it will run like JavaScript, as with the division example. There’s a lot of overlap between the two languages, but they’re not exactly the same, so the main drawback here is that you might see some unexpected, but predictable, behavior. This is very easy though if you know JavaScript, and if not, it’s easy to learn the differences.

There are some nice benefits to using a translator. Your input code and output code will look very similar. They will be roughly the same size and it will be easy to map a line with an error in the JavaScript output back to the Pythonic input for easy debugging. The output will be on the order of kB instead of MB and will run faster.

So those are the main differences between compilers and translators like RapydScript. RapydScript is really JavaScript behind the scenes so it will behave differently than Python, which lets it run a lot more efficiently.

Which is right for you?

The choice of which to use comes down to a few things:

  • First, something I have not mentioned, libraries. In general, JavaScript libraries work better with translators and Python libraries work better with compilers. The one caveat is compilers can only translate Pure Python, so if you’re using something like numpy, which uses C, there’s no easy answer for you.
  • Second, if you don’t know any JavaScript, you will have a tougher time with a translator. Speaking from experience though, JavaScript is not very different from Python, and I encourage you to try a translator because you’ll save time debugging, and your app will be more maintainable in the long term.
  • Lastly is performance requirements. If performance is important, you will want to use a translator over a compiler.

I think it makes sense for a beginner who may writing a simple internal app that won’t have any performance requirements to use a compiler. But if you’ll be writing many apps, or even a complex one, I would go with a translator. Having picked up the differences between JavaScript and Python, I now exclusively use RapydScript, a translator, even if I don’t need the performance. It’s easier to debug in the browser where it actually runs, which saves me time.

RapydScript II

Building New Compiler A while back I mentioned that I wanted to rewrite RapydScript to rely on an internal AST (Abstract Syntax Tree) structure rather than PyMeta, which would allow it several advantages, including code transformations prior to output, more consistent parsing, and better error detection/handling.

I have looked at several tools for helping me accomplish this. One of the most promising seemed to be CoffeeScript, that is until I actually started converting its source code into RapydScript (via an automated AST-to-RapydScript generating script I put together). Having discovered that the output is very ugly and unpythonic, I gave up on the idea. After some more searching, however, I found the right tool for the job, ironically it was UglifyJS, a tool designed to make your JavaScript less readable.

UglifyJS code-base is very clean, and although it uses its own format that doesn’t directly translate into Python, the logic is well-structured and it’s easy to see where the code could be replaced by Python/RapydScript constructs. The code does a good job being explicit rather than relying on various JavaScript hacks, like CoffeeScript does.

Using UglifyJS as a base, I rewrote the compiler to convert RapydScript into native JavaScript. You can get it here: https://github.com/atsepkov/RapydScript. There are a couple minor features from the original compiler RapydScript II doesn’t support yet, but it’s already better than the original in several ways:

  • It compiles code much faster (on the order of milliseconds instead of seconds)
  • It handles leading whitespace much better now (same way as native Python would), so it will not complain if comments or secondary lines mix spaces and tabs
  • It will properly localize outer-scope variables when using module wrapper
  • It’s a single-pass compiler that uses consistent logic to identify tokens, whereas the original RapydScript would sometimes try to perform the same logic in Python stage that it would later do using PyMeta, resulting in potential parsing inconsistencies in certain special cases
  • It supports some notations that the original does not (such as the code shown later in this post)
  • It handles if val in array the same way Python would, rather than JavaScript, checking for existence of the value rather than the index
  • It’s more resistant to bad code, reporting errors that are more relevant and identifying the exact line/column number that caused the error
  • It’s more resistant to valid code that doesn’t follow typical syntax conventions, parsing it correctly rather than erroring out
  • The class-parsing logic has been improved to allow better durability and some requirements of the original compiler no longer apply (the __init__ function no longer needs to be first in the class body, for example, classes can now be nested and even declared inside functions)
  • It is written in JavaScript, which means it can eventually be ported to native RapydScript, it also means it can be modified to run in a browser, and that we can actually test function rather than structure in our unit and integration tests
  • It’s AST-aware, so it doesn’t need to generate excessive parentheses in the output for safety like the original compiler
  • Since the compiler is AST-based, it doesn’t need to generate output at the time of parsing, which makes the compiler more flexible and allows various code transformations and analysis to take place before the output is produced. This also makes it easier to make changes to the way output is generated in the future, should we need to do so.

The compiler already supports most of the features of the original, the only features that aren’t yet supported are:

  • (UPDATE: these now work) Chained Comparisons (1 < x < 5)
  • (UPDATE: these now work) List Comprehensions
  • (UPDATE: these now work) Inline Functions

Everything else should work at least as well as the original, and in some cases much better. For example, you can now do the following:

(def(a):
    return a
)(1)

Here is a list of other features that the original compiler does not support:

  • Logic such as 5 in [5,6,7] now correctly returns True
  • stdlib is no longer required for using for loops, len() or print(), more functions will be moved out of stdlib with time
  • Negative array indices now compile to arr[arr.length-1] instead of using slices, this allows assignment to them and not just referencing them (you still can’t use variables as negative indices)
  • Simple minifier is now built into the compiler (it will remove whitespace and unnecessary punctuation)
  • Pythonic imports are now supported, use them via --namespace-imports flag (experimental, some code will probably break, star-imports are not supported yet, you can set variables in global scope but not modify them)

I can’t yet guarantee that your code will work correctly with RapydScript II, but all RapydScript II test cases (which are more rigorous than original RapydScript tests, in that they also test that the code performs correct logic after compilation) as well as the examples from original (with slight modifications to remove the unsupported features) seem to work. And as a present for those afraid of GPL, I’m releasing the entire compiler for RapydScript II under Apache 2.0 license.

RapydScript by Example

With RapydScript gaining popularity, I figured it’s about time for a blog post talking about an example app. After all, I learn fastest by observing examples, many others probably do as well.

Let’s imagine we wanted to write a game using RapydScript. Since I’m not feeling particularly creative today, let’s concentrate on an existing game, rather than coming up with an idea from scratch. For this example, I’ll use a game called Chip’s Challenge.

Chip's Challenge

Looking at the original game, we quickly notice that the environment is a grid of blocks. Each block type has a corresponding image, and some have an effect on the character stepping on them. If we assume that we’ll use a canvas element for drawing the grid, then each block will need to track its coordinates in the grid and the url of the image corresponding to this block. Let’s assume we created a canvas with id #canvas in our document using html. We now simply need to create a reference to it in RapydScript:

CANVAS = document.getElementById('canvas').getContext('2d')

Since canvas is not only an object, but also a module containing methods for creating canvas-compatible objects (like images and patterns), we will use a global to reference it rather than properly abstracting it. Next, let’s create a simple block:

BLOCK_SIZE = 32 #pixels
NUM_X_BLOCKS = 21 # horizontal blocks on a field
NUM_Y_BLOCKS = 21 # vertical blocks
NORMAL_BLOCK = 0 # enum identifying block type

class Block:
    def __init__(self, x, y, image_url):
        self.x = x
        self.y = y
        img = new Image()
        img.src = image_url
        self.type = NORMAL_BLOCK
        self.blockPattern = CANVAS.createPattern(img)

    def redraw(self):
        CANVAS.save()
        CANVAS.fillColor = self.blockPattern
        CANVAS.fillRect(self.x*BLOCK_SIZE, self.y*BLOCK_SIZE, BLOCK_SIZE, BLOCK_SIZE)
        CANVAS.restore()

Now that we have a basic block, let’s create advanced blocks that can affect chip in some way. For example, Chip’s Challenge has a water block that Chip can drown in unless he’s wearing flippers. There is also an ice block, that makes chip slide to the next one unless he’s wearing ice skates. Let’s create these two blocks:

WATER_BLOCK = 1
ICE_BLOCK = 2

class WaterBlock(Block):
    def __init__(self, x, y):
        Block.__init__(self, x, y, 'images/water.jpg')
        self.type = WATER_BLOCK

    def effect(self, unit):
        if not unit.hasItem('flippers'):
            unit.die() # drown

class IceBlock(Block):
    def __init__(self, x, y):
        Block.__init__(self, x, y, 'images/ice.jpg')
        self.type = ICE_BLOCK

    def effect(self, unit):
        if not unit.hasItem('skates'):
            unit.move(unit.direction) #slip

Let’s add a character that can move around. We will use Block as a base class since a character is really just another block drawn on top of the existing block. Unlike a regular block, the picture could change depending on the direction the character is facing:

LEFT = 0
UP = 1
RIGHT = 2
DOWN = 40

class Character(Block):
    def __init__(self, x, y, images):
        self.patterns = [CANVAS.createPattern(img) for img in images]
        self.direction = DOWN # default facing direction
        self.blockPattern = self._patterns[self.direction]

    def move(self, direction):
        self.direction = direction
        self.blockPattern = self.patterns[direction]
        if direction == DOWN:
            self.y += 1
        elif direction == UP:
            self.y -= 1
        elif direction == LEFT:
            self.x -= 1
        else:
            self.x += 1
        self.redraw()

This character can now serve as the base class both for Chip himself and the monsters that roam around. If we wanted to create a typical monster, for example, we would write:

MONSTER = 10

class Monster(Character):
    def __init__(self, x, y, redrawCallback):
        images = [Image() for i in range(4)]
        images[DOWN].src = 'image/monsterDown.jpg'
        images[UP].src = 'image/monsterUp.jpg'
        images[LEFT].src = 'image/monsterLeft.jpg'
        images[RIGHT].src = 'image/monsterRight.jpg'
        self.type = MONSTER
        Character.__init__(self, x, y, images)

        # have the monster move randomly every 0.5 seconds
        main = self
        window.setInterval(def(): main.move(Math.round(Math.random()*4)); redrawCallback();, 500)

    def die(self):
        pass    # monsters don't die

Similarly, let’s create chip:

CHIP = 11

class Chip(Character):
    def __init__(self, die=False):
        # center Chip
        self.x = int(NUM_X_BLOCKS/2)+1
        self.y = int(NUM_Y_BLOCKS/2)+1
        self.items = {}

        if not die:
            self.deaths = 0
            images = [Image() for i in range(4)]
            images[DOWN].src = 'image/chipDown.jpg'
            images[UP].src = 'image/chipUp.jpg'
            images[LEFT].src = 'image/chipLeft.jpg'
            images[RIGHT].src = 'image/chipRight.jpg'
            self.type = CHIP
        Character.__init__(self, x, y, images)

    def die(self):
        self.deaths += 1
        self.__init__(True)

    def hasItem(self, item):
        return self.items[item]

    def getItem(self, item):
        self.items[item] = True

We now have most of the basics done. Let’s start putting the pieces together by creating the actual class that renders these. Our main class (which we will call Field) will need to create the grid and populate it. We’ll pass it a matrix of blocks to create the field, and an array of monsters.

class Field:
    def __init__(self, grid, numMonsters):
        self.grid = grid
        self.monsters = []
        for i in range(numMonsters):
            monster = Monster(
                Math.round(Math.random()*NUM_X_BLOCKS),
                Math.round(Math.random()*NUM_Y_BLOCKS),
                self.redraw
            )
            self.monsters.append(monster)
        self.chip = Chip()
        main = self
        moveChip = def(event):
            main.chip.move(event.keyCode - 37)
            while main.grid[main.chip.x][main.chip.y] != NORMAL_BLOCK:
                main.grid[main.chip.x][main.chip.y].effect(main.chip)

            for monster in main.monsters:
                if monster.x == main.chip.x and monster.y == main.chip.y:
                    chip.die()
            main.redraw()
        window.addEventListener("keydown", moveChip)

    def redraw(self):
        for x_array in self.grid:
            for block in x_array:
                block.redraw()
        for monster in self.monsters:
            monster.redraw()
        self.chip.redraw()

We now have an almost complete game (although inefficient due to redrawing the entire field every time a monster or the user generates an event). The only thing left to do is to create blocks that that provide chip with flippers and skates. I will leave that exercise to the reader. These items should disappear after picked up, I recommend using Character base class for those and making them die() after Chip runs into them. As you can see RapydScript code is easy to follow, which is the main benefit of the language.

Why GPL?

Recently I received an email suggesting that I change the license for RapydScript to something other than GPL. I understand that no matter how powerful, a language is useless if no one wants to use it. That’s why it makes sense not to keep it proprietary, it might have worked for Oracle back in 1977, but it will no longer work today. You might be able to convince corporations to use your platform, but not people working on hobby projects. Likewise, the GPL license might scare those who intend to write commercial software. This is not my intent, I do not wish to hurt either group, which is why the RapydScript libraries are already licensed under Apache license. Part of the fear of the GPL license comes from misunderstanding its terms (especially by those not used to dealing with open-source). The work created using a product under GPL license is not itself subject to GPL license. This is important, GPL is not a cancer that keeps spreading through your tools to your product. Rather, GPL is a way to protect your own work from getting stolen and repackaged as someone else’s for profit (i.e. Cedega building upon Wine). Basically, GPL is a way to protect open-source work from plagiarism.

This protection becomes even more important when the plagiarist has the ability to hurt your project in some way. He could steal your product and then damage your ability to work on your own branch of the it in an attempt to get rid of the competition. For example, imagine that a fictional ACME Corporation writes an operating system. People try it out, they like it, and soon all devices end up running it. Years go by, ACME becomes rich, and many of the devices evolve. Instead of rewriting the OS for new devices, however, ACME decides to keep patching the original OS. They also realize that they could outsource part of the development to further cut costs. Eventually ACME OS code becomes an unmaintanable mess, and the software itself becomes buggy. Then a few guys, frustrated by the bugs in ACME OS, develop an alternative operating system in the basement of their home, call it FREE OS, and release it as open-source. FREE OS immediately becomes popular. ACME, noticing a loss in its profits, decides to repackage FREE OS as a new version of their own. This in itself might hurt FREE OS by stealing its users (and removing the incentive for developers to work on it). But even if FREE OS already got enough traction such that people stick with it, ACME, having much bigger pockets, could go to hardware manufacturers and pay them to add DRM capability using piracy prevention as an excuse (ironically, piracy is exactly what ACME is doing). ACME could then add a proprietary driver to their OS that is able to play files encrypted with this DRM.

Fortunately, RapydScript is not an OS, and as a language it’s not in danger of being affected by a DRM. That doesn’t mean it can’t be ruined by a corporation, however. The compiler aims to generate fast, lightweight code, compatible with various JavaScript obfuscation mechanisms (it should in theory be compatible with advanced mode of Google Closure, although I haven’t tested this). In fact, the new version of Grafpad itself uses an obfuscator based on RapydScript code (which I haven’t released yet) in combination with UglifyJS. The only thing preventing a 3rd party from coming in and selling their own proprietary obfuscator/optimizer based on RapydScript code without contributing anything back is the GPL license of the compiler. This in itself would not actually hurt the language, but if they start breaking backwards compatibility with RapydScript, that could in fact hurt the community. Once the RapydScript community grows and the language gains some initial popularity, this will not be as much of an issue anymore (the obfuscator will be unlikely to gain traction without full support for RapydScript). At that time I won’t mind changing the license to Apache for the entire project.

I’m not strongly attached to the GPL license, and the libraries are already licensed under Apache, allowing companies to build their own private implementations on top of them. Only the compiler itself is licensed under GPL, and I want to encourage that code to stay open to benefit everyone. GPL seemed like the right license for the job. So far the arguments I hear against GPL from other developers are either due to it being used in the wrong places (the APIs/libraries) or unfounded paranoia due to misunderstanding how the license works (people assuming that work created via GPLed product is subject to GPL as well).

Eventually, I do plan to release both, RapydML and RapydScript under Apache license. If you believe I should do so now, I would love to hear your argument. As of yet, I do not see a legitimate case of how the GPL license could hurt a company deciding to use RapydScript (aside from their legal department getting paranoid about the ‘GPL’ acronym). If they wish to use RapydScript as a compiler to create proprietary work, the GPL license does not affect their own code. If they wish to reuse RapydScript libraries in their proprietary code, the Apache license of the libraries will allow them to do that. If they wish to make changes to the compiler that will only be used internally by the company, the GPL license will not affect them. If they wish to release a stand-alone tool to be used with RapydScript, RapydScript’s license does not apply. If they wish to make changes to the compiler that would affect the rest of the community, then they have to release the source code for these changes.

RapydScript in a Nutshell

RapydScript Logo My last post mentioned RapydScript, a JavaScript variant with Pythonic syntax which I’ve come up with to speed up web-development. I haven’t had much free time to post since then, but I have been updating RapydScript this whole time. The bitbucket repository for the language started drawing some followers, and I am now getting questions in my inbox regarding plausibility of developing large-scale application in RapydScript. To address these concerns, and explain how to leverage RapydScript, I decided to write this post.

Before starting your project, it’s important to understand what RapydScript is and what it is not. RapydScript does not aim to emulate Python in JavaScript like most other Python-to-JS compilers such as Pyjamas, Pyjaco, and Skulpt. RapydScript to Python is what CoffeeScript is to Ruby (actually, RapydScript and Python have more in common). The point is, it’s important to understand that when writing RapydScript, you’re still using JavaScript. The code doesn’t get abstracted into a sandbox or wrapped in special types. This has a few advantages and disadvantages.

Advantages

The main advantage of such approach is closer integration with JavaScript. Your code will load and run faster than similar code written for one of the other Python compilers (except PyvaScript, which will work just as fast). Additionally, your code doesn’t need hacks or wrappers to invoke or create native JavaScript and DOM objects, anything JavaScript has access to, RapydScript has access to. Drawing its power from JavaScript’s prototype, RapydScript’s class inheritance system is more intuitive than Python’s. RapydScript can create object literals (similar to C structs), create anonymous multi-line functions, and do some other goodies that pure Python can not. You can use $ in your variable names and use jQuery (or any other JS framework) without hacks. You can even debug your code by reading generated JavaScript that works and looks almost the same way as original code. Basically, RapydScript has all the advantages of native JavaScript. In fact, it’s very easy to debug RapydScript using Chrome’s Developer Tools or Firefox’s Firebug, other Pythonic solutions don’t have this benefit.

The other advantage is that while RapydScript does have a standard library, it does not rely on it to be usable. This means that even if I decide to abandon the project tomorrow for some reason, you will not be stuck with a half-functional solution for your front-end. Even with a limited subset of Python library implemented, RapydScript is 100% usable and capable of building large-scale web apps, because its main advantage isn’t in its standard library, but in cleaning up JavaScript syntax. In fact, you can forego RapydScript’s stdlib.js in your project altogether and replace it with underscore.js. You will also automatically benefit from new JavaScript features as they appear and can continue using the compiler regardless of how old it is.

Disadvantages

The disadvantage of RapydScript (when compared to other Python-to-JS compilers, not counting PyvaScript) is also its close integration with JavaScript. RapydScript does not aim to catch errors JavaScript ignores. If you access an out-of-bounds cell in an array, you will get an undefined variable, not an IndexError (likewise, 1/0 is infinity). You can’t use negative indexes to traverse arrays in reverse (you can use negative indexes via list.__getitem__(), however). It turns out, however, that for properly written code, these disadvantages aren’t a problem. As long as you understand that the language is not pure Python, and the coding style a bit different, you will not have problems with RapydScript.

Why is it better than JavaScript?

If RapydScript has the same advantages and disadvantages as JavaScript, the question you might then have is “Why use it instead of JavaScript?”. The answer is “Because it brings many of the same benefits Python programmers already enjoy into JavaScript”. These benefits include all of the following and more:

  • Classes and Inheritance
  • Python Standard Library (only a portion of it is included so far, but more will be with time)
  • Clean, easy-to-read, Pythonic syntax
  • Better Variable Scoping (and Variable Shadowing)
  • Implicit Tuple Packing/Unpacking
  • Better Optional Argument Implementation
  • List comprehensions
  • Ability to import multiple modules into a single chunk of code (allowing easier code reuse)

If you spend a few minutes coding in RapydScript, or even checking out its examples, you will notice that once the ugly parts are removed from JavaScript, it’s actually a very beautiful language (allowing for code that is even cleaner than its Python-equivalent). And that’s exactly what RapydScript does, it cleans up the language so you can enjoy its true potential.

So who should use RapydScript?

The audience that will enjoy it the most is probably Python developers who want to write JavaScript code. There is a reason many Python companies choose to do their JavaScript development in CoffeeScript instead of using one of the existing Python-to-JavaScript compilers. The reason is performance, debugging ability, and integration with other JavaScript. RapydScript shares all of the advantages of CoffeeScript without introducing messy and confusing syntax.

RapydScript in Commercial Projects

I have no problem with people profiting from the code they write in RapydScript. Both, your code and the JavaScript you generate is yours to do whatever you want with. RapydScript itself is licensed under GPL, but all of its current libraries are covered by Apache license. This is to allow you to import those libraries into your project without being forced to open-source your entire front-end. I ask that other developers submitting new libraries for RapydScript also use a permissive license, but that is up to the developer.