RapydScript II

Building New Compiler A while back I mentioned that I wanted to rewrite RapydScript to rely on an internal AST (Abstract Syntax Tree) structure rather than PyMeta, which would allow it several advantages, including code transformations prior to output, more consistent parsing, and better error detection/handling.

I have looked at several tools for helping me accomplish this. One of the most promising seemed to be CoffeeScript, that is until I actually started converting its source code into RapydScript (via an automated AST-to-RapydScript generating script I put together). Having discovered that the output is very ugly and unpythonic, I gave up on the idea. After some more searching, however, I found the right tool for the job, ironically it was UglifyJS, a tool designed to make your JavaScript less readable.

UglifyJS code-base is very clean, and although it uses its own format that doesn’t directly translate into Python, the logic is well-structured and it’s easy to see where the code could be replaced by Python/RapydScript constructs. The code does a good job being explicit rather than relying on various JavaScript hacks, like CoffeeScript does.

Using UglifyJS as a base, I rewrote the compiler to convert RapydScript into native JavaScript. You can get it here: https://github.com/atsepkov/RapydScript. There are a couple minor features from the original compiler RapydScript II doesn’t support yet, but it’s already better than the original in several ways:

  • It compiles code much faster (on the order of milliseconds instead of seconds)
  • It handles leading whitespace much better now (same way as native Python would), so it will not complain if comments or secondary lines mix spaces and tabs
  • It will properly localize outer-scope variables when using module wrapper
  • It’s a single-pass compiler that uses consistent logic to identify tokens, whereas the original RapydScript would sometimes try to perform the same logic in Python stage that it would later do using PyMeta, resulting in potential parsing inconsistencies in certain special cases
  • It supports some notations that the original does not (such as the code shown later in this post)
  • It handles if val in array the same way Python would, rather than JavaScript, checking for existence of the value rather than the index
  • It’s more resistant to bad code, reporting errors that are more relevant and identifying the exact line/column number that caused the error
  • It’s more resistant to valid code that doesn’t follow typical syntax conventions, parsing it correctly rather than erroring out
  • The class-parsing logic has been improved to allow better durability and some requirements of the original compiler no longer apply (the __init__ function no longer needs to be first in the class body, for example, classes can now be nested and even declared inside functions)
  • It is written in JavaScript, which means it can eventually be ported to native RapydScript, it also means it can be modified to run in a browser, and that we can actually test function rather than structure in our unit and integration tests
  • It’s AST-aware, so it doesn’t need to generate excessive parentheses in the output for safety like the original compiler
  • Since the compiler is AST-based, it doesn’t need to generate output at the time of parsing, which makes the compiler more flexible and allows various code transformations and analysis to take place before the output is produced. This also makes it easier to make changes to the way output is generated in the future, should we need to do so.

The compiler already supports most of the features of the original, the only features that aren’t yet supported are:

  • (UPDATE: these now work) Chained Comparisons (1 < x < 5)
  • (UPDATE: these now work) List Comprehensions
  • (UPDATE: these now work) Inline Functions

Everything else should work at least as well as the original, and in some cases much better. For example, you can now do the following:

(def(a):
    return a
)(1)

Here is a list of other features that the original compiler does not support:

  • Logic such as 5 in [5,6,7] now correctly returns True
  • stdlib is no longer required for using for loops, len() or print(), more functions will be moved out of stdlib with time
  • Negative array indices now compile to arr[arr.length-1] instead of using slices, this allows assignment to them and not just referencing them (you still can’t use variables as negative indices)
  • Simple minifier is now built into the compiler (it will remove whitespace and unnecessary punctuation)
  • Pythonic imports are now supported, use them via --namespace-imports flag (experimental, some code will probably break, star-imports are not supported yet, you can set variables in global scope but not modify them)

I can’t yet guarantee that your code will work correctly with RapydScript II, but all RapydScript II test cases (which are more rigorous than original RapydScript tests, in that they also test that the code performs correct logic after compilation) as well as the examples from original (with slight modifications to remove the unsupported features) seem to work. And as a present for those afraid of GPL, I’m releasing the entire compiler for RapydScript II under Apache 2.0 license.

RapydScript by Example

With RapydScript gaining popularity, I figured it’s about time for a blog post talking about an example app. After all, I learn fastest by observing examples, many others probably do as well.

Let’s imagine we wanted to write a game using RapydScript. Since I’m not feeling particularly creative today, let’s concentrate on an existing game, rather than coming up with an idea from scratch. For this example, I’ll use a game called Chip’s Challenge.

Chip's Challenge

Looking at the original game, we quickly notice that the environment is a grid of blocks. Each block type has a corresponding image, and some have an effect on the character stepping on them. If we assume that we’ll use a canvas element for drawing the grid, then each block will need to track its coordinates in the grid and the url of the image corresponding to this block. Let’s assume we created a canvas with id #canvas in our document using html. We now simply need to create a reference to it in RapydScript:

CANVAS = document.getElementById('canvas').getContext('2d')

Since canvas is not only an object, but also a module containing methods for creating canvas-compatible objects (like images and patterns), we will use a global to reference it rather than properly abstracting it. Next, let’s create a simple block:

BLOCK_SIZE = 32 #pixels
NUM_X_BLOCKS = 21 # horizontal blocks on a field
NUM_Y_BLOCKS = 21 # vertical blocks
NORMAL_BLOCK = 0 # enum identifying block type

class Block:
    def __init__(self, x, y, image_url):
        self.x = x
        self.y = y
        img = new Image()
        img.src = image_url
        self.type = NORMAL_BLOCK
        self.blockPattern = CANVAS.createPattern(img)

    def redraw(self):
        CANVAS.save()
        CANVAS.fillColor = self.blockPattern
        CANVAS.fillRect(self.x*BLOCK_SIZE, self.y*BLOCK_SIZE, BLOCK_SIZE, BLOCK_SIZE)
        CANVAS.restore()

Now that we have a basic block, let’s create advanced blocks that can affect chip in some way. For example, Chip’s Challenge has a water block that Chip can drown in unless he’s wearing flippers. There is also an ice block, that makes chip slide to the next one unless he’s wearing ice skates. Let’s create these two blocks:

WATER_BLOCK = 1
ICE_BLOCK = 2

class WaterBlock(Block):
    def __init__(self, x, y):
        Block.__init__(self, x, y, 'images/water.jpg')
        self.type = WATER_BLOCK

    def effect(self, unit):
        if not unit.hasItem('flippers'):
            unit.die() # drown

class IceBlock(Block):
    def __init__(self, x, y):
        Block.__init__(self, x, y, 'images/ice.jpg')
        self.type = ICE_BLOCK

    def effect(self, unit):
        if not unit.hasItem('skates'):
            unit.move(unit.direction) #slip

Let’s add a character that can move around. We will use Block as a base class since a character is really just another block drawn on top of the existing block. Unlike a regular block, the picture could change depending on the direction the character is facing:

LEFT = 0
UP = 1
RIGHT = 2
DOWN = 40

class Character(Block):
    def __init__(self, x, y, images):
        self.patterns = [CANVAS.createPattern(img) for img in images]
        self.direction = DOWN # default facing direction
        self.blockPattern = self._patterns[self.direction]

    def move(self, direction):
        self.direction = direction
        self.blockPattern = self.patterns[direction]
        if direction == DOWN:
            self.y += 1
        elif direction == UP:
            self.y -= 1
        elif direction == LEFT:
            self.x -= 1
        else:
            self.x += 1
        self.redraw()

This character can now serve as the base class both for Chip himself and the monsters that roam around. If we wanted to create a typical monster, for example, we would write:

MONSTER = 10

class Monster(Character):
    def __init__(self, x, y, redrawCallback):
        images = [Image() for i in range(4)]
        images[DOWN].src = 'image/monsterDown.jpg'
        images[UP].src = 'image/monsterUp.jpg'
        images[LEFT].src = 'image/monsterLeft.jpg'
        images[RIGHT].src = 'image/monsterRight.jpg'
        self.type = MONSTER
        Character.__init__(self, x, y, images)

        # have the monster move randomly every 0.5 seconds
        main = self
        window.setInterval(def(): main.move(Math.round(Math.random()*4)); redrawCallback();, 500)

    def die(self):
        pass    # monsters don't die

Similarly, let’s create chip:

CHIP = 11

class Chip(Character):
    def __init__(self, die=False):
        # center Chip
        self.x = int(NUM_X_BLOCKS/2)+1
        self.y = int(NUM_Y_BLOCKS/2)+1
        self.items = {}

        if not die:
            self.deaths = 0
            images = [Image() for i in range(4)]
            images[DOWN].src = 'image/chipDown.jpg'
            images[UP].src = 'image/chipUp.jpg'
            images[LEFT].src = 'image/chipLeft.jpg'
            images[RIGHT].src = 'image/chipRight.jpg'
            self.type = CHIP
        Character.__init__(self, x, y, images)

    def die(self):
        self.deaths += 1
        self.__init__(True)

    def hasItem(self, item):
        return self.items[item]

    def getItem(self, item):
        self.items[item] = True

We now have most of the basics done. Let’s start putting the pieces together by creating the actual class that renders these. Our main class (which we will call Field) will need to create the grid and populate it. We’ll pass it a matrix of blocks to create the field, and an array of monsters.

class Field:
    def __init__(self, grid, numMonsters):
        self.grid = grid
        self.monsters = []
        for i in range(numMonsters):
            monster = Monster(
                Math.round(Math.random()*NUM_X_BLOCKS),
                Math.round(Math.random()*NUM_Y_BLOCKS),
                self.redraw
            )
            self.monsters.append(monster)
        self.chip = Chip()
        main = self
        moveChip = def(event):
            main.chip.move(event.keyCode - 37)
            while main.grid[main.chip.x][main.chip.y] != NORMAL_BLOCK:
                main.grid[main.chip.x][main.chip.y].effect(main.chip)

            for monster in main.monsters:
                if monster.x == main.chip.x and monster.y == main.chip.y:
                    chip.die()
            main.redraw()
        window.addEventListener("keydown", moveChip)

    def redraw(self):
        for x_array in self.grid:
            for block in x_array:
                block.redraw()
        for monster in self.monsters:
            monster.redraw()
        self.chip.redraw()

We now have an almost complete game (although inefficient due to redrawing the entire field every time a monster or the user generates an event). The only thing left to do is to create blocks that that provide chip with flippers and skates. I will leave that exercise to the reader. These items should disappear after picked up, I recommend using Character base class for those and making them die() after Chip runs into them. As you can see RapydScript code is easy to follow, which is the main benefit of the language.