Using Angular Rapydly

RapydScript with AngularJS I’ve been asked a few times if RapydScript works with Angular. The answer is the same as with any other JavaScript framework/library/etc. – of course! RapydScript takes the same approach to compilation/abstraction as CoffeeScript, so there really isn’t anything JavaScript can do that RapydScript can’t. Salvatore has already proven this with multiple demos using frameworks I haven’t even thought of. I urge any developer with similar questions to just try it out, it’s easier (and more fun) to spend an hour building an app than hesitating about it for weeks.

With that said, I decided to put together this tutorial by cloning the 5 examples shown here via RapydScript. I suggest you read the original article before this post. I should also mention that I myself have not used AngularJS before, and decided to learn it through these examples as well. I was a bit disappointed to see that a big chunk of AngularJS is handled magically from their html templates rather than JavaScript itself. This means that there is very little actual JavaScript (and hence RapydScript) used. As far as html itself, you could either write it directly, or use RapydML or Jade (both will have similar syntax). For the purposes of this post, I will leave css code alone, although you can easily convert it to sass as well. Let’s get started.

Setup (Only Needed for RapydML)

Looking at the first example, we immediately notice Django-like variables in the html code. That’s a sign that we may want to use template engines in RapydML. So let’s slap one together. Glancing through the examples we see that most AngularJS variables follow this format:

{{var}}

Doesn’t look like they’re doing anything fancy, so let’s create a very simple template engine in a new pyml file:

angular = TemplateEngine('{%s}')
angular.js = create('{%s}')

I should also mention that you can use AngularJS from RapydML directly, without creating a template engine, I chose to do so for clarity. But nothing prevents you from writing this:

div:
    'This is an angular var: {{myvar}}'

Example 1 (Navigation Menu)

This one doesn’t require any JavaScript. We’ll just need to slap together a RapydML template, leveraging the 2-line template engine we just wrote:

import angular

div(id="main", ng-app):
    nav(class="angular.js(active)", ng-click="\$event.preventDefault()"):
        for $name in [home, projects, services, contact]:
            a(href='#', class="$name", ng-click="active='$name'"):
                python.str.title('$name')

    p(ng-hide='active'):
        'Please click a menu item'
    p(ng-show='active'):
        'You choose'
        b:
            "angular.js(active)"

That’s it, compile it and plug into AngularJS. You will need the most recent version of RapydML, older versions did not handle template engine calls within strings, the new one handles them in double-quoted strings and ignores them in single-quoted strings.

Example 2 (Inline Editor)

As before, we start with the RapydML code:

import angular

div(id='main', ng-app, ng-controller='InlineEditorController', ng-click='hideTooltip()'):
    div(class='tooltip', ng-click='\$event.stopPropagation()', ng-show='showtooltip'):
        input(type='text', ng-model='value')
    p(ng-click='toggleTooltip(\$event)'):
        "angular.js(value)"

Next, let’s add some RapydScript:

def InlineEditorController($scope):
    $scope.showtooltip = False
    $scope.value = 'Edit me.'

    $scope.hideTooltip = def():
        $scope.showtooltip = False

    $scope.toggleTooltip = def(e):
        e.stopPropagation()
        $scope.showtooltip = not $scope.showtooltip

Once again, after compiling the first block of code via RapydML and second via RapydScript, we’ll have a fully-working example.

Example 3 (Order Form)

In this example, we see a new way of hooking into AngularJS. Note the {active: service.active} class tag. We could either reference is in the code verbatim (since it is quoted), or add a new method to the angular template engine for consistency:

angular.active = create('active: %s')

We can now continue with the 3rd example, first the RapydML code:

import angular

form(ng-app, ng-controller='OrderFormController'):
    h1:
        'Services'
    ul:
        li(ng-repeat='service in services', ng-click='toggleActive(service)', ng-class="angular.active(service.active)")
        "angular.js(service.name)"
        span:
            "angular.js(service.price | currency)"

    div(class='total'):
        'Total: '
        span:
            "angular.js(total() | currency)"

Finally, let’s complement it with RapydScript:

def OrderFormController($scope):
    $scope.services = [
        {
            name: 'Web Development',
            price: 300,
            active: True
        }, {
            name: 'Design',
            price: 400,
            active: False
        }, {
            name: 'Integration',
            price: 250,
            active: False
        }, {
            name: 'Training',
            price: 220,
            active: False
        }
    ]

    $scope.toggleActive = def(s):
        s.active = not s.active

    $scope.total = def():
        total = 0
        angular.forEach($scope,services, def(s):
            nonlocal total
            if s.active:
                total += s.price
        )
        return total

As usual, looks similar to the JavaScript version. RapydScript did prevent us from shooting ourselves in the foot by accidentally declaring total as global, something you’d need to be careful about in native JavaScript version.

Example 4 (Instant Search)

For this example, I’ll actually show alternative RapydML code, that doesn’t use our template engine:

div(ng-app='instantSearch', ng-controller='InstantSearchController'):
    div(class='bar'):
        input(type='text', ng-model='searchString', placeholder='Enter your search terms')

    ul:
        li(ng-repeat='i in items | searchFor:searchString')
            a(href='{{i.url}}'):
                img(ng-src='{{i.image}}')
            p:
                '{{i.title}}'

That was even easier than the template engine version, I think I’ll stick to this for my next example as well. Now the RapydScript portion:

app = angular.module('instantSearch', [])
app.filter('searchFor', def():
    return def(arr, searchString):
        if not searchString:
            return arr

        result = []
        searchString = searchString.toLowerCase()
        angular.forEach(arr, def(item):
            if item.title.toLowerCase().indexOf(searchString) != -1:
                result.push(item)
        )
        return result
)

def InstantSearchController($scope):
    $scope.items = [
        {
            url: 'http://tutorialzine.com/2013/07/50-must-have-plugins-for-extending-twitter-bootstrap/',
            title: '50 Must-have plugins for extending Twitter Bootstrap',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/07/featured_4-100x100.jpg'
        }, {
            url: 'http://tutorialzine.com/2013/08/simple-registration-system-php-mysql/',
            title: 'Making a Super Simple Registration System With PHP and MySQL',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/08/simple_registration_system-100x100.jpg'
        }, {
            url: 'http://tutorialzine.com/2013/08/slideout-footer-css/',
            title: 'Create a slide-out footer with this neat z-index trick',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/08/slide-out-footer-100x100.jpg'
        }, {
            url: 'http://tutorialzine.com/2013/06/digital-clock/',
            title: 'How to Make a Digital Clock with jQuery and CSS3',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/06/digital_clock-100x100.jpg'
        }, {
            url: 'http://tutorialzine.com/2013/05/diagonal-fade-gallery/',
            title: 'Smooth Diagonal Fade Gallery with CSS3 Transitions',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/05/featured-100x100.jpg'
        }, {
            url: 'http://tutorialzine.com/2013/05/mini-ajax-file-upload-form/',
            title: 'Mini AJAX File Upload Form',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/05/ajax-file-upload-form-100x100.jpg'
        }, {
            url: 'http://tutorialzine.com/2013/04/services-chooser-backbone-js/',
            title: 'Your First Backbone.js App – Service Chooser',
            image: 'http://cdn.tutorialzine.com/wp-content/uploads/2013/04/service_chooser_form-100x100.jpg'
        }
    ]

On to the last example.

Example 5 (Switchable Grid)

First the RapydML code, as usual:

def layout($type, $imgtype):
    ul(ng-show="layout == '$type'", class="$type"):
        li(ng-repeat='p in pics'):
            a(href='{{p.link}}', target='_blank'):
                img(ng-src="{{p.images.$imgtype.url}}")

div(ng-app='switchableGrid', ng-controller='SwitchableGridController'):
    div(class='bar'):
        for $icon in [list, grid]:
            a(class='$icon-icon', ng-class="{active: layout == '$icon'}", ng-click="layout = '$icon'")
    layout(list, low_resolution)
    layout(grid, thumbnail)
            p:
                '{{p.caption.text}}'

Notice that I chose to be a little more DRY and create an extra function, you don’t have to do that if you believe it hurts readability. Next, the RapydScript code:

app = angular.module('switchableGrid', ['ngResource'])
app.factory('instagram', def($resource):
    return {
        fetchPopular: def(callback):
            api = $resource('https://api.instagram.com/v1/media/popular?client_id=:client_id&callback=JSON_CALLBACK', {
                client_id: '642176ece1e7445e99244cec26f4de1f'
            }, {
                fetch: { method: 'JSONP' }
            })

            api.fetch(def(response):
                callback(response.data)
            )
    }
)

def SwitchableGridController($scope, instagram):
    $scope.layout = 'grid'
    $scope.pics = []
    instagram.fetchPopular(def(data):
        $scope.pics = data
    )

Conclusion

While the RapydScript code above might be a bit cleaner than the original JavaScript code, it’s hard to argue that significant value was gained from porting it. The main benefit of RapydScript comes into play when building larger-scale applications that leverage classes and use numerous variables. Such code increases the chances of shooting yourself in the foot or getting confused when dealing with native JavaScript. Indeed, if you look at some of the jQuery plugins, you’ll see that the code itself seems unmaintainable. After a few layers of anonymous functions it’s hard to figure out what’s being passed around where. AngularJS keeps the code short by handling a lot of the logic for you on the background, RapydScript allows you to keep your own logic readable as it grows. No matter how awesome a framework is, eventually you’ll be writing a lot of your own logic as well.

Pyjamas and Web2py

UPDATE: Pyjamas has since been renamed Pyjs and is under new leadership. Everything is still backwards compatible.

At this point, if you’ve been following along the posts, you should know how to create a simple web2py application. In this post I’m going to describe how to write a page that can connect to a backend written in web2py. This is another step on the path of having an app on GAE, in fact, the code we write here will get deployed on GAE. This code will also run on any system, including your own computer, and avoids the lock-in some people experience when developing for GAE.

There are 2 ways that I use for communicating with a web server are RESTful JSON calls, and JSON-RPC calls. Instead of covering both I plan on just showing how to use JSON-RPC. I suggest when you design your app first search online for comparisons between REST & JSON-RPC to see the trade-offs.

Before we get into the code, I also have a slight curve-ball. I originally wrote this code using Pyjs, but I’ve switched to using Rapydscript for all my development. In a future post I’ll show how to connect Rapydscript to GAE, which I plan to link here. I suggest reading Alex’s earlier post https://blogs-pyjeon.rhcloud.com/?p=301 for a good summary of various Python to JS compilers and their trade-offs.

web2py Services

Time to code!

First you need to setup your web2py app to use web2py’s services module. This lets your application work as a web service so it can respond to calls from clients, including support for JSON-RPC calls. Inside of models/db.py add the following lines:

from gluon.tools import Service
service = Service()

In the controller, add a call function that returns the service. Then all you have to do is add a decorator to each functions you want to act as services receiving jsonrpc calls and web2py handles the rest. In controllers/default.py I have added the following:

@service.jsonrpc
def myfun(data_from_JSON):
    return data_from_JSON.upper()

def call():
    return service()

The example function here, myfun, will make everything uppercase. Also worth nothing is data_from_JSON is already decoded data from the JSON request.

To access this service use the URI ‘/cyborg/default/call/jsonrpc’. For more information on services check out http://web2py.com/books/default/chapter/29/10#Remote-procedure-calls.

Pyjs Clients

I’ve had this code floating around my computer for a couple years now. I’ve made several minor changes, some just because I wanted slightly lighter code, and some because of changes in Pyjs, but it was originally based on code by Amund Tveit on his blog (http://amundblog.blogspot.com/2008/12/ajax-with-python-combining-pyjs-and.html). This is a simple page with a text area for sending text to a JSON-RPC service.

from pyjamas.ui.RootPanel import RootPanel
from pyjamas.ui.TextArea import TextArea
from pyjamas.ui.Label import Label
from pyjamas.ui.Button import Button
from pyjamas.ui.VerticalPanel import VerticalPanel
from pyjamas.JSONService import JSONProxy


class JSONExample:
    def onModuleLoad(self):
        self.rpc_service = JSONProxy('/cyborg/default/call/jsonrpc', ['myfun'])

        self.text_area = TextArea()
        self.text_area.setText(r"Hello World")
        self.text_area.setCharacterWidth(80)
        self.text_area.setVisibleLines(4)

        button = Button('Send to Server', self)

        self.status = Label()

        panel = VerticalPanel()
        panel.add(self.text_area)
        panel.add(button)
        panel.add(self.status)

        RootPanel().add(panel)


    def onClick(self, sender):
        print('sending to server')
        self.status.setText('Waiting for response...')

        textarea_txt = self.text_area.getText()
        if self.rpc_service.myfun(textarea_txt, self) < 0:
            self.status.setText('Server Error')


    def onRemoteResponse(self, response, request_info):
        print('good response')
        self.status.setText(response)


    def onRemoteError(self, code, message, request_info):
        print('error')
        print(code)
        print(message)
        self.status.setText("Server Error or Invalid Response: ERROR  - " + str(message))


if __name__ == '__main__':
    app = JSONExample()
    app.onModuleLoad()

To make the call, you need an instance of the JSONProxy class, so I have the line

self.rpc_service = JSONProxy('/cyborg/default/call/jsonrpc', ['myfun'])

Then call your function using that instance, along with passing in an object/variable (which Pyjs encodes) to send to the service, and a reference to an instance of a class with onRemoteResponse and onRemoteError methods.

self.rpc_service.myfun(data, response_class)

In the example code above, response_class is self, so the JSON response comes through onRemoteResponse.

Putting it Together

Believe it or not, this was the trickiest part for me when I first got this working. Figuring out the correct URIs to use, and having code listening for the call was tough to debug. Luckily for anyone following along, the example code here already has everything setup correctly :).

We can start out just by making sure everything ties together. Go to your web2py folder then applications\cyborg\static. This is the static directory for the cyborg app. Now take all the output from Pyjs and put it there. I called my Pyjs file JSONExample.py, which generated JSONExample.html, so I access this using http://127.0.0.1:8000/cyborg/static/JSONExample.html. The ‘Send to Server’ button on the page should work. If it doesn’t, make sure you followed every step exactly. There might have also been changes/bugs in Pyjs or web2py – it’s not likely, but it has happened.

I personally wanted my main app to call the JSON-RPC service, not a file in the static director. I am going to have the index load the JSON page. First thing to do is cleanup the index function in default.py. I just want index to return the view, so index now returns an empty dictionary:

def index():
    return {}

I then replace the contents of applications\cyborg\views\default\index (the view for index() in the default controller) with the contents of the main Pyjs output file, JSONExample.html in my case.

The compiled html/js files are still in the static dir though. Remember, users are accessing the files in the views directory by accessing controllers. So if the other Pyjs files were in the views, you would still need to have a controller function to access them. There is no clean, simple way to put all your Pyjs cache files in the views directory. Instead we need to modify the new index.html to point to the files in the static dir. So the index.html code gets modified once for the module:

<meta name="pygwt:module" content="/cyborg/static/JSONExample">

and, in several places, for bootstrap.js:

<script language="javascript" src="/cyborg/static/bootstrap.js"></script>

Now you can visit http://127.0.0.1:8000/cyborg/ and send JSON-RPC calls!

One thing to note is that with the default routes calling /cyborg/, /cyborg/default, and /cyborg/default/index all load the same view/controller. If your JSONProxy class uses a relative link it might only work when visiting 1 of these URIs. That is why, in the Pyjs code, I refer to URI starting from /. When I was learning how to do this, I set everything using relative URI’s, like ../static/JSONExample, and ../default/call/jsonrpc, and that made everything difficult, so I stay away from that.

A Simple web2py App

I know this is waaaaay overdue, but better late than never. Over the last year and a half I’ve switch over to using Linux and away from Pyjamas (I’m still using python though). I’ll have more info in future posts over the next couple weeks. Don’t worry if you’re using Windows though, the steps here work in both Linux and Windows (thank you Python!).

Installing and Running web2py

Start by downloading web2py (source – version 2.2.1 for me). I am doing this in XP, so I extracted the code to C:\Projects\web2py. Open a command window, navigate to your web2py dir and start it up with the command

> python web2py.py -a tmp_pass

This starts up web2py on 127.0.0.1:8000 with the admin password set to tmp_pass. You can use the -h option to see how to set web2py up in other ways. One thing to note is if the server is running on 127.0.0.1 you won’t be able to access it using your real IP address. If you want to test your server using external computer have web2py use the IP 0.0.0.0.

With web2py running, I could then visit http://127.0.0.1:8000 where an sort of hidden admin interface button lets me login using my admin password, tmp_pass.

Creating an App

There is a panel on the right, with a section called “New simple application” which you can use to create an app. This sets up all the template files for you. In my case I created a program called cyborg.

The server shows a list of files which I could edit. It’s a lot of code, definately more than I wanted for my app, but that should be easy to cleanup later on. With the web2py server up, I navigated to http://127.0.0.1:8000/cyborg/. which showed a page with some interesting bullets, including:

  • You visited the url /cyborg/
  • Which called the function index() located in the file web2py/applications/cyborg/controllers/default.py
  • The output of the file is a dictionary that was rendered by the view web2py/applications/cyborg/views/default/index.html

These are the MVC files discussed in my previous post.

Simplifying the App

I decided to investigate each of the steps taken to run the code and try to trim unneeded code: 1) Well, this one is obvious

2) The default code has calls to use databases, and uploading files, etc. Actions I don’t plan on supporting, at least just yet. First, I want to get comfortable with everyting. I changed default.py to be the simplest function possible:

# -*- coding: utf-8 -*-

def index():
    return dict(message="Hello World", message2="How's it going?")

3) The view should get the dictionary from index() and render it. Really whats happening is the view is loaded, and any python code in the index uses the dictionary from index(). Python code is embedded between {{}}

{{if 'message' in globals():}}
<h3>{{=message}}</h3>
{{pass}}
<br>
{{if 'message' in globals():}}
<h3>{{=message2}}</h3>
{{pass}}

The only kind of gotcha I found was that you need to have a pass aligned with every if. I think this is because there isn’t a way to unindent in the html files.

Cleanup

Since I only plan on using a couple function to start, I want to remove all the files that seem unnecessary. I started by removing and checking the page still worked: I removed these folders completely (I think most get recreated by web2py when the app runs, but with just dummy files):

  • databases (this folder has the database files)
  • errors (this is a logs folder)
  • languages (translation files)
  • models (usually where you put information about your database)
  • private
  • static
  • uploads

And cleaned the folders:

controllers: Only kept default.py views: only kept views\default\index.html

I realoaded http://127.0.0.1:8000/cyborg/, and I was happy to see the messages I passed in through the dict.

Pyjamas Alternatives for Web Development

In my last post I mentioned that I’m switching away from Pyjamas. In this one, I will explain the alternatives I looked into, as well as cons and pros of each one. While doing my research (which started almost half a year ago), I’ve stumbled upon multiple Python-inspired alternatives to JavaScript, many of which have been abandoned, and only few of which have reached the stage where they’re suitable for large projects. Out of those I have settled on a solution that works for me, but there is no perfect framework/language, your choice could be different than mine. Let’s look at a few frameworks and decide how usable each one is for writing a full application.

Pyjamas

While I might have issues with how Pyjamas approaches web development, I have to admit that it has gotten quite far, and it’s quite possible to write a usable application with it.

Verdict: usable – if overhead is not a problem

Skulpt

Skulpt does not process your Python code at all until execution time, this is a unique approach that I haven’t seen any other framework take. It’s essentially like a browser-based eval() for your Python. This is a neat concept, since it can be used to extend another framework to support Python-like eval() (something that even Pyjamas doesn’t do), allowing your script to write and execute code on the fly. While I’m not a big fan of eval() function, this would definitely be neat for a framework that aims to achieve complete Python compatibility. Similarly, one of Skulpt’s major disadvantages is having to include the entire “compiler” as a script in your page, if this project ever grows to have the same functionality as Pyjamas, the overhead of loading the extra logic could be enormous. The second disadvantage is speed, Python interpreted and executed in the browser on the fly will never be as fast as even boilerplate-based Pyjamas, let alone pure JavaScript. The final problem is that you can’t obfuscate/compress your code at all, since its interpreter needs to make sense of your Python code (including the whitespace), this might be a non-issue to open-source projects.

Verdict: not usable

pyxc-js

Like Skulpt, this compiler is unsuitable for any large-scale web development, but it has an elegant import mechanism, which is something many other frameworks/compilers lack. Just like Skulpt can be used to implement eval(), this compiler can be used to implement proper import in your compiler (I know Pyjamas has proper import, but it’s heavily integrated into the rest of the framework). This project has been abandoned and the author does not respond to emails. Aside from its import mechanism (which is based on a directed graph the compiler builds at compile time, eliminating any unused and repeated imports), this project doesn’t have much to offer. The documentation is lacking, it took me a while (and some tweaks) to be able to compile anything, the included test cases don’t work, and the generated code is rather messy (it often spits out multiple semi-colons at the end of the line and blank lines get semicolons too, and if I remember correctly there are occasional bugs in how your code gets compiled). On the plus side, it offers an internal compression mechanism that can shorten your code further (although I’m not sure how much I trust it, given the quality of the non-compressed code it generates).

Verdict: not usable

PyCow

This was the first framework aside from Pyjamas that I felt could support large projects. PyCow compiles Python code into MooTools-based JavaScript. More importantly, its source code is cleanly laid out and the original developer is very responsive and helpful (despite having abandoned the project). It relies on templates for many of its conversions rather than hard-coding them in the source or implementing alternative names for JavaScript functions. For example, list.append() gets converted to list.push() at compile time using a template specified in a hash table. This is a great idea, since it makes the compiler code cleaner and the final output stays closer to real JavaScript, introducing less overhead (and less need for large stdlib). The disadvantage of this approach is that we have no good way of knowing whether a certain method is part of our language and needs to be replaced by JavaScript equivalent, or if it belonds to another API and should be left alone. Here is an example where replacing the append() would break our code:

container = $('#element'); // jQuery used to create a container of elements
...
container.append(item);    // replace this append() and jQuery will complain

PyCow is also MooTools’-based, and I’d prefer a language that doesn’t tie me to a certain framework. I investigated rewriting it such that it generates classes using pure JavaScript, but later realized it would not be worth the effort. One other minor pet-peeve is that PyCow is written for Python 2.5, which happened to be around the same time as Python developers were rewriting their AST parser. As a result, PyCow uses a transitional AST package and needs to be rewritten to support the later one that came after 2.6 (2.5 is still closer to the current AST, however, than the AST parser Pyjamas uses form compiler module, which has been deprecated as of Python 2.6). This framework also doesn’t support any importing of modules, but I managed to fix that in my local version by ripping out the import mechanism from pyxc-js. Unfortunately, I’ve moved on since then, abandoning my PyCow enhancements (let me know if you’re interested in continuing from where I left off, however, I can send you the source code).

Verdict: usable

py2js (Variation 1)

There are at least 3 different frameworks by this name, and they’re only vaguely related to each-other. The first one has been abandoned since 2009, but it impressed me for several reasons. First of all, it takes PyCow’s templating one step further, allowing you to map virtually any function/method using more complex filtering. For instance, %0 represents the class that the method is being called on, %1 reprensents the first argument, %* represents all of them. You could use this to add very powerful templating. For example, if you want to output a list of elements as a comma-delimited string in Python, you could use join() as follows: ', '.join(list). In JavaScript, however, join() is a method of array/list (which actually makes more sense to me) rather than a string, requiring you to do list.join(', ') instead. But say you wanted to auto-convert your Python joins to JavaScript, all you would have to do is add the following rule to your template hash:

'%0.join(%1)' : '%1.join(%0)'

I was able to extend this variation of py2js to support multiple Python features with JavaScript alternatives this way. As mentioned earlier, however, this method is susceptible to renaming functions whose names you have no control over.

This compiler relies on templates heavily, introducing another advantage most other compilers lack. Once the code has been converted to AST, it can be manipulated almost entirely using templates (with little need to write additional code). With some tweaks, it can become a universal source-to-source compiler (well, universal is a bit strong of a word here, statically typed languages are a whole other animal). Imagine being able to convert Python to Perl, Ruby, or any other dynamically typed language just by generating a map of logic equivalents. Something like this, for example, could allow you to convert the syntax of a Python function to Perl (block would then further get interpreted by similar templates for loops, conditionals, etc.):

### Input Template
def <method>(%*):
    <block>

### Output Template
sub <method>{
    my (%*) = @_;
    <block>;
}

Admittedly, the more fancy you get with the language (i.e. using eval()), the more likely you’re to run into problems with the template breaking. This compiler was inspired by py2py, another project designed to clean up Python code. In theory, any input languages can be supported as well, as long as you write your own AST parser that converts the code to Pythonic AST tree (or a subset of). This is a fun project in itself, and I might continue it when I have more time. As a compiler, however, this project is not yet suitable for prime-time, lacking a lot of functionality, including classes.

Verdict: not usable

py2js (Variation 2)

The only similarity between this py2js and the previous is the name. They share nothing else (well… they do now, keep reading). Variation 2 just happens to be another project by the same name as variation 1. In fact, even the compile mechanism for this py2js is very different from other compilers mentioned before. The compiler works by “running” your code at compile time. You write your function/class in Python, and then add @JavaScript decorator to it if you want it to be included in the compiled output. You then “run” your program and it dumps JavaScript to STDOUT. While it’s a bit awkward to add @JavaScript decorator to every single function, this offers something most other compilers (including Pyjamas) lack. Since it actually runs your code, you get the benefit of partial “compile-time” error catching, a feature typically only seen in statically-typed languages. You can be sure, for example, that all syntax errors will be caught, as well as undeclared variables/methods. The disadvantage is that this compiler is a bit too strict, all your code has to be pure Python and all your variables (or stubs for them) have to exist at compile time. So, for example, if you want to use something from jQuery, you better have a jQuery stub written in Python (it doesn’t need to mimic jQuery functionality, however, only to have the method names defined). This project has also been abandoned several years ago in a premature stage, which is where the 3rd variation comes in.

Verdict: not usable

py2js (Variation 3) (now Pyjaco)

The 3rd variation started off as a fork of the 2nd variation, after 3 developers decided to take over the original, abandoned, project. Over time, this variation morphed into a powerful compiler, whose functionality is only outdone by Pyjamas itself. It added support for Pythonic assertions, tuples, classes, static methods, getattr()/setattr(), *args/**kwargs, better integration with JavaScript (allowing you to call JavaScript logic by wrapping it inside JS() or automatically handling the conversion at compile time if you use one of predefined globals (document, window, etc.)) as well as many other features you would expect from Python. It has its own standard library with most Python stdlib implemented.

It, however, also falls short in a few areas. Like Pyjamas, it tries to build Pythonic error handling on top of JavaScript. Admittedly, this task might be much easier here, due to partial “compile-time” error catching, which Pyjamas lacks. It also doesn’t properly translate Python’s math methods to JavaScript, a task that shouldn’t be too hard, however.

Additionally, while it’s convenient to have document and window be treated like special classes, this can add confusion when coding. In some cases your strings and numbers will get converted to JavaScript equivalents automatically, while in the others you have to do so manually (when invoking a native JavaScript method and passing it a string, for example). A py2js string is a wrapper around native JavaScript string with additional Python functionality, same applies to many other objects. This is a good thing, since it allows support for Python’s methods without having to overwrite native objects (and I believe Pyjamas takes the same approach for its primitive objects (arrays/strings/etc.). This is a minor pet-peeve, however, and would not be a problem most of the time.

Another annoyance that I already mentioned earlier, is having to explicitly define which functions you want compiled via @JavaScript tag + str() call on the function or class (but this is just a matter of personal preference).

The biggest problem with this third variation, however, is copyright issues (which the developers are trying to resolve now – they could already have been resolved). In its quest to acquire similar functionality as Pyjamas, this compiler has “borrowed” a lot of code from other projects (including the first py2js I mentioned). This py2js project itself carries MIT license, not all of the projects it borrows from, however, are compatible with that license. Additionally, some of these projects were not given proper mention in the copyright notice due to an oversight by one of the developers. As a result, the developers had a falling out, and now forked a couple alternative versions of this project, one of which is still maintained and aims to rewrite the code in question and/or get proper authorization to use it.

Additionally, if this project joins forces with the new Pyjamas, I believe both projects will benefit a lot. The code generated by py2js is significantly smaller (Pyjamas averages 1:10 ratio for non-compiled:compiled code, while py2js is closer to 1:2 – not counting stdlib) and more readable than Pyjamas, while retaining most of Pyjamas’ features. Pyjamas, on the other hand, already has its own implementation for most of py2js code that’s subject to copyright issues, which could be used to solve the biggest problem with py2js. Finally, py2js’ compile-time error catching makes it easier to build Pythonic exception handling on top of JavaScript. And to remedy the annoyance of @JavaScript decorators, the AST parser can be used to append them automatically to temporary version of the code. This can also be used as an opportunity to update Pyjamas to use the latest AST parser implementation, which it will need anyway if it ever wants to be compatible with Python 3 (which is missing the deprecated AST implementation).

Verdict: usable

UPDATE: I’ve been informed that this project now became Pyjaco, and the copyright is no longer an issue. So for those who want to stay closer to Python, this is a very solid alternative. Christian, the project leader, also informed me that not all of the details I mentioned are accurate. Also, apparently the developers seem to have misinterpreted the original post as me claiming that RapydScript (my own compiler) is the best for everything. That was not my intent, and I tried to avoid this issue by mentioning in the first paragraph of the original article that my choice is based on my own projects and the flexibility they need (mostly Grafpad), even stating that “your choice could be different than mine”. I hope they don’t hold a grudge at me because of this misunderstanding.

PyvaScript

PyvaScript is the opposite extreme of Pyjamas. It’s perhaps the closest to pure JavaScript out of all other Python-like languages. It was one of the first alternatives I looked into, and admittedly I originally decided it would be unsuitable for large projects. There are numerous disadvantages compared to other Python frameworks. Its stdlib is puny, and most of the Python methods you would want aren’t available. It not only lacks import logic, but even classes aren’t implemented. It can’t handle negative indexes for arrays. Its compiler isn’t even based on Python AST (it uses Ometa), as a result it’s often oblivious to errors other compilers would catch, and at other times it chokes on issues that other compilers have no problems with.

Its advantages, however, provide the perfect fit for the niche that Pyjamas left open. It’s the most JavaScript-compatible solution, supporting almost all JavaScript functionality out of the box. I can manipulate other JavaScript and DOM without the need of any wrappers or special syntax. Need to use jQuery? Just do it the same way you would in JavaScript: $(element).method(), the compiler won’t complain about the dollar sign. PyvaScript also supports anonymous functions, something regular Python lacks (except 1-liner lambdas), which makes it easier to write your code like JavaScript (I didn’t see the advantage to this at first, but believe me, when writing web apps, it really helps – especially when using other JavaScript logic as a tutorial).

Since PyvaScript adds no bells or whistles to your code, or requires any sort of wrappers, your code behaves a lot more like native JavaScript, allowing you to use closures more effectively and to bind functions to objects dynamically without the language complaining. At first this seemed like a bad idea, but after playing with it, I realized that with proper use of this, I can actually write cleaner code than with Python itself. Most importantly, PyvaScript is extremely light-weight, adding almost no overhead to the generated code and does not force a framework (MooTools, jQuery, etc.) on the user. I also realized, that PyvaScript’s lazy compilation has its own advantages (which I will explain later).

Verdict: usable – but doesn’t alleviate much pain from plain JS

CoffeeScript

Sharing a structure similar to that of Python, it deserves a mention as well. If you ignore its ugly syntax, and poor choice of variable scoping (preferring globals over locals), you will see that it has all the same advantages as PyvaScript, which makes it a good candidate for Pyjamas replacement as well. It has similar feel to Python (although it feels closer to Ruby), introduces list comprehensions, and even adds classes (something PyvaScript does not). It also adds namespaces, preventing variables in different modules from interfering. If I invert the scoping design, remove all the junk variables like on/off (synonyms for true/false), and modify the syntax to use Python tokens, this will be my ideal language for web development, but that’s a project for later.

Verdict: usable

And the winner is…

RapydScript, which I didn’t even mention yet. RapydScript (best described as PyvaScript++) is my wrapper around PyvaScript that provides the best of both worlds (for me at least). I’ve made a few enhancements (abusing PyvaScript’s lazy compilation to allow me to auto-generate broken Python code that becomes proper JavaScript once compiled by PyvaScript) that achive almost everything I would want, and just about all of CoffeeScript’s functionality. Some of the new RapydScript features include support for Pythonic classes including inheritance (single inheritance only, but you can bind functions from any class to fake multiple inheritance), Pythonic optional function arguments, anonymous functions now supported in dictionaries/object literals (something PyvaScript chokes on), beefed up stdlib (also optimized already implemented methods from PyvaScript’s stdlib), support for multi-line comments/doc-strings, preliminary (compile-time) checking for syntax errors and issues that PyvaScript chokes on (because of minor bugs in PyvaScript), module importing (currently everything gets dumped into a single namespace, but compiler does warn about conflicting names), checking proper use of Math functions, automatic insertion of ‘new’ keyword when creating an object (not sure why CoffeeScript doesn’t already do the same).

To me, the main advantages of RapydScript over PyvaScript are ability to break down my modules into separate files (like I would in Python), easier time to build large projects due to proper class implementation (class declaration is done the same way as in native Python), Pythonic declaration of optional arguments to a function (I’m not a big an of JavaScript’s solution for optional arguments), and support for anonymous functions as hash values (which allows me to build object literals the same way as in JavaScript). As for other projects, the main advantages of RapydScript are seamless integration with the DOM and other JavaScript libraries/modules (just treat them like regular Python objects), ability to use both Python and JavaScript best practices as well as rely on JavaScript tutorials (one of the biggest problems for projects in their alpha stage is lack of documentation, for RapydScript you really don’t need any), and lack of bloat (RapydScript gets me as close to Python as I need without forcing any boilerplate on me). As a nice bonus, I think I can add suppport for advanced compilation mode of Google’s Closure compiler with only minor tweaks.

If you want to give RapydScript a try, you can find it in the following repository: https://bitbucket.org/pyjeon/rapydscript. Keep in mind, this is still early alpha version and some of the functionality might change. Chances are, I will eventually rewrite this, using CoffeeScript as base, but the functionality should stay very similar (after all, I am rewriting Grafpad in this, and I don’t want to rewrite it a third time). In my next post, I will dive into more detail about RapydScript, for those interested in using it and needing a tutorial.

Why Pyjamas Isn’t a Good Framework for Web Apps

Earlier this week, I stated that Pyjamas no longer seems like a viable solution for Grafpad (or many other web-apps for that matter). In this post, I will explain the flaws with Pyjamas that ultimately made me decide to switch away from it. I’m aware that Pyjamas project is currently getting an overhaul, and I hope that these flaws get addressed in the upcoming Pyjamas releases. Before I go any further into bashing Pyjamas, I want to mention that I’ve been using Pyjamas for several years, writing over 20,000 lines of Python code that runs inside the browser (as well as several Pyjamas wrappers for extending its functionality). I appreciate the problem Pyjamas is trying to solve, and I definitely think it’s a useful tool. Perhaps one day Pyjamas will be good enough for the browser, unfortuantely it has a lot of issues to solve before that’s the case.

Experienced JavaScript developers might already be familiar with many of the points I will bring up. To summarize Pyjamas’ flaws in one sentence, it basically assumes that JavaScript is still a joke of a language it was several years ago and tries to apply outdated solutions that don’t scale well. Today’s JavaScript, however, can run circles around its predecessor, both in terms of performance and functionality. Many innovative design patterns have also been posted for keeping JavaScript code clean and object-oriented. In some ways, JavaScript has even surpassed Python in terms of design, which still lacks proper private variables, for example. So what are some of the big offenders in Pyjamas?

Browser Detection instead of Feature Detection

Many of you are probably familiar with Pyjamas’ compilation scheme. If not, it basically creates multiple versions of the JavaScript code, one for each major browser (IE, Firefox, Safari/Chrome, Opera) and serves the appropriate one depending on your user-agent string. A quick Google search will reveal thousands of pages explaining the problems with this technique (called browser detection), so there is really no point for me to go into much detail here. The first problem with browser detection is that we assume that the user will be using one of the browsers we’re detecting (sorry Konqueror). The second problem is that we’re assuming the user is using one of the versions of this browser that still has the same issues/functionality. I’ve already posted about the changes I had to make in Pyjamas to make it use IE9 properly, which has full canvas support, yet Pyjamas still treats it like IE6 (ironically, IE9 actually behaves more like WebKit than IE6). The third problem is that many browsers spoof the user-agent string, pretending to be a different browser (for various reasons). These browsers may support features that the spoofed browser doesn’t support and vice versa, forcing us to use an unnecessary work-around for a feature that the browser supports natively (just like IE9 being forced to use VML instead of canvas), or preventing the feature from working altogether (imagine if Chrome, with no VML support, spoofed IE6 user-agent string).

Bloat and Boilerplate Hell

If you’ve peeked at Grafpad’s JavaScript, you probably saw 80,000 lines of code in a 3.5MB file. But did you know that the pre-compiled version of Grafpad front-end is only about 8,000 lines of code? We have 10 times the needed code just to pretend like we’re still using Python. What’s worse, most of that code is only there to support obscure Python functionality most of us are never going to use in a web-app anyway. Pyjamas has become the most complete Python framework for the browser, unfortunately it has also become the most bloated one, with most other frameworks (such as py2js) only needing to generate 1.5 lines of JavaScript for each line of Python code. You can see the 80/20 principle at work here, where 20% of Python’s features account for 80% of Pyjamas’ boilerplate. In my opinion, it would make a lot more sense to only support the commonly used features of Python, allowing the user to rewrite the bits that don’t work well for JavaScript. After all, the most tedious things to port between languages are the algorithms, not the object structure.

Debugging

In theory, Pyjamas is much easier to debug than JavaScript. Unlike JavaScript, which either throws vague errors or worse yet, silently fails a block of code and continues execution like nothing happened, Pyjamas throws Pythonic exceptions, which most of the time do a very good job pinpointing the exact line that caused the problem… at least when you run your program through Pyjamas Desktop. The problem is, Pyjamas Desktop has been broken for almost 3 years now, requiring you to either use a 3-year old Linux distribution (last known version to have support for python-hulahop) or rely on WebKit or MSXML implementations, neither of which supports canvas.

Alternatively, you can debug your code directly in the browser. Pyjamas sports a good set of Python exceptions emulated in JavaScript through clever use of try/catch blocks. Unfortunately, this alternative not only lacks proper stack trace, but also the original code (and compiling your Pyjamas app with any of the debug modes doesn’t solve this, regardless of what various outdated posts on the mailing list claim). Needless to say, the errors raised by Pyjamas in the browser are not very useful. If you made an IndexError on line 50 of your code by referencing “object.array[5]“, for example, expect Pyjamas to throw some weird error (that’s right, chances are it won’t even realize it’s an IndexError – or at least won’t report it well, the except blocks seem to work correctly) on line 30,000 of your compiled JavaScript, which will reference $p['getattr'](object, 'array').__getitem__(5), among a bunch of other boilerplate which could have caused an error as a result of an earlier error in your code or a Pyjamas bug. Even when debugging using Pyjamas Desktop, the browser errors can occasionally be inconsistent with normal Python (usually due to a bug in Pyjamas), and it’s a pain to troubleshoot these. And there is really not much Pyjamas can do to remedy this, in my opinion.

Adding additional assertions to catch every possible case to throw Pythonic errors is a fool’s errand no different than trying to parse HTML using regular expressions. Python’s ability to throw relevant assertions stems from its fundamental design. It’s very strict about using non-existing/undefined variables and comparison of irrelevant types. JavaScript, on the other hand, is very lazy/permissive about these, much like Perl. Python is proactive about its assertions, Pyjamas tries to be reactive. It’s unrealistic to forsee every special case that could arise and account for it with an assertion the same way Python would. Even if you manage to do so, you will have added even more code to Pyjamas’ already large chunk of boilerplate (not to mention potential for new Pyjamas bugs). One option is to compile these assertions away when the debug flag isn’t set, but even then you would be doing the exercise of examining all possible errors that Python could throw in each case, plugging in more “reactive” logic to make JavaScript work the same way. Instead, we should make the framework easy to debug in the environment it’s meant to be in. Since we can’t make JavaScript behave like Python, and we can’t do compile-time debugging like we would with C++ or Java, we should make the output easy to understand, so that we can map it back to the original code.

Python is not Java, DOM is not a Desktop

This brings me to my next point. GWT (the original inspiration for Pyjamas) might be more bloated than Pyjamas, but there is something it can do that Pyjamas can’t: compile-time error catching. If it wasn’t for Python being a dynamically-typed language, a lot of my rant in the previous section about debugging would be irrelevant. Additionally, I don’t feel that Pyjamas is approaching the problem from the right angle. Python has the advantage of being much more similar to JavaScript than Java ever will, and a lot of Pyjamas’ wrapper logic wouldn’t even be necessary if Pyjamas didn’t try to pretend to be GWT (in addition to pretending to be Python). GWT was designed to make web development similar to Desktop GUI development, since that’s the background many Java developers come from. What other purpose is there to fake MouseListener and KeyboardListener in an environment that wasn’t designed to need either (KeyboardListener, by the way, is another source of grief for Pyjamas – it’s what makes the keyboard pop-up all over the place on mobile devices, it also attaches a fake input element to the current element, pretending like they’re the same element, adding even more boilerplate and wrappers to the code)? What other purpose is there to build the entire DOM dynamically (which, by the way, is also extremely inefficient)? The browser page was not designed to function the same way your Desktop calculator app does. Anyone who has taken a few minutes to learn how the DOM works probably agrees that it’s actually superior to the old-fashioned Desktop way of writing the GUI. I’m lazy (otherwise I wouldn’t have written my front-end in Python), so when a new technology comes along that clearly makes my life easier, why ignore it?

If it wasn’t for trying to fake a Desktop GUI, Pyjamas wouldn’t need all these wrappers. Most other Python-faking frameworks allow one to invoke JavaScript logic as if it was a regular Python object/function. Pyjamas, on the other hand, requires one to first write a wrapper for Pyjamas Desktop using Python, then for the browser using some limbo version of Python/JavaScript hybrid (where you can’t even access elements of array using standard indexing), and finally rewrite a separate version of your limbo code for each non-compliant browser (definitely IE, and possibly some others). This wrapping might have been necessary in Java, but should not be needed for Python at all, and could have been prevented with better design. But wait, there IS an alternative! You can put raw JavaScript in your code using JS() method and passing it one giant string of JavaScript code. Unfortunately, that chunk of the code will get completely ignored in Pyjamas Desktop (which you’re using to debug your entire app, since the browser debugger is no help at all), and to actually reference anything from this chunk of code in the browser, you will need to reference these variables the same way: “a = JS(‘a’)” (again, don’t expect “a” to get set in Pyjamas Desktop). Oh, and don’t try to modify any of the DOM elements created by Pyjamas from anything other than Pyjamas, you will run into object state sync issues. Pyjamas wraps each DOM element in a Python object, which then stores the element’s state as a set of variables, and assumes it doesn’t change without Pyjamas’ permission. Pyjamas plays well with other JavaScript frameworks… as long as they don’t touch any portion of the DOM Pyjamas uses.

JavaScript has its Strengths

JavaScript might not be the cleanest language, and I still much prefer Python to it. But I must give it credit where credit is due. First of all, it integrates the DOM into itself really well. I can take any DOM element, assign a function to onMouseDown event as if it was a regular JavaScript object, and all of a sudden I got an element that reacts to my mouse clicks. No need for complicated ClickHandlers.

Pyjamas has a lot of abstraction layers, both to hide JavaScript inconsistencies, and make it easier to build widgets. However, native JavaScript libraries, like jQuery, do a much better job at both. Yes, jQuery doesn’t scale well for larger projects, but there are libraries that do, like MooTools (which, by the way, was inspired by Python). But realistically, if you create a simple wrapper for generating classes (or loot one from John Resig’s blog – the same guy who wrote jQuery), even jQuery becomes good enough for creating large projects. Pyjamas, on the other hand, adds so much abstraction, that sometimes I need hacks just to manipulate the DOM. If you look at the DOM of a typical Pyjamas app, you will notice layers of unnecessary elements: images wrapped in divs, wrapped in more divs, placed inside some table that resides inside yet another table. When I try to render my app on a tablet, it often crashes due to the DOM bloat.

Pyjamas also assumes that JavaScript is slow, which was true when the project first started. As a result, it duplicates parts of its boilerplate code to avoid an extra function call (while adding excessive function calls and abstractions in other places). Ironically, JavaScript engines have come a long way, and a lot of Pyjamas’ optimizations are no longer relevant (such as using object["property"] instead of object.property). In fact, a quick paint app I wrote in Pyjamas actually runs faster in Chrome than Pyjamas Desktop. That same app runs faster still when written in pure JavaScript. It’s especially noticeable when using the paint-bucket tool, which works by pixel-scanning and takes a couple seconds in Pyjamas yet almost instantaneous in JavaScript.

Summary

While Pyjamas is the most complete Python emulation in a browser, it has become a very bloated and brittle framework. It doesn’t embrace any portion of JavaScript, nor the DOM, trying to hide them away like some sort of deformed beast. By pretending to be pure Python, it not only puts unrealistic expecations on itself, but also fails to make use of good parts of JavaScript. Instead, Pyjamas embraces a solution designed for a statically-typed language, favoring a GUI structure that should have died a decade ago.

So What’s The Alternative?

I did mention that I am porting Grafpad away from Pyjamas. However, I’m not crazy enough to rewrite the entire project in pure JavaScript. Rewriting all the code in a language with different quirks and troubleshooting differences like division rounding and modulo signage is not my idea of fun. I also still prefer to keep my front-end code interchangeable with the back-end (more or less), which has already provided multiple advantages, such as moving the proprietary clipping and recognition algorithms to the back-end in just a few hours of work. I happen to have another ace up my sleeve. In the next post, I will review multiple alternatives for Pyjamas and explain the solution I’ve chosen.

Touch Gestures in Pyjamas

Before I continue, I wanted to apologize for the lack of updates. To pay the bills, I got a new job in October, which took away a lot of my free time. You also may be familiar with the current state of Pyjamas from the mailing list. Before I go further, I wanted to mention that I have realized Pyjamas is not the best API for Grafpad in November of last year, and have been working on alternative solutions (I will write up a detailed post explaining this soon). However, for those still using Pyjamas, I wanted to provide my solution for handling touch events on mobile devices.

Some of you probably noticed that Grafpad has had support for iPad/Android devices for almost a year now despite Pyjamas still lacking support for touch gestures. Some of you may have even seen the communication between Luke and me where I was trying to figure out how to write a proper touch event wrapper for Pyjamas. Alas, it has been almost a year, and the wrapper is nowhere to be seen, yet Grafpad got support for touch gestures within a few weeks of that conversation. So what happened?

To summarize what happened, after a few weeks of beating my head against the wall, I gave up on writing the wrapper. I have updated a dozen or so different Pyjamas modules, all the pieces seemed to be there. I’ve made sure that all places responsible for triggering mouse-event logic also triggered touch events, I even made sure my logic was consistent with GWT way of triggering touch events. Pyjamas wasn’t throwing any errors, but my touch handlers just weren’t firing. I could’ve spent more time to debug this correctly, but my main priority was Grafpad, and I couldn’t afford to spend more time on this.

In my frustration, I came up with a hack based on another set of blogs I saw (I no longer have the links to them, so if you recognize them from the source code, please reply to this post and I will add the link). The solution is actually quite simple, and required only some understanding of how the browsers already handle touch events.

Since most websites aren’t designed for mobile devices, mobile browsers have adopted some work-arounds, faking some mouse events to achieve sane behavior on most websites. For many websites these fake events work fine, for many web-apps they do not. In particular, when the user clicks an element on a website, the mobile browser first sends onMouseOver event (that’s right, not onMouseDown). The mobile browser then checks for a change in the DOM, if no change occurs, the browser then sends onMouseDown event. This is actually pretty smart, and in most cases works pretty well. Unfortunately for Grafpad, this triggers a tooltip and requires a second press to actually activate the option being clicked on. The next annoyance is due to how onMouseMove works. Unlike PCs/Macs, mobile devices do not send onMouseMove events continuously while the mouse/finger is pressed down. Instead, they send a single onMouseMove event when the mouse is released (simultaneously with onMouseUp). My guess is that this is done to conserve CPU resources, since continuous onMouseMove events can get pretty resource intensive. As a side-effect, Grafpad would refuse to draw the actual Shape the user would doodle, and instead produce only a straight line from the point of touch-start to the end. The last problem is the keyboard listener. Grafpad relies on it for keyboard shortcuts, having it active at all times. Mobile devices, unfortunately, assume that the website expects keyboard input form the user, activating a keyboard that takes up half of already scarce real-estate. So how did Grafpad deal with these issues?

Apparently, mobile browsers will only try to fake real mouse events if touch gestures aren’t already handled by the website itself. Fortunately, touch gestures are very easy to detect in pure JavaScript, when you don’t have to rely on additional wrappers. To summarize my solution in one sentence, I added some JavaScript code to detect real touch events, and fake mouse events at the same coordinates. This solved all issues except the keyboard. For the keyboard, I modified the keyboard handler CSS display style to none, which seemed to have worked on the iPad. Unfortunately, this didn’t do the trick on Android devices. While I still haven’t addressed those, an easy hack (I mean solution) would be to detect the browser/device from the user-agent string (or better yet, as soon as we notice a touch occur) and disable keyboard handler entirely (we know the user isn’t going to use keyboard shortcuts on a tablet anyway).

Now to the actual solution, here is the code that I added for faking mouse events:

var TOUCHDEVICE = false;
var LASTTOUCH = 0;

function touchHandler(event)
{
    TOUCHDEVICE = true;
    var touches = event.changedTouches,
        first = touches[0],
        type = "",
        ctrl = false;
         switch(event.type)
    {
        case "touchstart":
            var d = new Date()
            var current = d.getTime();
            if (current-LASTTOUCH <= 500){
                ctrl=true;
                event.preventDefault(); //prevent zoom-in lens on double-click
            }
            else {
                LASTTOUCH = current;
            }
            type = "mousedown";
            break;
        case "touchmove":
            type="mousemove";
            event.preventDefault(); //prevent screen dragging
            break;      
        case "touchend":
            type="mouseup";
            break;
        default: return;
    }

    var simulatedEvent = document.createEvent("MouseEvent");
    simulatedEvent.initMouseEvent(type, true, true, window, 1,
                              first.screenX, first.screenY,
                              first.clientX, first.clientY, ctrl,
                              false, false, false, 0/*left*/, null);

    first.target.dispatchEvent(simulatedEvent);
}

function init()
{
    document.addEventListener("touchstart", touchHandler, true);
    document.addEventListener("touchmove", touchHandler, true);
    document.addEventListener("touchend", touchHandler, true);
    document.addEventListener("touchcancel", touchHandler, true);  
}

The above code does several things to make touch events behave the way a user would expect. First of all, once init() gets called, we map every touch event to our function, preventing default behavior from the mobile browser. The actual touchHandler function then decides how to handle each touch event. There are a few things to note in the code. First, the TOUCHDEVICE boolean. This variable was never actually used, but was initially added for the purpose of disabling the keyboard handler on Android devices. The idea is that it gets triggered the first time a touch event occurs. On a PC/Mac such event will never occur, thus the user will not be affected by this functionality. On a mobile device, this event will occur as soon as the user performs the first touch, allowing us to disable the keyboard handler before it has any effect.

The next item of interest is the LASTTOUCH variable. This variable stores the time of the last touch event. As you can see from the code, it’s used to simulate the right-click event (ctrl+click) in case two taps are detected within 0.5 seconds of eachother.

The above logic should be sufficient for single-touch events. Multi-touch events are somewhat more complicated, and I will not cover them in this post (although you should be able to rewrite the above logic to handle those too, if needed – complete documentation for those is available on HTML5Rocks website). One last thing to add, to make the above code work, make sure to add the following attribute to your body tag:

<body ... onLoad="init();" >

Strategy pattern in Pyjamas

I was recently trying to implement a Strategy pattern inside my Pyjamas web app. It’s basically a design pattern that allows the programmer to use multiple algorithms interchangeably without relying on inheritance, and it’s meant for cases when inheritance would add too much complexity.

For example, imagine that we’re implementing a class for a video game character, which has an attack() method. This method would probably vary depending on the weapon the character is holding. If the character is bare-handed he will probably punch the opponent. If the character is holding a sword, he’ll swing the sword at the opponent instead. If the character is holding a spear, he’ll perform a throwing motion. In this case, inheritance is clearly a poor choice, we’re not going to morph the character into a new object each time he switches a weapon. One other way to implement this is by using if/elif/else statements, but this solution isn’t much better, since we’ll end up with a behemoth method spanning several hundred lines of code as soon as we have a decent arsenal of weapons. We could also implement if/elif/else setup that calls other methods, but even then we’re adding unnecessary overhead. The cleanest solution is a Strategy pattern, which essentially maps the correct method at run-time.

Unfortunately, Pyjamas doesn’t seem to support either the types module or get() method for binding a function/method to a class method. Without it, Strategy pattern seems impossible to implement in Python.

For example, using types module, a clean way to implement this pattern in Python would be as follows:

import types

def strategyA(possible_self):
...

instance = OrigObject()
instance.strategy = types.MethodType(strategyA, instance)
instance.strategy()


This runs fine in Pyjamas Desktop, but throws an import error in the browser since types module does not exist in current Pyjamas implementation:

ImportError: No module named types, types in context None


There is an internalast fork of Pyjamas that does implement the types module, but does not implement the MethodType function within it. Essentially, the only difference would be the type of error you get.

There is an alternative way to implement the Strategy pattern. Every function/method in Python has descriptor methods that allow you to customize when you use an attribute. One of such methods is get() which allows you to rebind the method as follows:

instance.strategy = strategyA.__get__(self, OrigObject)


Unfortunately this solution also doesn’t work, throwing a type error in Pyjamas (since descriptors aren’t yet implemented in Pyjamas):

TypeError: Object function ... has no method '__get__'



The main problem is that Pyjamas will not let you bind an unbound method to an instance in Pyjamas for use as a bound method later. Which brings us to a somewhat hackerish (is that a word?) workaround. The idea is to define all alternative methods inside the class itself, and then swap between them, as if using variables. For example, if we have two alternative algorithms, strategyA and strategyDefault, we can switch back and forth between them as follows or each instance of the class independently:

self.strategyDefault, self.strategyA = self.strategyA, self.strategyDefault

If you need more than 2 methods, you can still use the same approach, but instead of swapping 2 methods, set the active one to point to one of the alternatives, keeping a backup of the original in another variable.

I must admit that even despite being the language where “there should only be one obvious way to do it” Python allows the programmer great flexibility in terms of implementation. And that flexibility is one of Pyjamas’ best features, in that even if Pyjamas drops the ball, Python tends to have plenty of firepower to pick up the slack.

Adding True Internet Explorer 9 support to Pyjamas

About two months ago, Rich Newpol added Internet Explorer 9 support to pyjamas. Before then, pyjamas tried to use old mozilla (pre-3.5 version) format whenever it detected IE9, which would result in an epic fail. One small problem, however, is that Rich’s solution was not good enough for the app I’m developing. The solution was to tell IE9 to render the website the same way previous versions of IE would render it. Since my app relies heavily on HTML5 canvas element, this means I’d be stuck with old crappy VML. Instead I have started examining pyjamas to add true IE9 support. In the end, it turned out simpler than I thought. This post talks about how I did it, as well as the quirks I had to address.

Setting up ie9 directory

To maintain at least partial support in older IE browsers, I decided to create a new __ie9__ directory inside pyjamas library. My first goal was to get pyjamas to compile a separate cache file for IE9 (*.ie9.cache.html). To do so, I had to modify home.nocache.html file inside of boilerplate to detect ie9:

...
20 else if (ua.indexOf('msie7.0') != -1) {
21 return 'ie6';
22 }
23 else if (ua.indexOf('msie 8.0') != -1) {
24 return 'ie6';
25 }
26 else if (ua.indexOf('msie 9.0') != -1) {
27 return 'ie9';
28 }
...
50 window["prop$user.agent"] = function() {
51 var v = window["provider$user.agent"]();
52 switch (v) {
53 case "ie9":
54 case "ie6":
55 case "mozilla":
56 case "oldmoz":
57 case "opera":
58 case "safari":
59 return v;
60 default:
61 parent.__pygwt_onBadProperty("%(app_name)s", "user.agent",
["ie9", "ie6", "mozilla", "oldmoz", "opera", "safari"], v);
62 throw null;
63 }
64 };
...

This tells the website to load *.ie9.cache.html if the browser identifies itself as ‘msie 9.0′. The next task is to modify pyjamas so it actually generates *.ie9.cache.html file for us. The script responsible for this is pyjs/src/pyjs/browser.py. Here are the changes I made to it:

...
21 AVAILABLE_PLATFORMS = ('IE6', 'IE9', 'Opera', 'OldMoz', 'Safari', 'Mozilla')
...
45 class BrowserLinker(linker.BaseLinker):
46
47 # parents are specified in most-specific last
48 platform_parents = {
49 'mozilla':['browser'],
50 'ie6':['browser'],
51 'ie9':['browser'],
52 'safari':['browser'],
53 'oldmoz':['browser'],
54 'opera':['browser'],
55 }
...

With these files modified, pyjamas should now generate *.ie9.cache.html file for us, that the *.nocache.html file will load if you visit the page in IE9. The problem is that there is nothing in our ie9 folder yet.

Populating ie9 directory

The obvious starting point would probably be to copy the contents of __ie6__ directory into __ie9__ and start tweaking it from there. However, after playing with that approach for a while, I started realizing that __ie9__ is so different from previous versions that the best approach is to actually leave __ie9__ blank and start troubleshooting from there. It turns out that if you leave __ie9__ directory empty, the compiled *.ie9.cache.html file is actually somewhat usable. After some testing, I found that the only methods that did not work as expected were DOM.eventGetButton(), DOM.getAbsoluteLeft(), DOM.getAbsoluteTop() (it’s possible I missed some, but my very complex web app seems to be working fine in IE9 now). After writing my own implementations of those, I noticed that DOM.py from __mozilla__ implements all 3 of those methods similar to my personal implementation, and all 3 implementations fix IE9 as well. This is not too much of a surprise, since Opera 11 now seems to run just fine with mozilla cache file as well, as I reported on pyjamas mailing list here. Perhaps __mozilla__ should be renamed to __default__ or __compliant__? Anyway, if you’ve been following along, chances are that even with all these changes you still might have issues with your web-app layout. The next section is designed to help you troubleshoot those.

Layout and appearance issues

First of all, IE9 will automatically throw you into ‘Quirks Mode’ if you don’t define a correct DOCTYPE. Quirks mode is designed to help with rendering non-W3C-compliant webpages. Its purpose is to render pages that were designed for older browsers and are no longer compliant with today’s standards. You can find more about it on Wikipedia. Actually, all browsers have a quirks mode, and they will all throw you into it if you don’t define a valid DOCTYPE. And if you look through pyjamas examples, you’ll probably notice a consistent lack of DOCTYPE. That’s right, most of pyjamas examples actually run in quirks mode when opened inside a browser (and not just in IE). This is usually not a big deal, unless you happen to be running IE9.

It turns out IE9 does not support event handler when in quirks mode. Unfortunately, event handler is one of the most important building blocks for pyjamas, as it handles just about all user interaction. So to make your web app compatible with IE9, you will have to make it W3C-compliant, this is something that will affect all browsers, not just IE9. Most of your changes will be in your app’s main html file itself and your css stylesheet, although it’s possible that you will need some slight code changes as well (in particular if you used DOM.setStyleAttribute() method).

First of all, we need to define a valid DOCTYPE. To do so, open up your app’s main html file (should be located in your app’s public directory) and add this line at the very top (before tag):

<!DOCTYPE html>

This tells it to render your app using the latest html standard (which would be HTML5). If your app looks and works great, congratulations, you know your html and can probably stop reading now. If you’re someone like me, however, you’re probably staring at a deformed version of your app while scratching your head. First of all, let’s address the issues that are common across all browsers. To do so, read my previous post about making your app HTML5-compliant. Once done, your app should appear correctly in all browsers (except maybe some glitchtes in IE9). Here are the last few fixes for the annoyances with IE9.

If you use a text area element (pyjamas’ TextArea() class), IE9 seems to override the width and height you set in the stylesheet unless you use ‘!important’ tag. It also seemed to add a scrollbar for me, so I had to set ‘overflow’ property to auto as well. I also used this oportunity to prevent Chrome from making my text area resizable (that little corner in bottom-right corner that the user can drag) by setting ‘resize’ to none:

.gwt-TextArea {
    overflow: auto;
    resize: none;
    width: 300px !important; /* important tag needed for IE9 */
    height: 100px !important; /* important tag needed for IE9 */
}

The second annoyance seems to be that unlike other browsers, IE9 (when operating in strict mode) does not assume that a single value for a field should be applied to both dimensions. This means that to center a background image, for example (as I needed to do with my loading icon), you’d need to specify “center” twice in CSS stylesheet. That should be pretty much it, I have to admit that for the most part IE9 is actually more standards compliant than Safari or Chrome. There might be other quirks, which I haven’t ran into with my app. Feel free to mention them to me so I can update this post.

Making Your Pyjamas App HTML5-compliant

I recently decided to update my pyjamas app so that it’s HTML5-compliant. I did this because I recently customized my pyjamas version to work in Internet Explorer 9 (I needed true IE9 support, not the backwards compatibility mode current version of pyjamas provides). Unfortunately, IE9 seems to be very anal about it’s rendering modes, it will either render the page in strict HTML5, or revert to half-broken ‘Quirks Mode’ that doesn’t even support event handler. It seemed easier to tweak my app to be compatible with HTML5 (besides I’d have to do so eventually anyway) than to try to write a special version of event handler for IE just so that I can run my app in a mode that’s not even meant to be used by modern websites anyway.

First of all, I had to tell the browser to render the page in HTML5 mode. To do so, I opened the main .html file for the app (in project’s public directory) and added the DOCTYPE at the very top (before <html> tag):

<!DOCTYPE html>

This tells the browser to render the page using the latest html standard (which happens to be HTML5). Second, I noticed that my fancy “loading” gif no longer appeared in the center of the page (not vertically at least). Apparently, setting my pyjamas frame to 100% height was no longer good enough. In HTML5, I now have to set both, <body> and <html> tags to take up 100% height:

html, body {
    height:100%;
}

The next issue with HTML5 that I noticed is that all my images now seemed to have a small footer below them (a few pixels of space below the image). This is probably not a big deal for most, but when you use images as icons, or need to fit exactly within a certain container, it can definitely mess up your layout. The reason for that is that HTML5 modified default display style for images to ‘inline’, which treats them the same way as text, giving them a ‘line-height’ property. There are two ways to fix the images back to normal. One way is to define ‘line-height’ for the image as 0:

.gwt-Image {
    line-height: 0;
}

This was my initial solution, but there is a problem with it. There is a bug in Firefox that overrides line-height, even if you specify ‘!important’ after it. The other alternative is to change image back to the way it was rendered before HTML5:

.gwt-Image {
    display: block;
}

This solution works in all browsers. Additionally, I noticed that the the text I placed inside tables using pyjamas’ Label() class rendered with too much spacing, as if it put every cell inside

tags. Turns out the work around for that was the same as for the images (even though the problems differ). To render the tables correctly, I modified the above code as follows:

.gwt-Table > tbody, .gwt-Image {
    display: block; /* HTML5 effects panel alignment fix, and footer removal fix for images */
}

Note that for the above fixes to work, you must not be overriding the .gwt-Image that pyjamas assigns to images, and should add .gwt-Table style to tables:

table.setStyleName('gwt-Table')

Finally, I noticed that my fonts were no longer consistent and that tables whose dimensions were defined via CSS were rendering incorrectly as well. The reason for that is that HTML5 no longer assumes that unitless numbers correspond to pixels. Which means that for all fonts, you now need to define “pt”, “px”, or “em” after the number, unless the number is zero. So something like

.text {
    font-size: 10;
}

should be changed to

.text {
    font-size: 10px;
}

The same change also applies to tables, divs, etc. If you have something as follows in your CSS:

.some-box {
    width: 200;
    height: 150;
}

you should change it to

.some-box {
    width: 200px;
    height: 150px;
}

I realize that some web designer is probably reading this now laughing at me, since I should have known to define the units to begin with. However, coming from a programming background, I see CSS more as a necessary evil, and usually stop tweaking it as soon as the page renders as expected (and up until HTML5, it rendered as expected without ‘px’ defined). Along the same lines, if you use Widget.setSize(), Widget.setWidth(), or Widget.setHeight() methods in pyjamas, make sure that you pass in the units as well (Widget.setSize('100px', '50px')) or use Widget.setPixelSize() instead. That’s all you need to do to make your pyjamas app HTML5-compliant.

To summarize:

  1. Add <!DOCTYPE html> tag at the top of the main .html file
  2. Set html and body to 100% height in CSS
  3. set images and table’s tbody to display: block
  4. Add units to every number in CSS (unless the number is 0)

There are a few IE9-specific quirks that this will not solve, but I’ll discuss them in my next post.

Using Web2py

As promised in the GAE post, here is part 1 of using JSONRPC calls on GAE. I originally intended to write more of a how-to, but now that I’m halfway through, this seems like it should be it’s own post, and I’ll write a how-to part later.

If you’re reading this blog, you probably know how to write Pyjamas apps, but you might not know about web2py, and why or how to use it. I will give a quick overview on how web2py works in the 2nd half of this post, but if you want to just know about using web2py with Pyjamas, or deploying to GAE (on windows), skip to the next post.

I want to start out with a warning. You MUST be careful if you plan to develop an app on GAE. Let’s say hypothetically you have an idea for a business using webapps. You start out with no money and no users. GAE looks really appealing because you can get reliable hosting for free. A few months later you’ve grown and you’re ready to move to your own server. This is where you run into issues. All the server code was written for GAE. If you were good and your code is clean, you can copy several large chunks of code directly over to your new server, but interfaces for things like JSON calls, and interfaces to your database will all need to be rewritten. If you look around, you can find a lot of talk about “portability” and “lock in” and ways to get around this issue. It’s definately a hot topic when it comes to using GAE.

Lock in only happens if you aren’t careful up front. You don’t have to develop using GAE directly, instead I suggest using a web application framework, and build your code on top of that. At the very least, this will protect your code. I’m working on a small game, so I haven’t looked at all into moving database info, but this is something you might want to research. Worst case, you could write a web app to print out all your db info on a web page in a format that you can import to your new server. If anyone knows any good ways to move databases, feel free to leave a comment!

Back to frameworks. There are a couple frameworks available that support GAE. The two most popular are web2py and django. From what I read, neither framework is clearly better – each has its own strenths. It’s more about finding a framework that fits your needs more. For hobbyists, I’d recommend web2py. Web2py supports quicker development. A lot of people say it’s more intuitive and you can be more productive. I feel it aligns better with the Python philosophy. Django has it’s own strengths though. It gives you more control as a developer. Sometimes you’ll want that extra control, it’s really up to you.

If you’ve decided to go with web2py, or even if you didn’t, here is a quick overview of how it works from my non-SW perspective. There are 4 areas I heavily use inside each of my applications – model, controller, view, and the static dir. With that in mind, I want to start with the web2py URLs which take on the form of http://domain/application/controller/function (where function is closely related to view). You also have a static directory which can be accessed using http://domain/application/static/. The static dir is where you’ll have website images, Pyjamas apps, and other static files. There are some defaults setup so if someone visits http://domain/ your server will load something, but most URLs will take on the form http://domain/application/controller/function.

So what do controllers and functions do, and how are they related to views? Well you have a “controllers” folder in your application’s directory which is full of controller scripts made up of several functions. If someone visits http://domain/application/game/loadGame the server will call the game.py file in your controllers directory and run the loadGame() function in that file.

This leads to views. Some functions will be used to handle forms, or handle JSON calls, but your most important functions will return a dict(). When you return a dictionary, it is really returning a reference to the view with the same name as the function. Let’s say this loadGame() function returns a dictionary. In your views directory there should be a html file game/loadGame.html. This is what the server sends users when they visit http://domain/application/game/loadGame.

This views page will probably include links to js. files, images and other static files. This is the 3rd area, the static dir. When you have static files, you will host them here.

There’s one last area – models. The models area has the setup for any databases you’re using, and the setup for other things like the JSONRPC interface. When I tried adding users and storing high scores, I was using this file a lot.

So that’s web2py in a nutshell.