Equality Operator

Matrix-agent-Smith-clones One thing that always bothered me, and many others, is how useless the == operator is in JavaScript. It seems to combine the worst of both worlds from equality and identity:

  • When comparing primitive types, it acts as a lax equality operator
  • When comparing objects, it acts as an identity operator

I can’t think of any use case for which such behavior is desired. In fact, reversing this behavior would produce a much more useful operator since the concept of “equals” tends to relax as data gets more complex because some dimensions are simply irrelevant. Do I care that two objects were not created at the same exact time? Do I care that their IDs differ? Do I care if the measured length is 4.00000000001 inches rather than 4? When it comes to equality comparison, JavaScript fails miserably. This is actually one of Douglas Crockford’s pet peeves about the language, which he explains in more detail in his book: JavaScript: The Good Parts.

Fortunately, there is also a more consistent identity operator (===), often erroneously called “strict equality” by people who don’t understand the difference between identity and equality. Unfortunately, as I just mentioned in previous paragraph, there is no concept of equality operator in the language at all. For those unfamiliar with these concepts, identity answers the question of “Does A and B refer to the same object?” while equality answers the question of “Is B a clone of A?“.

Naturally, both of these questions come up in programming a lot, so it’s a shame that only one can be easily answered in JavaScript. You probably already stumbled into this problem if you ever tried something like this:

new Date('1/1/2000') == new Date('1/1/2000')

or this:

{"foo": 1} == {"foo": 1}

The problem is not unique to JavaScript, many other languages lack equality operator as well. Typically, however, those languages have an alternative mechanism for handling this case, such as operator overloading. Indeed, there are cases when operator overloading is actually superior – such as when objects contain meta data that one wishes to omit from comparison (i.e. exact creation time, unique id, etc.). Unfortunately, JavaScript doesn’t allow for operator overloading either, and herein lies the problem. While one can easily roll a proper equality function (and many libraries such as underscore already include one) you would still have to decide whether it’s worth using on a case-by-case basis.

We’re not in FORTRAN age anymore, where developers had to tweak each operation. Today we’re spoiled, we often can get away with simply telling the compiler what to do rather than how, and enjoy optimal performance 90% of the time with negligible overhead. One such example is the sort function. When was the last time you had to wonder if you should use merge or insertion sort? You simply use the built-in sort and assume that unless there is something very special about your data, you’re better off moving on to other things rather than attempting to optimize the last 10% out of the algorithm. In most modern languages equality operator falls in the same category. Sure, you’d be able to shave off a few microseconds by replacing that == (equality) with is (identity) for certain comparison operations in Python, but is it worth the extra brain cycles? Most of the time the answer is “No”.

Why then do we have to be explicitly aware of this in JavaScript? Moreover, why can’t RapydScript compile == into deep equality? First, I should mention that, like some other libraries/languages for JavaScript, RapydScript already has a deep equality comparison via the inbuilt eq function. Yes, eq(a, b) (previously called deep_eq) has been supported for years. The problem is that (up until recently) I wasn’t able to make the decision for you of whether you want eq or ==. The issue boils down to the fact that if I decide to compile == to equality across the board for the developer, I effectively introduce enormous overhead (about 700% according to jsperf) on the user in cases where he/she expected identity comparison instead (which is about 90% of the time, since primitives are a lot more common). There has been a lot of discussion about this, which you can follow in issues 93 and 94 on my github page.

As you probably already guessed from previous paragraph, I’m now able to bridge that gap. Effective last month (that’s right, I snuck a change in without anyone noticing), RapydScript is the first JavaScript-based transcompiler to support high-performance deep equality. How performant is this operation, you may ask? Well, take a look for yourself:

Screen Shot 2015-12-05 at 2.59.38 PM

According to jsperf, the overhead is negligible (that’s right, the measurement noise is greater than any visible difference – as you can see from the deep-equality version outperforming identity). So what changed? Why am I suddenly able to blow the doors off of typical deep equality? Well, nothing new happened in JavaScript world. What changed is my approach. I decided to think about the problem creatively and had a sudden eureka moment (which I’m sure other compilers will copy in the future). Unfortunately you’ve probably already looked at the code from JsPerf above, ruining my surprise. But in case you haven’t, here is the pattern I came up with:

A === B || typeof A === "object" && eq(A, B)

How does it work? As you probably learned in your introductory programming class (assuming it was a legitimate C/Java class rather than an online tutorial about JavaScript), binary operators have short-circuit ability in just about all languages. This means that if left-hand side is truthy for || (or) operator, or falsy for && (and), the rest of the line is ignored (it’s as if it’s not there). That means that if A and B are primitives that are equal, the above will be equivalent to a simple A === B comparison. That handles the positive case, but negative is a bit trickier. I was struggling with it for a while (yes, the above equation makes it seem simpler than it really is – everything seems easy in hindsight), until I found a performant operation that works in both, browser and node (my first attempt was to check if A.constructor exists, which doesn’t work on all platforms). Fortunately, the very same “feature” that makes typeof useless in most cases (the fact that every non-primitive is an object) becomes the saving grace of this operation. As you can see from the JsPerf test, the overhead for this operation is negligible as well.

The best part is that unlike native JavaScript, where the developer would have to type that out by hand (because hiding it in a function call introduces the 700% overhead we’re trying to avoid), RapydScript can unroll == operator into that magic automatically. You’re probably wondering, then, how is it that this change has been in RapydScript for over a month if the == still compiles to ===? Well, for safety I’ve hidden this operator behind an import. If you wish all your == operators to compile to proper equality test shown above, add the following line at the top of your file:

from danger_zone import equality

That’s it. Now all your == will compile to the magic shown above, and != will compile to its inverse (thanks to DeMorgan’s Law). The above import will also tweak implementation of indexOf and lastIndexOf to perform deep equality as well, which in turn makes tests like if a in arr be based on deep equality (delivering a consistent experience across the board). As before, the identity operator is still there as well, via is and is not, which compile to === and !==, respectively. In the future, I also want to add an optimizer to the compiler, which will strip away the remainder of the line at compile time if one of the operands is a constant.

Scoping in JavaScript

I do a lot of programming in Perl, not because I like it, but because the company I work for uses it as its main language. In fact, I hate Perl (it tries to be overly implicit), but I do like how it handles scoping. Any variable declared inside a block (anything surrounded by {} brackets) will be local to that inner-most block. This means that variables declared inside loops, conditionals or even stand-alone {} will not be seen from the outside.

JavaScript will not localize variables like this. Any var declaration will simply be scoped to the inner-most function. That doesn’t mean, however, that you can’t use (function() {})(); same way you would use the brackets in Perl. In fact, you’re probably already familiar with wrapping chunks of code in the above pattern. Many developers do it to prevent leaking variables into global scope, including Facebook’s Like button:

(function(d, s, id) {
  var js, fjs = d.getElementsByTagName(s)[0];
  if (d.getElementById(id)) return;
  js = d.createElement(s); js.id = id;
  js.src = "//connect.facebook.net/en_US/all.js#xfbml=1";
  fjs.parentNode.insertBefore(js, fjs);
}(document, 'script', 'facebook-jssdk'));

Similar pattern, however, can also be applied anywhere else in your code. Consider a page, for example, with multiple elements sharing similar element ID structure, varying only by the index used within the ID. We then want to iterate through these elements, giving them all a click-handler. The following jQuery-based code seems like it will do the job:

for (var i=0; i<n; i++) {
    var cachedi = i;
    $('#' + i + '-element').click(function() {
        $('#' + cachedi + '-popup').show();
    })
}

At first glance, this code looks fine. We made sure to cache the index so that cachedi gets the value of i at the time of the function creation rather than using i directly, which would use i at the time of function call (after the loop terminates and i is set to n). However, running the above code we still get all elements attempting to trigger the popup with [n]-popup ID. The problem is that our declaration of cachedi gets moved outside the for loop and the same instance of the variable gets used in every single closure we generate inside the loop. There is an easy work-around, however:

for (var i=0; i<n; i++) {
    (function() {
        var cachedi = i;
        $('#' + i + '-element').click(function() {
            $('#' + cachedi + '-popup').show();
        })
    })();
}

Now our code works as expected. This is a handy trick for anyone wishing saner scoping in JavaScript. In fact, I’d prefer that RapydScript would scope things this way too, but that would contradict Python’s loop scoping. An alternative to this trick (and probably a more orthodox solution in JavaScript) would be to move the cachedi declaration inside the function making use of this closure.

Touch Gestures in Pyjamas

Before I continue, I wanted to apologize for the lack of updates. To pay the bills, I got a new job in October, which took away a lot of my free time. You also may be familiar with the current state of Pyjamas from the mailing list. Before I go further, I wanted to mention that I have realized Pyjamas is not the best API for Grafpad in November of last year, and have been working on alternative solutions (I will write up a detailed post explaining this soon). However, for those still using Pyjamas, I wanted to provide my solution for handling touch events on mobile devices.

Some of you probably noticed that Grafpad has had support for iPad/Android devices for almost a year now despite Pyjamas still lacking support for touch gestures. Some of you may have even seen the communication between Luke and me where I was trying to figure out how to write a proper touch event wrapper for Pyjamas. Alas, it has been almost a year, and the wrapper is nowhere to be seen, yet Grafpad got support for touch gestures within a few weeks of that conversation. So what happened?

To summarize what happened, after a few weeks of beating my head against the wall, I gave up on writing the wrapper. I have updated a dozen or so different Pyjamas modules, all the pieces seemed to be there. I’ve made sure that all places responsible for triggering mouse-event logic also triggered touch events, I even made sure my logic was consistent with GWT way of triggering touch events. Pyjamas wasn’t throwing any errors, but my touch handlers just weren’t firing. I could’ve spent more time to debug this correctly, but my main priority was Grafpad, and I couldn’t afford to spend more time on this.

In my frustration, I came up with a hack based on another set of blogs I saw (I no longer have the links to them, so if you recognize them from the source code, please reply to this post and I will add the link). The solution is actually quite simple, and required only some understanding of how the browsers already handle touch events.

Since most websites aren’t designed for mobile devices, mobile browsers have adopted some work-arounds, faking some mouse events to achieve sane behavior on most websites. For many websites these fake events work fine, for many web-apps they do not. In particular, when the user clicks an element on a website, the mobile browser first sends onMouseOver event (that’s right, not onMouseDown). The mobile browser then checks for a change in the DOM, if no change occurs, the browser then sends onMouseDown event. This is actually pretty smart, and in most cases works pretty well. Unfortunately for Grafpad, this triggers a tooltip and requires a second press to actually activate the option being clicked on. The next annoyance is due to how onMouseMove works. Unlike PCs/Macs, mobile devices do not send onMouseMove events continuously while the mouse/finger is pressed down. Instead, they send a single onMouseMove event when the mouse is released (simultaneously with onMouseUp). My guess is that this is done to conserve CPU resources, since continuous onMouseMove events can get pretty resource intensive. As a side-effect, Grafpad would refuse to draw the actual Shape the user would doodle, and instead produce only a straight line from the point of touch-start to the end. The last problem is the keyboard listener. Grafpad relies on it for keyboard shortcuts, having it active at all times. Mobile devices, unfortunately, assume that the website expects keyboard input form the user, activating a keyboard that takes up half of already scarce real-estate. So how did Grafpad deal with these issues?

Apparently, mobile browsers will only try to fake real mouse events if touch gestures aren’t already handled by the website itself. Fortunately, touch gestures are very easy to detect in pure JavaScript, when you don’t have to rely on additional wrappers. To summarize my solution in one sentence, I added some JavaScript code to detect real touch events, and fake mouse events at the same coordinates. This solved all issues except the keyboard. For the keyboard, I modified the keyboard handler CSS display style to none, which seemed to have worked on the iPad. Unfortunately, this didn’t do the trick on Android devices. While I still haven’t addressed those, an easy hack (I mean solution) would be to detect the browser/device from the user-agent string (or better yet, as soon as we notice a touch occur) and disable keyboard handler entirely (we know the user isn’t going to use keyboard shortcuts on a tablet anyway).

Now to the actual solution, here is the code that I added for faking mouse events:

var TOUCHDEVICE = false;
var LASTTOUCH = 0;

function touchHandler(event)
{
    TOUCHDEVICE = true;
    var touches = event.changedTouches,
        first = touches[0],
        type = "",
        ctrl = false;
         switch(event.type)
    {
        case "touchstart":
            var d = new Date()
            var current = d.getTime();
            if (current-LASTTOUCH <= 500){
                ctrl=true;
                event.preventDefault(); //prevent zoom-in lens on double-click
            }
            else {
                LASTTOUCH = current;
            }
            type = "mousedown";
            break;
        case "touchmove":
            type="mousemove";
            event.preventDefault(); //prevent screen dragging
            break;      
        case "touchend":
            type="mouseup";
            break;
        default: return;
    }

    var simulatedEvent = document.createEvent("MouseEvent");
    simulatedEvent.initMouseEvent(type, true, true, window, 1,
                              first.screenX, first.screenY,
                              first.clientX, first.clientY, ctrl,
                              false, false, false, 0/*left*/, null);

    first.target.dispatchEvent(simulatedEvent);
}

function init()
{
    document.addEventListener("touchstart", touchHandler, true);
    document.addEventListener("touchmove", touchHandler, true);
    document.addEventListener("touchend", touchHandler, true);
    document.addEventListener("touchcancel", touchHandler, true);  
}

The above code does several things to make touch events behave the way a user would expect. First of all, once init() gets called, we map every touch event to our function, preventing default behavior from the mobile browser. The actual touchHandler function then decides how to handle each touch event. There are a few things to note in the code. First, the TOUCHDEVICE boolean. This variable was never actually used, but was initially added for the purpose of disabling the keyboard handler on Android devices. The idea is that it gets triggered the first time a touch event occurs. On a PC/Mac such event will never occur, thus the user will not be affected by this functionality. On a mobile device, this event will occur as soon as the user performs the first touch, allowing us to disable the keyboard handler before it has any effect.

The next item of interest is the LASTTOUCH variable. This variable stores the time of the last touch event. As you can see from the code, it’s used to simulate the right-click event (ctrl+click) in case two taps are detected within 0.5 seconds of eachother.

The above logic should be sufficient for single-touch events. Multi-touch events are somewhat more complicated, and I will not cover them in this post (although you should be able to rewrite the above logic to handle those too, if needed – complete documentation for those is available on HTML5Rocks website). One last thing to add, to make the above code work, make sure to add the following attribute to your body tag:

<body ... onLoad="init();" >

Strategy pattern in Pyjamas

I was recently trying to implement a Strategy pattern inside my Pyjamas web app. It’s basically a design pattern that allows the programmer to use multiple algorithms interchangeably without relying on inheritance, and it’s meant for cases when inheritance would add too much complexity.

For example, imagine that we’re implementing a class for a video game character, which has an attack() method. This method would probably vary depending on the weapon the character is holding. If the character is bare-handed he will probably punch the opponent. If the character is holding a sword, he’ll swing the sword at the opponent instead. If the character is holding a spear, he’ll perform a throwing motion. In this case, inheritance is clearly a poor choice, we’re not going to morph the character into a new object each time he switches a weapon. One other way to implement this is by using if/elif/else statements, but this solution isn’t much better, since we’ll end up with a behemoth method spanning several hundred lines of code as soon as we have a decent arsenal of weapons. We could also implement if/elif/else setup that calls other methods, but even then we’re adding unnecessary overhead. The cleanest solution is a Strategy pattern, which essentially maps the correct method at run-time.

Unfortunately, Pyjamas doesn’t seem to support either the types module or get() method for binding a function/method to a class method. Without it, Strategy pattern seems impossible to implement in Python.

For example, using types module, a clean way to implement this pattern in Python would be as follows:

import types

def strategyA(possible_self):
...

instance = OrigObject()
instance.strategy = types.MethodType(strategyA, instance)
instance.strategy()


This runs fine in Pyjamas Desktop, but throws an import error in the browser since types module does not exist in current Pyjamas implementation:

ImportError: No module named types, types in context None


There is an internalast fork of Pyjamas that does implement the types module, but does not implement the MethodType function within it. Essentially, the only difference would be the type of error you get.

There is an alternative way to implement the Strategy pattern. Every function/method in Python has descriptor methods that allow you to customize when you use an attribute. One of such methods is get() which allows you to rebind the method as follows:

instance.strategy = strategyA.__get__(self, OrigObject)


Unfortunately this solution also doesn’t work, throwing a type error in Pyjamas (since descriptors aren’t yet implemented in Pyjamas):

TypeError: Object function ... has no method '__get__'



The main problem is that Pyjamas will not let you bind an unbound method to an instance in Pyjamas for use as a bound method later. Which brings us to a somewhat hackerish (is that a word?) workaround. The idea is to define all alternative methods inside the class itself, and then swap between them, as if using variables. For example, if we have two alternative algorithms, strategyA and strategyDefault, we can switch back and forth between them as follows or each instance of the class independently:

self.strategyDefault, self.strategyA = self.strategyA, self.strategyDefault

If you need more than 2 methods, you can still use the same approach, but instead of swapping 2 methods, set the active one to point to one of the alternatives, keeping a backup of the original in another variable.

I must admit that even despite being the language where “there should only be one obvious way to do it” Python allows the programmer great flexibility in terms of implementation. And that flexibility is one of Pyjamas’ best features, in that even if Pyjamas drops the ball, Python tends to have plenty of firepower to pick up the slack.

Making Your Pyjamas App HTML5-compliant

I recently decided to update my pyjamas app so that it’s HTML5-compliant. I did this because I recently customized my pyjamas version to work in Internet Explorer 9 (I needed true IE9 support, not the backwards compatibility mode current version of pyjamas provides). Unfortunately, IE9 seems to be very anal about it’s rendering modes, it will either render the page in strict HTML5, or revert to half-broken ‘Quirks Mode’ that doesn’t even support event handler. It seemed easier to tweak my app to be compatible with HTML5 (besides I’d have to do so eventually anyway) than to try to write a special version of event handler for IE just so that I can run my app in a mode that’s not even meant to be used by modern websites anyway.

First of all, I had to tell the browser to render the page in HTML5 mode. To do so, I opened the main .html file for the app (in project’s public directory) and added the DOCTYPE at the very top (before <html> tag):

<!DOCTYPE html>

This tells the browser to render the page using the latest html standard (which happens to be HTML5). Second, I noticed that my fancy “loading” gif no longer appeared in the center of the page (not vertically at least). Apparently, setting my pyjamas frame to 100% height was no longer good enough. In HTML5, I now have to set both, <body> and <html> tags to take up 100% height:

html, body {
    height:100%;
}

The next issue with HTML5 that I noticed is that all my images now seemed to have a small footer below them (a few pixels of space below the image). This is probably not a big deal for most, but when you use images as icons, or need to fit exactly within a certain container, it can definitely mess up your layout. The reason for that is that HTML5 modified default display style for images to ‘inline’, which treats them the same way as text, giving them a ‘line-height’ property. There are two ways to fix the images back to normal. One way is to define ‘line-height’ for the image as 0:

.gwt-Image {
    line-height: 0;
}

This was my initial solution, but there is a problem with it. There is a bug in Firefox that overrides line-height, even if you specify ‘!important’ after it. The other alternative is to change image back to the way it was rendered before HTML5:

.gwt-Image {
    display: block;
}

This solution works in all browsers. Additionally, I noticed that the the text I placed inside tables using pyjamas’ Label() class rendered with too much spacing, as if it put every cell inside

tags. Turns out the work around for that was the same as for the images (even though the problems differ). To render the tables correctly, I modified the above code as follows:

.gwt-Table > tbody, .gwt-Image {
    display: block; /* HTML5 effects panel alignment fix, and footer removal fix for images */
}

Note that for the above fixes to work, you must not be overriding the .gwt-Image that pyjamas assigns to images, and should add .gwt-Table style to tables:

table.setStyleName('gwt-Table')

Finally, I noticed that my fonts were no longer consistent and that tables whose dimensions were defined via CSS were rendering incorrectly as well. The reason for that is that HTML5 no longer assumes that unitless numbers correspond to pixels. Which means that for all fonts, you now need to define “pt”, “px”, or “em” after the number, unless the number is zero. So something like

.text {
    font-size: 10;
}

should be changed to

.text {
    font-size: 10px;
}

The same change also applies to tables, divs, etc. If you have something as follows in your CSS:

.some-box {
    width: 200;
    height: 150;
}

you should change it to

.some-box {
    width: 200px;
    height: 150px;
}

I realize that some web designer is probably reading this now laughing at me, since I should have known to define the units to begin with. However, coming from a programming background, I see CSS more as a necessary evil, and usually stop tweaking it as soon as the page renders as expected (and up until HTML5, it rendered as expected without ‘px’ defined). Along the same lines, if you use Widget.setSize(), Widget.setWidth(), or Widget.setHeight() methods in pyjamas, make sure that you pass in the units as well (Widget.setSize('100px', '50px')) or use Widget.setPixelSize() instead. That’s all you need to do to make your pyjamas app HTML5-compliant.

To summarize:

  1. Add <!DOCTYPE html> tag at the top of the main .html file
  2. Set html and body to 100% height in CSS
  3. set images and table’s tbody to display: block
  4. Add units to every number in CSS (unless the number is 0)

There are a few IE9-specific quirks that this will not solve, but I’ll discuss them in my next post.

Loading Notification for Web Apps

If you fooled around with some of the more complex Pyjamas examples, such as GChartTestApp, you probably noticed that they take quite a while to load. My apps too have been steadily growing in size and can take up to several seconds to load. It makes sense to inform the user that the app is loading and the browser is not frozen. One way to do so is by doing exactly what GChartTestApp does, displaying text that says “Please wait, the app is loading”. But since this is an app after all, I wanted something a bit more elegant than static test. At the same time, since the app is already large enough, I didn’t want to add any overhead to it by writing a new module for the loading screen. My solution was pretty simple, I decided to go find an animated .gif image that looks like a loading bar or an hour glass and slap it as background image in the middle of the screen. It uses no resources (aside from loading the image to begin with) and the best part is your app will load on top of it, hiding the image after the load completes (assuming your app background is not transparent).

First I decided to go find an image. What I found, however, was even better. http://ajaxload.info is a loading .gif generator that lets you choose a template and customize the color to fit your website. Then I wrote a quick body tag for my CSS stylesheet to setup the image properly:

body {
   height:100%;
   background-color: white;
   background:url(loading.gif) no-repeat center;
}

Height is necessary to make sure the image appears in the center of the screen, otherwise HTML defaults to height of zero and expands it as more content appears (since there is no content, the background image will appear at height of 0, with half of it cut off). If you’re using the standard CSS style sheets from examples, most apps don’t need more work than that, since the opaque background of the panels will hide the loading image. If you’ve picked one of the panels that don’t have a style sheet, or GWTCanvas/HTML5Canvas (which has transparent background by default), you can either create a new style sheet and assign it to the main panel, or use DOM to change the background of the panel without creating a style sheet:

DOM.setStyleAttribute(self.getElement(), "background", "white")

Now just place the loading image in your output directory and reload the page.

Modulo, String, Import Bugs

UPDATE: Most (if not all) of these bugs have been fixed since I started using Pyjamas. This is here mainly for reference.

Pyjamas has a very useful migration guide which summarizes the current issues/differences between JS and Python that the developers did not yet resolve. It’s a very useful read for anyone using Pyjamas, both with JS and Python background and a must-read for anyone wishing to make an app that works both in browser and in Pyjamas-Desktop. I did find a few issues, however, that weren’t yet documented. So I decided to post them here, including the work-arounds.

Importing radians function from math

Description:

For some reason, the following code will cause an ImportError in JS:

import math
math.radians(degs)

However, doing the same with math.pi, math.sin, math.cos, math.sqrt works fine

Solution:

Either use the following:

from math import radians
radians(degs)

Or:

from math import pi
degs*pi/180

Concatenating two strings using +

Description:

The following works fine in Python and Pyjamas-Desktop, but throws an exception (that Pyjamas catches, and gives “inform the webmaster” error):

b="jamas"
a="py"+b

Ironically, passing the same into a function as an argument works fine in both:

b="jamas"
fun("py"+b)

Solution:

The following alternative works in both:

b="jamas"
a=''.join(["py",b])

Alternatively, if you want a cleaner-looking concatenation:

def strcat(*args):
  return ''.join(args) b="jamas"
a=strcat("py",b)

Different modulo implementation

Description:

When performing the modulo command, Python uses the sign of the divisor for the result. JS, however uses the sign of the dividend. The following code in Pyjamas-Desktop will set a to 0.6, while in JS/browser it will set it to -0.4:

a=(-0.4)%1

Solution:

So far, my solution is to do the following:

a=dividend%divisor if a<0:
   a+=divisor

I also tried using math.fmod() instead, but ran into the same ImportError issue as described above for radians, unfortunately using the work-around described above only works for radians(). For fmod() Pyjamas will throw “$module.math.fmod is not a function” exception