Touch Gestures in Pyjamas

Before I continue, I wanted to apologize for the lack of updates. To pay the bills, I got a new job in October, which took away a lot of my free time. You also may be familiar with the current state of Pyjamas from the mailing list. Before I go further, I wanted to mention that I have realized Pyjamas is not the best API for Grafpad in November of last year, and have been working on alternative solutions (I will write up a detailed post explaining this soon). However, for those still using Pyjamas, I wanted to provide my solution for handling touch events on mobile devices.

Some of you probably noticed that Grafpad has had support for iPad/Android devices for almost a year now despite Pyjamas still lacking support for touch gestures. Some of you may have even seen the communication between Luke and me where I was trying to figure out how to write a proper touch event wrapper for Pyjamas. Alas, it has been almost a year, and the wrapper is nowhere to be seen, yet Grafpad got support for touch gestures within a few weeks of that conversation. So what happened?

To summarize what happened, after a few weeks of beating my head against the wall, I gave up on writing the wrapper. I have updated a dozen or so different Pyjamas modules, all the pieces seemed to be there. I’ve made sure that all places responsible for triggering mouse-event logic also triggered touch events, I even made sure my logic was consistent with GWT way of triggering touch events. Pyjamas wasn’t throwing any errors, but my touch handlers just weren’t firing. I could’ve spent more time to debug this correctly, but my main priority was Grafpad, and I couldn’t afford to spend more time on this.

In my frustration, I came up with a hack based on another set of blogs I saw (I no longer have the links to them, so if you recognize them from the source code, please reply to this post and I will add the link). The solution is actually quite simple, and required only some understanding of how the browsers already handle touch events.

Since most websites aren’t designed for mobile devices, mobile browsers have adopted some work-arounds, faking some mouse events to achieve sane behavior on most websites. For many websites these fake events work fine, for many web-apps they do not. In particular, when the user clicks an element on a website, the mobile browser first sends onMouseOver event (that’s right, not onMouseDown). The mobile browser then checks for a change in the DOM, if no change occurs, the browser then sends onMouseDown event. This is actually pretty smart, and in most cases works pretty well. Unfortunately for Grafpad, this triggers a tooltip and requires a second press to actually activate the option being clicked on. The next annoyance is due to how onMouseMove works. Unlike PCs/Macs, mobile devices do not send onMouseMove events continuously while the mouse/finger is pressed down. Instead, they send a single onMouseMove event when the mouse is released (simultaneously with onMouseUp). My guess is that this is done to conserve CPU resources, since continuous onMouseMove events can get pretty resource intensive. As a side-effect, Grafpad would refuse to draw the actual Shape the user would doodle, and instead produce only a straight line from the point of touch-start to the end. The last problem is the keyboard listener. Grafpad relies on it for keyboard shortcuts, having it active at all times. Mobile devices, unfortunately, assume that the website expects keyboard input form the user, activating a keyboard that takes up half of already scarce real-estate. So how did Grafpad deal with these issues?

Apparently, mobile browsers will only try to fake real mouse events if touch gestures aren’t already handled by the website itself. Fortunately, touch gestures are very easy to detect in pure JavaScript, when you don’t have to rely on additional wrappers. To summarize my solution in one sentence, I added some JavaScript code to detect real touch events, and fake mouse events at the same coordinates. This solved all issues except the keyboard. For the keyboard, I modified the keyboard handler CSS display style to none, which seemed to have worked on the iPad. Unfortunately, this didn’t do the trick on Android devices. While I still haven’t addressed those, an easy hack (I mean solution) would be to detect the browser/device from the user-agent string (or better yet, as soon as we notice a touch occur) and disable keyboard handler entirely (we know the user isn’t going to use keyboard shortcuts on a tablet anyway).

Now to the actual solution, here is the code that I added for faking mouse events:

var TOUCHDEVICE = false;
var LASTTOUCH = 0;

function touchHandler(event)
{
    TOUCHDEVICE = true;
    var touches = event.changedTouches,
        first = touches[0],
        type = "",
        ctrl = false;
         switch(event.type)
    {
        case "touchstart":
            var d = new Date()
            var current = d.getTime();
            if (current-LASTTOUCH <= 500){
                ctrl=true;
                event.preventDefault(); //prevent zoom-in lens on double-click
            }
            else {
                LASTTOUCH = current;
            }
            type = "mousedown";
            break;
        case "touchmove":
            type="mousemove";
            event.preventDefault(); //prevent screen dragging
            break;      
        case "touchend":
            type="mouseup";
            break;
        default: return;
    }

    var simulatedEvent = document.createEvent("MouseEvent");
    simulatedEvent.initMouseEvent(type, true, true, window, 1,
                              first.screenX, first.screenY,
                              first.clientX, first.clientY, ctrl,
                              false, false, false, 0/*left*/, null);

    first.target.dispatchEvent(simulatedEvent);
}

function init()
{
    document.addEventListener("touchstart", touchHandler, true);
    document.addEventListener("touchmove", touchHandler, true);
    document.addEventListener("touchend", touchHandler, true);
    document.addEventListener("touchcancel", touchHandler, true);  
}

The above code does several things to make touch events behave the way a user would expect. First of all, once init() gets called, we map every touch event to our function, preventing default behavior from the mobile browser. The actual touchHandler function then decides how to handle each touch event. There are a few things to note in the code. First, the TOUCHDEVICE boolean. This variable was never actually used, but was initially added for the purpose of disabling the keyboard handler on Android devices. The idea is that it gets triggered the first time a touch event occurs. On a PC/Mac such event will never occur, thus the user will not be affected by this functionality. On a mobile device, this event will occur as soon as the user performs the first touch, allowing us to disable the keyboard handler before it has any effect.

The next item of interest is the LASTTOUCH variable. This variable stores the time of the last touch event. As you can see from the code, it’s used to simulate the right-click event (ctrl+click) in case two taps are detected within 0.5 seconds of eachother.

The above logic should be sufficient for single-touch events. Multi-touch events are somewhat more complicated, and I will not cover them in this post (although you should be able to rewrite the above logic to handle those too, if needed – complete documentation for those is available on HTML5Rocks website). One last thing to add, to make the above code work, make sure to add the following attribute to your body tag:

<body ... onLoad="init();" >
This entry was posted in Frameworks, Hacks and Tricks and tagged , by Alexander Tsepkov. Bookmark the permalink.

About Alexander Tsepkov

Founder and CEO of Pyjeon. He started out with C++, but switched to Python as his main programming language due to its clean syntax and productivity. He often uses other languages for his work as well, such as JavaScript, Perl, and RapydScript. His posts tend to cover user experience, design considerations, languages, web development, Linux environment, as well as challenges of running a start-up.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>