Lean Startup Challenge: Weekly Reflection 2

For redundancy, I’ve been running Google Analytics and KISSmetrics along-side Performable the entire time. One thing I noticed is that all 3 seem to be tracking visitors a bit differently, so I wouldn’t rely on just one of them alone. Performable, for example, has been over-reporting new visitors and under-reporting returning ones (there have also been blatant inconsistencies in the numbers it’s been reporting the last 2-3 days, although it’s been accurate before -it’s possible that it’s getting confused by my new website). KISSmetrics has been hiding the visitors who navigate the funnel in a different order than it expects. Google Analytics is delayed by 1 day and doesn’t seem to have the idea of funnels. For a more detailed comparison, check out my blog after the competition is over, I will be comparing all 3 in much greater detail.

But before I jump into the metrics I wanted to go over the changes/experiments from this week. Last week I realized that people were getting confused by Grafpad, both the website and the app itself. To remedy that I have made a number of changes to the website, as well as the app.

I’ve gotten some very useful qualitative feedback about the website last week. There were recurring patterns in the feedback, which is what I concentrated on addressing first. The original paragraph on the old website, for example, would tell the user that Grafpad is now live and he/she is free to try it. It doesn’t, however, tell the user what Grafpad is and why the visitor should care about it. In other words, it doesn’t pass the 5-second test (which is about how much time a typical visitor will spend on the site unless there is something to grab the attention).

The visitors also mentioned that the page felt a bit plain, and doesn’t feel like a website. So after browsing around the web for a bit, looking at other websites, such as http://yamlabs.com/http://www.memrise.com/welcome/, or http://www.lucidchart.com/, I started noticing that many websites have 3 main bullets or options to grab the user’s attention on the main page. I decided to rearrange my bullets to do the same. The results of my tweaking can be seen below.

I’ve essentially picked the 3 key features of Grafpad people were getting confused about last week and moved them to the very top. The other features I’ve put in a slideshow container right below the main menu, figuring those features are secondary and many first time visitors might not even pay attention to them. I’ve also added a feedback form at the bottom of the page (not visible on the screenshot above) to let users mention their first impressions. So far, however, only 2 people have used the form, which leads me to wonder if I should move it closer to the top of the page.

As I mentioned earlier, I’ve been running Google Analytics on my site during the entire duration of the competition. It has a very nice feature, tracking the most popular search results that lead visitors to your site from Google. One thing I noticed last week is that the number one keyword was the IP address of my website. And some of you might remember that the domain name would change to the IP as the page would load (the issue occurred because we were using port 8000 instead of the regular port 80). My theory is that people were searching for the IP because they were assuming it’s a phishing site, and wanted to confirm that it’s not the case using Google.

Since then, I have moved my hosting to Amazon, and as you can see from the screenshot on the left, the IP address is no longer the most popular term. In fact, the number of searches for it has not increased at all since URL issue has been fixed.


The application also got a few minor enhancements based on user Feedback from last week. One of the first changes you’ll notice are the help bubbles that now pop up whenever you try to do some action for the first time that has confusing features.

I’ve gotten a lot of positive feedback about them, and I’m thinking of expanding them to cover help in more detail soon. At the same time I don’t want to swamp the user with help messages, because then he/she is not likely to read them either. Perhaps I should delay some help messages until the second time user plays with the application? I’m open to suggestions.

In addition to the help bubbles, I’ve modified success/fail messages to appear in similar bubbles for the sake of consistency. I’ve also done a bunch of minor bug fixes and interface changes since I got feedback that the login/sign-up prompts looked a bit plain. I definitely did not have as much time as I’d like to work on the app this week. Between addressing various usability issues that the users brought up and meeting with other entrepreneurs/groups this week to get a better understanding of what people actually expect/want from an application like Grafpad, I barely even had time to sleep this week.

This week we also setup a Facebook page for Pyjeon (the name of the company, since Grafpad is just a product name) to further help improve the awareness about Grafpad.

Now that we’ve covered the basic changes for the week, let’s talk about the quantitative data. I have to admit that it seems like a week is not long enough of a period for this. The metrics feel like they lag by a week when compared to my actions (the demo, for example, which I put up in the middle of last week, has hurt my sign-up rate for this week).

Acquisition: 7.15%
This week’s acquisition rate was significantly lower than last week’s. Out of 3000 people who could have seen the website on Web2py mailing list, 70 in the Front-End Meetup where I was doing a short talk on Pyjamas framework yesterday, and over 1400 friends on Facebook between me and my sister,  there were only 324 visitors to the site. I should also mention that Facebook seemed the least effective, with Web2py being the most, judging by daily numbers (I staggered the announcements to be able to measure their effectiveness). I suspect part of last week’s acquisition rate might have been the hype I built up about Grafpad during the months when the landing page was active but the web app was not yet available. I also think the number from last week might have been inflated since I was not able to get the member count for Pyjamas mailing list, but instead had to base my counts on the members who post on the list (as I mentioned last week, people who just read it and don’t actually post didn’t get counted).

Activation: 4.32%
As I mentioned earlier, after putting up the demo, the number of sign-ups dropped significantly. This week the demo has been up for the entire week, and I believe that’s one of the main reasons why the sign-up rate is lower. Nonetheless, it’s better to show the user right away what the app is about rather than forcing them to sign up. Even though only 14 people signed up this week, the number of people who tried the demo is an astonishing 140. I believe this could also be due to our audience. Many people coming from Pyjamas and Web2py mailing lists are curious to see how the technology they use is  used at another project, but that doesn’t mean that they want to use the product (this could also explain why the mailing lists are a lot more effective at getting people to come to the site than something like Facebook). It’s also possible that people don’t see a the advantage of having an account over the demo. I’m considering allowing free accounts online storage, but I feel that it would discourage people form getting a paid account (although it’s not like anyone has gotten a paid account anyway).

Retention: 35.71%
Five people who signed up this week have come back to use Grafpad more than once during the same week. I believe there are too few users to be trying to analyze this metric. Last week there was only 1 more person coming back than this week, so the fact that this number dropped does not necessarily mean that Grafpad is doing worse on retention.

Referral: 0%
It’s pretty disappointing that people are not mentioning Grafpad to their friends. It could imply that the app isn’t as great as I originally thought. It’s also possible that the sharing last week was actually a result of the landing page building up the hype for several months (similar to the acquisition metric). At this time I’m not sure what I can do to improve this metric, and I believe it’s more important to concentrate on the other metrics first, since they’re not doing too well either.

Revenue: 0%
Just as last week, no one has signed up for a paid account yet. As I mentioned before, I think I need to improve other areas of the funnel before concentrating on this one.

Plans for Next Week
Next week I will be trying other acquisition channels to test their effectiveness as well. I’m thinking of using Google AdWords, for example, or mentioning Grafpad in other forums/mailing lists (perhaps non-technical ones to test my hypothesis that people are visiting the site only because they want to see how we’re using Web2py/Pyjamas/Python).

I also want to improve activation. Perhaps I should give users more benefits for having a free account, or maybe I should set up a temporary promotion for users who sign up now. Maybe I should lay out the benefits of having an account on the front page (this might also help with getting users to sign up for a paid account).

I must admit that I’m a bit worried that my metrics actually got worse this week. But I’m not discouraged by this. This is an experiment after all, and not everything I do is guaranteed to improve my conversion funnel. The point is not to be right, but to realize when I’m not, and adjust accordingly the week after.

Lean Startup Challenge: Weekly Reflection 1

Now that the first week of the challenge is coming to an end, it’s time to look at the results and analyze what they mean (I’ll be doing this every week). Whereas my earlier post concentrated on the qualitative feedback, this one will concentrate on my observations of the quantitative data and define goals for next week.

Acquisition: 40.86%
This is the most bizarre metric. In this week I’ve received a total of 190 visitors to the website out of 465 that could have actually heard about it (by my calculations), since I only mentioned Grafpad so far through word of mouth (and as I found out, my sister helped too), twitter, blog, and Pyjamas mailing list. According to Google Analytics, there were 57 visitors to my blog during the same period, about double the average. Which leads me to believe that the 465 is also a bit inflated since I can’t tell if those  people came to my blog from the website or the other way around.

I will admit, however, that there are some inaccuracies in these numbers. For example, since Pyjamas mailing list does not reveal the member names in one nice list (correct me if this is wrong), I instead took all the emails from the duration I’ve been signed up for the list, downloaded them to my local machine through Thunderbird, and wrote a quick Python script to parse out the name of every person who has sent an email to pyjamasdev. It returned 150 names. Obviously the number is inaccurate, since there could be lurker members who do not send emails, but read other posts. There are also members signed up with multiple email accounts, and there are members who have posted throughout the year but have since left the mailing list.

Overall, 41% is a very high number, and not representative of what I expect to see when advertising through Google AdWords or a similar service. It’s quite possible that the number is only this high because other Pyjamas developers wanted to see what another member of Pyjamas community is up to. The number is probably also inflated because I’m underestimating the number of total people who heard of the website (it’s hard to see the true number of impressions when you’re not dealing with actual ads). I suspect this number will drop dramatically as I start advertising rather than showing it to communities that already know me.

Activation: 7.37%
This is an interesting metric. Two people signed up on day 1 (Tuesday), 8 people on day 2, and 4 people on day 3 (Thursday). Thursday was the first day that the demo was up, and in that one day 14 people tried the demo. I suspect that having the demo actually caused the number of sign ups to go down, since people no longer needed to register to play with Grafpad. On the other hand, the demo is a good indicator that there are a lot more interested people than just those who sign up for an account, most just don’t want to bother with registration.

I’m sure there will always be people who try the demo and leave, but the fact that so few actually register is in my opinion further reinforcement of the feedback I got earlier (that users are getting confused by the interface and leaving without registering). I will need to address this for the upcoming week.

Retention: 42.86%
The number of returning users so far is 6. I believe it’s too small of a sample to make any conclusion about retention rate. I’m not surprised that I got 6 people returning, the same people who went through the pain of registering are likely to come back at least once.

Referral: 1.58%
I was hoping this number would be much higher, if for no other reason than the novelty factor alone. Not every day you see a website intelligent enough to figure out what you’re drawing and help you out with it.  My goal is to get this number a lot higher. I think the same fixes that improve activation will improve referral as well. After all, if a user doesn’t like the product enough to register for it, he’s not likely to send it to friends either.

Revenue: 0%
No surprises here, I need to make the product addictive enough for the free users for anyone to even consider upgrading.

Goals for Next Week
For next week I will need to significantly improve the activation rate. While I do not yet plan to make big changes to the interface, I will need to work on a first-timer help module that will detect certain user actions and give advice regarding the areas of confusion from this week’s feedback. I will also need to add a feedback form to the main page, as well as clean up the front page so it looks more like a website and less like a landing page. I believe those two items will imrove both the activation, and the referral rate.

Reflecting on User Feedback (Lean Startup Challenge)

While Grafpad has been slow in terms of getting visitors (as most sites initially are), those visitors have been really great about feedback and even finding creative ways to give it to me (Twitter, Linked In, forums, Email). This made me realize that I’ve made it very hard for the users to provide feedback. It would probably make sense to add a feedback box directly to the main page.

Additionally, based on feedback itself it seems like the main page needs an overhaul. From the feedback I’ve gathered, it needs to be cleaner, more sexy, and less confusing. It also needs a “Try Now” button (and I knew I’d need to add one from the start, there is no excuse why it’s not there yet).

As far as the web app itself, I’ve gotten feedback that some of the things are not as intuitive as I thought. The menus, for example, will eventually need a complete overhaul. Due to their proximity, a couple people have mentioned that they don’t associate local bar with global bar. Since local bar is directly related to the currently selected option in global bar, that can be pretty annoying to the user. It probably makes sense to combine local and global bars into one.

Another area for improvement is right clicks. Currently Grafpad relies on right-clicks to rotate figures, modify figure and page properties, as well as perform undo/redo actions (it also supports keyboard shortcuts but from what I’m seeing not as many people use them as I thought). This seemed like a non-issue before a Mac user tried Grafpad (obviously, no right-click). To remedy that, the version I will be deploying in a bit allows Ctrl+click as well for all right-click operations.

Also, some of the keyboard shortcuts I implemented don’t even map to the same keys on a Mac. Delete key for example, is replaced with Backspace (they’re the same key on a Mac). One of the users also expected the color menu to apply to the figure when a figure is selected. That’s actually a pretty intuitive feature and I’m surprised it hasn’t occurred to me.

The good news is that some of these changes are minor tweaks that I was able to implement last night. For example, I moved undo/redo options to the global bar. Color menu now applies to selected figure (or document itself when no figure is selected in selection mode). While local bar and global bar are still two separate bars, I have stacked local bar on top of global bar to let the user know right away that they’re related. I must admit, it does seem to make a lot more sense now, the users were right. I also fixed several minor bugs I never caught myself but the new users were able to find within minutes of using the app. I will be deploying a new version of Grafpad with these changes shortly. If you haven’t yet, please give Grafpad a try and if you have, please keep playing with it as I improve the interface.

Also, I wanted to thank @leonartdesign and @lucchase for helping us spread the word about Grafpad by twitting about it.

Grafpad is now live! (Lean Startup Challenge Update)

Grafpad is now officially live. The .com address is still linking to the old landing page because I’m having some problems with NameCheap’s DNS servers (they’re saying it takes their server up to 72 hours to change to a different DNS), so as a temporary hack I’ve modified my Unbounce page to link to the main Grafpad page. Grafpad.net works correctly, if you want to use that link instead. You can sign up for a free account, and start using it now.

As part of the Lean Startup Challenge, I have setup various metrics using KISSmetrics to track the 5 Pirate Metrics I mentioned in my last post. For the most part it was pretty easy. The only part I struggled with was tracking YouTube views, so I will write another blog post about how I set that up later. For this week the goal is to start getting visitors to the site, so I’d appreciate it if you guys helped me spread the word.

To get a better understanding of what users want from Grafpad, I’ve also asked current alpha testers for their opinion on it. Only a couple have replied so far, so I’m not seeing a trend yet. I don’t really have much of a budget for ads, so I plan to work on my “acquisition” through the word of mouth.

As I mentioned earlier, I plan to primarily concentrate on activation and retention throughout the challenge. I’m treating this week as a control group for the experiment. The goal for this week is to get people to the site and see what my current conversation rates are, so that I have something to compare to in the upcoming weeks.

As I mentioned, the point of this week is to set up the conversion funnel correctly and get meaningful information. For that I’ve tied each Pirate Metric to a specific event at my website. Here is how I’m tracking each of the 5 metrics right now:

Acqusition: A visit to the main page

Activation: Sign up for a free account

Retention: Repeated login into the account

Referral: Sharing the website URL via the social buttons on the page (Twitter, Facebook, etc.)

Revenue: Upgrade to a paid account after playing with the free one

There are also some other metrics I use to judge user interest, such as users watching the YouTube video (as I mentioned earlier in the post). As I go through the competition, I might tweak the way I’m measuring these metrics as I make updates to the website.

Lean Startup Challenge Kickoff: Introducing Grafpad

For the last year or so I’ve been working on a hobby project of mine called Grafpad. Grafpad is a web app that runs completely in a browser and lets the user draw legible diagrams, wireframes, logos, and other vector graphics by hand. The app is essentially a piece of paper with brains, it uses image recognition to beautify the sketch as you’re drawing it. You can draw your concept as easily as you would on a white board, with the added benefit of not having to recreate your image in PowerPoint, InkScape or Visio when adding it into a presentation or a document.

Until recently, I tried to work on Grafpad and my startup called Pyjeon in stealth mode but learned that it’s one of the easiest ways for a startup to fail. Most startups today don’t fail because of technology limitations, but rather from lack of customers (or more specifically customer fit). Having learned about the Lean Startup Competition, I decided to use this 6-week challenge as an opportunity to turn Grafpad development on its head to see how my product can benefit from this.

For those unfamiliar with Lean Startup concept, it’s based on the observation that most learning about customer needs happens after the product is released. So if you shorten your release cycle, you can adjust your product much better to your target customers. Running a startup is essentially a race, where you try to achieve the right customer fit before running out of money and/or time.

The problem with typical projects is that it takes months (or years) to release the product. As a result, very little learning about the customer needs happens until the product becomes hard to modify. You can keep rolling out new versions or service packs, but if there is a fundamental problem with the idea itself, you’ll have to abandon ship eventually and pivot to a different concept. This can be a disaster for someone who spent years working on a concept that has no need

That’s where Lean Startup process comes in. The idea is to shrink down the amount of time needed for each cycle and allow you to iterate through your versions faster. As an added bonus, you can test how much the customers like each feature/tweak individually by comparing the product adoption rate between weeks when those features were added.

To measure that adoption rate, I will be using Dave McClure’s Pirate Metrics (they’re called so because they abbreviate to “AARRR”). Each week I will be running an experiment by tweaking Grafpad product itself or the website to see if it improves either of the 5 metrics. I will be updating my blog every Sunday with a summary of results from the previous week and talk about experiment I plan to run during the upcoming week.

During this competition, I plan to concentrate on activation and retention rate, but will be measuring all 5 of the metrics. As part of the competition, I will be releasing Grafpad app that I’ve been working on to the general public. I will track the metrics as follows:

Acquisition: Unique users visiting the grafpad.com.

Activation: Unique users playing with the Demo and/or registering for a free Grafpad account.

Retention: Users logging into their account more than once and/or creating an image they save using the account.

Referral: Users sharing grafpad.com with their friends using on-page twitter/facebook/etc. buttons or users mentioning another user’s name in the sign up form.

Revenue: Users upgrading their account to a paid version with more features.

I’d appreciate any help from readers, alpha testers, or random visitors during the competition in terms of feedback and/or helping me improve my metrics (although I obviously don’t want people singing up just for the sake of signing up if they don’t like the product). The product can only be as good as the feedback I get, so feel free to challenge my ideas. I’ll be updating you guys with the current status on this blog, as well as my personal twitter account (@ATsepkov) and my startup company’s twitter account (@Pyjeon). I will be tweeting under #LSChallenge and #leanstartup hashtags as part of the contest. The grafpad.com web app will be going live tomorrow, please check it out.

Strategy pattern in Pyjamas

I was recently trying to implement a Strategy pattern inside my Pyjamas web app. It’s basically a design pattern that allows the programmer to use multiple algorithms interchangeably without relying on inheritance, and it’s meant for cases when inheritance would add too much complexity.

For example, imagine that we’re implementing a class for a video game character, which has an attack() method. This method would probably vary depending on the weapon the character is holding. If the character is bare-handed he will probably punch the opponent. If the character is holding a sword, he’ll swing the sword at the opponent instead. If the character is holding a spear, he’ll perform a throwing motion. In this case, inheritance is clearly a poor choice, we’re not going to morph the character into a new object each time he switches a weapon. One other way to implement this is by using if/elif/else statements, but this solution isn’t much better, since we’ll end up with a behemoth method spanning several hundred lines of code as soon as we have a decent arsenal of weapons. We could also implement if/elif/else setup that calls other methods, but even then we’re adding unnecessary overhead. The cleanest solution is a Strategy pattern, which essentially maps the correct method at run-time.

Unfortunately, Pyjamas doesn’t seem to support either the types module or get() method for binding a function/method to a class method. Without it, Strategy pattern seems impossible to implement in Python.

For example, using types module, a clean way to implement this pattern in Python would be as follows:

import types

def strategyA(possible_self):

instance = OrigObject()
instance.strategy = types.MethodType(strategyA, instance)

This runs fine in Pyjamas Desktop, but throws an import error in the browser since types module does not exist in current Pyjamas implementation:

ImportError: No module named types, types in context None

There is an internalast fork of Pyjamas that does implement the types module, but does not implement the MethodType function within it. Essentially, the only difference would be the type of error you get.

There is an alternative way to implement the Strategy pattern. Every function/method in Python has descriptor methods that allow you to customize when you use an attribute. One of such methods is get() which allows you to rebind the method as follows:

instance.strategy = strategyA.__get__(self, OrigObject)

Unfortunately this solution also doesn’t work, throwing a type error in Pyjamas (since descriptors aren’t yet implemented in Pyjamas):

TypeError: Object function ... has no method '__get__'

The main problem is that Pyjamas will not let you bind an unbound method to an instance in Pyjamas for use as a bound method later. Which brings us to a somewhat hackerish (is that a word?) workaround. The idea is to define all alternative methods inside the class itself, and then swap between them, as if using variables. For example, if we have two alternative algorithms, strategyA and strategyDefault, we can switch back and forth between them as follows or each instance of the class independently:

self.strategyDefault, self.strategyA = self.strategyA, self.strategyDefault

If you need more than 2 methods, you can still use the same approach, but instead of swapping 2 methods, set the active one to point to one of the alternatives, keeping a backup of the original in another variable.

I must admit that even despite being the language where “there should only be one obvious way to do it” Python allows the programmer great flexibility in terms of implementation. And that flexibility is one of Pyjamas’ best features, in that even if Pyjamas drops the ball, Python tends to have plenty of firepower to pick up the slack.