For redundancy, I’ve been running Google Analytics and KISSmetrics along-side Performable the entire time. One thing I noticed is that all 3 seem to be tracking visitors a bit differently, so I wouldn’t rely on just one of them alone. Performable, for example, has been over-reporting new visitors and under-reporting returning ones (there have also been blatant inconsistencies in the numbers it’s been reporting the last 2-3 days, although it’s been accurate before -it’s possible that it’s getting confused by my new website). KISSmetrics has been hiding the visitors who navigate the funnel in a different order than it expects. Google Analytics is delayed by 1 day and doesn’t seem to have the idea of funnels. For a more detailed comparison, check out my blog after the competition is over, I will be comparing all 3 in much greater detail.
But before I jump into the metrics I wanted to go over the changes/experiments from this week. Last week I realized that people were getting confused by Grafpad, both the website and the app itself. To remedy that I have made a number of changes to the website, as well as the app.
I’ve gotten some very useful qualitative feedback about the website last week. There were recurring patterns in the feedback, which is what I concentrated on addressing first. The original paragraph on the old website, for example, would tell the user that Grafpad is now live and he/she is free to try it. It doesn’t, however, tell the user what Grafpad is and why the visitor should care about it. In other words, it doesn’t pass the 5-second test (which is about how much time a typical visitor will spend on the site unless there is something to grab the attention).
The visitors also mentioned that the page felt a bit plain, and doesn’t feel like a website. So after browsing around the web for a bit, looking at other websites, such as http://yamlabs.com/, http://www.memrise.com/welcome/, or http://www.lucidchart.com/, I started noticing that many websites have 3 main bullets or options to grab the user’s attention on the main page. I decided to rearrange my bullets to do the same. The results of my tweaking can be seen below.
I’ve essentially picked the 3 key features of Grafpad people were getting confused about last week and moved them to the very top. The other features I’ve put in a slideshow container right below the main menu, figuring those features are secondary and many first time visitors might not even pay attention to them. I’ve also added a feedback form at the bottom of the page (not visible on the screenshot above) to let users mention their first impressions. So far, however, only 2 people have used the form, which leads me to wonder if I should move it closer to the top of the page.
As I mentioned earlier, I’ve been running Google Analytics on my site during the entire duration of the competition. It has a very nice feature, tracking the most popular search results that lead visitors to your site from Google. One thing I noticed last week is that the number one keyword was the IP address of my website. And some of you might remember that the domain name would change to the IP as the page would load (the issue occurred because we were using port 8000 instead of the regular port 80). My theory is that people were searching for the IP because they were assuming it’s a phishing site, and wanted to confirm that it’s not the case using Google.
Since then, I have moved my hosting to Amazon, and as you can see from the screenshot on the left, the IP address is no longer the most popular term. In fact, the number of searches for it has not increased at all since URL issue has been fixed.
The application also got a few minor enhancements based on user Feedback from last week. One of the first changes you’ll notice are the help bubbles that now pop up whenever you try to do some action for the first time that has confusing features.
I’ve gotten a lot of positive feedback about them, and I’m thinking of expanding them to cover help in more detail soon. At the same time I don’t want to swamp the user with help messages, because then he/she is not likely to read them either. Perhaps I should delay some help messages until the second time user plays with the application? I’m open to suggestions.
In addition to the help bubbles, I’ve modified success/fail messages to appear in similar bubbles for the sake of consistency. I’ve also done a bunch of minor bug fixes and interface changes since I got feedback that the login/sign-up prompts looked a bit plain. I definitely did not have as much time as I’d like to work on the app this week. Between addressing various usability issues that the users brought up and meeting with other entrepreneurs/groups this week to get a better understanding of what people actually expect/want from an application like Grafpad, I barely even had time to sleep this week.
This week we also setup a Facebook page for Pyjeon (the name of the company, since Grafpad is just a product name) to further help improve the awareness about Grafpad.
Now that we’ve covered the basic changes for the week, let’s talk about the quantitative data. I have to admit that it seems like a week is not long enough of a period for this. The metrics feel like they lag by a week when compared to my actions (the demo, for example, which I put up in the middle of last week, has hurt my sign-up rate for this week).
This week’s acquisition rate was significantly lower than last week’s. Out of 3000 people who could have seen the website on Web2py mailing list, 70 in the Front-End Meetup where I was doing a short talk on Pyjamas framework yesterday, and over 1400 friends on Facebook between me and my sister, there were only 324 visitors to the site. I should also mention that Facebook seemed the least effective, with Web2py being the most, judging by daily numbers (I staggered the announcements to be able to measure their effectiveness). I suspect part of last week’s acquisition rate might have been the hype I built up about Grafpad during the months when the landing page was active but the web app was not yet available. I also think the number from last week might have been inflated since I was not able to get the member count for Pyjamas mailing list, but instead had to base my counts on the members who post on the list (as I mentioned last week, people who just read it and don’t actually post didn’t get counted).
As I mentioned earlier, after putting up the demo, the number of sign-ups dropped significantly. This week the demo has been up for the entire week, and I believe that’s one of the main reasons why the sign-up rate is lower. Nonetheless, it’s better to show the user right away what the app is about rather than forcing them to sign up. Even though only 14 people signed up this week, the number of people who tried the demo is an astonishing 140. I believe this could also be due to our audience. Many people coming from Pyjamas and Web2py mailing lists are curious to see how the technology they use is used at another project, but that doesn’t mean that they want to use the product (this could also explain why the mailing lists are a lot more effective at getting people to come to the site than something like Facebook). It’s also possible that people don’t see a the advantage of having an account over the demo. I’m considering allowing free accounts online storage, but I feel that it would discourage people form getting a paid account (although it’s not like anyone has gotten a paid account anyway).
Five people who signed up this week have come back to use Grafpad more than once during the same week. I believe there are too few users to be trying to analyze this metric. Last week there was only 1 more person coming back than this week, so the fact that this number dropped does not necessarily mean that Grafpad is doing worse on retention.
It’s pretty disappointing that people are not mentioning Grafpad to their friends. It could imply that the app isn’t as great as I originally thought. It’s also possible that the sharing last week was actually a result of the landing page building up the hype for several months (similar to the acquisition metric). At this time I’m not sure what I can do to improve this metric, and I believe it’s more important to concentrate on the other metrics first, since they’re not doing too well either.
Just as last week, no one has signed up for a paid account yet. As I mentioned before, I think I need to improve other areas of the funnel before concentrating on this one.
Plans for Next Week
Next week I will be trying other acquisition channels to test their effectiveness as well. I’m thinking of using Google AdWords, for example, or mentioning Grafpad in other forums/mailing lists (perhaps non-technical ones to test my hypothesis that people are visiting the site only because they want to see how we’re using Web2py/Pyjamas/Python).
I also want to improve activation. Perhaps I should give users more benefits for having a free account, or maybe I should set up a temporary promotion for users who sign up now. Maybe I should lay out the benefits of having an account on the front page (this might also help with getting users to sign up for a paid account).
I must admit that I’m a bit worried that my metrics actually got worse this week. But I’m not discouraged by this. This is an experiment after all, and not everything I do is guaranteed to improve my conversion funnel. The point is not to be right, but to realize when I’m not, and adjust accordingly the week after.