Lean Startup Challenge: Weekly Reflection 4

Last week, some members of the Lean Startup Challenge (including me) attended an AdWords workshop by Google. We were told some basics about how to use the ads, that it helps to group the keywords into categories with other similar words, and that a good click-through-rate to aim for is 1%. This week I started using Google AdWords. I’ve also managed to learn a few important lessons about the AdWords system that weren’t covered during the workshop.

For example, the ad “quality” for a given keyword (which is actually more like a keyphrase) seems to depend on how many words from the keyword appear in the ad itself. At first I didn’t realize that, thinking I’d bundle all diagram-related keywords under my diagram add, figuring that the people seeing the ad would understand that UML is a type of diagram. Unfortunately, Google penalized my quality score heavily for not having “UML” in the ad. I now have 3 different ads with 51 enabled keywords (a good rule of thumb is to have about 15 keywords per group and a unique ad for each group) and seem to be doing better in terms of my quality score. For those unfamiliar with Google Ads, quality score factors into the ads position on the page. Bid amount is the other component that factors into the ad position. So if you want your ad to appear at the top (where it’s much more likely to get noticed), you can either improve the ad quality or throw a lot of money at it. However, if the quality is too low for a given keyword, no amount of money will force the ad to show up (McDonald’s for example, will never show up when you’re looking for a new laptop).

While the quality score depends on your ad alone, the position of the ad also depends on quality of the competition (as well as the size of their pockets). As a result, I started sometimes ignoring the quality score if my rank is in the top 3 and I am getting clicks from the keyword. Other times my quality score is high (7/10) but I get no clicks and Google throws me in position 11. In those cases it’s time to disable the keyword before the quality score drops. Playing with AdWords is a lot like tweaking the skill points of a character in RPG, you don’t control the damage output directly but it’s a combination of dexterity, strength and a number of other factors that you might not be able to control. The trick is to test out your build in a real environment and analyze when you’re seeing diminishing returns.

There are also some important gotchas I stumbled into that hurt my click-through-rate quite a bit. First of all, don’t use generic single-word keywords, no matter how high your rank/quality is for them (there are probably exceptions, just that this didn’t work for me). For example, somehow I got a quality score of 6/10 for the term “graphics” and I also had position 3 for it (despite it seemingly being such a popular search term). I was very happy about this and let that keyword run for the first day. When I checked my campaign the next day, this keyword got over 8,500 impressions, but only 20 clicks (about a 5th of the 1% Google recommends). My other keywords were barely shown at all since this one was so much more popular in search terms. Out of those 20 visits, I’ve received no sign ups. Needless to say, that was the first keyword to get disabled.

The second gotcha I ran into is display network. These are the ads that appear on other people’s blogs, websites, and news stories. Google recommends leaving them on for inexperienced AdWords users (so that your ad can appear in those places as well), and I did just that in the beginning. I was told at the AdWords workshop that the click-through rate for display network is lower, somewhere around 0.3%, so I was expecting less clicks there. I figured that whatever clicks I get there would just be a bonus. So the next day (after disabling “graphics” and other generic terms) I noticed that the number of impressions for my other terms on Google has not increased too much, but I managed to get about 9000 impressions on display network, with only 1 click-through. That’s when I decided that display network needs to go before it hurts my quality score on Google as well. My guess is that the click-through rate is this low because people are actively engaged in the website, whereas with Google they’re still searching for something to engage in. It’s a lot easier for the ad to be more interesting than other searches than the article that the person already clicked on.

After addressing those issues, my campaign started doing a lot better. I get less impressions now, but my click-through rate is way higher, not to mention that visitors actually sign up to use Grafpad. I still check my AdWords account every day, disabling words for which my position is low and tweaking my ad wording if I see a trend of all keywords getting a low quality vallue compared to my other ads.

App and Website
There haven’t been major changes to Grafpad this week, I have been slowly fixing bugs and other annoyances however. I also observed several first-time users play with Grafpad, which was even more revealing than reading the feedback. I noticed, for example, that several users got confused with how right-click works. They expected that right-clicking outside of right-click menu while it’s active would close the menu instead of opening it in a new place. Perhaps I need to replace the right-click menu with “figure” menu that becomes active when a figure is selected (the menu can either appear inside the selection box or above the two other menus).

Acquisition: 0.17%
There were 34 visitors to my website this week out of 19,878 who saw the ad impression. As I mentioned earlier, it was the use of generic terms and display network that hurt me the most (costing me about 17,000 impressions with virtually no clicks). I believe next week I will be getting much closer to the 1% click-through Google recommends. Already I’m seeing 2-3% click-through for some search terms I’m using.

Activation: 14.71%
Since I haven’t done anything this week to improve activation, I would say that last week’s experiment is confirmed. Giving people free storage and comparing free account to paid one on main page does improve activation rate.

Retention: 40%
Out of 5 people that signed up this week, 2 used their account more than once within the same week. I still don’t want to draw any conclusions since the percentages are marginally different between the weeks and the number of people is in the single digits.

Referral: 0%
No referrals again, although I do seem to be getting more and more people following me on twitter. I’m not sure whether it’s my blog, Grafpad, or simply common interests.

Revenue: 0%

Plans for Next Week
I started trying to add touch-gesture support to Grafpad last week, to make Grafpad usable with iPad and Android devices. So far I’ve been unsuccessful (I tried to add it cleanly via Pyjamas, the same way mouse and keyboard handlers work). Next week I might try adding it via a hack, a Javascript hook that triggers Pyjamas mouse handlers. I’ve been getting a lot of feedback about making my app iPad-compliant. If I can at least add partial support for that, I can gauge the popularity of that idea. I do believe iPad market could be its own niche, and I want to try concentrating on it since many people don’t seem to like drawing with a mouse.

Lean Startup Challenge: Weekly Reflection 3

At the end of last week, Lean Startup Challenge started eliminating teams. Luckily, Grafpad is still in the challenge, but things are getting harder. I’m excited to see that there is a group of users that keeps coming back to use Grafpad repeatedly, but this group is currently pretty small (less than 20 people). There has also been a number of suggestions for Grafpad improvements. Two of the most popular ones are ability to collaborate on the same file online and being able to use Grafpad on a tablet PC. I want to assure the readers that there are definitely plans for collaboration system and touch screen functionality is already being worked on, but both will take some time to implement.

Second, I wanted to talk about the changes made to Grafpad during the current week. The biggest change some of you might have noticed is the new interface, or rather new icon theme.

I have been getting some complaints that the old icons were tiny and hard to see. I think the new ones should be a lot more descriptive. I am, however, considering adding ability in the settings to switch the icons to smaller size for those with less real estate on the screen (netbooks, for example). The second change is the color scheme, it seems like orange and black works a lot better than pink and purple. Now that the main page has already been changed to the new color scheme, modifying the app itself to be consistent was a no-brainer.

The second change many of you might be aware of is that I decided to let free users store one SVG file per account on our server. I’m still trying to decide whether this will be a temporary, promotional service during the alpha or if I will allow those who sign up later to use it as well.

I want free users to enjoy the app, but this is a business I’m trying to run after-all, and it doesn’t look like ads alone can pay the bills yet. Speaking of which, as part of this week’s changes, ads now do appear in the app as well.

Another change, that many users will probably appreciate, is that I finally managed to fix zoom on Safari. While at it, I added some other minor enhancements to zoom as well, one of them being a text box that allows the user to zoom in/out by typing in zoom percentage.

This week’s metrics were trickier to gather, there were a lot of inconsistencies and I had to rely on internal web2py hooks to obtain accurate data rather than KISSmetrics or Performable. Maybe the guys running those services didn’t like what I said about them last week and decided to take revenge (I’m joking). Most-likely, it’s actually flaws in the way I’ve been measuring the data that are now getting exposed as I get more return visitors.

Acquisition: 0.58%
There were 111 new visitors to the site this week. However, this week the main source of visitors was from an announcement on Linked In Python Community group. The group has 19,121 members, but the chances of a member seeing the announcement are pretty slim (they don’t get shown to the members unless they happen to be visiting the group during the 2-day period that it stays on the main page). Many people are members of 30+ groups and visit Linked In less than once a week, let alone one of the groups there. So I definitely would not trust that all 20k members saw the announcement (probably less than a quarter of them actually saw it), but this is the only data I have, so I will have to go with it.

Activation: 9.91%
There were 11 new registrations this week. In a proper statistical experiment, I wouldn’t trust a data set this small. Unfortunately, it’s the only data I have to go on now, and it’s double the percentage from last week. I will leave this experiment running for another week before making a final decision, but it does look like changing the look and giving users free storage increased the rate of new sign ups.

Retention: 36.36%
Out of 11 people who signed up, 4 of them logged in more than once. Last week, 5/14 people signed in more than once. Just as last week, I don’t think this data is statistically significant. I must admit, however, that people who signed in before, seem to be returning more now, there were 38 return logins this week.

Referral: 0%
It still seems like no one is spreading the word about Grafpad yet, it’s possible that some people are, just not through the social media links.

Revenue: 0%
Looks like Ramen noodles for me this week again.

Plans for Next Week
Next week I plan to try using Google AdWords for acquisition instead. It’s becoming clear that my current methods alone don’t seem to be working anymore. I’m also thinking of removing the extra steps from sign up process to encourage better retention rate. For example, instead of showing users the EULA and asking them to finalize the registration through their email, it might make sense to just give them a link to the EULA and allow them the first login without having to do email confirmation. I’m also not getting as much new feedback about the app anymore, so I might need to start actively asking other engineers, students, and designers (the target audience) about the kind of functionality they’re looking for (either through forums or meetup groups).