Back in February, I posted about running a little experiment looking at pay-what-you-want (PWYW) verses free products. This is the promised follow up post talking about the numbers.
Let’s start with a quick recap of the plan for the experiment. If you just reread the original article, you can skip this part and jump right to the “Expected Outcomes” section. The experiment was going to be done with my new line of Two Sheet Locations.
The plan was as follows:
- Release two products, one sci-fi focused and the other fantasy focused, both of which would be PWYW with no minimum amount specified. In other words, it was completely acceptable to download them for free but you could pay for them if you wanted.
- Release two products, again one sci-fi and one fantasy, but this time completely free with no PWYW option.
- Look at the results and try to draw some conclusions
That was the plan, lets see what happened.
Before we dive into the actual results, let’s talk for a moment about what I expected the results to be. This was my thinking before the experiment started. I guess you could call this my hypothesis for the experiment.
- The fantasy products will get more downloads than the sci-fi ones. – Fantasy just has a bigger audience so I expected that these will be more popular. I had no idea on how much more popular as there are a number of factors that affect this.
- The completely free ones would generate more downloads than the PWYW items. – This was the driving force behind the experiment in the first place. I believe there is a psychological barrier imposed by the PWYW option. In addition, I had done a similar thing with the Frontier Explorer. Between issue 7 and 8, which both had about 1200 downloads in the first 45 days after release, I released a free PWYW product as a special edition designed as a fund-raiser for the magazine. It was completely free, but you could give us some money if you wanted to. In its first 45 days, there were a just over 300 downloads of that special edition (and only a fraction of those gave us money), even though it was still completely free if you didn’t want to pay for it. This was a factor of nearly 4 between the free and PWYW items. It wasn’t apples to apples as the special edition was a single topic compared to the broad topics of the magazine but it was still targeted at the same audience.
There would be a lot of variables going into this that would be out of my control or imposed by the release schedule but I figured we could see some trends even if we couldn’t get really hard numbers.
Problems with the Experimental Design
There are a number of things that biased the results and that could have been controlled slightly better if I had thought about it sooner.
The primary problem is that I’m building a community at the same time I’m running the experiment. I’m a relative (more likely a complete) nobody in the on-line game publishing world. No one really has any reason to be interested in anything I produce. So I’m building a community around these products as I release them. Basically all of my advertising will come from contacting people that have already purchased one of the items, a number that will grow with each release. Thus the “customer base” is growing with each release which I expect will skew the downloads in favor of the later locations.
In light of this, from an experimental point of view, I released all of the products in exactly the wrong order. Given the expected results above, and the bias expected from the growing customer base, I think the ideal release order would have been:
- Free Fantasy
- Free Sci-fi
- PWYW Fantasy
- PWYW scifi
Thus the PWYW barrier, if it exists would have resulted in lower numbers even with the larger customer base relative to the free products and the sci-fi release would have resulted in a dip relative to the fantasy one despite the growing customer base as well. But of course I did it in the exact opposite order so that the natural biases I expect are amplified by the growing customer base. Oh well, it wasn’t going to be completely rigorous anyway.
The other problem was that I couldn’t completely control the amount of advertising each product received. Well, I could have, and arranged for there to be none, but I was looking to grow the customer base. If the customer base had been tiny, the results wouldn’t really mean anything. So I had to try to get as much visibility for the products as possible and that could be variable.
But that’s enough about problems and expected outcomes, let’s look at some numbers.
The Results are in!
First of all, if you haven’t already, and want to grab a copy of the first four Two Sheet Locations, here’s a link the product category page on DriveThruRPG where you can get your copies.
Let’s start with the totals as these are quite interesting in and of themselves.
|TSL001 – Mr. M’s Equipment Emporium||PWYW-scifi||4||332|
|TSL002 – Blue Pearl Grotto||PWYW-fantasy||3||213|
|TSL003 – Gloria’s||free-scifi||2||386|
|TSL004 – Shrine of the Harvest Goddess||free-fantasy||1||420|
The total downloads count is since the product was released. You can see that the later ones are definitely generating more downloads than the earlier ones, even though they have been out for less time.
Interestingly, the Blue Pearl Grotto, a fantasy offering, has done worse than the the two sci-fi offerings (at least in total downloads, more on that later). However, this one is unique for a couple of reasons.
First, of the four products, this is the only one that didn’t get advertised in the weekly DTRPG newsletter. The other three all appeared in the “Newest Free Products” section at the bottom of the newsletter that list the three newest products when the newsletter was published. In the case of the Blue Pearl Grotto, I posted it on DTRPG a little too soon for it to make it into the newsletter. So this one lacked the larger advertising reach of the others.
Additionally, this was the first item to be promoted by me only to those who had bought other Two Sheet Location products, i.e. Mr. M’s Equipment Emporium. For the Mr. M’s release, I emailed everyone who had bought my book or my card game. That e-mail went out to 859 people. Maybe not the target demographic but a reasonable number nonetheless. Additionally I also ran an article about it in issue 15 of the Frontier Explorer which came out a week after Mr. M’s got released, that issue has had over 1000 downloads as well. As a contrast, the e-mail that went out with the release of the Blue Pearl Grotto only went to 170 people (of the 225 that had downloaded Mr. M’s at the time and allowed themselves to be contacted). So the overall advertising reach for the Blue Pearl Grotto was much smaller. And it still performed fairly well.
Number of Downloads by Day
Let’s look at the number of downloads of each product as a function of number of days since release. (click on the image for a full sized view)
As you can see, most of the downloads come right after the release date, up to about 30 days afterwards. The little spikes you see later on occur with the release of the later products. This always drives a small download boost to the earlier ones as new people find the product line for the first time and download all of them.
Numbers Downloaded in First Month
Since most downloads occur during the first month, with there just being a trickle of downloads after that, let’s look at the numbers for just the first four weeks of each product.
|TSL001 – Mr. M’s Equipment Emporium||221|
|TSL002 – Blue Pearl Grotto||124|
|TSL003 – Gloria’s||339|
|TSL004 – Shrine of the Harvest Goddess||407|
Again we see the huge drop for the Blue Pearl Grotto. Again, I think this is due to the much smaller reach of advertising for that item. The e-mails that went out about Gloria’s and the Shrine of the Harvest Goddess went out to 219 and 408 people respectively. I released Mr. M’s before I even conceived this experiment and so was pushing on as many channels as I could to get the word out. It dialed that back a bit for the Blue Pearl Grotto
The free products have definitely out-performed the PWYW products but not by the large margins I was expecting. Although with the uniqueness of the advertising for both of the first two products it’s hard to tell exactly the cause.
Interestingly the fantasy items haven’t outperformed the sci-fi ones as much as I expected either. This is probably due to a couple of reasons. First, the product line has received more advertising (in the form of the Frontier Explorer article) in sci-fi forums than in fantasy forums. So the customer base might be a little skewed. Second, and this is purely theoretical on my part, is that although the number of people interested in fantasy is much larger, there are also many, many more fantasy products and the market is saturated. So with so many products to chose from, they simply aren’t interested in another offering. On the other hand, there aren’t as many sci-fi products and thus new ones capture the attention of a greater percentage of those looking at that kind of product. The Shrine of the Harvest Goddess did out-perform Gloria’s but considering my targeted e-mail went out to nearly twice as many people for that item, a 20% increase in downloads isn’t that significant.
I think the free products definitely out-performed the PWYW items but not nearly as much as I was expecting. This is probably at least partially due to the order I released the items and the growing customer base for the products. I think if you’re going for maximum reach as quickly as possible, and not at all concerned about generating revenue, don’t put the PWYW on there. More people will download the products.
As to the fantasy vs sci-fi being more popular, the jury’s still out. There definitely wasn’t as much discrepancy as I expected here either. It might look different down the road looking at longer term trends but for now it’s a toss up.
Getting into the New Product list can be a big boon. However, there’s a lot of random chance involved here as you don’t know how many other people are pushing out new products and when exactly the newsletter is written. But if you can make it into that forum, you happen to get a much larger reach which can be beneficial if you’re new and trying to grow your customer base. People will at least come and take a look at it. The Blue Pearl Grotto didn’t make the cut and I believed the downloads suffered as a result.
Some of you are probably wondering how much money I made on the first two products that were PWYW. The answer is not a whole lot, but then, I wasn’t expecting to. These were supposed to be “free samples” after all. This is the one area where the Blue Pearl Grotto out-performed Mr. M’s. The Blue Pearl Grotto had a total of 12 paid downloads netting me a whopping $3.62 (average of ~$0.30 per download) after DTRPG took out their fees while Mr. M’s had 15 paid downloads bringing in $3.09 in total (average of ~$0.21 per download).
Way back when I conceived the Two Sheet Location project, I had always planned that the first few would be free to generate a customer base and then I’d start charging for the later ones. That plan is still in place and starting with location five (Per’s Gadgets and Computers), they will be selling for $0.99 each. The first four will always be “free samples” so people that are interested can see what they are like before downloading the ones with a price tag.
I’ll be changing the ones that were free (Gloria’s and the Shrine of the Harvest Goddess) to be PWYW products as well. However, I won’t be doing that immediately. I’ll probably wait a month or two before I make the swap so that I have several months of free after the initial release before changing the model. That way I can compare the impact of the two different models on the exact same product during the time that they are not new but just in the catalog.
I fully expect the $0.99 price tag to tank the number of download of future products. However, based on the amount of money raised by the PWYW model, it would only take 5-6 purchases to generate a larger revenue than I’ve already generated. It will be interesting to see what happens. I’ll probably do another post in 3-4 months to talk about those results.
Any thoughts, ideas, or suggestions? Have you had similar experiences in self publishing? Let us know in the comments below.