Kickstarter Fulfillment and Product Development: A story of Dogfood and Data Validation

You could think of this post as telling the story of two Kickstarter projects. Since its a long post, here’s a quick summary:

  1. I recently ran a Kickstarter project.
  2. I wanted to share all the financials and details of how I shipped my rewards.
  3. I discovered we could do a better job helping creators process their backer’s addresses.
  4. We recently deployed a change to backer surveys that should do just that.

So I hope this post will educate Kickstarter creators on how to smoothly fulfill their rewards, but also shed a little light on how we do product development at Kickstarter.

The Kickstarter project was pretty simple — I FOUGHT SOPA AND ALL I GOT WAS THIS STUPID T-SHIRT— and the other project was actually a Kickstarter built by our product team, which we (and I use “we” loosely, Jed, Tieg, Daniella and Meaghan did all the work) shipped last week:

The address confirmation tool helps backers validate their addresses when filling out reward surveys from creators.

Just as Netflix or Amazon ask you to confirm your shipping address if it doesn’t precisely match an exact address, Kickstarter will now ask backers to confirm a more precise one so that creators can feel more confident about shipping their rewards off. This feature also has the benefit of cutting down some of the data correction creators might face when shipping rewards.

The development of this feature is a good example of the of “dogfooding” your own product, which is software jargon for the act of actually using the tools you’ve built.

Dogfooding is one of the best possible ways to understand and improve your product, so I’m always interested in getting feedback from people that have run their own Kickstarter projects.

The SOPA shirt project was pretty straightforward. It only really had two reward tiers, but as with all Kickstarter fulfillment, there were a lot of details to get right.

Getting one of those details right — backer addresses — made me realize we needed a better way to ensure we were delivering valid backer mailing addresses to project creators at the most crucial part of their project: reward fulfillment.

I also want this post to serve as a bit of a guide for fulfilling Kickstarter projects on a similar scale. So first, some details on how I planned to fulfill my rewards.

Estimating Income and Costs

To determine my actual net dollars from Kickstarter, Amazon, and the credit card fees, I had to download a CSV of my account activity from Amazon’s FPS platform, sum the pledges and subtract the credit card fees for each pledge.

The total cost of CC fees was 4.8%, Kickstarter’s fee was 5%, so I yielded 90.2% of actual pledges, or $3,497.

I had a chicken and egg problem trying to estimate which shirts to even offer in the survey.

Since different shirt sizes and different colors were different prices (the cost per shirt varies from $6.32 for a Grey Small, to $12.95 for a White XXL), it was crucial to a rough estimate of what the shirt size and color choice distribution would be.

I knew a lot of people were going to order large and medium shirts, but if I had too many XXLs, it might not have been affordable to offer the white shirts. And if it turned out that I couldn’t afford the white shirts, I didn’t want to even offer the white shirts in the survey.

So I took a look at Yancey’s shirt distribution. For his project he had the following distribution of sizes:

  • S: 13%
  • M: 26%
  • L: 30%
  • XL: 21%
  • XXL: 10%

These ratios enabled me to estimate what the distribution of sizes would be for my shirts. I guessed that 2/3rds of my backers would want the grey shirt, and the rest would want the white shirt.

Using these numbers and the bulk costs from ApparelSource.com, I was able to determine that I could afford to offer both the grey and white shirts in my survey.

Here’s what the actual distribution of sizes turned out to be:

  • S 13.85%
  • M 29.44%
  • L 32.90%
  • XL 15.58%
  • XXL 4.33%

My numbers weren’t far off from Yancey’s — I ended up having fewer XXL than me which surprised me. The grey / white distribution ended up being 77% grey, 23% white.

I threw together a quick graph using ggplot2:

One observation is that the distribution of white shirts looks like it skews smaller. Anecdotally, that might be explained by women preferring the white shirt (again, anecdotal), and since women’s sizes tend to run smaller, the distribution of white shirts had proportionally more smaller sizes.

Buying the Shirts

ApparelSource seems to operate multiple websites with basically the same layout and offerings. They had the best prices on CANVAS shirts I could find, though oddly, the prices varied between their different sites by a couple of cents. Some sites had them in stock and others didn’t. This made no sense to me because they’re all being shipped from the same warehouse.

I ordered one of each shirt style to confirm they were the style I wanted, and then prepared to make the bulk order.

My order was extremely specific (13 Small White, etc.) so I think it raised some flags on ApparelSource’s side. I had to spend some time working out the best way to pay them. It turned out paying them via PayPal was the easiest way.

This is a version of the sheet I used to calculate costs. I think working with something like this sheet is crucial for staying sane when fulfilling any Kickstarter project.

Printing

Yancey had also suggested I check out Kayrock so I dropped them a line to get a quote. Kayrock had a source they recommended for buying the actual shirts which would have saved me over $1,000, but I wanted to stick with my choice of CANVAS shirts from ApparelSource because I had already tested their quality.

In the original email Kayrock had quoted the print job as “plastisol ink, no shirt rolling, no double hit, no handling fee, no rush fee, no production prepress, no color change fee“, and gave me the quote of $479 + $80 for printing and setup.

What they didn’t tell me, was that if I didn’t use their apparel quote and instead had the shirts shipped to them directly, they were going to charge me $0.40 cents per shirt of handling. This added $100 to the total cost, so my Kayrock fees were $660.

Despite this issue, their work and responsiveness over e-mail was very good, and I would recommend checking them out if you’re interested in some great NY screen printing.

Once Kayrock finished printing the shirts that I had shipped directly to them, I headed to Greenpoint to pick them up myself so I could start packing and sending t-shirts from Kickstarter HQ.

Backer Addresses and Postage

In general, backer addresses tended to be malformed in a couple of ways. The first was that people tended confuse “Line 1” and “Line 2” — they either put their Apt # first, then street, or their Street then company.

I also ran into problems with 20+ addresses from the North East because their zip-codes were had their leading zeros removed.

Zip codes like 06897 became 6897 when opened in Google Docs or Excel because they were converted to numbers.

In addition someone put “Meow, Meow” as their address. They didn’t get their shirt.

To save on shipping, I did my research and chose First Class mail, which let me specify the ounces for each package. After weighing each style of shirt in an envelope (I correctly assumed the labels weren’t going to add any more weight), I had three tiers:

  • S / M white – 5oz – $1.98 postage
  • M / L shirts – 6oz – $2.15 postage
  • XL, XXL shirts – 7oz – $2.31 postage

Sorting by whose shirts had already been delivered (all KSR staff, some friends and family), and removing those people from my CSV yielded 183 packages, summing to $370. I’d recommend keeping careful track of the rewards you ship vs. the rewards you intend to hand deliver.

The cost to ship the same shirts using flat rate envelopes would have been $942.45, so I managed to save a lot by carefully weighing and picking First Class mail instead.

After having lunch with my mom on a Saturday, she offered to help me stuff envelopes and ship the shirts. This was actually a lot of fun and I always forget how rewarding manual labor can be when the majority of my day to day work involves keyboards and LCD screens.

My mom suggested to come up with a color coded system to manage shirt sizes & colors (envelopes marked with Blue represented a Large White shirt, for example). This worked well, but only because I had access to many different colored Sharpies.

Fulfillment would have been chaos without some kind of organized system for tracking sizes and colors.

Tyvek envelopes are great, until you realize they’re a little pricey; Staples sold 50 for $28, so $0.56 per envelope which added to the shipping cost. I probably could have ordered them for free from USPS if I had planned the shipping session in advance. In general I highly recommend using Tyvek envelopes as they’re much lighter, waterproof, and probably more durable than the cardboard flat-rate envelopes.

Endicia Software

I’m not sure how I would have addressed and processed all the envelopes without Endicia which is batch mailing software. I get the feeling many other creators have used it to fulfill their rewards. Endicia is free to use for 30 days, and then $15 a month to keep using it afterwards.

The software allows you to import a CSV, but the requirements are very strict. The fields must be (in the following order):

  • Name
  • Company
  • Address1
  • Address2
  • City
  • State
  • Zip

In order to correlate each person with a label and envelope with a t-shirt, I merged their size and color selection into the name field. So it looked like this for a XXL Grey shirt:

John Smith XXL/G
3105 Sturges Ridge
Santa Rosa, California 95401

This might have been a little odd for backers (are there privacy implications for exposing their shirt size to the post master?), but Endicia didn’t allow any other fields, so there was no where else to put this information on the label.

I bought $500 worth of postage, but only ended up using $370, and was able to get a refund for the left over balance when I canceled my Endicia account.

Printing Labels

Printing labels from Endicia is very scary. You are shown this dialog box (but replace 7 with 180, worth $370):

I did one test print of a label and it seemed OK, so I went with through with the big batch. After that Endicia automatically opened a file inside my /tmp/private directory with a number of rejected addresses, most of which had the ZIP code problem. This was a time consuming and stressful process because any mistake would mean lost money on postage.

After I was done printing the postage from Endicia, I realized I could have sent the batch to a PDF instead of the actual printer, and it would have mitigated some of the risk. Without storing my labels somewhere, if my printer had gone off line or my computer would have crashed, it was unclear how I would recover labels that were still stuck in the queue. I hated trusting the printer software this much, but the overall process went pretty smoothly.

Internal Kickstarter Post-Mortem

Once I shipped my shirts, I wrote up an internal email to Kickstarter staff detailing my fulfillment process. That email became the kernel for a number of discussions about how we could better improve the data processing creators face when delivering their rewards to backers.

One solution to the dirty-address problem was to use an external service to validate the addresses supplied by backers.

Using an external service to validate backer addresses turned out to solve two problems.

First, it would ensure backers had the chance to confirm a valid address, which is always a good thing. Second, it would add a hyphen and the 4-digit add-on to US zip-codes, converting zip-codes like 11217 to strings such as 11217-1142.

That extra hyphen would prevent applications like Google Docs and Excel from converting zip-codes like 06897 into numbers like 6897 (technically, the application would recognize the cell as containing a string so it wouldn’t attempt integer coercion).

Preventing zip-codes from losing their preceding zero would mean software like Endicia wouldn’t reject addresses from the North East. So we decided to give it a shot.

First, Jed did research on what external services we might like to use. Strike Iron emerged as one of the best options, so we began sending legacy addresses to their API in order to evaluate how many they’d be able to validate and correct. This back testing showed that Strike Iron would be able to supply valid addresses for the vast majority of the sample of the surveys we tried. It also showed that Strike Iron would even be able to suggest corrections for many otherwise undeliverable addresses.

Since then, its been really exciting to watch Jed and Tieg build the actual data flow, and I’m super proud of how it turned out.

So even though this is a somewhat behind-the-scenes change in the way Kickstarter processes data, its exciting to know that it will make the lives of many creators just a little bit easier.

14 comments
  1. While that’s awesome for the small scale and no infrastructure, I work for Fangamer and shipping out the Double Fine Adventure shirts+posters right now. Needless to say, it requires a ton more tech to get those out the door. Think I should do a blog post on that?

    Quick question: How do you do the address check? What’s your data source?

  2. This is information should be mandatory reading for Kickstarters. As a former Ebay power seller and past COO of Wally Famous Amos’ latest cookie company in Hawaii which shipped fresh-baked cookies, I often look at KS projects and wonder how much thought the project creators have put into fulfillment. Nice job!

  3. Intense. I had a similar experience where I tried to export email addresses to a spreadsheet and many of the addresses were missing letters. Makes a quick data dump turn into a long manual process. I wouldn’t want to deal with physical inventory too!

  4. @Ryan Alyea, Did you read the post? He named the address correction source as a company called “Strike Iron.”

  5. I have a small product company that ships to about 14 countries. I’ve worked with a few people doing Kickstarter things and I try and tell them that making, printing, and mailing ‘freebies’ is super expensive and time consuming.

    Just glancing at your gDoc it look like you took in $3600 and only profited $468? Ouch.

    Printing labels and postage certainly sucks. Once you automate as much as you can, as it seems you did, the rest is trivial.

    Excellent read!

  6. As a Kickstarter backer, I find the whole survey process to be very annoying. My biggest gripe is surveys that get sent right after a campaign is over but where the reward isn’t available for months later. I’ve moved twice in the last year, and had to update projects several times with a new address. In several cases, I never heard back, so I’ve no way of knowing if they really took the new data. Having some way to just tell Kickstarter that “give the project my current address” and only update it once on Kickstarter would be most excellent.

  7. Phenomenal analysis here, and it’s great to hear how this major chapter of the Kickstarter experience is evolving. Having just rafted through the white-knuckle rapids of rewards fulfillment, I think you’ve addressed all my difficulties and made it a whole lot easier for project leaders to keep their promises. Very swell.

  8. @Jason: I wasn’t intending on making a profit on the shirts — I could have adjusted their price ($15 is pretty low for a nice t-shirt, IMHO), and probably doubled it, but wanted the project to be a labor of love. Plus I have 10-20 surplus shirts :)

    @Ben: thanks for the feedback, I’ll be sure to pass it along.

  9. […] Kickstarter Fulfillment and Product Development: A story of Dogfood and Data Validation is a great post by Fred Benenson showing how they test the tools they build. It’s also just interesting to see what can be required to handle Kickstarter reward shipping […]

  10. later. I’ve moved twice in the last year, and had to update projects several times with a new address. In several cases, I never heard back, so I’ve no way of knowing if they really took the new data. Having some way to just tell Kickstarter that “give the project my current address” and only update it once on Kickstarter would be most excellen

  11. What an amazing detailed account of fulfillment. Thanks so much for posting this.

  12. […] Kickstarter project of my own, and in the process I came across Fred Benenson’s blog post “Kickstarter Fulfillment and Product Development: A Story of Dogfood and Data Validation.” This past summer, Benenson created a project to make T-shirts satirizing the SOPA laws being […]

  13. […] Dave Ghallagher notes that Kickstarter fixed the bad data problem by implementing an address validation system back in 2012, too late for the MaKey MaKey project, but in place for existing projects. […]

  14. Fred,
    Thank you for posting such detailed info about the shipping process!! Even though we are right in the middle of our own Kickstarter campaign it was nice to hear about shipping. I don’t know if you have changed the protocols after a successful campaign but I would consider your post mandatory reading.

    Cheers,
    Arron
    TRU Margarita

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.