Preface

For the past few years, a friend of mine has been hosting a game which required distributing many large files to their players.

These files in total would range between 20 to 25GB depending on the patch sizes at the time of release. Everytime a new patch gets distributed several larger files in the range of up to 5GB may need a redownload depending on the files changed.

The frequency of these patches, as well as newer players discovering the game, meant that in total, over the course of 6 months, around $1,800/month was racked up in S3 fees with an average of 20TB each month downloaded by players.

But this was odd. The game didn’t have that many players, up to 100 at peak times. It soon became evident that the high churn rate of players meant that newer players were coming in, while older ones were leaving along with their GBs. Typical for any industry, but a big hit when it came to the S3 bill.

Enter Backblaze 🔗

A high level of the architecture on how the files are served shows that, so long as we're able to swap out the S3 bucket for another compatible S3-storage provider, we're good to go.

The main technical requirement was that the service had to be S3 compatible, as the way in which the files were distributed used s3 sync under the hood.

After a bit of research, Backblaze’s B2 service was offering significant storage cost reductions with bandwidth at almost 90% of AWS’ on paper. What’s more, is that it claims the 11x 9’s gold standard that S3 does too for annual durability.

From the costs, though the reduction in storage fees is nice, the biggest eye catcher is the $0.09 to $0.01 per GB reduction, ignoring the 3x storage freebie that gets thrown in.

The only downside was that B2 isn’t a Content Delivery Network in itself. This means that files requested directly aren’t served at the edge for users, though this wasn’t a deal breaker, it just meant users would have less than optimal retrieval times for files.

I suppose this was one of the ways they were able to keep costs down by keeping it to a couple of data centres.

The migration itself was relatively straightforward, it effectively non existent because players would always be served the latest game updates. All that was needed was to ensure the next update applied was pointed to Backblaze instead of S3.

The cost of bandwidth was still high at around $200/month for 20TB downloaded, but it was a fraction of the cost compared to AWS.

After the rollout, a couple of the users that were located in Asia and Europe took a small hit to their download speeds- this was expected given the location of the bucket was in Virginia (US East). Looking back, initially expanding into the other data centers Backblaze offered would have helped counter this, or a CDN integration.

Overall, the gains were significant. 88% of the bandwidth costs were saved when compared with S3.

CloudFlare's R2 Service 🔗

A year after the switchover to Backblaze had passed, and in that time CloudFlare had released their R2 service to general availability in late 2022.

CloudFlare’s R2 service claims an 11x 9’s on annual durability, on par with Backblaze’s B2 and AWS’ S3.

The biggest differentiator is that egress fees are non-existent in R2. I had to re-read this twice but it’s true. There are no egress fees.

There’s usually a catch with these things, and the catch here is that you’ll provided with a *.r2.dev domain, which are rate limited- a non starter for anything other than development.

However, all you need to sign up to a custom CloudFlare domain for production use and attach it to your R2 bucket. A perfectly acceptable tradeoff given the generous no egress fee policy.

It does make me wonder how they were able to get away with this. CloudFlare being the goliath they are it’s not unimaginable that they were able to soak all this up, given their global presence and edge locations.

Following the same migration strategy going from AWS to B2, all future game updates were forwarded to R2 and saw the immediate cost reductions from day 1. It was incredible.

The R2 Super Slurper was released at that time, and although it wasn’t needed, it is a viable option to keep AWS egress fees to a minimal while making that shift to R2 for more large scale applications: https://blog.cloudflare.com/cloudflare-r2-super-slurper/.

The Final Bill (Conclusion) 🔗

The R2 bill eventually came in for that month. $10 was paid for storage fees, and as expected, $0 for egress fees on the 20TB of bandwidth that was used. Plus the cost of the domain registration.

A huge victory. When looking back, it’s rather ludicrous that AWS have been able to maintain their S3 egress pricing when competitors are offering it for free.

However, costs alone aren’t usually enough to force an organisation down a different storage provider- S3 has a lot of great features such as S3 Batch Operations, as well as AWS specific native integrations (particularly around event driven architectures), but is it a matter of time before competitors catch up?