Spotify International Pricing Index

The Big Mac Index for Spotify Premium pricing

See the project here: Spotify International Pricing Index

Just after moving to Malaysia it shocked me how big the price difference was between Spotify Premium in Denmark and Malaysia.

One of the main reasons for the large price difference can be found in the particular catalog size in each country. I scoured the web looking for a way to get the total number of tracks available for each country, unfortunately Spotify doesn’t seem to make this data publicly available.1

I’ve always wanted to do a simple project with data visualization with D3.js, so this a perfect opportunity to play around with that.

I wanted to replicate the charts from Big Mac Index, but with simpler interaction. The goal was basically a simple Javascript-generated infographic. I tried to match the styling of the Spotify OS X and iOS app as best as possible with similar colors and fonts.

Getting the pricing data

I wrote a simple little script in Node.js that crawls all the individual country pages for the pricing info. It then adds a little more relevant country info to each country from restcountries.eu and lastly converts the local price to USD with the help of money.js. Check it out on GitHub: spotify-crawler

Luckily Spotify keeps a nice and simple list of all their available countries. They also don’t force a location based on IP which would make crawling much harder without the use of proxies in different countries.

I exposed a simple API with all the countries on Heroku if you want to look at the data: http://spotify-pricing.herokuapp.com/api/

World map choropleth

I started out creating a map from the ground up, but quickly discovered datamaps.js.2.

The datamaps.js library includes features like hover, popup, legend, etc. – all features which I would have had to reproduce if I followed the tutorial.

I hit some limits in the use of the library and had to make some edits to the main source file - I’ll try contributing the changes back to the source.

spotify world map

I associated the different fill colors based on which price-difference quantile each country fell into – still not convinced that this is the most correct way of doing it but the result looks reasonable. Getting a specific quantile with D3.js is as simple as:

var sortedDifferences = _.map(_.sortBy(countries, 'priceDifference'), 'priceDifference');

var quantile = [
  d3.quantile(sortedDifferences, 0.2),
  d3.quantile(sortedDifferences, 0.4),
  d3.quantile(sortedDifferences, 0.6),
  d3.quantile(sortedDifferences, 0.8)
];

Assigning a fillKey/fillColor can then be done by comparing each priceDifference and observe which quantile it falls into.

The fill colors themselves are calculated with the small Javascript library tinycolor.js which provides powerful color methods like .lighten and .darken – you may have seen similar methods in Sass.

Bar chart with negative values

The original chart from The Economist that I was trying to reproduce: negative bar chart

This was the first chart I started working on, since it was my first time working with D3.js it was also the chart that took the longest to replicate.

I followed this brilliant example from Mike Bostock showing how to build a simple bar chart with negative values.3

I added some additional text elements and labels, but the end result is pretty close to the example.

Scatter plot

I originally didn’t plan on reproducing this chart, but after making the first two charts I could just create this one as well - a fun little challenge. I started off with yet another great example from Mike Bostock.

GDP data varies widely between different sources, be it IMF, CIA or World Bank. I ended up using the GDP per capita, PPP (current international $) data from World Bank, and their API was fairly easy to use. All data are from 2012 because it was the latest year without estimated numbers in the World Bank API.

5 countries still returned null from the World Bank API: Andorra, Argentina, Liechtenstein, Monaco and Taiwan. These countries are not shown in the scatter plot chart. The CIA list of GDP (PPP) per capita does contain these countries, but the reported numbers are from different years.

This whole project gave me an opportunity to dust off some of the math and basic statistics I had learned years ago – and never really used since. I was about to create my own trend line calculation when I found yet another example from Mike Bostock showing how to calculate a trendline.

Wrap up

I’m currently paying for the most expensive Spotify Premium plan in the world when converted to USD.

So will I be making the switch? Probably not, I still want to listen to Danish tracks which aren’t available in the cheaper countries like Philippines and Malaysia.

Check out the GitHub repo for the project here: spotify-pricing

Further additions that I want to implement are recording price changes over time, and automatically adding new countries as Spotify expands further internationally.


Update: Corrected small typo spotted by @nikreiman, thanks!

  1. If you know any way to get this data, please let me know.

  2. There is a great tutorial on how to create maps with D3.js here: http://bost.ocks.org/mike/map/

  3. Mike Bostock is the creator of D3.js

Reverse-Engineering Sonos API to Create Own iOS and OS X app  ∞ 

Nathan Borror wrote a great post about reverse-engineering the Sonos API in order to create a custom iOS application, OS X client and turning off speakers when he leaves home.

Nathan also documented the design- and code-wise progress on GitHub.

Sonos custom app

I enjoy posts like these where somebody is scratching an itch by reverse-engineering an already existing product. Wish more of the posts on HN were like this one.

SummaList RSS feeds OPML file

SummaList promises to “summarize all the best tech & startup newsletters.” sending you one weekly e-mail with summaries of the best newsletters.

I don’t want to receive any more emails, and generally just like to have all my feeds/newsletters combined in my RSS reader1 - so I crawled the RSS feeds from all the sites currently on SummaList.

For a similar service that offers summary of interesting content do check out Blinkist.

Grab the OPML file with RSS feeds of all2 the newsletters from SummaList below.

  1. Just switched to the new beta of Reeder 2 for Mac, and syncing with Feedbin

  2. Sites without RSS feed: Corbett Barr

Sass Unit Testing

We all know that testing is good, and most of us hopefully already write tests for our front-end JavaScript code – but is it needed for our Sass code as well?

We need to think about long-term maintenance as our Sass projects and libraries gets more and more powerful. Just look at some of the things Sass can do today, things nobody would’ve imagined just a few years ago:

The term Sass unit testing is not referring to visual regression testing of the visual end-result, but instead logical unit tests focused on the core functions of a library. The term is also not referring to linting of sassor scss, if that’s what you’re looking for scss-lint may be for you.

The logical testing of functions that an entire project relies on, just makes plain sense – additionally creating tests upfront prevent regressions later. Testing your individual functions will force you to break up your code into small modular functions, and maybe even adopting techniques like object oriented css.

Continuous integration

If you’re using continuous integration in your deployment process you are probably already running existing tests before deploying, the first step to making sure your Sass doesn’t break on deployment is making sure that it can compile with errors.

In case you’re already using Grunt for your local development workflow it would make sense to just make Grunt run your test and build tasks. If you’re using Travis it’s just a few simple steps to make your Grunt tasks run on every new build.

No matter what build system you are using, the key is that the build will fail if compilation of Sass/Compass fails - the last thing we want is for broken output css to make it into production.

Bootcamp

A BDD-style framework that doesn’t rely on any frameworks.

Bootcamp is developed by James Kyle who have done some crazy things with Sass, checkout his GitHub. The syntax for Bootcamp is clean and obvious, especially if you’ve worked with BDD frameworks in other languages before. You begin a new test suite by calling the mixin describe with the title for the spec suite and then the @content block inside:

@include describe("A suite") {
  @include it("contains spec with an expectation") {
    @include should(expect(true), to(be(true)));
  }
}

Specs are called by the mixin it which just like describe takes the name/title and a @content block. The spec contains at least one expectation which evaluates to either true or false, a spec with at least one false assertion is a failing spec.

The syntax for defining suites and specs are important, but the matchers that are available in assertions are just as important. Bootcamp comes with a bunch of rich matchers already included. An extensive list and demo of each matcher is right here.

It’s possible to disable suites and specs by changing the mixins to xdescribe and xit, just like Jasmine.

Installation

You can install Bootcamp from either npm or Bower:

$ npm install bootcamp --save-dev
$ bower install bootcamp --save-dev

After installing Bootcamp you’ll probably want to add it to your Sass :load_paths, instructions for Grunt and Compass:

npm   './node_modules/bootcamp/dist'
bower './bower_components/bootcamp/dist'

tests.scss

@import "bootcamp"; // or "[node_modules|bower_components]/bootcamp/dist/bootcamp" if you didn't set load_paths

@include runner-start;
@import "specs/example";
@include runner-end;

To run Bootcamp and get the results of the test suite simply compile the tests.scss file. If you’re using Grunt in your workflow please see a full working example in the official documentation.

The wiki for Bootcamp includes a full working example.

True

A TDD-style framework inspired by QUnit.

True is developed by the talented Eric M Suzanne who also gave a great talk at SassConf ‘13.

In True you define a suite with the mixin test-module with the title of the suite/module and again utilizing the @content block:

@include test-module('Column Math') {
  @include test('[function] is-symmetrical()') {
    @include assert-true(true, 'Number should be symmetrical');
  }
}

Individual specs are defined with the mixin test which again takes a name/title. True contains 4 different assertion-types: assert-true, assert-false, assert-equal and assert-unequal.

Installation

True can be installed with RubyGems:

$ gem install true

After installation you’ll need to add the following line to your config.rb:

require 'true'

It can also be installed with Bower or by copying the Sass directory from the project.

After install creating a test suite is a simple as creating a main test file. tests.scss

@import "functions/all"; // functions we want to test
@import "true";

@include test-module('Column Math') {
  @include test('[function] is-symmetrical()') {
    $s: is-symmetrical(12); // externally defined function

    @include assert-true($s, 'Number should be symmetrical');
  }
}

A great production example of True can be seen found Susy Next.

Going forward

As Chris Eppstein mentioned in his talk “It Takes a Village to Raise a Website” at SassConf ‘13, the community/ecosystem really needs to rally around these new unit testing frameworks.

We as a community need to figure out what it is that we want to test. What is missing in the current frameworks and maybe even in the Sass language. How are we going to do unit testing in Sass in the future.

Do we want a standard set of assertions and matchers in the Sass language, which we as a community can build different language/syntaxes on top of?


Update: Corrected mistakes in section about True, especially about dependencies on Compass, Ruby etc. Thanks to Eric for correcting me.

Free Our Bank Data

Full disclaimer: I worked for Spiir a couple of years back.

Why is it that you as a banking customer aren’t the owner of the data you generate?

The reason for this post was a recent incident in my home country Denmark, a Danish startup Spiir was reported to the police by the third largest bank in Denmark, Jyske Bank who is accusing Spiir of using “hacker-methods” to unlawfully gain access to users’ financial data.

Background

Spiir enables users to get a complete overview of their economy by analyzing transactions. In the early days of Spiir, the only way of getting your transactions from your bank into Spiir was by exporting, one account at a time, into CSV files and then importing these into Spiir. It worked, but it was too much of a hassle and broke the immediate feedback-loop that was important if users were to spend time on something as unsexy as bank data.

Spiir started a partnership with another Danish bank, Lån & Spar Bank, that allowed for automatic transfer of data from the bank to Spiir — if the customer wanted so. Most of the other banks were not as keen on this kind of partnership, even though their users were publicly asking for the integration.

Spiir then started looking at alternative solutions, so that their users could do what they wanted with their own bank data. It was clear that the banks were using relatively simple APIs for their own clients (mobile applications, etc.), which could potentially allow users to programmatically access their own data more easily than the manual CSV export.

After substantial investments into the legal aspects of this new kind of integration, Spiir launched an automatic synchronization of the bank data to Spiir. The legal aspects of this synchronization are clear because Spiir is only accessing the bank’s API on instructions from the user and not in a way that the user himself couldn’t access it. Furthermore, Spiir is only accessing the user’s data for the use of that specific user. The legal justification from Spiir can be found right here.

It is this particular integration that Jyske Bank find illegal. We are talking about a service that simply accesses an API on behalf of a customer who wants to use their own data in a way that the bank doesn’t approve of.

Even though Jyske Bank keeps calling Spiir’s methods for hacker-methods, it is clear to anybody who have the slightest technical knowledge that those claims are without merit. An example of the hack can be seen in this Gist by @soerenr.

Market disruption

This is a market ripe for a significant technological disruption, and we are already seeing players attacking this market from multiple positions.

We see it in the customer-facing sides of the market with startups such as Simple, Numbrs, Moven, Holvi, Tink, RocketBank, and so on. But I think, just as importantly, we are seeing startups focused on making the data from financial-institutions more easily accessible for developers and users:

  • Plaid recently raised a considerable funding round
  • figo seems to be gaining momentum in Germany
  • Open Bank Project sounds very promising
  • Standard Treasury(YC S13) aims at easing the hassle for businesses integrating with banks

It is time for the banking world to wake up and realize that the world around them have changed completely.

Free your own data

We should demand that our providers, no matter what type (bank, email, social network, etc.), set our data free. After all, we are the ones who generated and accumulated the data in the first place - is it not only fair that we can do whatever we want with it?

Every time I see company fighting this hard to keep user data to themselves, I can’t help but think that either they are relying on a dying business-model or they just simply don’t get what is happening around them. I don’t see how companies that actively fight against their customers’ wishes can survive in this day and age.

I, of course, know that change takes time, especially when it comes to big slow-moving companies such as banks, but that shouldn’t stop us from pursuing more open data from the services we use everyday.