Release: grunt-xcode v2

complete rewrite + new features: grunt-xcode

I started building the initial version of grunt-xcode around November last year. The primary reason for the project was that I was getting tired of manually having to do iOS builds for our clients - it needed to be a part of our Grunt build task.

I stumbled upon shenzhen which actually worked fairly well for me as a command-line tool, so I thought “hey let me just wrap this in JS”. That was probably not the wisest decision, but grunt-xcode v1 ended up sort of working for us (not so much for everybody else).

To be honest the code was quite ugly and I didn’t like the dependency on a RubyGem - it just didn’t feel right. I also started getting some bug reports on GitHub and email.

Rewrite

The primary goal of the rewrite was to remove the dependency on shenzhen and use the built-in xcodebuild tool instead.

Features:

  • remove dependency on shenzhen
  • added support for all parameters available in xcodebuild
  • show progress indicators for archiving and export tasks
  • show stdout if Grunt is run with the --verbose flag

grunt-xcode build

Usage

I’m actively using grunt-xcode for a client project and it’s been working out great for me.

Installing grunt-xcode is just as simple as any other npm module:

$ npm install grunt-xcode --save-dev
require('load-grunt-tasks')(grunt);

grunt.initConfig({
    xcode: {
        options: {
          project: '/path/to/my/awesome/App/App.xcodeproj',
          scheme: 'Release'
        }
    }
});

grunt.registerTask('default', ['xcode']);

For a list of all options please refer to the README.

Hopefully the codebase is a lot more readable now, that was at least the goal.

Please do let me know if you run into any issues with grunt-xcode, I’m sure there are multiple cases I haven’t tested properly.

Download YouTube videos and songs

youtube-dl allows you to download any video or song from YouTube, Vimeo, SoundCloud and 410 more sites.

youtube-dl have quickly become one of my favorite command line tools. Say goodbye to faulty browser extensions and sketchy websites, and say hello to downloading everything from any of the major video/audio sites right from your command line.1

Installation

You can download the binary on the official download page - I always choose the easier Homebrew installation:

# Homebrew installation (preferred)
$ brew install youtube-dl

# UNIX installation with curl
$ sudo curl https://yt-dl.org/downloads/2014.09.06/youtube-dl -o /usr/local/bin/youtube-dl
$ sudo chmod a+x /usr/local/bin/youtube-dl

Usage

$ youtube-dl http://www.youtube.com/watch?v=DwYPG6vreJg
[youtube] Setting language
[youtube] Confirming age
[youtube] DwYPG6vreJg: Downloading webpage
[youtube] DwYPG6vreJg: Downloading video info webpage
[youtube] DwYPG6vreJg: Extracting video information
[download] Destination: Douglas Crockford - Advanced JavaScript-DwYPG6vreJg.mp4
[download] 100% of 169.41MiB in 00:43

Supported sites

Most of the leading video sites like YouTube, Vimeo, Dailymotion are supported, including a bunch of news, sports, trailers and of course porn sites. For a full list please see supported sites on GitHub.

# List all available sites
$ youtube-dl --extractor-descriptions

Is your favorite site missing from youtube-dl? Then please do head over to the GitHub repo and check out the Adding support for a new site section. Open source contributors are constantly improving the tool with support for new sites.

Video formats

youtube-dl supports a long range of formats and resolutions, that are passed in the -f, --format FORMAT parameter.

Every format has a format code that can be specified when downloading a video, including special name codes like: "best", "bestvideo", "bestaudio", "worst", "worstvideo" and "worstaudio". If no specific format is specified the best quality format is chosen.

To list down all available formats for a video URL use -F, --list-formats, the output will look like this:

$ youtube-dl -F http://www.youtube.com/watch?v=DwYPG6vreJg
[youtube] Setting language
[youtube] Confirming age
[youtube] DwYPG6vreJg: Downloading webpage
[youtube] DwYPG6vreJg: Downloading video info webpage
[youtube] DwYPG6vreJg: Extracting video information
[info] Available formats for DwYPG6vreJg:
format code extension resolution  note
171         webm      audio only  DASH audio , audio@128k (worst)
140         m4a       audio only  DASH audio , audio@128k
160         mp4       144p        DASH video , video only
242         webm      240p        DASH video , video only
133         mp4       240p        DASH video , video only
243         webm      360p        DASH video , video only
134         mp4       360p        DASH video , video only
244         webm      480p        DASH video , video only
135         mp4       480p        DASH video , video only
17          3gp       176x144
36          3gp       320x240
5           flv       400x240
43          webm      640x360
18          mp4       640x360     (best)

Rip music from videos - YouTube songs to Spotify

In addition to the video formats, youtube-dl also supports a whole range of post-processing options. Check out the GitHub repo for a full list of options.

I primarily use the -x, --extract-audio option to convert videos files into audio-only files - do note that the options has a couple of extra dependencies: ffmpeg or avconv and ffprobe or avprobe.

Shell alias yt

I pretty much only use Spotify for listening to music, unfortunately more obscure remixes and new releases often aren’t available for streaming through Spotify, so I end up needing an offline copy. Luckily youtube-dl supports downloading tracks from SoundCloud, where I find a lot of great new music - but as described earlier it supports extracting music from videos as one of the post-processing options.

Because I end up extracting audio from video files fairly often, I’ve created a quick little shell alias for downloading the audio from for example YouTube:

# alias for youtube-dl extracting audio
alias yt='youtube-dl --extract-audio --audio-format mp3'

Additional parameters

I’ve mentioned a couple of the parameters for youtube-dl in this post, but it supports a ton more. All options are well documented in the options section of the README.

One of my favorite hidden features is the ability to download your entire Watch Later playlist from YouTube or Vimeo, perfect entertainment for a long-haul flight.

Tool updates

The rate of updates to youtube-dl is very frequent, and there seems to be a lot of active contributors to the project. The original creator of the project Ricardo Garcia have long since stepped down as a maintainer, check out his post regarding the team working on it now: The fantastic youtube-dl team.

I once encountered a problem where I couldn’t download a video off YouTube because of a problem with encrypted signatures, I apparently had an old version of the tool and a simple update fixed the problem:

# Update youtube-dl
$ youtube-dl -U
  1. I shall avoid all talk about copyright laws and other legal implications of using this tool.


Update: Corrected small typo spotted by @soerenr, thanks!

Generate .gitignore files with Bash function

No more copy/pasting lines from old .gitignore files

Every Git repo needs a .gitignore file, these often vary from project to project and creating them can be a tedious process.

This process used to, at least for me, involve copy/pasting from previous projects only to end up with a new project-specific Frankenstein-like file. Around a year ago I stumbled upon the site gitignore.io which makes it super simple to generate .gitignore files without resorting to copy/pasting from old projects.

The only problem was that when I start new projects I’ll usually be in my Terminal or editor of choice, and I don’t want to switch contexts by going to a website to generate the file.

gitignore.io offers an official and very simple command line utility, that’s documented here - but it doesn’t quite do as much as I wanted it to.

Luckily I got a link sent from @soerenr to this great little Gist by kqr.

I haven’t written much Bash besides a couple of small scripts here and there, but I thought I could add one little feature to the original gist by kqr. The ability to append existing .gitignore files.

Here is the full slightly edited function:1

function gi() {
  result=$(curl "http://www.gitignore.io/api/$1" 2>/dev/null)

  if [[ $result =~ ERROR ]]; then
    echo "Query '$1' has no match. See a list of possible queries with 'gi list'"
  elif [[ $1 = list ]]; then
    echo "$result"
  else
    if [[ -f .gitignore ]]; then
      result=`echo "$result" | grep -v "# Created by http://www.gitignore.io"`
      echo ".gitignore already exists, appending"
      echo "$result" >> .gitignore
    else
      echo "$result" > .gitignore
    fi
  fi
}

The code should be fairly self explanatory and straight-forward to understand. To use the function in your Bash shell:

# Add the function to your ~/.bash_profile

# Remember to source your ~/.bash_profile after adding the function
$ source ~/.bash_profile

# Show available keyword types
$ gi list

# Create new, or append to existing, .gitignore file
$ gi sass
  1. I created a PR with my edit and kqr wrote a great suggestion. I’ll update this post with improvement from kqr.

Spotify International Pricing Index

The Big Mac Index for Spotify Premium pricing

See the project here: Spotify International Pricing Index

Just after moving to Malaysia it shocked me how big the price difference was between Spotify Premium in Denmark and Malaysia.

One of the main reasons for the large price difference can be found in the particular catalog size in each country. I scoured the web looking for a way to get the total number of tracks available for each country, unfortunately Spotify doesn’t seem to make this data publicly available.1

I’ve always wanted to do a simple project with data visualization with D3.js, so this a perfect opportunity to play around with that.

I wanted to replicate the charts from Big Mac Index, but with simpler interaction. The goal was basically a simple Javascript-generated infographic. I tried to match the styling of the Spotify OS X and iOS app as best as possible with similar colors and fonts.

Getting the pricing data

I wrote a simple little script in Node.js that crawls all the individual country pages for the pricing info. It then adds a little more relevant country info to each country from restcountries.eu and lastly converts the local price to USD with the help of money.js. Check it out on GitHub: spotify-crawler

Luckily Spotify keeps a nice and simple list of all their available countries. They also don’t force a location based on IP which would make crawling much harder without the use of proxies in different countries.

I exposed a simple API with all the countries on Heroku if you want to look at the data: http://spotify-pricing.herokuapp.com/api/

World map choropleth

I started out creating a map from the ground up, but quickly discovered datamaps.js.2.

The datamaps.js library includes features like hover, popup, legend, etc. – all features which I would have had to reproduce if I followed the tutorial.

I hit some limits in the use of the library and had to make some edits to the main source file - I’ll try contributing the changes back to the source.

spotify world map

I associated the different fill colors based on which price-difference quantile each country fell into – still not convinced that this is the most correct way of doing it but the result looks reasonable. Getting a specific quantile with D3.js is as simple as:

var sortedDifferences = _.map(_.sortBy(countries, 'priceDifference'), 'priceDifference');

var quantile = [
  d3.quantile(sortedDifferences, 0.2),
  d3.quantile(sortedDifferences, 0.4),
  d3.quantile(sortedDifferences, 0.6),
  d3.quantile(sortedDifferences, 0.8)
];

Assigning a fillKey/fillColor can then be done by comparing each priceDifference and observe which quantile it falls into.

The fill colors themselves are calculated with the small Javascript library tinycolor.js which provides powerful color methods like .lighten and .darken – you may have seen similar methods in Sass.

Bar chart with negative values

The original chart from The Economist that I was trying to reproduce: negative bar chart

This was the first chart I started working on, since it was my first time working with D3.js it was also the chart that took the longest to replicate.

I followed this brilliant example from Mike Bostock showing how to build a simple bar chart with negative values.3

I added some additional text elements and labels, but the end result is pretty close to the example.

Scatter plot

I originally didn’t plan on reproducing this chart, but after making the first two charts I could just create this one as well - a fun little challenge. I started off with yet another great example from Mike Bostock.

GDP data varies widely between different sources, be it IMF, CIA or World Bank. I ended up using the GDP per capita, PPP (current international $) data from World Bank, and their API was fairly easy to use. All data are from 2012 because it was the latest year without estimated numbers in the World Bank API.

5 countries still returned null from the World Bank API: Andorra, Argentina, Liechtenstein, Monaco and Taiwan. These countries are not shown in the scatter plot chart. The CIA list of GDP (PPP) per capita does contain these countries, but the reported numbers are from different years.

This whole project gave me an opportunity to dust off some of the math and basic statistics I had learned years ago – and never really used since. I was about to create my own trend line calculation when I found yet another example from Mike Bostock showing how to calculate a trendline.

Wrap up

I’m currently paying for the most expensive Spotify Premium plan in the world when converted to USD.

So will I be making the switch? Probably not, I still want to listen to Danish tracks which aren’t available in the cheaper countries like Philippines and Malaysia.

Check out the GitHub repo for the project here: spotify-pricing

Further additions that I want to implement are recording price changes over time, and automatically adding new countries as Spotify expands further internationally.

  1. If you know any way to get this data, please let me know.

  2. There is a great tutorial on how to create maps with D3.js here: http://bost.ocks.org/mike/map/

  3. Mike Bostock is the creator of D3.js


Update: Corrected small typo spotted by @nikreiman, thanks!