youtube-dl allows you to download any video or song from YouTube, Vimeo, SoundCloud and 410 more sites.

youtube-dl have quickly become one of my favorite command line tools. Say goodbye to faulty browser extensions and sketchy websites, and say hello to downloading everything from any of the major video/audio sites right from your command line.¹

Installation

You can download the binary on the official download page - I always choose the easier Homebrew installation:

# Homebrew installation (preferred)
$ brew install youtube-dl

# UNIX installation with curl
$ sudo curl https://yt-dl.org/downloads/2014.09.06/youtube-dl -o /usr/local/bin/youtube-dl
$ sudo chmod a+x /usr/local/bin/youtube-dl

Usage

$ youtube-dl http://www.youtube.com/watch?v=DwYPG6vreJg
[youtube] Setting language
[youtube] Confirming age
[youtube] DwYPG6vreJg: Downloading webpage
[youtube] DwYPG6vreJg: Downloading video info webpage
[youtube] DwYPG6vreJg: Extracting video information
[download] Destination: Douglas Crockford - Advanced JavaScript-DwYPG6vreJg.mp4
[download] 100% of 169.41MiB in 00:43

Supported sites

Most of the leading video sites like YouTube, Vimeo, Dailymotion are supported, including a bunch of news, sports, trailers and of course porn sites. For a full list please see supported sites on GitHub.

# List all available sites
$ youtube-dl --extractor-descriptions

Is your favorite site missing from youtube-dl? Then please do head over to the GitHub repo and check out the Adding support for a new site section. Open source contributors are constantly improving the tool with support for new sites.

Video formats

youtube-dl supports a long range of formats and resolutions, that are passed in the -f, --format FORMAT parameter.

Every format has a format code that can be specified when downloading a video, including special name codes like: "best", "bestvideo", "bestaudio", "worst", "worstvideo" and "worstaudio". If no specific format is specified the best quality format is chosen.

To list down all available formats for a video URL use -F, --list-formats, the output will look like this:

$ youtube-dl -F http://www.youtube.com/watch?v=DwYPG6vreJg
[youtube] Setting language
[youtube] Confirming age
[youtube] DwYPG6vreJg: Downloading webpage
[youtube] DwYPG6vreJg: Downloading video info webpage
[youtube] DwYPG6vreJg: Extracting video information
[info] Available formats for DwYPG6vreJg:
format code extension resolution  note
171         webm      audio only  DASH audio , audio@128k (worst)
140         m4a       audio only  DASH audio , audio@128k
160         mp4       144p        DASH video , video only
242         webm      240p        DASH video , video only
133         mp4       240p        DASH video , video only
243         webm      360p        DASH video , video only
134         mp4       360p        DASH video , video only
244         webm      480p        DASH video , video only
135         mp4       480p        DASH video , video only
17          3gp       176x144
36          3gp       320x240
5           flv       400x240
43          webm      640x360
18          mp4       640x360     (best)

Rip music from videos - YouTube songs to Spotify

In addition to the video formats, youtube-dl also supports a whole range of post-processing options. Check out the GitHub repo for a full list of options.

I primarily use the -x, --extract-audio option to convert videos files into audio-only files - do note that the options has a couple of extra dependencies: ffmpeg or avconv and ffprobe or avprobe.

Shell alias yt

I pretty much only use Spotify for listening to music, unfortunately more obscure remixes and new releases often aren’t available for streaming through Spotify, so I end up needing an offline copy. Luckily youtube-dl supports downloading tracks from SoundCloud, where I find a lot of great new music - but as described earlier it supports extracting music from videos as one of the post-processing options.

Because I end up extracting audio from video files fairly often, I’ve created a quick little shell alias for downloading the audio from for example YouTube:

# alias for youtube-dl extracting audio
alias yt='youtube-dl --extract-audio --audio-format mp3'

Additional parameters

I’ve mentioned a couple of the parameters for youtube-dl in this post, but it supports a ton more. All options are well documented in the options section of the README.

One of my favorite hidden features is the ability to download your entire Watch Later playlist from YouTube or Vimeo, perfect entertainment for a long-haul flight.

Tool updates

The rate of updates to youtube-dl is very frequent, and there seems to be a lot of active contributors to the project. The original creator of the project Ricardo Garcia have long since stepped down as a maintainer, check out his post regarding the team working on it now: The fantastic youtube-dl team.

I once encountered a problem where I couldn’t download a video off YouTube because of a problem with encrypted signatures, I apparently had an old version of the tool and a simple update fixed the problem:

# Update youtube-dl
$ youtube-dl -U

No more copy/pasting lines from old .gitignore files

Every Git repo needs a .gitignore file, these often vary from project to project and creating them can be a tedious process.

This process used to, at least for me, involve copy/pasting from previous projects only to end up with a new project-specific Frankenstein-like file. Around a year ago I stumbled upon the site gitignore.io which makes it super simple to generate .gitignore files without resorting to copy/pasting from old projects.

The only problem was that when I start new projects I’ll usually be in my Terminal or editor of choice, and I don’t want to switch contexts by going to a website to generate the file.

gitignore.io offers an official and very simple command line utility, that’s documented here - but it doesn’t quite do as much as I wanted it to.

Luckily @soerenr sent me a link to this great little Gist by kqr.

I haven’t written much Bash besides a couple of small scripts here and there, but I thought I could add one little feature to the original gist by kqr. The ability to append existing .gitignore files.

Here is the full slightly edited function:¹

function gi() {
  result=$(curl "http://www.gitignore.io/api/$1" 2>/dev/null)

  if [[ $result =~ ERROR ]]; then
    echo "Query '$1' has no match. See a list of possible queries with 'gi list'"
  elif [[ $1 = list ]]; then
    echo "$result"
  else
    if [[ -f .gitignore ]]; then
      result=`echo "$result" | grep -v "# Created by http://www.gitignore.io"`
      echo ".gitignore already exists, appending"
      echo "$result" >> .gitignore
    else
      echo "$result" > .gitignore
    fi
  fi
}

The code should be fairly self explanatory and straight-forward to understand. To use the function in your Bash shell:

# Add the function to your ~/.bash_profile

# Remember to source your ~/.bash_profile after adding the function
$ source ~/.bash_profile

# Show available keyword types
$ gi list

# Create new, or append to existing, .gitignore file
$ gi sass

The Big Mac Index for Spotify Premium pricing

See the project here: Spotify International Pricing Index

Just after moving to Malaysia it shocked me how big the price difference was between Spotify Premium in Denmark and Malaysia.

One of the main reasons for the large price difference can be found in the particular catalog size in each country. I scoured the web looking for a way to get the total number of tracks available for each country, unfortunately Spotify doesn’t seem to make this data publicly available.¹

I’ve always wanted to do a simple project with data visualization with D3.js, so this a perfect opportunity to play around with that.

I wanted to replicate the charts from Big Mac Index, but with simpler interaction. The goal was basically a simple Javascript-generated infographic. I tried to match the styling of the Spotify OS X and iOS app as best as possible with similar colors and fonts.

Getting the pricing data

I wrote a simple little script in Node.js that crawls all the individual country pages for the pricing info. It then adds a little more relevant country info to each country from restcountries.eu and lastly converts the local price to USD with the help of money.js. Check it out on GitHub: spotify-crawler

Luckily Spotify keeps a nice and simple list of all their available countries. They also don’t force a location based on IP which would make crawling much harder without the use of proxies in different countries.

I exposed a simple API with all the countries on Heroku if you want to look at the data: http://spotify-pricing.herokuapp.com/api/

World map choropleth

I started out creating a map from the ground up, but quickly discovered datamaps.js.².

The datamaps.js library includes features like hover, popup, legend, etc. – all features which I would have had to reproduce if I followed the tutorial.

I hit some limits in the use of the library and had to make some edits to the main source file - I’ll try contributing the changes back to the source.

I associated the different fill colors based on which price-difference quantile each country fell into – still not convinced that this is the most correct way of doing it but the result looks reasonable. Getting a specific quantile with D3.js is as simple as:

var sortedDifferences = _.map(_.sortBy(countries, 'priceDifference'), 'priceDifference');

var quantile = [
  d3.quantile(sortedDifferences, 0.2),
  d3.quantile(sortedDifferences, 0.4),
  d3.quantile(sortedDifferences, 0.6),
  d3.quantile(sortedDifferences, 0.8)
];

Assigning a fillKey/fillColor can then be done by comparing each priceDifference and observe which quantile it falls into.

The fill colors themselves are calculated with the small Javascript library tinycolor.js which provides powerful color methods like .lighten and .darken – you may have seen similar methods in Sass.

Bar chart with negative values

The original chart from The Economist that I was trying to reproduce:

This was the first chart I started working on, since it was my first time working with D3.js it was also the chart that took the longest to replicate.

I followed this brilliant example from Mike Bostock showing how to build a simple bar chart with negative values.³

I added some additional text elements and labels, but the end result is pretty close to the example.

Scatter plot

I originally didn’t plan on reproducing this chart, but after making the first two charts I could just create this one as well - a fun little challenge. I started off with yet another great example from Mike Bostock.

GDP data varies widely between different sources, be it IMF, CIA or World Bank. I ended up using the GDP per capita, PPP (current international $) data from World Bank, and their API was fairly easy to use. All data are from 2012 because it was the latest year without estimated numbers in the World Bank API.

5 countries still returned null from the World Bank API: Andorra, Argentina, Liechtenstein, Monaco and Taiwan. These countries are not shown in the scatter plot chart. The CIA list of GDP (PPP) per capita does contain these countries, but the reported numbers are from different years.

This whole project gave me an opportunity to dust off some of the math and basic statistics I had learned years ago – and never really used since. I was about to create my own trend line calculation when I found yet another example from Mike Bostock showing how to calculate a trendline.

Wrap up

I’m currently paying for the most expensive Spotify Premium plan in the world when converted to USD.

So will I be making the switch? Probably not, I still want to listen to Danish tracks which aren’t available in the cheaper countries like Philippines and Malaysia.

Check out the GitHub repo for the project here: spotify-pricing

Further additions that I want to implement are recording price changes over time, and automatically adding new countries as Spotify expands further internationally.

Mobile HTML5-based apps that feel native.

Update: Since I published this blog post, AppGyver have changed quite a few things(and I’ve not kept up with all the changes). Check the AppGyver website for the newest updates.

For a long time mobile app development have been in two very separate camps, native development and HTML5 development. In between these two camps there has been multiple attempts at trying to bridge the gap between Web code and native elements.

Maintaining completely separate codebases often isn’t feasible for many smaller teams, so many teams often opt for one of the countless cross-platform development frameworks like Adobe AIR, Appcelerator Titanium, Intel App Framework or Xamarin.¹

The most popular and widely used cross-platform framework is without a doubt Cordova/PhoneGap.² This particular framework enables developers to use standards-based web technologies to bridge traditional web applications and mobile devices. It’s always been stated that the larger goal of Cordova is to at some point make itself obsolete as browser vendors eventually implement many of the APIs.

The downside to Cordova have long been that the feel of the resulting apps were clearly not native, it was easy to spot the differences between these hybrid-apps and native apps.

http://www.tipsyandtumbler.co.uk/blog/boydlee/what-is-native

Description

Steroids promises to bridge this gap by providing rapid development with the ease of HTML5/JS portability at native code speeds.

If you’ve already used Cordova before you’ll feel right at home with Steroids since it builds on the foundation laid out by the Cordova project. This also means that you have access to all the APIs exposed by Cordova, and not to forget the growing list of plugins for the platform.

Steroids gives you access to a number of native UI elements such as the navigationBar, tabBar, modal, drawer but most important of all is the native views which allows you to have multiple native-wrapped WebViews running at the same time. The ability to have multiple WebViews stacked really gives the illusion of a high performance native app, especially when you destroy a WebView to for example go back from a child-view.³

One important feature about Steroids that’s often forgotten in the coverage of the development platform, is that it enables non-Mac users to develop and compile iOS apps since all compilation and deployment is done in the cloud. No more need to have a local version of Xcode to compile and deploy applications to the App Store.

Setup process

Installing Steroids and getting everything up and running is quite a simple process.

• Create a free AppGyver account here
• Install the required tools
  • Git
  • Python
  • Node.js
• If you’ve used npm before then this process should be familiar to you:

# Install the Steroids CLI package:
$ npm install steroids -g

# Connect the CLI tool to your AppGyver account:
$ steroids login

# Now you're ready to create your first project:
$ steroids create ProjectName

• To run your project on a real device simply download the AppGyver Scanner for iOS or Android, and run the following command:

$ steroids connect

For full instructions please refer to AppGyver’s guide.

Android support

I’m not sure if AppGyver was more focused on iOS in the beginning, or if supporting iOS native UI elements was simply just easier – but it’s clear that support for native UI elements on Android is seriously lacking behind that of the iOS side. This is unfortunate since equal support on both platform would make Steroids the perfect tool for developing cross-platform web apps.

Preliminary support for Android 4.4(KitKat) was just released last week, but there’s still lots of bugs and problems with the support. There has been a lengthy discussion on this GitHub issue, please refer to it for the newest developments.

AppGyver currently seems to be focused on the development of an Android Simulator, since there is only support for the iOS simulator at this point.

Possiblities

One of the great things about Steroids is the before-mentioned ability to spawn new native WebViews for each of your views. This means that you don’t have to build traditional single-page apps like you would’ve done before. Using multiple WebViews however comes with some adjusting since communicating between each view is a little more complicated compared to the communication in a single-page app.⁴

With each WebView running a completely separate JavaScript runtime you can’t share global variables or data between them, and singletons like AngularJS services would be maintained separately for each instance of ng-app.

To help developers overcome these limitations Steroids have implemented a couple of HTML5 standards that should let you do most of the things a traditional single-page app could do.

window.postMessage

Steroids implements a simplified version of the postMessage standard, most notable differences being no event.origin and event.source attributes, and no targetOrigin in the API call.

A Steroids app running in the app package is walled-off from the general Internet, so all postMessage calls are automatically sent to every WebView in memory. Even though it’s a simplified version of the postMessage standard it’s tremendously powerful and useful in the building of Steroids apps.

Steroids includes and example of how postMessage can be used with the native drawer UI element:

$ steroids generate example drawer

window.localStorage

The implementation in Steroids makes the localStorage available in all your views, which makes it a great way to share data between separate views. Storing objects in localStorage can be done with JSON.stringify(object) and JSON.parse(string).

One key difference to using localStorage in Steroids is that the users won’t have access to developer tools, which means that they can’t manipulate the data you store in localStorage. The stored items will also be persistent through app updates, but it will be removed if the app cache is emptied.

Steroids also provides a more robust database for sharing data between WebViews, the AppGyver Scanner comes bundled with the PhoneGap SQLite plugin.

See this guide by the AppGyver team on prepopulating the database.

Background service-layer

One of the suggested ways of handling the data between different views is to implement a service-layer always running in the background. A document that runs all your applications singletons, making use of the above-mentioned standards it’s quite simple to accomplish.

Steroids have the ability to preload WebViews with the steroids.views.WebView.prototype.preload function. Running a background view can be done like this:

var backgroundView = new steroids.views.WebView("services.html")

backgroundView.preload();

Workflow and deployment flow

If you download the before-mentioned AppGyver Scanner app (iOS or Android)

workflow/deployment http://marcgg.com/blog/2013/08/29/appgyver-steroids-iphone-hybrid-javascript/#the-scanner-app

Features coming

AppGyver announced a couple of weeks ago that they had raised $2.5M in funding to expand the Steroids development platform. AppGyver said that they were planning on using this newly raised capital to expand the team and build out their strongly requested enterprise features.

As described above AppGyver allows developers to use the already existing Cordova plugins on the platform, some of these plugins are either outdated, poorly supported or simply just doesn’t work very well with the Steroids platform. That’s why AppGyver will also be expanding their own add-ons, the idea is to provide a complete library of maintained and well-supported plugins to help developers quickly build their applications.

At this point AppGyver have published 4 add-ons: Facebook, Urban Airship for push notifications, OAuth.io and their own GPS add-on. The following add-ons are announced, but not yet published.

Another new feature that AppGyver is teasing on their website is something called App Reload which promises “Continuous Integration with App Stores”. This sounds close to something they’ve earlier discussed, dynamic updating of assets over-the-air without submission to the App Stores. It’ll be very interesting to see how this’ll work once it’s available.