youtube-dl allows you to download any video or song from YouTube, Vimeo, SoundCloud and 410 more sites.
youtube-dl have quickly become one of my favorite command line tools. Say goodbye to faulty browser extensions and sketchy websites, and say hello to downloading everything from any of the major video/audio sites right from your command line.1
You can download the binary on the official download page - I always choose the easier Homebrew installation:
Most of the leading video sites like YouTube, Vimeo, Dailymotion are supported, including a bunch of news, sports, trailers and of course porn sites. For a full list please see supported sites on GitHub.
youtube-dl supports a long range of formats and resolutions, that are passed in the -f, --format FORMAT parameter.
Every format has a format code that can be specified when downloading a video, including special name codes like: "best", "bestvideo", "bestaudio", "worst", "worstvideo" and "worstaudio". If no specific format is specified the best quality format is chosen.
To list down all available formats for a video URL use -F, --list-formats, the output will look like this:
Rip music from videos - YouTube songs to Spotify
In addition to the video formats, youtube-dl also supports a whole range of post-processing options. Check out the GitHub repo for a full list of options.
I primarily use the -x, --extract-audio option to convert videos files into audio-only files - do note that the options has a couple of extra dependencies: ffmpeg or avconv and ffprobe or avprobe.
Shell alias yt
I pretty much only use Spotify for listening to music, unfortunately more obscure remixes and new releases often aren’t available for streaming through Spotify, so I end up needing an offline copy. Luckily youtube-dl supports downloading tracks from SoundCloud, where I find a lot of great new music - but as described earlier it supports extracting music from videos as one of the post-processing options.
Because I end up extracting audio from video files fairly often, I’ve created a quick little shell alias for downloading the audio from for example YouTube:
I’ve mentioned a couple of the parameters for youtube-dl in this post, but it supports a ton more. All options are well documented in the options section of the README.
One of my favorite hidden features is the ability to download your entire Watch Later playlist from YouTube or Vimeo, perfect entertainment for a long-haul flight.
The rate of updates to youtube-dl is very frequent, and there seems to be a lot of active contributors to the project. The original creator of the project Ricardo Garcia have long since stepped down as a maintainer, check out his post regarding the team working on it now: The fantastic youtube-dl team.
I once encountered a problem where I couldn’t download a video off YouTube because of a problem with encrypted signatures, I apparently had an old version of the tool and a simple update fixed the problem:
I shall avoid all talk about copyright laws and other legal implications of using this tool. ↩
Update 1: Corrected small typo spotted by @soerenr, thanks!
No more copy/pasting lines from old .gitignore files
Every Git repo needs a .gitignore file, these often vary from project to project and creating them can be a tedious process.
This process used to, at least for me, involve copy/pasting from previous projects only to end up with a new project-specific Frankenstein-like file. Around a year ago I stumbled upon the site gitignore.io which makes it super simple to generate .gitignore files without resorting to copy/pasting from old projects.
The only problem was that when I start new projects I’ll usually be in my Terminal or editor of choice, and I don’t want to switch contexts by going to a website to generate the file.
gitignore.io offers an official and very simple command line utility, that’s documented here - but it doesn’t quite do as much as I wanted it to.
Just after moving to Malaysia it shocked me how big the price difference was between Spotify Premium in Denmark and Malaysia.
One of the main reasons for the large price difference can be found in the particular catalog size in each country. I scoured the web looking for a way to get the total number of tracks available for each country, unfortunately Spotify doesn’t seem to make this data publicly available.1
I’ve always wanted to do a simple project with data visualization with D3.js, so this a perfect opportunity to play around with that.
Getting the pricing data
I wrote a simple little script in Node.js that crawls all the individual country pages for the pricing info. It then adds a little more relevant country info to each country from restcountries.eu and lastly converts the local price to USD with the help of money.js. Check it out on GitHub: spotify-crawler
Luckily Spotify keeps a nice and simple list of all their available countries. They also don’t force a location based on IP which would make crawling much harder without the use of proxies in different countries.
I started out creating a map from the ground up, but quickly discovered datamaps.js.2.
The datamaps.js library includes features like hover, popup, legend, etc. – all features which I would have had to reproduce if I followed the tutorial.
I hit some limits in the use of the library and had to make some edits to the main source file - I’ll try contributing the changes back to the source.
I associated the different fill colors based on which price-difference quantile each country fell into – still not convinced that this is the most correct way of doing it but the result looks reasonable. Getting a specific quantile with D3.js is as simple as:
Assigning a fillKey/fillColor can then be done by comparing each priceDifference and observe which quantile it falls into.
Bar chart with negative values
The original chart from The Economist that I was trying to reproduce:
This was the first chart I started working on, since it was my first time working with D3.js it was also the chart that took the longest to replicate.
I followed this brilliant example from Mike Bostock showing how to build a simple bar chart with negative values.3
I added some additional text elements and labels, but the end result is pretty close to the example.
I originally didn’t plan on reproducing this chart, but after making the first two charts I could just create this one as well - a fun little challenge. I started off with yet another great example from Mike Bostock.
GDP data varies widely between different sources, be it IMF, CIA or World Bank.
I ended up using the GDP per capita, PPP (current international $)data from World Bank, and their API was fairly easy to use. All data are from 2012 because it was the latest year without estimated numbers in the World Bank API.
5 countries still returned null from the World Bank API: Andorra, Argentina, Liechtenstein, Monaco and Taiwan. These countries are not shown in the scatter plot chart. The CIA list of GDP (PPP) per capita does contain these countries, but the reported numbers are from different years.
This whole project gave me an opportunity to dust off some of the math and basic statistics I had learned years ago – and never really used since. I was about to create my own trend line calculation when I found yet another example from Mike Bostock showing how to calculate a trendline.
I’m currently paying for the most expensive Spotify Premium plan in the world when converted to USD.
So will I be making the switch? Probably not, I still want to listen to Danish tracks which aren’t available in the cheaper countries like Philippines and Malaysia.
Update: Since I published this blog post, AppGyver have changed quite a few things(and I’ve not kept up with all the changes). Check the AppGyver website for the newest updates.
For a long time mobile app development have been in two very separate camps, native development and HTML5 development. In between these two camps there has been multiple attempts at trying to bridge the gap between Web code and native elements.
The most popular and widely used cross-platform framework is without a doubt Cordova/PhoneGap.2 This particular framework enables developers to use standards-based web technologies to bridge traditional web applications and mobile devices. It’s always been stated that the larger goal of Cordova is to at some point make itself obsolete as browser vendors eventually implement many of the APIs.
The downside to Cordova have long been that the feel of the resulting apps were clearly not native, it was easy to spot the differences between these hybrid-apps and native apps.
Steroids promises to bridge this gap by providing rapid development with the ease of HTML5/JS portability at native code speeds.
If you’ve already used Cordova before you’ll feel right at home with Steroids since it builds on the foundation laid out by the Cordova project. This also means that you have access to all the APIs exposed by Cordova, and not to forget the growing list of plugins for the platform.
Steroids gives you access to a number of native UI elements such as the navigationBar, tabBar, modal, drawer but most important of all is the native views which allows you to have multiple native-wrapped WebViews running at the same time. The ability to have multiple WebViews stacked really gives the illusion of a high performance native app, especially when you destroy a WebView to for example go back from a child-view.3
One important feature about Steroids that’s often forgotten in the coverage of the development platform, is that it enables non-Mac users to develop and compile iOS apps since all compilation and deployment is done in the cloud. No more need to have a local version of Xcode to compile and deploy applications to the App Store.
Installing Steroids and getting everything up and running is quite a simple process.
If you’ve used npm before then this process should be familiar to you:
To run your project on a real device simply download the AppGyver Scanner for iOS or Android, and run the following command:
For full instructions please refer to AppGyver’s guide.
I’m not sure if AppGyver was more focused on iOS in the beginning, or if supporting iOS native UI elements was simply just easier – but it’s clear that support for native UI elements on Android is seriously lacking behind that of the iOS side. This is unfortunate since equal support on both platform would make Steroids the perfect tool for developing cross-platform web apps.
Preliminary support for Android 4.4(KitKat) was just released last week, but there’s still lots of bugs and problems with the support. There has been a lengthy discussion on this GitHub issue, please refer to it for the newest developments.
AppGyver currently seems to be focused on the development of an Android Simulator, since there is only support for the iOS simulator at this point.
One of the great things about Steroids is the before-mentioned ability to spawn new native WebViews for each of your views. This means that you don’t have to build traditional single-page apps like you would’ve done before. Using multiple WebViews however comes with some adjusting since communicating between each view is a little more complicated compared to the communication in a single-page app.4
To help developers overcome these limitations Steroids have implemented a couple of HTML5 standards that should let you do most of the things a traditional single-page app could do.
Steroids implements a simplified version of the postMessage standard, most notable differences being no event.origin and event.source attributes, and no targetOrigin in the API call.
A Steroids app running in the app package is walled-off from the general Internet, so all postMessage calls are automatically sent to every WebView in memory. Even though it’s a simplified version of the postMessage standard it’s tremendously powerful and useful in the building of Steroids apps.
Steroids includes and example of how postMessage can be used with the native drawer UI element:
The implementation in Steroids makes the localStorage available in all your views, which makes it a great way to share data between separate views.
Storing objects in localStorage can be done with JSON.stringify(object) and JSON.parse(string).
One key difference to using localStorage in Steroids is that the users won’t have access to developer tools, which means that they can’t manipulate the data you store in localStorage. The stored items will also be persistent through app updates, but it will be removed if the app cache is emptied.
Steroids also provides a more robust database for sharing data between WebViews, the AppGyver Scanner comes bundled with the PhoneGap SQLite plugin.
See this guide by the AppGyver team on prepopulating the database.
One of the suggested ways of handling the data between different views is to implement a service-layer always running in the background. A document that runs all your applications singletons, making use of the above-mentioned standards it’s quite simple to accomplish.
AppGyver announced a couple of weeks ago that they had raised $2.5M in funding to expand the Steroids development platform. AppGyver said that they were planning on using this newly raised capital to expand the team and build out their strongly requested enterprise features.
As described above AppGyver allows developers to use the already existing Cordova plugins on the platform, some of these plugins are either outdated, poorly supported or simply just doesn’t work very well with the Steroids platform. That’s why AppGyver will also be expanding their own add-ons, the idea is to provide a complete library of maintained and well-supported plugins to help developers quickly build their applications.
At this point AppGyver have published 4 add-ons: Facebook, Urban Airship for push notifications, OAuth.io and their own GPS add-on. The following add-ons are announced, but not yet published.
Another new feature that AppGyver is teasing on their website is something called App Reload which promises “Continuous Integration with App Stores”. This sounds close to something they’ve earlier discussed, dynamic updating of assets over-the-air without submission to the App Stores. It’ll be very interesting to see how this’ll work once it’s available.
See PropertyCross for comparisons and further information about cross-platform frameworks. ↩