Uploading videos to YouTube with perl

As part of a perl project recently I needed to generate a load of videos (using the excellent MLT framework) and upload them to YouTube. As the bulk of the project was already in perl, I needed to write a YouTube upload module for it. Here’s a rough overview of how to authenticate and upload a video (chunked rather than loading it all into memory at once) with perl. Note as well it was quite a struggle to get long-term auth tokens from the Google services – one slight mistake with a parameter and they only give temporary grants which last for an hour or two rather than indefinitely.

package Youtube::Upload;
use Moo;

use LWP::Authen::OAuth2;
use Path::Tiny 'path';
use URI::QueryParam;
use JSON::XS qw< encode_json decode_json >;
use HTTP::Message;

# API described at https://developers.google.com/youtube/v3/docs/videos/update

has auth_id => is => 'ro', required => 1;
has auth_secret => is => 'ro', required => 1;
has redirect_uri => is => 'ro', required => 1;

# If you havn't used this for a while, blank these and re-run and you'll
# probably need to do some auth against google.
has auth_code => is => 'ro';
has auth_token => is => 'ro';

has auth => is => 'lazy', builder => \&_google_auth;

sub upload {
    my ($self, $details, $youtube_code, $video_file) = @_;

    die "No id to update, but also nothing to upload" if !$youtube_code && !$video_file;

    my %body = %$details;

    # Allow all embedding
    $body{status}{embeddable} = JSON::XS::true;

    my $magic_split = 'BlA123H123BLAH'; # A unique string...

    my $is_new = !defined $youtube_code;
    my ($content, %headers, $uri);
    if( !$is_new ) {
        $body{id} = $youtube_code;

        $content = encode_json(\%body);
        $headers{'Content-Type'} = 'application/json';
        $uri = URI->new( 'https://www.googleapis.com/youtube/v3/videos' );
    } else {
        my $msg = HTTP::Message->new([
            'Content-Type' => 'multipart/related',
        ]);

        $msg->add_part(
            HTTP::Message->new([
                'Content-Type' => 'application/json',
            ], encode_json(\%body) )
        );

        my $video_msg = 
            HTTP::Message->new(
                [
                    'Content-Type' => 'video/*',
                ],
                $magic_split,
            );
        $msg->add_part( $video_msg );
            
        $content = $msg->as_string;
        (my $head, $content) = split /\r?\n\r?\n/, $content, 2;
        my ($k, $v) = split /:\s*/, $head, 2;
        $headers{$k} = $v;
        $uri = URI->new( 'https://www.googleapis.com/upload/youtube/v3/videos' );
    }

    delete $body{id};
    $uri->query_form_hash({ 
        part => join(',', keys %body), 
    });

    my $res;
    if( $is_new ) {
        my @content = split /\Q$magic_split/, $content;
        die "Magic split failed" if @content != 2;

        my $content_fh = path($video_file)->openr;
        my $request = HTTP::Request->new( 'POST', $uri, HTTP::Headers->new( %headers ), sub {
            #warn "chunk uploaded";
            return '' if !@content;

            if( @content > 1 ) {
                return shift @content;
            } else {
                my $read = read $content_fh, my $data, 1_000_000;
                if( !$read ) {
                    return shift @content;
                }
                return $data;
            }
        } );
        $res = $self->auth->request( $request );
    } else {
        $res = $self->auth->put( $uri,
            %headers,
            Content => $content
        );
    }

    my $cont = $res->decoded_content;
    my $ret;
    if( !$res->is_success ) {
        if($res->code != 403) {   # not our video
            die "Response not success: $cont for " . ( $youtube_code || '' );
        }
    } else {
        $ret = decode_json $cont;
    }

    return ref($ret) ? $ret : {};
}

sub _google_auth {
    my ($self) = @_;

    my $auth = LWP::Authen::OAuth2->new(
        service_provider => 'Google',
        redirect_uri => $self->redirect_uri,
        client_type => "web server",

        client_id      => $self->auth_id,
        client_secret  => $self->auth_secret,

        save_tokens => sub {
            say "Save token string: $_[0]" if !$self->auth_token;
        },

        token_string => $self->auth_token,
    );

    # For debug:
    #$auth->user_agent->add_handler("request_send",  sub { shift->dump(maxlength => 10_000); return });
    #$auth->user_agent->add_handler("response_done", sub { shift->dump(maxlength => 10_000); return });

    if( !$self->auth_code ) {
        say $auth->authorization_url(
            scope=> 'https://www.googleapis.com/auth/youtube https://www.googleapis.com/auth/youtube.upload',
            # Need these two to get a refresh token
            approval_prompt => 'force',
            access_type => 'offline',
        );
        exit;
    }

    $auth->request_tokens( code => $self->auth_code ) if !$self->auth_token;

    return $auth;
}

As per this bug report you need to hack LWP::Authen::OAuth2::AccessToken::Bearer to enable the chunked uploads to work, otherwise it throws an error.

The auth_id and auth_secret parameters are given by the google code console when you sign up for YouTube API access, and the redirect_uri should be where the web app would redirect to after displaying the Google oauth permission grant screen. The first time you run the script, you’ll need to save the auth_code / auth_token parts and use them in future when calling. Even though it should grant a new auth token each time the script is run you still need to present the original one from the look of things.

Element scrolling within Angular (1) pages

Back to Angular 1 for today’s post as I’ve been doing some work on an older project today. As we all know, in the old days of the web, you could scroll to items within the page by using an <a> element with a hash reference, like:

<a href="#thing-to-scroll-to">scroll here</a>

<h2 id="thing-to-scroll-to">My title</h2>

With the advent of single-page sites and multiple ajax pages under them however, the hash section of a query parameter is increasingly unable to be used. On Angular 1 with angular-route this becomes impossible.

So what if for example you want to have some references at the top of a page within an Angular application which will scroll the user down to certain sections below, just like the hash references used to do? There are a number of suggestions on the internet for this but they are not very reusable, so I decided to create a simple directive that does this:

app.directive('clickScrollTo', function () {
    return {
        restrict: 'A',
        scope: {
            clickScrollTo: '@'
        },
        link: function (scope, el, attr) {
            el.click(function(event) {
                var main_page = $('.main-page');
                event.preventDefault();
                main_page
                    .stop()
                    .animate(
                        { scrollTop: main_page.scrollTop() + $(scope.clickScrollTo).offset().top },
                        500,
                        'linear'
                    );
            });
        }
    }
});

This is slightly complicated by the fact that I am using an element with class="main-page" with overflow: auto set to contain the scrollable section of the app. If you don’t have a layout like this just replace the $('.main-page') part with $('body').

Then you can easily create elements like:

<span class="a" click-scroll-to="#more-benefits">click here</span>
...
<div id="more-benefits">...</div>

Ionic 3 – Allowing back button action to be specified on a page-by-page basis

Ionic 3 is great as I’ve said in previous posts, however there are a few issues with its out-of-the box handling of back button on Android. Normally it just goes back to the previous page which is usually what you want, however this is not what you want when you are in a popup or modal on the page, or in some other cases you may want to capture the back event and do something else with it on a page-by-page basis. Fortunately it’s pretty straight forward to override this as I discovered.

Firstly, we want to override the back button action for the whole app. Open up app/app.component.ts:
and in the constructor:

import { App, Platform } from 'ionic-angular';
...
  constructor( ..., private app: App, private platform : Platform ) {
    platform.registerBackButtonAction(() => {
        let nav = app.getActiveNavs()[0];
        let activeView = nav.getActive();

        if(activeView != null){
          if(nav.canGoBack())
            nav.pop();
          else if (typeof activeView.instance.backButtonAction === 'function')
            activeView.instance.backButtonAction();
          else
            nav.parent.select(0); // goes to the first tab
        }
      });
  }

This basically defaults to going back if it is possible to, if not then it will take you to the first tab if it is a tab view. However if your active page has a backButtonAction() function, it will delegate to that.

So for example in a modal class you can add something like:

import { ViewController } from 'ionic-angular';
...
    constructor( private viewCtrl : ViewController ) {}

    backButtonAction() {
        this.viewCtrl.dismiss();
    }

which will dismiss the modal and go back to the page that called it, rather than the default action of simply going back a page.

Remove Windows Phone CSS in Ionic 2/3 to save space

Lets face it, no-one uses Window Phone any more. However, in Ionic 3 there are several skins which are built in to the CSS to emulate the display of the platform that the app is running on. These are ios (iOS, apple devices), md (Material Design – Android devices and Ionic default skin) and wp (window phone). As each of these is different and affects almost all widgets, they are each responsible for roughly 25% of the CSS size of ionic (which can be big – 250kb or so).

So, as no-one uses windows phone and we don’t want to have to test multiple platforms we can easily remove it from the CSS build to save time, space and complexity.

Firstly, copy the SASS config into our custom config directory; we will use this to override the default Ionic SASS config:

mkdir config
cp ./node_modules/@ionic/app-scripts/config/sass.config.js config

Then edit your package.json file and create or add a config dictionary to it like:

    "config": {
        "ionic_sass": "./config/sass.config.js"
    },

Finally, open up config/sass.config.js, find the excludeFiles section and add the following:

  excludeFiles: [
      /\.(wp).(scss)$/i
  ],

If you don’t want to match different platforms with different Ionic themes/skins (which while nice takes quite a bit of time to fully test), you can choose to use eg the Material Design skin only by doing something like:

  excludeFiles: [
      /\.(ios|wp).(scss)$/i
  ],

CKEditor Plugin Validation for Multiple Requirements

I’ve been working on a few CKEditor plugins recently for some projects. When building the elements of a dialog you create a data structure like:

elements: [{
    type: 'text',
    id: 'book',
    label: 'Book',
    validate: CKEDITOR.dialog.validate.notEmpty( "Book field cannot be empty" ),
}]

This is great, but what if you want multiple validation criteria for example it must be a number and not empty? There are a handful of hacks and examples of this online, but looking through the code I found an undocumented (and otherwise unused) function which enables you to combine multiple validation functions:

validate: CKEDITOR.dialog.validate.functions(
        CKEDITOR.dialog.validate.notEmpty(),
        CKEDITOR.dialog.validate.integer(),
        "(Start) chapter must be a number and is required",
    ),

You can pass as many validation functions into this as you want and it will combine them, but only allows you to specify one validation message unfortunately.

Ionic 3 Page Overlay

I’ve been doing a project recently in ionic 3 (WatchEm) and I must say I’m pretty impressed. I never tried using ionic 1 as it looked like quite a lot of overhead and it wasn’t certain whether it was going to turn into a popular platform, but ionic 3 seems good, stable and is developing well.

One thing we wanted to do was to provide a help screen the first time you access each page in the app as for example some google apps do. The aim was to present an overlay which provides some textual and visual pointers as to what you can do on the page, and some hints about it.

Fortunately it wasn’t hard to do; here is the code I wrote which lets you produce flexible help pages of the format:

<ion-header>
...
</ion-header>

<overlay>
    <h2>Player Buttons and Features</h2>

    <p>
        Pressing
        <ion-icon name="ios-skip-backward"></ion-icon>
        <ion-icon name="ios-skip-forward"></ion-icon>, or using your left/right keys you can move to the next key event in the game.
    </p>
</overlay>

<ion-content>
...
</ion-content>

We’re going to produce this as a component, so create components/overlay/overlay.ts like:

import { Input, Component, ElementRef, Renderer2, AfterViewInit } from '@angular/core';
import { Storage } from '@ionic/storage';
import { NavController } from 'ionic-angular';
    
@Component({
  selector: 'overlay',
  templateUrl: 'overlay.html'
})      
export class OverlayComponent implements AfterViewInit {
    private _force :boolean = false;
        
    @Input()
    set force(val) {
        this._force = val == '' ? true : !!val;
    }   
    
    constructor( private elementRef : ElementRef, private renderer : Renderer2, private _storage: Storage, public navCtrl: NavController ) {
    }

    get storage_key() {
        return `shown-overlay-${this.navCtrl.getActive().id}`;
    }

    ngAfterViewInit() {
        // Check local storage to see if we already displayed this...
        this._storage.get(this.storage_key).then( (val) => {
            if( !val || this._force )
                this.renderer.addClass( this.elementRef.nativeElement, 'shown' )
        });
    }

    hide_overlay() {
        this._storage.set(this.storage_key, 1);
        this.renderer.removeClass( this.elementRef.nativeElement, 'shown' );
    }
}

Pretty straight forwards – if the force= attribute is set on the <overlay> tag then it will always show it (useful for debugging). Otherwise if it is the first time the page has been opened it will show and then store in localStorage to say it shouldn’t be shown again.

Next, the HTML for the component in components/overlay/overlay.html:

<ion-grid full-height (click)="$event.stopPropagation(); hide_overlay()" ion-text color=white text-center>
    <ion-row full-height align-items-center>
        <ion-col col-md-8 push-md-2>
            <ng-content></ng-content>
        
            <button ion-button>Got it</button>
        </ion-col>
    </ion-row>
</ion-grid>

Obviously feel free to do what you want here with text/layout. We call stopPropagation() in order to prevent any stuff on the main page from receiving the click, especially if you have click-handlers further up the chain eg on the body element.

Finally a bit of styling in components/overlay/overlay.scss to make it look like an overlay and handle visibility changes correctly:

overlay {
    display: none;
        
    &.shown {
        position: fixed;
        display: block;
        padding: 40px 20px;
        top: 0;
        left: 0;
        bottom: 0;
        right: 0;
        background: rgba( 0, 0, 0, 0.7 );
        z-index: 9999;
        overflow-y: auto;
        overflow-x: hidden;
    }
}

Note that the overlay must be placed outside of any <ion-content tags as they provide for automatic scrolling of their content etc which is not wanted as the overlay itself needs to scroll.

Successfully downloading big files from Dropbox via Linux command-line

Recently, someone was trying to send me a 20Gb virtual machine image over dropbox. I tried a couple of times to download using chrome, however it got to 6-8Gb and then came up with a connection error. Clicking on the resume button failed and then removed the file (!). Very strange as I didn’t have any connection issues, but perhaps a route changed somewhere. I saw a number of dropbox users complaining about this on the internet. Obviously there are other approaches such as adding to your own dropbox account and using their local program to do the sync, however because I’m just on a standard free account I couldn’t add in such a large file.

Because I was using btrfs and snapper I still had a version of the half-completed download around, and so I tried seeing if standard linux tools would be able to continue the download where it left off. It turns out that simply using wget -c enables you to resume the download (it dropped a couple of times during the download but just restarting it with the same command let the whole file download just fine. So, to download a large dropbox file even if your internet connection is a bit flakey, simply go to the dropbox download link and then paste it into the terminal (may require the ?dl=1 parameter after it) like:

wget -c .https://dl.dropbox.com/...?dl=1

Apache configuration for WkWebView API service (CORS)

Switching from UIWebView to WKWebView is great, but as it performs stricter CORS checks than standard Cordova/Phonegap it can seem at first that remote API calls are broken in your app.

Basically, before WKWebView does any AJAX request, it first sends a HTTP OPTIONS query and looks at the Access-Control-* headers that are returned to determine if it is able to access the service. Most browsers can be made to allow all AJAX requests via a simple “Access-Control-Allow-Origin: *” header, however WKWebView is more picky. It requires that you expose which methods (GET, POST, etc); and which headers are allowed (eg if you are using JSON AJAX requests you probably need to use a “Content-Type: application/json” header in your main request).

Rather than having to update your API service, you can work around this in a general way using the following Apache config:

    # Required configuration for iOS WkWEBVIEW

    # Allow any location to access this service
    Header always set Access-Control-Allow-Origin "*"

    # Allow the following headers in requests (X-Auth is a custom header, also allow Content-Type to be specified)
    Header always set Access-Control-Allow-Headers "X-Auth, content-type, origin"
    Header always set Access-Control-Expose-Headers "X-Auth"

    # Allow the following methods to be used
    Header always set Access-Control-Allow-Methods "GET, POST, OPTIONS"

    # WkWebView sends OPTIONS requests to get CORS details. Don't tie up the API service with them;
    # just answer them via apache itself
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} =OPTIONS
    RewriteRule .* - [R=204,END]

Note the last line answers any HTTP OPTIONS request with blank content and returns it straight away. Most API services would cause a lot of CPU processing just to handle a single request whether it is a true request or an OPTIONS query, so we just answer this straight from Apache without bothering to send it through to the API. The R=204 is a trick to specify that we don’t return any content (HTTP 204 code means “Success, but no content”). Otherwise if we used something like R=200 it would return a page talking about internal server error, but with a 200 response which is more bandwidth, more processing and more confusing for any users.

Using Ionic’s WKWebView Engine with standard cordova/phonegap projects

The guys at ionic have created a branch of phonegap’s WKWebView Engine (for iOS) with a number of improvements such as local AJAX request ability. This is great for ionic projects, however I wanted to see if it could be used in a standard phonegap project for which the standard WKWebView plugin wouldn’t work.

For those who don’t know, older versions of iOS had an implementation of webview which was called UIWebView. This had a number of bugs and performance issues, however because cordova/phonegap strive to maintain backwards-compatibility, it is still the default within phonegap projects. WKWebView is faster, more stable and has newer features such as IndexedDB, however it has some additional security restrictions (especially with CORS and local file loading) meaning it is not simply a drop-in replacement for UIWebView.

Installation is simple, more details can be found on the github project page, however basically just run:

cordova plugin add https://github.com/ionic-team/cordova-plugin-wkwebview-engine.git --save

and it will be included in your iOS project. Make sure that your app works (and detects it’s being run from within cordova) when the window.location is set to 'http://localhost:8080/' rather than the usual file:// path. Then, add the necessary configuration into config.xml:

<platform name="ios">
    <access origin="http://localhost:8080/*" />
    <allow-navigation href="http://localhost:8080/*" />
    <feature name="CDVWKWebViewEngine">
        <param name="ios-package" value="CDVWKWebViewEngine" />
    </feature>
    <preference name="CordovaWebViewEngine" value="CDVWKWebViewEngine" />
</platform>

However, even if you do this, when you open the test app you’ll find that you are for some strange reason unable to scroll! This is because the ionic framework doesn’t use body scrolling, it has subelements with overflow: auto set on them. However for most non-ionic apps you probably want body scrolling. Simply change add the following into the ios platform configuration section in config.xml:

    <preference name="ScrollEnabled" value="true" />

If you are accessing remote API services, you’ll also need to modify some CORS settings on your server. In my post tomorrow I’ll show you how to do this easily without increasing the load on your API service.

Tracking down Lua JSON decoding issues

I’ve recently been doing quite a bit of Lua scripting for a client wanting some PowerDNS customizations. I’ve actually grown to quite like Lua, even though it’s very simple and quick, you can do some very complex programming with it reasonably straight forwardly. I think it could perhaps be compared to a stripped-down version of Perl which is also a language that I very much like because of its incredible flexibility.

Anyway, as part of this work are wanting to look up incoming IP addresses in a table of non-overlapping IP address ranges. For high performance I recommended LMDB as I’ve used it extensively before and I know that for its quirks and tendency to crash if you mishandle any aspect of its API, it is very high performance, low over head, scales very well to multiple cores, and can do pretty much anything you ask of it.

So basically the problem was “how do we store an IP range as an indexed key in LMDB” (which is just a key-value database where all keys are b-tree indexed). In the future we may want to support IPv6, and we may also want to support IP ranges which cannot be expressed in subnet-mask representation. The solution I came up with is to store the first IP in raw binary format (ie 4 bytes for IPv4, or 16 bytes for IPv6) as the key, and then as part of the value we store the end IP address. In order to see if a given IP is within a subnet, you look up you open a cursor on the table, seek to the position of the IP you are trying. If you get a direct hit, then obviously it has found the first IP in the subnet and so you know it is valid. If it does not get a direct hit you seek back to the previous entry (this is a great feature of LMDB and is found in surprisingly few indexed key-value data store APIs, even though it should be very simple to implement). You then take the value of that, get the end IP of the range and check to see if the requested IP is within the start and end of the range.

Because we wanted a very flexible and easily extensible data storage format for the values in this table we decided to encode it all as JSON. Lua has a number of JSON decoders and lua-cjson seemed pretty quick and easy, and was also available as a pre-built ubuntu package so we went with that. As we were storing the key’s IP address in raw binary notation, we figured it would make the code-path simplest if we stored the end IP address in the same manner. So, we did this, wrote a test suite with some non-public IPv4 addresses (10.xxx and 127.xxx) and verified that it was all working correctly, and then launched the code.

A few days later we started getting some complaints from customers that some IP addresses in their network ranges were not being identified correctly. But when we added the exact same details into the test suite with our private IP ranges, it clearly worked fine.

Finally I started trying to use the exact IP addresses that the customers were reporting issues with in the test scripts and discovered that there was actually a problem. Basically, whenever a component of the address was greater than 127 and the code did not go down the direct hit code path (ie the address was part of a subnet larger than a /32 and not the first entry) the decoded end IP address would be incorrect. Very strange! So, our test code which was using ranges like 127.0.0.1-127.1.2.3 worked fine, but an IP range like 1.0.0.0-1.0.129.0 would fail!

Looking more closely at the cjson Lua documentation I saw the line “cjson.decode will deserialise any UTF-8 JSON string into a Lua value or table”. And reading through the C code I saw that the routines were hard-coded to treat any JSON escaped \uXXXX value that was greater than 127 as part of a UTF-8 encoded character. This is because Lua uses the platform’s underlying char[] to store strings which means usually each character in a string can only be 8-bits, meaning that in order to store wider characters the bytes need to be encoded into a single character which is what UTF-8 is for. With our encoding we knew that all parts of the string would fit into 8-bits, but there was no way to tell the decoder this. Because cjson is aiming to be a fast module, this is all hard-coded and there is no way that I could see to easily work around this utf-8 decoding. We tried some other Lua JSON modules but they either had the same problem, or were orders of magnitude slower than cjson.

Eventually a colleague suggested just hex-encoding the end IP address prior to including it in the JSON data which was the simplest solution we could find. It should also reduce the storage required for an IP address as assuming 50% of the characters are usually encoded with a \uXXXX escape sequence in JSON, an average IPv4 address would take 14 bytes in the database, whereas with hex this would be a fixed 8 bytes per IPv4 address.

If the encoding program had been using perl we could probably have used some of the features of the JSON::XS module (specifically, the utf8 flag) to write characters directly as bytes into the string, which although is perhaps not technically valid JSON, from my reading of the Lua module should have bypassed the UTF-8 encoding of escaped values. However we wern’t using perl in our encoding routines so this wasn’t possible.