All posts by Mark

I'm a full-stack Linux consultant from the UK specializing in high performance systems, DNS and databases. I have also written and lead teams producing a number of web/mobile apps. I'm fluent in English and Turkish.

Fan/Light remote control teardown and documentation

I recently bought a few ceiling fan controllers from AliExpress, I’m guessing there are a lot more out there with a similar electrical design so I thought perhaps my notes on tearing these down would be useful for someone as I didn’t find this elsewhere on the internet. As I was taking them apart to replace the wireless section with an ESP8266 for WiFi remote control I thought I’d document the design of these devices a bit.

Firstly, the main circuits run at 12v – the relays are triggered by that voltage, and the remote control uses a small 12v battery too. However the radio is only rated for 5v, so I think there must be a 5v circuit there as well probably obtained via resistors from the 12v circuit. The basic block layout is that the radio receiver connects to an unmarked chip (in some models this connects to an AT24C02N which presumably holds the last state of the relays). The unmarked chip has 5 output tracks running from it to an ULN2003A which boosts the signal to 12v to trigger the relays. 4 tracks go to the 4 relays (one for the light, and 3 for fan control which has live, 3uF and 2.5uF (2uF in some models) capacitors to regulate the speed). The 5th track goes to a buzzer which is triggered when it gets powered on or receives a command from the remote controller. Unfortunately I can’t remember the order that fan relays were triggered as I wrote it down lost it, but it was something along the lines of:

high live
medium 3uF + 2.5uF
low 3uF

The radio receiver in the ceiling fan based on a SYN470R chip and the transmitter uses an EV1527 to encode the data to send to the fan. It sends 24-bits – a 20 bit id and a 4 bit code. Here are the hex codes for the commands as discovered by my patches to allow this device to work on a C.H.I.P board:

1

3

a

Command Hex code
low 8
medium 2
high 6
stop c
toggle light on/off 5
timer 1h 4
timer 2h
timer 4h
timer 8h

Uploading videos to YouTube with perl

As part of a perl project recently I needed to generate a load of videos (using the excellent MLT framework) and upload them to YouTube. As the bulk of the project was already in perl, I needed to write a YouTube upload module for it. Here’s a rough overview of how to authenticate and upload a video (chunked rather than loading it all into memory at once) with perl. Note as well it was quite a struggle to get long-term auth tokens from the Google services – one slight mistake with a parameter and they only give temporary grants which last for an hour or two rather than indefinitely.

package Youtube::Upload;
use Moo;

use LWP::Authen::OAuth2;
use Path::Tiny 'path';
use URI::QueryParam;
use JSON::XS qw< encode_json decode_json >;
use HTTP::Message;

# API described at https://developers.google.com/youtube/v3/docs/videos/update

has auth_id => is => 'ro', required => 1;
has auth_secret => is => 'ro', required => 1;
has redirect_uri => is => 'ro', required => 1;

# If you havn't used this for a while, blank these and re-run and you'll
# probably need to do some auth against google.
has auth_code => is => 'ro';
has auth_token => is => 'ro';

has auth => is => 'lazy', builder => \&_google_auth;

sub upload {
    my ($self, $details, $youtube_code, $video_file) = @_;

    die "No id to update, but also nothing to upload" if !$youtube_code && !$video_file;

    my %body = %$details;

    # Allow all embedding
    $body{status}{embeddable} = JSON::XS::true;

    my $magic_split = 'BlA123H123BLAH'; # A unique string...

    my $is_new = !defined $youtube_code;
    my ($content, %headers, $uri);
    if( !$is_new ) {
        $body{id} = $youtube_code;

        $content = encode_json(\%body);
        $headers{'Content-Type'} = 'application/json';
        $uri = URI->new( 'https://www.googleapis.com/youtube/v3/videos' );
    } else {
        my $msg = HTTP::Message->new([
            'Content-Type' => 'multipart/related',
        ]);

        $msg->add_part(
            HTTP::Message->new([
                'Content-Type' => 'application/json',
            ], encode_json(\%body) )
        );

        my $video_msg = 
            HTTP::Message->new(
                [
                    'Content-Type' => 'video/*',
                ],
                $magic_split,
            );
        $msg->add_part( $video_msg );
            
        $content = $msg->as_string;
        (my $head, $content) = split /\r?\n\r?\n/, $content, 2;
        my ($k, $v) = split /:\s*/, $head, 2;
        $headers{$k} = $v;
        $uri = URI->new( 'https://www.googleapis.com/upload/youtube/v3/videos' );
    }

    delete $body{id};
    $uri->query_form_hash({ 
        part => join(',', keys %body), 
    });

    my $res;
    if( $is_new ) {
        my @content = split /\Q$magic_split/, $content;
        die "Magic split failed" if @content != 2;

        my $content_fh = path($video_file)->openr;
        my $request = HTTP::Request->new( 'POST', $uri, HTTP::Headers->new( %headers ), sub {
            #warn "chunk uploaded";
            return '' if !@content;

            if( @content > 1 ) {
                return shift @content;
            } else {
                my $read = read $content_fh, my $data, 1_000_000;
                if( !$read ) {
                    return shift @content;
                }
                return $data;
            }
        } );
        $res = $self->auth->request( $request );
    } else {
        $res = $self->auth->put( $uri,
            %headers,
            Content => $content
        );
    }

    my $cont = $res->decoded_content;
    my $ret;
    if( !$res->is_success ) {
        if($res->code != 403) {   # not our video
            die "Response not success: $cont for " . ( $youtube_code || '' );
        }
    } else {
        $ret = decode_json $cont;
    }

    return ref($ret) ? $ret : {};
}

sub _google_auth {
    my ($self) = @_;

    my $auth = LWP::Authen::OAuth2->new(
        service_provider => 'Google',
        redirect_uri => $self->redirect_uri,
        client_type => "web server",

        client_id      => $self->auth_id,
        client_secret  => $self->auth_secret,

        save_tokens => sub {
            say "Save token string: $_[0]" if !$self->auth_token;
        },

        token_string => $self->auth_token,
    );

    # For debug:
    #$auth->user_agent->add_handler("request_send",  sub { shift->dump(maxlength => 10_000); return });
    #$auth->user_agent->add_handler("response_done", sub { shift->dump(maxlength => 10_000); return });

    if( !$self->auth_code ) {
        say $auth->authorization_url(
            scope=> 'https://www.googleapis.com/auth/youtube https://www.googleapis.com/auth/youtube.upload',
            # Need these two to get a refresh token
            approval_prompt => 'force',
            access_type => 'offline',
        );
        exit;
    }

    $auth->request_tokens( code => $self->auth_code ) if !$self->auth_token;

    return $auth;
}

As per this bug report you need to hack LWP::Authen::OAuth2::AccessToken::Bearer to enable the chunked uploads to work, otherwise it throws an error.

The auth_id and auth_secret parameters are given by the google code console when you sign up for YouTube API access, and the redirect_uri should be where the web app would redirect to after displaying the Google oauth permission grant screen. The first time you run the script, you’ll need to save the auth_code / auth_token parts and use them in future when calling. Even though it should grant a new auth token each time the script is run you still need to present the original one from the look of things.

Element scrolling within Angular (1) pages

Back to Angular 1 for today’s post as I’ve been doing some work on an older project today. As we all know, in the old days of the web, you could scroll to items within the page by using an <a> element with a hash reference, like:

<a href="#thing-to-scroll-to">scroll here</a>

<h2 id="thing-to-scroll-to">My title</h2>

With the advent of single-page sites and multiple ajax pages under them however, the hash section of a query parameter is increasingly unable to be used. On Angular 1 with angular-route this becomes impossible.

So what if for example you want to have some references at the top of a page within an Angular application which will scroll the user down to certain sections below, just like the hash references used to do? There are a number of suggestions on the internet for this but they are not very reusable, so I decided to create a simple directive that does this:

app.directive('clickScrollTo', function () {
    return {
        restrict: 'A',
        scope: {
            clickScrollTo: '@'
        },
        link: function (scope, el, attr) {
            el.click(function(event) {
                var main_page = $('.main-page');
                event.preventDefault();
                main_page
                    .stop()
                    .animate(
                        { scrollTop: main_page.scrollTop() + $(scope.clickScrollTo).offset().top },
                        500,
                        'linear'
                    );
            });
        }
    }
});

This is slightly complicated by the fact that I am using an element with class="main-page" with overflow: auto set to contain the scrollable section of the app. If you don’t have a layout like this just replace the $('.main-page') part with $('body').

Then you can easily create elements like:

<span class="a" click-scroll-to="#more-benefits">click here</span>
...
<div id="more-benefits">...</div>

Ionic 3 – Allowing back button action to be specified on a page-by-page basis

Ionic 3 is great as I’ve said in previous posts, however there are a few issues with its out-of-the box handling of back button on Android. Normally it just goes back to the previous page which is usually what you want, however this is not what you want when you are in a popup or modal on the page, or in some other cases you may want to capture the back event and do something else with it on a page-by-page basis. Fortunately it’s pretty straight forward to override this as I discovered.

Firstly, we want to override the back button action for the whole app. Open up app/app.component.ts:
and in the constructor:

import { App, Platform } from 'ionic-angular';
...
  constructor( ..., private app: App, private platform : Platform ) {
    platform.registerBackButtonAction(() => {
        let nav = app.getActiveNavs()[0];
        let activeView = nav.getActive();

        if(activeView != null){
          if(nav.canGoBack())
            nav.pop();
          else if (typeof activeView.instance.backButtonAction === 'function')
            activeView.instance.backButtonAction();
          else
            nav.parent.select(0); // goes to the first tab
        }
      });
  }

This basically defaults to going back if it is possible to, if not then it will take you to the first tab if it is a tab view. However if your active page has a backButtonAction() function, it will delegate to that.

So for example in a modal class you can add something like:

import { ViewController } from 'ionic-angular';
...
    constructor( private viewCtrl : ViewController ) {}

    backButtonAction() {
        this.viewCtrl.dismiss();
    }

which will dismiss the modal and go back to the page that called it, rather than the default action of simply going back a page.

Remove Windows Phone CSS in Ionic 2/3 to save space

Lets face it, no-one uses Window Phone any more. However, in Ionic 3 there are several skins which are built in to the CSS to emulate the display of the platform that the app is running on. These are ios (iOS, apple devices), md (Material Design – Android devices and Ionic default skin) and wp (window phone). As each of these is different and affects almost all widgets, they are each responsible for roughly 25% of the CSS size of ionic (which can be big – 250kb or so).

So, as no-one uses windows phone and we don’t want to have to test multiple platforms we can easily remove it from the CSS build to save time, space and complexity.

Firstly, copy the SASS config into our custom config directory; we will use this to override the default Ionic SASS config:

mkdir config
cp ./node_modules/@ionic/app-scripts/config/sass.config.js config

Then edit your package.json file and create or add a config dictionary to it like:

    "config": {
        "ionic_sass": "./config/sass.config.js"
    },

Finally, open up config/sass.config.js, find the excludeFiles section and add the following:

  excludeFiles: [
      /\.(wp).(scss)$/i
  ],

If you don’t want to match different platforms with different Ionic themes/skins (which while nice takes quite a bit of time to fully test), you can choose to use eg the Material Design skin only by doing something like:

  excludeFiles: [
      /\.(ios|wp).(scss)$/i
  ],

CKEditor Plugin Validation for Multiple Requirements

I’ve been working on a few CKEditor plugins recently for some projects. When building the elements of a dialog you create a data structure like:

elements: [{
    type: 'text',
    id: 'book',
    label: 'Book',
    validate: CKEDITOR.dialog.validate.notEmpty( "Book field cannot be empty" ),
}]

This is great, but what if you want multiple validation criteria for example it must be a number and not empty? There are a handful of hacks and examples of this online, but looking through the code I found an undocumented (and otherwise unused) function which enables you to combine multiple validation functions:

validate: CKEDITOR.dialog.validate.functions(
        CKEDITOR.dialog.validate.notEmpty(),
        CKEDITOR.dialog.validate.integer(),
        "(Start) chapter must be a number and is required",
    ),

You can pass as many validation functions into this as you want and it will combine them, but only allows you to specify one validation message unfortunately.

Ionic 3 Page Overlay

I’ve been doing a project recently in ionic 3 (WatchEm) and I must say I’m pretty impressed. I never tried using ionic 1 as it looked like quite a lot of overhead and it wasn’t certain whether it was going to turn into a popular platform, but ionic 3 seems good, stable and is developing well.

One thing we wanted to do was to provide a help screen the first time you access each page in the app as for example some google apps do. The aim was to present an overlay which provides some textual and visual pointers as to what you can do on the page, and some hints about it.

Fortunately it wasn’t hard to do; here is the code I wrote which lets you produce flexible help pages of the format:

<ion-header>
...
</ion-header>

<overlay>
    <h2>Player Buttons and Features</h2>

    <p>
        Pressing
        <ion-icon name="ios-skip-backward"></ion-icon>
        <ion-icon name="ios-skip-forward"></ion-icon>, or using your left/right keys you can move to the next key event in the game.
    </p>
</overlay>

<ion-content>
...
</ion-content>

We’re going to produce this as a component, so create components/overlay/overlay.ts like:

import { Input, Component, ElementRef, Renderer2, AfterViewInit } from '@angular/core';
import { Storage } from '@ionic/storage';
import { NavController } from 'ionic-angular';
    
@Component({
  selector: 'overlay',
  templateUrl: 'overlay.html'
})      
export class OverlayComponent implements AfterViewInit {
    private _force :boolean = false;
        
    @Input()
    set force(val) {
        this._force = val == '' ? true : !!val;
    }   
    
    constructor( private elementRef : ElementRef, private renderer : Renderer2, private _storage: Storage, public navCtrl: NavController ) {
    }

    get storage_key() {
        return `shown-overlay-${this.navCtrl.getActive().id}`;
    }

    ngAfterViewInit() {
        // Check local storage to see if we already displayed this...
        this._storage.get(this.storage_key).then( (val) => {
            if( !val || this._force )
                this.renderer.addClass( this.elementRef.nativeElement, 'shown' )
        });
    }

    hide_overlay() {
        this._storage.set(this.storage_key, 1);
        this.renderer.removeClass( this.elementRef.nativeElement, 'shown' );
    }
}

Pretty straight forwards – if the force= attribute is set on the <overlay> tag then it will always show it (useful for debugging). Otherwise if it is the first time the page has been opened it will show and then store in localStorage to say it shouldn’t be shown again.

Next, the HTML for the component in components/overlay/overlay.html:

<ion-grid full-height (click)="$event.stopPropagation(); hide_overlay()" ion-text color=white text-center>
    <ion-row full-height align-items-center>
        <ion-col col-md-8 push-md-2>
            <ng-content></ng-content>
        
            <button ion-button>Got it</button>
        </ion-col>
    </ion-row>
</ion-grid>

Obviously feel free to do what you want here with text/layout. We call stopPropagation() in order to prevent any stuff on the main page from receiving the click, especially if you have click-handlers further up the chain eg on the body element.

Finally a bit of styling in components/overlay/overlay.scss to make it look like an overlay and handle visibility changes correctly:

overlay {
    display: none;
        
    &.shown {
        position: fixed;
        display: block;
        padding: 40px 20px;
        top: 0;
        left: 0;
        bottom: 0;
        right: 0;
        background: rgba( 0, 0, 0, 0.7 );
        z-index: 9999;
        overflow-y: auto;
        overflow-x: hidden;
    }
}

Note that the overlay must be placed outside of any <ion-content tags as they provide for automatic scrolling of their content etc which is not wanted as the overlay itself needs to scroll.

Successfully downloading big files from Dropbox via Linux command-line

Recently, someone was trying to send me a 20Gb virtual machine image over dropbox. I tried a couple of times to download using chrome, however it got to 6-8Gb and then came up with a connection error. Clicking on the resume button failed and then removed the file (!). Very strange as I didn’t have any connection issues, but perhaps a route changed somewhere. I saw a number of dropbox users complaining about this on the internet. Obviously there are other approaches such as adding to your own dropbox account and using their local program to do the sync, however because I’m just on a standard free account I couldn’t add in such a large file.

Because I was using btrfs and snapper I still had a version of the half-completed download around, and so I tried seeing if standard linux tools would be able to continue the download where it left off. It turns out that simply using wget -c enables you to resume the download (it dropped a couple of times during the download but just restarting it with the same command let the whole file download just fine. So, to download a large dropbox file even if your internet connection is a bit flakey, simply go to the dropbox download link and then paste it into the terminal (may require the ?dl=1 parameter after it) like:

wget -c .https://dl.dropbox.com/...?dl=1

Apache configuration for WkWebView API service (CORS)

Switching from UIWebView to WKWebView is great, but as it performs stricter CORS checks than standard Cordova/Phonegap it can seem at first that remote API calls are broken in your app.

Basically, before WKWebView does any AJAX request, it first sends a HTTP OPTIONS query and looks at the Access-Control-* headers that are returned to determine if it is able to access the service. Most browsers can be made to allow all AJAX requests via a simple “Access-Control-Allow-Origin: *” header, however WKWebView is more picky. It requires that you expose which methods (GET, POST, etc); and which headers are allowed (eg if you are using JSON AJAX requests you probably need to use a “Content-Type: application/json” header in your main request).

Rather than having to update your API service, you can work around this in a general way using the following Apache config:

    # Required configuration for iOS WkWEBVIEW

    # Allow any location to access this service
    Header always set Access-Control-Allow-Origin "*"

    # Allow the following headers in requests (X-Auth is a custom header, also allow Content-Type to be specified)
    Header always set Access-Control-Allow-Headers "X-Auth, content-type, origin"
    Header always set Access-Control-Expose-Headers "X-Auth"

    # Allow the following methods to be used
    Header always set Access-Control-Allow-Methods "GET, POST, OPTIONS"

    # WkWebView sends OPTIONS requests to get CORS details. Don't tie up the API service with them;
    # just answer them via apache itself
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} =OPTIONS
    RewriteRule .* - [R=204,END]

Note the last line answers any HTTP OPTIONS request with blank content and returns it straight away. Most API services would cause a lot of CPU processing just to handle a single request whether it is a true request or an OPTIONS query, so we just answer this straight from Apache without bothering to send it through to the API. The R=204 is a trick to specify that we don’t return any content (HTTP 204 code means “Success, but no content”). Otherwise if we used something like R=200 it would return a page talking about internal server error, but with a 200 response which is more bandwidth, more processing and more confusing for any users.

Using Ionic’s WKWebView Engine with standard cordova/phonegap projects

The guys at ionic have created a branch of phonegap’s WKWebView Engine (for iOS) with a number of improvements such as local AJAX request ability. This is great for ionic projects, however I wanted to see if it could be used in a standard phonegap project for which the standard WKWebView plugin wouldn’t work.

For those who don’t know, older versions of iOS had an implementation of webview which was called UIWebView. This had a number of bugs and performance issues, however because cordova/phonegap strive to maintain backwards-compatibility, it is still the default within phonegap projects. WKWebView is faster, more stable and has newer features such as IndexedDB, however it has some additional security restrictions (especially with CORS and local file loading) meaning it is not simply a drop-in replacement for UIWebView.

Installation is simple, more details can be found on the github project page, however basically just run:

cordova plugin add https://github.com/ionic-team/cordova-plugin-wkwebview-engine.git --save

and it will be included in your iOS project. Make sure that your app works (and detects it’s being run from within cordova) when the window.location is set to 'http://localhost:8080/' rather than the usual file:// path. Then, add the necessary configuration into config.xml:

<platform name="ios">
    <access origin="http://localhost:8080/*" />
    <allow-navigation href="http://localhost:8080/*" />
    <feature name="CDVWKWebViewEngine">
        <param name="ios-package" value="CDVWKWebViewEngine" />
    </feature>
    <preference name="CordovaWebViewEngine" value="CDVWKWebViewEngine" />
</platform>

However, even if you do this, when you open the test app you’ll find that you are for some strange reason unable to scroll! This is because the ionic framework doesn’t use body scrolling, it has subelements with overflow: auto set on them. However for most non-ionic apps you probably want body scrolling. Simply change add the following into the ios platform configuration section in config.xml:

    <preference name="ScrollEnabled" value="true" />

If you are accessing remote API services, you’ll also need to modify some CORS settings on your server. In my post tomorrow I’ll show you how to do this easily without increasing the load on your API service.