Category Archives: Web Apps

The hardest task to do well in every frontend/backend system – paging

What is the hardest code to write efficiently in pretty much every frontend and backend system? I think it is paging.

Nearly every system I’ve worked on as full-stack developer has had some sort of reporting whether it is searching and filtering or news-feed style item display, all of these depend on paging. Whilst simple paging is trivial to implement, making it work quickly at scale is a very tricky challenge.

A simple approach (based on SQL, but could be with any backend data store) would be something like:

items = SELECT * FROM <table> WHERE <query> ORDER BY <order> LIMIT <rows>,<offset>;
count = SELECT COUNT(*) FROM <table> WHERE <query>;

It looks simple and it works in small-scale development systems. But there a massive number of issues and inefficiencies with this approach:

  • Whilst you need to redo the items query for every page, the count query only needs to be done once for each <query>. Counting is a very expensive operation as it essentially has to fetch all the rows, unless there are some very well optimized keys on your table. This especially the case if you are running over a table larger than a few megabytes on a mobile device, so you need to cache this as much as possible.
  • Often times as the <order> logic has gotten more and more complex I’ve found myself in a situation where some rows are equal in ordering and so they can move around as you switch between pages or redo the query. Always ensure that the last item of ordering is something which is well ordered to avoid this, for example an integer primary key
  • What happens with a LIMIT/OFFSET type query if you get a row inserted before the OFFSET? All the rows shift forwards by one so when you scroll down on your infinite feed you get a duplicate, or the user goes to the next page and sees a repeat of the last entry of the previous page. What happens when a row prior to the OFFSET is deleted? A row between the pages is not seen by the user. In some databases it may be possible to use a long-lived cursor but often in stateless backends for webapps this is not possible. Wherever possible you should try to get a column on the <order> which is simply increasing or decreasing and redo the previous query with use a > against that to ensure you fetch all the rows from the previous point the user saw regardless of what changed above it. This is easy when sorting by primary key, difficult or even impossible otherwise.
  • How do you handle a laggy backend when the user presses the next button multiple times? Does it re-fire the request for the next page multiple times to the backend (consuming unnecessary resources), ignore any presses after the first (better), or does it proceed forward multiple pages, potentially running past the last page and onto blank pages? I’d usually opt for the second approach but limiting so that it always stops on the last page, but it may vary depending on usage case (would users typically want to skip multiple pages quickly, or always want to page through one-at-a-time), backend latency (ie is the data stored locally or remotely, what is the connectivity like)
  • Following on from the above, if you are firing multiple requests for pages, or for infinite scrolling how do you cope with the fact that responses can come back out-of-order? Many times I see it would simply display the content from the last page returned, but this is open to a number of bugs if you are talking about remote services over wireless
  • How do you efficiently abort in-fly calls and database queries if the user changes page – on a large system this could be a significant performance gain

There are quite a large number of optimizations that can be done with paging code, whether this is with a remote backend or locally, for example:

  • Prioritize getting the first page of data to display, say “Showing results 1-50 of …” while the count is loading
  • Track historical data on how long it takes to generate counts or paging, and especially if running single-threaded (eg on a mobile device), delay the count until the main queries have completed and there is some idle time
  • Is it possible to parallelize the two querys against the database so that the query takes half the real-time for the user? Can you use non-blocking IO to open two database connections, one for each query and return a response when both have completed? Can you run 2 separate queries one for count and one for items and update your pager and results separately? For a few rows in a test system this won’t make any difference, but over millions of rows, especially if you need an accurate count, this can save seconds of time
  • Do you actually need a totally accurate count, or can you approximate it, does your database backend support something like this for queries? This can save significant time and resources
  • Have you set up good indexes and verified that they are correctly being used at the scale that you will have in your live environment? Do you optimize well for the common use case?
  • You don’t always need to bother running a count query – if the items returned are less than the items requested you know that you have finished paging and you can automatically count the total from that. This is true whether on the first on the 100th page.
  • Fetch the number of rows that you want to display plus one, so that you know if there is a next page or not. If the number of items is less than you requested you know there won’t be a next page no matter what the total actually is
  • If running on mobile devices or html, limit the number of items that are displayed on an infinite scroller as this can be very memory intensive. Perhaps allow infinite scrolling over 500-1000 rows and then put a next button at the bottom of the page. Or you can do some trickery to replace the top items with a large blank box but this can be very complex and difficult to do well
  • Don’t display a spinner when infinite scrolling (you are starting to load more items before the user hits the bottom of the list right?) Only display the spinner after a short time period or when the user hits the very bottom of the list to give visual feedback that there are more items loading to display

If this looks like a lot to think about, I don’t disagree. Simple, buggy and inefficient paging is easy to do, but fully-optimized, highly responsive complex paging is very tricky to do well. But if you want to scale your app you need to focus on this.

Styled cross-platform number input (in Angular but applicable to any HTML/CSS app)

Native elements in HTML 5 such as the number input are great, but unfortunately our designers often want them to render the same between different browsers and operating systems. For example on linux/chrome the number input has an up/down spinner on the right hand side which is always visible. On Mac the spinner is only visible on mouse over, on firefox it is rendered differently and on mobile the spinner is not there at all usually. In this case, my designer wanted number input with up/down buttons always available. This was for an order quantity input, so it should be integers from 1 upwards. With bootstrap you can easily add buttons etc to the left or right of an input, however I couldn’t see an easy way in ‘native’ bootstrap to have two buttons stacked as one of the addons, so I created my own html/less. This is for Angular 1, bootstrap 3 and fontawesome but it should be very easy to change for different platforms.

Here’s the HTML

<div class="number-input-group">
    <input class="noscroll" type="number" min=1 max=100 step=1 ng-model="extra.quantity"/>
    <div class="buttons">
        <div ng-click="extra.quantity = extra.quantity + 1"><i class="fa fa-caret-up"></i></div>
        <div ng-click="extra.quantity = extra.quantity > 1 ? extra.quantity - 1 : extra.quantity"><i class="fa fa-caret-down"></i></div>
    </div>
</div>

And the LESS:

.number-input-group {
    display: table; 
    width: 100%;    // fill container - remove if you want it as effectively an inline-block
    position: relative;
    border-collapse: separate;
    border-spacing: 0px;    
                                
    > input[type=number] {  
        display: table-cell;
        width: 100%; // keep biggest
        -moz-appearance:textfield;
        border-right: none;         
        &::-webkit-inner-spin-button, &::-webkit-outer-spin-button {
            -webkit-appearance: none;   
            margin: 0;                  
        }                           
    }                           
    > .buttons {            
        display: table-cell;
        width: 1%;      // shrink to smallest size
        vertical-align: top;
        border: @input-transparent-border-width solid @bespoke-light-black;
        //color: @bespoke-light-black;
        > div { 
            @number-input-group-arrow-box-size: (@input-padding-top * 2 + @input-line-height - @input-transparent-border-width - 1) / 2;
            line-height: @number-input-group-arrow-box-size * 0.8;  // make a bit smaller because ff and chrome mobile add a few px for some reason
            font-size: @number-input-group-arrow-box-size * 1;
            padding: 0 7px;
                            
            &:hover {
                background-color: lighten(@background-color, 15%);
            }

            &:last-child {
                border-top: @input-transparent-border-width solid @bespoke-light-black;
            }
        }
    }
}

Transparently serving WebP images from Apache

I’ve recently been working on a website where we are creating a tool to customize a product. We have various renders from the designers with lots of transparency and then combine these together on the frontend to produce the customized render. As a result of needing transparency we can’t use the jpeg format so we need to use PNG format, however as this is lossless it means the image sizes tend to be very big. Fortunately the WebP format can compress transparent images including the transparency layer (but this is not set by default). Running the WebP converter with light compression over our PNG assets for this projects produced a set of WebP’s which were in total only 25% of the size of the PNG assets and still a high quality. This means much faster loading for the site, especially when displaying multiple renders of the customized product and its 5-10 layers per render.

However, WebP support is only available in about 70% of the browsers today. Rather than trying to test for it on the client side, it would be great to just keep the browser-side code the same but serve different assets depending on whether the browser supports it or not.

I found a good start for apache support for transparent loading of WebPs on github, however there were a few bugs in the script. Here is the final version that I used – you need to put it under a <VirtualHost> section.

AddType image/webp .webp
<ifmodule mod_rewrite.c>
      # Does browser support WebP? 
      RewriteCond %{HTTP_ACCEPT} \bimage/webp\b

      # Capture image name
      RewriteCond %{REQUEST_URI}  (.*)(\.(jpe?g|png|gif))$

      # if you don't have all jpg/png images available
      # as webp then you want to uncomment the next line
      # so apache first checks if there is a webp file
      # otherwise leave it disabled as it removes the
      # need to query the disk
      RewriteCond %{DOCUMENT_ROOT}%1.webp -f

      # Route to WebP image 
      RewriteRule .* %1.webp [L,T=image/webp]
</ifmodule>

And here is a script to convert all png, jpg or gif files under your image directories to WebP format in such a way that they will be automatically served by the code above.

#!/bin/bash
# Convert all images to WebP
IMAGE_PATHS="assets/ imgs/"
for SRC in $(find $IMAGE_PATHS -name "*.png" -o -name "*.jpg" -o -name "*.jpeg" -o -name "*.gif"); do
    WEBP="${SRC%.*}.webp"
    if [ "$SRC" -nt "$WEBP" ]; then
        echo "Converting to $WEBP"
        convert "$SRC" -define webp:alpha-compression=1 -define webp:auto-filter=true -define webp:alpha-quality=90 -quality 95 "$WEBP"
        
    fi
done

Note the -nt comparison that only updates files if the source has changed. You could add this script to git post-checkout and post-merge hooks to automatically keep your WebP assets in sync with the images in the code (and add a .gitignore entry for *.webp – no need to keep 2 copies of each resource in the repository).

Important note: If you’re using an older version of imagemagick such as on Ubuntu 14.04 (imagemagick 6.7.7), it doesn’t pass the alpha compression arguments through correctly so if you have a lot of transparency you won’t see much in the way of compression happening. Switch the convert line to be something like the below, however you need to remove the gif support as that requires using the gif2webp command to convert:

cwebp -quiet "$SRC" -metadata none -alpha_q 80 -q 90 -o "$WEBP"

Also note that this causes some issues when you have for example a jpg and png of the same base name whose contents are different (I found a few in the old code I inherited). You can find the base name of any of these clashes clashes using the following command:

find $IMAGE_PATHS -name "*.png" -o -name "*.jpg" -o -name "*.jpeg" -o -name "*.gif" | perl -pe 's,\.[^.]+$,\n,' | sort |uniq -d

Awesome Angular 4 form validator routine

One of the things I like the most about Angular is the ability to make even complex forms relatively simple. In Angular 1 I had a library of form helpers that I wrote, the basic idea was initially show a blank form, when user fills in (dirties) an entry it should do validation. When the user click on the submit button it should do validation of all items on the form, submit the form if no errors and show any errors if there were any. This sounds simple enough but in reality it was a few hundred lines of code to do it correctly.

Angular 2+ (Angular 4 on Ionic 3 in this case) make life a lot easier, especially once you’ve got to terms with the FormBuilder framework. To force whole-form validation (including any subforms) simply create a module called form-tools.ts looking like:

// Force a form to show any invalid entries and return if form is valid or not. Recurse through any subforms.
export const validateForm = (form) => {
    Object.keys( form.controls ).forEach( control_name => {
        let control = form.controls[control_name];

        control.markAsTouched();
        if( 'controls' in control )
            validateForm( control );
    });

    return form.valid;
};

Then you can easily use it from another class as follows:

import { FormBuilder } from '@angular/forms';
import { validateForm } from '../../form-tools';

class ...Page {
    constructor(private fb : FormBuilder ) {
        this.userForm = this.fb.group({
            form elements...
        });
    }

    submit_form() {
        if( !validateForm(this.userForm) )
            return;
        send the form request to the server(this.userForm.value)
    }
}

Element scrolling within Angular (1) pages

Back to Angular 1 for today’s post as I’ve been doing some work on an older project today. As we all know, in the old days of the web, you could scroll to items within the page by using an <a> element with a hash reference, like:

<a href="#thing-to-scroll-to">scroll here</a>

<h2 id="thing-to-scroll-to">My title</h2>

With the advent of single-page sites and multiple ajax pages under them however, the hash section of a query parameter is increasingly unable to be used. On Angular 1 with angular-route this becomes impossible.

So what if for example you want to have some references at the top of a page within an Angular application which will scroll the user down to certain sections below, just like the hash references used to do? There are a number of suggestions on the internet for this but they are not very reusable, so I decided to create a simple directive that does this:

app.directive('clickScrollTo', function () {
    return {
        restrict: 'A',
        scope: {
            clickScrollTo: '@'
        },
        link: function (scope, el, attr) {
            el.click(function(event) {
                var main_page = $('.main-page');
                event.preventDefault();
                main_page
                    .stop()
                    .animate(
                        { scrollTop: main_page.scrollTop() + $(scope.clickScrollTo).offset().top },
                        500,
                        'linear'
                    );
            });
        }
    }
});

This is slightly complicated by the fact that I am using an element with class="main-page" with overflow: auto set to contain the scrollable section of the app. If you don’t have a layout like this just replace the $('.main-page') part with $('body').

Then you can easily create elements like:

<span class="a" click-scroll-to="#more-benefits">click here</span>
...
<div id="more-benefits">...</div>

Ionic 3 – Allowing back button action to be specified on a page-by-page basis

Ionic 3 is great as I’ve said in previous posts, however there are a few issues with its out-of-the box handling of back button on Android. Normally it just goes back to the previous page which is usually what you want, however this is not what you want when you are in a popup or modal on the page, or in some other cases you may want to capture the back event and do something else with it on a page-by-page basis. Fortunately it’s pretty straight forward to override this as I discovered.

Firstly, we want to override the back button action for the whole app. Open up app/app.component.ts:
and in the constructor:

import { App, Platform } from 'ionic-angular';
...
  constructor( ..., private app: App, private platform : Platform ) {
    platform.registerBackButtonAction(() => {
        let nav = app.getActiveNavs()[0];
        let activeView = nav.getActive();

        if(activeView != null){
          if(nav.canGoBack())
            nav.pop();
          else if (typeof activeView.instance.backButtonAction === 'function')
            activeView.instance.backButtonAction();
          else
            nav.parent.select(0); // goes to the first tab
        }
      });
  }

This basically defaults to going back if it is possible to, if not then it will take you to the first tab if it is a tab view. However if your active page has a backButtonAction() function, it will delegate to that.

So for example in a modal class you can add something like:

import { ViewController } from 'ionic-angular';
...
    constructor( private viewCtrl : ViewController ) {}

    backButtonAction() {
        this.viewCtrl.dismiss();
    }

which will dismiss the modal and go back to the page that called it, rather than the default action of simply going back a page.

Ionic 3 Page Overlay

I’ve been doing a project recently in ionic 3 (WatchEm) and I must say I’m pretty impressed. I never tried using ionic 1 as it looked like quite a lot of overhead and it wasn’t certain whether it was going to turn into a popular platform, but ionic 3 seems good, stable and is developing well.

One thing we wanted to do was to provide a help screen the first time you access each page in the app as for example some google apps do. The aim was to present an overlay which provides some textual and visual pointers as to what you can do on the page, and some hints about it.

Fortunately it wasn’t hard to do; here is the code I wrote which lets you produce flexible help pages of the format:

<ion-header>
...
</ion-header>

<overlay>
    <h2>Player Buttons and Features</h2>

    <p>
        Pressing
        <ion-icon name="ios-skip-backward"></ion-icon>
        <ion-icon name="ios-skip-forward"></ion-icon>, or using your left/right keys you can move to the next key event in the game.
    </p>
</overlay>

<ion-content>
...
</ion-content>

We’re going to produce this as a component, so create components/overlay/overlay.ts like:

import { Input, Component, ElementRef, Renderer2, AfterViewInit } from '@angular/core';
import { Storage } from '@ionic/storage';
import { NavController } from 'ionic-angular';
    
@Component({
  selector: 'overlay',
  templateUrl: 'overlay.html'
})      
export class OverlayComponent implements AfterViewInit {
    private _force :boolean = false;
        
    @Input()
    set force(val) {
        this._force = val == '' ? true : !!val;
    }   
    
    constructor( private elementRef : ElementRef, private renderer : Renderer2, private _storage: Storage, public navCtrl: NavController ) {
    }

    get storage_key() {
        return `shown-overlay-${this.navCtrl.getActive().id}`;
    }

    ngAfterViewInit() {
        // Check local storage to see if we already displayed this...
        this._storage.get(this.storage_key).then( (val) => {
            if( !val || this._force )
                this.renderer.addClass( this.elementRef.nativeElement, 'shown' )
        });
    }

    hide_overlay() {
        this._storage.set(this.storage_key, 1);
        this.renderer.removeClass( this.elementRef.nativeElement, 'shown' );
    }
}

Pretty straight forwards – if the force= attribute is set on the <overlay> tag then it will always show it (useful for debugging). Otherwise if it is the first time the page has been opened it will show and then store in localStorage to say it shouldn’t be shown again.

Next, the HTML for the component in components/overlay/overlay.html:

<ion-grid full-height (click)="$event.stopPropagation(); hide_overlay()" ion-text color=white text-center>
    <ion-row full-height align-items-center>
        <ion-col col-md-8 push-md-2>
            <ng-content></ng-content>
        
            <button ion-button>Got it</button>
        </ion-col>
    </ion-row>
</ion-grid>

Obviously feel free to do what you want here with text/layout. We call stopPropagation() in order to prevent any stuff on the main page from receiving the click, especially if you have click-handlers further up the chain eg on the body element.

Finally a bit of styling in components/overlay/overlay.scss to make it look like an overlay and handle visibility changes correctly:

overlay {
    display: none;
        
    &.shown {
        position: fixed;
        display: block;
        padding: 40px 20px;
        top: 0;
        left: 0;
        bottom: 0;
        right: 0;
        background: rgba( 0, 0, 0, 0.7 );
        z-index: 9999;
        overflow-y: auto;
        overflow-x: hidden;
    }
}

Note that the overlay must be placed outside of any <ion-content tags as they provide for automatic scrolling of their content etc which is not wanted as the overlay itself needs to scroll.

Apache configuration for WkWebView API service (CORS)

Switching from UIWebView to WKWebView is great, but as it performs stricter CORS checks than standard Cordova/Phonegap it can seem at first that remote API calls are broken in your app.

Basically, before WKWebView does any AJAX request, it first sends a HTTP OPTIONS query and looks at the Access-Control-* headers that are returned to determine if it is able to access the service. Most browsers can be made to allow all AJAX requests via a simple “Access-Control-Allow-Origin: *” header, however WKWebView is more picky. It requires that you expose which methods (GET, POST, etc); and which headers are allowed (eg if you are using JSON AJAX requests you probably need to use a “Content-Type: application/json” header in your main request).

Rather than having to update your API service, you can work around this in a general way using the following Apache config:

    # Required configuration for iOS WkWEBVIEW

    # Allow any location to access this service
    Header always set Access-Control-Allow-Origin "*"

    # Allow the following headers in requests (X-Auth is a custom header, also allow Content-Type to be specified)
    Header always set Access-Control-Allow-Headers "X-Auth, content-type, origin"
    Header always set Access-Control-Expose-Headers "X-Auth"

    # Allow the following methods to be used
    Header always set Access-Control-Allow-Methods "GET, POST, OPTIONS"

    # WkWebView sends OPTIONS requests to get CORS details. Don't tie up the API service with them;
    # just answer them via apache itself
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} =OPTIONS
    RewriteRule .* - [R=204,END]

Note the last line answers any HTTP OPTIONS request with blank content and returns it straight away. Most API services would cause a lot of CPU processing just to handle a single request whether it is a true request or an OPTIONS query, so we just answer this straight from Apache without bothering to send it through to the API. The R=204 is a trick to specify that we don’t return any content (HTTP 204 code means “Success, but no content”). Otherwise if we used something like R=200 it would return a page talking about internal server error, but with a 200 response which is more bandwidth, more processing and more confusing for any users.

Angular 4 API service with automatic retries and Ionic 3 integration

In the bad old days of the web, you’d submit a form and if there was a problem with your internet connection it would loose the form and display an error page in the browser. These days you don’t need to worry about this quite so much, but handling errors with sending AJAX form-submits or other API requests is still a difficult topic. Fortunately, the way that Angular 4 uses Observables makes retrying requests quite a bit easier.

In the app I was building for a client recently, we wanted the default process flow to be as follows. Any API request should display a spinner (via Ionic 3), and send the request to the server. If we got an error like login failure then it should return this error to the client. If the error is with the network connection timing out it should automatically retry a couple of times. For other errors such as internal server (ie API side) or not connected at all, it should fail straight away. However if it was an API or network connection failure, it should display a popup prompting the user to opt to retry or cancel the request (eg ‘Turn your internet connection on and hit retry’) rather than making them hit a form resubmit button again.

As Observables remember all the data and options they were submitted with, it’s pretty easy to retry the request and there are a number of bits of code on the internet for this. However I couldn’t find any good examples of this being written in a reusable fashion, and with options of asking prompting the user without forgetting the request. So, here is an example of how you can do this within the framework of Ionic, however it should work in general for anything based on Observables especially under Angular 2+. Below I’ll walk through some of the harder parts of this code.

Create the API service (app/api.service.ts) looking like:

import { Injectable } from '@angular/core';
import { Http, Headers, Response } from '@angular/http';

import { Observable } from 'rxjs/Observable';
import { Subject } from 'rxjs/Subject';
import 'rxjs/add/operator/map';
import 'rxjs/add/operator/scan';
import 'rxjs/add/operator/do';
import 'rxjs/add/operator/delay';
import 'rxjs/add/operator/retryWhen';
import 'rxjs/add/operator/finally';
import 'rxjs/add/operator/delayWhen';
import 'rxjs/add/operator/timeout';
import 'rxjs/add/observable/throw';
import 'rxjs/add/observable/onErrorResumeNext';
 
@Injectable()
export class APIService {
  public inprogress_requests : Subject<any> = new Subject();

  public error_handler : (message :string, err:any) => Observable<any> = (err) => Observable.throw(err);

  private requests_active :number = 0;

  constructor (private http: Http) {}
 
  request(path: String, data = {}, options :any = {}): Observable<any> {
    if( !options.headers )
        options.headers = new Headers();
    options.headers.append('Content-Type', 'application/json');

    let base_url = this.config.baseApiUrl();

    let timeout = options.timeout || 10000;
    let max_retries = 'retries' in options ? options.retries : 3;

    let url = `${base_url}/api/${path}`;
    let request = this.http.post(
            url, JSON.stringify(data),
            {
                headers: options.headers,
            }
        )

        // Add a timeout and retry the request after specified time and 3
        // attempts (but only if it is a timeout error)
        .timeout(timeout)
        .retryWhen((errors) =>
            errors.scan( ( errorCount, err ) => {
                if( errorCount < max_retries && err.name == 'TimeoutError' )
                    return errors.delay(500);
                throw err;
            }, 0)
        );

    if( options.interceptor )
        request = request.do( options.interceptor );

    request = request.map(this.extractData);

    // User-visible error handler now
    if( options.auto_fail )
        request = request.onErrorResumeNext();  // enable mergeMap etc to keep working
    else
        request = request.retryWhen( (errors) =>
            errors.delayWhen( (error) => {
                let message = this.log_error(error);
                console.error( `URL was: ${url}, request body: ${stringified_data}` );
                return this.error_handler(message, error);
            })
        );

    if( !options.nonblocking ) {
        this.add_blocking_request( options.loading_msg ? { reason: options.loading_msg } : {} );
        request = request.finally( () => this.finish_blocking_request() );
    }

    return request;
  }

  private add_blocking_request(details :any = {}) {
    // Re-issue a request if the details have been updated
    if( this.requests_active++ == 0 || Object.keys(details).length ) {
        details.active = true;
        this.inprogress_requests.next( details );
    }
  }

  private finish_blocking_request() {
    if( --this.requests_active == 0 )
        this.inprogress_requests.next( { active: false } );
  }

  private extractData(res: Response) {
    // Decode errors will be handled automatically by Observable
    let body = res.json();
    return body || {};
  }

  private log_error(error: Response | any) {
    let errMsg: string;
    if (error instanceof Response) {

      // Ignore any decode errors
      let body :any = {};
      try {
        body = error.json() || {};
      } catch(e) {}

      const err = body.error || JSON.stringify(body);
      errMsg = `${error.status} - ${error.statusText || ''} ${err}`;

      // No internet, probably
      if( error.status == 0 ) {
        console.error(errMsg);
        errMsg = 'Your internet connection is offline. Please connect and hit retry';
      }
    } else {
      errMsg = error.message ? error.message : error.toString();
    }
    console.error(errMsg);
    return errMsg;
  }
}

Lets walk through some potentially confusing bits of this service.

The main request observable is the request variable, we perform actions on this (saving the result in the request variable again) as the user requests, Initially we just set the request to have a timeout (several multiples of time of the maximum time you expect the API to respond in, otherwise you may get multiple resubmissions of the same request if the API gets a bit laggy).

Then, we come to this piece of code:

        .retryWhen((errors) =>
            errors.scan( ( errorCount, err ) => {
                if( errorCount < max_retries && err.name == 'TimeoutError' )
                    return errors.delay(500);
                throw err;
            }, 0)
        );

This basically keeps a log of all the errors that occurred and each time there is an error with the request, it first checks to see how many times we already retried, and ensure that it was actually a timeout error (as opposed to an internal server error or so). If that was the case then it waits 500ms and retries, otherwise it re-throws the error which will cause the Observable to continue as an error response.

    if( options.auto_fail )
        request = request.onErrorResumeNext();  // enable mergeMap etc to keep working

If the user passes an auto_fail option to the request, we want the request to happily silently fail (perhaps we are just sending some usage stats to the server and we don’t want errors popping up about them). This basically returns a successful Observable whether or not it was actually a success so that it doesn’t short-circuit anything due to an error being raised.

However, under normal circumstances we want to raise a frontend error:

        request = request.retryWhen( (errors) =>
            errors.delayWhen( (error) => {
                let message = this.log_error(error);
                console.error( `URL was: ${url}, request body: ${stringified_data}` );
                return this.error_handler(message, error);
            })
        );

This code says to shell out to an external function (the error_handler function reference which can be set somewhere in the main code that builds the API) with the error, and expects it to return an item such as a Subject or a true/false value indicating whether the whole of the above work should be retried again or not. This is a bit messy – you should perhaps have multiple different instances of API depending on whether you want this functionality or not, but because the API is a global service and we want a standard piece of retry code I thought to put it like this. However because it needs to interact with the frontend, I set this elsewhere as I’ll show in a bit.

Finally, we want to wrapper most requests with some code to display a spinner (optionally with a message), unless it is a non-blocking request:

    if( !options.nonblocking ) {
        this.add_blocking_request( options.loading_msg ? { reason: options.loading_msg } : {} );
        request = request.finally( () => this.finish_blocking_request() );
    }

The add_blocking_request and finish_blocking_request issue an Observable message (via this.inprogress_requests) when there are requests active or when the last active request finishes, which avoids having the spinner popping on and off again every time a request is redone or a sub-request is triggered.

Finally, in the main app constructor we hook into these two Observables to do the UI-facing work (app/app.component.ts in ionic – this is ionic-specific but you should be able to replace with your own framework easily enough). Firstly, the spinner:

    // Loader needs creating each time it is displayed in ionic...
    let loader; 
    api.inprogress_requests.subscribe(
        details => {
            if( loader )
                loader.dismiss();
            loader = null;  
                            
            if( details.active ) {
                let loader_options :any = {};
                if( details.reason )
                    loader_options.content = details.reason;
                loader = loadingCtrl.create(loader_options);
                loader.present();
            }           
        }           
    );

Simple enough – if there is a loader get rid of it, and if there should be one then create it with the message. This enable us to update the message displayed easily enough although I’ve not really used this functionality much in the code I’ve written.

Finally, lets look at the dialogs presented to the user to prompt retries. This handler should be simple enough providing different dialogs and messages depending on what the error was exactly. Note that we are returning a Subject which we effectively use like a Promise to handle the asynchronous nature of user interaction with the dialog:

    // Handle errors with a popup and offer retry functionality
    api.error_handler =
        (message, error) => {
            // Just in case it is the first request..
            this.hide_splashscreen();

            let retry_subject = new Subject();
            let retry;

            // Unauthorized
            if( error.status == 401 ) {
                retry = alertCtrl.create({
                    title: 'Logged Out',
                    message: `You have been logged out and need to log in again`,
                    buttons: [
                        {
                            text: 'OK',
                            handler: () => {
                                retry.dismiss().then( () => this.navCtrl.push('login') );
                                retry_subject.error( error );
                                retry_subject.complete();

                                return false;
                            }
                        },
                    ]
                });
            } else {
                let title = 'Server Error';
                let display_message = `We got an error from the remote server: ${message}. Do you want to retry?`;

                // 400's are nicer errors - not a server code issue but a user input problem most likely
                if( error.status == 400 || error.status == 0 ) {    // 0 = no internet
                    display_message = `${message}. Do you want to retry?`;
                    title = "Error";
                }
                retry = alertCtrl.create({
                    title,
                    message: display_message,
                    buttons: [
                        {
                            text: 'No',
                            handler: () => {
                                retry.dismiss();
                                retry_subject.error( error );
                                retry_subject.complete();
                                return false;
                            }
                        },
                        {
                            text: 'Retry',
                            handler: () => {
                                retry.dismiss();
                                retry_subject.next( 1 );
                                retry_subject.complete();
                                return false;
                            }
                        }
                    ]
                });
            }
            retry.present();
            return retry_subject;
        };

Prompt before opening an external link in AngularJS

On a recent project of creating an Angular app which would be both a website and a cordova-packaged app, we had a number of links which opened to external websites (terms and conditions, links to some process flows which couldn’t be contained within the app, etc). However because some of the branding on the sites was very similar to the app itself some test users were getting confused about whether they were still in the app, or had been redirected into a browser.

Because of these issues the client wanted us to create a small popup for some external links that would prompt the user to see if they wanted to move off the site/app. Below is a small angular directive that does this. Usage like:

<a href="https://..." target=_blank prompt-before-open="Do you want to go to an external web page?">...</a>

app.directive('promptBeforeOpen', function($window) {
    return {
        link: function(scope, elem, attr) {
            $(elem).click(function(event) {
                if( !$window.confirm( attr.promptBeforeOpen ) )
                    event.preventDefault();
            });                      
        }                            
    };                               
});