Remove Windows Phone CSS in Ionic 2/3 to save space

Lets face it, no-one uses Window Phone any more. However, in Ionic 3 there are several skins which are built in to the CSS to emulate the display of the platform that the app is running on. These are ios (iOS, apple devices), md (Material Design – Android devices and Ionic default skin) and wp (window phone). As each of these is different and affects almost all widgets, they are each responsible for roughly 25% of the CSS size of ionic (which can be big – 250kb or so).

So, as no-one uses windows phone and we don’t want to have to test multiple platforms we can easily remove it from the CSS build to save time, space and complexity.

Firstly, copy the SASS config into our custom config directory; we will use this to override the default Ionic SASS config:

Then edit your package.json file and create or add a config dictionary to it like:

Finally, open up config/sass.config.js, find the excludeFiles section and add the following:

If you don’t want to match different platforms with different Ionic themes/skins (which while nice takes quite a bit of time to fully test), you can choose to use eg the Material Design skin only by doing something like:

CKEditor Plugin Validation for Multiple Requirements

I’ve been working on a few CKEditor plugins recently for some projects. When building the elements of a dialog you create a data structure like:

This is great, but what if you want multiple validation criteria for example it must be a number and not empty? There are a handful of hacks and examples of this online, but looking through the code I found an undocumented (and otherwise unused) function which enables you to combine multiple validation functions:

You can pass as many validation functions into this as you want and it will combine them, but only allows you to specify one validation message unfortunately.

Ionic 3 Page Overlay

I’ve been doing a project recently in ionic 3 (WatchEm) and I must say I’m pretty impressed. I never tried using ionic 1 as it looked like quite a lot of overhead and it wasn’t certain whether it was going to turn into a popular platform, but ionic 3 seems good, stable and is developing well.

One thing we wanted to do was to provide a help screen the first time you access each page in the app as for example some google apps do. The aim was to present an overlay which provides some textual and visual pointers as to what you can do on the page, and some hints about it.

Fortunately it wasn’t hard to do; here is the code I wrote which lets you produce flexible help pages of the format:

We’re going to produce this as a component, so create components/overlay/overlay.ts like:

Pretty straight forwards – if the force= attribute is set on the <overlay> tag then it will always show it (useful for debugging). Otherwise if it is the first time the page has been opened it will show and then store in localStorage to say it shouldn’t be shown again.

Next, the HTML for the component in components/overlay/overlay.html:

Obviously feel free to do what you want here with text/layout. We call stopPropagation() in order to prevent any stuff on the main page from receiving the click, especially if you have click-handlers further up the chain eg on the body element.

Finally a bit of styling in components/overlay/overlay.scss to make it look like an overlay and handle visibility changes correctly:

Note that the overlay must be placed outside of any <ion-content tags as they provide for automatic scrolling of their content etc which is not wanted as the overlay itself needs to scroll.

Successfully downloading big files from Dropbox via Linux command-line

Recently, someone was trying to send me a 20Gb virtual machine image over dropbox. I tried a couple of times to download using chrome, however it got to 6-8Gb and then came up with a connection error. Clicking on the resume button failed and then removed the file (!). Very strange as I didn’t have any connection issues, but perhaps a route changed somewhere. I saw a number of dropbox users complaining about this on the internet. Obviously there are other approaches such as adding to your own dropbox account and using their local program to do the sync, however because I’m just on a standard free account I couldn’t add in such a large file.

Because I was using btrfs and snapper I still had a version of the half-completed download around, and so I tried seeing if standard linux tools would be able to continue the download where it left off. It turns out that simply using wget -c enables you to resume the download (it dropped a couple of times during the download but just restarting it with the same command let the whole file download just fine. So, to download a large dropbox file even if your internet connection is a bit flakey, simply go to the dropbox download link and then paste it into the terminal (may require the ?dl=1 parameter after it) like:

Apache configuration for WkWebView API service (CORS)

Switching from UIWebView to WKWebView is great, but as it performs stricter CORS checks than standard Cordova/Phonegap it can seem at first that remote API calls are broken in your app.

Basically, before WKWebView does any AJAX request, it first sends a HTTP OPTIONS query and looks at the Access-Control-* headers that are returned to determine if it is able to access the service. Most browsers can be made to allow all AJAX requests via a simple “Access-Control-Allow-Origin: *” header, however WKWebView is more picky. It requires that you expose which methods (GET, POST, etc); and which headers are allowed (eg if you are using JSON AJAX requests you probably need to use a “Content-Type: application/json” header in your main request).

Rather than having to update your API service, you can work around this in a general way using the following Apache config:

Note the last line answers any HTTP OPTIONS request with blank content and returns it straight away. Most API services would cause a lot of CPU processing just to handle a single request whether it is a true request or an OPTIONS query, so we just answer this straight from Apache without bothering to send it through to the API. The R=204 is a trick to specify that we don’t return any content (HTTP 204 code means “Success, but no content”). Otherwise if we used something like R=200 it would return a page talking about internal server error, but with a 200 response which is more bandwidth, more processing and more confusing for any users.

Using Ionic’s WKWebView Engine with standard cordova/phonegap projects

The guys at ionic have created a branch of phonegap’s WKWebView Engine (for iOS) with a number of improvements such as local AJAX request ability. This is great for ionic projects, however I wanted to see if it could be used in a standard phonegap project for which the standard WKWebView plugin wouldn’t work.

For those who don’t know, older versions of iOS had an implementation of webview which was called UIWebView. This had a number of bugs and performance issues, however because cordova/phonegap strive to maintain backwards-compatibility, it is still the default within phonegap projects. WKWebView is faster, more stable and has newer features such as IndexedDB, however it has some additional security restrictions (especially with CORS and local file loading) meaning it is not simply a drop-in replacement for UIWebView.

Installation is simple, more details can be found on the github project page, however basically just run:

and it will be included in your iOS project. Make sure that your app works (and detects it’s being run from within cordova) when the window.location is set to 'http://localhost:8080/' rather than the usual file:// path. Then, add the necessary configuration into config.xml:

However, even if you do this, when you open the test app you’ll find that you are for some strange reason unable to scroll! This is because the ionic framework doesn’t use body scrolling, it has subelements with overflow: auto set on them. However for most non-ionic apps you probably want body scrolling. Simply change add the following into the ios platform configuration section in config.xml:

If you are accessing remote API services, you’ll also need to modify some CORS settings on your server. In my post tomorrow I’ll show you how to do this easily without increasing the load on your API service.

Tracking down Lua JSON decoding issues

I’ve recently been doing quite a bit of Lua scripting for a client wanting some PowerDNS customizations. I’ve actually grown to quite like Lua, even though it’s very simple and quick, you can do some very complex programming with it reasonably straight forwardly. I think it could perhaps be compared to a stripped-down version of Perl which is also a language that I very much like because of its incredible flexibility.

Anyway, as part of this work are wanting to look up incoming IP addresses in a table of non-overlapping IP address ranges. For high performance I recommended LMDB as I’ve used it extensively before and I know that for its quirks and tendency to crash if you mishandle any aspect of its API, it is very high performance, low over head, scales very well to multiple cores, and can do pretty much anything you ask of it.

So basically the problem was “how do we store an IP range as an indexed key in LMDB” (which is just a key-value database where all keys are b-tree indexed). In the future we may want to support IPv6, and we may also want to support IP ranges which cannot be expressed in subnet-mask representation. The solution I came up with is to store the first IP in raw binary format (ie 4 bytes for IPv4, or 16 bytes for IPv6) as the key, and then as part of the value we store the end IP address. In order to see if a given IP is within a subnet, you look up you open a cursor on the table, seek to the position of the IP you are trying. If you get a direct hit, then obviously it has found the first IP in the subnet and so you know it is valid. If it does not get a direct hit you seek back to the previous entry (this is a great feature of LMDB and is found in surprisingly few indexed key-value data store APIs, even though it should be very simple to implement). You then take the value of that, get the end IP of the range and check to see if the requested IP is within the start and end of the range.

Because we wanted a very flexible and easily extensible data storage format for the values in this table we decided to encode it all as JSON. Lua has a number of JSON decoders and lua-cjson seemed pretty quick and easy, and was also available as a pre-built ubuntu package so we went with that. As we were storing the key’s IP address in raw binary notation, we figured it would make the code-path simplest if we stored the end IP address in the same manner. So, we did this, wrote a test suite with some non-public IPv4 addresses (10.xxx and 127.xxx) and verified that it was all working correctly, and then launched the code.

A few days later we started getting some complaints from customers that some IP addresses in their network ranges were not being identified correctly. But when we added the exact same details into the test suite with our private IP ranges, it clearly worked fine.

Finally I started trying to use the exact IP addresses that the customers were reporting issues with in the test scripts and discovered that there was actually a problem. Basically, whenever a component of the address was greater than 127 and the code did not go down the direct hit code path (ie the address was part of a subnet larger than a /32 and not the first entry) the decoded end IP address would be incorrect. Very strange! So, our test code which was using ranges like 127.0.0.1-127.1.2.3 worked fine, but an IP range like 1.0.0.0-1.0.129.0 would fail!

Looking more closely at the cjson Lua documentation I saw the line “cjson.decode will deserialise any UTF-8 JSON string into a Lua value or table”. And reading through the C code I saw that the routines were hard-coded to treat any JSON escaped \uXXXX value that was greater than 127 as part of a UTF-8 encoded character. This is because Lua uses the platform’s underlying char[] to store strings which means usually each character in a string can only be 8-bits, meaning that in order to store wider characters the bytes need to be encoded into a single character which is what UTF-8 is for. With our encoding we knew that all parts of the string would fit into 8-bits, but there was no way to tell the decoder this. Because cjson is aiming to be a fast module, this is all hard-coded and there is no way that I could see to easily work around this utf-8 decoding. We tried some other Lua JSON modules but they either had the same problem, or were orders of magnitude slower than cjson.

Eventually a colleague suggested just hex-encoding the end IP address prior to including it in the JSON data which was the simplest solution we could find. It should also reduce the storage required for an IP address as assuming 50% of the characters are usually encoded with a \uXXXX escape sequence in JSON, an average IPv4 address would take 14 bytes in the database, whereas with hex this would be a fixed 8 bytes per IPv4 address.

If the encoding program had been using perl we could probably have used some of the features of the JSON::XS module (specifically, the utf8 flag) to write characters directly as bytes into the string, which although is perhaps not technically valid JSON, from my reading of the Lua module should have bypassed the UTF-8 encoding of escaped values. However we wern’t using perl in our encoding routines so this wasn’t possible.