All posts by Mark

I'm a full-stack linux consultant from the UK but currently living in Istanbul. I specialize in rapid development of web/mobile apps using HTML5 typically using Bootstrap, AngularJS, Modern Perl (Catalyst/DBIC) and PostgreSQL but have worked with many other technologies and languages as well. I'm fluent in English and Turkish.

Creating flexible highlight rules in Ace that change according to editor mode

I’ve only recently started using the excellent Ace editor in projects, but I’m really enjoying it so far. It has very flexible and well designed custom highlighting rulesets that developers can extend to various different syntax. However it does not currently support in-editor spell-checking (I believe this functionality is planned for a release soon though – hopefully!).

One of the situations I’m using the editor in is site which has input from many different languages and character sets. As part of this I wanted to ensure that the language that a given article is claimed to be written in matches the character set of various content sections of the editor. For example if the article is in Thai, but you are using Latin script characters it should highlight them as errors. A big issue we have is that there is a lot of Cyrillic content, but as a number of characters render the same or very similarly in Cyrillic and Latin (for example Р and P), some of the users input mostly Cyrillic, but with the occasional character of Latin. This wouldn’t be a problem if we were just rendering the text, but we are also unidecoding it for searching – in this case Latin P goes to p, but the Cyrillic character Р encodes to r (as it is pronounced). This means that it throws off the searching.

I wrote a script which parses the CLDR data’s exemplarCharacters data to get the expected character sets for a language, adds a few in (as the CLDR is not totally complete unfortunately, especially for languages that use extended Cyrillic sets such as Karakalpak, lacks Traditional Mongolian entirely, and is incomplete for some ideographic languages such as Chinese). It then adds some general punctuation and other characters and generates a javascript regexp for matching characters that should not exist. For example for Armenian, the regexp is /[^\u0020-\u0022\u0025-\u0029\u002b-\u003b\u003f\u005b-\u005d\u00ab\u00b4\u00b8\u00bb\u0531-\u0556\u055a-\u055f\u0561-\u0587\u058a\u2030]/.

So far so good, but how to integrate this with Ace editor? Usually you only have a mode for Ace which specifies the language (eg PHP, HTML), but I don’t want to create a new mode for each language/script that this app wants to support.

Initially I tried basing some code on this code which adds spell checking outside of Ace, however there are a number of limitations with this approach namely it doesn’t integrate with the existing highlighting system so if you are mean to be editing eg HTML document containing only Thai characters it doesn’t know which parts are HTML and which parts should be checked, without redoing the whole highlighting run a second time. It is also needs to reprocess the entire document every change.

Digging around in the Ace source I found that you can actually pass an object into the setMode function which enables you to pass new parameters, such as the invalid-characters regexp for the current language. However the highlighting functions are usually static which means it is complex to update this on-the-fly, especially after normalizing it for something based on the Text Highlight Rules. My solution is as follows:

You can then just do .setMode({ path: 'ace/mode/my_mode', regex: /.../ }) changing the regex for each different language or character set that you wish to validate.

Recovering from unmountable btrfs filesystem issues

Here are some notes of how I recovered most of the data after my btrfs disk got horribly corrupted by bad memory. Fortunately I had upgraded the disk 6 months ago so I was able to start from that image left behind on the old disk, copied over using the excellent btrfs-clone tool.

After that I could restore most of my files to the last backup (a month or two back) and git repositories from the main server. But I still had a number of documents and other bits that I needed to recover.

The first thing prior to formatting the disk (I don’t have another spare fast SSD lying around) was to take a backup of the entire btrfs disk. However it was quite a bit larger than I easily had spare on another disk. So, I stored it in squashfs which reduced size by 50%.

After that I tested that it was mountable:

And erased and cloned the old btrfs disk to it.

I then started using the btrfs restore tool to try to recover the data. First you need to list the roots, usually the highest number will be the latest snapshot and it may have consistent data:

Then you can get a listing of the files under that root and whether they may be recoverable using the -v -D flags (-v means list files, -D means don’t actually try to restore any data. For example:

If that looks good then you can run the command with a few extra flags to try to get the files back as much as possible:

This can take a while but it seems to work well on smaller files. Unfortunately some virtual machine images (60gb or so each) didn’t recover because they had got corrupted in the middle.

If you want to recover only a particular point under the tree you can use the --path-regex parameter to specify this, however writing the regexps is very difficult. Here is a short bit of code which will generate the path regex correctly:

You can then restore just those files like:

Diagnosing faulty memory in Linux…

For the past year I’ve had very occasional chrome crashes (segfaults in rendering process) and an occasional bit of btrfs corruption. As it was always easily repairable with btrfs check --repair I never thought much about it, although I suspected it may be an issue with the memory. I ran memtest86 overnight one time but it didn’t show up any issues. There were never any read or SMART issues logged on the disk either, and it happened to another disk within the machine as well.

Recently though I was seeing btrfs corruption on a weekly basis, especially after upgrading to ubuntu 18.04 (from ubuntu 16.04). I thought it may be a kernel issue so I got one of the latest kernels. It seemed to happen especially when I was doing something quite file-system intense, for example browsing some cache-heavy pages while running a vm with a long build process going on.

Then, earlier in the week the hard drive got corrupted again, much more seriously and after spending some time fixing, running btrfs check --repair a few times it suddenly started deleting a load of inodes. Force rebooting the machine I discovered that the disk was un-mountable, although later I was able to recover quite a lot of key data from btrfs restore as documented in this post.

memtest86 was still not showing any issues, and so my first thought was that assuming the hard disk was not at fault it may be something to do only when the memory had a lot of contention (memtest86 was only able to run on a single core on my box). I booted a minimal version of linux and ran a multi-process test over a large amount (not all) of the memory:

where 8 is the number of processor/threads and 1400 is the amount of free memory on the system divided by that number (in my case I was testing 16gb of memory). 10 is the number of runs. It took about 45 min to run once over the 16gb, or about 25 min to run over 8gb (each of the individual sodimms in my laptop).

Within about 10 minutes it started showing issues on one of the chips. I’ve done a bit of research since this and seen that if a memory chip is going to fail then it would usually do it within the first 6 months of being used. However this is a kingston chip that has been in my laptop since I bought it 2 or 3 years back. I added another 8gb samsung chip a year ago and it seemed to be after that that the issues started, however that chip works out as fine. Perhaps adding another chip in broke something, or perhaps it just wore out or overheated somehow…

ESP8266 minimal setup

I’m sure there are many notes out there, but I often get confused about the minimal setup required to run an ESP8266. You actually only need 4 pins connected:

Connect GND to 0v, VCC and EN to +3.3v.

Then connect GPIO15 via a 2-10k (I usually use 3k3) resistor to GND to specify boot from flash.

And you’re good to go.

Obviously in order to do the initial flash of the device you need to connect the TX/RX and also connect GPIO0 to GND.

Whatsapp upgraded, crashes on start

Somehow today my wifes’ phone had managed to upgrade to a new version of WhatsApp. When she opened it it just said that the applicaiton had crashed. This also started happening recently with ‘Google Play Services’ and some other apps on her phone.

(As an aside, this is why I turn off auto-update where at all possible because you never know when something will break)

However after much research and debugging I learnt that the problem is not so much with WhatsApp itself as with the Cyanogenmod (custom ROM) that we use on our phones and will happen increasingly. Fortunately there is a relatively easy way to fix this – skip to the bottom of this article if you want to just fix the issue.

The technical root cause is documented on the google issue tracker and is caused by a change in the way apps are being built when they are upgraded to using the gradle 3 build-chain. It seems to be fixed in the latest versions of google build-tools so hopefully in the next 6 months this problem will go away but for the moment it will only increase as teams upgrade their android build chains. Basically in my quick scanning of the bug ticket the problem is that the implementation of some low-level part of reading an apk package on cyanogenmod and many other derived custom ROMs is slightly faulty. That code-path is not normally used but the new appt2 build-process creates some outputs that trigger the condition in libandroidfw which then cause the apps to not load.

This means that we just need to patch the library and it fixes the problem:

Download fix for cyanogenmod 12.1.

Download fix for cyanogenmod 13 (untested)

To install this fix you can put it onto your SD card and install via TWRP or whichever bootloader you use. Alternatively you can do it by hand if you have rooted your phone by connecting to your phone’s shell with adb shell and setting up the following:,

Then run the following from your computer to update (after having extracted the zip file):

Then reboot your phone and it should all work again.