All posts by Mark

I'm a full-stack Linux consultant from the UK specializing in high performance systems, DNS and databases. I have also written and lead teams producing a number of web/mobile apps. I'm fluent in English and Turkish.

The hardest task to do well in every frontend/backend system – paging

What is the hardest code to write efficiently in pretty much every frontend and backend system? I think it is paging.

Nearly every system I’ve worked on as full-stack developer has had some sort of reporting whether it is searching and filtering or news-feed style item display, all of these depend on paging. Whilst simple paging is trivial to implement, making it work quickly at scale is a very tricky challenge.

A simple approach (based on SQL, but could be with any backend data store) would be something like:

items = SELECT * FROM <table> WHERE <query> ORDER BY <order> LIMIT <rows>,<offset>;
count = SELECT COUNT(*) FROM <table> WHERE <query>;

It looks simple and it works in small-scale development systems. But there a massive number of issues and inefficiencies with this approach:

  • Whilst you need to redo the items query for every page, the count query only needs to be done once for each <query>. Counting is a very expensive operation as it essentially has to fetch all the rows, unless there are some very well optimized keys on your table. This especially the case if you are running over a table larger than a few megabytes on a mobile device, so you need to cache this as much as possible.
  • Often times as the <order> logic has gotten more and more complex I’ve found myself in a situation where some rows are equal in ordering and so they can move around as you switch between pages or redo the query. Always ensure that the last item of ordering is something which is well ordered to avoid this, for example an integer primary key
  • What happens with a LIMIT/OFFSET type query if you get a row inserted before the OFFSET? All the rows shift forwards by one so when you scroll down on your infinite feed you get a duplicate, or the user goes to the next page and sees a repeat of the last entry of the previous page. What happens when a row prior to the OFFSET is deleted? A row between the pages is not seen by the user. In some databases it may be possible to use a long-lived cursor but often in stateless backends for webapps this is not possible. Wherever possible you should try to get a column on the <order> which is simply increasing or decreasing and redo the previous query with use a > against that to ensure you fetch all the rows from the previous point the user saw regardless of what changed above it. This is easy when sorting by primary key, difficult or even impossible otherwise.
  • How do you handle a laggy backend when the user presses the next button multiple times? Does it re-fire the request for the next page multiple times to the backend (consuming unnecessary resources), ignore any presses after the first (better), or does it proceed forward multiple pages, potentially running past the last page and onto blank pages? I’d usually opt for the second approach but limiting so that it always stops on the last page, but it may vary depending on usage case (would users typically want to skip multiple pages quickly, or always want to page through one-at-a-time), backend latency (ie is the data stored locally or remotely, what is the connectivity like)
  • Following on from the above, if you are firing multiple requests for pages, or for infinite scrolling how do you cope with the fact that responses can come back out-of-order? Many times I see it would simply display the content from the last page returned, but this is open to a number of bugs if you are talking about remote services over wireless
  • How do you efficiently abort in-fly calls and database queries if the user changes page – on a large system this could be a significant performance gain

There are quite a large number of optimizations that can be done with paging code, whether this is with a remote backend or locally, for example:

  • Prioritize getting the first page of data to display, say “Showing results 1-50 of …” while the count is loading
  • Track historical data on how long it takes to generate counts or paging, and especially if running single-threaded (eg on a mobile device), delay the count until the main queries have completed and there is some idle time
  • Is it possible to parallelize the two querys against the database so that the query takes half the real-time for the user? Can you use non-blocking IO to open two database connections, one for each query and return a response when both have completed? Can you run 2 separate queries one for count and one for items and update your pager and results separately? For a few rows in a test system this won’t make any difference, but over millions of rows, especially if you need an accurate count, this can save seconds of time
  • Do you actually need a totally accurate count, or can you approximate it, does your database backend support something like this for queries? This can save significant time and resources
  • Have you set up good indexes and verified that they are correctly being used at the scale that you will have in your live environment? Do you optimize well for the common use case?
  • You don’t always need to bother running a count query – if the items returned are less than the items requested you know that you have finished paging and you can automatically count the total from that. This is true whether on the first on the 100th page.
  • Fetch the number of rows that you want to display plus one, so that you know if there is a next page or not. If the number of items is less than you requested you know there won’t be a next page no matter what the total actually is
  • If running on mobile devices or html, limit the number of items that are displayed on an infinite scroller as this can be very memory intensive. Perhaps allow infinite scrolling over 500-1000 rows and then put a next button at the bottom of the page. Or you can do some trickery to replace the top items with a large blank box but this can be very complex and difficult to do well
  • Don’t display a spinner when infinite scrolling (you are starting to load more items before the user hits the bottom of the list right?) Only display the spinner after a short time period or when the user hits the very bottom of the list to give visual feedback that there are more items loading to display

If this looks like a lot to think about, I don’t disagree. Simple, buggy and inefficient paging is easy to do, but fully-optimized, highly responsive complex paging is very tricky to do well. But if you want to scale your app you need to focus on this.

Creating flexible highlight rules in Ace that change according to editor mode

I’ve only recently started using the excellent Ace editor in projects, but I’m really enjoying it so far. It has very flexible and well designed custom highlighting rulesets that developers can extend to various different syntax. However it does not currently support in-editor spell-checking (I believe this functionality is planned for a release soon though – hopefully!).

One of the situations I’m using the editor in is site which has input from many different languages and character sets. As part of this I wanted to ensure that the language that a given article is claimed to be written in matches the character set of various content sections of the editor. For example if the article is in Thai, but you are using Latin script characters it should highlight them as errors. A big issue we have is that there is a lot of Cyrillic content, but as a number of characters render the same or very similarly in Cyrillic and Latin (for example Р and P), some of the users input mostly Cyrillic, but with the occasional character of Latin. This wouldn’t be a problem if we were just rendering the text, but we are also unidecoding it for searching – in this case Latin P goes to p, but the Cyrillic character Р encodes to r (as it is pronounced). This means that it throws off the searching.

I wrote a script which parses the CLDR data’s exemplarCharacters data to get the expected character sets for a language, adds a few in (as the CLDR is not totally complete unfortunately, especially for languages that use extended Cyrillic sets such as Karakalpak, lacks Traditional Mongolian entirely, and is incomplete for some ideographic languages such as Chinese). It then adds some general punctuation and other characters and generates a javascript regexp for matching characters that should not exist. For example for Armenian, the regexp is /[^\u0020-\u0022\u0025-\u0029\u002b-\u003b\u003f\u005b-\u005d\u00ab\u00b4\u00b8\u00bb\u0531-\u0556\u055a-\u055f\u0561-\u0587\u058a\u2030]/.

So far so good, but how to integrate this with Ace editor? Usually you only have a mode for Ace which specifies the language (eg PHP, HTML), but I don’t want to create a new mode for each language/script that this app wants to support.

Initially I tried basing some code on this code which adds spell checking outside of Ace, however there are a number of limitations with this approach namely it doesn’t integrate with the existing highlighting system so if you are mean to be editing eg HTML document containing only Thai characters it doesn’t know which parts are HTML and which parts should be checked, without redoing the whole highlighting run a second time. It is also needs to reprocess the entire document every change.

Digging around in the Ace source I found that you can actually pass an object into the setMode function which enables you to pass new parameters, such as the invalid-characters regexp for the current language. However the highlighting functions are usually static which means it is complex to update this on-the-fly, especially after normalizing it for something based on the Text Highlight Rules. My solution is as follows:

define('ace/mode/my_highlight_rules', ... {
    var MyHighlightRules = function(regex) {
        var rules = function () {
            this.$rules = {
                start: [ ... ],
                words: [ ... something with regex ... ],
                ...
            this.normalizeRules();
        };

        rules.metaData = { ... };

        oop.inherits(rules, TextHighlightRules);
        return rules;
    };

    exports.MyHighlightRules = MyHighlightRules;
});

define("ace/mode/my_mode", ... {
    var Mode = function (opts) {
        this.HighlightRules = MyHighlightRules(opts.regex);
        ...
    };
});

You can then just do .setMode({ path: 'ace/mode/my_mode', regex: /.../ }) changing the regex for each different language or character set that you wish to validate.

Recovering from unmountable btrfs filesystem issues

Here are some notes of how I recovered most of the data after my btrfs disk got horribly corrupted by bad memory. Fortunately I had upgraded the disk 6 months ago so I was able to start from that image left behind on the old disk, copied over using the excellent btrfs-clone tool.

After that I could restore most of my files to the last backup (a month or two back) and git repositories from the main server. But I still had a number of documents and other bits that I needed to recover.

The first thing prior to formatting the disk (I don’t have another spare fast SSD lying around) was to take a backup of the entire btrfs disk. However it was quite a bit larger than I easily had spare on another disk. So, I stored it in squashfs which reduced size by 50%.

mkdir empty-dir
mksquashfs empty-dir squash.img -p 'sdb3_backup.img f 444 root root dd if=/dev/sdb3 bs=4M'

After that I tested that it was mountable:

mount squash.img /mnt/tmp
btrfs restore -l /mnt/tmp/sdb3_backup.img

And erased and cloned the old btrfs disk to it.

I then started using the btrfs restore tool to try to recover the data. First you need to list the roots, usually the highest number will be the latest snapshot and it may have consistent data:

btrfs restore -l /mnt/tmp/sdb3_backup.img

Then you can get a listing of the files under that root and whether they may be recoverable using the -v -D flags (-v means list files, -D means don’t actually try to restore any data. For example:

btrfs restore -r 290 -v -D sdb3_backup.img /laptop/restore/

If that looks good then you can run the command with a few extra flags to try to get the files back as much as possible:

btrfs restore -r 290 -x -m -i sdb3_backup.img /laptop/restore/

This can take a while but it seems to work well on smaller files. Unfortunately some virtual machine images (60gb or so each) didn’t recover because they had got corrupted in the middle.

If you want to recover only a particular point under the tree you can use the --path-regex parameter to specify this, however writing the regexps is very difficult. Here is a short bit of code which will generate the path regex correctly:

perl -E 'for(@ARGV){ $p = () = m!/!g; s!/!(|/!g; $_.= "(|/.*))" . ")" x $p; say "--path-regex '\''^/(|$_\$'\''" }' 'mark/data/documents'

You can then restore just those files like:

btrfs restore -x -m -i  -r 290 --path-regex  '^/(|mark(|/data(|/documents(|/.*))))$' sdb3_backup.img /laptop/restore/

Diagnosing faulty memory in Linux…

For the past year I’ve had very occasional chrome crashes (segfaults in rendering process) and an occasional bit of btrfs corruption. As it was always easily repairable with btrfs check --repair I never thought much about it, although I suspected it may be an issue with the memory. I ran memtest86 overnight one time but it didn’t show up any issues. There were never any read or SMART issues logged on the disk either, and it happened to another disk within the machine as well.

Recently though I was seeing btrfs corruption on a weekly basis, especially after upgrading to ubuntu 18.04 (from ubuntu 16.04). I thought it may be a kernel issue so I got one of the latest kernels. It seemed to happen especially when I was doing something quite file-system intense, for example browsing some cache-heavy pages while running a vm with a long build process going on.

Then, earlier in the week the hard drive got corrupted again, much more seriously and after spending some time fixing, running `btrfs check –repair` a few times it suddenly started deleting a load of inodes. Force rebooting the machine I discovered that the disk was un-mountable, although later I was able to recover quite a lot of key data from btrfs restore as documented in this post.

memtest86 was still not showing any issues, and so my first thought was that assuming the hard disk was not at fault it may be something to do only when the memory had a lot of contention (memtest86 was only able to run on a single core on my box). I booted a minimal version of linux and ran a multi-process test over a large amount (not all) of the memory:

parallel sh -c 'memtester 1400 10; echo EXIT: $?' -- `seq 8`

where 8 is the number of processor/threads and 1400 is the amount of free memory on the system divided by that number (in my case I was testing 16gb of memory). 10 is the number of runs. It took about 45 min to run once over the 16gb, or about 25 min to run over 8gb (each of the individual sodimms in my laptop).

Within about 10 minutes it started showing issues on one of the chips. I’ve done a bit of research since this and seen that if a memory chip is going to fail then it would usually do it within the first 6 months of being used. However this is a kingston chip that has been in my laptop since I bought it 2 or 3 years back. I added another 8gb samsung chip a year ago and it seemed to be after that that the issues started, however that chip works out as fine. Perhaps adding another chip in broke something, or perhaps it just wore out or overheated somehow…

ESP8266 minimal setup

I’m sure there are many notes out there, but I often get confused about the minimal setup required to run an ESP8266. You actually only need 4 pins connected:

Connect GND to 0v, VCC and EN to +3.3v.

Then connect GPIO15 via a 2-10k (I usually use 3k3) resistor to GND to specify boot from flash.

And you’re good to go.

Obviously in order to do the initial flash of the device you need to connect the TX/RX and also connect GPIO0 to GND.

Whatsapp upgraded, crashes on start

Somehow today my wifes’ phone had managed to upgrade to a new version of WhatsApp. When she opened it it just said that the applicaiton had crashed. This also started happening recently with ‘Google Play Services’ and some other apps on her phone.

(As an aside, this is why I turn off auto-update where at all possible because you never know when something will break)

However after much research and debugging I learnt that the problem is not so much with WhatsApp itself as with the Cyanogenmod (custom ROM) that we use on our phones and will happen increasingly. Fortunately there is a relatively easy way to fix this – skip to the bottom of this article if you want to just fix the issue.

The technical root cause is documented on the google issue tracker and is caused by a change in the way apps are being built when they are upgraded to using the gradle 3 build-chain. It seems to be fixed in the latest versions of google build-tools so hopefully in the next 6 months this problem will go away but for the moment it will only increase as teams upgrade their android build chains. Basically in my quick scanning of the bug ticket the problem is that the implementation of some low-level part of reading an apk package on cyanogenmod and many other derived custom ROMs is slightly faulty. That code-path is not normally used but the new appt2 build-process creates some outputs that trigger the condition in libandroidfw which then cause the apps to not load.

This means that we just need to patch the library and it fixes the problem:

Download fix for cyanogenmod 12.1.

Download fix for cyanogenmod 13 (untested)

To install this fix you can put it onto your SD card and install via TWRP or whichever bootloader you use. Alternatively you can do it by hand if you have rooted your phone by connecting to your phone’s shell with adb shell and setting up the following:,

# Set the system partition read-write
mount -o remount,rw /system

# Create backup copy of the old library in case anything goes wrong
cp /system/lib/libandroidfw.so /system/lib/libandroidfw.so.bkup

Then run the following from your computer to update (after having extracted the zip file):

adb push system/lib/libandroidfw.so /system/lib/libandroidfw.so

Then reboot your phone and it should all work again.

Talk from my dads funeral

A bit of a different topic for the post today, but as several people have asked here is the text of the talk I gave at my dads funeral last week. If you prefer, you can find the audio of the talk on the page with other audio recordings from the funeral.


Looking through some photos of my dad Charles as a child, many people have said how similar we looked. I guess we say “A chip off the old block”. In some respects this is true – we were both interested in computers and work in IT, we both love to travel. In other respects we were very different – I have never liked wine and for the past 8 years we’ve not had any car – I’m not sure those things were ever true of Charles! However in the area that was most precious to my dad – that of following and loving Jesus – he raised all of his children to follow his good example. So, I wanted to share a few words about what my dad taught me about Jesus, and in particular the hope that it gave him as he lived and as he was dying.


I remember my first experience of death. I must have been about 4 or 5 and our family had been given a friends’ pet gerbil to look after while they went on holiday. As my mother Joyce didn’t really like animals this was the first pet we had ever had in the house, and as it turns out, also the last! The first morning I went downstairs to look through the cage bars at it, but I found it on its’ back not moving. I went and asked my mum and she said it had died. I remember we made a small marker and buried it in the back garden – I don’t know what my parents said to our friends when they got home though!

I didn’t really understand death, but I started to be both fascinated and frightened by the idea. Shortly after this incident Charles had his wisdom teeth come through. I remember him lying in bed for a few days in pain and asking my mum “Is daddy dying?” and being scared and fearful of this.


A few years later when I was about 8 years old my great grandfather died. I remember beforehand having nightmares about him dying and was very scared. On several occasions I ran downstairs crying to my dad. He couldn’t explain much, and certainly couldn’t comfort me that my great-granddad would not die or that I would not die, but he read these words of Jesus to me from the gospel of John (14:1)

“Do not let your hearts be troubled. You believe in God; believe also in me. 2 My Father’s house has many rooms; if that were not so, would I have told you that I am going there to prepare a place for you? 3 And if I go and prepare a place for you, I will come back and take you to be with me that you also may be where I am. … 6 I am the way and the truth and the life. No one comes to the Father except through me”

As he read those words and prayed for me to know this comfort, I felt the peace of the Holy Spirit come on me even though I couldn’t have described it like this at the time.


For several weeks before Charles died we feared this would happen, and again, following my dads’ example I turned to the Bible for comfort, reassurance and guidance, and thinking about how Charles would understand these verses. Paul writes to the Corinthians:

1 Cor 15:20 … Christ has indeed been raised from the dead, the firstfruits of those who have fallen asleep. 21 For since death came through a man, the resurrection of the dead comes also through a man. 22 For as in Adam all die, so in Christ all will be made alive. 23 But each in turn: Christ, the firstfruits; then, when he comes, those who belong to him.

In the story told by the Bible, God created humans in a world without death, without pain, without sorrow and with the presence and glory of God in every area of it. But because of our natural inclination to disobedience and sin; decay and death – both physical and spiritual – came into the world. This is the first creation, perfected by God but ruined by humans, symbolized and summarized in this passage by our forefather Adam.

But God could not bare to see his good creation continually be ruined by people and sought out people who wanted to follow Him – not those who thought they were perfect, but ordinary fallen people like you and me and Charles, through whom He could display His Glory and Grace. Throughout thousands of years through the people of Israel he showed that no-one could achieve this on their own merit but rather everyone was in need of a saviour. And throughout this time he promised that one day a new creation, a new beginning, a new dawn would be ushered in by the True King of the world. And in time Jesus Christ, Son of God and Son of Man came into the world and redeemed humanity dying, rising from the dead and ascending into heaven. Just as decay and death entered the world through our forefather Adam, so true renewal and life for all who wish entered through Jesus.

Paul says here that Jesus was the first example of new creation – one day God will finally return to recreate and restore this beautiful artwork of a world to its intended state. Charles didn’t believe that we would all sit in heaven strumming harps, he had a sure and certain knowledge that at the end of time God would return and usher in the new creation with no more pollution, no more ill health, no more grief or pain and no more death.


24 Then the end will come, when he hands over the kingdom to God the Father after he has destroyed all dominion, authority and power. 25 For he must reign until he has put all his enemies under his feet. 26 The last enemy to be destroyed is death.

As Jesus died on the cross he dealt the death blow to Satan and evil by turning their own weapons against them. Satan and evil are still in the world until the new creation, however they know are defeated and their time is limited. Their end will soon come.

35 But someone will ask, “How are the dead raised? With what kind of body will they come?” 36 How foolish! What you sow does not come to life unless it dies. 37 When you sow, you do not plant the body that will be, but just a seed, perhaps of wheat or of something else. 38 But God gives it a body as he has determined, and to each kind of seed he gives its own body.

Here Paul explains what the resurrection will be like, and why physical death is necessary. It’s like the difference between an apple pip and an apple tree. No-one looks at an apple pip and says “if I plant this I’ll get a giant apple pip” they say “if I plant an apple pip I’ll get an apple tree which will make lovely apples”. An apple tree, an apple pip and an apple itself all have the same essence – they are all the same thing – but how they look is totally different from each other. And Paul continues:

42 So will it be with the resurrection of the dead. The body that is sown is perishable, it is raised imperishable; 43 it is sown in dishonor, it is raised in glory; it is sown in weakness, it is raised in power; 44 it is sown a natural body, it is raised a spiritual body.

How can someone come to this more amazing form if they have not been planted, if they have not died and been buried? Charles understood that physical death is a necessary passage – like planting a grain of wheat, an apple pip or some other seed; or perhaps like a caterpillar going into a cocoon and then emerging from the chrysalis as a butterfly leaving behind its shell to rot and decay as it has no more need of it.

In the resurrection Charles will be the same essence of how we knew him in his life but a more perfected and glorious form. In Charles’ final days when he was unable to say more than a few words at a time, unable to move from his hospital bed we were so upset and sad because we remembered a Charles at the peak of his life. But in the resurrection, even the Charles we knew at the peak of his life will seem like this compared to the awesome, amazing, completed, glorified and purified Charles that we shall see then. As my brother read earlier

50 I declare to you, brothers and sisters, that flesh and blood cannot inherit the kingdom of God, nor does the perishable inherit the imperishable. 51 Listen, I tell you a mystery: We will not all sleep, but we will all be changed— 52 in a flash, in the twinkling of an eye, at the last trumpet. For the trumpet will sound, the dead will be raised imperishable, and we will be changed. 53 For the perishable must clothe itself with the imperishable, and the mortal with immortality. 54 When the perishable has been clothed with the imperishable, and the mortal with immortality, then the saying that is written will come true: “Death has been swallowed up in victory.”
55 “Where, O death, is your victory?
Where, O death, is your sting?”
56 The sting of death is sin, and the power of sin is the law. 57 But thanks be to God! He gives us the victory through our Lord Jesus Christ.

One day the end will come, one day the new creation will come, one day death will be rendered dead and Jesus will be shown to be victorious, the true King over all creation. Today as believers in Jesus we can look forward at that time and spit in the face of death – yes it hurts now, but this is part of a greater process of renewal of this world and of defeat of evil. Charles realised this – he wasn’t afraid of death – saddened by the thought but not worried or scared by it because he knew it is a necessary step in the great picture of God’s plan for the universe. He knew that death was not a permanent fixture here.

58 Therefore, my dear brothers and sisters, stand firm. Let nothing move you. Always give yourselves fully to the work of the Lord, because you know that your labor in the Lord is not in vain.

This is what Charles believed in, standing firm, letting nothing move him he gave himself fully to the work of the Lord because he know that this was the one thing that lasts. This didn’t mean he shied away from things of the world – work, enjoyment, relationships – but rather in all that he did wether it was starting his own company and looking after and mentoring his employees, enjoying a good game of Cricket or hosting people at his house and cooking for them; he did this with the aim of pleasing God and as a labour of love to Jesus, living life to the full with Him.

He did not fear death, knowing as he read to me all those years ago that Jesus had said:

“Do not let your hearts be troubled. You believe in God; believe also in me. 2 My Father’s house has many rooms; if that were not so, would I have told you that I am going there to prepare a place for you? 3 And if I go and prepare a place for you, I will come back and take you to be with me that you also may be where I am.”

Charles is in this place now, resting from his labours and enjoying the presence of his king and redeemer, Jesus. And one day all who follow Jesus will meet him in the new creation, with a renewed body, and realise that what we saw in this life was but a shadow of the true Charles, and get to enjoy an eternity together with him. We come today to celebrate his life, to say goodbye to this pip, this chrysalis of a body in the ground, and to assure each other that one day we will see him again in the fullness of life in the glorious new creation.

Styled cross-platform number input (in Angular but applicable to any HTML/CSS app)

Native elements in HTML 5 such as the number input are great, but unfortunately our designers often want them to render the same between different browsers and operating systems. For example on linux/chrome the number input has an up/down spinner on the right hand side which is always visible. On Mac the spinner is only visible on mouse over, on firefox it is rendered differently and on mobile the spinner is not there at all usually. In this case, my designer wanted number input with up/down buttons always available. This was for an order quantity input, so it should be integers from 1 upwards. With bootstrap you can easily add buttons etc to the left or right of an input, however I couldn’t see an easy way in ‘native’ bootstrap to have two buttons stacked as one of the addons, so I created my own html/less. This is for Angular 1, bootstrap 3 and fontawesome but it should be very easy to change for different platforms.

Here’s the HTML

<div class="number-input-group">
    <input class="noscroll" type="number" min=1 max=100 step=1 ng-model="extra.quantity"/>
    <div class="buttons">
        <div ng-click="extra.quantity = extra.quantity + 1"><i class="fa fa-caret-up"></i></div>
        <div ng-click="extra.quantity = extra.quantity > 1 ? extra.quantity - 1 : extra.quantity"><i class="fa fa-caret-down"></i></div>
    </div>
</div>

And the LESS:

.number-input-group {
    display: table; 
    width: 100%;    // fill container - remove if you want it as effectively an inline-block
    position: relative;
    border-collapse: separate;
    border-spacing: 0px;    
                                
    > input[type=number] {  
        display: table-cell;
        width: 100%; // keep biggest
        -moz-appearance:textfield;
        border-right: none;         
        &::-webkit-inner-spin-button, &::-webkit-outer-spin-button {
            -webkit-appearance: none;   
            margin: 0;                  
        }                           
    }                           
    > .buttons {            
        display: table-cell;
        width: 1%;      // shrink to smallest size
        vertical-align: top;
        border: @input-transparent-border-width solid @bespoke-light-black;
        //color: @bespoke-light-black;
        > div { 
            @number-input-group-arrow-box-size: (@input-padding-top * 2 + @input-line-height - @input-transparent-border-width - 1) / 2;
            line-height: @number-input-group-arrow-box-size * 0.8;  // make a bit smaller because ff and chrome mobile add a few px for some reason
            font-size: @number-input-group-arrow-box-size * 1;
            padding: 0 7px;
                            
            &:hover {
                background-color: lighten(@background-color, 15%);
            }

            &:last-child {
                border-top: @input-transparent-border-width solid @bespoke-light-black;
            }
        }
    }
}

Transparently serving WebP images from Apache

I’ve recently been working on a website where we are creating a tool to customize a product. We have various renders from the designers with lots of transparency and then combine these together on the frontend to produce the customized render. As a result of needing transparency we can’t use the jpeg format so we need to use PNG format, however as this is lossless it means the image sizes tend to be very big. Fortunately the WebP format can compress transparent images including the transparency layer (but this is not set by default). Running the WebP converter with light compression over our PNG assets for this projects produced a set of WebP’s which were in total only 25% of the size of the PNG assets and still a high quality. This means much faster loading for the site, especially when displaying multiple renders of the customized product and its 5-10 layers per render.

However, WebP support is only available in about 70% of the browsers today. Rather than trying to test for it on the client side, it would be great to just keep the browser-side code the same but serve different assets depending on whether the browser supports it or not.

I found a good start for apache support for transparent loading of WebPs on github, however there were a few bugs in the script. Here is the final version that I used – you need to put it under a <VirtualHost> section.

AddType image/webp .webp
<ifmodule mod_rewrite.c>
      # Does browser support WebP? 
      RewriteCond %{HTTP_ACCEPT} \bimage/webp\b

      # Capture image name
      RewriteCond %{REQUEST_URI}  (.*)(\.(jpe?g|png|gif))$

      # if you don't have all jpg/png images available
      # as webp then you want to uncomment the next line
      # so apache first checks if there is a webp file
      # otherwise leave it disabled as it removes the
      # need to query the disk
      RewriteCond %{DOCUMENT_ROOT}%1.webp -f

      # Route to WebP image 
      RewriteRule .* %1.webp [L,T=image/webp]
</ifmodule>

And here is a script to convert all png, jpg or gif files under your image directories to WebP format in such a way that they will be automatically served by the code above.

#!/bin/bash
# Convert all images to WebP
IMAGE_PATHS="assets/ imgs/"
for SRC in $(find $IMAGE_PATHS -name "*.png" -o -name "*.jpg" -o -name "*.jpeg" -o -name "*.gif"); do
    WEBP="${SRC%.*}.webp"
    if [ "$SRC" -nt "$WEBP" ]; then
        echo "Converting to $WEBP"
        convert "$SRC" -define webp:alpha-compression=1 -define webp:auto-filter=true -define webp:alpha-quality=90 -quality 95 "$WEBP"
        
    fi
done

Note the -nt comparison that only updates files if the source has changed. You could add this script to git post-checkout and post-merge hooks to automatically keep your WebP assets in sync with the images in the code (and add a .gitignore entry for *.webp – no need to keep 2 copies of each resource in the repository).

Important note: If you’re using an older version of imagemagick such as on Ubuntu 14.04 (imagemagick 6.7.7), it doesn’t pass the alpha compression arguments through correctly so if you have a lot of transparency you won’t see much in the way of compression happening. Switch the convert line to be something like the below, however you need to remove the gif support as that requires using the gif2webp command to convert:

cwebp -quiet "$SRC" -metadata none -alpha_q 80 -q 90 -o "$WEBP"

Also note that this causes some issues when you have for example a jpg and png of the same base name whose contents are different (I found a few in the old code I inherited). You can find the base name of any of these clashes clashes using the following command:

find $IMAGE_PATHS -name "*.png" -o -name "*.jpg" -o -name "*.jpeg" -o -name "*.gif" | perl -pe 's,\.[^.]+$,\n,' | sort |uniq -d

Using wildcards in ssh configuration to create per-client setups

In my role as a linux consultant, I tend to work with a number of different companies. Obviously they all use ssh for remote access, and many require going through a gateway/bastion server first in order to access the rest of the network. I want to treat these clients as separate and secure as possible so I’ll always create a new SSH key for each client. Most clients would have large numbers of machines on their network and rather than having to cut and paste a lot of different configurations together you can use wildcards in your ~/.ssh/config file.

However this is not amazingly easy – as SSH configuration requires the most general settings to be at the bottom of the file. So here’s a typical setup I might use for an imaginary client called abc:

# Long list of server names & IPs
host abc-server1
hostname 10.1.2.3

host abc-server2
hostname 10.2.3.4
...

# Gateway box through which all SSH connections need routing
host abc-gateway
hostname gateway.example.org

# Generic rule to access any box on ABC's network. Eg ssh abc-ip-10.2.3.4 is the same as ssh abc-server2.
# You could also use hostnames like ssh abc-ip-foo.local assuming these resolve from the abc-gateway box.
host abc-ip-*
ProxyCommand ssh abc-gateway -W $(echo %h | sed 's/^abc-ip-//'):22

# Proxy all ssh connections via the gateway machine
host !abc-gateway !abc-ip-* abc-*
ProxyCommand ssh abc-gateway -W %h:22

# Settings for all abc machines - my username & private key
host abc-*
user mark.zealey
IdentityFile ~/.ssh/abc-corp