Making a BTRFS read-only snapshot writable

For the past few years I’ve been using btrfs on most filesystems that I create, whilst it’s pretty slow on rotating disk media now that most of my hardware is SSD-based there’s not much of a performance penalty (as long as you’re not using quotas to track filesystem usage). The massive advantage is the ability to have proper snapshot history (unlike any LVM snapshotting hacks that you may suggest) going back a long time with very little overhead. With a tool like snapper (which admittedly is tricky to get set up) you can automatically rotate your snapshots and easily recover any files that you accidentally changed or deleted. Alongside always using git for code repositories, this has saved my skin repeatedly!

Anyway, by default snapper creates read-only snapshots. But when trying to diagnose some database server file corruption I recently experienced I wanted to change a btrfs snapshot from read-only to read-write so I could update some files. After spending a while looking around in the manual and on stack overflow I couldn’t see any way to do this with the kernel/toolchain versions that I was using.

Then, the solution struck me. Simply create a read-write snapshot of the read-only snapshot and work off that. Sometimes it’s very easy to look at the more complicated way of doing things and forget about some of the easier solutions that there might be!

UPDATE: For newer versions of btrfs tools you can toggle read-onlyness of snapshots by running the following command against the subvolume directory:

btrfs property set -ts /path/to/snapshot ro false

Protecting an Open DNS Resolver

As another piece of work I’ve been doing for the excellent Strongarm anti-malware team we recently converted the service so that it can be used to get instant protection wherever you are. Part of this involved my work in converting the core (customized) DNS server into an open resolver. This is usually strongly advised against as you can unwittingly become part of some very serious Denial of Service attacks, however in this blog post I show you how to implement some pretty simple restrictions and limitations to prevent this from happening so you can run a DNS open resolver without running this risk.

Here’s a copy of the article:

One of the challenges of running an open DNS resolver is that it can be used in a number of different attacks, compared to a server that is only allowed access from a known set of IPs. One of the most well known is the DNS amplification attack. As this article explains, “The fact that a DNS reply may be many times larger than a DNS query allows the attacker to achieve amplification by spoofing a relatively small query that is known to generate a large answer in response”. That means that if I can send a DNS question that takes 50 bytes, and I send it pretending to be the computer that I want to attack, and the answer to that question is 1000 bytes, then I have effectively multiplied the traffic that I can attack with by 20 times. Especially as DNSSEC (Domain Name System Security Extensions) become more common, the RRSIG and DNSKEY DNS response codes can contain a lot of data that can be used in this type of attack.

In this post, I’d like to present a couple of ways to easily protect your open DNS resolver from being involved in malware attacks like the DNS amplification attack.

Configuring a DNS Resolver

Many DNS servers, or frontends such as PowerDNS or dnsdist, have the built-in or user-configurable ability to limit some types of attacks. In the case of dnsdist, the loadbalancer sits in front of the DNS servers and monitors the traffic going to and from them in order to blacklist hosts that are abusing the platform.

However, when configuring this within Strongarm’s servers, we wanted the ultimate scalability and flexibility on our DNS infrastructure, so we decided not to use dnsdist but instead use a pure networking approach. Here are a few steps that you can take to protect your DNS infrastructure no matter whether you use a DNS loadbalancer or servers interfacing directly to the internet.

The first step you can take in protecting your server is to ensure that ANY queries cannot be used in an attack. An ANY query returns all the records of a particular domain so naturally it returns more data than a standard query. This is usually easy to configure with an option like ‘any-to-tcp’ in PowerDNS. This setting says that if the recursive server receives an ANY query, it will automatically send back a small redirect: “TCP is required”.

To understand why this helps prevent attacks we need to understand the following three things.

  1. An ANY query will usually return larger responses as it asks for all records under a particular domain.
  2. 99% of the time, an ANY query is not legitimate traffic. Usually, a host will only want a specific type of record such as A or MX.
  3. Whereas it’s easy to spoof UDP traffic, it’s virtually impossible to spoof TCP. This is because establishing a TCP connection requires a 3-way handshake. For example, if the client says “I’d like to open a connection”, and the server says “Okay, you’d like to open a connection, it’s now open”, then the client says, “Thanks, the connection is now open”. While you can spoof the initiation of the connection, when the server says “Okay, you’d like to open a connection, it’s now open,” the host that has been spoofed will reply “What?! I didn’t ask to open a connection!” and it won’t go any further.

Putting this all together, we can see that this can be a very effective preventative measure for abusing an open DNS resolver. Legitimate clients will fall back to using TCP and attackers will simply give up. We can’t use this for all connections because having to do every DNS lookup over TCP would noticeably slow down internet browsing speed, but we can do this easily enough on connections that have a high probability of being attack traffic.

In a similar vein, another useful option for many DNS servers is the ability to limit the size of a return packet over UDP. Typically, you would configure this to say, “If the return packet is more than X bytes, send a TCP redirect and only allow this over TCP.”

Firewall Limiting of Potential Attack Traffic

In addition to doing the above, we implemented a pure firewall-based approach to throttling attack traffic. To do this, we needed to configure our firewall to be stateless, as we described how to do in a previous post.

As opposed to dnsdist or other frontend servers, this allows you to deploy either on a single server or on a frontend router that covers multiple resolvers. This also should be much more efficient as all processing occurs in-kernel via netfilter rather than having to go through a program which may crash or be somehow limited in the speed at which it can process data. As we showed in a previous post this is very efficient at packet processing.

We start by creating an ‘ipset’ of IPs that we have currently blacklisted. We’ll use the ‘timeout’ option to specify that after we have added an IP into this blacklist, it will automatically expire after a certain time. We’ll also limit it to a maximum 100,000 IPs so that an attacker cannot use this to take our server offline:

ipset create throttled-ips hash:ip timeout 600 family inet maxelem 100000

Then, if an IP is on this list, we’ll block it from doing any UDP traffic to our server:

iptables -t raw -A PREROUTING -p udp -m set --match-set throttled-ips src -j DROP

Now for the clever part: we’ll look for DNS responses that are over a certain threshold packet size (700 bytes) and start monitoring them to see the rate at which someone is sending them:

iptables -N LARGE_DNS_PACKET_TRACKING # Create the destination chain
iptables -A OUTPUT -p udp --sport 53 \
        -m length --length 700:0xffff \
        -j LARGE_DNS_PACKET_TRACKING

This points to a new iptables chain called “LARGE_DNS_PACKET_TRACKING” which we’ll set up as follows:

iptables -A LARGE_DNS_PACKET_TRACKING -m hashlimit --hashlimit-mode dstip --hashlimit-dstmask 32 \
   --hashlimit-upto 50kb/min --hashlimit-burst 10 --hashlimit-name large-dns-packets --hashlimit-htable-max 100000 \
   -j ACCEPT

This first rule allows up to 50kb of large DNS responses per minute to a single IP (the 32 means a /32, i.e. a single IP address), and always allows the first 10 large response packets through. Again, it tracks, at most, 100,000 IPs in order to avoid an attack vector against our server.

After a host goes over this threshold, we’ll pass the traffic through to the next stage of the chain:

iptables -A LARGE_DNS_PACKET_TRACKING -j SET --add-set throttled-ips dstip --timeout 600 --exist

This is where the magic happens. If the client breaches the threshold set above, then it will add its IP to the ipset we created earlier, meaning that it will be blocked for 10 minutes. Finally, let’s note this in the system log and then drop the packet:

iptables -A LARGE_DNS_PACKET_TRACKING -j LOG --log-prefix "DNS-amplification protection: "
iptables -A LARGE_DNS_PACKET_TRACKING -j DROP

Conclusions

With the right protection in place, it’s not such a bad thing to run an open DNS resolver on the internet. If you look in your server’s configuration manual, you should find a few options that can also help in preventing attacks. Additionally, we recommend setting up a firewall-based system like I detailed above so that you can limit the amount of traffic you send out. Otherwise, you may easily find your server being disconnected by your ISP for being part of an attack.

Easily extending Cordova’s WebView in your Android app

I’ve recently been working on producing a AngularJS-based financial web app for a client which will also be packaged and distributed via cordova/phonegap. As we are only targeting relatively new browsers, and as we’re aiming to be mobile-first, I decided to use HTML5 inputs such as number as this causes virtual keyboards on iOS and Android to reflect the fact that they can only enter numbers.

This was working fine in Chrome and on various different Android phones via the phonegap build, but then we got feedback that on a certain Android 4.x Samsung phone you could only enter numbers and not a decimal point! This was the first time I’d heard about this bug as normally when I’ve used number inputs before they have only been integral, but it seems that this is a relatively well-known bug on most Samsung Android phones. D’oh.

I searched for quite a while for a plugin or work-around for phonegap, and discovered some code that could be used on a WebView component to work around but no instructions for how to replace this function in the cordova WebView subclass. Fortunately it turned out to be relatively simple, and this is also a generic way of customizing a cordova build’s Android WebView in such a way that you can keep rebuilding the app without it getting overwritten.

Firstly, create a new Java class under your main package called HackedWebViewEngine as at the bottom of this post. The key line is

        this(new HackedWebView(context), preferences);

which changes phonegap’s engine to use your own subclassed WebView rather than using the default one. You need to tell phonegap to use this customised Engine by placing the following in your config.xml file:

    <platform name="android">
       <preference name="webView" value="com.myapp.HackedWebViewEngine" />
    </platform>

Here’s the full code of the Java class to handle the overriding (as an aside, I hate how many imports Java programs need!)

package ...;

import android.content.Context;
import android.text.InputType;
import android.util.AttributeSet;
import android.view.inputmethod.EditorInfo;
import android.view.inputmethod.InputConnection;

import org.apache.cordova.CordovaPreferences;
import org.apache.cordova.engine.SystemWebView;
import org.apache.cordova.engine.SystemWebViewEngine;

public class HackedWebViewEngine extends SystemWebViewEngine {
    public static class HackedWebView extends SystemWebView {
        public HackedWebView(Context context) {
            super(context);
        }
        public HackedWebView(Context context, AttributeSet attrs) {
            super(context, attrs);
        }

        @Override
        public InputConnection onCreateInputConnection(EditorInfo outAttrs) {
            InputConnection connection = super.onCreateInputConnection(outAttrs);

            // Many Samsung phones don't show decimal points on html number inputs by default.
            if ((outAttrs.inputType & InputType.TYPE_CLASS_NUMBER) == InputType.TYPE_CLASS_NUMBER)
                outAttrs.inputType |= InputType.TYPE_NUMBER_FLAG_DECIMAL;

            return connection;
        }
    }

    /** Used when created via reflection. */
    public HackedWebViewEngine(Context context, CordovaPreferences preferences) {
        this(new HackedWebView(context), preferences);
    }

    public HackedWebViewEngine(SystemWebView webView) {
        super(webView);
    }
    public HackedWebViewEngine(SystemWebView webView, CordovaPreferences preferences) {
        super(webView, preferences);
    }
}

Prompt before opening an external link in AngularJS

On a recent project of creating an Angular app which would be both a website and a cordova-packaged app, we had a number of links which opened to external websites (terms and conditions, links to some process flows which couldn’t be contained within the app, etc). However because some of the branding on the sites was very similar to the app itself some test users were getting confused about whether they were still in the app, or had been redirected into a browser.

Because of these issues the client wanted us to create a small popup for some external links that would prompt the user to see if they wanted to move off the site/app. Below is a small angular directive that does this. Usage like:

<a href="https://..." target=_blank prompt-before-open="Do you want to go to an external web page?">...</a>

app.directive('promptBeforeOpen', function($window) {
    return {
        link: function(scope, elem, attr) {
            $(elem).click(function(event) {
                if( !$window.confirm( attr.promptBeforeOpen ) )
                    event.preventDefault();
            });                      
        }                            
    };                               
});

Using ImageMagick to manipulate PNGs stably

This seems to be an issue that has been talked about in a number of places, however I found it very hard to find the correct solution, which is why I have documented it here.

Often as part of the build process for a webapp you’ll want to take original images and shrink them down to be the correct dimensions (either because they require certain dimensions to be accepted, such as icons, or because you want to save space by stripping out unnecessary data). For JPGs you can do this pretty easily like

convert orig.jpg -strip -resize 500x500 build/out.jpg

The -strip removes any EXIF header information both anonymizing the image and saving potentially a few Kb of asset size.

This process is ‘stable’ because if you repeat it (within the same version of ImageMagick), the resulting file’s data will be identical. This means that you won’t get a new version of the built image in your (git) repository each time you run this command.

However recently when trying to do the same for PNGs (because I required transparency) I noticed that each time they were being built, git was committing a new version into the repository. This is bad news because it both grows the size of the repository by storing pointless identical versions of the file, and also makes it a lot harder tracking through history to see what changed because you have loads of PNG images being committed each time you do a build.

Looking at the output of identify -verbose I could see that the part that was changing each time was below:

  Properties:
    date:create: 2016-12-01T19:24:13+03:00
    date:modify: 2016-12-01T19:24:13+03:00
    png:tIME: 2016-12-01T16:24:13Z

So it appears that PNG format wants to store the update/create time in the image’s header itself. That was what was changing each time.

Searching on the internet I found a number of suggestions about how to strip these out with the convert command, and I saw that the header changed a bit but I couldn’t find any that were also removing the ‘png:tIME’ element. Finally I managed to come up with the following flags which convert the image stably:

convert orig.png -strip -define png:exclude-chunks=date,time build/out.png

The identify command still outputs the date: property sections but these are now being taken from the create time (ctime) and modify time (mtime) of the file itself rather than from the header and so are not stored in version control.

You might be wondering why I don’t just create a lazy build system that only updates the asset if the mod time of the source asset is greater than that of the built asset – if I was doing this on a bigger project that would be the best way, but as this was just for a small project I wanted to do quickly I thought that doing this would be the easiest way!

Easily switch between KVM and VirtualBox virtual machines

I’ve done quite a bit of development recently in Android and also been working with a client who has a local virtual environment using Oracle/Sun’s VirtualBox vm. So, I found myself switching between the two platforms quite frequently which unfortunately requires removing and reinstalling kernel modules. So, I wrote the below shell script to switch between the two platforms. Simply put in a directory in $PATH (for me I always have ~/bin as a directory there for my user-local scripts) and call the script something like switch_vm. Use it like:

switch_vm virtualbox
switch_vm kvm

Here’s the script:

#!/bin/bash
VM=$1

if [ "$VM" = "kvm" ]; then
    sudo rmmod vboxpci vboxnetflt vboxnetadp vboxdrv
    sudo modprobe kvm_intel
fi

if [ "$VM" = "virtualbox" ]; then
    sudo rmmod kvm_intel kvm
    for i in vboxdrv vboxpci vboxnetflt vboxnetadp; do
        sudo modprobe $i
    done
fi

KUbuntu double login screen

More a note to myself, but hopefully it will be helpful to others too! After updating to KUbuntu 16.04 a few months back, I had an issue where when I closed the lid to suspend the laptop, and then opened it again it presented both a gnome login/password screen and then a KDE5 password screen. When I used the button, or some other times when I suspended it only showed the KDE 5 Plasma unsuspend prompt. After some googling around I found a SO thread which pointed me in the right direction.

Going in to the org.gnome.desktop.screensaver of dconf-editor I deselected the idle-activation-enabled, lock-enabled and ubuntu-lock-on-suspend options and I’ve not had this issue ever since