Ultra-high performance with PowerDNS

I love PowerDNS, it’s so flexible through the multiple backends that you can do pretty much whatever you want with it. When we had a nightmare weekend at 123-reg several years ago, I designed and built the next generation of the DNS infrastructure on PowerDNS because it would connect straight into our main DNS databases and we could easily change schema etc without having issues.

However, because of this flexibility sometimes PowerDNS has some performance issues. When designing the second generation of high-performance DNS servers for the HostEurope group I tested a number of other opensource servers. The highest performing by far was nsd however it used a lot of memory and wasn’t able to give us the flexibility that we required.

Looking at PowerDNS it had a number of scaling issues – up to 4 or 6 cores it was fine but after that it just couldn’t scale any further. So, working with a number of tools such as valgrind and perf I identified a number of scaling issues with the distributor code and created a series of patches which were rolled in to PowerDNS Authoritative 3.3 and 3.4 and enable PowerDNS to scale easily to 32 or 64 cores. I also came across a Linux kernel issue with locking (actually an improvement in the packet handling code) which meant that the boxes couldn’t handle more than about 250k PPS before the thread contention on the socket lock hit a limit. Fortunately with Linux 3.9 the patches that the Google guys had produced a few years ago finally got incorporated and so the SO_REUSEPORT patch was also added to PowerDNS.

Finally, I found a very high performance key/value database (lmdb) which is very reliable and fast (sqlite was limited to about 3 or 4 cores, BDB always corrupts itself, KoyotoDB is nice but updates cause a lot of contention and I’m not sure if it’s being actively developed any more). And the result is a high performance backend for PowerDNS (lmdbbackend) which can scale to over 1m QPS/server with instant updates and low response times. The best way to avoid an embarrassing DNS DDOS is to have enough capacity to respond to all queries that are thrown at the server, after that you can try to filter the big hitters. Unfortunately many anti-DDOS appliances that we tested turned out to fail quite badly on UDP-based attacks, or attacks at > 1m PPS so it’s often best to not put them in front of infrastructure that may sustain such an attack.

If you want professional consulting and assistance with setting up a high-performance resilient DNS infrastructure please contact me through my consultancy company dns-consultants.com

The little-known toJSON javascript function

I say little-known, perhaps it is not but at least I struggled for a while to figure out how to do a really simple thing – change the format that dates are sent out via a JSON call. Typically a date will be sent in ISO string format eg “2015-01-19T10:30:33.732Z” which is not ideal for my very thin backend service to dump into a database – I usually use a unix timestamp. Rather than having to add a whole load of parsing code and figuring out which was meant to be a date field and which is just a normal string I really wanted any dates to automatically serialize to a unix timestamp (or javascript timestamp which is unixtime*1000). The solution?

Thanks to Javascript’s great prototype concept (this must be a good day – usually it winds me up quite a bit!) this means that any date objects will now get serialized into the format that I wanted. To send them all as unix-times you can just do:

DLI Turkish Language Learning Resources

So, these have been on my server for a few years but they are still a great resource. I believe they were originally written by the US army for troops stationed in Turkey, then improved by some friends of mine these are a 2 year very comprehensive course for learning Turkish. Totally free of charge you can work through them yourself or with a language helper or teacher. You can download them all for free at http://mark.zealey.org/dli_books/ – I’d especially recommend reading the “How to use DLI Turkish Basic Course” document first.

Having both a require and a controller in a directive

I’ve been playing around with AngularJS directives a lot recently trying to modularize out some commonly used bits of code – particularly form validation stuff. As part of this I often want a controller in the directive (so I have child scopes reach in and add elements or trigger events), but I also want to be able to access some parent angular things such as the ‘form’ directive’s controller.

Typically if you have a controller in a directive it gets passed in as the 4th argument to the link function eg:

However if you have a require argument it also gets passed in as the 4th element in link which unfortunately trumps our controller:

I couldn’t find anything else referencing this problem on the internet, so as a hunch I tried (what turned out to be correct) multiple require arguments including requiring the directive itself:

Hopefully this helps someone!

Formatting time inputs nicely with AngularJS

Unfortunately the AngularJS code uses a fixed formatter for the time when a browser doesn’t support the time input natively (eg firefox). The code specifies ‘HH:mm:ss.sss’ which is pretty pointless for most users I would imagine (I’m building a calendering app). Fortunately we can hook into this formatter and fix it, although the ideal would be to override it unfortunately there are local variables that we can’t override particularly easily. So I took the easy route of just modifying the string after it had already been parsed: