All posts by Mark

I'm a full-stack Linux consultant from the UK specializing in high performance systems, DNS and databases. I have also written and lead teams producing a number of web/mobile apps. I'm fluent in English and Turkish.

crypt() function potential insecurities with invalid salts

So, yesterday I discovered quite a serious vulnerability in an application using the crypt() function (it was in perl, but perl just calls through to the system library so this application design flaw may be found in anything using the crypt() function for authentication).

Firstly why use crypt() at all? It’s well known that the DES-style crypt is very weak, for example the salt is only 2 characters and it only takes the first 8 characters of the password and ignores everything after that. However the modern glibc implementation of crypt() include a number of very secure hashing functions prefixed with $, particularly $6$ which is hashed SHA-512 and I’d advise everyone to use.

Anyway, back to the issue at hand. Your standard crypt() password check would look like this (taken from perl’s Catalyst::Authentication::Credential::Password module):

return $storedpassword eq crypt( $password, $storedpassword );

ie you use as the salt the encrypted version of the password to encrypt the user specified password, and then check that against the crypted password itself. If they match it means the password was the same.

However in this application for certain pre-created accounts or accounts that had only been logged in to using an oauth mechanism (eg facebook login) the user’s password field was an empty string. This seems reasonable enough – crypt() of a blank password should always return something non-blank (unless you’re using an older version of mysql that has a crypt() inconsistency that I reported 2 years ago). eg

$ perl -le 'print crypt("", "aa")'
aaQSqAReePlq6

Unless that is, that you specify an invalid (or blank) salt:
$ perl -le 'print crypt("any password", "a")'

$ perl -le 'print crypt("any password", "")'


D’oh. This basically means that if you are using a blank password column to specify no password login allowed in reality someone can log in with ANY password! So in the case of the app in question if you knew the registered email address of someone who had a precreated (but locked out) account, or the address that someone who logged in using oauth you could do a login with any password. I’m guessing that because this is not particularly widely known amongst developers there are probably a number of apps where this is possible today but no-one has tested it.

Some workarounds/mitigations:

  • Set your database password field to default to something that is non-blank (eg ‘a’) – even if crypt() classes the salt as invalid it will return blank which will not match this field.
  • Use something that overrides crypt() to auto-generate a salt if none is specified (eg Crypt::Password::Util’s crypt function). This won’t cover you on the case where the specified salt is invalid and so crypt() just returns a blank.
  • Assert that the user’s password field is not blank before allowing login (but you need to make sure you do this everywhere in your application).
  • For language designers: Either make sure that your crypt() function doesn’t ever return blank (throws an error on invalid output for example), OR that it automatically generates a valid salt regardless of whether or not a salt is specified (including the case where a salt is specified but is invalid).

Extracting old weird format audio files

So, I had a friend who has a load of recordings from about 10 years ago which were done on a weird dictophone. The files had the extension .FC4 which according to the internet is a legacy Amiga audio format with no more support. Great.

First thing was to run file on it:

$ file t.FC4 
t.FC4: data

Great. Let’s see if we can do a better job looking at a hex dump (with xxd):

0000000: 4649 4c45 0103 0101 0333 0fff ffff ffff  FILE.....3......
0000010: ffff ffff ffff ffff ffff ffff ffff ffff  ................
0000020: aa10 1f40 01ff ffff ffff ffff ffff ffff  ...@............
0000030: ffff ffff ffff ffff ffff ffff ffff ffff  ................
0000040: ffff ffff ffff ffff ffff ffff ffff ffff  ................
0000050: 4d49 2d53 4334 ffff ffff ffff ffff ffff  MI-SC4..........
0000060: 4456 522d 3030 37ff ffff ffff ffff ffff  DVR-007.........
0000070: 4130 322d 3033 3031 3031 3033 3531 3135  A02-030101035115
0000080: ffff ffff ffff ffff ffff ffff ffff ffff  ................
0000090: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000a0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000b0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000c0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000d0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000e0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000f0: 4643 34ff ffff ffff ffff ffff ffff ffff  FC4.............
0000100: 5249 4646 501f 0000 5741 5645 666d 7420  RIFFP...WAVEfmt 
0000110: 1400 0000 5003 0100 401f 0000 2910 0000  ....P...@...)...
0000120: 1e00 0000 0200 3a00 6461 7461 0000 0000  ......:.data....
0000130: 00fe ffff feff fffe ffff feff ffef 55ff  ..............U.
0000140: feff feff feff feff feff effe feff efef  ................
0000150: efef efef efef efef efef efef 55ef efef  ............U...
0000160: effe feff efef feff effe ffff ffff ffff  ................
0000170: ffff ffff ffff ffff ffff 55ff ffff ffff  ..........U.....
0000180: ffff ffff ffff ffff ffff ffff ffff ffff  ................
0000190: ffff ffff ffff ffff 55ff ffff ffff ffff  ........U.......
00001a0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00001b0: ffff ffff ffff 55ff ffff ffff ffff ffff  ......U.........
00001c0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00001d0: ffff ffff 55ff ffff ffff ffff ffff ffff  ....U...........
00001e0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00001f0: ffff 55ff ffff ffff ffff ffff ffff ffff  ..U.............
0000200: ffff ffff ffff ffff ffff ffff ffff ffff  ................
0000210: 55ff ffff ffff ffff ffff ffff ffff ffff  U...............
...
00006e0: ffff ffff fd88 3fe1 e1e1 e1ef d21e 1e1e  ......?.........
00006f0: 1fff ffff c821 e1e1 bb32 f2d1 55f1 e1e1  .....!...2..U...
0000700: dbac f61f ffff ac15 fe2e 1e1d 8f2f 1e1e  ............./..
0000710: dc2d 4fe1 d9d4 f3ef ed31 55b2 e219 b4fb  .-O......1U.....

So, looks like at offset 0x100 (256) we have something that is a RIFF/WAV file, then the stuff that shows as U is probably a chunk-size block or somesuch. Given the blocks of data afterwards it could probably be 16-bit single channel at a guess. Perhaps something can read that if we cut the initial header off and re-save:

$ xxd -s -256 -r t out.wav
$ file out.wav 
out.wav: RIFF (little-endian) data, WAVE audio, mono 8000 Hz

Ah-ha looks like file has a clue now. Let’s try to play it:

$ mplayer out.wav
...
Requested audio codec family [sc4] (afm=acm) not available.
Enable it at compilation.
Cannot find codec for audio format 0x350.
...

D’oh. Opening as a raw file in audacity shows pretty much white-noise (whereas you’d have expected it to be something vaguely like speach but with blips in every so often if it was any sort of valid PCM or wave type encoding).

After searching around for a long time I discovered this post which talked about a very similar looking header and especially WAV encoding 0x350. This linked to an mplayer plugin with an acm and inf file however the ubuntu version of mplayer doesn’t support w32codecs. I tried installing this in several different ways in a windows 7 vm but couldn’t get it to work.

I then tried compiling mplayer from source only to be greeted with:

cc -MMD -MP -Wundef -Wall -Wno-switch -Wno-parentheses -Wpointer-arith -Wredundant-decls -Werror=format-security -Wstrict-prototypes -Wmissing-prototypes -Wdisabled-optimization -Wno-pointer-sign -Wdeclaration-after-statement -std=gnu99 -Werror-implicit-function-declaration -D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -D_ISOC99_SOURCE -I. -Iffmpeg -O4 -march=native -mtune=native -pipe -ffast-math -fomit-frame-pointer -fno-tree-vectorize -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE  -fpie -DPIC -D_REENTRANT  -I/usr/include/freetype2 -DZLIB_CONST -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -c -o loader/wrapper.o loader/wrapper.S
loader/wrapper.S: Assembler messages:
loader/wrapper.S:31: Error: `pusha' is not supported in 64-bit mode
loader/wrapper.S:34: Error: operand type mismatch for `push'
loader/wrapper.S:38: Error: operand type mismatch for `push'
loader/wrapper.S:40: Error: operand type mismatch for `push'
loader/wrapper.S:45: Error: operand type mismatch for `push'
loader/wrapper.S:46: Error: operand type mismatch for `push'

D’oh. Rather than mess around with trying a 32-bit compile or hacking the assembly I remembered I had a 10-year old laptop lying around with a very old 32-bit install of gentoo. Power it up, install the codec files and it plays them!

I then try to extract some proper PCM WAV file from the FC4 file using mencoder. But mencoder doesn’t support audio-only. I also try using the -dumpstream option in mplayer but that just dumps the encoded audio. So finally I come across the -ao pcm option which puts out a nice plain wav file that I can encode into mp3 or any other format.

Migrating Drupal to new server breaks login

So I just cloned a drupal site onto a new server. It all worked fine but then I couldn’t log in. I found this post which said you need to perhaps change the $cookie_domain variable but that didn’t make any difference. Finally I found the root cause of the problem – mod_rewrite wasn’t active so even though the user/login page was displaying (it was status 404 which redirects through to index.php hence into drupal) it wasn’t accepting POST requests.

a2enmod rewrite
service apache2 restart

Job done.

Add placeholder to all input fields in Drupal

Although this blog is wordpress I’d typically choose Drupal for doing any proper content-based website as it’s so easily flexible and supports multiple languages really easily out of the box. I was doing a site recently where we wanted to have mostly placeholders everywhere rather than form labels. This was unfortunately a bit tricky so I wrote the following code for the theme’s template.php to automatically put labels into the placeholders of input elements:

function THEME_form_alter(&$form, &$form_state, $form_id) {
    if( $form_id == 'user_register_form' ) {
        $form['account']['mail']['#title'] = t('Your Email Address');
    }
    if( $form_id == 'views_exposed_form' ) {
        foreach( $form as $key => &$config ) {
            if( !is_array($config) )
                continue;
            if( array_key_exists('#attributes', $config) )
                $config['#attributes']['placeholder'] = $config['#attributes']['title'];
        }

        $form['keys']['#attributes']['placeholder'] = $form['#info']['filter-keys']['label'];
        $form['#action'] = '';
    }

    //error_log( print_r( $form, TRUE ) . ' ' . $form_id );
    add_placeholder_check($form);
}

function add_placeholder_check(&$array) {
    if( !is_array($array) )
        return;

    foreach( $array as $key => &$config ) {
        if( !is_array($config) )
            continue;
        add_placeholder_check( $config );
        if( substr($key, 0, 1) != '#' && array_key_exists( '#title', $config ) ) {
            if( !array_key_exists('#attributes', $config) )
                $config['#attributes'] = array();
            $config['#attributes']['placeholder'] = $config['#title'];
        }
    }
}

Ultra-high performance with PowerDNS

I love PowerDNS, it’s so flexible through the multiple backends that you can do pretty much whatever you want with it. When we had a nightmare weekend at 123-reg several years ago, I designed and built the next generation of the DNS infrastructure on PowerDNS because it would connect straight into our main DNS databases and we could easily change schema etc without having issues.

However, because of this flexibility sometimes PowerDNS has some performance issues. When designing the second generation of high-performance DNS servers for the HostEurope group I tested a number of other opensource servers. The highest performing by far was nsd however it used a lot of memory and wasn’t able to give us the flexibility that we required.

Looking at PowerDNS it had a number of scaling issues – up to 4 or 6 cores it was fine but after that it just couldn’t scale any further. So, working with a number of tools such as valgrind and perf I identified a number of scaling issues with the distributor code and created a series of patches which were rolled in to PowerDNS Authoritative 3.3 and 3.4 and enable PowerDNS to scale easily to 32 or 64 cores. I also came across a Linux kernel issue with locking (actually an improvement in the packet handling code) which meant that the boxes couldn’t handle more than about 250k PPS before the thread contention on the socket lock hit a limit. Fortunately with Linux 3.9 the patches that the Google guys had produced a few years ago finally got incorporated and so the SO_REUSEPORT patch was also added to PowerDNS.

Finally, I found a very high performance key/value database (lmdb) which is very reliable and fast (sqlite was limited to about 3 or 4 cores, BDB always corrupts itself, KoyotoDB is nice but updates cause a lot of contention and I’m not sure if it’s being actively developed any more). And the result is a high performance backend for PowerDNS (lmdbbackend) which can scale to over 1m QPS/server with instant updates and low response times. The best way to avoid an embarrassing DNS DDOS is to have enough capacity to respond to all queries that are thrown at the server, after that you can try to filter the big hitters. Unfortunately many anti-DDOS appliances that we tested turned out to fail quite badly on UDP-based attacks, or attacks at > 1m PPS so it’s often best to not put them in front of infrastructure that may sustain such an attack.

If you want professional consulting and assistance with setting up a high-performance resilient DNS infrastructure please contact me through my consultancy company dns-consultants.com

The little-known toJSON javascript function

I say little-known, perhaps it is not but at least I struggled for a while to figure out how to do a really simple thing – change the format that dates are sent out via a JSON call. Typically a date will be sent in ISO string format eg “2015-01-19T10:30:33.732Z” which is not ideal for my very thin backend service to dump into a database – I usually use a unix timestamp. Rather than having to add a whole load of parsing code and figuring out which was meant to be a date field and which is just a normal string I really wanted any dates to automatically serialize to a unix timestamp (or javascript timestamp which is unixtime*1000). The solution?

// All Dates want to get sent as timestamps...
Date.prototype.toJSON = Date.prototype.getTime;

Thanks to Javascript’s great prototype concept (this must be a good day – usually it winds me up quite a bit!) this means that any date objects will now get serialized into the format that I wanted. To send them all as unix-times you can just do:

Date.prototype.toJSON = function() { return Math.floor( this.getTime() / 1000 ) }

DLI Turkish Language Learning Resources

So, these have been on my server for a few years but they are still a great resource. I believe they were originally written by the US army for troops stationed in Turkey, then improved by some friends of mine these are a 2 year very comprehensive course for learning Turkish. Totally free of charge you can work through them yourself or with a language helper or teacher. You can download them all for free at http://mark.zealey.org/dli_books/ – I’d especially recommend reading the “How to use DLI Turkish Basic Course” document first.

Having both a require and a controller in a directive

I’ve been playing around with AngularJS directives a lot recently trying to modularize out some commonly used bits of code – particularly form validation stuff. As part of this I often want a controller in the directive (so I have child scopes reach in and add elements or trigger events), but I also want to be able to access some parent angular things such as the ‘form’ directive’s controller.

Typically if you have a controller in a directive it gets passed in as the 4th argument to the link function eg:

angularApp.directive('ngValidSubmit', function() {
    return {
        controller: function($scope, $element, $attrs) {
        },
        link: function(scope, elem, attr, ngValidSubmitCtrl) {
        },
    }
});

However if you have a require argument it also gets passed in as the 4th element in link which unfortunately trumps our controller:

angularApp.directive('ngValidSubmit', function() {
    return {
        require: 'form',
        controller: function($scope, $element, $attrs) {
        },
        link: function(scope, elem, attr, formCtrl) {
        },
    }
});

I couldn’t find anything else referencing this problem on the internet, so as a hunch I tried (what turned out to be correct) multiple require arguments including requiring the directive itself:

angularApp.directive('ngValidSubmit', function() {
    return {
        require: [ 'form', 'ngValidSubmit' ],
        controller: function($scope, $element, $attrs) {
        },
        link: function(scope, elem, attr, ctrls) {
            var formCtrl = ctrls[0];
            var ngValidSubmitCtrl = ctrls[1];
        },
    }
});

Hopefully this helps someone!

Formatting time inputs nicely with AngularJS

Unfortunately the AngularJS code uses a fixed formatter for the time when a browser doesn’t support the time input natively (eg firefox). The code specifies ‘HH:mm:ss.sss’ which is pretty pointless for most users I would imagine (I’m building a calendering app). Fortunately we can hook into this formatter and fix it, although the ideal would be to override it unfortunately there are local variables that we can’t override particularly easily. So I took the easy route of just modifying the string after it had already been parsed:

/* Attach to input time elements and switch their formatting to be HH:MM
 */             
angularApp.directive('ngModel', function( $filter ) {
    return {
        require: '?ngModel',
        link: function(scope, elem, attr, ngModel) {
            if( !ngModel )
                return;
            if( attr.type !== 'time' )
                return;
                    
            ngModel.$formatters.unshift(function(value) {
                return value.replace(/:00\.000$/, '')
            });
        }
    }   
});