Category Archives: Linux

Hiding the list of sites on your server

Following on from switching my server to use HTTPS/SSL with the excellent Lets Encrypt free SSL certificate authority, after I enabled SSL on the first domain and you connect via HTTPS to any of the other domains I noticed that the browser comes up with an error like “You tried to get to site xxx.com but the certificate was issued for yyy.com”. I’m not sure about the specifics of the HTTPS protocol and certificates, but I don’t really want people easily being able to get a list of all the virtual hosts that are on my server. If you use the default Lets Encrypt client to just get one certificate for all domains on your server then when the certificate is passed to the client they will be able to see all the domains anyway, however if you issue a certificate for each virtual host as per the script in my other post, at least you can restrict what people see.

To make it even more secure and disallow even one valid certificate from being shown by default, you can create a new default vhost which will display a dummy certificate. To do this, you first need to create a random self-signed certificate:

openssl req -x509 -nodes -days 2000 -newkey rsa:2048 -keyout /etc/apache2/default.key -out /etc/apache2/default.crt

Just hit enter to all of those questions. Then, create a file in /etc/apache2/sites-enabled called 00default-ssl.conf and place the following commands in it:

# Default self-signed cert to mask what certificates are on the server
<VirtualHost *:443>
    SSLEngine on
    SSLCertificateFile /etc/apache2/default.crt
    SSLCertificateKeyFile /etc/apache2/default.key
</VirtualHost>

Job done!

A source of random perl crashes

Recently we rolled out a new version of code to production and started observing a lot of segfaults being recorded to do with freeing memory:

Dec  5 14:11:06 XXX kernel: [268032.637417] perl[62948]: segfault at 7ffc3e26dff8 ip 00007f296a21f6ef sp 00007ffc3e26e000 error 6 in libc-2.19.so[7f296a1a3000+19f000]

Researching a bit further, I found that this was caused by some code which was meant to automatically save an object to the database when it was destroyed, if it had not already been saved:

# Catch objects being destroyed that have not been correctly flushed
sub DESTROY {
  shift->flush_to_database
}

Researching further, I discovered that the issue is to do with the way perl handles global destruction. During a normal object destroy phase this code will work just fine; however when the process is exiting and objects are being destroyed you don’t know what order they will be destroyed in. In this case I assume that the database object that we tried sending the object to had already been DESTROY/garbage collected which is what was causing the crash. Fortunately there’s quite a simple work-around:

# Catch objects being destroyed that have not been correctly flushed.
sub DESTROY {
  if( ${^GLOBAL_PHASE} eq 'DESTRUCT' ) {
    warn "Unflushed objects cannot be flushed when process is exiting";
  } else {
    shift->flush_to_database
  }
}

If you want to support perl < 5.14 you should use the Devel::GlobalDestruction module. Moo provides this more easily via the DEMOLISH subroutine:

# Catch objects being destroyed that have not been correctly flushed.
sub DEMOLISH {
  my ($self, $in_destruction) = @_;
  if( $in_destruction ) {
    warn "Unflushed objects cannot be flushed when process is exiting";
  } else {
    $self->flush_to_database
  }
}

We had another similar issue when using Log4perl – if you try to access it from within the DESTROY scope the logger object may already have been destroyed and the process baloons in memory until it is OOM’d. Again just a check if you are in global destruction from within your logging function will handle this just fine.

Easy guide to free SSL with Lets Encrypt

I was excited to hear the other day that the beta of the lets encrypt project had gone live. For those of you that don’t know this is a great project that provides free SSL certificates to any website with the aim of finally moving much of the internet to SSL. However it’s not quite as easy to use as I had hoped, so I documented the process and the issues I ran in to below with the aim of helping some other people out that are wanting to do the same. My setup is apache on linux ubuntu but it should be true for most other flavours of linux.

Firstly, I downloaded and installed letsencrypt on my server as per the instructions:

git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt

I had thought that simply running their advised command of:

./letsencrypt-auto --apache

would convert my apache setup to use https and provide a certificate for each domain on my server. It doesn’t. Whilst it’s a great tool it is still in beta and even after that I’m not sure if the auto configuration functionality would be included. My first issue was that it presented a long list of all the domians in my apache config; some of which were live and some wern’t. You have to select which you want to get a certificate for. However if you select any that are not accessible or not supported (i18n domains ie those beginning xn-- or wildcard domains) then the process dies with an error and you have to restart selecting which domains you want. This is clearly an issue they could fix pretty simply in the code and hopefully that will happen before long. Also, you have to renew the SSL certs every three months and I believe that you have to go through this same process again each time rather than just hitting a renew button.

Another issue that I ran against which is probably more of a design issue is the idea that it issues you one certificate per server. This is fine if you are just hosting one set of domains; however if you have multiple virtual host entries or host multiple sites that are logically separate you probably don’t want to go down this route.

So, because of these issues I gave up on the automatic apache client and wrote a little script that gets a certificate for each virtual server configuration file and excludes i18n domain names, wildcard, localhost and optionally any others. You should be able to just run this script once every few months and it will automatically regenerate all your SSL certificates without having to change any of your config:

for conf in /etc/apache2/sites-enabled/*; do
    cmd=$(perl -nE 'next unless /Server(Name|Alias)/i; next if /xn--|\*|localhost|127|my-other-nonregistered-domain.com.../i; $d=(split " ", $_)[1]; print "-d $d "' $conf)
    if [ ! -z "$cmd" ]; then
        echo $conf
        /root/letsencrypt/letsencrypt-auto certonly --apache $cmd
    fi
done

Just save that to a file like /etc/apache2/regen_ssl.sh and run it with bash, or paste it directly into your shell and you will then get a load of directories under /etc/letsencrypt/live/ with your certificates in. Each directory should be called after the first ServerName configuration option in the particular virtual host configuration file.

To get the certificates auto-renewing you want to edit the crontab to execute this every few months:

# Refetch all certificates every 2 months
4 4 1 */2 * bash /etc/apache2/regen_ssl.sh

BUT, the process doesn’t end there – you have to set up apache for each domain you want to enable ssl on. First, enable SSL globally:

a2enmod ssl
service apache2 restart

Also make sure to update your firewall to allow HTTPS (port 443) traffic. Then for each virtual host you should change the header line from:

<VirtualHost *:80>

to

<VirtualHost *:80 *:443>

Then you need to enable SSL and specify the certificates:

    SSLEngine on
    SSLCertificateFile    /etc/letsencrypt/live/mark.zealey.org/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/mark.zealey.org/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/mark.zealey.org/fullchain.pem

A service apache2 graceful should then mean you can access your site via https.

If you want to force traffic to use HTTPS there are a number of different routes; the correct way is to use HSTS; however if for some reason you discover that your site doesn’t work correctly using HTTP (eg you have iframes or javascript that is not on SSL-enabled servers) you want to be able to quickly revert to not force clients to use SSL. For the initial deployment then I used the amazing apache2 mod_rewrite to handle this:

    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]

If you are running a blog and you just want to force areas where secure information is transferred (eg user login page) to use SSL so people can’t snoop on your password, you could use something like this (for drupal)

    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteCond %{REQUEST_URI} ^/(user|admin)/
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]

If you’re running a site where you have some mobile or legacy clients accessing your API, but you want new web clients forced to use SSL, you could have some config like this:

    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteCond %{REQUEST_URI} !^/api/.*
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]

Blacklisting domains using PowerDNS Recursor

I recently had a client that was wanting to provide a recursive DNS service within his company, however wanted to blacklist a lot of domains to redirect internally. And I mean a lot – over 1 million porn/spam/… domains. It’s one thing to use the excellent unbound recursive DNS software and set up the blocks using the local-data argument, but it was requiring over 6gb of memory to load the list and crashing the process because of that.

As it turns out, yet again PowerDNS to the rescue. I love PowerDNS for its flexibility (so much so that I created a very high performance DNS backend for it and also run a company consulting on DNS deployments).

As the full list of domains would not fit in memory we had to use a database, I took inspiration from a previously posted Lua script which used a tinycdb. Unfortunately tinycdb requires manual compilation and so wasn’t an option, and as the client already had the list of domains in a MySQL deployment we ended up using that. Both PowerDNS authoritative server and recursor can support Lua scripting to do pretty much anything you need, and there are a number of database options that Lua can use. I started off using luadbi as it seemed to have a nicer interface, however unfortunately luadbi only supports lua 5.1 whereas the debian build of PowerDNS uses lua 5.2. This meant switching to use luasql which is a bit lower-level.

So, the following Lua script will redirect any blacklisted subdomains (in the mysql table domains with field name) to 127.0.0.1:

driver = require "luasql.mysql"
env = assert( driver.mysql() )

function preresolve ( remoteip, domain, qtype )
        con = assert(env:connect("database_name", 'username', 'password'))

        domain = domain:gsub("%.$", "")

        while domain ~= "" do
                local sth = assert (con:execute( string.format("SELECT 1 FROM domains WHERE name = '%s'", con:escape( domain )) ) )
                if sth:fetch() then 
                        return 0, { { qtype=pdns.A, content="127.0.0.1" } }
                end

                domain = domain:gsub("^[^.]*%.?", "")
        end

        return -1, {}
end

As establishing a MySQL connection each request is quite a high overhead it might well be worth switching to use SQLite in the future, this should be very simple to do by just changing the driver name and connection string.

Switching to specific KDE activites using shortcut keys

Today I’ve been playing around with an incredibly powerful feature of KDE – activities. This basically means you can switch between entire sets of applications and virtual desktops very easily. As a consultant I usually have several different clients that I work for at the same time plus a set of programs I use personally. KDE Activities allows you to separate these and switch between them very easily. The default keypress to circle around the different activities is meta-tab but I wanted to be able to switch directly to a specific one, however this doesn’t have any specific support in KDE. Fortunately with Linux and KDE being so flexible I found a solution to this from this thread from 2011. As I was implementing it I noticed a number of changes that have been made so here is an updated version of the instructions:

Firstly, get a list of activity ID’s and their human-readable names; from the console:

$ for i in $(qdbus org.kde.kactivitymanagerd /ActivityManager/Activities ListActivities); do echo -n "$i : "; qdbus org.kde.kactivitymanagerd /ActivityManager/Activities ActivityName $i; done
872ecadb-1388-41fe-89c5-4475c7fd137f : AAA
05104b7f-5df9-4ba2-80bf-0b3eca8a501d : BBB
a20e63e3-6ac6-4e0c-9f0f-2eebb3dfcc8b : CCC

Then open up the “Custom Shortcuts” program and create a new shortcut using “Edit -> New -> Global Shortcut -> D-Bus Command”. Fill out the D-Bus settings as follows:

Remote application: org.kde.kactivitymanagerd
Remote object: /ActivityManager/Activities
Function: SetCurrentActivity
Arguments: 872ecadb-1388-41fe-89c5-4475c7fd137f

And you’re done!

Recovering from strange mysql crash

So, yesterday my server started emailing some cron errors over. One particular script that runs every hour was throwing an error about 10 seconds in to running that appeared to be the server hanging up the query. I thought I may have set some timeout too low such that the server wouldn’t allow queries longer than 10 seconds (as this is a web server then nothing should take that long apart from a few analytics scripts that run overnight). Running the query by hand showed the same problem, so I started doing some analysis of the component parts to see which was taking so long. Then, I looked at the timeouts set and noticed:

mysql> show status like "%time%";
+----------------------------+-------+
| Variable_name              | Value |
+----------------------------+-------+
| Uptime                     | 105   |
| Uptime_since_flush_status  | 105   |
+----------------------------+-------+

Hmmm that looks bad. Looking in dmesg confirms that mysql has not been hanging up – it’s actually been crashing!

[504510.549172] init: mysql main process (15317) terminated with status 1
[504510.549186] init: mysql main process ended, respawning

I ran mysqlcheck on the database in question and the server crashed again, even though I was able to query the table fine and even added an index before realizing that there was some issue with it. So, rather than restore from a backup I thought I’ll just clone the table and replace the existing one with it:

mysql> create table t like client_songs;
mysql> insert t select * from client_songs;
mysql> check table t;

New table looks fine, lets do a final update (as it’s continually being inserted into)

mysql> insert t select * from client_songs cs where client_song_ts >= ( select max(client_song_ts) from t ) on duplicate key update select_count= cs.select_count, download_count = cs.download_count, rating = cs.rating, client_song_ts = cs.client_song_ts, print_count = cs.print_count;

Then put it live:

mysql> drop table client_songs;
mysql> rename table t to client_songs;
mysql> check table client_songs;

Everything working again. I wish mysql (5.5.37 from ubuntu 14.04 LTS) was more reliable that’s why I tend to use postgres for new projects these days. It’s really strange that the table could be read fine but one particular query caused it to crash – probably a case of the particular index that was being for the query being corrupted but not the row-data.

Getting HP p1005 and associated LaserJet printers working with Raspberry Pi

The standard hplip doesn’t seem to work very well on the Raspberry Pi with printers like the HP LaserJet 1000/1005/1018/1020/P1005/P1006/P1007/P1008/P1505 as they need a custom firmware downloading to them before being able to print anything. This seems like an issue with the firmware download or page formatting tool which just make the printer give a click sound and do no more. This is annoying because I was wanting to use my Raspberry Pi as a file and print server – my all-in-one HP DeskJet F2420 set up really easily but I was struggling for a long time to make the LaserJet work correctly.

The basic way to make this work is to download and compile the foo2zjs open source printer driver (as the apt sources don’t seem to be working properly at least on my slightly older version of Raspian). After having downloaded and compiled you install using the following commands:

rm /etc/udev/rules.d/*hpmud* # remove any existing hotplug from HP
./getweb P1005 # change depending on your printer
make install
make install-hotplug
make cups

For some reason the install script didn’t seem to correctly install the firmware on my system so I had to work around it:

mkdir -p /usr/share/foo2xqx/firmware/
mv sihp* /usr/share/foo2xqx/firmware/

If you want to test without having to use cups, try the following (after turning your printer off and on again):

foo2xqx-wrapper testpage.ps > testpage.xq # convert to wire-format
cat /usr/share/foo2xqx/firmware/sihpP1005.dl > /dev/usb/lp0 # bang firmware over to printer (wait 5 sec for it to install)
cp testpage.xq /dev/usb/lp0 # print the page (hopefully)

Depending on printer you may need to use foo2zjs-wrapper instead of foo2xqx-wrapper.

Then you have proved it works so you can set it up in cups nice and easily and use it as a remote printer!

Convert emf files to png/jpg format on Linux

For a project recently I was sent some excel files with images embedded in them. Not fun. Then I discovered that these were in some random windows format of emf or wmf (depending on whether I exported as .xlsx or .ods from libreoffice) which I think was just wrapping a jpg/png file into some vector/clipart format. Fortunately there’s a great script called unoconv that uses bindings into libreoffice/openoffice to render pretty much anything, however it doesnt seem possible to change page size/resolution. If you use the PDF output though you can get the image simply embedded in the PDF, then use the pdfimages command to extract the original images out of there. Finally some of these had different white borders so I cropped these and converted to png. Full commands below:

rm -fr out; mkdir out;
for i in xl/media/image*.emf; do
  unoconv -f pdf -o t.pdf "$i";
  pdfimages t.pdf out;
  convert out-000.ppm -trim out/$(basename "$i").png;
done

Solved: Problems with connecting ath9k to 802.11n network

So, I was at a friends house and tried to connect my Qualcomm Atheros AR9285 Wireless Network Adapter (ath9k driver on linux) to their wireless network (D-link DIR-615). It was connecting and then 10 seconds later disconnecting without ever properly establishing a connection. Output as below:

[ 6350.957601] wlan0: authenticate with XXX
[ 6350.971542] wlan0: send auth to XXX (try 1/3)
[ 6350.973230] wlan0: authenticated
[ 6350.976927] wlan0: associate with XXX (try 1/3)
[ 6350.980936] wlan0: RX AssocResp from XXX (capab=0xc31 status=0 aid=3)
[ 6350.981006] wlan0: associated
[ 6350.981376] cfg80211: Calling CRDA for country: GB
[ 6350.984168] ath: EEPROM regdomain: 0x833a
[ 6350.984172] ath: EEPROM indicates we should expect a country code
[ 6350.984174] ath: doing EEPROM country->regdmn map search
[ 6350.984175] ath: country maps to regdmn code: 0x37
[ 6350.984177] ath: Country alpha2 being used: GB
[ 6350.984178] ath: Regpair used: 0x37
[ 6350.984179] ath: regdomain 0x833a dynamically updated by country IE
[ 6350.984207] cfg80211: Regulatory domain changed to country: GB
[ 6350.984209] cfg80211:  DFS Master region: unset
[ 6350.984210] cfg80211:   (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time)
[ 6350.984213] cfg80211:   (2402000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
[ 6350.984215] cfg80211:   (5170000 KHz - 5250000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
[ 6350.984217] cfg80211:   (5250000 KHz - 5330000 KHz @ 40000 KHz), (N/A, 2000 mBm), (0 s)
[ 6350.984218] cfg80211:   (5490000 KHz - 5710000 KHz @ 40000 KHz), (N/A, 2700 mBm), (0 s)
[ 6350.984220] cfg80211:   (57240000 KHz - 65880000 KHz @ 2160000 KHz), (N/A, 4000 mBm), (N/A)
[ 6360.987225] wlan0: deauthenticating from XXX by local choice (Reason: 3=DEAUTH_LEAVING)

Not very nice. I browsed around on the internet but couldn’t find anything obvious, eventually by looking at the different options the ath9k kernel driver accepts I found the ath9k_hw_btcoex_disable option which seems to do the trick.

echo options ath9k nohwcrypt=1 ath9k_hw_btcoex_disable > /etc/modprobe.d/ath9k.conf
sudo rmmod ath9k ath9k_hw ath9k_common
sudo modprobe -v ath9k ath9k_hw ath9k_common

and it all works again.