Category Archives: Apache

Transparently serving WebP images from Apache

I’ve recently been working on a website where we are creating a tool to customize a product. We have various renders from the designers with lots of transparency and then combine these together on the frontend to produce the customized render. As a result of needing transparency we can’t use the jpeg format so we need to use PNG format, however as this is lossless it means the image sizes tend to be very big. Fortunately the WebP format can compress transparent images including the transparency layer (but this is not set by default). Running the WebP converter with light compression over our PNG assets for this projects produced a set of WebP’s which were in total only 25% of the size of the PNG assets and still a high quality. This means much faster loading for the site, especially when displaying multiple renders of the customized product and its 5-10 layers per render.

However, WebP support is only available in about 70% of the browsers today. Rather than trying to test for it on the client side, it would be great to just keep the browser-side code the same but serve different assets depending on whether the browser supports it or not.

I found a good start for apache support for transparent loading of WebPs on github, however there were a few bugs in the script. Here is the final version that I used – you need to put it under a <VirtualHost> section.

AddType image/webp .webp
<ifmodule mod_rewrite.c>
      # Does browser support WebP? 
      RewriteCond %{HTTP_ACCEPT} \bimage/webp\b

      # Capture image name
      RewriteCond %{REQUEST_URI}  (.*)(\.(jpe?g|png|gif))$

      # if you don't have all jpg/png images available
      # as webp then you want to uncomment the next line
      # so apache first checks if there is a webp file
      # otherwise leave it disabled as it removes the
      # need to query the disk
      RewriteCond %{DOCUMENT_ROOT}%1.webp -f

      # Route to WebP image 
      RewriteRule .* %1.webp [L,T=image/webp]
</ifmodule>

And here is a script to convert all png, jpg or gif files under your image directories to WebP format in such a way that they will be automatically served by the code above.

#!/bin/bash
# Convert all images to WebP
IMAGE_PATHS="assets/ imgs/"
for SRC in $(find $IMAGE_PATHS -name "*.png" -o -name "*.jpg" -o -name "*.jpeg" -o -name "*.gif"); do
    WEBP="${SRC%.*}.webp"
    if [ "$SRC" -nt "$WEBP" ]; then
        echo "Converting to $WEBP"
        convert "$SRC" -define webp:alpha-compression=1 -define webp:auto-filter=true -define webp:alpha-quality=90 -quality 95 "$WEBP"
        
    fi
done

Note the -nt comparison that only updates files if the source has changed. You could add this script to git post-checkout and post-merge hooks to automatically keep your WebP assets in sync with the images in the code (and add a .gitignore entry for *.webp – no need to keep 2 copies of each resource in the repository).

Important note: If you’re using an older version of imagemagick such as on Ubuntu 14.04 (imagemagick 6.7.7), it doesn’t pass the alpha compression arguments through correctly so if you have a lot of transparency you won’t see much in the way of compression happening. Switch the convert line to be something like the below, however you need to remove the gif support as that requires using the gif2webp command to convert:

cwebp -quiet "$SRC" -metadata none -alpha_q 80 -q 90 -o "$WEBP"

Also note that this causes some issues when you have for example a jpg and png of the same base name whose contents are different (I found a few in the old code I inherited). You can find the base name of any of these clashes clashes using the following command:

find $IMAGE_PATHS -name "*.png" -o -name "*.jpg" -o -name "*.jpeg" -o -name "*.gif" | perl -pe 's,\.[^.]+$,\n,' | sort |uniq -d

Apache configuration for WkWebView API service (CORS)

Switching from UIWebView to WKWebView is great, but as it performs stricter CORS checks than standard Cordova/Phonegap it can seem at first that remote API calls are broken in your app.

Basically, before WKWebView does any AJAX request, it first sends a HTTP OPTIONS query and looks at the Access-Control-* headers that are returned to determine if it is able to access the service. Most browsers can be made to allow all AJAX requests via a simple “Access-Control-Allow-Origin: *” header, however WKWebView is more picky. It requires that you expose which methods (GET, POST, etc); and which headers are allowed (eg if you are using JSON AJAX requests you probably need to use a “Content-Type: application/json” header in your main request).

Rather than having to update your API service, you can work around this in a general way using the following Apache config:

    # Required configuration for iOS WkWEBVIEW

    # Allow any location to access this service
    Header always set Access-Control-Allow-Origin "*"

    # Allow the following headers in requests (X-Auth is a custom header, also allow Content-Type to be specified)
    Header always set Access-Control-Allow-Headers "X-Auth, content-type, origin"
    Header always set Access-Control-Expose-Headers "X-Auth"

    # Allow the following methods to be used
    Header always set Access-Control-Allow-Methods "GET, POST, OPTIONS"

    # WkWebView sends OPTIONS requests to get CORS details. Don't tie up the API service with them;
    # just answer them via apache itself
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} =OPTIONS
    RewriteRule .* - [R=204,END]

Note the last line answers any HTTP OPTIONS request with blank content and returns it straight away. Most API services would cause a lot of CPU processing just to handle a single request whether it is a true request or an OPTIONS query, so we just answer this straight from Apache without bothering to send it through to the API. The R=204 is a trick to specify that we don’t return any content (HTTP 204 code means “Success, but no content”). Otherwise if we used something like R=200 it would return a page talking about internal server error, but with a 200 response which is more bandwidth, more processing and more confusing for any users.

Hiding the list of sites on your server

Following on from switching my server to use HTTPS/SSL with the excellent Lets Encrypt free SSL certificate authority, after I enabled SSL on the first domain and you connect via HTTPS to any of the other domains I noticed that the browser comes up with an error like “You tried to get to site xxx.com but the certificate was issued for yyy.com”. I’m not sure about the specifics of the HTTPS protocol and certificates, but I don’t really want people easily being able to get a list of all the virtual hosts that are on my server. If you use the default Lets Encrypt client to just get one certificate for all domains on your server then when the certificate is passed to the client they will be able to see all the domains anyway, however if you issue a certificate for each virtual host as per the script in my other post, at least you can restrict what people see.

To make it even more secure and disallow even one valid certificate from being shown by default, you can create a new default vhost which will display a dummy certificate. To do this, you first need to create a random self-signed certificate:

openssl req -x509 -nodes -days 2000 -newkey rsa:2048 -keyout /etc/apache2/default.key -out /etc/apache2/default.crt

Just hit enter to all of those questions. Then, create a file in /etc/apache2/sites-enabled called 00default-ssl.conf and place the following commands in it:

# Default self-signed cert to mask what certificates are on the server
<VirtualHost *:443>
    SSLEngine on
    SSLCertificateFile /etc/apache2/default.crt
    SSLCertificateKeyFile /etc/apache2/default.key
</VirtualHost>

Job done!

Easy guide to free SSL with Lets Encrypt

I was excited to hear the other day that the beta of the lets encrypt project had gone live. For those of you that don’t know this is a great project that provides free SSL certificates to any website with the aim of finally moving much of the internet to SSL. However it’s not quite as easy to use as I had hoped, so I documented the process and the issues I ran in to below with the aim of helping some other people out that are wanting to do the same. My setup is apache on linux ubuntu but it should be true for most other flavours of linux.

Firstly, I downloaded and installed letsencrypt on my server as per the instructions:

git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt

I had thought that simply running their advised command of:

./letsencrypt-auto --apache

would convert my apache setup to use https and provide a certificate for each domain on my server. It doesn’t. Whilst it’s a great tool it is still in beta and even after that I’m not sure if the auto configuration functionality would be included. My first issue was that it presented a long list of all the domians in my apache config; some of which were live and some wern’t. You have to select which you want to get a certificate for. However if you select any that are not accessible or not supported (i18n domains ie those beginning xn-- or wildcard domains) then the process dies with an error and you have to restart selecting which domains you want. This is clearly an issue they could fix pretty simply in the code and hopefully that will happen before long. Also, you have to renew the SSL certs every three months and I believe that you have to go through this same process again each time rather than just hitting a renew button.

Another issue that I ran against which is probably more of a design issue is the idea that it issues you one certificate per server. This is fine if you are just hosting one set of domains; however if you have multiple virtual host entries or host multiple sites that are logically separate you probably don’t want to go down this route.

So, because of these issues I gave up on the automatic apache client and wrote a little script that gets a certificate for each virtual server configuration file and excludes i18n domain names, wildcard, localhost and optionally any others. You should be able to just run this script once every few months and it will automatically regenerate all your SSL certificates without having to change any of your config:

for conf in /etc/apache2/sites-enabled/*; do
    cmd=$(perl -nE 'next unless /Server(Name|Alias)/i; next if /xn--|\*|localhost|127|my-other-nonregistered-domain.com.../i; $d=(split " ", $_)[1]; print "-d $d "' $conf)
    if [ ! -z "$cmd" ]; then
        echo $conf
        /root/letsencrypt/letsencrypt-auto certonly --apache $cmd
    fi
done

Just save that to a file like /etc/apache2/regen_ssl.sh and run it with bash, or paste it directly into your shell and you will then get a load of directories under /etc/letsencrypt/live/ with your certificates in. Each directory should be called after the first ServerName configuration option in the particular virtual host configuration file.

To get the certificates auto-renewing you want to edit the crontab to execute this every few months:

# Refetch all certificates every 2 months
4 4 1 */2 * bash /etc/apache2/regen_ssl.sh

BUT, the process doesn’t end there – you have to set up apache for each domain you want to enable ssl on. First, enable SSL globally:

a2enmod ssl
service apache2 restart

Also make sure to update your firewall to allow HTTPS (port 443) traffic. Then for each virtual host you should change the header line from:

<VirtualHost *:80>

to

<VirtualHost *:80 *:443>

Then you need to enable SSL and specify the certificates:

    SSLEngine on
    SSLCertificateFile    /etc/letsencrypt/live/mark.zealey.org/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/mark.zealey.org/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/mark.zealey.org/fullchain.pem

A service apache2 graceful should then mean you can access your site via https.

If you want to force traffic to use HTTPS there are a number of different routes; the correct way is to use HSTS; however if for some reason you discover that your site doesn’t work correctly using HTTP (eg you have iframes or javascript that is not on SSL-enabled servers) you want to be able to quickly revert to not force clients to use SSL. For the initial deployment then I used the amazing apache2 mod_rewrite to handle this:

    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]

If you are running a blog and you just want to force areas where secure information is transferred (eg user login page) to use SSL so people can’t snoop on your password, you could use something like this (for drupal)

    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteCond %{REQUEST_URI} ^/(user|admin)/
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]

If you’re running a site where you have some mobile or legacy clients accessing your API, but you want new web clients forced to use SSL, you could have some config like this:

    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteCond %{REQUEST_URI} !^/api/.*
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]