Category Archives: Shell scripting

High Performance Linux, Linux, Networking, Shell scripting, Virtual Machines

Complex network setup on Centos/Redhat 8 via Network Manager

1 Comment

Here are some notes, mostly for my future self about how to set up bond + vlan + bridge networks in Centos 8 in order to create a highly resilient virtual machine host server.

Firstly, create the bond interface and set the options:

nmcli con add type bond ifname bond0 bond.options "mode=802.3ad,miimon=100" ipv4.method disabled ipv6.method ignore connection.zone trusted

Note the 802.3ad option – this says we are going to be using the LACP protocol which requires router support, however you can look at the different mode options in the kernel documentation.

Then, we add the required interfaces into the bond – in this case enp131s0f0 and enp131s0f1

nmcli con add type ethernet ifname enp131s0f0 master bond0 slave-type bond connection.zone trusted
nmcli con add type ethernet ifname enp131s0f1 master bond0 slave-type bond connection.zone trusted

If you want to create a standard vlan interface over the top of the bond for vlan tag 123 we can do this just with 1 more command:

nmcli con add type vlan ifname bond0.123 dev bond0 id 123 ipv4.method manual ipv4.addresses a.b.c.d/24 ipv4.gateway a.b.c.d ipv4.dns 8.8.8.8 connection.zone public

Alternatively if you want to set up a bridge on this interface:

nmcli con add ifname br.123 type bridge ipv4.method manual ipv4.addresses a.b.c.d/24 ipv4.gateway a.b.c.d ipv4.dns 8.8.8.8 \
        bridge.stp no bridge.forward-delay 0 connection.zone public
nmcli con add type vlan ifname bond0.123 dev bond0 id 123 master br.123 slave-type bridge connection.zone trusted

If you don’t want the default route or DNS, simply remove the ipv4.gateway and ipv4.dns options above.

If you want to create a bridge which doesn’t have any IP address on the host (just bridges through to the host vm’s) then replace all the ipv4 settings with: ipv4.method disabled ipv6.method ignore.

Linux, Shell scripting, Sysadmin

Percent signs in crontab

Leave a comment

As this little-known ‘feature’ of cron has now bitten me several times I thought I should write a note about it both so I’m more likely to remember in future, but also so that other people can learn about it. I remember a few years ago when I was working for Webfusion we had some cronjobs to maintain the databases and had some error message that kept popping up that we wanted to remove periodically. We set up a command looking something like:

0 * * * * mysql ... -e 'delete from log where message like "error to remove%"'

but it was not executing. Following on from that, today I had some code to automatically create snapshots of a certain btrfs filesystem (however I recommend that for serious snapshotting you use the excellent (if a bit hard to use) snapper tool):

0 5 * * 0 root /sbin/btrfs subvol snap -r /home/ /home/.snapshots/$(date +%Y-%m-%d)

But it was not executing… Looking at the syslog output we see that cron is running a truncated version of it:

May 14 05:00:02 localhost /USR/SBIN/CRON[8019]: (root) CMD (/sbin/btrfs subvol snap -r /home/ /home/.snapshots/$(date +)

Looking in the crontab manual we see:

Percent-signs (%) in  the  command,  unless  escaped
with backslash (\), will be changed into newline characters,
and all data after the first % will be sent to the command
as standard input.

D’oh. Fortunately the fix is simple:

0 5 * * 0 root /sbin/btrfs subvol snap -r /home/ /home/.snapshots/$(date +\%Y-\%m-\%d)

I’m yet to meet anyone who is using this feature to pipe data into a process run from crontab. I’m also yet to meet even very experienced sysadmins who have noticed this behaviour making this a pretty good interview question for a know-it-all sysadmin candidate!

AngularJS, git, grunt, Shell scripting, Web Apps

Using ImageMagick to manipulate PNGs stably

1 Comment

This seems to be an issue that has been talked about in a number of places, however I found it very hard to find the correct solution, which is why I have documented it here.

Often as part of the build process for a webapp you’ll want to take original images and shrink them down to be the correct dimensions (either because they require certain dimensions to be accepted, such as icons, or because you want to save space by stripping out unnecessary data). For JPGs you can do this pretty easily like

convert orig.jpg -strip -resize 500x500 build/out.jpg

The -strip removes any EXIF header information both anonymizing the image and saving potentially a few Kb of asset size.

This process is ‘stable’ because if you repeat it (within the same version of ImageMagick), the resulting file’s data will be identical. This means that you won’t get a new version of the built image in your (git) repository each time you run this command.

However recently when trying to do the same for PNGs (because I required transparency) I noticed that each time they were being built, git was committing a new version into the repository. This is bad news because it both grows the size of the repository by storing pointless identical versions of the file, and also makes it a lot harder tracking through history to see what changed because you have loads of PNG images being committed each time you do a build.

Looking at the output of identify -verbose I could see that the part that was changing each time was below:

  Properties:
    date:create: 2016-12-01T19:24:13+03:00
    date:modify: 2016-12-01T19:24:13+03:00
    png:tIME: 2016-12-01T16:24:13Z

So it appears that PNG format wants to store the update/create time in the image’s header itself. That was what was changing each time.

Searching on the internet I found a number of suggestions about how to strip these out with the convert command, and I saw that the header changed a bit but I couldn’t find any that were also removing the ‘png:tIME’ element. Finally I managed to come up with the following flags which convert the image stably:

convert orig.png -strip -define png:exclude-chunks=date,time build/out.png

The identify command still outputs the date: property sections but these are now being taken from the create time (ctime) and modify time (mtime) of the file itself rather than from the header and so are not stored in version control.

You might be wondering why I don’t just create a lazy build system that only updates the asset if the mod time of the source asset is greater than that of the built asset – if I was doing this on a bigger project that would be the best way, but as this was just for a small project I wanted to do quickly I thought that doing this would be the easiest way!

Linux, Shell scripting

Convert emf files to png/jpg format on Linux

Leave a comment

For a project recently I was sent some excel files with images embedded in them. Not fun. Then I discovered that these were in some random windows format of emf or wmf (depending on whether I exported as .xlsx or .ods from libreoffice) which I think was just wrapping a jpg/png file into some vector/clipart format. Fortunately there’s a great script called unoconv that uses bindings into libreoffice/openoffice to render pretty much anything, however it doesnt seem possible to change page size/resolution. If you use the PDF output though you can get the image simply embedded in the PDF, then use the pdfimages command to extract the original images out of there. Finally some of these had different white borders so I cropped these and converted to png. Full commands below:

rm -fr out; mkdir out;
for i in xl/media/image*.emf; do
  unoconv -f pdf -o t.pdf "$i";
  pdfimages t.pdf out;
  convert out-000.ppm -trim out/$(basename "$i").png;
done

Bash, Linux, Shell scripting

Multi-line commands with comments in bash

Leave a comment

As part of the last post I initially used a bash script to generate the commands to output the individual videos. As per usual, when I finally got fed up of the limitations and syntax issues in bash I switched to a proper programming language, perl. However this time I learnt a neat trick to doing multi-line commands in bash with comments embedded using the array feature of bash. A multi-line command typically looks like:

        melt \
            color:black \
                out=$audiolen \
            ...

However what if you want to add comments into the command? You can’t.

To solve this create an array:

    cmd=(
        # Take black background track for same number of seconds as the MP3, then add 10 seconds of another image
        melt
            color:black
                out=$audiolen
        ...
    )

and then use the following magic to execute it:

"${cmd[@]}"

Using this you can also conditionally add in extra statements if you’re using a pipeline-type program such as imagemagick (convert) or melt: 

    cmd+=(
        # Output to the file
        -consumer avformat
            target="$target"
            mlt_profile="hdv_720_25p"
            f=mpeg acodec=mp2 ab=96k vcodec=mpeg2video vb=1000k
    )