All posts by Mark

I'm a full-stack linux consultant from the UK but currently living in Istanbul. I specialize in rapid development of web/mobile apps using HTML5 typically using Bootstrap, AngularJS, Modern Perl (Catalyst/DBIC) and PostgreSQL but have worked with many other technologies and languages as well. I'm fluent in English and Turkish.

Ripping unfinalized DVDs from Linux

Recently we had some DVD’s with old family videos that had been recorded directly but never finalized, the device that recorded them then broke so it was impossible to finalize them, and it seemed to be pretty much impossible for anything else to read them. So, I figured out a way to recover them using linux…

First I installed the dvd+rw-tools package on ubuntu and used that to get various info and prove that the DVD itself was readable even though nothing on the system could see it as a video disk or filesystem.

I then pulled the entire DVD into a file on the local computer for easier processing later. The command will produce lots of errors (as there are parts of the DVD that are not readable as they were never written to), but the output file (image.iso) will contain a full dump of the DVD eventually

I then put together a short perl script to search through this file for 1kb blocks beginning with the magic tag DVDVIDEO – these seemed to be the starts of individual chapters, which avconv (also called ffmpeg on some distributions) can then extract into proper video/audio.

Save that as extract_dvd_tracks.pl and then run it to extract them as files named like out0.mp4. Note that it won’t process tracks that are less than about 1Mb because there seemed to be a number of small sections like this which we wanted to skip over.

Job done! Note that there are probably many different formats and layouts for unfinalized DVD’s, this may be just one of many but hopefully the principle remains the same.

Extracting all PHP code from a file

In checking over a project recently, I wanted to extract all PHP code from a set of files and combine it into a single output so I could easily assess what was being used. The eventual command I ended up with was as follows, hopefully it will be useful to someone else in the future:

Implementing correct modal navigation with react-router

I’ve recently been converting a big project from jquery mobile into React and Material-UI. One area that seems pretty weak compared to other frameworks is with regards to having a proper mobile-focused infrastructure for navigation. One of the big issues for me was that of modals (for example a dialog, alert or popup page components). For example when you see an alert you want to be able to click on the ok button to dismiss it. If you are on android you also expect to be able to press the back button to dismiss. However if you have separate routes for your modals such that the url changes when they are open, if you refresh the page at that point you will have a modal but no previous state to explore. There seem to be certain hacks with react-router-dom which allow you to change the page but keep the URL the same, assuming you are using BrowserRouter, however because this was a legacy project I want to keep on using HashRouter. So, I whipped up a quick HOC hack to wrap around a modal which will allow both the back navigation to work as expected, and also for refreshes to go to the main page rather than opening the modal.

I already had a standard base class which had the handleClose() method to close off a dialog and signal to the parent that it was done, so I expanded it to include a state listener as below

You then just wrap your component with this – for example

This works because react-router-dom doesn’t attempt to parse query strings etc, so when the app is first loaded you want to just have something to remove any of the ?dialog path hacks:

Dumping all remote mysql queries to a server

For a project recently we needed to see which users were accessing which databases on a remote mysql server. After a little bit of reading and experimenting I came up with this command – hopefully it will be useful for someone in the future.

This uses the excellent wireshark tool to capture traffic. The -f flag specifies a filter to only inspect inbound traffic to the server (as we don’t care about the responses).

The line beginning -T specifies the fields we wish to dump (in a tab-separated fashion), -d ensures that all traffic will attempt to be decoded as mysql. The final -Y flag ensures that only packets with queries or logins will be dumped, rather than TCP overhead or placeholder binding etc.

Note that user information is only given at the beginning of a session. Schema (database name) information may be on that same line, or you may have to scan the queries to look for a USE database type command as there are two different ways of setting the current database that will be accessed.

How to generate good-looking geographical heatmaps

For a project recently I needed to produce a geographical heatmap with millions of data points. I first tried using R with OpenStreetMap rendering, but I couldn’t make the heatmap display as flexibly as I wanted. Then, a friend suggested I try using python with the geopandas library. Even then, inspite of some examples in the manual and even a heatmap-based example, I couldn’t make the map look how I wanted with the interfaces that were exposed.

There were a number of issues:

  • My data was weighted, but there is no support for a weighted heatmap – all points are considered equal
  • I want to have a nicer map which has town names on etc. Ideally the cartodb light style
  • My data was heavily weighted to certain locations, so I wanted to run some logarithmic function on the weights generated by the heatmap algorithm to make the data more easily visible
  • The standard heatmap setup assumes there is nothing other than a map outline so by default is not transparent at all
  • I want to include this data in a report so all I want is a single png image output, rather than having the axis lines, various paddings etc on the graph

So, looking through the geopandas code (which mostly relies on seaborn kdeplot) to generate heatmaps I pulled out various bits into my program so that I could have full control over these processes. Here is a short runthrough of the resulting code:

First, we include libraries, read command-line values and set up the area of our map as a geopandas object

Then, we pull out the bounds both in latlng format and convert the area into Spherical Mercator for web display. We need to plot everything in this so that we can pull in open street map tiles for the background later.

We now read in the CSV and drop any data that we wouldn’t display:

NOTE: I tried doing .to_crs() on the CSV data using normal lat/lng representation to convert into Spherical Mercator but it was very slow. As I’m generating the data itself from PostGIS I just run the conversion in that instead. The data I have is basically using the freely available MaxMind geoip database to generate a list of lat/lng/radius. I then pick a random point within that radius to output to the CSV file using a query like:

Then comes the hard bit – converting these values into the heatmap using an algorithm called KDE. These bits were lifted and simplified from the kdeplot routines mentioned above, but fortunately although not exposed by seaborn the underlying scipy.stats module contains a weighted KDE which is easy to use.

The levels output variable is then the weights of all points on the heatmap we are going to output. We can apply a mathematical function to this (in this case take the quintic root) in order to scale the data appropriately, and output:

By generating a palette ourselves we can add an alpha channel to each aspect so that the underlying map will be visible. However there is quite a lot of noise at the lower levels so we want to hide all of them completely:

We then add in the OSM base tiles, which is a bit of a mess as the contextily library seems a bit immature:

Finally we can tweak some of the output settings and save it as a png file: