One of the features of this blog is that it includes technical posts about the NZ Rail Maps project with which converser.nz blogs are affiliated. These posts are less suitable for a general audience mainly interested in the map content and so are posted on NZTechonverse instead.
The NZ Rail Maps live webmaps project is now almost three months old (various previews and tests go back about another month or two, but are no longer operational) and a lot has been learned in that time about the ins and outs of setting up this kind of thing. The first real change is to be completely hosted on our own web server which is essentially set up from a basic Apache/Php virtual host with nothing on it. However, web tiles are very simple, either just a straight image server or in this case groups of images stored in single files (mbtiles). The main issue has been getting used to being a webmaster again and revising skills not used in quite a number of years.
The nature of webtiles presents a number of different challenges due to their small size and consequently the large number of them needed to cover a map to any extent. Some servers impose a limit as to how many files can be stored on them, with a common one for a web hosting package of this size of storage being around 250,000 inodes (the Linux term for a directory entry). Since the maps have actually got millions of tiles in them so far this was one of the first obstacles to be overcome, the solution being found in the mbtiles format aggregating tiles into sqlite databases, a utility for creating and extracting mbtiles files called mb-util, and Mapbox’s free tileserver PHP script. So far this has worked more or less flawlessly and only results in a slight delay in serving tiles.
Still, there is the problem when building tiles locally, that they waste a lot of disk space. On our server with the disk array using ext4, the block size is 4 KB. As most tiles are much less than this size, this means our currently around 40 GB of raw tiles will use around 88 GB of actual disk space. We can add another 40 GB for the storage of the generated mbtiles files. Backing up which is done regularly to another computer independently of whole-computer backups that are less frequent, only copies the mbtiles files to reduce the overhead of copying a large number of small files.
At the start of the project we had thought it would be desirable to have a versioning server installed to track multiple versions of the maps down to the level of individual tiles. Another early idea was to build the maps into individual volume paths, to be combined before building the mbtiles. Over time, the need to roll back has not proved to be such an imperative, because generally there has not been found a need to deal with issues at a tile level; the use of scripting makes it possible to automate the rebuilding of tiles. The main build strategies used are fivefold:
- Use the mbtiles as an archive. This means when we set out to build new versions of the maps, we still have the option at any time of rolling back to the previous edition in the mbtiles db. Of course, once we have committed a new version of the mbtiles we no longer can do this but we still have a backup on our backup computer until we do the next backup run.
- Have a local web server installed to enable us to check the maps at tiles and mbtiles levels before uploading to the web server. So the maps can be checked for issues as soon as the tiles have been captured.
- Run the capture on a separate computer from the one we edit the maps on. This has its own completely separate copy of the GIS files. This gives a degree of redundancy whilst also eliminating the inconvenience of the editing computer being unable to edit whilst maps are being captured.
- Preview the maps in the GIS before they are captured. The styles needed to create the webmaps are not compatible with the task of editing and producing other map formats, so a separate GIS project is used combining all webmap sources (except for Aerial Photos) into a single project with its own special layer styling. This project can also be used to preview what the maps will look like in order to check for issues either in the GIS data or the styles.
- Run a development version of the website on the live web server. The mbtiles and client changes for the browser are first tested on this development site before going live. When ready to go live, we can log directly into a shell in the web server and simply copy the mbtiles to the live website. This has the great advantage over FTP direct upload of minimising disruption to users (when each mbtiles takes around an hour to upload, it means all of the data in that db is unavailable for that whole period).
Part 2 will look at some of the issues affecting the live webmaps and how they might be able to be addressed.