I use a Python script called mb-util to build the mbtiles files for the NZ Rail Maps webmaps site. It takes the raw tiles and builds what is effectively a SQLite database that encapsulates the millions of little tiles into one file, which is much easier to handle for file storage and uploading, although slightly slower to stream on a web server.
mb-util can be obtained from the GitHub site. There is a multi-step process to follow to get it installed into a standard location so that it can just be typed as a standalone command in Bash:
- First of all download from https://github.com/mapbox/mbutil – click on that green Code button – or use the command line:
git clone git://github.com/mapbox/mbutil.git - Go into the mbutil directory
- To install needs a python script and additional Python libraries such as setuptools or pip. If you haven’t got those, Debian provides them in its package library:
apt install python3-venv python3-pip - Finally you can install mbutil by changing into the directory where you downloaded it to (e.g. ~/mbutil) and running this command:
python3 setup.py install - Then check you can run commands by typing mb-util, it should output the help screen.
When using mb-util I normally use –silent and –do_compression as well as –image_format as appropriate. –silent just tells mb-util not to output informational messages. There is only one level of informational messaging on mb-util and that is a message for every source file scanned (there could be millions of these) as well as the finalisation steps of producing the database. You are probably just as well served to turn off the messages or capture them into a text file if it’s really important to see them.
–do_compression enables mbtiles compression, which can save a lot of file space where there are lots of identical tiles in your source. The most obvious type of tile duplication you are going to see is tiles that are completely blank, and there are a lot of these in the NZ Rail Maps webtiles because the current build process captures relatively large rectangular areas that have a lot of white space around the actual narrow rail corridor. At the moment using the tile compression capability of mb-util is the easiest way to eliminate the tile wastage for a lot of the tile sources that only capture rail tracks or location symbols that have lots of white space around them in the layers. The Non Rail overlay and topographic base layers are examples of where there is less white space and thus, less compression likely to happen as the data in these layers covers areas outside the rail corridor so there will be less compression possible with these layers. As an example of file compression optimisation, I was able to compress the Locations overlay layer from a raw tile size of 2.9 GB, down to 219 MB, which is more than 90% compression. The much smaller size file takes up less space on the web server and also takes less time to upload.
In the case where your source tiles are only one file type, –image_format is actually not necessary. It is only relevant if you have multiple file types in your source folder tree and want to capture only one type of file into your mbtiles file, or the reverse when extracting. The only image format issues I have run into is when Qgis produced the wrong type of tile file when running its the tilesxyz script. Using a find command to find and delete the wrong files then regenerating them from scratch was necessary. At the time of writing this the Topographic layer has had to be rebuilt completely from scratch for every one of the current 10 volumes that are in the webmaps. This is very slow, fortunately the script can just be left to run automatically on a spare computer while getting on with other things. This happened because somehow the file type exported was png instead of jpg, jpg files are smaller resulting in a smaller space usage which is important as compression is not likely to be effective on this mbtiles layer which is currently building to around 6.5 GB.
In order to help streamline the process of building tiles I have now written a build script which calls mb-util with default options and it automatically renames or deletes an existing mbtiles file before calling mbutil as an existing mbtiles file cannot be updated. This built script has now been made into its own executable command instead of just being a Python file that has to be run by calling the Python interpreter and this is easy to accomplish:
- Rename the script removing the .py extension off the file name so it looks just like any other command:
e.g. mv mb-build.py mb-build - Make the script executable
chmod +x mb-build - Add the Python interpreter directive to the first line of the script:
#!/usr/bin/env python3 - Put the script into the path so it doesn’t need to be executed from a specific directory with ./ prefix.
- Make a bin directory in your profile to hold scripts e.g. mkdir -p ~/bin
- copy your script into this new bin directory e.g.
cp mb-build ~/bin - Debian bullseye and possibly earlier versions in ~/.profile for Bash automatically look for this directory and add it to the PATH if it exists, so the next time you open a shell window, the new path is automatically recognised. To have it recognised in a current shell window type in source ~/.profile