Blog

The other day we were asked to quickly throw together an “offline map” for laptop use by crisis responders in the Philippines. We had a couple of hours to create something that was simple and rugged, something that ideally runs on any operating system without having to install software.

We knew that you can easily use OpenLayers or Leaflet plus a directory of pre-produced tile PNG images and then access that with a “file://” type URL from your local browser. This solution can even be run from an USB stick without any software installation, and any halfway recent browser works – be it OSX, Windows, Linux, or even more exotic operating systems. There are many more sophisticated solutions – storing vector data on the device gives you more interactivity and detail at a fraction of the disk space, or if using raster tiles, storing them in MBTiles format is less clumsy because it’s just a single file. All these options require some software installation though. So we decided to go with the old-style technology – but we made one minor tweak that made our solution much more useful on certain systems.

Because our set of tiles might be used on computers that use FAT file systems – crucially, most USB sticks or external USB drives will come formatted with that -, we had to take into account that FAT allocates space in full clusters. Even a small 500-byte PNG will consume one full cluster; and the cluster size on FAT systems can go up to 32 kB. With many high-zoom tiles being near empty, this means we’re wasting a lot of space.

The area we were exporting was about 2 GB of image data. The red and orange lines in the above image show how much disk space this would have used had we stored single 256×256 pixel tiles (the standard size) on a FAT system with 8 kB or 32 kB cluster size. Zoom level 18 tiles for the area in question would have taken 25 GB on a volume with an 8kB cluster size, and almost 100 GB (shooting out of the graph) with a 32kB cluster size.

Instead, we opted for making non-standard tiles with 1024×1024 pixels, 16 times as big as standard tiles. This puts more of the map in every single image, and reduces the waste incurred by “rounding up” to the nearest cluster size dramatically. OpenLayers accepts a “tileSize” parameter and processes the non-standard tiles flawlessly.

Unpacking our tiles down to z18 now only takes 6 GB on a volume with a 32 kB cluster size, and 3 GB if the cluster size is 8 kB – perfect for a “map-on-a-stick”!

If you find yourself in a similar situation – there are probably many ways to generate tiles like these, but we used Tirex which we configured to use large tiles (config option tilesize=1024), and then converted the resulting meta tiles to simple PNGs with the meta2tile utility. Remember that when doing this, you’ll only compute tiles down to a nominal zoom level of 16 which, because each tile has 1024×1024 pixels, gives you the same amount of detail that you’d normally see at zoom level 18. Then, in OpenLayers, you’ll have something like var map = new OpenLayers.Map('map', { tileSize: new OpenLayers.Size(1024, 1024) }) to work with these tiles.

(Hat tip to Matt Amos and Ian Dees on whose ideas we based this.)

(Deutsche Version in eigenem Artikel.)

Geofabrik is a household name in OpenStreetMap circles, but even insiders often ask us: What is it that you do at Geofabrik, apart from all the handy things for the community? What’s your business model?

We like to respond like this: At Geofabrik, we’re essentially craftspeople. Something like carpenters or cabinetmakers. Clients come to us and say “can you build me a cupboard”, and then we discuss the details with them, make them an offer, and when they like what we suggest, then we’ll build the cupboard. Of course the analogy only works so far – at Geofabrik, we might be able to build a standard cupboard for a client, which makes things easier. For another client we might, after a longer discussion, suggest that they get a nightstand instead because it serves them better, or we might offer to store their stuff in one of our cupboards. Or maybe we find out that they’re rather good at woodworking themselves and we just give them a blueprint for building their own cupboard… but what we like about the analogy is the the “honest work” aspect. We’re not “in the cloud”, we’re a solid engineering company, and we’re masters of our trade.

Since its inception in 2007, Geofabrik has done lots of different projects, sold products or services, and we’ve finally come up with a web page that presents a selection of them so that you can see for yourself what it is that we do.

For those that are interested further, Frederik has held a talk at the State of the Map US with an inside view of what it is like to do business in OSM.

And no, there aren’t any cupoards on that page, or in that talk.

(For an English version, see other post.)

Obwohl die Geofabrik in der OpenStreetMap-Szene weithin bekannt ist, werden wir selbst von Insidern oft gefragt: Was macht ihr eigentlich außer all den praktischen Sachen für die Community – was ist Euer Geschäftsmodell?

Wir sagen dann gern: Die Geofabrik ist im Grunde ein Handwerksbetrieb. Sowas wie eine Schreinerei – der Kunde kommt und sagt “bauen Sie mir einen Schrank”, und dann unterhält man sich über die Details, bietet ihm was an, und wenn das für den Kunden gut klingt, dann bauen wir ihm den Schrank. Natürlich trägt die Analogie nur begrenzt – für einige Kunden können wir möglicherweise einen Standard-Schrank bauen, was die Sache vereinfacht. Anderen empfehlen wir vielleicht nach einem längeren Gespräch, dass sie lieber einen Nachttisch kaufen sollen, oder dass sie ihre Sachen doch auch in einem von unseren Schränken aufbewahren können. Oder wir geben ihnen eine Bauanleitung für einen Schrank. Was uns aber an der Analogie gut gefällt, ist das Handfeste: Wir sind nicht “in der Cloud”, sondern ein solider Ingenieurbetrieb, und wir beherrschen unser Handwerk.

Seit ihrer Gründung 2007 hat die Geofabrik eine ganze Reihe verschiedener Projekte durchgeführt und Produkte oder Dienstleistungen verkauft, und wir haben jetzt endlich eine Auswahl davon auf unserer Webseite zusammengestellt, damit man leichter sehen kann, was wir eigentlich machen.

Echte Schränke sind übrigens keine dabei.

Wer sich für einen kleinen Einblick in das Tagesgeschäft der Geofabrik interessiert, dem seien die (englischen) Folien zu Frederiks Talk “Open for Business” von der State of the Map US empfohlen.

We’ve overhauled the download server (download.geofabrik.de) a couple months ago but never really wrote anything on our blog so, let me quickly mention that for a few months now we do not only have a nicer user interface with overview maps and MD5 sums, but we also have selected older versions (usually, data from the past couple of days plus the 1st of each month for the current year) for every extract, as well as incremental updates (diff files). These updates allow you, for example, keep a country extract current by downloading only a fraction of the full file every day, or update an osm2pgsql database that you run with Europe only.

Last weekend we added free daily shape files for Canada’s provinces/territories, and from today we’ll also be serving free daily shape files for all US states. (Larger shape files, e.g. for all of Canada, all of the US, other continents, or the whole world, as well as shape files with more detail or tailored to a specific use case, are made to order for a fee – see http://www.geofabrik.de/data/shapefiles.html).

Would you like to see me explaining the download server (and the technology behind it) in a 20 minute video? Then watch my talk from the SOTM-US conference on Vimeo! The slides are available as well.

The OSM Inspector’s “routing” view has always been the “odd one out” among the OSMI views. It higlights potential problems for routing, and was initially sponsored by Skobbler. At first it was available only for the EU; later we rolled it out world-wide but lacked the resources to actually update the world-wide view regularly, leaving us with a daily updated “Europe” view and a less frequently updated “non-Europe” view. We were hoping to attract someone to sponsor that section of OSMI but since nobody was forthcoming we’ve now put it on a more solid footing (read: better hardware) ourselves. The routing view should now update regularly for the whole world, and is also available as one single layer.

(If you should have bookmarked the old “routing-non-eu” view, drop the “non-eu” to use the unified routing view.)

This is also a good opportunity to thank OSM’s routing experts Pascal Neis (inventor of the original openrouteservice.org, and developer of the “unconnected” and “duplicate” layers in OSMI’s routing view) and Dennis Luxen (the man behind OSRM, and supplier of the “islands” layer in OSMI’s routing view).

The new hardware will allow us to add a couple more improvements to OSMI in the coming months.

Die Geofabrik ist umgezogen. Die neuen Räume haben wir schon im Januar angemietet, aber diverse Renovierungsarbeiten haben uns dann doch noch einen Monat aufgehalten.

Eine kleine Abstellkammer haben wir durch den Einbau einer Klimaanlage in einen Serverraum umfunktioniert. Neben den Servern, die für unsere täglichen Updates der OpenStreetMap-Daten verantwortlich, findet hier auch unser wichtiger Grossformat-Drucker Platz.

Im grossen Büro sah es zwischenzeitlich recht chaotisch aus, weil wir hier alles reingestopft haben, was in den anderen Räumen bei den Umbauarbeiten gestört hat. In einem seltenen Moment bot sich aber auch eine Gelegenheit, den Raum einmal leer zu fotografieren.

In der Küche haben wir alles herausgerissen. Die Verkleidung der Außenwand und das Küchenmobiliar haben wir neu gemacht.

Das, was hier noch eine Werkstatt ist, wird irgendwann mal ein Meetingraum.

Rechtzeitig zum Hack-Weekend dieses Wochenende musste alles fertig sein – und es hat auch gerade so alles geklappt.

Das Hack-Weekend

Plangemäß konnte das Februar-Hackweekend stattfinden. Die neuen Räumlichkeiten wurden standesgemäß eingeweiht.

We’re making two new map styles publicly available on our map web site, map.geofabrik.de – one is the German style that we mentioned a while ago, and the other is our “topo style”:

The topo style is a relatively simple style that blends OSM data with hillshading and contour lines generated from enhanced SRTM data that we have licensed from CIAT. This style is preferred by clients who want to overlay other information (like hiking trails or cycle routes) on the map and have little use for a colourful style like the standard Mapnik map. On the topo style, all roads are grey and we only use two different shades of green for forests and meadows; buildings are added at higher zoom levels.

The topo style is available world-wide but due to the limitations of SRTM data, hillshading and contours are only visible between 60°S and 60°N.

Our map styles are freely viewable on map.geofabrik.de, where tiles come under a CC-BY-SA license. If you want to use these maps in your own application, please contact us or see our web page on Tile Servers.

If we were a little slow in responding to inquiries during the past two weeks, that’s because the majority of Geofabrik staff – read: Christine and myself – were in Japan, combining participation in this year’s State of the Map conference with a little bit of tourism.

Japan is a very interesting country with lots of attractions, and around this time of the year rather warmer and more humid than our home climate. We had been preparing for the trip for quite a while, taking lessons in Japanese and reading a lot about Japan’s culture (and cuisine).

The 'ootori' gate on Mijayima island.

The Golden Pavillion of Kinkakuji Temple in Kyoto.

The conference took place on the campus of Tokyo University. The facilities were excellent; someone mentioned that Tokyo University is not just any university but a rather elite affair where they only take the brightest (or best-connected) students. There were lots of students helping with the organisation on the local team, and they all did that without compensation.

A few of the many volunteer helpers at the conference.

The conference bag and T-Shirt were beautifully designed, with Kanji characters on them that reference historic land surveying in Japan. To make sure we would return our empty food and drink containers, everyone received two specially minted “way” coins which had to be deposited whenever you fetched something to eat or drink, and were returned in exchange for your garbage.

The 'way' coins and the design of the conference bag.

Food at the conference was plentiful, with a large selection of different “Bento Boxes” to select from for lunch.

A Bento Lunchbox.

Calpis, a popular Japanese refreshment drink.

On Thursday evening there was a sponsored welcome party with a mixture of Italian and Japanese food, and on Friday we had a sponsored boat cruise with a multi-course dinner of various Japanese specialties like sashimi and tempura.

Bjarne, Frederik, and Dennis on the Friday night boat cruise.

Talks at the conference covered a wide range of topics; a recurring theme was the use of OSM in disaster response and the humanitarian context, but we also heard about technology and OSM’s social fabric.

The one thing that brought tears to everyone’s eyes was the shooting of the group photo which involved looking at a high-flying unmanned aerial vehicle for extended periods of time 😉

Shooting a group photo at noon on Friday.

We greatly enjoyed our time in Japan and at this impeccably well-organised conference. Thank you to everyone who helped!

Some time in the future I’d like to have a giant button on my desk and every time I press it, a new tile server is ordered, bootstrapped, configured, and seamlessly added to our infrastructure. And if you read the glitzy announcements from various “cloud computing” providers you might think that this is actually not very far away.

However at Geofabrik we’re still doing things like our grandparents did, oh well, maybe not, but we do things the old-fashioned way: We select and order physical hardware, individual machines that have a rack number and a physical location, and we configure them and grill them and test them and run them ourselves. We don’t even have a setup script that “does everything” because if we set up another batch of tile servers in half a year’s time then technology will have progressed again and we’ll be using different hardware or software…

The tile servers we set up are generally of the dynamic kind like the one OSM operates – we have a lot of tiles pre-rendered but we can render any tile, in any style, in any location, on demand if required. If you add a telephone booth on a rock in the Atlantic to OSM and load the respective tile on zoom level 18 from Geofabrik a few minutes later, the telephone booth will be there. (Please don’t do that though. Only if you have verified that the telephone booth really exists.) That means that our servers usually need a lot of SSD storage to keep a copy of OSM’s database. For the current generation, we’re using fairly standard Hetzner servers but equipped with 3x120GB of SSD, just enough to store a full OSM planet for the next year or so.

Today we’ve phased out the previous generation of machines, and also updated to the new Mapnik2 rendering engine. Other than that, our tile servers are running the classic OSM rendering stack – PostgreSQL/PostGIS (with hstore extension for access to seldom-used tags), osm2pgsql, and the Tirex queue manager.

We’ve also updated our map styles to reflect the latest OSM Mapnik style and its German variant, which we have slightly adapted to add German names to local names where they are present in OSM:

We’ll contribute that change to the openstreetmap.de site as well if people want to use it. We’re also working on a couple of new tile styles that our clients can use. Mapnik2 should also allow us to offer nice high-resolution tiles for “retina” displays or even print use in the future.

If you already are a user of Geofabrik tiles then you shouldn’t notice anything, apart from perhaps a minor style difference here or there. If you’re thinking of using our tiles then read more on our web site.

Langsam ist bei uns nach dem Hack-Weekend wieder der Alltag eingekehrt. Ein sehr angenehmes Treffen ganz unterschiedlicher Leute mit ganz unterschiedlichen Projekten. Die Teilnehmer kamen von Bonn bis München und sogar aus Zürich zu uns nach Karlsruhe. Und auch die Karlsruher Community war zahlreich vertreten.

Es wurde viel gefachsimpelt und über Dinge geredet, die man endlich mal in Angriff nehmen sollte (und ein paar davon wurden auch tatsächlich erledigt oder zumindest angefangen): Frederik hat einen neuen Relationen-Layer für die Lizenzwechsel-Anzeige im OSM Inspector gebastelt; Jochen hat ein paar Bugs im Taginfo repariert. Ich habe dem Bigmap-Tool ein Frontend mit Karte verpasst, Sarah wurde mit Nominatim-Verbesserungswünschen überhäuft, und Sven hat am deutschen Mapnik-Stil gearbeitet. Andi und Stephan sah man ODbL-Statistiken ausrechnen, und Christoph musste zwischendurch immer mal wieder vor die Tür, um den neuesten Build der gesprochenen Fahranweisungen in seinem MoNav zu testen, und vieles vieles mehr.

Der große Tintenstrahldrucker der Geofabrik hat dafür gesorgt, dass der eine oder andere ein nettes Souvenir mit nach Hause nehmen konnte.

Am Samstag Abend wurde in grosser Runde im Büro zu Abend gegessen.

Wir hatten viel Spass dabei, und es soll nicht das letzte Wochenende dieser Art gewesen sein: Als Termin für das nächste Hack-Weekend in Karlsruhe hier bei der Geofabrik haben wir den 2. und 3. Juni 2012 anvisiert. Tragt Euch das schonmal in den Kalender ein – wir melden uns rechtzeitig hier und auf der Mailingliste mit einer genaueren Planung.