RSS Feed icon


Have your OSM updates suddenly stopped?

16.03.2017 | Frederik Ramm

bomb_iconIf you’re running a world-wide OSM tile or Nominatim server and you are consuming updates more frequently than once per hour, you might find that your update process got stuck on around 21:00 UTC on March 12, 2017. At that time, a relation (#7066589) was uploaded to OSM that had more than 215-1 members. This is currently allowed by the API, but triggers a failure in the osm2pgsql utility used to update tile server and Nominatim databases. Depending on how your update process is constructed and when it runs, you could be lucky – the relation was deleted 55 minutes later and might therefore never have reached your osm2pgsl. But if your update process is constructed such that it downloads a diff file from OSM and then tries to apply that no matter what, then your process will be stuck because the file containing the large relation can never be successfully applied.

How can I find out if my OSM database is affected?

Your database will not be affected if you do not use the minutely, global updates from OSM. If you use them, you can either check your log files, or look at your database and check if an object created shortly after the problem polygon is there or not. If it is there, then everything is good; if not, then you need to investigate (of course it is possible that your updates are broken for a different reason and maybe for longer).

For a Nominatim database, try this:

nominatim=# select class from place where osm_type=’R’ and osm_id=7066630;
(1 row)

For a tile server database, try this:

osm=# select boundary from planet_osm_polygon where osm_id=-7066630;
(1 row)

In each case, if you don’t get a result back then your database does not include a small boundary relation that was created shortly after the problem relation. (To those who read this long after 16 March 2017, note that the test relation we’re using here could have been changed deleted on OSM meanwhile and that would of course mean it’s normal that it wouldn’t be in your data – check before you panic).

How can I repair the problem?

Nominatim, when run in an update loop with “–import-osmosis-all”, will create a file called “osmosischange.osc” in its data subdirectory, and will try to apply that every time it wakes up from its sleep. Fix the file by removing everything between <relation id="7066589"... and the matching </relation>.

Tile servers will use different ways to update and potentially accumulate changes; one method that we frequently use when setting up servers is to collect un-applied changes in a file named /srv/osm-replicate/var/merged.osc. If you have such a file then both the addition and the deletion of the relation are likely in that file, and you can either delete the relation manually (as described above), or run osmosis --read-xml-change merged.osc --sort-change --simc --write-xml-change sorted.osc && mv sorted.osc merged.osc to squash the relation from the file. If you don’t have such a file, try to find out where your tile server downloads change files to, and get rid of the relation.

Learn more

The issue has led to a patch in osm2pgsql discussed here on GitHub. This patch will make osm2pgsql ignore too-large relations.

Improving Geofabrik OSM Extracts

23.01.2017 | Frederik Ramm

We’ve just rolled out a small enhancement to the OSM extracts available on our download server, concerning the way we deal with objects that cross an extract boundary. In the last couple of years we’ve used a program called osm-history-splitter to create the extracts. This program is based on an older version of the Osmium library and is able to ensure referential integrity on the way/node level only.

Referential Integrity when making extracts

Referential Integrity when making extracts. Cases a and b covered until now, case c additionally covered from now on.

The ways shown as a and b in the sketch above would have been fully contained in the extract cut our along the dotted boundary. A polygon formed by a relation c in which at least one way lies fully outside the boundary, however, would not be constructable from the extract because those ways would be missing.

We’ve now switched to the newest version of osmium, the command line companion to the libosmium library, for producing the extracts and deriving the change files. This allows us to offer referential integrity for boundary-crossing multipolygon relations, while other (non-multipolygon) relations are still handled the same as before. This is important because otherwise a large boundary or route relation crossing a small extract would blow up the size of that extract too much.

With the new complete multipolygon relations (called the “smart” strategy in osmium), the extracts we offer have seen a size increase of just 0.5% on average. Some very small extracts with lots of border-crossing multipolygons have become much larger – the Andorra extract, for example, is now three times as big as before. But we believe it is worth it! If you process nightly updates you’ll likely see a small spike today, with today’s update bringing in all those extra ways needed to complete polygons.

Using the new software has also brought down the overall processing time from around 10 hours to under 4 hours, a 60% speedup. Kudos to Jochen Topf and Mapbox for their tireless improvements to Osmium! This is a nice proof that even in times of the ever-scalable cloud, solid engineering still has its place.

Our little band of Geofabrik people is back from this year’s State of the Map conference. Christine and Frederik, Geofabrik directors, were there as part of the SotM working group (Christine) and speaker (Frederik); Rory, a Geofabrik employee since 2014, even did two talks, and Michael, who currently writes his master thesis at Geofabrik, was a speaker too.

Frederik, Rory, and Michael proudly wearing their #craftmapper t-shirts

SotM started out with a really nice keynote from the US ambassador to Turkmenistan, Allan Mustard, who sounded like a proper hacker when he playfully said: “And then I realized that I was the ambassador, and all these people were working for me, and I could tell them what to do!” (and sent them mapping Ashgabat). We had overheard a few people fearing the worst: “Well, a keynote by a diplomat, that sounds tiring” – they were most positively surprised.

As always, State of the Map provided amazing insights into what goes on in the OpenStreetMap universe. The project is growing steadily, and new people bring new ideas. There are new communities, new fields of endeavour, and new technologies every year – and we were lucky enough with the weather to be able to have lunch on the landuse=grass outside!

Rory presented on vector tile work he’d been doing for Geofabrik, as well as on his long-time project of Irish townland mapping. Michael was at his best explaining the nuts and bolts of railway mapping (including exactly which places in which trains were the best spots). Frederik held forth, as he likes to do, on mechanical edits in OpenStreetMap and why he didn’t encourage them.

The social event was held at the “Event Brewery” serving nice food (including Belgian waffles) in a great atmosphere.

We all had a good time and we’d like to thank the whole SotM team for running the conference. We’re looking forward to next year. We even heard the idea of an African SotM being discussed.

New Shapefiles

12.08.2016 | Frederik Ramm

For years, the Geofabrik download server has been, and still is, the go-to resource for anyone wanting OSM data neatly sliced up in country or even smaller bites, and for those not into OSM tools we’ve also offered free shape files to open OSM data quickly and easily in tools like ArcGIS, QGIS and so on.

The shape files have always been “quick and dirty”, in a way, designed more for a quick preview than for serious use. We kept them simple because we wanted to be able to refresh them daily, and also because we get some income from selling more complex shape files with a fuller feature set.

One of the main shortcomings of the free shapes was that they didn’t support multipolygons – a data type that is becoming more and more frequent in OSM, being used not only for large forests but also for buildings that have a courtyard or other comparably smaller structures.

We’ve now come up with a new structure for our free shape files that is a slightly reduced version of the shape files we sell for money, but still one that has all the major objects and properties, nicely sorted into layers, and support for multipolygons.

Starting today, you’ll find “preview” links to the new shape files on our download server. We’ll keep the old files around for a while though because we know that many people are used to them. Here’s more info on the move and here’s the detailed layer description for the free shape files.

The paid shape files also receive an upgrade (from v0.6 to v0.7), covering more features and more properties than before, and also for the first time exporting routes on a separate layer. Here’s the new detailed layer description for the 0.7 shape files.

Because of the similarity between the free and paid shape files, it will now be easy for someone working with the free shapes to upgrade to the paid version if they want more features or properties, or if they need a larger area (the free shapes are only available for smaller areas).

A frequent problem in OpenStreetMap is that someone dreams up a project that involves generating good routes for pedestrians, decides that OSM’s standard way of routing pedestrians along streets isn’t good enough, and proceeds to add every single pavement (or sidewalk, for our American friends) as a highway=footway – causing a headache for other mappers who now have to move three lines instead of one when they refine the road geometry, among other things.

A very promising solution to this problem has been developed by Nathanel Lang in his Bachelor Thesis (at HfT Stuttgart) available in German here: He’s synthesizing footways on both sides of streets (unless the highway type or particular tags say otherwise), and then takes care to properly link them to the rest of the network – or drop them again if it is found that footways have already been mapped as separate geometries. Road crossings are also synthesized because it is assumed that smaller roads may be crossed by pedestrians anywhere. That way, he can generate a pedestrian-friendly routing graph to be used as the basis for routing – without actually adding lots of unnecessary extra geometries to OSM.

The software that generates the routing graph is called “Pedro”, and available from GitHub at the URL given above. There are some open ends that might need attention before it can be used in a world-wide production environment, but already it is an excellent demonstration of the possible future of pedestrian routing with OSM.

(The thesis was mentored by Frederik Ramm at Geofabrik.)