RSS Feed icon

Blog

Rework of the OSM Inspector Routing View

14.06.2019 | Michael Reichert

Screenshot of the OSM Inspector Routing view around Landau/Pfalz, Germany

We recently rewrote the OSM Inspector Routing View. The old views were used quite a lot by mappers to find potential routing errors. Unfortunately, they made use of PostGIS to find nearby, unconnected roads which meant having to import the whole road network into a PostGIS database every day. To make matters worse, the island and duplicate segment detection was implemented using OSRM which also has relatively heavy system requirements, especially if you do not limit yourself to automobile routing. This made us have a closer look at using GraphHopper to generate the routing view.

The new backend code was implemented from scratch and does not aim to be a one-by-one reimplementation. It uses GraphHopper to read the planet, build up the graph, analyse it and find nearby edges. Our code uses a forked version of GraphHopper 0.12 with the following changes:

  • The original GraphHopper lacks some callback functions in its OSMReader class. Our changes allow us to build additional indexes and populate maps mapping from internal edge or node IDs to OSM object IDs.
  • The PrepareRoutingSubnetworks class which removes islands from the graph is not extensible but we need to write the edges being removed to a GeoJSON file.

In addition to forking, we implemented a new routing profile (“flag encoder” in GraphHopper speak) accepting any road and ignoring access restrictions. It is used to find duplicated edges, unconnected roads, and islands.

The backend code is available on GitHub.

The new routing view has the following groups of layers:

Duplicated Edges

These layers show edges which exist twice in the graph. The blue ones are almost always real errors and should be fixed by comparing the tags and history of both ways. The purple ones are locations where two edges have equal geometry and at least one of them is an area. Opinions whether areas should share nodes with neighbouring roads are diverse among the OSM community. Therefore, these cases are shown in a separate layer at high zoom levels only.

The results of the duplicated edges layer (not involving areas) are similar to those of OSRM. Minor differences are possible where OSRM excluded a road due to access restrictions while our new implementation includes it.

Islands

Screenshot of the OSM Inspector Routing view in Normal, Illinois, US
The use of GraphHopper allows us to identify routing islands for multiple types of vehicles in one go. The OSM Inspector now provides islands for cars and bicycles using slightly modified version of the original profiles of GraphHopper (we use fewer bits to store the speed because island detection does not take travel times or distances into account). In addition, a layer with islands inaccessible to any vehicle is provided. This layer contains fewer entries in total but those contained are more serious because they are not caused by correct or incorrect access restriction tagging in OSM.

Unconnected Nodes

Most development time was spent finding a set of unconnected nodes layers with a proper separation of very likely, likely, potential errors and likely false positives. We are grateful to the folks on the German OSM forum who pointed out many bugs and too high rates of false positives.

Nodes where a single edge ends but with other edges within 15 meters are assigned to one of 6 priority classes. A distance below 2 meters makes a unconnected node appear in the top layer. The further composition of the layers depends on the road class, the distance, and access restrictions. The priority of unconnected nodes involving private access roads is reduced by 1. Service roads and footpaths get low default levels. In addition, the following rules avoid too many false positives:

  • noexit=yes and entrance=* make unconnected nodes disappear.
  • Nearby edges on different layers/levels are ignored.
  • Linear barriers (fences, walls, embankments) are taken into account.
  • Unconnected open ends of parking aisles are ignored if they snap to a parallel edge.
  • If the distance between the open end node and its snap point on the graph is less than twice as long as the beeline distance, no point is shown.

The “snap points” layer shows the snap points of open ends helping to understand the situation.

Not all entries in the unconnected nodes layers are mistakes. Often, adding a linear barrier is helpful. The more blueish a point is, the less likely it is a mistake at all. Don’t feel forced to add noexit=yes to every point the OSM Inspector complains about. You are not mapping for the validator 😉

16. OpenStreetMap-Hackweekend in Karlsruhe

23.10.2018 | Christine Karch

Seit 2012 veranstalten wir zweimal bis dreimal im Jahr ein OpenStreetMap-Hackweekend in unseren Büroräumen.

Es wird am Freitag mit einem gemütlichen Abend in der Kneipe eingeleitet. Im Laufe des Abends trudeln die meisten Gäste ein. Viele haben eine weite Anreise hinter sich. Sie kommen aus allen Teilen Deutschlands, und oft ist auch der ein oder andere Gast aus dem Ausland dabei.

Von Samstag früh bis Sonntag abend geht es dann zur Sache. Es wird diskutiert, manchmal auch Flipcharts und Whiteboards vollgeschrieben. Dann wieder ist alles still, weil alle auf ihre Arbeit am Notebook konzentriert sind. Dazwischen unterbrechen wir unsere Gäste mit Pizza, Keksen, Gummibärchen oder Kaffee. Auch den Samstag Abend verbringen wir gemeinsam im Restaurant.

Aufgrund der Routine, die sich im Lauf der Jahre eingestellt hat, mussten wir dieser Veranstaltung zuletzt nur noch sehr wenig Aufmerksamkeit in der Vor- und Nachbereitung schenken. Die Wiki-Seite ist in zwei Minuten aufgesetzt (Copy und Paste vom letzten Jahr, die Termine werden angeglichen, die alten Teilnehmernamen werden quasi nur pro forma herausgelöscht). Wir recyclen sogar den alten Einkaufszettel.

Doch unser 16. Hackweekend stellte alles auf den Kopf. Ein zu Martin Raifer leicht dahingesagtes “wir können uns ja auf dem nächsten Hackweekend darüber unterhalten” verursachte ein 9-köpfiges Vorbereitungstreffen für die SotM in Heidelberg im nächsten Jahr. Und meine Bemühungen, den Austausch mit den Franzosen zu intensivieren, hatte Früchte getragen. Vincent Privat (aus Toulouse) machte Werbung auf der JOSM-dev-Liste. Sieben JOSM-Entwickler aus ganz Europa (von Österreich, über Polen und Belgien bis Frankreich) meldeten sich an.

Teilnehmer des OSM-Hackweekend Karlsruhe 2018 (Teil 1)

Plötzlich war klar, dass die angemeldeten 32 Gäste überhaupt nicht in unser Büro passen würden. Glücklicherweise hat uns Prof. Dr. Günther-Dieringer ganz kurzfristig und unkompliziert einen Raum in der Hochschule Karlsruhe (University of Applied Sciences) vermittelt. Hierfür danken wir ihm alle ganz herzlich.


Teilnehmer des OSM-Hackweekend Karlsruhe 2018 (Teil 2)

Am Ende war das Hackweekend wieder ein großer Erfolg. Auch diejenigen unter den Besuchern, die befürchtet hatten, dass alles nicht mehr so schön gemütlich sein würde, waren am Ende zufrieden. Der Karlsruher und Heidelberger Teil der SotM-Working-Group haben viele Fragen geklärt, den Zeitplan für das nächste Jahr aufgestellt und das Konferenzformat festgelegt. Ein Protokoll wird demnächst auf den Wiki-Seiten der OSMF zu finden sein. Auch die JOSM-Entwickler, die sich teilweise noch nie getroffen hatten, waren sichtlich produktiv und haben endlos Blätter auf dem Flipchart vollgeschrieben.

Wir freuen uns schon auf das nächste Hackweekend im Februar 2019!

Download Server News

4.05.2018 | Frederik Ramm

As mentioned two months ago, we have made some changes to our download server at download.geofabrik.de. The publicly available files now don’t contain user or changeset information any longer (see below for details). You can still get everything that was there before, but files that do contain user information were moved to a server that requires you to have an OSM login.

The public server continues to be download.geofabrik.de, and the URLs and paths have been unchanged. The new server that requires a login is called osm-internal.download.geofabrik.de, and while it looks largely the same as the public server, the files it offers will have a name component of -internal in order not to confuse them with the public files.

Here’s a table that explains which files to get where:

public server internal server
.osm.pbf files for all regions (original OSM data) yes, without user data yes, with user data
.osc.gz files (diff updates) yes, without user data yes, with user data
.osh.pbf files for all regions (original OSM data with full history) no yes, with user data
.osm.bz2 files (old XML OSM data format) yes, without user data no
.shp.zip files for all regions (ESRI shape files) yes no

Update: backward compatibility with Osmosis and older versions of Libosmium

The change was originally made on 03 May. Unfortunately osmosis and older versions of osmium, and all programs depending on them (e.g. osm2pgsql < 0.96.0) don’t work with PBF files lacking metadata fields. They fail to read PBF files which don’t have any username, user and changeset field set. To work around that, starting 05 May, we’re now writing empty strings to all user fields and zeros to all user and uid fields.

Update: scripted access to extracts and diffs with full metadata

OSM extracs and diffs with full metadata are still available but you have to log in with your OSM account.
For security reasons, it is not enough to pick the cookie out of the developer console of your web browser. The cookie will become invalid after 48 hours and the server will redirect you to authorization form of openstreetmap.org if you send that cookie after it has expired. That’s why you have to retrieve a new cookie every 48 hours or any other time the server forces you to do so.

We have published a client program written in Python (read the documentation) automating the cookie retrieval. The repository containing the client also contains the server software if you are curious or want to use it on your own server providing sensitive data.

This blog post may be updated as things develop.

On May 1st this year, we’ll make the following changes to the standard download.geofabrik.de server:

  • all .osm.bz2, .osm.pbf, and .osc.gz files (regional extracts and diffs) will be without user information (no username, uid and changeset fields), and might* have their timestamp fields diluted
  • .osh.pbf files (full history extracts) will cease to be available

The general functioning of the server and all URLs will remain unchanged. We believe that the majority of users of our download server, who use the extracts to populate rendering, routing, or geocoding engines, will not be affected by the changes.

In case you want to double-check if your software supports the files with missing fields, here are sample files for the UK county of Rutland with user data removed: osm.pbf, osm.bz2, osc.gz

If you need more test data, you can generate these files on your own using the latest (unreleased) Osmium Tool and the latest (unreleased) Libosmium library. The following Osmium Tool command removes all metadata except version and timestamp fields from a file:

osmium cat -o without-user-data.osm.pbf --output-format pbf,add_metadata=version+timestamp input.osm.pbf
(and similar for .osm.bz2 or .osc.gz)

At the same time, we will introduce a new service, osm-internal.download.geofabrik.de, which will be like the “old” download service, offering full history files, regional extracts, and diffs with full user information and un-diluted time stamps. This service will be 100% free of charge like the main download server, but will require an OAuth login with the openstreetmap.org web site and might* also require a click-through user agreement in order to safeguard personal data.

*) Some details are still being worked out.

Relaunch of the Public Transport Views

24.10.2017 | Michael Reichert

We recently rewrote the Public Transport views of our OSM Inspector. The old views had been used rarely and were of limited use, so we went for a full rewrite. While the old views aimed to give an overview on both the infrastructure and the network, the new views are focussed on the routes and stops.

The public transport schema used today was proposed back in 2010/2011 but it was adopted rather slowly. Also, many mappers made lots of mistakes when trying to use the schema because the documentation on the wiki was confusing. To clearly distinguish public transport mapping that follows this schema from old-style public transport mapping, new schema is now called “public transport v2” – even though there really wasn’t ever a version 1.

The new views of OSM Inspector try to support good mapping according to the “public transport v2” scheme, and act as a validator.

Here’s how you can use the two new views to do some public transport mapping:

The Stops View

The Stops view shows stop positions and platforms (multipolygons are not supported yet). Go to an area you are familiar with and check if there are missing bus stops. Use this view on zoom levels 10 and 11 to spot (rural) areas where no bus stops are mapped at all. Just take your car or bicycle and visit them!

screenshot of view "Public Transport - Stops" in Thüringen, Germany

Platforms are rendered in blue, stop positions in green. Bus stops with highway=bus_stop but without a public_transport=* tag are rendered in orange. They are not errors if they are the only object that represent a bus stop (per direction).

The Routes View

The Routes view shows route relations which claim to be v2 route relations (they have the tag public_transport:version=2). The view shows whether a route is valid (green) or not (black) and highlights objects which are either the reason of invalidity or which are located next to the invalidity (orange and purple).

screenshot of view "Public Transport - Routes" in Wuppertal-Elberfeld, Germany

When is a route valid?

The tagging schema defines minimum requirements a route must comply with:

  • The order of the members of the route is important. The list of members is not an unordered collection.
  • One route relation represents only one variant of a line. Therefore, you need at least two route relations for most lines, one for the forward direction and one for the backward direction.
  • The first member is the first stop of the route (where it starts). It either uses the role stop or stop_entry_only.
  • The second member is the second stop of the route and so on.
  • If a stop is mapped by both a stop position and a platform, add both to the relation but the stop first and immediately followed by the platform. Platforms use the roles platform, platform_entry_only or platform_exit_only.
  • The list of the ways the vehicle uses is at the end of the relation member list (i.e. after the last stop). These ways use an empty role.

(Even though many mappers seem to think so, you do not have to map both the stop position and the platform. One is enough. Especially for vehicles where the platform and the vehicle are short, adding stop positions is overkill. But if a stop has both, both have to be added to the route relation.)

A correct route relation of a bus line would look like this (roles are given in parentheses):


- stop position of station 1 (stop)
- platform object of station 1 (platform)
- stop position of station 2 (stop)
- platform object of station 2 (platform)
- ...
- stop position of station n (stop)
- platform of station n (platform)
- first way with highway=* ()
- second way with highway=* ()
- ...
- last way with highway=* ()

That’s how it would look like in JOSM’s relation editor:

screenshot relation editor

As said above, mapping both stops and platforms is not mandatory. One of the two is enough. (If you know the location of the platform and/or the bus stop sign, add it. Otherwise the stop position is a good replacement.) Therefore, the route could also look like this:

screenshot relation editor showing a route relation with less members

What does OSMI currently check?

To get quickly a working validator, we decided not to implement all checks. Here is the list of implemented validation tests:

  • A route must have stops or platforms.
  • A route must have members which are neither stops nor platforms.
  • Member nodes must have one of the following roles: stop, stop_entry_only, stop_exit_only, platform, platform_entry_only, platform_exit_only.
  • Member ways must have one of the following roles: stop, stop_entry_only, stop_exit_only, platform, platform_entry_only, platform_exit_only or an empty role.
  • Member relations must have one of the following roles: platform, platform_entry_only, platform_exit_only.
  • The member list must begin with the list of stops and platforms.
  • There must not be any member with a non-empty role after the first member with an empty role.
  • Train, tram and subway routes must use railway tracks. Buses and trolleybuses must use roads. Trolleybuses need trolley_wire=yes (or trolley_wire:forward/backward=yes). Roads/tracks under construction trigger errors.
  • The route=* tag of a route relation must have one of the following values: train, tram, subway, bus, trolleybus, ferry, aerialway.

There are some ideas for more tests but they haven’t been implemented yet:

  • Correct order of stops and platforms is not checked.
  • It is not checked if all stop positions of a route are located on a member way with an empty role.
  • There is no test which checks if platforms are located next to the route/stop position.
  • Access tags of member ways are not checked.
  • If a platform is followed by another platform, users who follow the schema exactly will read two stations. If both platforms share some nodes, it is very likely an error.

osmi_pubtrans3 is free software, and patches are welcome.