5.03.2012 | Christine Karch
Langsam ist bei uns nach dem Hack-Weekend wieder der Alltag eingekehrt. Ein sehr angenehmes Treffen ganz unterschiedlicher Leute mit ganz unterschiedlichen Projekten. Die Teilnehmer kamen von Bonn bis München und sogar aus Zürich zu uns nach Karlsruhe. Und auch die Karlsruher Community war zahlreich vertreten.
Es wurde viel gefachsimpelt und über Dinge geredet, die man endlich mal in Angriff nehmen sollte (und ein paar davon wurden auch tatsächlich erledigt oder zumindest angefangen): Frederik hat einen neuen Relationen-Layer für die Lizenzwechsel-Anzeige im OSM Inspector gebastelt; Jochen hat ein paar Bugs im Taginfo repariert. Ich habe dem Bigmap-Tool ein Frontend mit Karte verpasst, Sarah wurde mit Nominatim-Verbesserungswünschen überhäuft, und Sven hat am deutschen Mapnik-Stil gearbeitet. Andi und Stephan sah man ODbL-Statistiken ausrechnen, und Christoph musste zwischendurch immer mal wieder vor die Tür, um den neuesten Build der gesprochenen Fahranweisungen in seinem MoNav zu testen, und vieles vieles mehr.
Der große Tintenstrahldrucker der Geofabrik hat dafür gesorgt, dass der eine oder andere ein nettes Souvenir mit nach Hause nehmen konnte.
Am Samstag Abend wurde in grosser Runde im Büro zu Abend gegessen.
Wir hatten viel Spass dabei, und es soll nicht das letzte Wochenende dieser Art gewesen sein: Als Termin für das nächste Hack-Weekend in Karlsruhe hier bei der Geofabrik haben wir den 2. und 3. Juni 2012 anvisiert. Tragt Euch das schonmal in den Kalender ein – wir melden uns rechtzeitig hier und auf der Mailingliste mit einer genaueren Planung.
2.01.2012 | Christine Karch
Ich habe die Zeit zwischen den Jahren genutzt und Best of OpenStreetMap um einige interessante Beispiele erweitert.
Wer Best of OpenStreetMap noch nicht kennt: Es dokumentiert Meilensteine der OpenStreetMap Entwicklung. Wir zeigen herausragende Mapping-Beispiele, wenn sie eine historische Bedeutung haben oder wenn es sich ganz einfach um besonders interessante Stellen handelt.
Besonders erwähnenswert finde ich den Mapping Trip in die Regionen Beshood und Surkhrod in der
Provinz Nangarhar. Werft auch einen Blick auf diesen durchaus informativen und interessanten Blog aus Afghanistan: Jalalalaa Good!
Gleichzeitig möchte ich diese Gelegenheit nutzen, um mich vorzustellen: Seit August 2011 bin ich Mit-Geschäftsführer-/Inhaberin der Geofabrik. Ich komme aus dem Bereich der Web-Entwicklung und werde die Geofabrik insbesondere hier verstärken. Davor habe ich 9 Jahre bei plan b und 3 Jahre bei der 1&1 gearbeitet. Der OSM Community bin ich zum größten Teil noch unbekannt. Ich hoffe aber, dass sich das im Lauf der Zeit ändert, und werde meinen Teil zum Vorankommen des Projekts beitragen.
3.10.2011 | Frederik Ramm
Our OSM Inspector web site is an important part of many a mapper’s toolbox. It has a series of, daily updated, debug views on OSM data, highlighting potential problems from invalid multipolygons over strange tagging or faulty geometries to missing address data.
The OSM Inspector, or OSMI for short, has several thematic views (like “geometry”, “tagging”, “boundaries”), each of which again supports a number of individually selectable layers that highlight a specific kind of problem. OSMI displays its results over an OSM base map, and you can select individual bug reports and get more information about them, or even download a list of errors for further processing. There’s extensive documentation on the OpenStreetMap Wiki.
Worldwide Routing View
While most views have always shown data computed in-house at Geofabrik, one of our views, the routing view, presents data that comes from an algorithm developed by Pascal Neis of OpenRouteService fame, and is sponsored by Skobbler. The routing view attempts to highlight obstacles to routing – mostly roads that are not connected when they should be, but also duplicate roads that may lead to strange routing results.
We do not have a sponsor for running this routing view world-wide on a daily basis right now, but we have run it once for the non-Europe part of the planet, and you can try it out here:
We’ll try to update this once a week or so. (The normal, sponsored routing view for Europe with daily updates is here.)
In all, OSMI has data collected by three different toolchains; one of them works globally and feeds e.g. the tagging, multipolygon, and general geometry debug views; the other two, including the routing view toolchain, currently do Europe only and would require re-writing or more hardware to cope with the whole planet.
Potlatch
It has always been possible to access OSMI views through WMS for use in other applications, especially as a background layer in editors. Users of the popular Potlatch editor were however unable to make use of this as Potlatch does not support WMS backgrounds. We have therefore configured OSMI to serve tiles as well. (We’re using the excellent open source MapProxy for that.)
If you are using Potlatch, you can now add any or all of the following background layers:
Those marked (*), and additionally all the public transport views offered by OSMI, are currently only available for Europe. For the public transport views, use the URL components “ptri” (rail infrastructure), “ptnri” (non-rail infrastructure), “ptf” (ferries), “pts” (stops), “ptn” (network).
Note that the tiles for each view will always show all available layers at the selected zoom level; when using tiles, you cannot toggle the visibility of individual layers like you can when using OSMI through its web interface. Also, the key to colours and symbols is only available within OSMI. Therefore, even if you plan to use the tiles in your editor, it might make sense to familiarize yourself with the data presentation on OSMI first!
We hope that these changes help to further improve quality awareness and, in consequence, data quality in OSM, and we’re happy to work with potential sponsors on extending these services to support more frequent updates, additional analyses, or a wider geographical coverage.
19.09.2011 | Frederik Ramm
For years, Geofabrik has been supplying the world with OpenStreetMap Shapefiles – two kinds of them, in fact; there’s our popular download page which has free shapefile downloads for many places, updated daily, and there’s our extended shapefile service where we sell shapefiles that require more work, and processing time, than the free nightly builds.
Whereas the free nightly builds are very close to the source, simply taking OSM objects and converting them into shapefiles, the extended service, which is based on a different toolchain here at Geofabrik, has a well-documented feature catalogue which we are today releasing in version 0.6 with a ton of enhancements over the previous version. The extended service also encompasses a few useful preprocessing steps, like automatically converting area POIs to points and proper handling of large multipolygons. The extended shapefiles are made to order; the standard feature catalogue contains almost anything in OSM that is widely used, but clients can also request extra features to be added. Extended shapefiles are available for any area – the whole planet, a single continent, a single country, a city, or anything in between.
In addition to releasing our new feature catalogue for our extended shapefiles, we’ve also made a change to the free nightly builds; these now have a much-asked-for “landuse” layer that contains information from OSM’s tag of the same name.
(Eine deutsche Version folgt.)
30.11.2010 | Frederik Ramm
We are sometimes asked how we produce the files on our download server. Read this if you are one of those asking.
It is 22:30 in Central Europe, and the lights come on in the Geofabrik office. (The HDD LEDs, that is.) One of the servers, named bonne, begins downloading the collected works of mappers around the world from the last 24 hours
osmosis --rri --simc --write-xml-change 2010-11-16-22:30.osc (duration 00:01:30)
and then applies them to a locally held, full copy of the OSM database (the “planet file”).
osmosis --read-xml-change 2010-11-16-22:30.osc --read-bin current-planet.osm.pbf --apply-change --write-bin new-planet.osm.pbf compress=none (00:28:09)
The newly created planet file is transferred to another server, named hammer, where a script converts it to a simple CSV-like format (the “tbf” format) and then analyzes and converts it in several steps, creating statistics and a number of shape files. The shape files are later copied to our off-site tools server where they are used by the OSM Inspector.
But bonne, of course, doesn’t sit idle. More …
14.10.2010 | Frederik Ramm
Today we finally received our author’s copies of our brand-new English OSM book. Old news, you might say; after all, this was featured on opengeodata.org a month ago. But it took a while for the books to reach us here at Geofabrik — we’ve been told that the first batch of books had to be dispatched to the US immediately because of the high demand there!
As you probably know, Jochen and I published the first OSM book ever in spring 2008, in German; a second edition came out in 2009, and a third in 2010. The book is of course a niche product as books go — with roughly 4,000 copies sold altogether. But we’ve been getting great feedback from the community and were encouraged to do an English version basically from day one. And so we did! At first we toyed with the idea of doing the book on our own with a print-on-demand place like lulu.com but we tried a few and were unhappy with the results. It took us a while to find a proper publisher with whom we could work, but we finally found a great, if small, publisher in UIT (Cambridge, UK). Just as with the German book, we not only wrote the content, but also did the typesetting, layout, and book cover ourselves, and we are very pleased with the finished product. It looks, feels, even smells like a proper book.
We had great help from one of the old OSM hands in England, Steve Chilton, who made sure that the English version properly reflects the international aspects of OSM, and fixed up our translation. Like the German book, the English one has 384 pages, 32 of them in full colour. (The English language is a bit more concise than German, so we managed to squeeze in a few extra illustrations and examples!) It covers everything there is to know about OSM, from the origins and the community to web services, editors (with detailed descriptions of Potlatch and JOSM), how to render your own maps (including how to write Mapnik and Osmarender style sheets), and how to access OSM data through API, planet files, or diff updates.
If you’re serious about OSM, you should really get this book. (If only to prove to others that the project you’re spending half your waking life on is something serious, you know, something people write books about!)
The sad thing is, like so often with publishing, the distribution, or more specifically, amazon.com; you can order the book at bookdepository.co.uk (free shipping) and it is also in stock at amazon.co.uk, but amazon.com says it “ships in 1 to 3 weeks”. (Also, both amazon sites have the wrong cover image, reflecting an early draft.) I hope that these initial distribution problems will soon be flushed out. This is the one drawback of doing a traditionally printed book – the stuff has to be shipped from printers to wholesalers to retailers and all that takes time.
More information on the book is also at www.openstreetmap.info.
22.09.2010 | Frederik Ramm
Geofabrik has been providing the OSM community with cut-down data extracts for various continents, countries, and smaller administrative divisions for over two years now. Our downloads at download.geofabrik.de form the basis of many community projects, and we’re happy to make working with OSM data easier for so many people.
Today we’re launching experimental downloads in a new binary format. The new “protobuf binary format” (.osm.pbf) is 30% smaller than the bzip2-compressed OSM XML, and it can be processed or extracted much faster than bzip2 files. Also, while we will continue supporting the bzip2 files for a while, we hope that we can ultimately free up some resources by dropping bz2 support, and use these resources to produce an even wider set of daily updated OSM extracts.
The protobuf binary format was developed by Scott Crosby and presented to the OSM community in April this year (wiki article with details). As the name implies, it relies on Google’s “Protocol Buffers” for its internal data representation. The format is supported by Osmosis starting with version 0.37; .osm.pbf files can be read directly by Osmosis, or converted to plain XML OSM first using a command like
osmosis --read-pbf myfile.osm.pbf --write-xml myfile.osm
The above command will run significantly faster than a bz2 decompression, and the .osm.pbf files made available by Geofabrik are 100% lossless. The format offers further compression options by stripping of metadata or minimally reducing precision, but Geofabrik extracts will remain lossless.
Not only are .osm.pbf smaller and faster to process than their bz2 counterparts; they are also likely to appear faster on the Geofabrik download site than the regular .osm.bz2 files. Everyone is encouraged to give them a try.
(Edit on 2010-11-16: Initially the command line options to use were called “read-bin” and “write-bin”, but later releases of Osmosis now use “read-pbf” and “write-pbf”.)
26.07.2010 | Frederik Ramm
OpenStreetMap WMS servers are hard to come by – because they are hard to operate. Running a WMS server always means a trade-off. You can have it render from vector data which makes it slow, or render from pre-computed raster tiles which makes intermediate zoom levels and reprojection ugly. You can use clever preprocessing but then you’re unlikely to be able to trace OSM updates quickly, or you opt for high-frequency updates but that again causes more load on your server. You can use Mapnik but the available WMS servers for that are not so good, or you can use the established UMN Map Server or Geoserver which have excellent WMS support but don’t achieve what has become the “standard” look and feel for OSM maps.
Geofabrik WMS displayed in Quantum GIS.
We think we have struck a nice balance between all these goals with our WMS server, which is available to customers immediately starting from EUR 35 per month. (We regret that we cannot make this available for free due to the cost involved for us, but will consider granting free access for OSM community projects.) The server runs from a continuously updated OSM database covering the whole world, and offers a clean Mapnik vector-based rendering in all projections and sizes. More on our product page.
At the same time, we’re making the software that runs our WMS – a special module for the Apache web server that talks directly to Mapnik – available as Open Source under the GPL license, which should allow anyone to install and run their own WMS server. There’s more on the OSM wiki.
13.07.2010 | Frederik Ramm
Wir sind zurück von der State of the Map 2010, der internationalen OpenStreetMap-Konferenz, die dieses Jahr in Girona in Spanien stattfand.
Insgesamt lässt sich beobachten, dass das Interesse von Unternehmen an der Konferenz wächst; es wurden dieses Mal grob doppelt so viel Sponsorengelder eingesammelt wie letztes Mal (verbunden allerdings auch mit grob doppelt so hohen Kosten für die Konferenz), und viele “beruflich” OSM-Interessierte waren auch das ganze Wochenende da.
Dennoch schien ein großer Teil der Anwesenden eher technisch interessiert – kaum jemand im Publikum, der sich nicht schon einmal wenigstens in irgendeinen Aspekt von OSM tiefer eingearbeitet oder sogar Software geschrieben hatte. So war dann auch das Vortragsprogramm fast weniger spannend als die zahlreichen Gespräche mit Bastlern aus der ganzen Welt, die in den (spanientypisch sehr ausgedehnten) Kaffee- und Mittagspausen oder auch im Foyer vor den Vortagssälen geführt wurden.
Auf besonderes Interesse stiessen am “Business Day”, dem Freitag, die Ankündigungen von Microsoft (bing maps soll bald einen wählbaren OSM-Layer bekommen) und Aol (Mapquest will vollständig auf OSM umstellen, für Karten und Routing, und im kommenden Jahr eine Million US-Dollar bereitstellen, um das Projekt vorallem in den USA voranzubringen).
Von der Geofabrik gab es einen Vortrag über Geschwindigkeitsoptimierung bei osm2pgsql/Mapnik-basiertem Tile-Rendering (Folien hier) sowie Kurzvorstellungen der Tirex-Software und der zum Wochenende von Dennis Luxen am KIT (Karlsruher Institut für Technologie, ehemals Universität Karlsruhe) veröffentlichten schnellen Routing-Engine für OSM-Daten (Demo-Seite bei der Geofabrik, Ankündigung auf der Mailingliste.)
Als Mitglied der “Data Working Group” bei OpenStreetMap stellte Frederik ausserdem kurz die Arbeit in diesem Gremium vor, das sich mit der Bekämpfung von Vandalismus und Urheberrechtsverletzungen bei OpenStreetMap beschäftigt.
Am Freitag abend gewann die Geofabrik überraschend einen kleinen Preis für den “Best Elevator Pitch” – fünf kleinere OSM-Firmen waren aufgefordert, in einer Minute ihr Geschäftsmodell zu umreißen und zu erklären, was man macht, wer die Kunden sind, was man diesen Kunden bietet und warum sie zu einem kommen sollen. Frederik sagte für die Geofabrik in etwa folgendes:
“Unser Geschäftsmodell ist einfach, altmodisch, und langweilig: Sie zahlen uns Geld, und dafür arbeiten wir für Sie. Wir verkaufen unseren Kunden keine Software oder Leistungen, die sie nicht auch selber mit freier Software und anhand von Wiki-Anleitungen hinbekommen – wir verkaufen einfach Zeitersparnis. Die Kunden beauftragen uns, wenn sie ein Problem lieber von erfahrenen Experten lösen lassen möchten, anstatt selbst ein mehrfaches der Zeit zu investieren. Wir verwenden die Hälfte unserer Zeit darauf, mit der und für die OSM-Community zu arbeiten – dadurch weiss die Community auch, was wir können, und schickt potentielle Kunden in unsere Richtung.”
Das Publikum fand diese Beschreibung erfrischend einfach, und die Geofabrik gewann mit Abstand (“by a country mile”, wie Steven Feldman, Moderator des Wettbewerbs, sagte).
Den Abschluss der Veranstaltung bildete am Sonntag wieder, wie letztes Jahr in Amsterdam, eine langwierige Versteigerung sämtlicher übriggebliebenen Veranstaltungsgegenstände – Poster, Banner, Taschen, sogar einige Hardware und Weinflaschen wurden an den Mann oder an die Frau gebracht.
Wir hatten vier Vorab-Exemplare unseres demnächst auf englisch erscheinenden OSM-Buches dabei, die uns förmlich aus der Hand gerissen wurden (“wann kann man das kaufen? wo?…”) – nun wird es höchste Zeit, das endlich in den Handel zu bringen!
8.03.2010 | Frederik Ramm
One of the work horses behind any OpenStreetMap web map is the tile server – people either use the existing openstreetmap.org server or set up their own. The software used to run the openstreetmap.org tile server is open source, but it proved insufficient for a sophisticated tile server installation we recently did for one of our clients. This led to the development of an advanced queuing and tile rendering system we named Tirex.
The client for whom we developed Tirex, Enaikoon GmbH in Berlin, is in the telematics business. They are OSM and Open Source friendly and allowed us to bundle the whole thing into a GPL release that is likely to benefit other users of OSM data. We have uploaded the full source code and documentation to the OpenStreetMap SVN. Details can be found on the OpenStreetMap Wiki.
