{"id":58,"date":"2010-09-22T20:26:05","date_gmt":"2010-09-22T20:26:05","guid":{"rendered":"http:\/\/blog.geofabrik.de\/?p=58"},"modified":"2010-09-22T20:26:05","modified_gmt":"2010-09-22T20:26:05","slug":"new-file-format-for-osm-downloads","status":"publish","type":"post","link":"https:\/\/blog.geofabrik.de\/index.php\/2010\/09\/22\/new-file-format-for-osm-downloads\/","title":{"rendered":"New file format for OSM downloads"},"content":{"rendered":"<p>Geofabrik has been providing the OSM community with cut-down data extracts for various continents, countries, and smaller administrative divisions for over two years now. Our downloads at <a href=\"http:\/\/download.geofabrik.de\/osm\/\">download.geofabrik.de<\/a> form the basis of many community projects, and we&#8217;re happy to make working with OSM data easier for so many people.<\/p>\n<p><img decoding=\"async\" src=\"\/wp-content\/schraubstock.jpg\" align=\"left\">Today we&#8217;re launching experimental downloads in a new binary format. The new &#8220;protobuf binary format&#8221; (.osm.pbf) is 30% smaller than the bzip2-compressed OSM XML, and it can be processed or extracted much faster than bzip2 files. Also, while we will continue supporting the bzip2 files for a while, we hope that we can ultimately free up some resources by dropping bz2 support, and use these resources to produce an even wider set of daily updated OSM extracts.<\/p>\n<p>The protobuf binary format was developed by Scott Crosby and presented to the OSM community in April this year <a href=\"http:\/\/wiki.openstreetmap.org\/wiki\/ProtocolBufBinary\">(wiki article with details).<\/a> As the name implies, it relies on Google&#8217;s &#8220;Protocol Buffers&#8221; for its internal data representation. The format is supported by Osmosis <a href=\"http:\/\/bretth.dev.openstreetmap.org\/osmosis-build\/\">starting with version 0.37;<\/a> .osm.pbf files can be read directly by Osmosis, or converted to plain XML OSM first using a command like<\/p>\n<pre>osmosis --read-pbf myfile.osm.pbf --write-xml myfile.osm<\/pre>\n<p>The above command will run significantly faster than a bz2 decompression, and the .osm.pbf files made available by Geofabrik are 100% lossless. The format offers further compression options by stripping of metadata or minimally reducing precision, but Geofabrik extracts will remain lossless.<\/p>\n<p>Not only are .osm.pbf smaller and faster to process than their bz2 counterparts; they are also likely to appear faster on the Geofabrik download site than the regular .osm.bz2 files. Everyone is encouraged to give them a try.<\/p>\n<p><em>(Edit on 2010-11-16: Initially the command line options to use were called &#8220;read-bin&#8221; and &#8220;write-bin&#8221;, but later releases of Osmosis now use &#8220;read-pbf&#8221; and &#8220;write-pbf&#8221;.)<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Geofabrik has been providing the OSM community with cut-down data extracts for various continents, countries, and smaller administrative divisions for over two years now. Our downloads at download.geofabrik.de form the basis of many community projects, and we&#8217;re happy to make working with OSM data easier for so many people. Today we&#8217;re launching experimental downloads in [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.geofabrik.de\/index.php\/wp-json\/wp\/v2\/posts\/58"}],"collection":[{"href":"https:\/\/blog.geofabrik.de\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.geofabrik.de\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.geofabrik.de\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.geofabrik.de\/index.php\/wp-json\/wp\/v2\/comments?post=58"}],"version-history":[{"count":0,"href":"https:\/\/blog.geofabrik.de\/index.php\/wp-json\/wp\/v2\/posts\/58\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.geofabrik.de\/index.php\/wp-json\/wp\/v2\/media?parent=58"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.geofabrik.de\/index.php\/wp-json\/wp\/v2\/categories?post=58"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.geofabrik.de\/index.php\/wp-json\/wp\/v2\/tags?post=58"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}