Skip to content
This repository was archived by the owner on Jul 30, 2026. It is now read-only.
This repository was archived by the owner on Jul 30, 2026. It is now read-only.

OpenStreetMap Viewer #25

Closed
Closed

Description

@davidar

Continued from ipfs-inactive/archives#11

Goal: create an OSM viewer with all of the assets (tiles) hosted on IPFS

Activity

  1. davidar commented on Jun 1, 2016

    @davidar
    MemberAuthor
  2. davidar commented on Jun 1, 2016

    @davidar
    MemberAuthor

    Thanks again to @osm2vectortiles @lukasmartinelli for making this possible :)

  3. lukasmartinelli commented on Jun 1, 2016

    @lukasmartinelli

    https://ipfs.io/ipfs/QmNUFNQXWVzbhePhj6bscW6TPu5azraCpbejbNY9zyfFjo/

    Really cool. Do you generate the vector tiles yourself or use osm2vectortiles downloads v1.0 or v2.0?
    We could even use that as CDN in our showcases.

    This is also only data for upper zoom levels? As I zoom in in Europe > zoom level 11 I miss data.

  4. davidar commented on Jun 1, 2016

    @davidar
    MemberAuthor

    Really cool. Do you generate the vector tiles yourself or use osm2vectortiles downloads v1.0 or v2.0?

    @lukasmartinelli I extracted them from the osm2vectortiles v1.0 download. What changes have been made in v2.0?

    We could even use that as CDN in our showcases.

    That would be cool :)

    This is also only data for upper zoom levels? As I zoom in in Europe > zoom level 11 I miss data.

    Currently it only goes up to level 9, but once ipfs/kubo#1924 is fixed I'll add the remaining zoom levels (10-14)

  5. Kubuxu commented on Jun 1, 2016

    @Kubuxu
    Member

    Currently the viever doesn't work on local gateways as it tires to access global gateway, which is blocked by no-CORS. I reported CORS issue ipfs/infra#166 and created patch to enable those options by default ipfs/kubo#2778.

    The best way of accessing data on IPFS using XHR is rooted relative link: /ipfs/Qm...AAA. Then if site is opened on global gateway, the global gateway will be used, and vice versa.

  6. lukasmartinelli commented on Jun 1, 2016

    @lukasmartinelli

    @lukasmartinelli I extracted them from the osm2vectortiles v1.0 download. What changes have been made in v2.0?

    Okay perfect. We are also just finished with v2.0, we will release a changelog.
    http://osm2vectortiles.org/downloads/

    In short: has much better quality, smaller filesize and biggest of all can now be updated in a weekly interval.

    Really happy to have IPFS host this 😁 Thanks @davidar for your work.

  7. self-assigned this
    on Jun 4, 2016
  8. btrask commented on Jun 4, 2016

    @btrask

    This sounds very promising!

    I would like to suggest a few features for preventing tracking and maximizing user privacy:

    • Provide an easy way to pin map data for large geographic areas (like cities), so that queries can be made privately on the user's own machine
    • Compress the map data as much as possible so that it's practical to store on e.g. mobile devices (not sure how large the OSM data is currently, if it's small this may not be an issue)
    • Structure the data so that efficient differential updates are possible (again OSM may take care of this for you, or not)
    • Have an optional warning before trackable remote requests are made (e.g. if the map isn't stored locally, don't just blindly fetch it, since that could leak information)

    Thanks @davidar for letting me know about this project!

  9. davidar commented on Jun 5, 2016

    @davidar
    MemberAuthor

    @lukasmartinelli Updated to osm2vectortiles v2.0 (z0-z8): https://ipfs.io/ipfs/QmTAEbxXKoBWDzNduLS9WmAoNkcuHgQbc3fm9oGBe3FfR4/?/ipfs/Qmf3KBYSv9H4xADgKqpvwWrXkiMh5BfZohFXCaaFh7u4nb/

    • Provide an easy way to pin map data for large geographic areas (like cities), so that queries can be made privately on the user's own machine

    @btrask Good idea. I'd have to think about the best way to do this.

    • Compress the map data as much as possible so that it's practical to store on e.g. mobile devices (not sure how large the OSM data is currently, if it's small this may not be an issue)

    It's about 50GB in total, but substantially less for lower levels of detail. Having said that, I don't think it's possible (or at least not recommended) to run an ipfs node on mobile devices yet.

    • Structure the data so that efficient differential updates are possible (again OSM may take care of this for you, or not)

    IPFS deduplicates tiles, so update efficiency would depend on how many tiles are changed.

    • Have an optional warning before trackable remote requests are made (e.g. if the map isn't stored locally, don't just blindly fetch it, since that could leak information)

    Yeah, the issue of tracking/privacy in the context of ipfs has been brought up before. In this case you could just use ipfs offline, but that would cause remote requests to silently fail rather than issue a visible warning, so probably isn't the best solution. @jbenet Thoughts?

  10. davidar commented on Jun 5, 2016

    @davidar
    MemberAuthor

    @lgierth Does it? The readme must be outdated

  11. btrask commented on Jun 5, 2016

    @btrask

    @davidar sounds good!

    Just one comment:

    IPFS deduplicates tiles, so update efficiency would depend on how many tiles are changed.

    Deduplication helps, but it's still up to the application to make sure that its data dedups effectively. That means making sure that small changes to the map result in localized changes within the database. (You probably already know this, but it doesn't hurt to document.)

  12. davidar commented on Jun 6, 2016

    @davidar
    MemberAuthor

    @btrask localised changes to the map should only change a small number of tiles. @lukasmartinelli would probably be able to give a more concrete answer than me though :)

  13. lukasmartinelli commented on Jun 6, 2016

    @lukasmartinelli

    @btrask localised changes to the map should only change a small number of tiles. @lukasmartinelli would probably be able to give a more concrete answer than me though :)

    There are two things that go on.

    A small number of tiles change very frequently. I can't say percent numbers but it feels like 20% of the tiles change every week and the others don't have any changes at all.

    There are butterfly effects when changing boundaries.

    Quote from @ImreSamu osm2vectortiles/osm2vectortiles#251

    OSM butterfly effect : https://en.wikipedia.org/wiki/Butterfly_effect
    The butterfly effect refers to a concept that small causes can have large effects.

    for example modifying small part of France - Schweiz border : http://www.openstreetmap.org/changeset/39252252

    Has an effect of other part of the French-administered territories ...
    france_osm_2202162

  14. Kubuxu commented on Jun 6, 2016

    @Kubuxu
    Member

    Thanks to de-duplication properties of IFPS it shouldn't really affect us, I might write a tool to calculate de-duplication ratio of two different hashes if there is need for that.

  15. davidar commented on Jun 6, 2016

    @davidar
    MemberAuthor

    for example modifying small part of France - Schweiz border has an effect of other part of the French-administered territories

    @lukasmartinelli why does that happen, out of interest?

  16. lukasmartinelli commented on Jun 10, 2016

    @lukasmartinelli

    @lukasmartinelli why does that happen, out of interest?

    Because all these lines are in the same superior relation "France".

  17. whyrusleeping commented on Jun 16, 2016

    @whyrusleeping
    Member

    @Kubuxu telling the deduplication factor between two hashes is pretty straightforward. The simplest way is to run ipfs refs -r on both, and use comm to take the differences of the set.

    Alternatively, using ipfs object diff if the data is formatted in a nice enough way would provide a nice view

  18. daviddias commented on Jun 24, 2016

    @daviddias
    Member

    Just had the time to play properly with this for the first time and my reaction was WOW! :D This looks and feels really good!

    @lukasmartinelli what about http://osm2vectortiles.org/downloads/ are served by IPFS also, it would really only requires to add an hash to be there on the webpage since we are already hosting it, making the download of those 50GB way faster :)

  19. lukasmartinelli commented on Jun 24, 2016

    @lukasmartinelli

    @lukasmartinelli what about http://osm2vectortiles.org/downloads/ are served by IPFS also, it would really only requires to add an hash to be there on the webpage since we are already hosting it, making the download of those 50GB way faster :)

    Not entirely sure whether I get it. Could IPFS then serve the downloads - instead of S3?
    But we still need to upload the downloads to S3? IPFS would mirror them.

    So we would need to append a shortened sha-1 of the file to the filename and then IPFS could serve the downloads?

  20. daviddias commented on Jun 24, 2016

    @daviddias
    Member

    We can have IPFS serving all the downloads or we can have the hybrid (the first option would reduce storage and bandwidth costs).

    It would be a sha2-256 and you can get it by doing an ipfs add <path to the blob>. Once you get that hash, you will be able to get it through ipfs.io or a user can also 'reprovide' it by booting up a node and loading it through their IPFS node. I'm pulling the data set right now to add it to my local node and so that I can share with you the IPFS Hash. However, if you have the data set already, you can do it too :)

  21. daviddias commented on Jun 24, 2016

    @daviddias
    Member

    Here you go :D

    planet_z0-z8.mbtiles

    » ipfs add planet_z0-z8.mbtiles
    added QmeiadPfZUHxZwTSU8nNQGjQQfaan2Yj7GvBVdg1muevAu planet_z0-z8.mbtiles
    

    planet_z0-z5.mbtiles

    » ipfs add planet_z0-z5.mbtiles
    added QmQ8hxi3wbdoYNG6UPHgZbWz72Ckgr2CtmG3SZQgSW9nW3 planet_z0-z5.mbtiles
    

    planet.mbtiles

    » ipfs add planet.mbtiles
    added QmVHzEAwUNow7jhABCZEc9KECfCvfJq7ucNx4eXCrP26Da planet.mbtiles
    

    If you run an local ipfs node, replace ipfs.io by localhost:8080 on the links :)

    The Qm... are base58 encoded sha2-256 hashes

  22. jbenet commented on Jul 5, 2016

    @jbenet
    Member
  23. davidar commented on Jul 30, 2016

    @davidar
    MemberAuthor

    Graduating to its own repo: https://github.com/davidar/ipfs-maps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions