Get the top HN stories in your inbox every day.
simonw
simonw
I turned this into a demo: https://simonw.github.io/hmb-map/ and a blog post: https://til.simonwillison.net/gis/pmtiles
globular-toast
Beat me to it. I was working on this last night but had to sleep before writing it up. I was trying to use parcel as a bundler and was thrown because whatever dev server it uses doesn't seem to support HTTP range requests properly. I was getting "invalid magic number" on the client side.
oritsnile
That's really cool. Cloud be really use full for local businesses. They often embed Google maps, but I feels like having a map from the local are would be sufficient, and cheaper.
pimlottc
Wouldn’t they need to keep updating this from time to time to stay current? That would likely kill any savings pretty quickly.
globular-toast
It depends what you use it for. I plan to use it for blog posts where I show the track of where I walked. I actually like that the map won't update. It wouldn't make sense to show me walking across a map of the future, would it? It's a record of the past and the map should be frozen just like the writing.
tppiotrowski
Protomaps generates daily world files I believe [1] but I haven't confirmed that they're used by the download tool yet. In theory you can download the data everyday to pick up updates.
[1] https://maps.protomaps.com/builds/
Edit: re-reading top comment it looks like they're using a daily file so you'd just have to create a cron job
globular-toast
I did the same and got a map of Gran Canaria which is 15MB! And it's not like a user is ever going to download the entire file. It's really quite cool and I'm quite excited to put maps on my static site.
botanical
I realise this isn't tech support but I have no idea why it doesn't work for me. The pmtile file you linked works, but creating my own smaller one doesn't.
But the concept is great to get a map for a project
mberg
I've been an avid Mapbox user since getting alpha access to Tilemill. From my perspective, this is the most important contribution to the open mapping space since the introduction of vector tiles and Mapbox GL / Map GL. Mapbox in my opinion left the door open in how they approached tile baking with MTS. While powerful it was way to confusing and expensive. While you could always bake tiles with Tippecanoe you still had to struggle with vector tile hosting hosting. With PMTIles vector tile hosting is arguably easier, more convenient and an order of magnitude cheaper then going with Mapbox (especially if you don't need fancy projection / 3d support and are ok with MapLibre). Felt is one of the major services that I believe uses the PMTile approach behind the scenes and they won't be the last. (Notably: they have continued to invest in Tippecanoe which is awesome). With Mapbox focus shifting, it appears, to car navigation - it's great to see disruptive innovation coming from new places. Even more amazing that is has come from a one person team.
Also to note, the emergence of PMTiles coincides DuckDB spatial query support will unlock a lot of innovation. The ability to query and process large geoparquet files and quickly stream them into baked PMTiles unlocks a lot of really compelling use cases.
bdon
At the bottom of this post is an example of creating PMTiles from GeoParquet via tippecanoe + gpq. Thanks to Tim Schaub for making this possible!
gpq convert Cairo_Governorate.parquet --to=geojson | tippecanoe Cairo.geojson -o Cairo.pmtiles
* https://cloudnativegeo.org/blog/2023/10/where-is-cog-for-vec...
NelsonMinar
It's kinda sad how we need open source alternatives to MapBox, the formerly open source company.
bdon
Mapbox is still the best choice where a polished suite of mapping APIs is a better fit for a project.
Mapbox is a venture-backed company with a SaaS business model, and has never been open source in total - it used to be open core with a FOSS frontend and proprietary backend. This SaaS model is absolutely the best way to fund huge companies and give investors a return. Mapbox has also done the bulk of innovation in open source web mapping over the past 10 years - the Protomaps project uses MapLibre (fork of Mapbox GL 1.0) and the MVT tile format. Both required teams of full-time developers - easily tens of millions in salaries and stock - and they have given away version 1.0 for free as a gift, even though 2.0 is not open source.
The ideal software economy is one in which innovators capture a good portion of the wealth they create. This is why it's important for Protomaps to focus on use cases underserved by SaaS, instead of just being a cheaper map API. The sibling comment on wildfire mapping https://news.ycombinator.com/item?id=37989059 is a good example of the applications I want the project to support.
pdntspa
> The ideal software economy is one in which innovators capture a good portion of the wealth they create.
Beg to differ, the ideal software economies maximally empowers end-users at the absolute minimal cost. Innovators can and should leave substantial cash on the table. They should see themselves as stewards of a public good.
jampekka
Ideal software economy would be all free (as in freedom) software. The current economy is hugely wasteful with lots of redundant work and generally bad outcomes.
Private ownership economy doesn't work for zero marginal cost products even in theory. It's a huge waste of our resources and a big hinderance to progress.
bcks
I lurk in an online community of journalist coders and some folks there are excited about PMTiles as cost-saving way to host your own custom-styled map tiles.
See, for instance, "How The Post is replacing Mapbox with open source solutions" https://www.kschaul.com/post/2023/02/16/how-the-post-is-repl...
wolverine876
How do the publications know this data is accurate?
lmeyerov
Most of these piggyback on the OSM for the data tier, which $T companies work on. Afaict the discussion is more about compute on top - tile generation, interactive renderers, ... .
wolverine876
$T companies aren't a source of truth. Much of journalism is reporting on their errors, including the intentional, reckless, and lazy kinds.
loughnane
Sounds like a neat community. Can you share where it is?
bcks
Sure thing, check out NICAR-L: https://www.ire.org/resources/listservs/
blutack
If anyone else was curious, their sample OSM derived PMTile file (protomaps-basemap-opensource-20230408.pmtiles) is 103GB.
pmontra
It's 107 GB now.
It would be great to be able to download only diffs and apply them to a daily, weekly, monthly baseline.
mycall
I wonder if the compressed Hilbert ordering inside the tiles make them hard to diff efficiently.
bdon
The ordering applies across tiles, and not within - it should be efficient to diff two archives because there is a single, well-defined ordering over the entire archive, and writing algorithms over the tileset is all 1D operations instead of 2D (a single zoom level plane) or 3D (multiple planes)
blutack
Hmm I was just going off the Firefox estimate, maybe there's some browser difference?
bdon
The newest major version has moved to a daily build channel (the last major version was last updated in April)
Today 10/23 is 107GB.
https://maps.protomaps.com/builds/
I'll update the documentation page today.
edhelas
Exactly the kind of thing that could be downloaded using Bittorrent!
ElectricalUnion
I assume that would require BitTorrent v2 (and I think more specifically something along BEP-46 - Updating Torrents Via DHT Mutable Items) or your map is gonna be out-of-sync and sad after a while.
nextaccountic
are the torrents in the wild that uses this? and are there bittorrent clients that implement this? how do they surface the mutable aspect to the user?
(is it like, your torrent client will silently update files, and if you dont want to update anymore you need to pause? can you still seed if you dont want to update?)
eternityforest
What are they using to compress tiles? Could they get even smaller with AVIF? Or AVIF+SVG overlaid for text?
bambax
Yes thank you I was wondering exactly this. Quite reasonable size one might say.
dinkblam
thanks for the info, weird that this isn't answered in the FAQ
robertlagrant
> The Google Maps API has a generous free tier, but high fees past 30,000 map loads. Protomaps is designed from the ground up to be a system you run yourself - as a 100% static backend on cloud storage - so you only incur the usage costs of storing and serving that data, which can be pennies for moderate usage.
It would be interesting to see a cost comparison: recommended hosting setup vs Google Maps, and at what point the line intersects.
elaus
I haven't found this info in the FAQ, but it would be interesting to know how large that PMTiles file is for a medium-sized country.
It seems to be a really attractive way of self-hosting maps that don't have to be 100% up to date and are only used in a specific region. "Recompiling" that file once a year and uploading it to a static file hoster would be an easy process with very little external dependencies.
Edit: Found the official download link[1] for the whole world, which is a bit over 100 GB. So I guess the answer to my question would be "hundreds of MB to a few GB", which seems totally fine.
Doctor_Fegg
Geofabrik already does this for vector tiles of each country, but as individual mvt files rather than in a container like pmtiles or mbtiles.
robertlagrant
That's true - I suppose the cost changes depending on the size of the area you're mapping. Maybe a representative set? A city (e.g. Johannesburg), a small country (e.g. UK), and a large country (e.g. USA)?
wiredfool
Monaco is 500k, Ireland +NI is 400M, and Europe is 40G, for their basemap output.
(Representative numbers from my runs)
tppiotrowski
Cloudflare: 100GB hosting is $1 and $.36 for 10 million requests.
If you read the Protomaps docs it explains how to cache the requests on Cloudflare CDN so you only have to pay for each tile request once per cache period. It's quite cheap.
Edit: previous submission title for Protomaps: Serverless maps at 1/700 the cost of Google Maps API
Aachen
Okay but the question was what it actually costs, not what our favorite mitm provider offers as a commercial service (they have the worldwide hardware and sufficiently deep pockets to offer this at a rate that guarantees gaining market share, below the cost price of normal hosting providers)
squeaky-clean
Are you asking how much it would cost to run this on self-hosted bare metal servers? Otherwise I'm also not sure what you're asking for.
Aeolun
I mean, one of the examples uses cloudflare to host the thing, so that is the pricing parent is talking about. What else do you want?
freyfogle
Brandon, the founder of Protomaps, was interviewed on the Geomob podcast a few months back: https://thegeomob.com/podcast/episode-176
crabmusket
And here on MapScaping: https://mapscaping.com/podcast/planet-scale-tiled-maps-witho...
AlbertoGP
For me the most interesting part is “PMTiles”:
> PMTiles is a single-file archive format for pyramids of tiled data. A PMTiles archive can be hosted on a storage platform like S3, and enables low-cost, zero-maintenance map applications.
> PMTiles is a general format for tiled data addressed by Z/X/Y coordinates. This can be cartographic basemap vector tiles, remote sensing observations, JPEG images, or more.
> PMTiles readers use HTTP Range Requests to fetch only the relevant tile or metadata inside a PMTiles archive on-demand.
> The arrangement of tiles and directories is designed to minimize the amount of overhead requests when panning and zooming.
jonotron
PMTiles is amazing. We discovered it when we were building a Canadian wildfire hotspot map this summer.
We were taking daily CSV files, turning them into vector tiles, syncing them with S3, and displaying in mapbox-gl map. The biggest cost was S3 file operations. We had to aggressively cluster to reduce our cost down and got to a point where the tiles numbered ~75,000. At the frequency we wanted to update this was costing $500/mo in S3 ops.
Now we process the files into a single PMTiles file, deploy that one file and cut the costs down to pennies.
For a map that we make public, it was incredibly enabling. You can checkout the Canadian hotspot data for all of 2023 if you're curious: https://lens.pathandfocus.com
wiredfool
Pmtiles are the vector counterpart to Cloud Optimized GEOTIFF (COG), allowing for efficient usage of mapping info from clients when the server supports a http range request.
Previous iterations of this on the vector side have either been a ton of small files(pbf) or a large file that needed a front end to serve the individual tiles (mbtiles).
At this point, you can take an OSM dump and convert it to a country level basemap in minutes on a stout machine, or hours for a continent, which you can then self host on a plain old web server with a custom style sheet.
bdon
Generating country level basemaps from scratch is a great approach!
I've made an even easier path which is to just extract the relevant tiles from a daily build: https://docs.protomaps.com/guide/getting-started
This can take seconds to minutes for small-medium areas and doesn't need a powerful computer at all. The drawbacks are that the build is only daily, and the lower-zoom tiles aren't clipped so include extra information beyond your specified area.
mycall
I've been using TileServer GL for a while and it looks like there is support for pmtiles coming soon.
kylebarron
pmtiles are only the counterpart to the "overviews" part of COG. pmtiles doesn't give you analysis-ready data. For that, look to FlatGeobuf or GeoParquet (once it gets a spatial index in v1.1 of the spec)
wiredfool
Yeah, I'm aware of that. I love what you're going with geoparquet, it's certainly quite nice for analysis and some viewing. But there are still going to be a lot of datasets and clients that aren't going to be able to handle that where the tiled approach will work.
I'm looking at this currently for a couple of clients, and I think that for what we're doing -- we're going to wind up serving both a view and analysis version separately.
Reason077
So, could PMTiles be used as a cheap/dirty/fast way to host other forms of geospatial data that isn't strictly "map tiles"? Such as a large-ish, infrequently updated POI database?
wiredfool
If the geometry is points/lines/polygon/multi* + properties on the features, yes. (In other words, if it's representable as geojson), with the caveat that it's more for viewing and less for analysis.
It's essentially what the internal layers are in the base layer -- there are land use, buildings, transit, physical, and border features that are combined with a stylesheet to make the base layer map tiles.
There are some subtleties for packing the overview data for the wider zooms, and what happens when you have tiles that are too big because of number of features or metadata size -- where you can drop on density or coalesce on density.
bambax
These being vector files, does it mean they can be styled on the fly?
bambax
(the answer is yes)
undefined
xxdiamondxx
Maplibre[1] + PMTiles + Felt's "tippecanoe"[2] (it can output .pmtiles) are an awesome combination for self-hosted web maps if you're ok with being locked into a Web Mercator projection
for pretty much any geospatial source you can convert to .pmtiles via GDAL[3] and tippecanoe (.shp .gpkg ...) | ogr2ogr -> .geojson | tippecanoe -> .pmtiles
for OpenStreetMap data there's planetiler[4], and and openmaptiles[5] styles that work with Maplibre
with those combinations you've got a great start to something you can host for pennies on AWS S3+CloudFront or Cloudflare R2, with an open source data pipeline
[2] https://github.com/felt/tippecanoe
[4] https://github.com/onthegomap/planetiler
[5] https://openmaptiles.org/styles/
ps I find GDAL/ogr2ogr documentation pretty hard to parse, as an example to get you started
ogr2ogr -f GeoJSON counties.json -t_srs EPSG:4326 -nln counties -sql "SELECT STATEFP, COUNTYFP, NAME FROM tl_2022_us_county" /vsizip/tl_2022_us_county.zip
https://www.census.gov/geographies/mapping-files/time-series/geo/cartographic-boundary.htmldoublemint2203
I'm a student trying to upskill in geospatial/compsci. What kinds of projects could you make with all of this? Any good starting points you'd recommend?
tony_cannistra
Not OP, but almost anything where "a map" is the output will be a hugely big learning opportunity for you. Static maps, print-quality maps, interactive web maps -- the experience of building any of those will bring lots of learning. (That's how I got started.)
xxdiamondxx
you can checkout https://github.com/maplibre/awesome-maplibre#users for some examples of what you can do with Maplibre
CSSer
I’ve contributed to this project, and we use it in production at work! It’s great. Brandon is very nice, and I really appreciate all the work that has been put into this.
tobilg
We've had quite some success with deploying "serverless" maps, leveraging PMTiles. Have a look at https://github.com/serverlessmaps/serverlessmaps if you're interested to deploy this on AWS CloudFront/Lambda@Edge/S3...
tobilg
I also wrote a blog post as introduction: https://tobilg.com/serverless-maps-for-fun-and-profit
globular-toast
I wonder if Github Pages supports HTTP range. Obviously putting a multi-GB blob on there would probably be abuse, but a small blob limited to a small region with limited detail would be really nice for having a map to accompany a blog post or something.
wcedmisten
I used it for this hospital accessibility map of Maryland. Sorry the legend looks terrible on mobile though
bdon
It does, but there is a limit of something like 500 MB for single files.
globular-toast
It would be cool to have an automatic tool to generate a blob of any size and it would include as much detail as possible within your given blob size limit...
bdon
You can kind of do that already with the pmtiles CLI:
pmtiles extract INPUT.pmtiles OUTPUT.pmtiles --bbox=BBOX --maxzoom=Z --dry-run
--dry-run will output the total size of the final archive without downloading any tiles, so you can adjust the maxzoom from the default of 15 (for basemaps) until it fits under your limit
nazgul17
I am curious about maps and the software to generate them and play with them, but when I tried - a few years ago, admittedly - to dabble in it (I wanted to generate those art-sy nice city maps pictures for my hometown [1]), I was overwhelmed with the ecosystem, all the steps involved, all the options etc.
Does anyone have a good introduction to paint a high level picture of this, so someone like me can navigate?
[1] https://www.etsy.com/nz/listing/1541532385/dusseldorf-print-...
ioshaan
[dead]
Get the top HN stories in your inbox every day.
I just used their pmtiles tool to grab a map of just the area around Half Moon Bay, south of San Francisco.
I grabbed the latest macOS Go binary from https://github.com/protomaps/go-pmtiles/releases
I found a rough bounding box using http://bboxfinder.com/#37.373977,-122.593346,37.570977,-122....
Then I ran this:
The result was a 2MB hmb.pmtiles file which, when dragged onto the "drop zone" on https://maps.protomaps.com/ showed me a detailed zoomable map, with full OpenStreetMap metadata (airport polygons etc) - all the way down to individual building outlines level.All that in 2MB! I'm really impressed.
My file is here: https://gist.githubusercontent.com/simonw/3c1a5358ca8f782f13...