- shinyZola2026-07-08T00:00:00+00:00https://pr-30.preview.ricochet.rs/blog/tags/shiny/atom.xmlScalable Spatial Viz: Lonboard and Shiny for Python2026-07-08T00:00:00+00:002026-07-08T00:00:00+00:00
Unknown
https://pr-30.preview.ricochet.rs/blog/high-performance-spatial/<p>With the release of ricochet v0.6 we've begun stabilizing support for <a rel="external" href="https://shiny.posit.co/py/">Shiny for Python</a>, rounding out our existing Shiny support across both R and Python.</p>
<p>To celebrate, we built a demo app (<a rel="external" href="https://github.com/ricochet-rs/test-apps/tree/main/python/shiny-py-lonboard">source</a>) that showcases modern scalable geospatial visualization using <a rel="external" href="https://developmentseed.org/lonboard/"><code>lonboard</code></a> from <a rel="external" href="https://developmentseed.org/">Development Seed</a>. The app visualizes the <a rel="external" href="https://www.kaggle.com/datasets/harlfoxem/housesalesprediction">King County housing sales dataset</a> (~22k home sales) on an interactive map with filtering by price, bedrooms, bathrooms, and square footage.</p>
<p>You can view the app live at <a rel="external" href="https://portal.ricochet.rs/app/shiny-lonboard">portal.ricochet.rs/app/shiny-lonboard</a>.</p>
<!-- TODO screenshot: full app view -->
<p>This blog post will discuss what makes lonboard <em>so fast</em> 🏎️💨.</p>
<h2 id="tl-dr">TL;DR</h2>
<ul>
<li>GeoJSON adds a <em>ton</em> bloat and is a real bottleneck</li>
<li>Using "end-to-end binary data pipeline" provides meaningful speed ups</li>
<li><a rel="external" href="https://geoparquet.org/">GeoParquet</a> results in smaller file sizes</li>
<li><a rel="external" href="https://geoarrow.org/">GeoArrow</a> native represention results in 0 cost data transfer</li>
</ul>
<h2 id="run-the-app-locally">Run the app locally</h2>
<p>If you want to run the app directly, the source code for the app is in our repo <a rel="external" href="https://github.com/ricochet-rs/test-apps/"><code>ricochet-rs/test-apps</code></a>.</p>
<p>Clone the example:</p>
<pre class="giallo" style="color: #E0DEF4; background-color: #191724;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #EBBCBA;">git</span><span style="color: #F6C177;"> clone</span><span style="color: #31748F;"> --filter=blob:none --sparse</span><span style="color: #F6C177;"> https://github.com/ricochet-rs/test-apps.git</span></span>
<span class="giallo-l"><span style="color: #EB6F92;font-style: italic;">cd</span><span style="color: #F6C177;"> test-apps</span></span>
<span class="giallo-l"><span style="color: #EBBCBA;">git</span><span style="color: #F6C177;"> sparse-checkout set python/shiny-py-lonboard</span></span>
<span class="giallo-l"><span style="color: #EB6F92;font-style: italic;">cd</span><span style="color: #F6C177;"> python/shiny-py-lonboard</span></span></code></pre>
<p>Install dependencies and run locally:</p>
<pre class="giallo" style="color: #E0DEF4; background-color: #191724;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #EBBCBA;">uv</span><span style="color: #F6C177;"> sync</span></span>
<span class="giallo-l"><span style="color: #EBBCBA;">uv</span><span style="color: #F6C177;"> run shiny run app.py</span></span></code></pre><h2 id="status-quo">Status Quo</h2>
<p>Interactive visualization of spatial data has traditionally been done using the library <a rel="external" href="https://leafletjs.com/">Leaflet</a> (via <a rel="external" href="https://python-visualization.github.io/folium/latest/">folium</a> in python and <a rel="external" href="https://rstudio.github.io/leaflet/"><code>{leaflet}</code></a> in R). There are other alternative such as <a rel="external" href="https://deckgl.readthedocs.io/en/latest/"><code>pydeck</code></a> and <a rel="external" href="https://github.com/keplergl/kepler.gl">kepler.gl</a>. The R community has also begun transitioning away from <code>{leaflet}</code> towards the newer and modern <a rel="external" href="https://walker-data.com/mapgl/"><code>{mapgl}</code></a>.</p>
<p>However, <strong>all of</strong> these libraries still suffer from the same bottleneck: <a rel="external" href="https://geojson.org/">GeoJSON</a>.
GeoJSON is an extension of <a rel="external" href="https://www.json.org/json-en.html">JSON</a> (JavaScript Object Notation).
It is a <em>pure text</em> representation of spatial data. And frankly, text <strong>is not</strong> an efficient way to store spatial data—a shapefile is likely better.</p>
<p>There are two primary issues with GeoJSON:</p>
<ol>
<li>It is text which grows in size very quickly</li>
<li>Data in GeoJSON needs to be <em>parsed</em> into an in-memory representation</li>
</ol>
<p>Converting an in memory object such as a data frame (e.g. <code>GeoDataFrame</code> or <code>sf</code>) to another format is called <strong>serialization</strong>.
Serializing to GeoJSON can be <em>very</em> slow. On the flip side, reading from GeoJSON—a.k.a. <strong>deserialization</strong>—itself is even slower.
Other file formats such as <a rel="external" href="https://flatgeobuf.org/">FlatGeobuf</a> enable users to read only relevant portions of the file (random access).</p>
<p>Most browser-based mapping tools (Leaflet, MapLibre, pydeck, etc) require your data to arrive as GeoJSON.</p>
<p>This results in a workflow like so:</p>
<pre class="mermaid">
flowchart LR
A[GeoDataFrame] --> B[GeoJSON]
B --> C[Send to the browser]
C --> D[Parse the GeoJSON]
D --> E[Render]
</pre>
<p>The (de)serialization steps take up the bulk of the time.</p>
<p>The whole workflow is fraught.
Serializing a GeoDataFrame toe GeoJSON is slow. The string itself is large.
Parsing it in the browser is slow. This is not noticeable for small data. But but the time you have tens of thousands of features, the round-trip becomes a bottleneck.</p>
<p>Lonboard's special sauce is that it uses an <a rel="external" href="https://nyhackr.blob.core.windows.net/presentations/Lonboard-Fast-Interactive-Geospatial-Vector-Data-Visualization-in-Jupyter-and-JavaScript_Kyle-Barron.pdf">"end-to-end binary data pipeline"</a> skipping GeoJSON entirely.</p>
<h2 id="lonboard-s-innovations">Lonboard's innovations</h2>
<p>There are 3 major innovations to lonboard that make it so powerful:</p>
<ol>
<li>Lonboard uses <a rel="external" href="https://geoparquet.org/"><strong>GeoParquet</strong></a> instead of GeoJSON</li>
<li>Data is stored as <strong>GeoArrow</strong> in memory</li>
<li>It uses deck.gl, a GPU powered renderer</li>
</ol>
<p>By relying on GeoParquet and GeoArrow throughout the entire pipeline, lonboard avoids expensive (de)serialization costs every step of the way.</p>
<h3 id="geoparquet">GeoParquet</h3>
<p>Lonboard eliminates the text serialization entirely by leaning on <a rel="external" href="https://geoparquet.org/">GeoParquet</a>. GeoParquet is an extension to the <a rel="external" href="https://parquet.apache.org/">Parquet</a> file format, which is itself a compressed binary columnar format. Instead of sending a multi-megabyte string across the wire and then parsing it, lonboard sends the GeoParquet file as binary.</p>
<p>The workflow looks like this:</p>
<pre class="mermaid">
flowchart LR
A[GeoDataFrame] --> B[Write to GeoParquet]
B --> C[Send to the browser]
C --> D[Read with parquet-wasm]
D --> E[Pass directly to deck.gl]
</pre>
<p>According to the <a rel="external" href="https://developmentseed.org/lonboard/latest/how-it-works/#how-is-it-so-fast">lonboard docs</a>:</p>
<blockquote>
<p>Saving GeoArrow to Parquet was 135x faster than saving a GeoDataFrame to GeoJSON. The resulting file was 26x smaller. And parsing the Parquet to Arrow on the frontend was 5.6x faster than <code>JSON.parse</code>.</p>
</blockquote>
<p>The format yields speedups orders of magnitude faster than we could achieve with GeoJSON.
This change makes it possible to ship datasets to the browser that were previously off the table.</p>
<div class="callout-tip">
Watch Kyle Barron's (author of lonboard) great talk on <a href="https://www.youtube.com/watch?v=KWQMa-y8CNg">Why GeoParquet Matters</a>.
</div>
<h2 id="geoarrow">GeoArrow</h2>
<p>The real benefit that we get from GeoParqeut comes from the usage of GeoArrow!
GeoArrow is an extension of the <a rel="external" href="https://arrow.apache.org/">Apache Arrow</a> (Arrow for short) specification.
Arrow specifies how data should be represented <em>in memory</em>.</p>
<p>Arrow is a <a rel="external" href="https://arrow.apache.org/docs/format/Intro.html"><em>columnar</em> storage format</a>. Simply put, it means that data is stored in columns rather than rows. Doing so make it so that only columns can be accessed without having to sift through every single row of a dataset.</p>
<div class="flex flex-col items-center gap-2 my-4">
<img src="https://arrow.apache.org/docs/_images/columnar-diagram_1.svg" class="w-full sm:w-[70%]"/>
<p class="text-sm text-muted-foreground text-center">Diagram from the <a href="https://arrow.apache.org/docs/format/Intro.html" class="underline hover:text-foreground">Apache Arrow columnar format specification</a></p>
</div>
<p>Arrow is a <em>standard</em> and not a library. The benefit of this is that Arrow data can be sent between tools / languages / runtimes without incurring a (de)serialization cost enabling "seamless" interoperability.</p>
<div class="flex flex-col items-center gap-2 my-4">
<img src="https://josiah.rs/posts/cross-language-ds/images/paste-8.png" class="w-full sm:w-[70%]"/>
<p class="text-sm text-muted-foreground text-center">Diagram from <a href="https://josiah.rs/posts/cross-language-ds" class="underline hover:text-foreground">Apache Arrow, Rust, and cross-language data science</a></p>
</div>
<p>GeoArrow builds upon the Arrow specification to create a standard for how spatial data shuold be represented in memory. Meaning any tool that can use GeoArrow can use the same data—i.e. GeoArrow from Python can be used by <a rel="external" href="https://github.com/geoarrow/geoarrow-r"><code>geoarrow-r</code></a> or <a rel="external" href="https://arrow.apache.org/nanoarrow/latest/r/"><code>{nanoarrow}</code></a>.</p>
<h3 id="geoarrow-handshake-geoparquet">GeoArrow 🤝 GeoParquet</h3>
<p>The <a rel="external" href="https://geoparquet.org/releases/v1.1.0/">GeoParquet 1.1</a> release added support for <strong>native</strong> GeoArrow encoding. When geometry is stored in native GeoArrow format inside a Parquet file, the browser can read the file and access the geometry directly without converting the data <em>to</em> GeoArrow.</p>
<p><a rel="external" href="http://deck.gl/">deck.gl</a>, the tool used by lonboard, has an extension that supports the use of native GeoArrow.
Thus, coupling GeoParquet and native GeoArrow enables transferring data from disk to GPU without any intermediate format.</p>
<p>This is what "zero-cost serialization" means in practice—the format you store the data in is the same format the renderer consumes.</p>
<h2 id="deckgl-and-geoarrow">DeckGL and GeoArrow</h2>
<p>The piece that ties this all together is <a rel="external" href="https://github.com/geoarrow/deck.gl-geoarrow"><code>@geoarrow/deck.gl-geoarrow</code></a>. It is a small "glue library" that lets deck.gl consume GeoArrow data. <code>deck.gl-geoarrow</code> takes an Arrow Table from JavaScript and copies it directly to the GPU. deck.gl supports GeoArrow natively (specifically the interleaved coordinate layout).</p>
<p>The deck.gl-geoarrow demo renders 3.2 million points with a <code>GeoArrowScatterplotLayer</code>. For context, that is roughly 150x the size of the King County dataset our demo app ships.</p>
<h2 id="gpu-rendering">GPU rendering</h2>
<p>The other half of the story is where rendering actually happens. Leaflet renders using the CPU. Every marker and every polygon is drawn using the CPU which works fine for a few hundred features, but it does not scale nearly as well as the GPU.</p>
<p>Why would one use the CPU to do graphics when modern GPUs are incredibly powerful? Modern enhancements to GPUs have made using them to render graphics very, very fast and very efficient.</p>
<p>Lonboard renders via <a rel="external" href="https://deck.gl/">deck.gl</a>, which runs on <a rel="external" href="https://www.khronos.org/webgl/">WebGL</a> and pushes geometry to the GPU.</p>
<p>The remaining piece is <a rel="external" href="https://github.com/kylebarron/parquet-wasm">parquet-wasm</a>, which reads the Parquet file in the browser via WebAssembly. <a rel="external" href="https://webassembly.org/">WebAssembly</a> (Wasm) is an instruction format that lets you run code written in languages like Rust or C++ inside the browser at near-native speed. <code>parquet-wasm</code> uses WASM to read Parquet files directly in the browser, so the data never has to be converted out of its native format before reaching deck.gl.</p>
<h2 id="building-the-shiny-app">Building the Shiny app</h2>
<p>Since lonboard integrates with Jupyter Widgets via <a rel="external" href="https://anywidget.dev/">anywidget</a>, it works in Shiny for Python through <a rel="external" href="https://github.com/posit-dev/py-shinywidgets">shinywidgets</a> with minimal glue code. The <code>@render_widget</code> decorator drops the map directly into the UI:</p>
<pre class="giallo" style="color: #E0DEF4; background-color: #191724;"><code data-lang="python"><span class="giallo-l"><span style="color: #908CAA;">@</span><span style="color: #EBBCBA;">render_widget</span></span>
<span class="giallo-l"><span style="color: #31748F;">def</span><span style="color: #EB6F92;font-style: italic;"> map</span><span style="color: #908CAA;">():</span></span>
<span class="giallo-l"><span style="color: #31748F;"> return</span><span> Map</span><span style="color: #908CAA;">(</span></span>
<span class="giallo-l"><span style="color: #C4A7E7;font-style: italic;"> layers</span><span style="color: #31748F;">=</span><span style="color: #908CAA;">[</span><span>layer</span><span style="color: #908CAA;">],</span></span>
<span class="giallo-l"><span style="color: #C4A7E7;font-style: italic;"> view_state</span><span style="color: #31748F;">=</span><span style="color: #908CAA;">{</span></span>
<span class="giallo-l"><span style="color: #F6C177;"> "longitude"</span><span style="color: #908CAA;">:</span><span style="color: #31748F;"> -</span><span style="color: #EBBCBA;">122.2</span><span style="color: #908CAA;">,</span></span>
<span class="giallo-l"><span style="color: #F6C177;"> "latitude"</span><span style="color: #908CAA;">:</span><span style="color: #EBBCBA;"> 47.5</span><span style="color: #908CAA;">,</span></span>
<span class="giallo-l"><span style="color: #F6C177;"> "zoom"</span><span style="color: #908CAA;">:</span><span style="color: #EBBCBA;"> 9</span><span style="color: #908CAA;">,</span></span>
<span class="giallo-l"><span style="color: #908CAA;"> },</span></span>
<span class="giallo-l"><span style="color: #C4A7E7;font-style: italic;"> show_side_panel</span><span style="color: #31748F;">=</span><span style="color: #EBBCBA;">False</span><span style="color: #908CAA;">,</span></span>
<span class="giallo-l"><span style="color: #908CAA;"> )</span></span></code></pre><h3 id="efficient-filtering">Efficient Filtering</h3>
<p>Transferring data is where much of the bottleneck comes from.
When adding interactivity, it's critical that data only be sent <em>once</em>.
Recreating the widget sends the full dataset from Python to the browser.
Lonboard's <code>DataFilterExtension</code> solves this by moving filtering to the GPU.
It sets the filter range inside of the browser and, critically, does not re-render the widget with the subsetted data.</p>
<pre class="giallo" style="color: #E0DEF4; background-color: #191724;"><code data-lang="python"><span class="giallo-l"><span style="color: #908CAA;">@</span><span style="color: #EBBCBA;">reactive</span><span style="color: #908CAA;">.</span><span style="color: #EBBCBA;">effect</span></span>
<span class="giallo-l"><span style="color: #31748F;">def</span><span style="color: #EBBCBA;"> _update_filter</span><span style="color: #908CAA;">():</span></span>
<span class="giallo-l"><span> layer</span><span style="color: #908CAA;">.</span><span>filter_range</span><span style="color: #31748F;"> =</span><span style="color: #908CAA;"> [</span></span>
<span class="giallo-l"><span style="color: #908CAA;"> [</span><span style="color: #EB6F92;font-style: italic;">input</span><span style="color: #908CAA;">.</span><span>price_min</span><span style="color: #908CAA;">(),</span><span style="color: #EB6F92;font-style: italic;"> input</span><span style="color: #908CAA;">.</span><span>price_max</span><span style="color: #908CAA;">()],</span></span>
<span class="giallo-l"><span style="color: #908CAA;"> [</span><span style="color: #31748F;">BED_CHOICES</span><span style="color: #908CAA;">[</span><span style="color: #EB6F92;font-style: italic;">input</span><span style="color: #908CAA;">.</span><span>bedrooms</span><span style="color: #908CAA;">()],</span><span style="color: #31748F;"> BEDS_MAX</span><span style="color: #908CAA;">],</span></span>
<span class="giallo-l"><span style="color: #908CAA;"> [</span><span style="color: #31748F;">BATH_CHOICES</span><span style="color: #908CAA;">[</span><span style="color: #EB6F92;font-style: italic;">input</span><span style="color: #908CAA;">.</span><span>bathrooms</span><span style="color: #908CAA;">()],</span><span style="color: #31748F;"> BATHS_MAX</span><span style="color: #908CAA;">],</span></span>
<span class="giallo-l"><span style="color: #908CAA;"> [</span><span style="color: #EB6F92;font-style: italic;">input</span><span style="color: #908CAA;">.</span><span>sqft_min</span><span style="color: #908CAA;">(),</span><span style="color: #EB6F92;font-style: italic;"> input</span><span style="color: #908CAA;">.</span><span>sqft_max</span><span style="color: #908CAA;">()],</span></span>
<span class="giallo-l"><span style="color: #908CAA;"> ]</span></span></code></pre>
<p>This ensures that the map doesn't re-render.</p>
<!-- gif: filtering in action -->
<h2 id="the-broader-ecosystem">The broader ecosystem</h2>
<p>This approach is not Python-only. In the R ecosystem, the <a rel="external" href="https://r-consortium.org/all-projects/2025-group-2.html#modernizing-rs-web-mapping-capabilities">r-consortium has funded active work</a> on <a rel="external" href="https://github.com/r-spatial/geoarrowWidget">geoarrowWidget</a> to bring the same GeoArrow-native pipeline to R.</p>
<p>The <a rel="external" href="https://walker-data.com/mapgl/">mapgl R package</a> is already fast and highly effective, though it <a rel="external" href="https://github.com/walkerke/mapgl/discussions/71">currently still uses GeoJSON</a> for data transfer. The direction of travel is clear.</p>
<h2 id="deploy-to-ricochet">Deploy to Ricochet</h2>
<p>Install the CLI, add the server, and deploy:</p>
<pre class="giallo" style="color: #E0DEF4; background-color: #191724;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #EBBCBA;">curl</span><span style="color: #31748F;"> -fsSL</span><span style="color: #F6C177;"> https://raw.githubusercontent.com/ricochet-rs/cli/main/install.sh</span><span style="color: #31748F;"> |</span><span style="color: #EBBCBA;"> sh</span></span>
<span class="giallo-l"><span style="color: #EBBCBA;">ricochet</span><span style="color: #F6C177;"> servers add try https://try.ricochet.rs</span></span>
<span class="giallo-l"><span style="color: #EBBCBA;">ricochet</span><span style="color: #F6C177;"> login</span><span style="color: #31748F;"> -S</span><span style="color: #F6C177;"> try</span></span>
<span class="giallo-l"><span style="color: #EBBCBA;">ricochet</span><span style="color: #F6C177;"> deploy</span></span></code></pre><h2 id="further-reading">Further reading</h2>
<ul>
<li><a rel="external" href="https://developmentseed.org/lonboard/latest/how-it-works/#how-is-it-so-fast">How lonboard works</a></li>
<li><a rel="external" href="https://www.youtube.com/watch?v=KWQMa-y8CNg">Why GeoParquet (Kyle Barron talk)</a></li>
<li><a rel="external" href="https://geoparquet.org/releases/v1.1.0/">GeoParquet 1.1 spec</a></li>
<li><a rel="external" href="https://github.com/kylebarron/parquet-wasm">parquet-wasm</a></li>
<li><a rel="external" href="https://github.com/r-spatial/geoarrowWidget">geoarrowWidget for R</a></li>
<li><a rel="external" href="https://blog.djnavarro.net/posts/2021-11-19_starting-apache-arrow-in-r/#overview-of-arrow">Getting started with Apache Arrow</a></li>
<li><a rel="external" href="https://josiah.rs/posts/cross-language-ds/">Apache Arrow, Rust, and cross-langauge data science</a></li>
</ul>