---
title: "Webmapping and massive statistical data, a democratization story"
author: "Etienne Côme, @comeetie@mapstodon.space, FOSSDEM 2023"
date: 03/02/2023
format: 
  revealjs:
    theme: [default, uge.scss]
title-slide-attributes:
  data-background-image: "./logo_univ_gustave_eiffel_rvb.svg"
  data-background-size: "25%"
  data-background-position: "bottom 50px right 100px"
---

## @comeetie

![](https://www.comeetie.fr/assets/img/portrait.png){ width=30% }

Researcher Gustave Eiffel University COSYS/GRETTIA </br>
Stats, Data, Visualisation, Carto, Transports,...</br>
R,JS,Python,Cpp,...

## Francepixels, some context {background-image="./images/feuilleimpotswhite2.jpg"}

<div style="background-color:#ffffffbb">
Going back in 2014 !

Massive Grided statistical data 


::: {.incremental}
- derived from tax sheets
- 200m x 200m pixels
- limit of the statistical secret (11 households)
:::

</div>

## {background-image="./images/insee2014v2.png"}

## {background-image="./images/inseeqgis.png"}


## Some context 

open data but :

::: {.incremental}
- heavy files with tricky file formats and projection problems
- only usable with <span style="color:var(--alert-color)">pain</span>
- on dedicated software mapinfo / arcgis / qgis
::: 



## An opportunity

Webmapping stack developped around the <span style="color:var(--alert-color)">OSM</span> project

- [openLayer](https://openlayers.org/)
- [leaflet](https://www.leaflet.org/), 
- [mapbox tools of the time (tilemill,...)](https://web.archive.org/web/20140605004029/https://www.mapbox.com/)

<span style="color:var(--alert-color)">$\Rightarrow$ would allow to renew the data diffusion/visualiation approaches</span>

##  An opportunity

Not the only one to see this opportunity </br>

Oliver O'Brien did a quite similar proposal in the Uk in 2015 with data-shine

![](./images/obrien.png){ width=80% }

## {background-image="./images/datshine.png"}




##  Technical details and choices

At that time <span style="color:var(--alert-color)">vector tiles</span> were slowly comming

- [http://mike.teczno.com/notes/postgreslessness-mapnik-vectiles.html](http://mike.teczno.com/notes/postgreslessness-mapnik-vectiles.html)
- [https://observablehq.com/@d3/vector-tiles](https://observablehq.com/@d3/vector-tiles)
- [https://gist.github.com/wboykinm/7393674](https://gist.github.com/wboykinm/7393674)

<span style="color:var(--alert-color)">$\Rightarrow$ Massive advantage for statistical data</span>


##  Francepixels, Technical details and choices

Tool-chain :

- R to process the data and export geojson vector tiles
- leaflet for raster tiles 
- d3 hook to render "vectors tiles" on canvas
- invisible canvas tiles for interactivivity 
- details matter (color scales/map background/labels)

[try it, still usable](https://www.comeetie.fr/galerie/francepixels/#map/p65/vardefault/12/48.860/2.333)


## MAUP and multiscale aggregation

The modifiable areal unit problem <span style="color:var(--alert-color)">(MAUP)</span> is a statistical bias that occurs when you aggregate point data.

![](./images/MAUP-Scale-Effect-425x212.png){height=50%}

[see, https://gisgeography.com/maup-modifiable-areal-unit-problem/](https://gisgeography.com/maup-modifiable-areal-unit-problem/)




## MAUP and multiscale aggregation

The modifiable areal unit problem <span style="color:var(--alert-color)">(MAUP)</span> is a statistical bias that occurs when you aggregate point data.

<span style="color:var(--alert-color)">$\rightarrow$ try aggregating the raw data at different scales and link them with zoom levels</span>

solve two problems :

::: {.incremental}
- vector tiles sizes <span style="color:var(--alert-color)">!! we need to keep them small</span>
- MAUP <span style="color:var(--alert-color)">!! we want to explore several aggregation scales</span>
:::


## {background-image="https://www.comeetie.fr/galerie/francepixels/assets/images/paris_vieux2.jpg"}


## Feedbacks and users

- INSEE
- Journalists, on specific topics eg. segregation and school
- Urbanists for territorial diagnosis
- Transportation researchers
- Grided data user 
- Curious people ...

## Francepixels, evolutions

Vector tiles have since become a mature technology 

- MVT format [specifications 1.0 and 2.0](https://mapbox.github.io/vector-tile-spec/)

and are no widely available

- [postgis can now produce mapbox vector tiles](https://postgis.net/docs/ST_AsMVT.html)
- new webmapping tools [mapbox-gl](https://www.mapbox.com/mapbox-gljs) -> [MapLibre](https://maplibre.org/projects/),...


## Francepixels, evolutions

A lot easier now :

- [pypixgrid](https://github.com/comeetie/pypixgrid) python script to export MVT tiles from postgis
- [customizable multiscale aggregation](https://github.com/comeetie/pypixgrid/blob/main/exemples/insee/reunion/config_reunion.json)
- mapbox-gl / mapLibre for webmapping


## Francepixels successors 

::: {.incremental}
- [FrancePixels 2019](https://www.comeetie.fr/galerie/francepixels2019/) new data from insee  
- [FrancePixels Bati](https://www.comeetie.fr/galerie/francepixelsbati) custom aggreation of IGN BD Topo on side topics
- [FrancePixels 2023](https://www.comeetie.fr/galerie/francepixels2023bis) new data from insee + temporal evolution
- [gridViz by @julgaf (eurostats)](https://eurostat.github.io/gridviz/) 
- [datashine successor](https://www.ons.gov.uk/census/maps/choropleth/identity/sexual-orientation/sexual-orientation-4a/lesbian-gay-bisexual-or-other-lgb)
- insee project
- ...
:::




# Thanks


Questions ? Feedback 

