# Multiplicity

_A collective photographic city portrait_

Published: 2018-04-03

Today, we collectively and continuously document our city experience on social media platforms, shaping a virtual city image. Multiplicity reveals a novel view of this photographic landscape of attention and interests. How does Paris look as seen through the lens of thousands of photographers? What are the hotspots of attraction, what are the neglected corners? What are recurring poses and tropes? And how well do the published pictures reflect your personal view of the city?

## Key visuals

- Overview of the image map, thousands of Paris photos scattered as a cloud clustered by visual similarity
- Mid-zoom of the map showing round photo thumbnails with a human silhouette for scale
- Cluster of visitor photos of Monet's Water Lilies panoramas in the Orangerie's curved rooms
- Tight cluster of tourist photos posing in front of the Eiffel Tower
- Close-up of the touchscreen console showing the annotated map, projected photo grid behind
- Installation view of the console before the wide projection of the glowing photo cloud and title text
- Zoomed annotated map with labeled clusters including Food, Misc, Louvre and Tour Eiffel
- Grid of thousands of photos in one cluster, shading from dark nightlife to bright fashion portraits
- Cluster of art and design photos, typographic signs, murals and colorful street-art patterns
- Zoomed cluster of muted grey-toned art and painting photos with a human silhouette for scale
- Tight cluster of tattoo photos, arms and bodies showing delicate line-work designs
- Cluster of people posing at the blue Wall of Love (Le mur des je t'aime) in Montmartre
- Tight cluster of near-identical photos of people striking the same dance and yoga poses
- Cluster of hairdresser photos showing blonde balayage and wavy hairstyles seen from behind
- Tight cluster of photos of the Grande Roue Ferris wheel at Place de la Concorde
- Multiplicity title logo spelled out in a grid of letters on black
- Visitors reading the French introduction beside the projected sparse photo cloud
- Two visitors at the console before a projected wall of Paris cityscape and landmark photos
- A visitor alone at the console facing the glowing photo cloud in the darkened room
- Two visitors from behind exploring the map at the console before the full photo-grid wall
- Visitor silhouetted at the console between projected walls of neon nightlife and fashion street style
- Projection wall filled with hundreds of Eiffel Tower photos around the touchscreen console
- Gallery view of the installation booth with ceiling projectors and a visitor entering
- Projected wall spanning Monet Water Lilies panoramas and gilt-framed Louvre paintings including the Mona Lisa
- Joystick console before a projected wall zoomed into the tattoos cluster of arms and body art
- Multiplicity title logo, the word emerging from a grid of repeated letters

![Mid-zoom view of the image map, thousands of Paris photos clustered by visual similarity, with a human silhouette for scale]()

This interactive installation provides an immersive dive into the image space spanned by hundreds of thousands of photos taken across the Paris city area. Using machine learning techniques, the images are organized by similarity and image contents, allowing to visually explore niches and microgenres of image styles and contents.

![Grid of thousands of Instagram photos in one cluster, shading from dark nightlife shots to bright fashion portraits]()

Special emphasis is put on tight clusters of very similar images around a specific location: collective re-enactments of typical poses — same, same, but different.

To me, these very tight clusters of almost identical images became the most interesting aspect. How often can people take the same photos? At the same time, each of them is slightly different indeed, and the continuous re-enactment of rituals and re-discovery of photo ideas has a comforting charme to it as well.

Here are some of my favorites (but there are many more to explore):

![Tight cluster of near-identical photos of people striking the same dynamic dance and yoga poses]()
![Cluster of visitor photos of Monet's Water Lilies panoramas in the curved rooms of the Orangerie]()
![Cluster of art and design photos, typographic signs, murals and colorful street-art patterns]()
![Tight cluster of photos of the Grande Roue Ferris wheel at Place de la Concorde, day and night]()
![Zoomed cluster spanning colorful macarons and pastries into the skulls and bones of the Catacombs]()

## Installation & Interaction

The projection spans three 1080p squares arranged in a slightly angled tryptich structure, allowing an immersive dive into the image space.

![Installation view, touchscreen console before a wide projection of the glowing photo cloud, with title and text on angled walls]()

Visitors can navigate the map using a touch device as well as a physical joystick. Manual annotations help with identification of the main map areas. The application goes to sleep and starts to dream of Paris after a while of inactivity.

![Installation view with joystick console, projected wall zoomed into the tattoos cluster of arms and body art]()

![Touchscreen interface showing the annotated overview map with handwritten cluster labels and joystick navigation controls]()

![Installation view, projected wall spanning Monet Water Lilies panoramas and gilt-framed Louvre paintings including the Mona Lisa]()
![Two visitors seen from behind exploring the map at the console before the full photo-grid projection wall]()

![Installation view, projection wall filled with hundreds of Eiffel Tower photos around the touchscreen console]()

The projected display seamlessly zooms from the cloudy overview map over a gridded version of the cloud to a full grid. This layering allows to understand the clustering and neighborhood structure well in the zoomed out view, while providing a tidy and efficient image display in zoomed views.

See a screen capture of the flight animations here:

## Content selection and data processing

Based on a sample of 6.2m geo-located social media photos posted in Paris in 2017, a custom selection of 25'000 photos was chosen and forms the basis for the map.

It consists of
- 1 part top liked photos
- 1 part uniform spatial sample
- ca. 1 part hand-selected clusters
- ca. 2 parts most recent images from most active locations

The images were analysed using neural networks which were trained for image classification (from [tensorflow](https://github.com/tensorflow/tensorflow) with [keras](https://keras.io/)). I used feature vectors normally intended for classification to calculate pairwise similarities between the images. The map arangement was calculated using [t-sne](https://lvdmaaten.github.io/tsne/) — an algorithm that finds an optimal 2D layout so that **similar images are close together**.

## Layout strategy

Bridging the cloudy overview and the griddy detail view was an interesting challenge. After a couple of [failed approaches](https://vimeo.com/257880299/8bf4b950ed), here's how I solved it:

![Animation cycling through the four layout stages, from the scattered tsne photo cloud to a filled grid]()

**1** The initial, zoomed out view lays out the images according to the tsne coordinates. We draw images, which are more central to local clusters (technically speaking: having the highest degree centralities in a k-nearest-neighbors network), a bit bigger, in order to emphasize the cluster structure. Finally, the image is treated with a bit of post-production: Photoshop's smart blur filter and some tweaks to image contrast provide a better overview and pleasant zooming behavior.

**2** The second, gridded view puts every image on a rasterized coordinate system, as close to their original coordinates as possible. In this layout, some images will be hidden behind others; again, we put the ones on top which are more central to local cluster structure. Also this image is slightly blurred, in order to avoid flickering artefacts when zooming and panning.

**3** Finally, the fully gridded view assigns all previously hidden images to the next best free spot, thus leading to an almost fully filled grid.

**4** At the corners, due to the round nature of the cloud, some images were missing, so we iteratively filled these with the next best match (i.e. most similar image to the neighbors) from a larger image pool.

Altogether, this strategy makes sure that as little pixels change between the different transitions, thus leading to a consistent and satisfying zoom experience. Check out a slow motion version of the zooms below:

## Data-less visualization

As a final remark — it has been my intention **not to measure, but portray the city**, but to portray it, using social media contents as material. Rather than statistics, the project presents a stimulating arrangement of qualitative contents, open for exploration and to interpretation — consciously curated and pre-arranged, but not pre-interpreted.

At the same time, data was instrumental in arranging these contents in a human digestible way. How else would one scan and assemble hundreds of thousands of images into a coherent whole?

So, one could say, data was used as a vehicle for the experience and design process, but not as the object nor the end point of the visualization.

![Overview map of the photo cloud with handwritten annotations marking clusters like Food, Tattoos, Louvre and Eiffel Tower]()

The interplay between automatic analysis, inspection of the results — what does the machine suggest and conclude — and my own actions — (in terms of layout, content selection, parameter tweaking…) was inspiring to explore.

As a design hint, the use of handwriting for the map annotations hints at the involvement of me as an active author and a subjective sense-making process.

The final result emerged from a dialogue between me and the city, the image contents and the algorithms, which actually managed to surprise and inspire me throughout the project.

![Animated Multiplicity title logo, the word spelled out through a shifting grid of letters]()

## Related work

This project is part of a series of investigations of digital city images:
- [Stadtbilder](/projects/stadtbilder) investigates the digital layers of the city, beyond physical infrastructure.
- [Selfiecity](/projects/selfiecity) takes a closer look at the selfie phenomenon, and among other things, aims to identify unique styles of self-photography in different cities.
- [On Broadway](/projects/on-broadway) starts with a single street, and stacks all kinds of quantitative and  qualitative information on top of each other.

I was also inspired by a couple of projects from other folks:
- [Exactitudes](http://www.exactitudes.com)
- [Diorama maps](https://web.archive.org/web/20250511152844/https://soheinishino.net/dioramamap)
- [City Scan](https://web.archive.org/web/20220814190146/http://www.spehr.ch/city_scan/made_in_paris/)
- [Terrapattern](https://web.archive.org/web/20250525222620/https://terrapattern.com/)
- [fw4](https://uclab.fh-potsdam.de/fw4/en/vis/)

## Thanks to

- [Christian Laesser](https://christianlaesser.com), [Dominikus Baur](https://do.minik.us), [Tobias Kauer](https://twitter.com/tobi_vierzwo), [Yuri Vishnevsky](http://yuri.is), [Jia Her Teoh](https://www.linkedin.com/in/herteoh), [Bernard Kerr](https://www.linkedin.com/in/bernardkerr), [Baldo Faieta](https://www.researchgate.net/profile/Baldo_Faieta) and [Christopher Pietsch](https://uclab.fh-potsdam.de/people/christopher-pietsch/)
for feedback and comments
- the communities behind [react.js](https://reactjs.org), [mobx](https://mobx.js.org), [pixi.js](http://www.pixijs.com), [d3](https://d3js.org/), [electron](https://electronjs.org), [parcel.js](https://parceljs.org), [keras](https://keras.io), [tensorflow](https://github.com/tensorflow/tensorflow), [pandas](http://pandas.pydata.org), [scikit-learn](http://scikit-learn.org), [jupyter](http://jupyter.org)
- [Mario Klingemann](http://quasimondo.com) and [Gene Kogan](http://genekogan.com)
for their extremely helpful [tutorials](http://ml4a.github.io/guides/) and [open source projects](https://github.com/Quasimondo/RasterFairy),
- the team at [Fondation EDF](https://web.archive.org/web/20191117153347/https://fondation.edf.com/en) (Catherine Jaffeux, Benoit Franqueville), [trafik](http://www.lavitrinedetrafik.fr) and [David Bihanic](http://www.davidbihanic.com)
for the opportunity and the production.

## Exhibition

The project is at display as part of the [123 data](http://123data.paris) exhibition at [Fondation EDF](https://web.archive.org/web/20191117153347/https://fondation.edf.com/en), Paris, May — September 2018.

![Gallery view of the installation booth with ceiling projectors and a visitor entering the projection room]()
![Two visitors at the console before a projected wall of Paris cityscape and landmark photos]()
![Visitors reading the French introduction beside the projected sparse photo cloud of the map overview]()
![A visitor alone at the console facing the glowing photo cloud projected across the darkened room]()

## Press material

You can find some hi-res screenshots and photos in a [dropbox folder](https://www.dropbox.com/sh/jage7v8i8d44t3t/AAC3gjK0Uyw-mbxIXTebysj2a?dl=0).
Also feel free to embed the [flight demo](https://www.youtube.com/watch?v=qvcaVKxyitA) and the [zoom layer](https://www.youtube.com/watch?v=RY4_jANyy0o) videos or download [social media teaser video](https://www.dropbox.com/s/cjxjvjbif1z3xrp/mutliplicity-teaser-square.mp4?dl=0).

<a href="mailto:moritz@stefaner.eu?subject=Hi there">Get in touch</a> for any questions.

## Credits

Comissioned by Fondation EDF, Paris on occasion of the exhibition [123 data](http://123data.paris)

Design and implementation: Moritz Stefaner

---
[View on truth-and-beauty.net](https://truth-and-beauty.net/projects/multiplicity)