Personal Photo

Livia Foldes

This is a photograph I took of my older sister holding my niece. I took it at her apartment, just after I moved to Brooklyn last summer.

When I look at it, I remember the thick feeling of the humid July air. I see my sister’s tender expression, and how much my niece has grown in the last six months. I remember how I felt: nervous and excited about starting a new phase in my life.

1. Pixels

‘Info’ panel showing photo metadata

But there are many other ways to see this photo. When I instruct my Macbook’s operating system to ‘Get Info,’ I can see when it was taken, the device it was taken on, and more. I can see that its 7,990,272 pixels use 1,287,199 bytes of memory. 

I can open the photo in Photoshop and enlarge it until I see the places where the algorithm in my iPhone’s camera guessed at the colors it couldn’t see in the living room’s dim, late afternoon light.

I can zoom in until I see the individual pixels.

I can select a single pixel, 

and see the different formulas my computer uses to translate that pixel’s data into color.

2. Place

My iPhone remembers where I took the photo. 

In Google Maps, I can see a photograph of the exact spot.

The photograph has so many pixels that I can see my sister’s living room where I took the photo.

This photo has metadata, too. It was taken by an employee of Google in June, 2019.

Over the last eleven years, someone has photographed her street eight times, at semi-regular intervals.

Google’s maps, the surveillance they facilitate, and the wealth they generate have remade cities like Brooklyn. Its Street View photographs are at once artifacts of the mapping process, documents of gentrification, and a memory bank of the spaces lost in its wake (Wu; Naik, Nikhil, et al.).

A Street View car; a closeup of its camera; and a “visualization of the spatial coverage of each camera.”

Credit: Google AI blog

The anonymous photographers who take these images were likely hired through an ad on Craigslist, and were likely paid $15-16/hour to drive a car like this one. (McNeil; “Google Street View Driver Hourly Pay”; “r/GoogleMaps – How Does One Get a Job as Google Street View Driver?”)

I found this image on Google’s AI blog, in a 2017 post written by people whose job titles are “Software Engineer” and “Research Scientist, Machine Perception” (Krainin and Liu). The blog and its authors point to Google’s motivation for capturing, storing, and publishing a decade of memories of my sister’s street. The photographs are, in fact, “leftovers that happen to be images” (McNeil)—the visual artifacts of a massive data set used to train artificial intelligence systems to “see” our world (Paglen).

Separate images from Street View’s “rosette” camera being algorithmically “stitched together”

Credit: Google AI blog

The rectangular boundaries of the images I pulled from Street View are arbitrary; the engineer and the research scientist demonstrate the algorithms they use to eliminate “visible seams” from this boundless digital representation of our world (Krainin and Liu).

Depth sensors added to Street View cameras

Credit: Wired, “Google Maps Is Upgrading Street View and You Can Help”

The post was published the same year Google added 3D depth sensors to the “rosette” of cameras mounted to its Street View cars (Amadeo). These sensors are used, they explained, not just “to position us in the world,” but to “build a deeper data set of knowledge.” They are amassing this knowledge “so that when people come to Google and ask questions about the world around them, we have more and more of the answers” (“Google Maps Is Upgrading Street View and You Can Help”).

3. Vision

Promotional image of Waymo car
Credit: https://waymo.com/

In its website’s FAQs, Waymo (a subsidiary of Alphabet) describes itself as a “self-driving technology company… [built] on technology developed in Google’s labs since 2009”—the year my sister’s apartment was first photographed.

Waymo’s website features a scrolling animation purporting to show us what its cars see. As I scroll, it raises and answers a series of existential questions:

Detail of a visualization from Waymo’s website showing how its cars “see”
Credit: https://waymo.com/tech/
Detail of a visualization from Waymo’s website showing how its cars “see”
Credit: https://waymo.com/tech/
Detail of a visualization from Waymo’s website showing how its cars “see”
Credit: https://waymo.com/tech/

Over an image of a strip mall in Southern California, it asks: Where am I?

What’s around me?

What will happen next?

What should I do?

The visualization imagines a world made simple and legible—a world where my car can answer these questions for me. 

As I scroll, the street scene is overlaid with streamlined blue paths, bubbly icons, and text labels. Oncoming cars announce their velocity and distance from me. In this world, labels are definitive and meanings are fixed. This world erases the labor of the humans paid to categorize the millions of photographs taken by other humans that taught it to see (Crawford and Paglen).

A lone pedestrian, in a box shaded a cautionary orange, stands out from the blue-tinged cars.

The orange gradient is a graphic flourish meant for my eyes, not the car’s sensors. It is there to remind me, perhaps, that Waymo’s algorithmic vision is safe—that it sees humans differently than machines or stop lights.

And it does.

4. Recognition

My iPhone’s Photos app interface

My iPhone’s photo app has trained itself to recognize the people I photograph most often. 

Slide from Apple’s 2018 launch event illustrates the iPhone’s photo processing algorithm
Credit: James Martin/CNET (https://www.cnet.com/pictures/apple-iphone-xs-iphone-xr-event-photos/29/)

According to Apple, it does this by executing 1 trillion operations.

Caption: Detail of a diagram from Facebook’s publication of its DeepFace algorithm
Credit: (Taigman, Yaniv, et al)

Many of these operations involve facial detection and analysis. These algorithms are a way for our machines to look at—and, more importantly, categorize—us (Paglen). Calista Flockhart’s face becomes data, and the data is reconstituted into the label, ‘Calista Flockhart.’

Apple’s decision to illustrate this with an image of a black woman was not accidental. It was, perhaps, a nod to AI systems’ well-documented failures to categorize people with dark skin as human (Joy and Gebru).

Left: Google’s PoseNet; right: face-api.js

Facial recognition software has no trouble recognizing my sister and niece. After all, its inventors trained it on images just like this one (Hill and Krolik).

This photograph that I took to remember a specific time and place looks vastly different to me than it does to a machine. Soon, the machine will be able to see much more of what I see. But our readings will never be the same.

The machine and I interpret the photograph’s use and meaning in accordance with our distinct cultures, histories, and logics. We read complex “epistemic boundaries” (Crawford and Paglen) into its pixels—mine informed by my own experiences, the machine’s determined by its engineers and the companies they work for.

My memory, in the moment of its capture, was at once flattened and augmented, made both less and more. The pixels holding my memory are as unique as an unrepeatable moment, and as infinitely replicable as binary code. They are situated in space, and distributed across global networks. My memory’s metadata doesn’t (yet) contain the texture of my sister’s table, but it can locate her table in relation to the rest of the world. In this ontologically layered artifact, my niece and sister are rendered unknowable, even as they are intimately known.

Works Cited

Amadeo, Ron. “Google’s Street View Cars Are Now Giant, Mobile 3D Scanners.” Ars Technica, 6 Sept. 2017, arstechnica.com/gadgets/2017/09/googles-street-view-cars-are-now-giant-mobile-3d-scanners/.

Buolamwini, Joy, and Timnit Gebru. “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification.” Proceedings of Machine Learning Research, vol. 81, 2018, pp. 1–15.

Crawford, Kate, and Trevor Paglen. Excavating AI: The Politics of Images in Machine Learning Training Sets. 19 Sept. 2019, www.excavating.ai/.

“Google Maps Is Upgrading Street View and You Can Help – WIRED Videos.” Wired, Conde Nast, 5 Sept. 2017, www.wired.com/video/watch/google-maps-is-upgrading-street-view-and-you-can-help.

“Google Street View Driver Hourly Pay.” Glassdoor, www.glassdoor.com/Hourly-Pay/Google-Street-View-Driver-Hourly-Pay-E9079_D_KO7,25.htm.

Hill, Kashmir, and Aaron Krolik. “How Photos of Your Kids Are Powering Surveillance Technology.” The New York Times, 11 Oct. 2019, www.nytimes.com/interactive/2019/10/11/technology/flickr-facial-recognition.html.

Krainin, Mike, and Ce Liu. “Seamless Google Street View Panoramas.” Google AI Blog, 9 Nov. 2017, ai.googleblog.com/2017/11/seamless-google-street-view-panoramas.html.

Mattern, Shannon. “Mapping’s Intelligent Agents.” Places Journal, Sept. 2017, placesjournal.org/article/mappings-intelligent-agents/.

McNeil, Joanne. “The Windshield and the Screen.” u n t h i n k i n g . p h o t o g r a p h y, Feb. 2019, unthinking.photography/articles/the-windshield-and-the-screen.

Naik, Nikhil, et al. “Understanding Urban Change with AI.” Streetchange, streetchange.media.mit.edu/about.html.

Paglen, Trevor. “Invisible Images (Your Pictures Are Looking at You).” The New Inquiry, 8 Dec. 2016, thenewinquiry.com/invisible-images-your-pictures-are-looking-at-you/.

“r/GoogleMaps – How Does One Get a Job as Google Street View Driver?” Reddit, www.reddit.com/r/GoogleMaps/comments/80q8bt/how_does_one_get_a_job_as_google_street_view/.

Taigman, Yaniv, et al. “DeepFace: Closing the Gap to Human-Level Performance in Face Verification.” Facebook Research, 24 June 2014, research.fb.com/publications/deepface-closing-the-gap-to-human-level-performance-in-face-verification/.

Wu, Katherine J. “Could Google Street View Help Track Gentrification?” PBS, Public Broadcasting Service, 13 Mar. 2019, https://www.pbs.org/wgbh/nova/article/could-google-street-view-help-track-gentrification/.

Leave a Reply

Your email address will not be published. Required fields are marked *