january 20th
New York City 2018


pop up windows
#42 metatorium


This is a retrospective, AKA postmortem of what was it like to create Metaroruim. Done for a class at ITP called “Pop-up Window Displays”. Metatorium was located at the NYU Broadway Windows.

A project done alongside 3 other people; Regina Cantú, Alexia Kyriakopoulos, and Ari Meliciano. It goes without saying that this post is from my point of view of the experience.


Origin Story

The main idea behind this was an observation post, a place where people could observe themselves and others, we wanted to play with the obsession most people unconsciously have towards themselves.

With that in mind I came up with a basic prototype to be able to test out the interaction.

For the last couple months I had personally been very interested in face detection, but the project to apply this never came, until this one. it was a good chance to try it.

A screen was set up to show a mirrored image of a webcam footage, this way people would see themselves and maybe, react to it. The footage above is a portion of the images taken during said test. This was not only a mirror, but a python script that with openCV would take captures every x amount of seconds.

From this captures, openCV would look for faces, and if one was found, it would crop it and save it in a different folder.

I am fully aware of the absence of any kind of technical explanation here. Soon enough that will be fixed. I will explain the very basics of openCV and how we made use of it for the setup we had.

Success!

The test was very successful, people flocked around it, even if nothing was apparently happening, they would still be there. I indeed took images and cropped them but that first time I didn’t do anything with them.

We realized that we needed some kind of response/reward for being there, besides just seeing your face in a big screen. So we made use of those cropped images.

The next time we set up the application a window was added showing the last image cropped. For obvious reasons the results were even better.

Surely, people used the “mirror” even more, and we observed that they would come back to have their face on the screen again.

At this point, what was being used to display the last faced that had been cropped, was a new window created with openCV, that showed to be quite problematic and the main cause for crashing the script when more than 1 face was found in a picture taken.

Said crash would not stop the whole process, but only part if ot, meaning that it would get stuck in a loop checking for faces in the same capture.

This was not the only problem we had, the detection was not precise enough, it would get stuck thinking that a piece of furniture or a smudge in the floor were faces. This was adjusted through the settings, but there was always that sneaky shadow that would still be there.

Building it

Now we knew how the interaction was, so the work was needed for this interaction was going to be shown.

Coming back to the idea of an observation post, lost of plausible representations came to mind, but the one that stuck, was the vision of a plant-like organism that would feed itself from faces, meaning that instead of sunlight, this plant would need people and their faces. The aura of this was a tech induced plant. With iridescent leafs, light producing branches, and screens as flowers.

This is the shape it had on the day of the opening.

“Front end”

This plant-like thing that has screens as flowers, needs a lot of them, and a Mac Mini can handle so many. That meant having 1 mac mini for every 2 displays. At one point we were planing to have up to 8 screens but that happened to be obviously too much. We needed to figure out two things, both of them equally important, the first one, the content in each of the screens, and the second one, how to arrange these.

To figure out the arrangement we set up a space to be a placeholder for the window-to-be. That way we were able to start playing around with the different possibilities.

the “mock-up” area

testing different configurations

After a while, we came to the conclusion that we needed/could fit 6 screens in the space we had. These would be placed with a speed-rail structure that later was hidden with black foamcore. Once this was done we needed to decide the what the contents of each screen would be.

Knowing that the central element was the mirror, it was clear that the main monitor would be it. After that, we had two more things that had to be shown, one was the last face that got taken, and the other one was an array of some of the other collected faces before the last one. The final content was the on shown in the following section. Besides the content of the screens, something else that had to be resolved was the communication between the three Mac Minis that we where gonna use.

“Back end”

Here we talk about how the screens and their respective computers talked to each other so the content would show as we wanted it to be. I’m fairly sure that there are way better ways of doing this and that this is not “long-term-proof”, but we had two weeks to make it work, and we went with what we had

This diagram has the 6 screens represented, each of them has a different number (1-6) and each of the 3 computers is represented by a letter (A-C).

Computer A it’s the main one. It runs the python script that captures the images from the camera feed seen in screen 2, analyses the images and crops the found faces displayed in screen 3. Screen 1 is showing the terminal for the script running, that way people would realise what is going on.

All these images are stored in a hard drive connected directly to A. To be able to have the images read by other computers and be able to make use of them, we had a switch that would connect all the machines together.

something to note here is that, because of your lack of experience, and even shorter time frame we needed to make sure all the system was as bulletproof as possible, to do so we had multiple simple programs running, instead of a big ass one that would do everything.

The only thing computer B would do was run a python and node script; a web server, the server and the .py script would be listening for newly added files to the directory where the cropped images were posted. This server would take the path of this new face, and send it to the clients, this would display it in two different ways; screen 3 would show only the last found face, and screen 4 would add it to an array of images creating a compilation of all the faces found since connected.

The python script would be connected through serial to an arduino that would trigger a set of LED strips and light up the “tree” and consequently bringing it to life.

Machine C and its screens were a different camera feed and two processing sketches that would apply effects to the footage.

the lights in action.

location/setup

The class this was made for, pop up window displays, historically has not been shown in these windows. Past years the place was not in Broadway, and the windows were way smaller. That meant that each team would have to do two windows. this year we had much more space to play with, but only one window. In my opinion I think we were lucky.

corner between 11st and Broadway

We had around 2 days to set up everything, it took us one to put everything together and almost two to figure out the whole aesthetic and arrangement of the elements shown around the screens. I know that that ups to 3 days… The biggest challenge my group had was coordination of thought. Some of the members had very strong opinions and bumped heads with others, all this is normal, it was a good experience nonetheless. but it was the final stretch and stress got the best of us in some occasions.

As stated before, the set up of the screens and the working system was fast, we where smart to mock up the space back in ITP, it made things much easier. The only problem we actually had was to make sure the camera would stay on the glass and not fall. Double sided tape fixed that in no time. Even though we had to find a special one that was way~ thicker and transparent.

I got to say though, I had a blast all the way through.

the final version of this project gave us great moments.

Aftermath

This project was up for 6 days. And we where taking a photo every couple seconds. If I do my math properly, We would have 144014 images. we do. And I didn’t do the math, it’s in my hard drive.

Of all those images, not all had faces in them. “only” … were found. The thing here is that not all of them had actual faces, there are plenty of weird shadows and other shapes that openCV thought to be faces. That is something that we tried to tweak to make sure it happened as little as possible, but it’s quite hard to reduce that number to zero in the amount of time we had at our disposal. And at the same time, we kind of liked the fact that everything wasn’t perfect, this “tech-being” was learning, so it was suiting that it would not know exactly how a face looked, therefore sometimes it would get it wrong.

I learned a lot for this project, python is not as hard as I thought it was (even though I don’t really know much about it). the main point is that I lost my fear of tackling new languages, I’ve been focusing a lot on javascript, not only because of my level with it, but also for the mentioned fear to change environments.

The second lesson would be how much fun it was to build this, and to have a bunch of strangers play with it and actually enjoy it. A funny anecdote was the faces people made when they realized that the window was being taken down. lots of sad faces. It was a very stressful process, but that made it so engaging, we were passionate about the project and we wanted to make it the best we could. It was a very intense experience.

Running around trying things, coordinating everything, and managing the time we had and making this possible was incredible. I learnt to realize how much I enjoy time constrictions.

This was a very long post, You probably skimmed through it, which is fine. I hope you learned something from it.


links that either, you should check out, or have been mentioned in this post

gP




Please, tell me how good or bad this post was. And comment on anything that comes up in that beautiful brain of yours.

0 Comments