Coding and storytelling with video sensing

Video sensing is a type of augmented reality that allows a software to "see" objects in front of the camera and respond to them on-screen with animations, sounds, or any code one can imagine. We use a video feed instead of using a MakeyMakey or other keyboard input devices. Interacting with a computer in such a playful and creative way seemed like a rich tool for computational tinkering activities. We decided to familiarize ourselves with this aspect of Scratch and have been exploring video sensing with Scratch for about a month now (see here and here).

Real world paper objects merged with computer animations on a computer screen.

I created Day to Night as an example of a starting place for visitors. The grass was handmade and imported as a sprite, and the Deanna sprite was left over from a Scratch project that AiR Tim Hunkin made.


Techniques and Scratch Assets
We learned that Scratch can detect motion as well as color. The background color changes to a 50% opacity blue when the video sensing picks up a specified color (in this case, the "moon" on a stick that is light yellow). The code block below is assigned to a large, transparent rectangle sprite. It also triggers the bat to come out and removes the sun.

Screen Shot 2018-04-09 at 4.24.08 PM
This code is assigned to the transparent sprite.

Screen Shot 2018-04-09 at 4.24.29 PM

The sun sprite appears on whether the transparent sprite detects the moon color.

Screen Shot 2018-04-09 at 4.24.40 PM
The bat sprite's appearance is also dependent on the presence of the moon color.

We believe that video sensing can be added to the library of computational tinkering projects. Video sensing encourages visual story telling along the same lines as some of our collaborators like Becca Rose.
As we think about bringing Scratch video sensing out on the floor for visitors, I'm interested in creating inviting set-ups for visitors that encourage a wide variety of possible outcomes.

Where Do We Go From Here?

Sounds. After sharing this prototype during a team share-out, we discussed what visitors (or Tinkering Studio members) might want to add to the set-up. Sounds were mentioned as a must have, including daytime sounds (chirping birds, feet walking on grass) and nighttime sounds (crickets, owl hoots).

Characters. With more sounds come the opportunity to add more characters, or sprites in the Scratch vernacular. Populate the scene with more animals and environmental details.

Half-baked Ideas

Scenes. A walking sprite could transition between scenes, like Eric Rosenbaum's Big Tape Adventure. Perhaps the Deanna sprite could go on a walking tour of San Francisco or the Exploratorium, or even go on a safari or to outer space.

Collaboration. Inspired by the drawing-style of exquisite corpse, each participant could contribute an object and code what happens when the object is placed on the screen. In a museum setting, we could add to the same project throughout the day and different visitors could contribute to a larger, collaborative story.


Paper Circuit Cards Video

After running the paper circuit cards workshop for donors last December, I've been interested in exploring other types of LEDs. Here is a video compilation of a variety of types of LEDs including a range of sizes, colors, and styles. Enjoy!

Paper Circuit Cards from The Tinkering Studio on Vimeo.


Tinkering with sound - developing tools to explore and share sounds

Sound Safari


Investigating sound as a physical phenomenon

We love to ground our activities in the direct investigation of phenomena. Recently, we have learned a lot about how to engage visitors in deep explorations of sounds made with everyday objects amplified with contact microphones. To read about our initial sound experiments with visitors, check out Luigi's blog post.


To allow participants to study everyday sounds in a playful way, we developed a new tool, a portable contact microphone with an IPad amplifier and headphones. Visitors can use this contact mic like a stethoscope to closely listen to otherwise inaudible sounds. We often describe it as a magnifying glass for sound.
Sound Safari

We observed a lot of curiosity and delight as visitors went on a "Sound Safari" to discover and record unexpected sounds from everyday materials.

sound-sheet blog

Our desire to better understand what visitors are thinking when they deeply investigate sounds from specific materials lead us to this documentation sheet. We uncovered that sounds were often described in imaginative and personally meaningful ways by visitors. We also overheard some visitors talking about stories derived from sounds they recorded.

sharing discoveries and ideas

Since it's conducive to the tinkering process to share and build on the ideas of others, we wanted to make the sound recordings sharable between participants. On a small scale, we found that working in pairs with headphones makes for great collaborations and supports the exchange of ideas. However, it was challenging to share sounds this way with facilitators and the recordings themselves needed a home to be stored and remixed.


Keina Konno a media artist and currently visiting artist at the Tinkering Studio conceived of an app that visually represents recorded sounds in a "sound field". She dove right into creating a prototype and we've since tried out different versions with our visitors. The screen becomes populated with sounds recorded during the workshop and visitors can share, combine and compare sounds on the touch screen monitor. This video shows a collection of sounds that were all recorded from Lianna's bicycle during a 45 minute long sound safari recording session.

How to create with recorded sound

Good tinkering activities invite participants to explore a topic and in the process develop their own path of understanding and their own ideas for projects. During the next phase of activity development, we will focus on adding context that encourages creating personally meaningful projects that incorporate the recorded sounds. As we often do as part of our R&D process, we invite our team and collaborators to create projects based on their own interests. As we all create and share our own sound-based creations over the next couple of weeks and months, we will move toward understanding the core concept of the activity and then decide on how to frame it to visitors. For a start we are thinking of sound collages, visual animations that incorporate sounds using the Scratch visual programming platform, sound sequencers similar to Eric Rosenbaum's MmmTsss that the sounds can be added to, and a collaborative "sound wall" where visitors contribute their sounds and trigger other sounds by touch or dancing.

As we transition to developing activities using recorded sounds as a material for story telling and composing, we will look more closely at computational thinking concepts and practices present in the activities, such as sequencing, modularizing, abstracting and iterating.


Observations from Circuit Boards in XTech

The Beginner XTech program at the museum kicked off a new semester and year with some exploration of Circuit Boards. It’s a foundational Tinkering Studio activity and we like to use it to kick off curriculum involving electricity because of its accessible construction and its openness to studying current within and between mechanical or electronic objects.

Crystal Schematic

Elena’s drawing of her circuit. You can see how she diagrammed the different boards and the alligator clips connecting them.

During our initial exploration of the circuit boards, we invited students to start simply with a set of batteries and lights. From there they expanded to incorporate various electronic parts such as motors, switches, and buzzers. These are made by attaching them to blocks of wood and then soldering the leads to conductive nails. They typically connect the components to each other using alligator clips between the nails, as in Elena’s illustration above.

I wanted to capture how our students were thinking about these concepts in their own words so I walked around the room and asked questions. I was particularly interested in the ways students used non-technical terms to tell a kind of narrative about what they thought happened with an electric current, and to identify specific words used to describe any aspect of the electronics or the behavior of a current. The questions I asked included, “how would you describe what has happened?” and “how do you think electricity behaves?”


“If you connect it at the solder, there’s more power.”

Jezzreal had spent the first several minutes of his exploration sticking with just a battery set and a light bulb. Once he had it light up the first time, he experimented with powering the bulb in a sustained way, then off and on in quick succession, then began investigating what would make the bulb brighter or dimmer. With the alligator clips, he pushed it against parts of the nails connected to the bulbs, and found that the solder was the best place to power it, and in a very tactile way discovered some behaviors of electrical conductivity and resistance.


“Putting a light AND motor doesn’t work if you set it up like a circle. But it did work when I stacked up the wires from the same battery.”

Crystal set up a challenge for herself with the condition of powering up a light bulb and motor from the same battery. At her area on the table were other components she had grabbed, including more battery sets, but she insisted on figuring out how one power source could make two objects function. She arranged the alligator clips in as many ways as she could think of that resembled a circle, with one wire going from a nail to another in one direction and another wire going in the opposite to the second nail. But nothing worked. After some frustration, she sat and thought about other ways to connect them, and in her fluster decided to add in more wires. Whereas before only one nail was connected to another on a different board, this time she started to add two wires from the battery going to the bulb with another pair going to the motor. She was hesitant at first, and it became clear that she thought she was going to get a short-circuit by doubling up on the wires in each direction between battery, bulb, and motor. But, once connected, they both started working! It was a wonderful moment where we figured out the difference between parallel and series, and Crystal’s arrangement of the components on the table really helped this discovery come through.


“If I change the way the wires are connected, the motor will move. You have to reverse the wires to get it to move in the other direction.” -Logan

“You have to switch it back and forth to get it to move.” -Zachary

“You know you can add a switch to do that, right?” -Addison

Logan had selected a motor that came from a salvaged portable CD player, and figured out that it moved a CD tray from an ‘open’ to a ‘close’ position, and vice versa, by changing the polarity of the current. He was thrilled by this and began to show those sitting around him. Seeing an opportunity, I asked Zachary to come over to Logan’s circuit and, without pointing to objects or touching anything, asked Logan to verbally assist Zachary in recreating the motor’s movement. “Try putting an alligator clip to one of the nails,” is how he began, and Zachary glanced at the clips in his hands before choosing which to connect. The motor stayed still. “Okay, now change it.” It moved. Zachary looked surprised, and a little confused. After Logan repeated his instruction, it dawned on Zachary and, in a quick and deliberate maneuver, compelled the motor to move the opposite direction with a change of the alligator clips’ position. Zachary began rapidly swapping the clips to the nails to get the motor to move back and forth in succession, and at this point we had garnered a small crowd. Addison, having spent that whole interaction between Zachary and Logan observing, chimed in with her own innovative thinking. “You know you can add a switch to do that, right?”


“The cords, the battery, the thing itself -- any of those things can prevent the circuit from working… the bulb could be busted, something could be jammed.”

Elena Instructions

Elena’s notes for “First-Time Users.”

One of our prompting questions for the students was to think about how the electrical currents behaved, or moved or didn’t move, and Johnathon noticed that much of the work is thinking about how to troubleshoot something that doesn’t go the way you expect or want it to go. When I asked him what he would recommend for someone else to start with when troubleshooting, he responded that first, the connection between power source and object, then said batteries don’t really indicate when they’re dead or not, then spoke of issues inherent to the component. Shortly after he shared his thoughts, his sister and investigation partner, Elena, started to write a set of instructions for “first-time users” that came complete with these thoughts and with diagrams. Both siblings recently joined us as facilitators in another program, so it was especially exciting to see them take such a pedagogical approach as part of their own experiences with learning.

Once we’d spent a substantial amount of time exploring circuits, we challenged students to construct just one using any number and variety of parts, then to diagram it or draw it. After that we cleaned up the boards and wires, restoring them into separated groupings, then we collected the drawings and handed each student a drawing someone else made. We asked them to recreate the circuits from the diagram, making needed edits to either the setup or the drawing until they were functioning.

Zach Schematic

Zachary’s drawing became a collaboration with some notes and color-coding.

We found that by having students draw their own working circuits and swapping those drawings with others to build new circuits from diagrams, they were able to see the variety of ways electrical components could be represented. Positive and negative nodes were often drawn similarly, so once a circuit board was fully connected in accordance with the drawing, it didn’t always work. Some drawings featured “+” and “-” symbols, evading these moments of confusion, but some of them did not and it was necessary to note the direction of currents. Altogether, the exercise proved very powerful as a way of reconciling differences in vocabulary used to describe the same thing, and noticing that diagramming was in itself a meticulous art seeking to capture complex science.

Linked here is an activity guide we wrote specifically with afterschool and out-of-school time programs in mind. There are additional resources (including how to make a set for yourself) available here on the Tinkering Studio website.


A Visit from Artist-In-Residence Noga Elhassid

Earlier this month we were lucky to have a brief but inspirational visit from our friend and fellow tinkerer Noga Elhassid. Similar to past residencies with her, we used the three days we had to explore making simple mechanisms out of everyday materials.

On Thursday night for After Dark we repurposed small cardboard boxes to make shaky sculptures. One of my favorite things about this activity is how that by adding paper elements onto the box flaps, it exaggerates the natural motion of wiggling the box. The results were often surprising, silly, and satisfying. I was really impressed by how long folks stayed to build their sculptures; many people stayed for almost an hour iterating and complexifying their designs.




Over the weekend we changed gears and hosted a Whirligig Factory workshop. In this activity visitors explored making a wind-powered crank mechanism that activates a paper sculpture or creature. There were four steps to building the whirligig: making a propellor, making a stand for the propellor, making a crank, and finally building a creature.

IMG_5914 IMG_5915 IMG_5916 IMG_5917

Building the creatures was one of the most popular parts of this activity. People came up with really elaborate designs for how they wanted their whirligigs to move.

I love how this video shows that the size of your sculpture and propellor design both make a difference in how fast your crank spins.


Since we're big fans of automata of all types, we're going to continue to explore the potential of wind-powered contraptions!