re:publica 2016 – Kate Stone: A New Feel for Print – Paper Music Instruments


Kate is an artist
in how sound and visionary can interact. She has created new ways
in mashing up the visual and the digital. Finding new ways to free
the wonders of technology from their devices to surfaces
we want to interact with. Kate is bringing back
some magic to our world. Please give a warm welcome
for Kate Stone “Paper Music Instruments”. (Applause) Hi, thank you for coming along
to listen to me speak. My name is Kate Stone. I want to talk about my journey
and why I do what I do. Also a vision and feeling
I developed for the future. I believe the future will look
more like the past than the present. It will be more like a magical world instead of the disturbing
minority report. That is because we are always nostalgic
about the past and fearful of the future. And the people
who will be living in the future will be more nostalgic about the past. When I talk I never have any slides. I have some things prepared
and a few videos. So bear with me. I need to set my timer, because I never know
how long I am going to speak for. So what I do is … I seem to have found immersed
in a world that is something binary. Is something this way or that way? Physical or digital? Are we thinking about
something futuristic, or something from the past? I started out working with electronics,
my background is that I am a scientist. But some say I am an artist. I’m not really sure
whether I am or I am not. It’s a question of art or science. I work with printing and electronics, there are often
two totally different worlds. The place where I end up
is somewhere in the middle. I show you something
I created some time ago. Otherwise nobody has a clue
what I’m talking about. We printed a poster,
a poster of a drum kid. That is a small one,
we can build large ones. And when you touch it: (Drum noises) It is kind of fun. (Drum noises) I can – thanks. (Applause) Some can have it and play with it. Can I pass this to you? So what we do: we print to create a touch device and have a small circuit board
to add capacity to the paper. We can print it in all sizes. We did something recently
that was 50 feet in the street in Austin. I say more about my background,
I don’t know how I ended up doing this. I remember as a child
I used to play around with electronics, and pulled toys of my siblings apart,
what they have never forgiven me for. I was not really fascinated
by the technology, but on how I use technology in a way
people didn’t know what I was doing. I hid speakers, microphones
and switches in my bedroom. I’d hide in my cupboard
and made some noises through a microphone
coming out somewhere else. I wasn’t playing with tech
for the sake of tech. I wanted to create experience,
but I didn’t know it at that time. It was just what I wanted to do. I remember, I got one of my dad’s books
and carved out the inside, which he wasn’t happy about. And I ordered one of the electronic kits
from the back of a magazine, where you could build
a radio transmitter with a microphone. I hid it in the book. Then I hid the book next to my parents
and snoop back to my bedroom and tunes in and listened. It was not good what I heard. I kind of regret that. But looking back from what I do now, I realized what I now do
is what I used to do as a child. So I ended up becoming a scientist
through different stages, disappearing to the desert,
chasing sheep, becoming a farmer. And then I ended up
having a PhD in Physics, where I looked at how electrons move. My fascination is how things are made,
how to make electronic devices. Something I have been asking myself: what is the difference
between a scientist and an artist? The conclusion is that science
is an understanding of the inside. Art seems to be
a description of the outside. If you add these things together, you can end up to something
more like nature. Science plus art is nature. That is what I want to create. I want to create more natural experiences
with a digital world by combining art and science. After I did my PhD in Physics,
look at electrons move around I was working for a company
who wanted to print transistors. How to change the way
we create electronics. I learned a lot, and how difficult it is
to try and do things totally differently. How difficult it is
to reinvent new materials, reinvent methods and new machines
to do those things with. And after working there
for 4 years, I left. I started on my own in my garage,
that was 11 years ago. I had been hiding away
in my garage and slowly building a team. There are 7 of us now,
we’re based in Cambridge. We’ve been quite disconnected
from the rest of the world in what we do. It is different what we do,
we are doing things quite differently. For quite a long time it was scary, because no one understood
why I was doing what I was doing. But now people seem to like it
and companies start to adopt it, which is for us exciting. The thing I became fascinated by
as a scientist was printing. And seeing how pervasive printing is. Printing is everywhere,
it’s all around us. There is more print
than electronical devices. Print is the highest form
of manufacturing. Printing has revolutionized
and democratized the world. When books went
from being written one by one. Few people that could read and translate
to something that anyone could have. An information that could be anywhere. I became fascinated with printing
and how to use this with electronics. In the beginning
I was printing my own transistors, I was very proud in the garage
that I had the greatest transistors. But with hindsight,
they really weren’t good. It was like going back to Intel 1969. I quickly realized,
that the right things to do was to combine printing with
everyday low-cost electronics. What you might find in a Happy Meal toy
or a greetings card. And how to combine this with a piece of print
to make something interactive. Was this idea… That poster is so noisy! This idea that we always think that what we need
is something we don’t have. And realizing that so often,
everything we have is our fingertips. We can repurpose, reuse and reimagine
the things around us to start to create
new things and experiences. There were two things I told myself: I’m going to have a vision at no mission. That set me free to be an explorer,
because I did not make any promises. I didn’t say:
“This is what I’m going to do”. The second thing was, I was going to create things
that my science friends laugh at. Not in a good way. Because I wanted to set myself free
from trying to impress people. I wanted to make them smile and see them
immersed in some kind of experience. Those two things
set me free from the beginning. It was quite a hard journey. It is always difficult to find funding. And it is difficult to do things when people say it won’t work
or nobody wants it. This is just a believe, in my innovation
is all to repurpose what’s around us. A little thought I had,
I was looking on an oak tree. That massive oak tree grew from a small acorn,
with some sense of evitability. It made me think how I have been. Always thinking how can I use
the printing machines as they are? How can I use the inks as they are? How can I use the electronics
as they are? And try to be creative.
That is what we have done. Ya, so. And I keep saying me, but it is a team. My team is quite diverse. I as a scientist realized, that it was at the beginning
of some kind of experiment. It was something I felt ashamed of, so the first person I hired
was a graphic designer. I needed a totally different perspective
on what I created. I needed the creative perspective. The first thing we created
was a greetings card. I asked her to do the design
and line work of the greeting card. Then I took it,
cut out parts in Photoshop to allow a circuit flow
through that line art. So you could pick up this card
with some flowers. And just by the way you held it,
you saw lighting candles. The current was floating
through a cupcake. When you blew on the cupcake, it could sense your breath
and it blew out the candles. You had to do that twice
and then it sang Happy Birthday. It is something we made a long time ago, but it told me how art and science
need to be combined to create art. To create creativity and something
that was a great experience. I say a little bit more
about how it works what we create. We are using screen-printing,
that’s the same way T-shirts are printed. Or we use flexorr-printing,
that is how labels are printed, like wine bottle labels. We can print a hundred meters a minute. We can print a full size X-Way
multi-touch trackpads on a press that is used
to print wine bottle labels. At a hundred meters a minute. Then all we needed to do
was add our electronic circuit board. Because we’ve been doing it
for a long time, we started using circuit chips
that weren’t available. We were using micro-controllers. When we had to write the code
to the assembler. There is no paperwork to go with it. It is really quite challenging. Doing things with those chips,
they said we couldn’t actually do. It’s about
bringing those two things together. I will show a little video which shows something what we made
for an advertising agency. I think it was last year. A lot we have been doing was working with ad agencies
to create experiences for people. The video is called “Sound of Taste”. Hopefully you will put up
that video for me. Do you have the “Sound of Taste” video? It’s not on my computer,
I gave you a memory stick. No! I need the videos. I don’t mind waiting for a minute. It is on my memory stick,
but please find the videos. I have four videos. The sound of taste. Somebody pass the message back,
I need my video. (Peaceful piano music) My wish was to visually articulate
flavor through color, pattern and forms. This is my attempt at trying
to capture that taste explosion. What I love about this project is: You’re not only seeing this poster,
but hearing it. And touch it. It is a complete multisensory experience. (Woman) What makes this ink special… (Stone) I think
this is some audio missing. I can tell you: This is showing how we are printing. This lady is Maria, she is the designer.
The first person to join my team. What we created
is the mix of digital and physical. So we printed and created
this beautiful artwork of color that represents a sound. When you touch the poster,
it connects to a smart phone. It plays beautiful piano chords. It combines the physical
and the digital world together. It’s a great experience. (Piano chords) Everything seems to be
on an iPad or iPhone. In a way we kind of hacked paper. We’re trying to find a way
to get that activity back to print, so it works hand in hand
with the digital world. I think, the missing audio is my fault. This is something,
what we have been doing. We have been working
with these advertising agencies. It was great. Once they’ve seen what we can do
they get excited about that. And started putting it
into their campaigns. I would like to have
to be working on it right now. The great thing about ad agencies is,
that they love to make beautiful videos. They like to win awards.
They like to show off. Just before I came on stage
somebody e-mailed me from McDonald’s. It is called McTracks. They’ve been DJing
from the paper tray in McDonald’s. It is funny because we started
using chips from McDonald’s toys, now our print is put onto that product. In trying to work
with this kind of advertising agencies and with other people, it was difficult
to make this platform open. The chips are not generally available. It’s not done in the way
that we can open source or open quote. There is no way to reprogram those chips. What we spent a lot of time
doing the last year to create a very easy way
that anyone wants to make anything with our technology. We’re doing this by printing stickers
in a range of sizes that have a fixed template. Creating a control module, this small circuit board
that has pads on the back. All you do is take a print
and press the circuit board on. That creates an interactive print. We also created software tools. I can’t code so I did not expect
anyone else to code with what we do. In our software all you do
is to drag in the sounds you want to use. You choose from a dropdown list
what you want each touchpoint to do. Whether you want it to play,
be a volume control. In our Bluetooth version, it can trigger
video or audio on a smartphone. Then you save that and put it
onto the circuit board via SD card or it’s pushed over Bluetooth
to this module. In that way someone can create
an interactive experience with a piece of print
in just a few minutes. Then, they can show to someone. Everyone who sees this seems to like it,
this is really exciting. This is the difference
in how we are trying to make things, is: once you created one thing with this,
the description of how it works, can be sent to a factory make a million. We’re trying to find
a very easy way not really for makers. I really want to empower creators, people who have an idea
what they want to make. They can instantly create something
that fits with their vision. Once they have done that, if they want
to make a million, a thousand or 10, there is no other development to do,
it can be instantly done. Our challenge has been
how to create the platform where we can let other people
start to do things. I’ll say a bit
about the Bluetooth version. We made a website that you can go to. On that website
you type in a new ad name. You choose what the touchpoints do,
upload your assets and click publish. Instantly it becomes available
to the player version of the app. Then you can touch this piece of print
and unfold the experience. When I’ve been thinking about digitals,
what it means, so much has been killed off by digitals. We think about books of physical music,
physical advertising. To me digital is three things: Touch. The ability to touch something
that has no moving parts. Connectivity. You can update it,
can give it new information. Data. The ability to capture data. The ability to know
who is touching that why. It’s a little bit scary. But so much which is digital,
the value comes from data. That is what we try to create. I call that: “Description a digital soul” The idea of our software is: You can define a digital soul
and put it into a physical item. I think is has potential
to change so much. What we say to the iPad generation:
print is just a broken touchscreen. You will have heard of two year olds
that tried to touch a magazine and swipe. Or try to pinch to zoom in
on the magazine page. They don’t know why it doesn’t work. And on the face of it
what we’re trying to create is to allow everyday objects
to become digital objects. We get asked
how people know where to touch, why would people want this. I think that is
what the next generation not only wants, but what they expect. Without them knowing that
this is what they want. We want to try to create a platform
that is easy to manufacture, so we can fulfill
desires of the next generation. In a way that it doesn’t look
like something from Minority Report. But more like something
out of Mary Poppins, something out of Harry Potter. So these two year olds can have
the experiences that they expect. We made a broschure recently
for an advertising agency for Audi and that shows how we can connect
the broschure to a smart phone. So hopefully we will play another video. It is called “Audi TT Broschure Hack” Find my video?
Audi TT Broschure Hack. (Video) In October 2014
Audio launched the new TT. The first with a revolutionary cockpit. Our task:
buy us customers with this feature. The catch:
do it with the sales brochure. The Audi TT Broshure Hack. An augmented journey through the cockpit. Simply place your phone
on the brochure to begin. (Broshure) Der Fuel Tester ermöglicht
Kontrolle auch bei schneller Fahrt. Touching the hot spots on the page
let’s you navigate the display. Using conductive print,
we turned the page into a smart surface. An embedded Bluetooth chip allows a seamless experience
between the broschure and the phone. (Stone) This is missing audio again,
I think. You can touch the brochure
and it connects to a smartphone. You can play around the brochure
by touching it. That smart phone becomes the dashboard. You can also
ask for a test drive as well. This was for the advertising agency. What I see is, so many products
and things are becoming commoditized. I think it is of so much value
instead of objects it is an experience. What we create
is more of an experienced than a product. I will show you something else
that we made. This was for a young singer songwriter. We made it over a few days,
we made 50 of them. She’s called “B. Sharla”. She had a debut single
launch a week before last. She wanted to make an invite
she could send out to the music industry. That would let them
experience music in a different way. This cassette here
is the invite to her gig. Which was really awesome. All you do is fold the box up. I should say how this came about. Me and Sharla sat in a coffee shop
and we were talking about how we could make an interactive invite. There was a box of chocolates
on the table. And I thought: Why don’t we turn a cardboard box
into a boom box? Can you hear it all? (Music) You can remix it from the box,
it is a full track audio. (Music) A bit difficult to see from here. So in designing that
it didn’t take long at all. All it is a sticker on the box,
a graphic stuck on top and using the software tool
you put a full track audio in this. So you can transform the audio
in this way. She sent that out to people
in the music industry. She had a successful gig,
which was awesome. The following week
she got her first play on BBC Radio one. It is also exciting
because she is actually my niece. So you found me out. I am trying to demo some things. I’m always bringing too much tech,
which makes it confusing for me. Oh ya. (Piano)
Oops. I have to keep
looking on my notebooks. But does anybody knows
why notebooks are named notebooks? Because they play notes. I can use the cover of my notebook (Piano notes) to play some tunes. I am not a pianist, I can’t play. (Piano notes) I would pass that around,
but actually I need my notes. (Siren) We made this album cover for DJ QBert. He is one of the best
scratch DJ’s in the world. He saw a talk I did, where I showed cardboard DJ decks
and tried to DJ from them. (Horns) And this is the world’s thinnest DJ deck. You can scratch a mix
and cross-fade from them. (Scratching) Ok, I am not a DJ! (Applause) (Siren) Oh. And I like to hack my hat as well. (Siren) I am doing this sound from my hat,
which is kind of fun. (Siren) I just make all this crazy shit
all the time and I have been doing this
since I am a kid. I keep saying that
I want to do this, people believe me and give me money
and me and my team make it. And every time I say it,
I expect to get laughed at. But people actually want it. I have to explain a bit of the backstory
of how I ended up doing this. There was something
I was saying last year, in the beginning of summer. Because I wanted to create
this natural experiences, in a way that moves the minds away from
how we use computers and smartphones, to something which is more different,
more like nature. The thing I had in my mind,
is to walk through a forest and what I naturally do
is a way to produce music. And I am not a music producer or a DJ. If you could touch a tree or leaves,
you start to trigger sound effects. I said this to people
in San Francisco and in LA. It seemed to fit with their dreams too,
but then they tend to smoke things, so these things
were more natural to them. They gave me money
and we built things at a big festival. We made three mandalas
on the floor and when you walked over, they triggered sound effects. We also made flowers out of paper
and built trees with cardboard tubes. You could touch the flowers
and get some sound effects. I also talked to people at NASA
and they liked the idea, too. They pointed me in the direction
of NASA space sounds. So we created a NASA
space sound soundscape. This could be triggered
by using these things. There is a video which hopefully works. The video is called
“Nocturnal Wonderland”. Please play that video. Yeah. Oh no. You can’t hear me, the video, yeah. “Nocturnal wonderland” You could all hear me saying that, right? (Loud music) (Mystic sounds) That was kind of fun. That was one of my designers,
she pulled a lot of this together. We only had a couple of weeks,
so we put it together quickly. In the beginning it shows us
in a hardware store, and we build it in two days. It was great and we had a lot of fun
playing around with that. Then I was talking
with some people in New York, they wanted to create
a musical experience. They want to take this idea
and make it bigger. So we built five installations
in March by Southwest. We built a 50 feet wall of music
in the street. You can touch this wall
and make sound effects. This is what I had in mind
while walking down the street. I wanted to touch the wall
and create music. I do, but nothing happens. And we made three other walls
that were 8 feet by 20 feet, that you can touch and where
Bluetooth midi interfaces make music. It enabled people to mix music,
who never made music before. I have a video of that. The video is called “South by Southwest”,
hopefully it will play, yes. (Electronic sounds) A way to keep my attention. (Sounds with voices) I hope you guys like this song. (Unintelligible) You can send Bluetooth
midi-messages to the board. You can control the light
and it can react to the music. That’s cool! I’m figuring out
what sounds I was making. (Music) Never mind. Maybe I try
to put the video online or something. What that posters were doing,
what you couldn’t hear. We had stamps from major artists. We had some from X Ambassadors
and a bunch of people and remixed it all. I tried to create a baby version of it. Those people that were interviewed
never touched anything with music before. They were doing remixes. (Poster)Hold on to me.I made this last week. It was a way to show
that instead of using this: I hope everyone can see? It scares me. It took me a year to use that. It is Ableton,
which is awesome software, but the fact that it is on a computer
scared me from touching it. Last week we made this. (Sounds) The audio is quantized,
each flower is a different instrument. You can touch it and change the loops,
each petal is a loop. (Electronic music) These flowers are vocal one shots. (Electronic music with vocals) (Poster)Hold on to meAnd the Flamingo
is the acapella for the whole song, I don’t know why
but it seemed like a good idea. (Electronic music) (Poster)Hold on to me.I’m a little understanding.
A little understanding.
Hold on. Hold on. Hold on to me.I’m a little understanding.
A little understanding.
Hold on. Hold on to me.(Electronical music) It is kind of fun, thanks. (Applause) It is so difficult for me to describe all the things we make
and how we make it. It is hard to explain
the journey quite some times. I tried to show some videos
of what we do. This is a poster with flowers on
but it sums up what we are trying to do. This is printed at our screen printer. The design can be designed anywhere. It is using a circuit board
which is generic and comes from Asia. We put it together,
like a stamp on the envelope. It is easy to make. We could go from now we make one,
we could make many. This is what a computer interface
could look like in the future. Something that a baby would understand. Because it is a flower and it talks to it in a way that it seems to mean
something more. Those people in the video
were doing something with music, something they never knew they do. We are trying to find a way in which products
can be designed anywhere. It can be made locally,
anywhere and try to democratize that. To use things that already exist. We are trying to create these things that could look much more
like the past than the present. This is the message I want to get across. With using the things that are around us
I think we can create a future that is going to look
more magical than scary. Maybe I’ll stop now
and take some questions, thank you. (Applause) Thank you so much, Kate. Please raise your hands now,
we will bring a microphone to you. Everybody is still impressed
by the magic. Over there. Thank you. Just thinking because we have
so many refugees coming over in Europe. We could design translating books. Everybody could be translating
more easily than otherwise, because everybody could listen to
what the sound could be like. It is a very practical application
for your ideas. Even though I appreciate very much
the beautiful aesthetic that you create. Yeah that is something that is magical. When we combine graphics and sound,
there is so much more that you can do. If you have just graphics,
it is limited to that space. Then you are often limited
to just that language. When design is done
where language is a challenge, because there’s so many languages. When you think about
something like the Olympics, so much thought
goes into designing the graphics. The graphics that tell you what to do. It creates
more beautiful and simple graphics. But if you could touch
these graphics and have sound on them, hearing something in any language, the greatest selection of languages
would be awesome. What we’re trying to create… Instead of a product being designed
by several thousand people in California, made by a million people in China,
products are designed by those refugees, not designed by me. I want to create a platform, where someone locally
for example in South Africa, designs something
based on the local needs. Nobody would have any clue that it would be
incredibly useful in Iceland. That is the amazing thing
about diversity. When we have more different people
designing and creating things, there’s so much more benefit
for everybody. (Applause) I have a technological question,
you are using conductive ink on paper. The difference between the paper and the circuit board is
that a circuit board is not flexible, but the paper is flexible. Do you use flexibility as an instrument? Yes, the circus body is rigid. At the moment
we use a stiff circuit board and we could use a flexible one. But we try to keep the cost low. There is two ways I could go with that. One is, that I really want to create
to print that circuit board. I want to print using flexorr
and fine lines, conductive tracks print resistors, print circuit boards. Instead of putting
a packaged chip on it and bend it, I’d love to create things
that look like posting stamps. Those are flexible,
so it becomes part of the piece of paper. I tried to do that years ago. It would cost so much money
and need so much development, so we use rigid circuit boards
and ink on paper. We can also print the ink on plastic. I can say a bit more about that. The biggest challenge we had was how to stick
circuit board and print together. It was difficult. We tried with conductive inks,
conductive glues. Putting holes in the circuit board
and poking it through. Using special tapes as well,
but none of them worked. Then I discovered,
that we can use double-sided sticky tape. Which was like:
“Why did I not think of that?” Our electronics can stick
on the reverse of the print. We have a very easy way
of sticking the two together. All things are binary, our challenges are to bring things
together in a seamless way. That is what we managed to do. Are there more questions? (Siren) There is one over there. (Man) Regarding the technological site. You said you choose Ableton
as a program for your midi controllers: It’s creative midi-controllers,
if I am right? Why Ableton? Ableton is not an intuitive interface. It is intuitive now. (Applause) (Man) Is it the most modular software
to adapt to? I understand what you’re saying.
It can be counter-intuitive to use it. Because you are used to this… We are used things to be linear. Software for music.
You produce put music linear. Ableton has two modes. The mode that is confusing
is the live mode. It has columns instead
and you put samples in that columns. Which is what this is. I studied that for a year
and it just confused me. Once I played with it, I got it quickly. I can tell how to use it in 15 minutes. It is counter-intuitive
but it is really easy to use. I find it brilliant for this. You can put different sound effects
and it works so well. What we did when we created
the Album cover and we used the DJ software “Algoriddim”. We were trying to link
straight into that software. There was a rumor of Apple
bringing out midi over Bluetooth. I joined the working group,
infiltrated it. We were able to adapt
the Apple midi-control to our print. It is not limited to Ableton or anything. It is just a midi-controller. You can use it with anything
with “Logic”, “Garage Band”. You can play on my notebook
with any music software. We have time for one more. This part of the room is very quiet. (Music) Maybe someone
wants to play around with it? I am sorry for some sound issues we had. (Man) Sorry?
– You were hidden in the back. We will take your question. (;an) Thank you for the last question. I’m a user interface designer. I don’t make music, I make interfaces. I use Photoshop and stuff like that. I thought it could be a good way to give people
more acceptable fields of color. Because if you have these huge gradients, it is very hard to accept
the point of color and show the people, this is the point of color and then
you have a digital other parts of it. Would it be possible
to make such a detailed platform where you can point a color
of the gradient and you can say: This color I want for my interface
or for my picture. That is a cool idea. With most of these things, they are screen-printed
and we create large touchpoints. Now we are describing
the flexor-printing. That has a millimeter resolution
and it’s 85 or so, where you create these touchpads. You could have these touchpads
paper-thin. You can use a slim battery. You could put a graphic on top
and that could be a perfect way of touching a printed graphic and have it send out a code
for that color. You can map it
to a color palette you want. You can touch the colors
and get it exactly. It’s not another part
of your computer screen. If you are using Photoshop
or whatever you are using, you have all these boxes,
cluttering the area you need for the music,
the video production, the image. We can create a little box of postcards
that you get out, lay around your computer
and touch these things and select it. For you it’s picking the designs, it makes your interfaces
be all around you. Or like for me playing piano
with my notebook. Or like a DJ who gets sound effects
on cards all around him. That would be really cool. (Siren) Or like my hat. Thanks. Thank you Kate for being with us. For everybody who likes to try this: You are giving a workshop
on Wednesday early afternoon. A big round of applause.
Thank you so much, Kate! (Applause) We got a 15 minutes break
and be back then with Richard Sennet.