This show was written and produced by Leigh McDonald.
Did you know your phone is a really good listener? Apps on your phone might be sending and receiving data over ultrasound. Ultrasonic communication is used for everything from tracking your daily habits to enabling light shows at music festivals. We hear from Yale Privacy Lab's Sean O’Brien and Michael Kwet, and privacy and technology counsel Katie McInnis. We also discuss the more positive uses of data over sound with LISNR CEO and co-founder Rodney Williams.
MUSIC IN THIS EPISODE
Autumn Eyes by The light The Heat
The Fairest Things by Chad Lawson
Fore by Steven Gutheinz
We Need Each Other by Dexter Britain
Gentle Without by Steven Gutheinz
Chasing Time by David A Molina
Butterflies (Night Hawk Remix) by Tony Anderson
Miles (instrumental) by Sonjo
Finding Glass by Steven Gutheinz
Check out Defacto Sound, the studios that produced Twenty Thousand Hertz, hosted by Dallas Taylor.
Follow the show on Twitter & Facebook. Our website is 20k.org.
Consider supporting the show at donate.20k.org.
To get your 20K referral link and earn rewards, visit 20k.org/refer.
Check out wetransfer.com for all of your file sending needs!
Go to forhims.com/20k for your $5 trial month.
View Transcript ▶︎
[Mall ambience at SFX]
You're listening to Twenty Thousand Hertz, the stories behind the world's most recognizable and interesting sounds. I'm Dallas Taylor.
[Music Start]
Put yourself in a shopping mall. What do you hear? Maybe the sound of clothes hangers sliding across a rack…
[Clothes hangers SFX]
… or a cash register ringing up a purchase…
[Cash register SFX]
...maybe it’s rustling shopping bags?
[Shopping bags SFX]
What you probably don’t hear is this?
[Macy’s signal SFX]
Just on the edge of human hearing, at around 18 thousand to 20 thousand hertz, data is being transmitted over sound. It’s called ultrasonic communication, even though it might be audible to a child or someone with excellent hearing. The sample you just heard has been pitched down so the rest of us can hear it.
[ Continue Macy’s single SFX]
<span data-preserve-html-node="true" style=“color:rgb(127,44,202)"> Sean: Ultrasonic tracking if sci-fi, right? It's the kind of thing that seems like it comes out of a comic book or a movie. And I think that gets under people's skin.
That’s Sean O’Brien, from Yale University’s Privacy Lab..
Sean: We do privacy and security work and we look at advertising trackers inside of mobile apps, such as the ultrasonic trackers.
Ultrasonic tracking is tracking that's done through your microphone. <span data-preserve-html-node="true" style=“color:rgb(127,44,202)"> Sean: If you have an app on your phone that allows microphone permissions, permissions to record onto your device, it can eavesdrop on you in the room and light up that microphone when you don't know.
Providers can embed their ultrasonic tones, or beacons, as they call them, into television shows and advertisements.
Sean: Let's say for example you're playing a game...
[Tiny Wings SFX]
...and you have the television on…
[Jessica Jones SFX]
Sean: There could be a signal coming through that television that's ultrasonic or near ultrasonic in most cases, that you can't hear. That sound can be picked up by your microphone, processed by the app, and then communicate with a server on the internet so that advertisers can gain data about what you're watching and potentially where you are.
If you’re listening to this, you’re probably using your phone at this exact moment. You carry that little device with you everywhere. And it might be spying on you. It can listen to what’s around you, and give that information to advertisers so they can get you to buy stuff.
[Music Start]
Sean: Hopefully at this point people have at least heard of binary and understand that there is zeros and ones inside computers. Which still sounds pretty mystical. The ability to discern between different tones can be correlated to zeros and ones. Or to use a simplified example, which we wouldn't see in the wild, 26 letters of the alphabet. You could have 26 different ultrasonic frequencies that are slightly different, so we call it frequency shift keying, because there's shifts in the frequency, and they could do A through Z with these tones.
You can look at the wave form, so a microphone and devices that are specifically designed to look at sound, can look at these waveforms and get data from them in this way.
[Music out]
This technology is also being used outside the home. You can find it at sporting events, music festivals, and yes, even the mall.
Sean: They take a look at people in retail outlets and they try to do things like, for example if you're walking by a rack of clothing, they might send you an advertisement for some clothing on that rack. It might be a 50% off coupon, it might be some other kind of promotion, that's going to try to motivate you to buy that piece of clothing.
Remember that sound at the top of the show?
[Macy’s Signal SFX]
[Music start]
That was found at a Macy’s department store nearby. The provider responsible for the ultrasonic beacon is ShopKick. They’re exclusively in retail stores. Shopkick has an app that lets you earn points and gift cards for walking into stores like Lord & Taylor, Yankee Candle and American Eagle. When you walk in, your phone picks up this ultrasonic beacon from the store speakers, and let’s the app know that you’re there.
The thing is, Macy’s doesn’t advertise integration with the Shopkick app. Shoppers can’t earn points for visiting. So it’s unclear how they’re using Shopkick’s technology. We reached out to Macy’s and Shopkick for interviews, but they declined.
[Music out]
While earning points and gift cards for simply walking around the mall is enticing, there’s a bigger picture here. And in this case, the bigger picture is big data.
Michael: There is an incredible amount of things that can be learned about an individual based on a small amount of data.
That’s Michael Kwet. He works with Sean at Yale Privacy Lab.
Michael: Companies can infer quite a bit about you. They can infer what your sexuality is, what your politics are, and we're learning that they're able to infer things about potentially your mental health based on the frequency of words you use, how often you swear.
And it’s not just ultrasonic tracking. Yale Privacy Lab found that over 75% of Android apps have some kind of tracker. Apps can use WiFi, Bluetooth and GPS to track your behaviors. And these trackers can work together to collect even more data from you. Here’s Sean again.
Sean: The message we're trying to bring is that this tracking is layer after layer after layer, really interwoven, very difficult to untangle the business relationships between these different trackers.
It's not just that when I go get an Android device or get an Apple device that Google or Apple are looking at me. It's that there's this entire ecosystem of trackers that are doing all kinds of nuanced things to track me, sharing data with each other, building profiles of us that, can usually be used to identify us backwards because it is unique to us.
Sean and Michael are confident that these trackers are in iOS apps, too. But Apple has more restrictions on their devices and software, so it’s harder to research.
Sean: We know that these trackers are also in iOS apps. We want to be very careful, at Yale Privacy Lab we want to always say that this is not a Google versus Apple thing. There are very strict laws in the United States specifically about circumventing DRM. That's digital restrictions management, or digital rights management as they like to call it. Not being able to get around pieces of software that lock down an iPhone because you could go to a federal prison, is a big barrier for us as researchers.
[Music start]
While there are strict laws protecting proprietary information, there isn’t much protection for the consumer.
Katie: So to some extent, this is a little of a wild west, right? Like, this is kind of brand new technology.
That’s Katie McInnis. She’s a privacy and technology attorney. With Katie’s help, the Federal Trade Commission issued warnings to apps using ultrasonic trackers. The FTC is the government agency that protects consumers.
Katie: We wrote comments to the FTC about how users are tracked, and one of these methods was ultrasonic beacons, which we were highly concerned about, because it was really unclear to the user that their activities across devices were being correlated using an ultrasonic audio beacon. And we felt like, unlike other methods of tracking, this one had the least amount of consumer exposure.
[Music out]
The FTC warned apps against SilverPush, which provides ultrasonic tracking in retail stores. And, when they got the warning, SilverPush said they’d end their tracking program. But because of how the FTC works, they couldn’t have prevented SilverPush from the start.
Katie: Unfortunately, in the U.S., we have a very fragmented system of privacy enforcement. The FTC, doesn't really have rule-making authority, unlike most of their agencies. And so they can't create prospective rules, then regulate future actions. They can only look at something, let's say, retroactively that was unfair and deceptive to user.
[Music start]
The researchers at Yale Privacy Lab found eight android apps that still use SilverPush. Most of them are international, though, and outside the scope of US law.
One of the few laws that does protect consumers in the US is the FTC act, which established the Federal Trade Commission. This act protects consumers against “unfair or deceptive acts or practices in or affecting commerce.” Basically, it protects consumers against the shady stuff business sometimes try to pull. This act was signed into law by President Wilson way back in 1914, so it’s pretty crazy it’s being used to regulate technology they never even dreamt of in the early 20th century.
[Music out]
In recent lawsuits against ultrasonic tracking providers, the Wiretap Act has been referenced.
[Music start]
This act not only protects our private conversations over the phone, but It also makes it illegal to spy on any kind of communication through a device. So it’s no surprise that this act has been brought up in lawsuits against ultrasonic communication providers.
We’ll hear from one of those providers after the break.
[Music out]
MIDROLL
[Music in]
Lawsuits against ultrasonic communication companies have been popping up lately. These apps use your phone’s microphone so it’s easy to see that this could be compared to wiretapping. But, not all ultrasonic communication companies are in the advertising or tracking business. There are genuinely useful ways to use this new technology to make people's lives easier - just like Wifi and Bluetooth has.
A company called LISNR describes their technology as “data-over-audio.”And like most of the providers in this field, they use these near-ultrasonic tones to transmit information.
[Music out]
Rodney: It's really a modulation across a frequency range.
That’s LISTNR’s CEO and Co-founder Rodney Williams.
Rodney: And we can push that frequency range up or we can push that frequency range down depending on the environment to ensure that it's gonna be reliable, but our core infrastructure is built between 18,000 and 20,000 kilohertz.
So, that frequency range is important, here’s why….
Rodney: the FCC says that all audio up to 21,000 kilohertz is safe audio - safe as in health, it’s not affecting your ear drum - We have competitors that actually use audio above 21,000 kilohertz. Technically that's not in the bandwidth of safe audio, and that's why really high frequency ranges outside of that bandwidth are regulated.
LISNR got their start as a marketing technology company, and they worked with some pretty big names.
For example, for Discovery Communication, as you watch MythBusters, little quiz overlays about the myth. Did they use water? And it would count down, and then your phone would start counting down, and vibrating, and then you had nine seconds to hurry up and answer it. One of my favorite, Budweiser Made in America music festival, What I loved was at the end of the night, if it recognized that you walked past a gate, it actually sent you a message to get a Uber, and it gave you a coupon offer on an Uber.
I thought it was perfect, right? I mean, it's a bunch of kids obviously at a festival. Obviously they just need a ride home, and I mean, I just think that's the power of understanding when a consumer's inside of an experience, and being able to help it.
In order for this experience to work, you had to download the festival’s app so it could listen for the ultrasonic signals.
[Play clip: Crazy in Love by Beyonce (Live at Made in America)]
Here’s an example of how it might work. But, in this example we’ve lowered the frequency of the signal by four octaves, to a range where you can hear it.
Rodney: Yeah, so it would be in the Budweiser Made in America app. What would technically be happening is that we would actually be playing our tones throughout the venue, and tones basically would have different location data so that if it heard a certain tone that mean you were in a certain area, and then it could understanding how long you were in that area, and if you went from area 45 to area 46, and then to 47, obviously you're walking, and then we just basically could trigger different messages based on where you are in relation to these tones.
[Music out]
Rodney: The magic behind it, which drove a lot of the engagement, is that this wasn't the battery drainage. It didn't use your cellular data, wifi data or GPS data to trigger you the message.
*[Music start]
Despite all their success, LISNR decided to end their marketing program and focus on other uses for the technology.
Rodney: All transparency, it was mainly because of a lawsuit that we got - that's actually just got dismissed, by the way, because our technology is fantastic and it does what we say it does - but it was a lawsuit that basically said that we were recording consumers' conversations for purpose of advertising, I can't say too much because I don't know what else has been released publicly, but I what I can say is our technology just doesn't do that, right? It doesn't interpret sound. It can't hear a voice. It's not voice recognition. It's true data over audio.
[Music out]
One of the concerns with ultrasonic communication today is that you have to let apps use your microphone. Sean from Yale Privacy Lab says it’s hard to know exactly what they are doing with your microphone, and it might be possible to collect more data than intended, like human voices, for example.
Sean: the processing is happening on a server somewhere. The app is not going to spend a lot of processing power or use the capabilities of your phone to make that waveform more privacy-respecting before it sends that audio to whatever server it's talking to.
But LISNR says they took their technology offline once they stopped using it to track.
Rodney: The moment we went offline, locally encoding and locally decoding, Lisnr has the inability to track. It’s a completely offline method of wireless transmission, so it does not connect to a wireless server. It does not connect to a cloud.
This is a complicated problem - “the cloud” wherever that may be, is actually the vulnerable part of the system. Ultrasonic communication is just a tool to collect information, which could be sent to “the cloud”.
If someone is going to try and steal information from you, they will most likely target a cloud server because they hold such massive amounts of information.
Back to Rodney.
Rodney: You can't hack the data transmission from a cloud server because we are no longer connected to a cloud, so the cloud does not initiate a transfer or decode the transfer, it's locally. Then you have to be locally there. You have to know the algorithm, you have to know the encryption, and you have to be able to understand the time token. And if you was to get all of that, then good for you. I think it should be that hard.
When this technology is offline and more secure, it’s better suited for things like authentication and payment purposes.
[Music Start]
Rodney: There's some unique advantages by using this as a authentication method, and that's probably the biggest area of interest and growth for us. Earlier last year we landed Ticketmaster, a consumer's mobile phone would actually broadcast real-time ticketing data, the same ticketing data that would be sitting in a barcode, and instead of walking up, and getting your screen brightness correct, and then getting the right angle, you would literally just have to place your phone within 12 inches of a scanning device, and your phone would immediately authenticate and turn green, and you are allowed in.
We want our data to be with the individuals that it's supposed to be with, not anyone else. In a perfect world, consumers locally have data, and when they want to transmit it, they control the transmission and they control who it's delivered to, it's not tracked by a third party like Amazon, Apple, anywhere, it's literally tracked by you.
When it comes to ultrasonic datate transmission, it’s up to each company to use their technology is ethical ways. For ultrasonic tracking companies like Silverpush or ShopKick, or really any company that tracks and collects data for advertising, transparency is especially important. And like everything else, transparency is on a spectrum, with open source code on one end, and a black box on the other.
[Music out]
Sean: A lot of this is black box, so we're making guesses from the outside, which is sort of the thing that's so scary. Inside the advertising industry, this kind of tracking is no secret to anyone. What the actual business practice are inside a specific business is the kind of thing that's hard for us to say.
We don’t really know what data ultrasonic tracking companies are collecting, or what they’re doing with it. And that means it’s hard to hold them responsible if they go too far.
Michael: What these advertisers want in this situation is of course to get people to buy their products.
But the degree of manipulation is pretty extensive and so I think as time marches on, the kinds of information and practices that we're seeing in the advertising industry are cause for alarm because nobody really wants to be manipulated in this way and a lot of this is being packaged into video games or into chat apps so in order for us just to carry out our day-to-day lives, we're all being subjected to a lot of surveillance that is very concerning for our rights and liberties.
And remember, 75% of android apps have some kind of tracker, whether it’s ultrasonic, Bluetooth, WiFi or GPS. And these apps are built around the trackers so that the apps won’t even function without them. And even if they could, the companies make it really hard to opt out.
Michael: it's extremely hard to opt out. For Tinder, if you want to use the app, you have to turn on the location tracking.
[Music start]
But, if you turn location tracking off, then you can't use your map service. So, the problem is these companies understand that instead of giving you a straightforward option to opt into these kinds of things, they construct their apps and their privacy policies to make it onerous and difficult for users to opt out, and when you have maybe 40 apps in your phone, the opt out process becomes overwhelming for an individual and their tactic in the industry is to overwhelm individuals so that they just throw in the towel and say, "I want to play games. I want to talk to my friends. I'm just going to install it and click-through."
What we need is stronger transparency from the industry and greater awareness from consumers. It’s easy to forget, but our phones are right there with us during the most intimate part of our lives. It’s worth keeping safe and trustworthy. So, if an app requests access to your microphone, but has no reason to do so - you should probably reconsider installing that app.
CREDITS
Twenty Thousand Hertz is produced out of the studios of Defacto Sound, a sound design team dedicated to making television, film, and games sound incredible. Find out more at defacto sound dot com.
This episode was written and produced by Leigh McDonald...and me, Dallas Taylor. With help from Sam Schneble. It was sound designed and mixed by Nick Spradlin.
Thanks to Sean O’Brien and Michael Kwet from Yale Privacy Lab. Also thanks to privacy and technology counsel Katie McInnis and LISNR CEO and co-founder Rodney Williams.
The music in this episode is from of our friends at Musicbed. Having great music should be an asset to your project, not a roadblock. Musicbed is dedicated to making that a reality. That’s why they’ve completely rebuilt their platform of world-class artists and composers with brand-new features and advanced filters to make finding the perfect song easier and faster. Learn more at musicbed.com/new.
Did this episode change the way you think about your phone? Let us know on Twitter at 20k org. You can also give us feedback, submit a show idea, read episode transcripts, or buy a super cool 20k t-shirt through our website at 20k dot org.
Finally the first person to decode the ultrasonic message we embedded at the top of the show will win a t-shirt. Just hit us up through our website, twitter or facebook with the message. Thanks for listening.
[Music out]
