Smartphone snoop: Even when phone sleeps, digital assistant always eavesdrops

Think Siri, only smarter. When it comes to smartphones, could you trust a digital assistant that was constantly lurking and listening so it could chime in at any time...even when the mobile phone was supposedly in 'sleep' mode? Or would such smartphone snooping be too much of a privacy invasion?

On your smartphone, do you use speech-to-text or other voice commands for convenience? While it is handy, so long as it actually correctly interprets the words, you turn it on by choice. If the mobile phone microphone, which happens to be about 10 times more powerful than it was even two years ago, activated by itself to covertly turn your voice to text, then sort of invasive spying would be downright creepy. What if your phone was just lurking and listening, even when you thought it was in "sleep" mode, waiting to leap into action by a digital assistant chiming in? No way you'd go for that app? What if such persistent listening capabilities were built into mobile phone chipsets?

Nuance Communications, Inc, best known for its Dragon voice recognition software, told Technology Review that it is "working with 'a number of' chip companies who are 'thinking very actively' about how to make this sort of persistent listening work in a low-power way." Just as Apple has the digital assistant Siri, Nuance has a virtual assistance via the company's Dragon Go! app.

In the world of biometric identification, you can be identified via face, iris, fingerprint or voice recognition. Nuance is currently on the path of persistent listening via its Dragon ID concept that uses your voiceprint, "a voice fingerprint," to authenticate yourself instead of using a password. It's supposed to "bring even more intelligence to a smartphone, tablet or TV" so you can "Forget Cryptic Passwords, Use Your Voice Instead!" The system can "wake up" by using a passphrase such as "Hello Dragon," according to the hype, "to quickly and securely turn on your smartphone or TV—or instantly launch your personalized home screen with favorite apps and content on a shared tablet or PC. And that's just the beginning of what you can do with Dragon ID!"

What Nuance is hoping to do with chip makers would go beyond what you saw in the video. While discussing if a person could trust their voice to secure their digital life, Jason Stirling from Nuance told CNET:

There's three pieces. Firstly, there is speech recognition, or capturing what was said. The second part to the problem is around semantic understanding, or natural language processing. So if I've captured what was said correctly, I'm also trying to capture the intent or what was meant. Let's say you asked, am I going to be late to my meeting? We then have to check a time reference and we have to know where the meeting is, so we have to hit a diary. And that moves into the third element of the problem...a disambiguation process. If [the system] is unclear about some part of the request, we might come back in a conversational way, the way you or I might speak, and try to clarify what information you were trying to get to.

Nuance Mobile can already understand 34 languages well enough to translate them into speech-to-text, as NDEV Mobile recently announced. There are about 12,000 developers and growing in its developer program since "the Dragon Mobile SDK is available for iOS 4.0 and 5.0 (iPhone/iPad/iPod touch), Android 2.1 and higher, Windows Phone 7.5, and via an HTTP web services interface." Regarding Nuance making a smartphone that is listening even when it's asleep, some of this will be up to the developers. Since we've seen over and again how some developers create apps with overreaching permissions, what if your eavesdropping assistant verbally inserts "ads" while you are cooking or making a grocery list?

Technology Review pointed out that "software developers will have to be careful to avoid making services that are annoying or creepy, or else they'll create more 'paper clip stories'—a reference to Clippy, the much maligned (and now extinct) digital helper in Microsoft's Office software."

Secretly snooping on private or business conversations sounds terrible from a privacy perspective. Could it be hacked by a competing enterprise or even an ex-signifcant other? What exact voice data is it storing? There might be one possible plus side—and only if it helped you "win" an argument—to have the always-wiretap-ready phone repeat back a time-stamped "promise," such as a kid's vow to clean his or her room.

In an oddly funny but potentially possible privacy pitfall, TelecomTV wrote, "The idea seems to be that simply by saying something or making some kind of a noise (such as a terrified whimper whilst having a nightmare or an appreciative grunt during the rigors of the Congress of the Aardvark, as prescribed in the Kama Sutra) that can be attributed by the software to something above and beyond background ambient noise, a handset will spark into instant wakefulness and ask what the squeaker would like to do next."

In a proverbial killing of two privacy birds with one stone, what if it were to flip to the "darkside" and decide to collect even a few seconds of people's voiceprint to build a VoiceGrid? Such as SpeechPro Russian biometric software which is helping law enforcement to store and match voices to identities. VoiceGrid has figured out how to distinguish between two people in a conversation, how to filter around background noises, and how to comprehend changes to a voice when a person has a cold or words are slurred by drinking. It only needs a few seconds to collect voice data which can come from audio or video, and your voice may already have been passively collected via "voicemail, recordings made while speaking to commercial service providers such as banks, cell phone companies, and cable TV companies, as well as 911 calls, suspect interviews and court recordings." When it comes to matching a voice with a real person, the company has a "90% voice match to identification accuracy within 15 seconds."

What Nuance is working on is supposed to be an always alert, always listening, digital assisant for your mobile device. If it could get past the problem of guzzling battery power, it could offer some hands-free convenient help. However the idea of a smartphone being too smart by always eavesdropping, by being a mobile snoop even while supposedly sleeping, will raise a lot of people's privacy-hackles. And think of how much feds would love this added "bonus" to cell phones—GPS could track activists and this new spy tech could listen in too. If a person couldn't turn it off, like you can with GPS or location services, and couldn't part with the smartphone by powering it off, then "airplane" mode will be much more popular.

Like this? Here's more posts:

Follow me on Twitter @PrivacyFanatic

Copyright © 2012 IDG Communications, Inc.

What is security's role in digital transformation?