Augmented Reality Research: 'Offline to Online Recognition'

"Offline to online recognition" starts with one photo taken in real life. Researchers then used "augmented reality" and data-mined social networks to link cyber faces to IRL people, including names, addresses, phone numbers, credit scores, and social security numbers. Will it alter how we think about privacy?

Current Job Listings

At the Black Hat security conference, Carnegie Mellon University researchers Alessandro Acquisti, Ralph Gross and Fred Stutzman presented "Faces of Facebook: Privacy in the Age of Augmented Reality" [PDF]. During their research, they took photos of students with a web cam and then used facial recognition software to compare the photos with Facebook to get students' names. After more online data mining that they called "augmented reality," like on creepy site Spokeo once they had a person's name, they were able find out addresses and phone numbers, interests, sexual orientation, credit scores, and in some cases social security numbers. This "offline to online recognition" all started by them having a single photo of a person. They called this "the emergence of personally predictable information from a person's face."

The researchers said "point-and-shoot" like done by cell phones on the street to capture images of "cooperative subjects" will become much less intrusive. For example, in preparation for the 2014 World Cup, Brazilian police have been testing "RoboCop" facial recognition glasses that can scan up to 400 faces per second and then match the faces against a data of criminals and terrorists. The Carnegie Mellon researchers' mentioned these facial recognition glasses and then asked, "How long before it can be done on.... contact lenses?"

Just look at the leaps and bounds of biometric collection that happened this summer. Police in Massachusetts have been busy scanning irises, taking fingerprints and capturing photos with a 12-ounce handheld device which is attached to a smartphone and then checking identities against national and local databases. By July, they had collected more than 70,000 photos for facial recognition that they could tap remotely. The ACLU of Massachusetts told The Boston Globe, "There's simply no good reason, in our view, to create detailed dossiers on every American resident. And increasingly it appears as if all of these surveillance programs and technological programs are doing just that."

The same facial biometrics collected by the Massachusetts DMV for a fraud prevention system ended up flagging an innocent man and revoking his driver's license. AxXiom for Liberty asked, "Remember the quaint notion of 'presumption of innocence'?" After the DMV fail, the man that was "fingered as a bad guy by the less than flawless technology" is now "suing for his trouble." The Boston Globe called it, "Caught in a dragnet."

In regard to "augmented reality" carrying "deep-reaching behavioral implications" in the future, the Carnegie Mellon University researchers asked, "Will we rely on our instincts, or on our devices, when mobile devices make their own predictions about hidden traits of a person we are looking at?"

In the UK, police are doing "on the spot" fingerprinting on people who are stopped for "moving traffic offenses." The device is the size of a mobile phone. Police run the biometric data against a national database of 8.5 million prints to "correctly identify suspects who lie about their details."

Then there are the license plate readers that police use to scan 30 plates a second in a search for people who are wanted or vehicles that are not insured or are not registered. Those Automatic License Plate Readers (ALPRs) are old-school, as ELSAG's plate readers can scan up to 1,800 plates a minute. Massachusetts is required to submit all that captured data to the state criminal justice database which is accessible by state and federal law enforcement, the FBI, etc. Other states are combining the thousands of surveillance cameras with advanced database mining programs for faces, names, personal info. The NYPD is creating a "ring of steel" which allows for "near total surveillance" over downtown Manhattan.

It's all being collected and dumped into databases whether you are cool with your privacy being invaded in such ways or not. It makes connecting the dots about you and everything about your life so much easier. The researchers used three experiments. Experiment 1 was about "online-to-online re-identification" such as taking photos from dating sites, where people used pseudonyms, and then using face recognition to re-identify people. Experiment 2 was about "offline-to-online re-identification" and successfully re-identified one-third of the people. Experiment 3 was about using augmented reality to prove "it is possible to start from an anonymous face in the street, and end up with very sensitive information about that person, in a process of data 'accretion.' In the context of our experiment, it is this blending of online and offline data - made possible by the convergence of face recognition, social networks, data mining, and cloud computing."

In the "augmented reality" case of the Carnegie Mellon researchers, most of data collected was willingly put on social networking sites by the people themselves. "Faces as conduits between online and offline data" is potentially the "democratization of surveillance" and creates the "emergence of PPI: 'personally predictable' information." The researchers stated, "The coming age of augmented reality, in which online and offline data are blended in real time, may force us to reconsider our notions of privacy. What will privacy mean in a world where a stranger on the street could guess your name, interests, SSNs, or credit scores?"

Like this? Here's more posts:

Follow me on Twitter @PrivacyFanatic

SUBSCRIBE! Get the best of CSO delivered to your email inbox.