Mind’s Eye surveillance to watch, identify and predict human behavior from video

If a person holding a gun were to walk up to you, what might you think would happen next? Researchers from Carnegie Mellon have created intelligent software that will identify human activities in videos and then predict what might happen next. It should come as little surprise that the spookily named ‘Mind’s Eye’ program is sponsored by DARPA’s Information Innovation Office.  

“A truly ‘smart’ camera would be able to describe with words everything it sees and reason about what it cannot see,” said DARPA. Visually intelligent technology previously ‘thought’ in terms of nouns to describe a scene, but Carnegie Mellon researchers have made smart software that can also think in terms of action verbs. “A video shows a woman carrying a box into a building. Later, it shows her leaving the building without it. What was she doing?” asked Carnegie Mellon University.

The Mind’s Eye software “will compare the video motion to actions it’s already been trained to recognize (such as walk, jump, and stand) and identify patterns of actions (such as pick up and carry). The software examines these patterns to infer what the person in the video is doing. It also makes predictions about what is likely to happen next and can guess at activities that might be obscured or occur off-camera.”

Carnegie Mellon University’s National Robotics Engineering Center explained the image below as: “The Mind’s Eye program will automate video analysis – recognizing current behavior, interpolating actions that occur off-camera, and predicting future behavior.”

The next step is to make the ‘Cognitive Engine’ even smarter. According to the report “Using Ontologies in a Cognitive-Grounded System: Automatic Action Recognition in Video Surveillance” [PDF], the researchers “plan to extend the system functionalities in order to support a wider range of action verbs and run tests on a large video dataset.”

DARPA explained, “In the first 18 months of the program, Mind’s Eye demonstrated fundamentally new capabilities in visual intelligence, including the ability of automated systems to recognize actions they had never seen, describe observed events using simple text messages, and flag anomalous behaviors.” Carnegie Mellon is one of twelve research teams and three commercial integrators participating in the five-year Mind’s Eye Program.

Previously, BRS Labs had the “smartest AI suspicious behavioral recognition system” with “the capability to learn from what it observes, remember activity patterns and adjust to changes in the environment, field of view and equipment – without manual interaction.” Phys reported that the Carnegie Mellon “researchers’ approach is designed to help prevent crimes or dangerous events from happening.” The newest visually intelligent software “system would eventually sound an alarm if it recognized that an action was not permitted, detecting anomalous behaviors. One example of such a scenario would be the cameras at an airport or bus station, flagging a bag abandoned for more than a few minutes.” This Army-funded AI research was disclosed “at the Semantic Technology for Intelligence, Defense, and Security conference at George Mason University.” 

The Mind’s Eye system could potentially be used to analyze live drone footage. Who knows, it might even be integrated to work with the unblinking surveillance stare of the Army’s 7-story flying football field-sized blimp? It will likely be embraced in the future by the police and by the military to keep soldiers out of harm’s way. It might have home security applications, watching the surveillance video and alerting home owners with a text message before burglars break in.

On the creepy privacy invasion side of the coin, it’s one more surveillance technology hunting suspicious behavior. Let’s hope the researchers get it right because when added to social media surveillance helping the government read your mind and future TSA plans to track all ‘daily travels to work, grocery store and social events’, the future surveillance society world could have a very Orwellian no-privacy flavor.

Like this? Here’s more posts:

• Time to disable Java AGAIN: 1 billion at risk from newest critical Java bug
• Feds Warn of Zombie Apocalypse! Buy emergency kit, but you might be a terrorist if…
• Famous Patriot Hacktivist The Jester Shares Battle Chest of OSINT Tools
• Smartphone snoop: Even when phone sleeps, digital assistant always eavesdrops
• Facebook Want Button: Collecting massive amounts of data about you has never been easier
• Busted! Forensic expert who recovered lurid SMS warns: Phone texts don’t die, they hide
• Deanonymizing You: I know who you are after 1 click online or a mobile call
• Does Microsoft Oppose Verizon Spying? Verizon allegedly wants more control over WP8
• Flame’s vicious sibling miniFlame malware, a cyber-espionage ‘surgical attack tool’
• Do you allow RDP connections? Cybercrime service sells hacked Fortune 500 access
• Surveillance State: From Inside Secret FBI Terrorist Screening Room to TrapWire Training
• Social media surveillance helps the government read your mind
• iPredators: Cyberbullying to cyber-harassing troll cop from hell

Follow me on Twitter @PrivacyFanatic