What's one of the biggest insider threats to the corporate network? The high-tech folks that put it together, make changes to it, and know more about what's on it and how it works than anybody else.When the database, network or systems administrator goes rogue -- stealing data, setting up secret access for themselves, even in anger planting logic bombs to destroy data , or just peeking at sensitive information they know is off limits -- they become the very insider threat that the IT department is supposed to be guarding against.Houston hotel on watch for threats from inside and out|6 tips for guarding against rogue sys admins|How to keep employees from stealing intellectual propertyIndeed, IT workers with privileged access to the network are often considered a greater risk and potential danger than other types of employees."They're different because there's a high risk associated with the potential damage they can do," says Donna Durkin, chief information security and privacy officer for Computershare, a global financial services company.Security quiz: How well do you know the insider threat?\u00a0Mike Theis, executive director of insider threat technology for Raytheon, agrees that sys admins and others with privileged-user access are a bigger potential threat than other types of organizational employees. He says his decades-long experience as an investigator in the federal government and commercial sector has shown him that half of detected anomalies -- lapses in accepted protocol -- come from such insiders."It doesn't mean they're guilty of anything," Theis adds. "Sometimes they're just trying to get the job done, but they're outside the bounds of the organizational policy."Sometimes IT workers are pushed by demanding users, such as business and sales managers, to perform tasks in a hurry or to violate official IT policy by, for instance, adding printers on network segments where that's not allowed.One main concern about privileged access is making sure that IT workers have appropriate access only to the resources they need even as their job function may change. In North America alone, Computershare has about 100 IT workers with privileged access to IT resources who undergo monthly "entitlement reviews" to make sure their access to systems and data is appropriate to the function of their role.Computershare is moving from what Durkin calls a more "manual process" for entitlement review -- which requires input from business and IT managers as well as the human resources department -- to an automated system from vendor Sailpoint, where software will take feeds from various applications to keep a database of predefined rules and roles.By focusing on the insider threat, Computershare is targeting a real and growing problem. Insiders participate in 48% of all data breaches, according to Verizon's 2010 Data Breach Investigations Report, an analysis of 275 data-breach cases that occurred in 2009. This figure is up from 26% the previous year.The Verizon report points out that external agents such as hackers are responsible for stealing far more records than insiders. Nonetheless, the report says that most insider cases -- 90% -- are deliberate and malicious, and they usually involved misuse of privileges. The report notes that employees often get more privileges than they need to perform their job duties, with monitoring usually insufficient. Another finding is that 24% of crime tied to internal agents was associated with those undergoing a job change, whether being fired or resigning, newly hired or changing roles within the organization.\u00a0Sys admins themselves know their power is substantial and can be abused by disgruntled coworkers."The insider threat -- it's tough. You don't know. It could be the person right next to you, something has gone wrong in their lives [and] they turn bitter," says Angelo DiPietro, senior systems engineer at Commerzbank, Germany's second-largest bank with offices in the United States.DiPietro acknowledges that IT workers with elevated rights and privileges in terms of network access to information can go astray. "Out of idle curiosity, they'll log into something," DiPietro says, noting that it's human nature to test the limits of systems.But to counter that inclination, and to detect any malicious activity, security experts say companies need to monitor those with special network privileges. To that end, Commerzbank relies on software to monitor employee behavior. The firm uses ArcSight Enterprise Security Manager, a security information and event management (SIEM) product, in conjunction with Lieberman Software's Enterprise Random Password Manager to keep an eye on actions taken by sys admins and others. "Since we put it in place, everyone has been behaving," DiPietro says.The Lieberman password-manager software is sometimes referred to as a type of "firecall" tool that gives top IT managers a way to approve accounts for elevated rights of domain temporarily. It requires multiple individuals with manager status to approve any elevation of account.The system being used at the bank will automatically re-set when the time limit expires. And requests for approval of elevated privileges can be denied for any variety of reasons. "I got denied plenty of times," DiPietro says. He knows why the controls are there. "I built the environment, but I do not manage the environment."That realization apparently never sank into Terry Childs, the former network administrator for the city of San Francisco. Childs is credited with having helped build the city's modern FiberWAN network but refused to hand over the passwords for the router network in a standoff with managers two years ago, apparently afraid he was about to lose his job or be reassigned. This act of rebellion got Childs arrested, judged guilty and sentenced to four years. \u00a0One juror in the trial, Jason Chilton, who happens to be a Cisco Certified Internetwork Expert, said he found the hardest part of the case involved pondering who was supposed to be an authorized person to have the passwords, particularly since the city of San Francisco didn't seem to have any procedures or policies that employees were supposed to follow.Chilton says he found Childs a sympathetic character, though "possibly a little paranoid. But the problem he had was that he didn't have good management to keep that in check. He was allowed free rein, which allowed engineering decisions over the years that made things worse and worse, and locked people out of possibly getting into this network."Watching the watchersIncreasingly, organizations appear mindful of the potential insider threat that privileged-access IT workers represent. And though trust in them is implicit by giving them responsibilities over critical data, there's awareness that exercising strong security controls benefits the organization as a whole."Because we are in the knowledge business, we have to lead by example," says Rudy Juarez, IT project manager for the City of McAllen, Texas, which has been building a modernized data center and networks associated with a new border-crossing point into Mexico called the Anzalduas International Bridge Facility. The city is also upgrading the network used at City Hall and other municipal buildings.Some of the toughest information security for the project has gone into restricted access to the IT department's data center, where IT employees and service providers use biometric fingerprint access based on the Bioscrypt fingerprint readers to gain access."We had the opportunity to bring in the technology," Juarez says, pointing out fingerprint-based access is better than a door-access proximity card since that could be used by someone who didn't actually own the card. There is also video monitoring surveillance in the data center room, as there is in other parts of the city's municipal buildings. The goal is to unite these physical-security systems using Matrix Systems' Frontier Standard software, including new HID integrated door-access and time\/attendance cards being distributed to city employees.Since the heart of the network and security controls is the data center, there are also controls placed on access to servers, with a small number of the IT department's employees granted logical access to IT systems. The issue isn't that the other IT employees aren't trusted. It's about reducing risk, especially when much of the city's network and security infrastructure is now centralized in a data center.IT security analysts say corporations should aim to structure their IT workforce operations around the notion of "separation of duties" (also called "segregation of duties,") so that there's a way to institute checks and monitoring that prevent power from being excessively concentrated in any individual. This is not easy to do and can be expensive. Adding to this conundrum is that sys admins and others are often involved in some aspects of enterprise security deployments, though efforts are made to keep the IT security managers and auditors in control.Phil Cox, analyst at consultancy SystemExperts, says he recently learned about a sys admin who moved from group to group in IT operations over a decade, in networking, telecom and desktop support. "Over time, he accumulated access he didn't need for his job," Cox says. He was a foreign national who ended up quitting his job, and the company started looking at the last weeks of his stored e-mail, which was a common exit strategy there. They saw something "weird" that got managers thinking he had taken data. They went looking at logs and saw he had been accessing data he had access to three years ago.The strategy Cox recommends is to "keep role-based access and provisioning current. And understand what normal usage patterns are." SIEM tools are a help, but also have meaningful policies, such as "nobody logs in as root on a machine. If you see a root logon, you investigate."Enterprise password management tools for temporary access hold appeal, but they can be hard to make work because "if a sys admin has to get authorization every time they want to do something, it's a huge overhead, and it's never going to work," Cox says.Cox points out there are third-party services, such as those from SecureWorks, which look for suspicious behavior of IT employees. And of course, the idea is to keep the monitoring process, whatever it may be, out of the hands of the IT workers being monitored.Although IT workers must be informed they are being monitored, it's important to hide the actual monitoring mechanism from them so they can't disable it, Raytheon's Theis says. He adds he's seen many instances where sys admins come up with reasons why monitoring tools shouldn't run on their machines and willfully try to disable whatever they can, so it should be as secret from them as possible.There's a lot of debate about whether monitoring and other technologies to exert controls over IT workers should automatically block suspicious actions. Raytheon insider-threat expert Theis favors monitoring without blocking because "of all the activity that occurs, only a small percentage of it is malicious."Forrester Research analyst Chenxi Wang says she knows of instances of privacy violations when the sys admin was looking at e-mail and files he shouldn't be looking at, adding this is hard to stop since it may simply happen to some degree "within the normal range of operation." What's important is the intent, she adds. "Are they solving an IT problem or doing it for their own fun and games?"Wang also emphasizes that IT workers with privileged access to network resources are themselves high-value attack targets. She points out the infamous attack on Google linked to China, which is known to be associated with a privileged user as a target.Another area where the IT worker gets eyed as a suspect is in outsourcing and third-party support arrangements where IT workers, though not specifically employees of an organization, have approximately the same access to the corporate network as if they were direct employees.Clearly, one of the most important things is to be able to "accurately judge trust in people," Raytheon's Theis says. In a formal sense, that might mean background checks where red flags would be poor recommendations or criminal record.Background checks of IT personnel are becoming more commonplace if only because of the Payment Card Industry's data-security standard that now requires it in some instances. "PCI DSS actually specifies if you have someone holding cardholder data, they have to go through a background check," Cox from SystemExperts points out.James Pu, CIO at the Los Angeles County Employee Retirement Agency, says background checks are considered a mainstay for hiring IT personnel at his agency. While this is primarily in the hands of the human resources department, a background check also could involve a look at credit history and a debtor's spending patterns. "It's important to be hiring well," Pu says. Red flags clearly include any criminal history, fraud, theft or poor ethics.Read more about wide area network in Network World's Wide Area Network section.