A tech leader's day can be unpredictable, but Ginny Davis, CIO at entertainment services company Technicolor, can rely on one thing: She's guaranteed to get an email from a new security provider urging her to check out its latest and greatest technology.Davis says she values "the evolution in the fight against hackers" and considers the many new options a positive trend, "but it's mind-numbing how quickly [the security landscape] is changing."Bob Lamendola, general manager of infrastructure services at IT services provider Mindshift, agrees. "The number of security-related products and services coming at you is almost frightening. The [security] marketplace is evolving at a frantic rate, making a complex situation even more complex to navigate."This back-and-forth escalation between the good-guy developers and bad-guy hackers is putting even more pressure on tech leaders. "Whenever you bring a new security product in, it's like putting up a 10-foot wall," says Tom Barnett, vice president of healthcare IT at NorthShore University HealthSystem in Evanston, Ill. "But the bad guys just come back with an 11-foot ladder." At a similar healthcare organization, Barnett says inbound hack attempts jumped from a few million per year in 2008 to around a million per month in 2012.Security consultant Larry Ponemon, founder of Ponemon Institute, acknowledges that the onslaught of new applications is a problem. "Generally, people are frustrated because their security environment is very complex," he says. "That leaves places where the bad guys can get in. The more we rely on these tools, the greater the possibility that they create more havoc than value."So what's a tech leader to do? And which group is poised to swamp you first, the hackers or the security vendors? The cumulative advice of the CIOs, chief information security officers and consultants interviewed for this story boils down to this: There are times when you might need both a belt and suspenders to protect your security portfolio, but prudence will always win out over panic.Where IT goes wrong with security applicationsThe high-profile demands of protecting the enterprise sometimes drive CIOs to stumble when it comes to procuring security applications. "The challenge is hard for a lot of organizations to grasp because they're running at 100 miles per hour," says Doug Davidson, U.K. cybersecurity CTO at consulting firm Capgemini. "Sometimes they procure rashly.""Vendors show up with a new product that they claim is innovative. It's shiny, sexy and exciting, and they try to tell me that I need it," says Deborah Blyth, CISO for the state of Colorado. She recalls being enticed by a security vendor's product that seemed cool, and even strategic \u2014 but after purchase, it ended up on the shelf. "Even though we didn't pay a lot for it, it didn't work consistently, and it had a lot of limitations," she says.Blyth's experience is typical. In Ponemon's 2015 Global Study on IT Security Spending & Investments, sponsored by SecureWorks, respondents said that in the past two years, an average of 37% of their security technology investments fell below expectations. The reasons included a lack of in-house expertise, installation costs, support and complexity issues, and sometimes just the effectiveness of the software (see chart, below).While this is a major issue, Mindshift\u2019s Lamendola says trial and error is part of the process. "We\u2019ve tried a lot of solutions with varying degrees of success. You think a particular product is right for you, then you find out it isn\u2019t. You have to break a lot of eggs to find out what works best within your environment."Another flaw in IT\u2019s security strategy is the trap of being seduced by features rather than products. "You get inundated with proposals and pitches," says NorthShore University HealthSystem\u2019s Barnett. "Many of them can be redundant, but they have one or two special features that look interesting."That's a problem, warns Bill Burns, CISO at data integration vendor Informatica. Before joining the company, he worked as the information security executive-in-residence at Scale Venture Partners, a Foster City, Calif.-based venture capital firm. There, he worked on creating a framework for investing in information security that closely replicated the issues that CIOs face. "The security startup market was growing quickly, but it was hard to separate the signal from the noise," he says.When looking at startups, Burns says, it's hard to distinguish a security feature from a full-fledged security product. "You have to ask yourself whether technologies really will help create companies or will they be folded into a larger platform?"He cites the technology behind machine learning that was so popular a few years ago. Many in IT expected that technology would become the next security information and event management tool. But instead, it turned out to be a feature, not a product, and something that's now incorporated into multiple products.Where security applications go wrongIT has the unenviable task of having to sift through a plethora of tools that rarely if ever exchange data, including analytics, mobile, threat detection, data loss prevention, email, web, cloud, encryption, endpoint security, enterprise security, network security and identity verification. That raises questions: Are there gaps between tools, and if so, is IT missing information? Or are there overlaps, and if so, is IT spending too much for security?Security applications that are knitted together would help. "That would be nirvana," says Lamendola. "The ability to have these products integrate from a common platform would be an enormous benefit for most organizations. It would remove a huge obstacle toward adoption. Tool sets integrated more tightly would improve the security profile of most organizations."Yet that vision is unlikely to materialize. "There is no Microsoft Office of security," says John Pescatore, director of emerging security trends at SANS Institute. "There is no one vendor whose applications work together and are good enough for what's necessary." He says vendors failed in this effort for a simple reason: Threats change, so security tools have to keep changing, too. Vendors can't be good at everything, and that in turn drives customers to best-of-breed products.Interoperability has obstacles, too, as Ponemon notes. If one application uses unencrypted cleartext and another uses encrypted ciphertext, they won't provide visibility. Pescatore says he considers interoperability unlikely for another reason: "The vendors don't like it because they smell commoditization. They'd be compliant at message level, but then they would extend their solution because they want a way to say they're unique."Not everyone agrees. Capgemini's Davidson notes that open-source tools for security information exchange such as TAXII and MAEC might help reveal when different applications identify the same malware. The National Institute of Standards and Technology is working on SCAP, the Security Content Automation Protocol for interoperability. FIDO, or Fast IDentity Online, is an open-source standard for authentication services.The state of Colorado's Blyth is already working on a standards-based framework that would enable her agency to identify and select tools that would work together."Some [state government] agencies will have more familiarity with some tools than others, so we want to be able to choose multiple tools based on their desires and not lock into one specific tool," she says. "We have to make sure that any solution we're implementing has a standards-based framework so that we can integrate tools more cleanly with other tools."Blyth says she's already looking at taking advantage of FIDO's two-factor U2F authentication service, which requires a separate token-generating device.Where IT can go right with security applicationsGiven all these challenges, how should IT executives proceed? CIOs and CISOs cite the importance of being rational about the decision-making process, rather than being driven by what Informatica's Burns calls the "attack du jour."One approach is to develop a framework for categorizing security needs. What's the most important data you're trying to protect? Who has access? What's the data connected to? Technicolor's Davis suggests identifying the various "jewels" of the organization and the tools necessary for protecting each of them, including encryption, authentication, workflows, firewall management and others."There's a layer of protection for each jewel," she says, and you have to constantly assess whether the tools are doing their job, and if not, where are the gaps.Many frameworks are readily available; Colorado's Blyth uses the one developed by the Center for Internet Security. "This [framework] lets me vet where I have gaps and where my needs are," she says. "That helps me focus on the most important controls, look at gaps in those areas, and rein myself in from the next shiny object."Blyth also recommends doing a proof of concept within your own infrastructure. "We want to make sure someone isn't selling us futureware. We want to see that it can do what we need to have done, not that we're banking on something being available sometime in the future," she says.Also, tech leaders suggest looking on the positive side of all the innovation happening in the security field \u2014 vendors are striving for new solutions. SANS Institute's Pescatore says new innovations might come along that can better protect your company and replace your current tools. NorthShore's Barnett says, "It's not to say that once you choose something, it stays in place forever. You may have to swap technologies out as the landscape changes."Tech leaders also agree that one way to avoid gaps in security protection is to go the extra mile on process and not overbuy on products. That approach includes hiring third parties to conduct regular risk assessments. Technicolor's Davis, for example, says she can't be too careful, especially after the 2014 breach at Sony, which is just across town.She works at a company that many hackers would love to target: Technicolor is a household name with a global presence of 35 locations, and its content is highly desirable and potentially monetized. Though Davis cautions that the company would never store an entire movie in one location, she acknowledges the desirability of clips or even images from certain high-profile movies.To address these issues, she created an office of assessment testing specifically to hack the system. "If their attempts are not flagged by our log management tool, they log a report and give it to the CISO." The team has already found issues with at least one tool."It's no accident that we don't have all our eggs in one basket," she says, adding that she also subscribes to two different sources of security reports that follow what's percolating in the security space of the Darknet and social media. (For a look at other organizations offering insight and assistance, see "Who's On Your Side?" below.)"It's a belt-and-suspenders approach," agrees Barnett, who also recommends keeping on top of the security blogs. "If you\u2019re thinking about how someone could attack you and doing constant threat assessment, you\u2019ll start thinking about how to put multiple hurdles in place for the bad guys."Who\u2019s On Your Side?Consider the following public sector organizations as sources for security insights:Center for Internet Security:Oversees the Multi-State Information Sharing & Analysis Center (MS-ISAC) for cyberthreat prevention, protection, response and recovery.Information Technology-Information Sharing and Analysis Center:IT-ISAC is an IT forum for managing risk to the IT infrastructure through cyber information sharing and analysis.Infragard:An association of businesses, academic institutions, state and local law enforcement agencies, and other participants that share information and intelligence to prevent hostile acts against the U.S.