• United States



#BlackHat #DefCon preview: Jericho reflects on 13 years of Errata

Jul 20, 20126 mins
Data and Information SecurityIT Leadership

One of the more popular websites in the security community is — particularly the Errata section, in which so-called charlatans of the industry are exposed. At Black Hat, we’ll hear from Brian Martin — a.k.a. Jericho – on the history of Errata and how the project has evolved over 13 years. In a phone interview and some email exchanges, Jericho gave me a preview.

CSO: What was the catalyst that made you decide to embark on this mission 13 years ago? Was it a specific incident or just a build up of frustration over the things you were observing in the industry?

Jericho: I think it was a slow boil starting in 1996 (when I first worked in security professionally) and leading into 1999. I kept seeing things that struck me as hypocritical or unethical, while the term “ethical hacking” was being thrown around more and more. I figured if I showed all these cases in one place, it would make people realize how absurd our industry was becoming and encourage companies to watch themselves. I was wrong. I was particularly bothered by what I was seeing in the media. Articles back then were all based on FUD and guesswork.

You’ve also made it a point to name companies that have done wrong.

Around 1998 and 99 there was a big ramp-up in the area of penetration testing and we saw charlatans focusing on that. At the same time, big companies were making disingenuous claims that they would not hire blackhats. I knew one at each of these companies. Around 2003-04 we started seeing many people claiming to be security experts with 15-20 years of experience, when in truth they were just IT administrators doing a small amount of security.

When you set your sights on a potential charlatan, what is your process for vetting the person’s work and background?

There is no formal process. From the very beginning, our interest in a person or organization is sometimes crowd-sourced. We see snide remarks or accusations on a mail list or Twitter, mentally note it, and move on. Sometimes we’ll take notes and start a file, which becomes what we have recently referred to as the “unpublished watch list”. Presence on that list does not mean someone is a charlatan, just that someone has a problem with them and it is worth digging further as time permits.

If there is enough information or incidents to warrant a write-up for the charlatan page, it becomes a simple but time consuming process.

Investigate every complaint, read every page (positive or negative) about the person, read everything they have written, and more. From that, several files are started, one for each transgression or problem. Eventually they are worked into the articles you see today. Before publication, they are reviewed a second time by the author at least 24 hours later, to help ensure emotion isn’t the strongest factor in publishing. Then they are reviewed internally, usually by two people; one person involved in Errata that is familiar with the process, and one person that is not involved other than being the first to read the article. This is done so that a set of fresh eyes, who has no real history in the security industry, gives it a balanced review. That person will check references, make sure that our logic and assertions are backed up, and help ensure that the article avoids statements that could be considered libelous or ambiguous.

After all of that, the article is published and open for peer review. From then on, feedback from anyone, including the subject of the article, is taken into account. The article will be modified or updated as we see appropriate.

Have you ever put people on the charlatan list but removed them after further consideration? Perhaps that person deserved to be there in the first place but they went on to redeem themselves?

This has happened one time so far. He was published on the “Watch List”, which typically have fewer articles than the full charlatan page. There were two major factors that led to him being included; a book he was responsible for, and some of his presentations at conferences. Two years ago he and I sat down at Defcon and went over everything. He gave me additional information that I did not have access to, and I gave him our perspective on why he was included to begin with. At the end of the sitdown, we agreed he would send over some additional information, and I would review additional conference presentations. If I found more errata in them, he would stay on the page. If I didn’t find more errata, then his primary page would be removed, but the portion related to the plagiarism in his book would stay up. He agreed, and ultimately the charlatan page was removed. The biggest factor in removing was that he truly sought to improve himself. He had handled the book plagiarism incident fairly well, in that he took responsibility in the end. While I personally have a problem with how the second edition got written (e.g., still largely not by the named author), that wasn’t grounds for inclusion on charlatans. Moving forward, he understands that if he ‘slips up’, we’ll be there watching and won’t hesitate to re-add him.

How many people and organizations are on the unpublished watch list right now?

Eighty-five – some companies but mostly individuals.

You mention in the talk description on the Black Hat website that you will talk about areas where you fell short. Can you offer some examples here?

One portion of the talk is titled “Why errata hasn’t lived up to expectations.” It briefly looks at how it hasn’t lived up to our expectations, the communities, and why. On our side, we think it fell short due to dismal community support. Almost no volunteers, getting people to report ‘bad things’ is often like pulling teeth, and ultimately very few people reference the site when doing background checks. The media still quotes charlatans because they say what the media outlet wants to hear, not what is rational and correct.

What would you say to those who live in fear of being put on your list?

The only reason you should have fear is if you know you’re doing things wrong and hiding it. If you don’t hide your mistakes and you’re the type that makes a constant effort to improve, you have nothing to worry about.

How many people does it take to pull this off? I know a core group of people contribute to, but are there any regular contributors from outside your circle?

There are two of us who do the daily updates and a bulk of the work. I am the only one who has stuck with it the entire duration of the project. In the 13 years, we have had several people that contributed heavily for short periods, and several volunteers that did research or contributed for short stints.