• United States




Trust and safety 101

Jul 07, 20174 mins
Data and Information SecurityIT SkillsTechnology Industry

Creating a trust and safety team, even if it consists of a small group of part-time employees, can pay dividends in brand equity and user trust.

teach train coach locker room chalkboard
Credit: Thinkstock

The world is becoming more connected every day. With increased connectivity comes increased productivity and, unfortunately, increased risk.

SaaS companies ask that customers export their sensitive data to the cloud. Transportation providers like Uber require that customers get into a stranger’s car. Mobile social apps give today’s teenagers unprecedented connectivity – along with exposure to bullying and inappropriate content. Online marketplaces like Craigslist want their customers to send money to someone they have never met. And e-commerce providers want customers to share and save personal information including credit card numbers to enable faster (and more personalized) purchasing next time.

Most people are happy to try new services they can access from their PCs and mobile devices. The only reason these services continue to exist is because customers trust them. Once trust is broken, the brand is irreparably harmed and customers will leave.

Successful businesses invest in trust and safety teams to protect their users, their brand, and their bottom line from malicious users. While trust and safety, risk, fraud, and application security have some things in common, there’s a key difference. Risk, fraud, and application security teams protect against attacks on the business. The trust and safety team protects against attacks on the user.

Trust and safety versus security: what’s the difference?

Unlike the application security team, the trust and safety team doesn’t worry about threats like SQL injection attacks or cross-site-scripting bugs. The trust and safety team worries about protecting user data to prevent compromised accounts and fraudulent transactions. They also worry about policy enforcement, including stopping spam, scams, harassment, and fake accounts. These malicious tactics often can’t be detected by your Web application firewall (WAF), which detects bugs in your application or massive bot activity.

Trust and safety teams have different metrics, too. As they’re protecting the customers of the business, they work closely with customer support teams to resolve issues at scale. These metrics include how many incidents are solved reactively, such as when users write in to complain, compared with proactively, before they are noticed by users. Trust and safety teams also need to keep track of how long it takes for an incident to be resolved, and what percentage of users are affected by false positives.

Trust and safety is cross-functional

Trust and safety teams are cross-functional groups comprised of analysts, customer success representatives, policy managers, product managers, designers, data scientists and software engineers. Most trust and safety teams start out reactively: a customer complains about a bad product experience and the team does their best to fix the issue after it happens. Soon, the team begins proactively reviewing high-risk behavior, like posting public content, making a purchase from a risky country, or making a critical settings change.

The problem with manual review is that it is expensive and slow. Even when companies choose to offshore their manual review to save money, the accuracy of the reviews becomes difficult to maintain. The solution is to introduce automation in the form of rules and machine learning models to mark certain behavior as clearly good, some behavior as clearly bad, and reduce the “gray area” that needs to be manually reviewed to a manageable amount. While this requires specialized expertise and infrastructure, it often pays for itself quickly in saved manual review time.

Automation isn’t the end of the story. A mature trust and safety team understands that user behavior often isn’t simply “good” or “bad”; it’s a spectrum. That’s why trust and safety teams may add checkpoints to the product. For example, if a user wants to download an archive of all of their private data from a suspicious IP, rather than outright block the activity, perhaps the product will require the user to re-verify their phone number with an SMS message. There are several such checkpoints that you can introduce, including CAPTCHA, user education, and email/SMS verification.

Trust and safety, like security, is absolutely critical when putting a service online in 2017. Fortunately, getting ahead of the ball isn’t difficult and can be approached incrementally. Creating a trust and safety team, even if it consists of a small group of part-time employees, can pay dividends in brand equity and user trust.


Pete Hunt is co-founder and CEO of Smyte, a cybersecurity startup based in San Francisco. Prior to founding Smyte, Hunt led the Instagram web team at Facebook and built Instagram’s suite of business analytics products. Before that, he was one of the original members of React.js, Facebook's largest open source project, and was key to taking it from an internal tool to a massive open source library.

Hunt earned a B.A. in Information Science and Masters in Computer Science from Cornell University, where he was also Sigma Phi Epsilon Vice President of Recruiting, Varsity Heavyweight in Rowing, and WVBR Radio DJ.

The opinions expressed in this blog are those of Pete Hunt and do not necessarily represent those of IDG Communications, Inc., its parent, subsidiary or affiliated companies.