Trust and safety 101

The world is becoming more connected every day. With increased connectivity comes increased productivity and, unfortunately, increased risk.

SaaS companies ask that customers export their sensitive data to the cloud. Transportation providers like Uber require that customers get into a stranger’s car. Mobile social apps give today’s teenagers unprecedented connectivity – along with exposure to bullying and inappropriate content. Online marketplaces like Craigslist want their customers to send money to someone they have never met. And e-commerce providers want customers to share and save personal information including credit card numbers to enable faster (and more personalized) purchasing next time.

Most people are happy to try new services they can access from their PCs and mobile devices. The only reason these services continue to exist is because customers trust them. Once trust is broken, the brand is irreparably harmed and customers will leave.

Successful businesses invest in trust and safety teams to protect their users, their brand, and their bottom line from malicious users. While trust and safety, risk, fraud, and application security have some things in common, there’s a key difference. Risk, fraud, and application security teams protect against attacks on the business. The trust and safety team protects against attacks on the user.

Trust and safety versus security: what’s the difference?

Unlike the application security team, the trust and safety team doesn’t worry about threats like SQL injection attacks or cross-site-scripting bugs. The trust and safety team worries about protecting user data to prevent compromised accounts and fraudulent transactions. They also worry about policy enforcement, including stopping spam, scams, harassment, and fake accounts. These malicious tactics often can’t be detected by your Web application firewall (WAF), which detects bugs in your application or massive bot activity.

Trust and safety teams have different metrics, too. As they’re protecting the customers of the business, they work closely with customer support teams to resolve issues at scale. These metrics include how many incidents are solved reactively, such as when users write in to complain, compared with proactively, before they are noticed by users. Trust and safety teams also need to keep track of how long it takes for an incident to be resolved, and what percentage of users are affected by false positives.

Trust and safety is cross-functional

Trust and safety teams are cross-functional groups comprised of analysts, customer success representatives, policy managers, product managers, designers, data scientists and software engineers. Most trust and safety teams start out reactively: a customer complains about a bad product experience and the team does their best to fix the issue after it happens. Soon, the team begins proactively reviewing high-risk behavior, like posting public content, making a purchase from a risky country, or making a critical settings change.

The problem with manual review is that it is expensive and slow. Even when companies choose to offshore their manual review to save money, the accuracy of the reviews becomes difficult to maintain. The solution is to introduce automation in the form of rules and machine learning models to mark certain behavior as clearly good, some behavior as clearly bad, and reduce the “gray area” that needs to be manually reviewed to a manageable amount. While this requires specialized expertise and infrastructure, it often pays for itself quickly in saved manual review time.

Automation isn’t the end of the story. A mature trust and safety team understands that user behavior often isn’t simply “good” or “bad”; it’s a spectrum. That’s why trust and safety teams may add checkpoints to the product. For example, if a user wants to download an archive of all of their private data from a suspicious IP, rather than outright block the activity, perhaps the product will require the user to re-verify their phone number with an SMS message. There are several such checkpoints that you can introduce, including CAPTCHA, user education, and email/SMS verification.

Trust and safety, like security, is absolutely critical when putting a service online in 2017. Fortunately, getting ahead of the ball isn’t difficult and can be approached incrementally. Creating a trust and safety team, even if it consists of a small group of part-time employees, can pay dividends in brand equity and user trust.