How Might a Social Media Crackdown on Self Harm Content Actually Work?

By Mark Brown

Social media as means of ordinary people sharing, discussing and meeting others has democratised the web. Anyone can post, respond, distribute anything they want. It has broken down barriers between people and information and made it possible for anyone, anywhere to discuss or learn about anything they want. There are no editors, no locked doors. This grand liberation of information has brought both great positives and also created great anxieties as what is online seeps offline, sometimes with fatal results.

On 12th February 2019 The UK Independent reported The National Society for the Prevention of Cruelty to Children’s call for a change in law to make social media companies that breach “duty of care” laws designed to keep children safe online face criminal investigation and unlimited fines. This call came after Secretary of State for Health and Social Care Matt Hancock’s January letter to social media companies prompted by the death of Molly Russell in 2017 telling them ““It is appalling how easy it still is to access this content online and I am in no doubt about the harm this material can cause, especially for young people. It is time for internet and social media providers to step up and purge this content once and for all.”

There are some precedents for how such an attempt to ‘crack down’ on a particular topic of content on social media platforms might play out. None of these prior attempts to manage user generated content fill me with great confidence that legitimate conversations and mutual support from peers around self harm won’t been nuked from orbit along with malicious or damaging material.

While platforms like Instagram, Facebook, tumblr, Youtube and Twitter are mainly public, other platforms like Snapchat and Whatsapp are private but are still mediums for the sharing of content, as indeed are email or text message. Reposting, remixing, quoting, copy and pasting, sending via private message or private platform makes content of any type mobile. Anything that is on the web can be shared and re-shared. Once it’s in the ecosystem, it’s there to stay.

The desire to eradicate damaging material from the web is a seductive one. The challenge with self harm related content is that it what is considered harmful about it is its subject as much as its intent. We might see a number of categories of intention in social media posts about self harm. First might be triggering but unintentional (discussion of own experience). Second might be socially endorsing (own experience plus endorsement). Third might be malicious triggering. Some self harm material is passive (it exists in a social media space and can be found). Some self harm material is targeted (sent as a form of harassment or online violence sending passive material purposely to a vulnerable individual). Some is discursive or shared conversation. The notion of self harm material as a vector for seducing the innocent removes any nuance of intention. If it’s there, the platform is liable.

The popular consciousness assumes that it’s possible to draw a very clear distinction between self-harm promoting material and material that is constructively autobiographical or supportive. This is problematic where the nature of self harm is that it is an embodied experience. It happens to people’s bodies, often with long lasting or permanent results. Self harm related material represents a variety of the ‘don’t think of an elephant’ conundrum, where the notion of the line between harmful and helpful is very much in the eye of the beholder.

The notion that every post on social medial will pass before a human set of eyes who will then give it a tick before allowing it onto the platform is a gloriously steampunk view of how social media works. The major challenge around moderation of social media content about anything, not just self harm, is volume. While we can take stats about volume of content shared or posted with a pinch of salt, interactions with social media content on a daily basis is in the hundreds of millions.

Social media companies have a limited number of options in ‘purging’ their platforms of self harm content. The first and primary step for a social media organisation is to declare certain forms of content against the terms and conditions of their platform and to set out a clear tariff of sanctions for posts or accounts that break those terms and conditions. Beyond that, the question for social media platforms is how material deemed problematic is detected. There are really only two mechanisms for this to occur: other users flagging content deemed to be of concern; or algorithmic or other means screening either flagged content or all content that passes through the platform via automatic means.

Twitter currently has a function for reporting tweets that cause others to worry about whether the tweeter is at risk of suicide, sending an automated message with helpline numbers. While this may be useful, the net effect for people who live suicidal feelings and discuss them openly as a form of peer support is that they get multiple messages regularly containing the same help. The UK Samaritans introduced their Samaritans Radar app in 2014, which allowed twitter users to monitor the accounts of other people for suicide or mental health related content, utilised an automated version of this flagging, sending a message not to the person who was tweeting but to the person using the app suggesting they ‘reach out’. Privacy concerns and concerns of twitter users who used twitter to discuss their mental health led the Samaritans to retire the app after a month. Given that the concern is self harm content, not the wellbeing of users of platforms who experience self harm, it’s likely that this form of moderation would aim at automatic takedowns of posts or content or the suspension of accounts with an appeals process after the fact.

In 2017, Facebook began a global roll out of AI-assisted content moderation to detect suicidal idea in user submitted content. Flagged content is brought to the attention of human moderators who can escalate the case. According to a Techcrunch report ““the moderator can then contact the responders and try to send them to the at-risk user’s location, surface the mental health resources to the at-risk user themselves or send them to friends who can talk to the user.” Facebook cannot implement this under EU law, but in the rest of the world could potentially send the police to your house for a ‘wellbeing check’. Removing self harm content would likely use a combination of AI and human moderation in the same fashion.

Given that the aim of the current discussion is to remove self harm content, rather than to assist or help those who post it, the most likely model to be implemented at scale will be something like Tumblr’s crack down on sex-related content. In March 2018 US Congress passed the Fight Online Sex Trafficking Act and the Stop Online Sex Trafficking Act (FOSTA/SESTA), a statute that gives prosecutors more powers to tackle sex traffiking. According to Paris Martineau writing for tech magazine WIRED, this opened the door for social media platforms to “be held criminally and civilly liable for the actions of their users. The law’s passage immediately led to the closure of several sex-work-related online venues, such as Craigslist’s personals section, numerous subreddits, and Patreon’s support for adult creators.”

The effect on Tumblr was immediate for those who used the site for sex and sexuality related content. Tumblr implemented what amounts to an all-out ban on ““photos, videos, or GIFs that show real-life human genitals or female-presenting nipples, and any content—including photos, videos, GIFs and illustrations—that depicts sex acts.” According to Martineau: “To draw the distinctions, the company says it will use a mix of machine learning and human moderation, and that all appeals for posts erroneously flagged as adult will be reviewed by “real, live human eye(s).” As the WIRED piece remarks, this is proving to be less than perfect in practice: “Tumblr users are already finding problems with the flagging system. Classical paintings of Jesus Christ were flagged, as were photos and GIFs of fully clothed people, patents for footwear, line drawings of landscape scenes, discussions about LGBTQ+ issues and more.” AI is not infallible and will struggle even more with the nuance of a complex issue such as self harm. Google’s own AI for reading images got into hot water when it labelled photos of black people as ‘gorillas‘. Gorillas and black people aren’t similar at all. Images or videos of self harm will likely prove even more difficult to label reliably.

There’s a difficult line of nuance to be walked with the issue of social media and self harm. Whatever we say or do via social media is content to someone if they consume it. Self harm remains a difficult and sometimes shameful experience and social media may be the only place where individuals can openly discuss their experiences with peers and others. These calls for legislation privilege the views of those alarmed by self harm over those with lived experience. If they are form focus for future action, there is a perverse risk those seeking legitimate peer support or helping others minimise risk or damage will be first to be ‘taken down’ and pushed out of social media spaces.

Mark Brown is development director of Social Spider CIC, ex-editor of One in Four magazine and is @markoneinfour on twitter. He writes regularly on mental health, technology and other subjects. He is currently writer in residence at Centre for Mental Health.

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: