Security risks
We have just had a warning about the risks of rushing supposedly ‘safe by design’ technology. The HMD Fuse child-safety phone, previously promoted by ministers, was taken off sale after serious security vulnerabilities were discovered in its safeguarding system, including flaws which a researcher said could expose a child’s live location and allow protections to be disabled.
That does not mean every on-device safety system will contain the same vulnerabilities. But it demonstrates why giving software deep access to our cameras, screens and private communications creates additional security risks that could harm children.
Privacy risks
Adults will be required to go through an ID check point with an unregulated age verification company to disable this software. Millions of people could be faced with either sharing their biometrics or official IDs or having a phone they paid for bricked with blocking software. Since the introduction of the Online Safety Act, the British public is being increasingly pushed to compromise their privacy before they can access content or use tech features. Despite this, the Government has refused to regulate the burgeoning age verification industry.
A disproportionate approach
While there might be merits to an opt-in system that parents could set for accounts that they setup for children requiring all adults to opt-out of on device content scanning through age verification is disproportionate to address the harms.
The privacy implications of having all photos subject to intrusive scanning or requiring people to submit their sensitive data for identification purposes are too high a cost.
There is a clear risk that once on device content-scanning software was enabled it would be fairly easy to expand they categories of content it detected or censored.
It could also go beyond scanning images to people’s private communication. If implemented the operating system would be empowered to prevent an end-to-end encrypted application such as WhatsApp or Signal from delivering an otherwise private communication on the basis of an automated assessment of its content.
Nudity is not always inappropriate
Blocking images on devices would rely on “nudity-detection algorithms” built into the operating system, but nudity is in itself not always of an inappropriate sexual nature. It can also be artistic, educational or of historic importance – or simply an innocent snap of a child’s bathtime. The BBFC recognises that nudity can appear at every mainstream classification, including U, with the rating depending heavily on whether the nudity is sexualised, its detail, duration and context.
As with any scanning technology, there’s a risk of both false positives, and weaponised false negatives that can have unintended consequences. Forcing adults to use broken scanning systems risks falsely blocking medical images that people need to share with their doctor, journalism about children, artistic expression through such mediums as life drawing, breast-feeding support, sex education, or historic reporting on atorcities like the ‘Napalm girl’.
It also creates the risk of “targeted adversarial false-positive attacks on the classifier.” In this type of attack a bad faith actor could potentially engineer lawful images to trigger blocking systems deliberately.
There are effective ways to protect children online without hard-coding surveillance into everyone’s device or strong arming everyone to verify their age. The focus should be on educational methods, specialised policing of sextortion and child abuse offences, and a targeted application of this technology as part of parental control packages or on devices marketed and sold as devices for children. An opt-in mechanism for parents and people who choose to use this technology is a more proportionate response, one that doesn’t ransom privacy for free expression on a population scale. Adult devices should not be treated as child devices by default.
