Image moderation without storing images
Not research on a model — the classifier is off-the-shelf and says so — but on the regime it runs in: zero retention, a score instead of a verdict, fail-closed under outage, and a refuted hypothesis about reactive moderation as a cheaper substitute for scanning. Includes what has not been measured.
Koch Laboratory — image moderation without storing images
The opening question was: what exactly does someone buy when they buy content moderation — given that the classifier itself is free and any developer can wrap it in a weekend? The answer we arrived at moves the whole problem away from model accuracy and towards the regime the model runs in: retention, failure behaviour, and where the decision is taken. Those choices are not settled by the choice of model — and they are the subject here.
Method. Problem → hypothesis → falsifiable criterion → decision with its rationale. Statuses stated openly: “implemented” means running, “design work” means no measured result, “rejected” means tested and abandoned with the reason given.
1. Direction 1 — moderation with no image retention
Research question. Can visual content moderation be run without storing a single moderated image — while still leaving an evidential trail sufficient for a regulator? Why it is hard. Two duties pull in opposite directions. Data minimisation says: do not keep other people’s images, least of all the suspect ones. Accountability says: show what was checked, when, and what was decided. The obvious solution — “let’s keep the image in case of a dispute” — turns the moderation vendor into an archive of its customers’ most sensitive content. State of the art (published). Cloud moderation services typically retain content samples for model development; documentation duties under the DSA; hash matching as a separate ecosystem for prohibited material. Success criterion. Zero content storage: processing in memory only, the log limited to metadata (time, size, score, decision). The evidential trail describes the decision, never the content. Result. Confirmed within the criterion: the image is not written at any stage; only metadata of the moderation event is recorded. Boundary condition stated openly: with zero retention there is no going back to the disputed image — a dispute is settled on the record of the decision and on whatever material remained with the customer. Anyone who needs a content archive needs a different product. Status: implemented.
2. Direction 2 — a score instead of a verdict, the threshold on the customer’s side
Research question. Who should set the line between “allowed” and “blocked” — the vendor of the classifier or the operator of the platform? Why it is hard. The cost of a mistake is asymmetric and depends on a context the vendor does not know: a children’s app will accept a hundred false alarms rather than one item slipping through; an adult forum the opposite. Returning only a verdict hides that decision and shifts responsibility to the vendor, who lacks the data to take it. State of the art (published). Classifiers returning a label; moderation interfaces returning category labels without raw values. Success criterion. The raw score (0.0–1.0) is always returned; the decision field is computed solely against the threshold set by the customer and explicitly marked as derived from it. Result. Confirmed: the interface returns scores and a decision computed at the customer’s threshold. An open consequence: setting that threshold honestly requires data on the score distribution in the customer’s actual traffic, which does not exist at the start — see direction 5. Status: implemented.
3. Direction 3 — behaviour under failure: silence or refusal
Research question. What should a moderation service sitting inside a customer’s critical path do when its own inference stops responding? Why it is hard. Both answers are expensive. “Let it through” (fail-open) keeps the customer’s service flowing but converts an outage into silent consent to publish unchecked content — with no trace that anything happened. “Refuse” (fail-closed) stops the flow and hands the decision back to the customer at the worst possible moment. State of the art (published). Fail-open/fail-closed patterns in security systems; degradation policies for third-party services. Success criterion. Unambiguous behaviour, documented before adoption, never silent. Result. The decision: when inference is unavailable the service returns 503 and the customer decides what to do — usually routing the case to a manual queue. The outage is visible in the response, not hidden inside a score. Boundary condition: the customer has to handle that case on their side; a service that does not admit this before adoption is selling them concealed risk. Status: implemented and documented.
4. Direction 4 — a refuted hypothesis: reactive moderation as a cheaper substitute for scanning
Research question. Instead of checking every upload, is it enough to check content only after users report it — cheaper, at comparable effect? Why it was tempting. The number of requests drops by orders of magnitude, inference cost stops growing with traffic, and the community points at exactly the cases a model cannot understand from the image alone (harmless content in the wrong context). Result: hypothesis refuted. For two independent reasons. First, the mechanism by definition lets content appear and then waits for reports — with dangerous or illegal material that is not compliance, it is documented omission. Second, a reactive layer requires holding references to reported, still unverified material — which turns the service into a register of suspect content, the exact opposite of direction 1. The saving was real; the price turned out to be unacceptable. What survives. User reports remain as a second layer above the proactive scan (re-check and a manual queue once a report threshold is crossed), never as its replacement. Separating those two roles is the principal result of this direction. Status: rejected as a substitute, retained as a supplementary layer (design work).
5. Direction 5 — what we have not measured
Research question. Does a model tuned on real traffic outperform the raw open model — and can that advantage be demonstrated publicly rather than asserted? Why it is hard. An honest benchmark requires data that, under zero retention (direction 1), simply does not exist; obtaining any of it must be agreed with the customer case by case and settled separately in legal terms. This is a genuine tension between two of our own principles, and it is not smoothed over here. Success criterion. A public, repeatable benchmark of “tuned model vs raw open model” on a dataset described as to its provenance — including negative results if there is no advantage. Result. None — no measurement has been carried out. What runs today is an off-the-shelf open model, quantised to int8 and baked into the container image; any claim of an advantage would be an assertion with nothing behind it. Status: design work, result unknown.
6. Direction 6 — boundaries drawn deliberately
The model as a component, not as a download. The weights are baked into the container image; the inference container is not exposed and runs on an internal network. Criterion: it starts with no internet access and no dependency on somebody else’s weight host — the service must not fail because another party withdrew a file. Licence as a design boundary. Model choice is bounded by licence terms, not accuracy figures alone; a model with excellent numbers whose licence cannot be reconciled with a closed installation at the customer’s site is useless for this product. That boundary was drawn deliberately and early, not after the fact. Prohibited material out of scope. Detection of child sexual abuse material is explicitly excluded: it is a separate, tightly regulated ecosystem built on hash matching and reporting duties, not a task for an in-house classifier. Naming that boundary plainly is part of the product, not small print. Status: implemented (boundaries in force).
Status of the direction. The operating regime — zero retention, a score instead of a verdict, fail-closed, self-hosted operation at the customer’s site — is implemented and running. There are no measurements of our own model, and there will be none until the benchmark described in direction 5 exists. Next step: a queue of borderline cases with a human in the loop, with the share of cases handed to manual decision measured openly.