From 127a9ef9baaf47d554aafe6c31b2e24ceb651e62 Mon Sep 17 00:00:00 2001 From: Adrian Gruntkowski Date: Wed, 18 Oct 2023 12:05:25 +0200 Subject: [PATCH] Document reasoning behind disabling deduplication in import tables (#3439) --- lib/plausible/purge.ex | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/lib/plausible/purge.ex b/lib/plausible/purge.ex index 4288f74b5..b774798ba 100644 --- a/lib/plausible/purge.ex +++ b/lib/plausible/purge.ex @@ -5,6 +5,30 @@ defmodule Plausible.Purge do Stats are stored on Clickhouse, and unlike other databases data deletion is done asynchronously. + All import tables have MergeTree's deduplication mechanism _disabled_ by setting + `replicated_deduplication_window` from default 100 to 0. When enabled, every insert + into a given table is compared against hashes of 100 previous inserts (as complete + parts, not concrete rows) and ignored when match is found. The prupose of that + mechanism is making inserts of exact same batches idempotent when retrying them + shortly after - for instance due to timeout, when the client can't easily tell if + previous insert succeeded or not. Deduplication, however, only considers inserts, + not mutations. Deletions do not affect stored hashes, so further inserts of parts + that were deleted will still be treated as duplicates. That's why this feature + is disabled for import tables. + + Although deletions are asynchronous, the parts to delete are "remembered", so there's + no risk of overlapping deletion causing problems with import following right after it. + + IMPORTANT: Deletion requires revision if/when import tables get moved to sharded CH + cluster setup. Mutation queries, which have to be run with `ON CLUSTER` in such setup, + dispatch independent queries across shards and those queries can start at different + times. This in turn means risk of deletions corrupting data of follow-up inserts + in some edge cases. Ideally, imported entries should be unique for a given import + - an extra `import_id` column can be introduced, holding identifier. Last processed + import identifier should be stored with other site data and should be used for scoping + imported stats queries. No longer used imports can then be safely removed fully + asynchronously. + - [Clickhouse `ALTER TABLE ... DELETE` Statement](https://clickhouse.com/docs/en/sql-reference/statements/alter/delete) - [Synchronicity of `ALTER` Queries](https://clickhouse.com/docs/en/sql-reference/statements/alter/#synchronicity-of-alter-queries) """