diff --git a/lib/plausible/purge.ex b/lib/plausible/purge.ex index 4288f74b5..b774798ba 100644 --- a/lib/plausible/purge.ex +++ b/lib/plausible/purge.ex @@ -5,6 +5,30 @@ defmodule Plausible.Purge do Stats are stored on Clickhouse, and unlike other databases data deletion is done asynchronously. + All import tables have MergeTree's deduplication mechanism _disabled_ by setting + `replicated_deduplication_window` from default 100 to 0. When enabled, every insert + into a given table is compared against hashes of 100 previous inserts (as complete + parts, not concrete rows) and ignored when match is found. The prupose of that + mechanism is making inserts of exact same batches idempotent when retrying them + shortly after - for instance due to timeout, when the client can't easily tell if + previous insert succeeded or not. Deduplication, however, only considers inserts, + not mutations. Deletions do not affect stored hashes, so further inserts of parts + that were deleted will still be treated as duplicates. That's why this feature + is disabled for import tables. + + Although deletions are asynchronous, the parts to delete are "remembered", so there's + no risk of overlapping deletion causing problems with import following right after it. + + IMPORTANT: Deletion requires revision if/when import tables get moved to sharded CH + cluster setup. Mutation queries, which have to be run with `ON CLUSTER` in such setup, + dispatch independent queries across shards and those queries can start at different + times. This in turn means risk of deletions corrupting data of follow-up inserts + in some edge cases. Ideally, imported entries should be unique for a given import + - an extra `import_id` column can be introduced, holding identifier. Last processed + import identifier should be stored with other site data and should be used for scoping + imported stats queries. No longer used imports can then be safely removed fully + asynchronously. + - [Clickhouse `ALTER TABLE ... DELETE` Statement](https://clickhouse.com/docs/en/sql-reference/statements/alter/delete) - [Synchronicity of `ALTER` Queries](https://clickhouse.com/docs/en/sql-reference/statements/alter/#synchronicity-of-alter-queries) """