|
|
|
|
@ -5,6 +5,30 @@ defmodule Plausible.Purge do
|
|
|
|
|
Stats are stored on Clickhouse, and unlike other databases data deletion is
|
|
|
|
|
done asynchronously.
|
|
|
|
|
|
|
|
|
|
All import tables have MergeTree's deduplication mechanism _disabled_ by setting
|
|
|
|
|
`replicated_deduplication_window` from default 100 to 0. When enabled, every insert
|
|
|
|
|
into a given table is compared against hashes of 100 previous inserts (as complete
|
|
|
|
|
parts, not concrete rows) and ignored when match is found. The prupose of that
|
|
|
|
|
mechanism is making inserts of exact same batches idempotent when retrying them
|
|
|
|
|
shortly after - for instance due to timeout, when the client can't easily tell if
|
|
|
|
|
previous insert succeeded or not. Deduplication, however, only considers inserts,
|
|
|
|
|
not mutations. Deletions do not affect stored hashes, so further inserts of parts
|
|
|
|
|
that were deleted will still be treated as duplicates. That's why this feature
|
|
|
|
|
is disabled for import tables.
|
|
|
|
|
|
|
|
|
|
Although deletions are asynchronous, the parts to delete are "remembered", so there's
|
|
|
|
|
no risk of overlapping deletion causing problems with import following right after it.
|
|
|
|
|
|
|
|
|
|
IMPORTANT: Deletion requires revision if/when import tables get moved to sharded CH
|
|
|
|
|
cluster setup. Mutation queries, which have to be run with `ON CLUSTER` in such setup,
|
|
|
|
|
dispatch independent queries across shards and those queries can start at different
|
|
|
|
|
times. This in turn means risk of deletions corrupting data of follow-up inserts
|
|
|
|
|
in some edge cases. Ideally, imported entries should be unique for a given import
|
|
|
|
|
- an extra `import_id` column can be introduced, holding identifier. Last processed
|
|
|
|
|
import identifier should be stored with other site data and should be used for scoping
|
|
|
|
|
imported stats queries. No longer used imports can then be safely removed fully
|
|
|
|
|
asynchronously.
|
|
|
|
|
|
|
|
|
|
- [Clickhouse `ALTER TABLE ... DELETE` Statement](https://clickhouse.com/docs/en/sql-reference/statements/alter/delete)
|
|
|
|
|
- [Synchronicity of `ALTER` Queries](https://clickhouse.com/docs/en/sql-reference/statements/alter/#synchronicity-of-alter-queries)
|
|
|
|
|
"""
|
|
|
|
|
|