* Reapply "Replace caching engine (#3878)" (#3883)
This reverts commit c5881cdc6d.
* Ensure hit rate is tracked on `get_or_store`
* Remove :wx and :observer
* Remove unused deps
* Use `:set` table type
* Dependencies: swap Cachex for ConCache
* Implement Cache adapter wrapping ConCache
* Implement cache stats tracker, for metrics
* Use Cache.Adapter in Plausible.Cache
Marking the test as not slow anymore
* Use Cache Adapter when tracking sessions
* Use Cache Adapter for UA parsing
* Rename child identifiers - cachex is obsolete now
* Test stats tracking
* Update grafana metrics
* Put all caches under common child specification
* Try less
* Shorten the function delegation path
* Migration: add country rules
* Add CountryRule schema
* Implement CountryRule cache
* Add country rules context interface
* Start country rules cache
* Lookup country rules on ingestion
* Remove :shields feature flag from test helpers
* Add nested sidebar menu for Shields
* Fix typo
* IP Rules: hide description on mobile view
* Prepare SiteController to handle multiple shield types
* Seed some country shield
* Implement LV for country rules
* Remove "YOU" indicator from country rules
* Fix small build
* Format
* Update typespecs
* Make docs link point at /countries
* Fix flash on top of modal for Safari
* Build the rule struct with site_id provided up-front
* Clarify why we're messaging the ComboBox component
* Re-open combobox suggestions after pressing Escape
* Update changelog
* Fix font size in country table cells
* Pass `added_by` via rule add options
* Display site's timezone timestamps in rule tooltips
* Display formatted timestamps in site's timezone
And simplify+test Timezone module; an input timestamp converted
to UTC can never be ambiguous.
* Remove no-op atom
* Display the maximum number of rules when reached
* Improve readability of remove button tests
* Credo
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Replace footer text
* add COPYING.txt file
* Add new logos
* Use new logos in all layouts
* New logos
* Check license key on startup
* Bypass license check when Mix.env == :dev
* Use new logos with smaller wordmarks
* Add generic logo_path/1 function
* Use new favicons everywhere
* Bypass license check in test env
* Use sha256 for license key hash
* Mix.env -> config_env()
* Use Mix.evn at compile time rather than runtime
* Mix format
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* Add Ecto.Network dependency
* Migration: Add ip block list table
* If Cachex errors out, mark the cache as not ready
* Add IPRule schema
* Seed IPRules
* Add Shields context module
* Implement IPRuleCache
* Start IPRuleCache
* Drop blocklisted IPs on ingestion
* Cosmetic rename
* Add settings sidebar item
* Consider IPRuleCache readiness on health checks
* Fix typo
* Implement IP blocklist live view
* Update moduledocs
* Extend contextual module tests
* Convert IPRules LiveView into LiveComponent
* Keep live flashes on the tabs view
* Update changelog
* Format
* Credo
* Remove garbage
* Update drop reason typespecs
* Update typespecs for cache keys
* Keep track of who added a rule and when
* Test if adding via LV prefills the updated_by tooltip
* Update ecto_network dependency
* s/updated_by/added_by
* s/drop_blocklist_ip/drop_shield_rule_ip
* Add docs link
* s/Updated/Added
* Simplify caches config
The intervals were proven to never change really,
and previous shape made them not very discoverable either.
We'll look into a more declarative setup.
* Aim for more declarative supervision tree setup
* Generalize `Plausible.Cache`
This makes the `Site.Cache` module implement the specifics,
leaving the common bits reusable for upcoming cache processes.
* Generalize `Plausible.Cache` warmers
* Fix typos
* Bump deps
* Bump stack
* Fix deprecation warnings
* Fix VCR cassettes mismatch due to OTP-18414
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Format & fix flaky tests
* Handle raw IPv4 hostnames; test public suffix TLD
* Configure locus db cache_dir
So that maxmind unavailability doesn't affect
application startup. PERSISTENT_CACHE_DIR env var is used
to point locus at the GeoIP DB file.
* WIP: Remove ExVCR
* Fix test env config
* Fixup exvcr
* Remove exvcr from deps
* Add convert script
* Remove exvcr cassettes
* Remove convert script
* Rename test
* Update moduledoc
* Update dockerfile
* Bump CI cache
* Tag more slow tests, why not?
* Use charlist for locus cache option
* Pin nodejs
* Merge google tests, make them async
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Clickhouse migration: add ingest_counters table
* Configure ingest counters per MIX_ENV
* Emit telemetry for ingest events with rich metadata
* Allow building Request.t() with fake now() - for testing purposes
* Use clickhousex branch where session_id is assigned to each connection
* Add helper function for getting site id via cache
* Add Ecto schema for `ingest_counters` table
* Implement metrics buffer
* Implement buffering handler for `Plausible.Ingestion.Event` telemetry
* Implement periodic metrics aggregation
* Update counters docs
* Add toStartOfMinute() to ordering key
* Reset the sync connection state in `after` clause
* Flush counters on app termination
* Use separate Repo with async settings enabled at config level
* Switch to clickhouse_settings repo root config key
* Add AsyncInsertRepo module
* Configure ingest repo access/pool size
If I'm not mistaken 3 is a sane default, the only
inserts we're doing are:
- session buffer dump
- events buffer dump
- GA import dump
And all are serializable within their scopes?
* Add IngestRepo
* Start IngestRepo
* Use IngestRepo for inserts
* Annotate ClickhouseRepo as read_only
So no insert* functions are expanded
* Update moduledoc
* rename alias
* Fix default env var value so it can be casted
* Use IngestRepo for migrations
* Set default ingest pool size from 3 to 5
in case conns are restarting or else...
* Ensure all Repo prometheus metrics are collected
* Move Endpoint errors setup to common config
* Implement naive Sentry link resolver
* Implement error report e-mail
* Delete static sentry script
* Implement user feedback form on server errors
* Re-arrange pipe
* Use Sentry.Config.dsn() where applicable
* Fix typo
* Use Map.replace/3
This PR replaces geolix with locus to simplify self-hosted setup. locus can auto-update maxmind dbs which are recommended for self-hosters if they want city-level geolocation. locus is also a bit faster.
This PR also uses a test mmdb file from https://github.com/maxmind/MaxMind-DB for e2e geolocation tests without stubs.
### Changes
This PR:
- pushes PromEx to the bottom of supervision stack to avoid Endpoint
instrumentation failure
- ensures the site cache is ready by exposing it through the health
check endpoint
- fixes event timestamps being calculated at compile time, with
regression unit and integration tests
### Tests
- [x] Automated tests have been added
- [ ] This PR does not require tests
### Changelog
- [ ] Entry has been added to changelog
- [x] This PR does not make a user-facing change
### Documentation
- [ ] [Docs](https://github.com/plausible/docs) have been updated
- [x] This change does not need a documentation update
### Dark mode
- [ ] The UI has been tested both in dark and light mode
- [x] This PR does not change the UI
* Update Sites.Cache
So it's now capable of refreshing most recent sites.
Refreshing a single site is no longer wanted.
* Introduce Warmer.RecentlyUpdated
This is Sites Cache warmer that runs only for
most recently updated sites every 30s.
* Validate Request creation early
* Rename RateLimiter to GateKeeper and introduce detailed policies
* Update events API tests - a provisioned site is now required
* Update events ingestion tests
* Make limits visible in CRM Sites index
* Hard-deprecate DOMAIN_BLACKLIST
* Remove unnecessary clause
* Fix typo
* Explicitly delegate Warmer.All
* GateKeeper.allwoance => GateKeeper.check
* Instrument Sites.Cache measurments
* Update send_pageview task to output response headers
* Instrument ingestion pipeline
* Credo
* Make event telemetry test a sync case
* Simplify Request.uri/hostname handling
* Use embedded schema, apply action and rely on get_field
* Implement sites by domain caching interface + warmer
* Add test
* Implement hit rate interface
* Add moduledocs
* Fix up typespec
* s/warmer/warmer_fn
* Extract measure_duration/2
* Fix up typespec
* Log errors and return nil on cache internal errors
* Fix up non-existing cache test
* Retrieve specific db columns when pre-filling the cache
* Reduce the subset of fields retrieved from the DB
See 63f3c6233d (r89871536)
This pull request improves the current OpenTelemetry implementation. Currently only 1% of the spans are sent, due to the high volume of ingestion requests to /api/event. I enabled the 1% sampling to /api/event only, recording 100% of the other traces.
* Add Custom telemetry for Plausible.Event.WriteBuffer, Plausible.Event.WriteBuffer and Cachex
Signed-off-by: Manu S Ajith <neo@codingarena.in>
* Rename telemetry.ex to avoid confusion with Phoenix Telemetry supervisor
Signed-off-by: Manu S Ajith <neo@codingarena.in>
* Remove duplicate event
Signed-off-by: Manu S Ajith <neo@codingarena.in>
Signed-off-by: Manu S Ajith <neo@codingarena.in>
* Introduce Finch for Sentry integration
* Make sure the DummyAgent can be started
* No need to sanitize the dsn, finch takes care of that
* Simplify the dummy child spec
* Annotate redirects clause
* Make use of new `get_int_from_path_or_env`
* Actually use finch in Sentry config
* Configure `excluded_domains` correctly for Sentry
The way sentry is configured currently, when we get an HTTP error it
will be logged twice - once from Sentry.PlugCapture and once from
Sentry.LoggerBackend. The logger backend module does the right thing
by default but for some reason we've been overriding the config
parameter that by default stops double-counting errors. This commit
returns to the default configuration which is better.
* Default to 15s timeout
* Attempt to send twice at most
* Warn in sentry client
* Use warn level in sentry client
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Plugin exceptions may be silently covering up database problems such
as failed cron job inserts.
* The previously tracked `:circuit` event is no longer emitted.
* Use clickhouse-ecto for stats
* Use clickhouse ecto instead of low-level clickhousex
* Remove defaults from event schema
* Remove all references to Clickhousex
* Document configuration change
* Ensure createdb and migrations can be run in a release
* Remove config added for debug
* Update plausible_variables.sample.env
* Add daily salt rotation for user ids
* Rotate salts on a cron schedule daily
* Keep old salt fo 48 hours
* Double write for plausible.io to test salt rotation
* Rotate salts for every site