Commit Graph

181 Commits (531dfb114bbdeb536751d7b12914fbf90ca40fdf)

Author SHA1 Message Date
Adam Rutkowski 8f85b110aa
Split Clickhouse pools into Read-Only and Read/Write (dedicated to writes) (#2661)
* Configure ingest repo access/pool size

If I'm not mistaken 3 is a sane default, the only
inserts we're doing are:

  - session buffer dump
  - events buffer dump
  - GA import dump

And all are serializable within their scopes?

* Add IngestRepo

* Start IngestRepo

* Use IngestRepo for inserts

* Annotate ClickhouseRepo as read_only

So no insert* functions are expanded

* Update moduledoc

* rename alias

* Fix default env var value so it can be casted

* Use IngestRepo for migrations

* Set default ingest pool size from 3 to 5

in case conns are restarting or else...

* Ensure all Repo prometheus metrics are collected
4 years ago
ruslandoga 7b2f4c99ee
Support alternative mailing services (Mailgun, Mandrill, Sendgrid) (#2649)
* more bamboo adapters

* add changelog

* add tests
4 years ago
Cenk Kücük f6ee17a400
Use hostname for server_name (#2642) 4 years ago
Adam Rutkowski 8f9f032968
Delay stats deletions (#2632)
* Implement Site removal transaction

* Implement Stats removal Oban worker

* Configure site removal queue

* Call Site.Removal.run() instead of Purge.delete_site!

* Test site/stats removal

* Remove FIXME - filed a ticket

* Over-communicate lenghty deletion process to the users
4 years ago
Adam Rutkowski ad12e1ef31
Show user feedback form on server errors (#2617)
* Move Endpoint errors setup to common config

* Implement naive Sentry link resolver

* Implement error report e-mail

* Delete static sentry script

* Implement user feedback form on server errors

* Re-arrange pipe

* Use Sentry.Config.dsn() where applicable

* Fix typo

* Use Map.replace/3
4 years ago
ruslandoga 166748dcf2
Replace Geolix with Locus (#2362)
This PR replaces geolix with locus to simplify self-hosted setup. locus can auto-update maxmind dbs which are recommended for self-hosters if they want city-level geolocation. locus is also a bit faster.

This PR also uses a test mmdb file from https://github.com/maxmind/MaxMind-DB for e2e geolocation tests without stubs.
4 years ago
Uku Taht 1785653b1e
Ignore unknown countries (#2556)
* Ignore XX and T1 countries

* Add fallback if country_code=nil

* Lookup city overrides directly in CityOverrides module

* Changelog

* Add empty moduledoc

* Remove redundant comment
4 years ago
Adam Rutkowski 5de43b758d
Run tests in async mode where applicable (#2542)
* Set pg pool size for MIX_ENV=test

* Include slow tests in CI run

* Exclude slow tests by default

* Mark tests slow/async where applicable

* Restructure captcha mocks

* Revert async where env is relied upon

* Add --max-failures=1 to CI run

* Set warnings as errors

* Disable async where various mocks are used

* Revert "Disable async where various mocks are used"

This reverts commit 2446b72a29.

* Disable async for test using vcr
4 years ago
ruslandoga 138e7c06d6
add BUILD_METADATA fallback when parsing (#2503)
### Changes

This PR adds a fallback to empty build metadata when BUILD_METADATA
contains invalid JSON.

Example `warning` log for `BUILD_METADATA={...}`:

```
20:57:57.872 [warning] failed to parse $BUILD_METADATA, reason: ** (Jason.DecodeError) unexpected byte at position 1: 0x2E (".")
```

Fixes https://github.com/plausible/analytics/issues/2491

### Tests
- [x] This PR does not require tests

### Changelog
- [ ] Entry has been added to changelog

### Documentation
- [x] This change does not need a documentation update

### Dark mode
- [x] This PR does not change the UI
4 years ago
Adam Rutkowski 356575ef78
Gatekeep ingestion pipeline (#2472)
* Update Sites.Cache

So it's now capable of refreshing most recent sites.
Refreshing a single site is no longer wanted.

* Introduce Warmer.RecentlyUpdated

This is Sites Cache warmer that runs only for
most recently updated sites every 30s.

* Validate Request creation early

* Rename RateLimiter to GateKeeper and introduce detailed policies

* Update events API tests - a provisioned site is now required

* Update events ingestion tests

* Make limits visible in CRM Sites index

* Hard-deprecate DOMAIN_BLACKLIST

* Remove unnecessary clause

* Fix typo

* Explicitly delegate Warmer.All

* GateKeeper.allwoance => GateKeeper.check

* Instrument Sites.Cache measurments

* Update send_pageview task to output response headers

* Instrument ingestion pipeline

* Credo

* Make event telemetry test a sync case

* Simplify Request.uri/hostname handling

* Use embedded schema, apply action and rely on get_field
4 years ago
Adam Rutkowski 457a558471
Kick off sites by domain cache implementation (#2434)
* Implement sites by domain caching interface + warmer

* Add test

* Implement hit rate interface

* Add moduledocs

* Fix up typespec

* s/warmer/warmer_fn

* Extract measure_duration/2

* Fix up typespec

* Log errors and return nil on cache internal errors

* Fix up non-existing cache test

* Retrieve specific db columns when pre-filling the cache

* Reduce the subset of fields retrieved from the DB

See 63f3c6233d (r89871536)
4 years ago
ruslandoga 0b7870dc4d
improve first launch experience for self-hosters (#2357)
* first launch

* dynamic children, wait for repo

* remove wait_for_repo and app env manipulations

* don't mention free trial in self-hosted pages

* add changelog

* assigns[:is_selfhost] -> @is_selfhost

* better changelog wording

* rm admin_user, admin_email, admin_pwd from app env

* rm DISABLE_AUTH

* redirect / to /login when not authenticated

* remove TODO

* Update lib/plausible_web/controllers/page_controller.ex

Co-authored-by: Uku Taht <Uku.taht@gmail.com>

* format

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
4 years ago
Adam Rutkowski 101e5a68b5
Allow Site DB lookups during ingestion phase (#2408)
* Implement FF-driven DB lookup for sites during ingestion

We like to see the impact of doing a simple postgres lookup on each
ingestion event. The percentage-based feature flag `:ingestion_pg_lookup`
must be set in order for lookups to be executed.

* Fix resolving Cachex stats metrics

* Enable PromEx on dev env
4 years ago
Vinicius Brasil b898642373
Double maximum header length (#2353)
This commit makes the permitted header length more permissive, 8,192
bytes, doubling the Phoenix default.

Related to https://github.com/4lejandrito/next-plausible/issues/67
4 years ago
Vinicius Brasil 9220d0034d
OpenTelemetry (OTEL) Implementation (#2317)
This pull request improves the current OpenTelemetry implementation. Currently only 1% of the spans are sent, due to the high volume of ingestion requests to /api/event. I enabled the 1% sampling to /api/event only, recording 100% of the other traces.
4 years ago
Adam Rutkowski e3ca3b32db
Include tests for Captcha success/failure scenarios (#2344)
* Include tests for Captcha success/failure scenarios

* DRY
4 years ago
RobertJoonas c0da024b23
Remove static tracker files (#2116)
* remove tracker files from git index

* generate tracker files on npm test

* generate tracker files for elixir tests/dev/CI

* update tracker/package-lock.json

* exclude npm run deploy from mix test + some docs
4 years ago
Uku Taht e849e03058
Fix favicons (#2257) 4 years ago
Adam Rutkowski 3f7c1ce549
Aggregate DBConnection.ConnectionError in Sentry (#2260) 4 years ago
Uku Taht e373799b01 Move fun_with_flags config from runtime.exs to config.exs
Getting this error when running the release:

ERROR! the application :fun_with_flags has a different value set for key :persistence during runtime compared to compile time. Since this application environment entry was marked as compile time, this difference can lead to different behaviour than expected:

  * Compile time value was not set
  * Runtime value was set to: [adapter: FunWithFlags.Store.Persistent.Ecto, repo: Plausible.Repo]
4 years ago
Uku Taht 3d54b88f0a
Make Finch pools lighter for self-hosting (#2250)
* Make Finch pools lighter

* Use standard http1 Finch pools
4 years ago
Vinicius Brasil d31db86b49
List all Google Analytics views during import (#2184)
* List all Google Analytics views during import

This commit fixes a bug where different Google Analytics views with the
same name and URI were not shown. This was caused because GA views were
stored as a map, that naturally doesn't support duplicate keys.

This change updates the GA views list to display view IDs, making it
clearer to know what is being imported. The dropdown is now grouped by
website URL.

* Put Google Analytics API URLs in app env

* Add controller test to GA view list
4 years ago
Vinicius Brasil 4d20c7ce70
Catch Google Search Console grant error (#2101)
* Remove invalid Jason.decode argument

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

* Add custom message to Google invalid grant error

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

* Test invalid_grant while refreshing Google token

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
4 years ago
Uku Taht 5c83ea77de Remove cache reporting to logs 4 years ago
Vinicius Brasil 4b9032d822
Google Analytics Import Refactor (#2046)
* Create separate module for GA HTTP requests

* Fetch GA data entirely instead of monthly

* Add buffering to GA imports

* Change positional args to maps when serializing from GA

* Create Google Analytics VCR tests
4 years ago
Vinicius Brasil b415ebe776
Fix geolocation subdivision pattern matching (#2063)
* Fix geolocation subdivision pattern matching

This commit fixes a bug where regions were not being saved. This was
caused because Geolix response was returning an additional
`:geolocation` map key. It also adds a test case for this.

Closes #2033

* Add geolocation database to .gitignore
4 years ago
Weslei Juan Novaes Pereira 0324d03da9
fix: Oban pruner max_age config (#2032) 4 years ago
Uku Taht 6fbb0a24a8 Do not log Sentry.CrashError to Sentry
Stops recursive error logging to sentry
4 years ago
Adam Rutkowski 3b82ba0e25
Upgrade to Geolix 2.0 (#1997)
* Upgrade geolix

* Remove geolix pool config

* Save unnecessary Task.async_stream roundtrip

Normally the Geolix API accepts `:where` keyword option that designates
the database to look up. In case no parameter is supplied, it'll spawn
a parallel map over all databases available. In this case we have only
one DB anyway, so there is no need for the extra instrumentation.

* Follow up on direct :geolocation lookups
4 years ago
Manu S Ajith 81f18ff0a5
Setup promex (#1999)
* Setup promex

Signed-off-by: Manu S Ajith <neo@codingarena.in>

* Cleanup promex config file

Signed-off-by: Manu S Ajith <neo@codingarena.in>
4 years ago
Uku Taht 2b8e3ea62a
Use finch in sentry client (#1996)
* Introduce Finch for Sentry integration

* Make sure the DummyAgent can be started

* No need to sanitize the dsn, finch takes care of that

* Simplify the dummy child spec

* Annotate redirects clause

* Make use of new `get_int_from_path_or_env`

* Actually use finch in Sentry config

* Configure `excluded_domains` correctly for Sentry

The way sentry is configured currently, when we get an HTTP error it
will be logged twice - once from Sentry.PlugCapture and once from
Sentry.LoggerBackend. The logger backend module does the right thing
by default but for some reason we've been overriding the config
parameter that by default stops double-counting errors. This commit
returns to the default configuration which is better.

* Default to 15s timeout

* Attempt to send twice at most

* Warn in sentry client

* Use warn level in sentry client

Co-authored-by: Adam Rutkowski <hq@mtod.org>
4 years ago
Uku Taht ac89d60808 Add sample rate to sentry config 4 years ago
Uku Taht 0553fa041b Parse geolix pool config as integers 4 years ago
Manu S Ajith 606c162138
Add option to configure sentry pool size, and geolix worker size (#1992)
Signed-off-by: Manu S Ajith <neo@codingarena.in>
4 years ago
Adam Rutkowski 45cc1d27a1
Fix dev environment startup errors (#1990)
* Include gelocation DB download in the development workflow

* Make sure `tls_certificate_check` is started ASAP

This prevents `:application_either_not_started_or_not_ready` errors
on application startup.

* Mark Makefile targets as PHONY

By default Make assumes the targets are files,
in this case none of them are.
4 years ago
Uku Taht 910efd849c Revert config changes 5 years ago
Uku Taht b667d65d52 Move ARG to running container instead of build container 5 years ago
Uku Taht d23f7d5358 Disable sentry if not configured 5 years ago
Uku Taht da93f2aa6e Remove dead code 5 years ago
Uku Taht 18e2711556 Package new db-ip library in the git repo 5 years ago
Vignesh Joglekar b7b69c6f62
Adds "invite_only" disable_registration config option (#1841)
* Adds tri-state disable_registration config

* Formatting

* Changes variable back to atom

* Changelog

* Uses atoms correctly :/

* Swaps to a more fitting value

* Formatting
5 years ago
Uku Taht e23cbfcb46 Only nodes that run cron should be elected as leader 5 years ago
Uku Taht f18a211dcc
Ingest throughput improvement test setup (#1867)
* Add OTEL and test Cachex for sessions

* Move load test

* Start apps in the appropriate order
5 years ago
Uku Taht 7c1d64458e Add fun with flags library 5 years ago
Uku Taht 680bd98bd1 Fix logic 5 years ago
Uku Taht a282478838 Update cron config 5 years ago
Uku Taht 83c407c016
Upgrade Oban & configure Stager plugin (#1822) 5 years ago
Uku Taht 06b165eb6d Run GA import in monthly batches 5 years ago
Uku Taht ae78444830 Add notice about feature preview 5 years ago
Uku Taht 4cc4e0d61b Add config flag for import testers 5 years ago