Commit Graph

160 Commits (con-cache-patch-test)

Author SHA1 Message Date
ruslandoga 279e89c693
CSV imports (no UI) (#3895)
* encode/decode date range in filenames

* Update lib/plausible/imported/csv_importer.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible/imported/csv_importer.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* drop unused functions

* send failure email if there is no data to export

* use PlausibleWeb.Email.mailer_email_from()

* ensure we get dates from minmax date query

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
ruslandoga 07b714a143
Update Sentry (#3843)
* update Sentry

* Sentry.HTTPClient.child_spec is now optional

* Sentry.EventFilter is deprecated

* update sentry to 10.2.0

* fix dialyzer warnings
3 years ago
ruslandoga c08aa29ad8
Remove DISABLE_AUTH deprecation warning (#3904)
* remove DISABLE_AUTH deprecation warning

* changelog
3 years ago
ruslandoga 5e74b1cf74
CSV exports (UI) (#3875)
* ui

* fix redirect link

* improve make minio

* use implicit button form for csv export

* add exports_bucket helper

* read S3_EXPORTS_BUCKET

* supply s3_bucket in export_csv job args

* make plausible_minio use unprotected port

* move s3_csv_export queue to base queues

* Update lib/plausible_web/controllers/site_controller.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
ruslandoga 5a3072ca21
CSV exports (no UI) (#3836)
* csv exports

* use ex_unit's tmp_dir
3 years ago
Cenk Kücük dea9287d74
Revert "Temporarily disable tzdata auto-updating (#3873)" (#3885)
This reverts commit 11368e5143.
3 years ago
ruslandoga 83a46fb47e
Dockerfile and co updates (#3811)
* wip

* more env, setup user before packages to avoid them picking our uid

* make entrypoint.sh executable by all again

* apparently it's a best practice to have executables be owned by root

* make entrypoint executable in COPY

* stop writing to /app, write to /tmp if needed

* fewer changes

* system user

* keep same style for multiline commands

* fewer changes

* add changelog entry

* fix group assignment for plausible user

* use gid=999

* no home

* no home

* add gecos

* add plausible user to nogroup instead of creating a custom one

* eh

* fewer changes

* fewer changes

* fewer changes

* use PERSISTENT_CACHE_DIR instead of STORAGE_DIR

* ignore more

* cleanup

* remove hex timeout env var

* use ERL_FLAGS=+JMsingle true in public builds

* fallback to /tmp and nest under /tzdata_data/ for tzdata

---------

Co-authored-by: Cenk Kücük <cenk@plausible.io>
3 years ago
Karl-Aksel Puulmann 11368e5143
Temporarily disable tzdata auto-updating (#3873)
This can cause issues as 1 day after a deploy the tzdata list is updated
from 2021 to 2024. Selecting a timezone thats not supported in 2021 can
then break dashboards temporarily following a deploy.

This can be reverted after https://github.com/plausible/analytics/pull/3811
3 years ago
ruslandoga f3423aefec
Add csv importer (#3795)
* add csv importer

* make table validation explicit

* update some docs

* improve docs

* add minio container to ci

* more tests

* eh

* continue

* add passing test

* add failing test

* add config test

* add minio to Makefile

* testcontainers

* remove extra whitespace

* explain the implementation a bit

* account for async deletes in tests

* bounces is UInt32

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
ruslandoga 31cf3e54f8
Add Bamboo.Mua (#3654) 3 years ago
Cenk Kücük 0f0f604d2e
Branding stuff (#3816)
* Replace footer text

* add COPYING.txt file

* Add new logos

* Use new logos in all layouts

* New logos

* Check license key on startup

* Bypass license check when Mix.env == :dev

* Use new logos with smaller wordmarks

* Add generic logo_path/1 function

* Use new favicons everywhere

* Bypass license check in test env

* Use sha256 for license key hash

* Mix.env -> config_env()

* Use Mix.evn at compile time rather than runtime

* Mix format

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
3 years ago
Karl-Aksel Puulmann ac7da6a9d4
Set materialized_views_ignore_errors (#3784)
We had an incident where a stray materialized view caused inserts to fail. This setting
causes stray mv errors to silently fail, which is okay for our current usecases.

Further reading: https://clickhouse.com/docs/en/sql-reference/statements/create/view#materialized-view
3 years ago
Adrian Gruntkowski f8b4d5066a
Add multiple imports per site (#3724)
* Clean up references to no longer active `google_analytics_imports` Oban queue

* Stub CSV importer

* Add SiteImport schema

* Rename `Plausible.Imported` module file to match module name

* Add `import_id` column to `Imported.*` CH schemas

* Implement Importer behavior and manage imports state using new entities

* Implement importer callbacks and maintain site.imported_data for UA

* Keep imports in sync when forgetting all imports

* Scope imported data queries to completed import IDs

* Mark newly imported data with respective import ID

* Clean up Importer implementation a bit

* Test querying legacy and new imported data

* Send Oban notifications on import worker failure too

* Fix checking for forgettable imports and remove redundant function

* Fix UA integration test

* Change site import source to atom enum and add source label

* Add typespecs and reduce repetition in `Plausible.Imported`

* Improve documentation and typespecs

* Add test for purging particular import

* Switch email notification templates depending on import source

* Document running import synchronously

* Fix UA importer args parsing and ensure it's covered by tests

* Clear `site.stats_start_date` on complete import to force recalculation

* Test Oban notifications (h/t @ruslandoga)

* Purge stats on import failure right away to reduce a chance of leaving debris behind

* Fix typos

Co-authored-by: hq1 <hq@mtod.org>

* Fix another typo

* Refactor fetching earliest import and earliest stats start date

* Use `Date.after?` instead of `Timex.after?`

* Cache import data in site virtual fields and limit queried imports to 5

* Ensure always current `stats_start_date` is used

* Work around broken typespec in Timex

* Make `SiteController.forget_imported` action idempotent

* Discard irrecoverably failed import tasks

* Use macros for site import statuses

There's also a fix ensuring only complete imports are considered
where relevant - couldn't isolate it as it was in a common hunk

* Use `import_id` as worker job uniqueness criterion

* Do not load imported stats data in plugins API context

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
Karl-Aksel Puulmann 625b2a2946
Use parallel_hash for joins in clickhouse (#3741) 3 years ago
hq1 18ddf1e7a0
Re enable accept traffic until notifications (#3737)
* Revert "Remove 0-sized queue"

This reverts commit 6d229b3f70.

* Revert "Temporarily remove notification cron worker (#3732)"

This reverts commit 447e3929cb.

* Revert "Temporarily disable accept_traffic_until actions (#3730)"

This reverts commit 58afe3376f.

* Send accept_traffic_until notifications at 8 UTC rather than 5

* Add option for dry-run

* Expose dry_run in a dedicated function
3 years ago
Adam Rutkowski 6d229b3f70 Remove 0-sized queue 3 years ago
hq1 447e3929cb
Temporarily remove notification cron worker (#3732)
* Temporarily remove notification cron worker

* Format
3 years ago
hq1 58afe3376f
Temporarily disable accept_traffic_until actions (#3730)
* Temporarily disable accept_traffic_until actions

* Remove commented code
3 years ago
Adrian Gruntkowski 822483c37c
Extract Universal Analytics import logic (#3700)
* Move imported tables schemas to separate modules outside Google ns

* Move buffer for imports to Imported ns

* fix schema newlines

* Extract UA import processing and persistence

* Decouple analytics worker implementation from UA

* Rename env variable for import buffer size

* Preserve old import queue until release
3 years ago
hq1 32456d4348
Upgrade stack (deps, elixir 1.16, OTP 26.2.1) (#3678)
* Bump asdf erlang & elixir

* Bump erlang/elixir in the dockerfile

* Remove Oban.Stager config as per https://github.com/sorentwo/oban/blob/main/guides/upgrading/v2.14.md

* Configure Oban for tests as per https://github.com/sorentwo/oban/blob/main/guides/upgrading/v2.14.md

* Mark sampling hint clause with unsafe fragment

Any other/recommended way to do that? cc @ruslandoga

* Address String.slice/2 deprecation

* Update deps

* Address 0.0 matching warning

* Make funnel settings work

although this is probably not the best way to do it.
Needs revisiting, not sure what broke it - could not find
any breaking changes in related dependencies.

* Migrate oban as per https://github.com/sorentwo/oban/blob/main/guides/upgrading/v2.17.md

* Update credo

* Bump CI cache

* Use `Bypass.pass` to avoid exit shutdown message
3 years ago
hq1 f755b20569
Lock traffic notifications (#3641)
* Update communication

* Remove an unreachable function (mistyped)

* [migration] Make accept_traffic_until a date

* Fix typo

* Set `accept_traffic_until` when creating a site

* Update sites `accept_traffic_until` on subscription change

* Add a note to yearly cancellation notification

* Rephrase annual e-mail for clarity

* Pass the small build test

* Add email notifications

* Fixup

* Implement `accept_traffic_until` notification worker

* Fixup - no need to test this for small build

* Update moduledoc

* Move moduletag

* s/sent_at/sent_on

* Use WHERE NOT EXISTS instead of LEFT JOIN

* Use upsert when tracking notifications sent

* Store sent marker before actually sending notification

* Prefer to keep `accept_traffic_until` on the user record

This gives us a single source of truth, addresses cases like
ownership transparently, simplifies the code and enables CRM toggles.
The only downside is that there's another join performed in the
Sites.Cache full refresh - in this case, small refreshes are
skipped - but this is fine, since the traffic will be let in
anyway.

* Expose `accepted_traffic_until` in the CRM

* Update lib/plausible/auth/user.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Preload owner in CRM

* Use the offset parameter in trial over e-mail contents

* Format

* Harden cache test

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
ruslandoga 6639d6af63
Flush Clickhouse buffers faster by encoding early (#3623)
* encode early

* rename: CLICKHOUSE_MAX_BUFFER_SIZE -> CLICKHOUSE_MAX_BUFFER_SIZE_BYTES

* deprecate CLICKHOUSE_MAX_BUFFER_SIZE

* cleanup

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
hq1 7b5c20db17
Revert "experiment in improving throughput of write buffers (#3511)" (#3614)
This reverts commit 16bfb11701.
3 years ago
Adam Rutkowski 69579272d6 Revert "dump buffer on invalid insert (#3613)"
This reverts commit 7a4cafac16.
3 years ago
ruslandoga 7a4cafac16
dump buffer on invalid insert (#3613)
* dump buffer on invalid insert

* dump to PERSISTENT_CACHE_DIR

* fix warning

* ensure dir is not nil
3 years ago
ruslandoga 16bfb11701
experiment in improving throughput of write buffers (#3511)
* wip

* fewer length/1 calls

* encode early

* cleanup

* add micro benchmark

* extract types at compilation

* remove specs to make credo happier

* cleanup is_bounce

* cleanup :D

* add eprof to buffer insert+flush benchmark

* add ci/bench.yml

* rm state.schema from write buffer

* rm noisy logs

* rm benches

* add :source to insert

* make CLICKHOUSE_MAX_BUFFER_SIZE mean bytes

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
ruslandoga 0eedf9aa98
rm Hammer (#3571) 3 years ago
hq1 c4eea9f8d5
`s/warn/warning` (`warn` is deprecated) (#3587) 3 years ago
hq1 a4b9c3b8ba
Remove custom domains support + update build options (#3559)
* Disable super-admin checks on small build

* Mute a test writing to stdout

* Move sampling outside of small build

* Convert waiting_first_pageview to heex and stop relying on env vars

* Set site limit unlimited on small build

* Stop relying on app env to get trial expiry

* Remove custom domains - including migration

* Remove is_selfhosted from layout view

* Quota fixup

* Stop relying on app env for self hosted registration

* Stop relying on app env for pass reset success

* Apply on_trial? check only on full build

* Update templates relying on app env

* Adjusts auth controller tests for small build

* Trial fixup

* Fixup

* Stop relying on app env

* Rest of the fsckn owl

* Update typespecs

* Fix dialyzer warning

* Remove unused module

* Credo + format

* GeoIP is not, for full build

* Use `small_build?()` where applicable

* Implement bypassing FirstLaunchPlug without insertions

* Get Marko's patch de58a18a85

* Test is-dbip=false presence

* Fix typespec

* Remove future hardcodes

* Handle `nil` from `Plausible.Geo.database_type()`

* Remove XXX marker

* Use one typespec for two clauses

* Introduce `MIX_ENV=small_dev`

* Revert "Use one typespec for two clauses"

This reverts commit 8d8cd21764.
3 years ago
Adrian Gruntkowski 65cc8980e0
Implement core logic for TOTP support (#3525)
* Add `nimble_totp`, `cloak` and `cloak_ecto` to project dependencies

* Setup Cloak-based secrets vault and create a dedicated Ecto type

* Add `totp_enabled|secret|last_used_at` fields to `User` schema

* Implement schema and stateless logic for TOTP recovery codes

* Implement core logic of TOTP auth

* Fix typos and improve style of doc comments

Co-authored-by: hq1 <hq@mtod.org>

* Fix moduledoc alignment

* Use more compact conditional expression

Co-authored-by: hq1 <hq@mtod.org>

* Disambiguate `I` as `7` when generating recovery codes (h/t @hq1)

* Fix a typo in runtime config error message

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
hq1 b9ec38038c
Add small build option (#3536)
* Update applications

* Clone community config

* Move modules to experimental dir

* Update runtime config

* Apply first set of compile-time conditionals

* Move funnel schemas to experimental

* Make funnel schema-less build compile

* Use experimental/lib for elixir code

* Move JS funnels to experimental

* Clean up conditional rendering

* Tidy up the pipeline

* Make two builds pass tests without warnings

* Reuse existing dotenvs

* Do a bunch of renames

* Clean up naming

* Run secondary CI

* Update router

* Remove RewriteFunnelDupes migration

Tests were disabled already and it was a one-off shot

* Fixup quota mixins

* Add moduledoc

* Change MIX_ENV for seconary test run

* Skip crm on small

* !fixup

* Exclude flags pipeline

* Update lib/plausible_web/controllers/stats_controller.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
Adrian Gruntkowski 3ba57a04fc
Make OpenTelemetry sampler ratio configurable via env (#3514) 3 years ago
hq1 671904aefd
Enable JSON log formatting (#3512)
* Enable JSON log formatting

if `LOG_FORMAT=json` is set, the app will start logging
JSON-formatted messages. By the way, the Repo :loggers configration
was removed since it's been already deprecated by Ecto v3
(https://hexdocs.pm/ecto/changelog.html#deprecations-4).

* Update changelog
3 years ago
hq1 117eef000d
Upgrade Erlang/Elixir stack (#3454)
* Bump deps

* Bump stack

* Fix deprecation warnings

* Fix VCR cassettes mismatch due to OTP-18414

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Format & fix flaky tests

* Handle raw IPv4 hostnames; test public suffix TLD

* Configure locus db cache_dir

So that maxmind unavailability doesn't affect
application startup. PERSISTENT_CACHE_DIR env var is used
to point locus at the GeoIP DB file.

* WIP: Remove ExVCR

* Fix test env config

* Fixup exvcr

* Remove exvcr from deps

* Add convert script

* Remove exvcr cassettes

* Remove convert script

* Rename test

* Update moduledoc

* Update dockerfile

* Bump CI cache

* Tag more slow tests, why not?

* Use charlist for locus cache option

* Pin nodejs

* Merge google tests, make them async

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
Uku Taht 97b24c0492
Nolt sso (along with a better nav dropdown) (#3395)
* Add SSO link with signed JWT token

* Falls back to Nolt URL without SSO if token cannot be generated

* Add profile image (gravatar) to Nolt SSO link

* Improve navbar dropdown

* Add 'contact support' link to nav dropdown

* Add CSS rule to prevent horizontal jumps

* Dark mode styling

* Close dropdown when link is clicked

* Clarify links in dropdown

* Clarify CSS comment

* Use Alpine.data() over window

* Rename suggestions_dropdown -> combo-box

* Mix format

* Make logout link look good on dark mode

* Use proxy for gravatar

* Do not use Gravatar proxy in self-hosted

* Changelog

* Add Github Repo link to nav dropdown

* Make dialyzer happy

* Add proxy for Gravatar

* Update assets/css/app.css

Co-authored-by: hq1 <hq@mtod.org>

* Update lib/plausible_web/controllers/avatar_controller.ex

Co-authored-by: hq1 <hq@mtod.org>

* Fix alpine <> Liveview integration

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
Adrian Gruntkowski 70c001099d
Improve and simplify email verification codes generation (#3407)
* Refactor email verification codes generation to avoid predictability

* Improve `Site.Memberships.any?` slightly

* Update tests

* Fix seeds

* Use `expired?` predicate for checking verification code validity in tests

* Store verification code as string in database to avoid unnecessary int casting
3 years ago
Cenk Kücük 4ae0c68397
verify tls connection to clickhouse and postgres (#3254) 3 years ago
hq1 b9fb6fd1a7
Revert "Revert "Reconfigure session cookie (#3274)" (#3284)" (#3285)
This reverts commit 65f7b4c5cc.
3 years ago
hq1 65f7b4c5cc
Revert "Reconfigure session cookie (#3274)" (#3284)
This reverts commit 63fabcbb5f.
3 years ago
Vini Brasil dad787cb4b
Rename allowance to monthly_pageview_limit (#3273)
* Remove unused site_limit_exempt list

* Rename allowance to monthly_pageview_limit
3 years ago
hq1 63fabcbb5f
Reconfigure session cookie (#3274)
* Reconfigure session cookie

* Allow configure secure cookie (#3277)

* Update config/runtime.exs

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* fix runtime conf

* Revert "fix runtime conf"

This reverts commit ff37e479e4.

* Revert "Update config/runtime.exs"

This reverts commit 5d9b310b02.

* Revert "Allow configure secure cookie (#3277)"

This reverts commit 7401a2ad3f.

* Read SECURE_COOKIE with defaults per cloud/selfhost

* Include environment in cookie name

* Remove redundant option

* Format

* s/Map.replace/Map.put

* up

* One more try

* Prevent browser refreshes on socket connection error

We'll keep the log and don't make the page look dumb
even if there's some misconfiguration going on.

---------

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
3 years ago
Vini Brasil 34f1ddfc8c
Refactor Plausible.Billing.Plans module (#3268)
This pull request introduces a series of improvements to Plausible.Billing.Plans, including:

* Tag the JSON file with the plan version
* Rename the JSON field limit to monthly_pageview_limit
* Move site_limit function to Billing.Plans
* Refactor subscription_interval, allowance and site_limit functions
* Remove unused AnalyzePlans task
3 years ago
risson 50fa1b059b
Allow optional IPv6 for clickhouse repo (#2970)
* Allow optional IPv6 for clickhouse repo

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* remove trailing comma

---------

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Co-authored-by: ruslandoga <rusl@n-do.ga>
3 years ago
hq1 eb397a6c25
Dynamically set up session domain (#3089)
* Dynamically configure session domain

* Fix up error message in runtime config
3 years ago
hq1 4ac83b6e5d
Reconfigure live websocket (#3087)
* Dynamically set :domain for the live socket

Ref: https://github.com/phoenixframework/phoenix_live_view/pull/2715

* Make runtime config raise before the user ends up in a reconnect loop

* hall of shame: remove console.info remnant

* Check origin on live websocket

* Get rid of single pipe
3 years ago
hq1 0c77b01a99
Allow configurable websocket URL for live view (#3082) 3 years ago
ruslandoga fd15853965
disable registration by default in self-hosted setups (#3014)
* disable registration by default in self-hosted setups

* add changelog entry

* add error flash
3 years ago
Vini Brasil e4d4f7d954
Revenue tracking: Ingestion and breakdown queries (#2957)
* Add revenue fields to ClickHouse events

This commit adds 4 fields to the ClickHouse events_v2 table:

* `revenue_source_amount` and `revenue_source_currency` store revenue in
  the original currency sent during ingestion

* `revenue_reporting_amount` and `revenue_reporting_currency` store
  revenue in a common currency to perform calculations, and this
  currency is defined by the user when setting up the goal

The type of amount fields is `Nullable(Decimal64(3))`. That covers all
fiat currencies and allows us to store huge amounts. Even though
ClickHouse does not suggest using `Nullable`, this is a good use case,
because otherwise additional work would have to be done to
differentiate missing values from real zeroes.

I ran a benchmark with the data pattern we expect in production, where
we have more missing values than real decimals. I created 100 million
records where 90% of decimals are missing. The difference between the
tables in storage is just 0.4Mb.

* Add revenue parameter to Events API

This commit adds support for sending revenue data in ingestion using the
`revenue` parameter - aliased to `$`.

* Add revenue parameter to mix send_pageview

* Add average and total revenue to breakdown queries
3 years ago
ruslandoga 40e95ffd3d
add LOG_FAILED_LOGIN_ATTEMPTS (#2936)
* add failed login logs

* put failed login attempt logs behind a config option

* add changelog entry

* add config test

* add auth_controller tests

* move tests to separate non-async test module

---------

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
4 years ago
ruslandoga ce7401dd83
add MAILER_NAME (#2937)
* add MAILER_NAME

* add mailer test

---------

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
4 years ago