Commit Graph

776 Commits (exclude-in-api)

Author SHA1 Message Date
Karl-Aksel Puulmann 88ee94754f Refactor: Exclude selected prop keys in API instead of in FE
Feedback from original PR: https://github.com/plausible/analytics/pull/3719#discussion_r1466583893

The FE approach would run into problems if you have >25 custom props, so excluding
in the BE allows avoiding problems
3 years ago
Karl-Aksel Puulmann f3509f2a17
Refactor spike detection top sources query (#3770)
* ORDER BY referrer_source for spikes job

This is more consistent with the rest of the queries

* Refactor top_sources -> top_sources_for_spike

* Remove more dead code

* Remove unused arguments

* Remove unused select arguments

* Add a test to top_sources_for_spike
3 years ago
Karl-Aksel Puulmann d1fe184cb7
Remove ORDER BY min(session.start) (#3771)
This adds a needless performance overhead. Note that all queries already
sort by the _name_ of the breakdown value besides the session start.
3 years ago
hq1 494ffbd622
Shields behind a FF (#3780) 3 years ago
hq1 99fe03701e
IP Block List (#3761)
* Add Ecto.Network dependency

* Migration: Add ip block list table

* If Cachex errors out, mark the cache as not ready

* Add IPRule schema

* Seed IPRules

* Add Shields context module

* Implement IPRuleCache

* Start IPRuleCache

* Drop blocklisted IPs on ingestion

* Cosmetic rename

* Add settings sidebar item

* Consider IPRuleCache readiness on health checks

* Fix typo

* Implement IP blocklist live view

* Update moduledocs

* Extend contextual module tests

* Convert IPRules LiveView into LiveComponent

* Keep live flashes on the tabs view

* Update changelog

* Format

* Credo

* Remove garbage

* Update drop reason typespecs

* Update typespecs for cache keys

* Keep track of who added a rule and when

* Test if adding via LV prefills the updated_by tooltip

* Update ecto_network dependency

* s/updated_by/added_by

* s/drop_blocklist_ip/drop_shield_rule_ip

* Add docs link

* s/Updated/Added
3 years ago
hq1 6a2d7fc0f5
Merge `Plugins.API.Router` into main one (#3767)
* Merge `Plugins.API.Router` into main one

In order to get grafana metrics reported
See: https://github.com/akoutmos/prom_ex/issues/224

* Format
3 years ago
hq1 f5129f1b0d
Turn Revenue Goals into Custom Events if the plan doesn't support them (#3768)
* Turn Revenue Goals into Custom Events if the plan runs out

* Tag test with full_build
3 years ago
Karl-Aksel Puulmann 1cb7982cd9
Filtering by multiple custom properties (#3719)
* WIP: PropFilterRow

* Get multi-behavior working

* Render multiple prop filters in one

* Modal reads from query string correctly

* Backend support for multiple custom property filters

* Add backend tests for multiple custom property filters

* Disable already selected options in property keys

We can't allow choosing the same property multiple times without changing the request
params, which we decided against

* Allow choosing any property under Behaviors > Custom props even if custom prop filter applied

This was a limitation (I believe) introduced by using ARRAY JOINs to query custom properties

* CHANGELOG.md

* Solve credo warning about too deep nesting

* Update assets/js/dashboard/stats/modals/prop-filter-modal.js

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>

* Refactor internal function for clarity

* Add another step -> Add another

* Solve 500 error

* Separate boxes per property filter

* Retain other filters in props table

* removeFilter behavior for props

* matches_member support for custom props

* filter_suggestions for prop keys should account for prop filter

* find over filter

* refactor appliedFilters

* FILTER_TYPES => FILTER_OPERATIONS

* Make add another link not wrap the whole page

* Unique keys

---------

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
3 years ago
hq1 6ce4d19931
Fix flaky test: wait till ets is emptied (#3755)
* Fix flaky test: wait till ets is emptied

* fixup
3 years ago
RobertJoonas 51f0e406a0
improve stats api upgrade required error message (#3747) 3 years ago
Adrian Gruntkowski 3738cd9578
Parse referrer filter value when passed separately (#3742)
* Parse referrer filter value when passed separately

* Remove unnecessary test setup
3 years ago
hq1 e28793a45a
Make `hello@plausible.io` clickable (#3746)
* Make `hello@plausible.io` clickable

* Format

* ...
3 years ago
hq1 9fb4ea0c3e
Bugfix: available features for expired trials, no subscriptions (#3740)
* Reorganize how subscriptions/trials are evaluated

* Bugfix: expired trial+no subscriptions should not have access to extra features

* Make self-hosted users always on trial

* Seed secondary user with password

* Format

* Fix docs

* Fix small_test run

* Run the test only on full_build

* More tweaks to small builds

* Allow [Goals] for expired trials with no subscription
3 years ago
hq1 931161f693
Generalize Cache/Warmers - extract only specifics to `Site.Cache` (#3716)
* Simplify caches config

The intervals were proven to never change really,
and previous shape made them not very discoverable either.
We'll look into a more declarative setup.

* Aim for more declarative supervision tree setup

* Generalize `Plausible.Cache`

This makes the `Site.Cache` module implement the specifics,
leaving the common bits reusable for upcoming cache processes.

* Generalize `Plausible.Cache` warmers

* Fix typos
3 years ago
hq1 18ddf1e7a0
Re enable accept traffic until notifications (#3737)
* Revert "Remove 0-sized queue"

This reverts commit 6d229b3f70.

* Revert "Temporarily remove notification cron worker (#3732)"

This reverts commit 447e3929cb.

* Revert "Temporarily disable accept_traffic_until actions (#3730)"

This reverts commit 58afe3376f.

* Send accept_traffic_until notifications at 8 UTC rather than 5

* Add option for dry-run

* Expose dry_run in a dedicated function
3 years ago
RobertJoonas 04b9067591
Add `event:goal` property to Stats API (#3718)
* add tests for filtering by goal in timeseries and aggregate

* refactor filter parsing

* stop returning custom props in event:goal breakdown

* test breaking down wildcard pageview goals

* extract filter utils

* parse more goal filter options

* add passing tests for new filter types

* do not allow querying session metrics with a goal filter

* remove unused page_match property

* test that non-configured goals are not returned in breakdown

* enforce filtered goals configured

* update changelog

* Allow simple filtering by revenue goals

This does not mean that revenue metrics are supported. If a revenue goal
is filtered by, we treat it like a simple custom event goal in the API.

* use List.wrap
3 years ago
hq1 712f60f37f
Update `accept_traffic_until` on payment succeeded notification (#3731)
* Update `accept_traffic_until` on payment succeeded notification

* full build only
3 years ago
hq1 58afe3376f
Temporarily disable accept_traffic_until actions (#3730)
* Temporarily disable accept_traffic_until actions

* Remove commented code
3 years ago
hq1 0fa7f3c1e1
Make subsequent query boundaries idempotent (#3723)
* Make subsequent query boundaries idempotent

This should prevent time-sensitive race conditions
on subsequent `Plausible.Stats.*` calls for the same query.
Not sure if there are more places where now() is
dynamic relative to query life cycle, but it's a start.

* Format

* Update lib/plausible/stats/query.ex

Co-authored-by: Wojtek Mach <wojtekmach@users.noreply.github.com>

* Match on `Query.now` in a test

---------

Co-authored-by: Wojtek Mach <wojtekmach@users.noreply.github.com>
3 years ago
Adrian Gruntkowski 822483c37c
Extract Universal Analytics import logic (#3700)
* Move imported tables schemas to separate modules outside Google ns

* Move buffer for imports to Imported ns

* fix schema newlines

* Extract UA import processing and persistence

* Decouple analytics worker implementation from UA

* Rename env variable for import buffer size

* Preserve old import queue until release
3 years ago
hq1 36a8b36382
CombBox bugfix: stale spinner on creatable options selection (#3713) 3 years ago
Karl-Aksel Puulmann 1b95433d6e
Support `contains`/`matches` for custom properties (#3687)
* Allow `matches` operator to work in BE for custom props

Note: No FE support yet, needs further testing

* feat: allow choosing `contains` for property filters in the UI

* no autocomplete on prop values if `contains` for consistency

* CHANGELOG.md

* Fix: Handle (none) property in property breakdowns when using matching

When matching we should always exclude (none)
3 years ago
hq1 0a124e69e7
Actually enforce proper WordPress spelling (#3710) 3 years ago
Uku Taht 730c8edc4a
Update ref_inspector database (#3697)
* Update ref_inspector database

* Use random referrer input
3 years ago
Uku Taht a5e0619053
Escape special regex characters (#3634)
* Escape special regex characters

* Add changelog entry
3 years ago
hq1 7deffcdeee
Prefer example.com in tests (#3690) 3 years ago
Cenk Kücük 44fca0c95d
Prioritize X-Plausible-IP header over everything else (#3689) 3 years ago
RobertJoonas 0a29679afa
Stop expecting `trial_expiry_date` to exist on upgrade (#3688)
* stop expecting trial_expiry_date to exist on upgrade

* add test
3 years ago
RobertJoonas d3094ffdb7
Prevent upgrades that would leave the customer locked (#3683)
* Remove allowance_required field from grace_period

Since we are now preventing customers from subscribing to a plan that
does not accommodate their pageview usage, there is no need for an extra
check on removing the grace period after a successful upgrade.

This extra check is the reason why the automatic unlocks have recently
failed in several cases.

* refactor outgrown subscription notices

* make a test actually test the described functionality

* Apply greater pageview allowance margin only for trial upgrades

...in order to prevent cancelled or paused subscriptions from subscribing
to plans that would still leave their account locked.

* Mark the entire ChoosePlanTest module full build only

* remove account locking guide

This is irrelevant for self-hosters, and the internal knowledge base is
a better place for this document. Moved it there.

* refactor Keyword get clause

* add a pattern matching assertion in code
3 years ago
RobertJoonas 7536f0285c
Fix weekly interval bug (#3686)
* Fix a bug

This commit fixes a bug where the timeseries weekly interval query for a
month returned 0 visitors for the first week.

This happened because the start of the month defining the first week was
missing the UTC -> site.timezone conversion.

* Update test/plausible_web/controllers/api/stats_controller/main_graph_test.exs

Remove dot from test description

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

---------

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
3 years ago
Adrian Gruntkowski eaa7020230
Latency-friendly LV modal implementation and Goals form refactor (#3649)
* Make modal for goal settings trigger without BE roundtrip

* Turn goal form into a live component and extract modal into a wrapper

* Further extract modal component and handle reset action

* Make ComboBox selection callback more flexible

* Add rudimentary loading state to dialog

* Make form unaware of being put inside a modal

* Make modal a live component and completely reset contents on open server-side

* Try to avoid race condition

* Fix race condition

* Remove unnecessary conditional on socket assigns

* Add typespecs and fix formatting

* Make goals form high latency friendly

* Fix tests to account for goal settings form becoming live component

* Fix goal settings form live component declaration

* Add documentation for modal

* Fix small build test

* Fix typo

Co-authored-by: hq1 <hq@mtod.org>

* Revert no longer necessary test changes from 46f65d9

* Fix and clean up modal styling

* Keep focus on dialog when open and show only spinner on backdrop when loading

* Adjust corners and shadows and implement open/close transitions

* Lock body scroll when modal is open

* Make modal top-aligned again to avoid jumping around on variable content height

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
Adrian Gruntkowski 954b70c79b
Run SiteLocker right after site ownership transfer (#3684)
* Run SiteLocker right after site ownership transfer

* Put site unlocking behind full build flag

* Tag regression tests as full build only
3 years ago
hq1 32456d4348
Upgrade stack (deps, elixir 1.16, OTP 26.2.1) (#3678)
* Bump asdf erlang & elixir

* Bump erlang/elixir in the dockerfile

* Remove Oban.Stager config as per https://github.com/sorentwo/oban/blob/main/guides/upgrading/v2.14.md

* Configure Oban for tests as per https://github.com/sorentwo/oban/blob/main/guides/upgrading/v2.14.md

* Mark sampling hint clause with unsafe fragment

Any other/recommended way to do that? cc @ruslandoga

* Address String.slice/2 deprecation

* Update deps

* Address 0.0 matching warning

* Make funnel settings work

although this is probably not the best way to do it.
Needs revisiting, not sure what broke it - could not find
any breaking changes in related dependencies.

* Migrate oban as per https://github.com/sorentwo/oban/blob/main/guides/upgrading/v2.17.md

* Update credo

* Bump CI cache

* Use `Bypass.pass` to avoid exit shutdown message
3 years ago
RobertJoonas 403f559b35
Limit Custom Properties access in Stats API (#3670)
* add new function and doc for allowed_props

* limit props access in Stats API queries

* use dot syntax instead
3 years ago
hq1 24a8aa2821
Reapply sentry context (#3675)
* Reapply "Sentry context in live views (#3672)"

This reverts commit 5449fead160064b8a0081c458cc5dcd34399eb0b.

* Make sure `:selection_made` is handled in `GoalSettings.Form`

That was a bit unepexcted.. normally `handle_info` is injected
by the LiveView use macro and it discards any message gracefully.
After switching to `use PlausibleWeb, :live_view` we're also
using `PlausibleWeb.Live.Flash` that happens to inject its own receive
clause for closing the flash. Which then renders the original,
overridable, `handle_info` catch-all obsolete.

* Update LV SentryContext only on connected sockets

(first mount already has the right context coming from Sentry plug)

* Make sure Live.ChoosePlan passes `current_user_id` session key
3 years ago
hq1 c87b165aef
Revert "Sentry context in live views (#3672)" (#3673)
This reverts commit 9bb2dc00d0.
3 years ago
hq1 9bb2dc00d0
Sentry context in live views (#3672)
* Add common LiveView macro to PlausibleWeb

* Keep peer data, URI and UA in /live websocket metadata

* Use new PlausibleWeb macro in existing LiveViews

* Implement adding some basic Sentry context `on_mount`

* Format

* Use macro in Live.FunnelSettings

* Update FunnelSettings.Form
3 years ago
hq1 9cb44291f7
Rate limit e-mail changes (#3667) 3 years ago
hq1 5c8f39ac4c
Prevent crashes on ambiguity when casting timezones (#3663)
* Fall back to UTC when timezone gap found

* bugfix 1

* bugfix 2

* Apply safe tz conversion to all cases where it's done on a ts other than now

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
hq1 72b4e05bbf
Always enable e-mail verification on full build (#3666)
* Always enable e-mail verification on full build

* s/change/set

* Update lib/plausible/auth/user.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
Vinicius Brasil 9e5d63ed96
Display premium features tabs on the dashboard (#3646)
* Display premium features tabs on the dashboard

This commit makes the funnels and props tab on the dashboard visible
even when the site owner's plan does not have access to that feature.
This is to raise awareness of those features, and if the site owner does
not want to see that tab, they can still click "Hide this report" to
hide it.

Previously, when the plan did not support a feature, the feature module
`enabled/1` function returned `false`, regardless of the
`sites.feature_enabled` toggle. This commit creates a new function
called `opted_out/1` to differentiate access from explicitly opting out
a feature.

* Remove unused data-conversions-enabled attribute

* DRY Plausible.Billing.Feature.check_availability/1 function

* Allow opting out features the user doesn't have access to

* add upgrade CTA to FeatureSetupNotice

* fix JS linting errors

* simplify notice.js

* fix behaviour when deleting funnels saved to localstorage

In case some other funnel exists, we will use that as the default
selected one. If not, a feature setup notice will be displayed again.

---------

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
3 years ago
RobertJoonas 155a7a56a7
Fix goal filter inconsistency (#3658)
* remove unused code

* add e.name == "pageview" condition to pageview goals

This fixes the weird behavior where filtering by a pageview goal would
also return custom events and vice versa.

* update changelog
3 years ago
hq1 b6a2acb57a
Update accept_traffic_until notifications (#3665)
* Update accept_traffic_until notifications:

  - extend has_stats to 48h
  - ensure the user is properly greeted

* Clarify the structure passed to email template function
3 years ago
hq1 21bbd3835a
Disallow funny business on timezone entry (#3662)
* Disallow funny business on timezone entry

* Add external API test
3 years ago
RobertJoonas 1342135908
Add `custom_props.csv` back to CSV export for Growth plans (#3656)
* restore special props access for growth plans

* format

* refactor if condition

Co-authored-by: hq1 <hq@mtod.org>

* format

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
hq1 8be9397199
Skip breakdown of imported pages with pageviews=0 (#3655)
* Skip breakdown of imported pages with pageviews=0

* Fixup

* Prove good impoted records are merged
3 years ago
hq1 73923404b0
Improve approaching accept_traffic_until e-mail (#3660)
* Improve approaching accept_traffic_until e-mail

* fixup

* format
3 years ago
hq1 f755b20569
Lock traffic notifications (#3641)
* Update communication

* Remove an unreachable function (mistyped)

* [migration] Make accept_traffic_until a date

* Fix typo

* Set `accept_traffic_until` when creating a site

* Update sites `accept_traffic_until` on subscription change

* Add a note to yearly cancellation notification

* Rephrase annual e-mail for clarity

* Pass the small build test

* Add email notifications

* Fixup

* Implement `accept_traffic_until` notification worker

* Fixup - no need to test this for small build

* Update moduledoc

* Move moduletag

* s/sent_at/sent_on

* Use WHERE NOT EXISTS instead of LEFT JOIN

* Use upsert when tracking notifications sent

* Store sent marker before actually sending notification

* Prefer to keep `accept_traffic_until` on the user record

This gives us a single source of truth, addresses cases like
ownership transparently, simplifies the code and enables CRM toggles.
The only downside is that there's another join performed in the
Sites.Cache full refresh - in this case, small refreshes are
skipped - but this is fine, since the traffic will be let in
anyway.

* Expose `accepted_traffic_until` in the CRM

* Update lib/plausible/auth/user.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Preload owner in CRM

* Use the offset parameter in trial over e-mail contents

* Format

* Harden cache test

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
Adrian Gruntkowski 66d4b3a966
Fix SiteLocker for case of user without trial expiry date but with active subscription (#3652) 3 years ago
RobertJoonas d2270f3c35
Small bugfix + refactor email reports (#3642)
* use more convenient testing functions

* do not display + sign with 0% change in emails

* Rename module/file/function names

before, `weekly_report` was also used for monthly reports and that was a
bit confusing to read in code.

* Refactor send_email_report.ex

This commit improves readability by refactoring the code into smaller
functions and reducing the number of arguments given to functions.

But more importantly, it stops making duplicate stats queries for every
email recipient by moving the queries out of the for loop.

* Refactor: move querying logic out of the worker module

and merge all stats information under a single `stats` assign.
3 years ago