Commit Graph

1521 Commits (exclude-in-api)

Author SHA1 Message Date
Karl-Aksel Puulmann 88ee94754f Refactor: Exclude selected prop keys in API instead of in FE
Feedback from original PR: https://github.com/plausible/analytics/pull/3719#discussion_r1466583893

The FE approach would run into problems if you have >25 custom props, so excluding
in the BE allows avoiding problems
3 years ago
Karl-Aksel Puulmann f5e15941de
Add migration for minmax indexes on session timestamps (#3772) 3 years ago
Karl-Aksel Puulmann f3509f2a17
Refactor spike detection top sources query (#3770)
* ORDER BY referrer_source for spikes job

This is more consistent with the rest of the queries

* Refactor top_sources -> top_sources_for_spike

* Remove more dead code

* Remove unused arguments

* Remove unused select arguments

* Add a test to top_sources_for_spike
3 years ago
Karl-Aksel Puulmann d1fe184cb7
Remove ORDER BY min(session.start) (#3771)
This adds a needless performance overhead. Note that all queries already
sort by the _name_ of the breakdown value besides the session start.
3 years ago
hq1 494ffbd622
Shields behind a FF (#3780) 3 years ago
hq1 99fe03701e
IP Block List (#3761)
* Add Ecto.Network dependency

* Migration: Add ip block list table

* If Cachex errors out, mark the cache as not ready

* Add IPRule schema

* Seed IPRules

* Add Shields context module

* Implement IPRuleCache

* Start IPRuleCache

* Drop blocklisted IPs on ingestion

* Cosmetic rename

* Add settings sidebar item

* Consider IPRuleCache readiness on health checks

* Fix typo

* Implement IP blocklist live view

* Update moduledocs

* Extend contextual module tests

* Convert IPRules LiveView into LiveComponent

* Keep live flashes on the tabs view

* Update changelog

* Format

* Credo

* Remove garbage

* Update drop reason typespecs

* Update typespecs for cache keys

* Keep track of who added a rule and when

* Test if adding via LV prefills the updated_by tooltip

* Update ecto_network dependency

* s/updated_by/added_by

* s/drop_blocklist_ip/drop_shield_rule_ip

* Add docs link

* s/Updated/Added
3 years ago
hq1 6a2d7fc0f5
Merge `Plugins.API.Router` into main one (#3767)
* Merge `Plugins.API.Router` into main one

In order to get grafana metrics reported
See: https://github.com/akoutmos/prom_ex/issues/224

* Format
3 years ago
hq1 f5129f1b0d
Turn Revenue Goals into Custom Events if the plan doesn't support them (#3768)
* Turn Revenue Goals into Custom Events if the plan runs out

* Tag test with full_build
3 years ago
Karl-Aksel Puulmann 499be9f0b3
Add more custom properties to be tracked (#3778)
We just released support for multiple custom property filters, but our demo/live setup
does not have many useful custom properties. This PR adds a few more to allow
better slicing-and-dicing.
3 years ago
Karl-Aksel Puulmann 1cb7982cd9
Filtering by multiple custom properties (#3719)
* WIP: PropFilterRow

* Get multi-behavior working

* Render multiple prop filters in one

* Modal reads from query string correctly

* Backend support for multiple custom property filters

* Add backend tests for multiple custom property filters

* Disable already selected options in property keys

We can't allow choosing the same property multiple times without changing the request
params, which we decided against

* Allow choosing any property under Behaviors > Custom props even if custom prop filter applied

This was a limitation (I believe) introduced by using ARRAY JOINs to query custom properties

* CHANGELOG.md

* Solve credo warning about too deep nesting

* Update assets/js/dashboard/stats/modals/prop-filter-modal.js

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>

* Refactor internal function for clarity

* Add another step -> Add another

* Solve 500 error

* Separate boxes per property filter

* Retain other filters in props table

* removeFilter behavior for props

* matches_member support for custom props

* filter_suggestions for prop keys should account for prop filter

* find over filter

* refactor appliedFilters

* FILTER_TYPES => FILTER_OPERATIONS

* Make add another link not wrap the whole page

* Unique keys

---------

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
3 years ago
hq1 9f2f894421
Fix flashing on reapplying theme (#3760)
* Fix flashing on reapplying theme

This seem to be making a difference on staging

* Add `blocking="rendering"` attribute
3 years ago
hq1 9ad5ecebc2
Improve print dashboard css (#3754)
* Improve print dashboard CSS

* More improvements
3 years ago
Marko Saric 03e1224693
improvements to the email copy (#3756)
* Update yearly_expiration_notification.html.eex

* Update yearly_renewal_notification.html.eex
3 years ago
RobertJoonas 51f0e406a0
improve stats api upgrade required error message (#3747) 3 years ago
Adrian Gruntkowski 3738cd9578
Parse referrer filter value when passed separately (#3742)
* Parse referrer filter value when passed separately

* Remove unnecessary test setup
3 years ago
hq1 e28793a45a
Make `hello@plausible.io` clickable (#3746)
* Make `hello@plausible.io` clickable

* Format

* ...
3 years ago
hq1 9fb4ea0c3e
Bugfix: available features for expired trials, no subscriptions (#3740)
* Reorganize how subscriptions/trials are evaluated

* Bugfix: expired trial+no subscriptions should not have access to extra features

* Make self-hosted users always on trial

* Seed secondary user with password

* Format

* Fix docs

* Fix small_test run

* Run the test only on full_build

* More tweaks to small builds

* Allow [Goals] for expired trials with no subscription
3 years ago
hq1 931161f693
Generalize Cache/Warmers - extract only specifics to `Site.Cache` (#3716)
* Simplify caches config

The intervals were proven to never change really,
and previous shape made them not very discoverable either.
We'll look into a more declarative setup.

* Aim for more declarative supervision tree setup

* Generalize `Plausible.Cache`

This makes the `Site.Cache` module implement the specifics,
leaving the common bits reusable for upcoming cache processes.

* Generalize `Plausible.Cache` warmers

* Fix typos
3 years ago
hq1 18ddf1e7a0
Re enable accept traffic until notifications (#3737)
* Revert "Remove 0-sized queue"

This reverts commit 6d229b3f70.

* Revert "Temporarily remove notification cron worker (#3732)"

This reverts commit 447e3929cb.

* Revert "Temporarily disable accept_traffic_until actions (#3730)"

This reverts commit 58afe3376f.

* Send accept_traffic_until notifications at 8 UTC rather than 5

* Add option for dry-run

* Expose dry_run in a dedicated function
3 years ago
RobertJoonas 04b9067591
Add `event:goal` property to Stats API (#3718)
* add tests for filtering by goal in timeseries and aggregate

* refactor filter parsing

* stop returning custom props in event:goal breakdown

* test breaking down wildcard pageview goals

* extract filter utils

* parse more goal filter options

* add passing tests for new filter types

* do not allow querying session metrics with a goal filter

* remove unused page_match property

* test that non-configured goals are not returned in breakdown

* enforce filtered goals configured

* update changelog

* Allow simple filtering by revenue goals

This does not mean that revenue metrics are supported. If a revenue goal
is filtered by, we treat it like a simple custom event goal in the API.

* use List.wrap
3 years ago
hq1 712f60f37f
Update `accept_traffic_until` on payment succeeded notification (#3731)
* Update `accept_traffic_until` on payment succeeded notification

* full build only
3 years ago
hq1 58afe3376f
Temporarily disable accept_traffic_until actions (#3730)
* Temporarily disable accept_traffic_until actions

* Remove commented code
3 years ago
hq1 0fa7f3c1e1
Make subsequent query boundaries idempotent (#3723)
* Make subsequent query boundaries idempotent

This should prevent time-sensitive race conditions
on subsequent `Plausible.Stats.*` calls for the same query.
Not sure if there are more places where now() is
dynamic relative to query life cycle, but it's a start.

* Format

* Update lib/plausible/stats/query.ex

Co-authored-by: Wojtek Mach <wojtekmach@users.noreply.github.com>

* Match on `Query.now` in a test

---------

Co-authored-by: Wojtek Mach <wojtekmach@users.noreply.github.com>
3 years ago
Adrian Gruntkowski 822483c37c
Extract Universal Analytics import logic (#3700)
* Move imported tables schemas to separate modules outside Google ns

* Move buffer for imports to Imported ns

* fix schema newlines

* Extract UA import processing and persistence

* Decouple analytics worker implementation from UA

* Rename env variable for import buffer size

* Preserve old import queue until release
3 years ago
Karl-Aksel Puulmann 11c5d3b251
Remove ARRAY JOINs (#3708)
* Use array indexing instead of left joins

This has multiple benefits:
- Allows us to more easily add support for multiple custom property
  filters
- Performance: Increased performance of some queries by ~20%

Work left to do:
- Check tests coverage
- Create macros for repeated fragments
- Quote and refer to right tables in fragments/macros

* Use macros to clean up code

* Use existing atoms to avoid memory leaks
3 years ago
RobertJoonas 041b0ab779
trace aggregate and timeseries queries too (#3712) 3 years ago
hq1 36a8b36382
CombBox bugfix: stale spinner on creatable options selection (#3713) 3 years ago
RobertJoonas 6ca4eaf5e2
Improve query tracing (#3704)
* Enable breakdown property tracing for event:goal

This is currently overwritten by event:name. The queries for event:name
will still be tracked if queried specifically.

* Add filters tracing to Query.trace/1

* make sure breakdown metrics are sorted and joined

* check opts skip tracing directly without passing it
3 years ago
hq1 8965ac40a7
Decorate site settings sidebar with heroicons (#3711)
* Decorate site settings sidebar with heroicons

* Capitalize "Danger Zone"

* Capitalize "Danger Zone"
3 years ago
Karl-Aksel Puulmann 1b95433d6e
Support `contains`/`matches` for custom properties (#3687)
* Allow `matches` operator to work in BE for custom props

Note: No FE support yet, needs further testing

* feat: allow choosing `contains` for property filters in the UI

* no autocomplete on prop values if `contains` for consistency

* CHANGELOG.md

* Fix: Handle (none) property in property breakdowns when using matching

When matching we should always exclude (none)
3 years ago
hq1 0a124e69e7
Actually enforce proper WordPress spelling (#3710) 3 years ago
Marko Saric fb80847d7e
capital P in WordPress (#3703) 3 years ago
Uku Taht a5e0619053
Escape special regex characters (#3634)
* Escape special regex characters

* Add changelog entry
3 years ago
Cenk Kücük 44fca0c95d
Prioritize X-Plausible-IP header over everything else (#3689) 3 years ago
RobertJoonas 0a29679afa
Stop expecting `trial_expiry_date` to exist on upgrade (#3688)
* stop expecting trial_expiry_date to exist on upgrade

* add test
3 years ago
RobertJoonas d3094ffdb7
Prevent upgrades that would leave the customer locked (#3683)
* Remove allowance_required field from grace_period

Since we are now preventing customers from subscribing to a plan that
does not accommodate their pageview usage, there is no need for an extra
check on removing the grace period after a successful upgrade.

This extra check is the reason why the automatic unlocks have recently
failed in several cases.

* refactor outgrown subscription notices

* make a test actually test the described functionality

* Apply greater pageview allowance margin only for trial upgrades

...in order to prevent cancelled or paused subscriptions from subscribing
to plans that would still leave their account locked.

* Mark the entire ChoosePlanTest module full build only

* remove account locking guide

This is irrelevant for self-hosters, and the internal knowledge base is
a better place for this document. Moved it there.

* refactor Keyword get clause

* add a pattern matching assertion in code
3 years ago
RobertJoonas 7536f0285c
Fix weekly interval bug (#3686)
* Fix a bug

This commit fixes a bug where the timeseries weekly interval query for a
month returned 0 visitors for the first week.

This happened because the start of the month defining the first week was
missing the UTC -> site.timezone conversion.

* Update test/plausible_web/controllers/api/stats_controller/main_graph_test.exs

Remove dot from test description

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

---------

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
3 years ago
Adrian Gruntkowski eaa7020230
Latency-friendly LV modal implementation and Goals form refactor (#3649)
* Make modal for goal settings trigger without BE roundtrip

* Turn goal form into a live component and extract modal into a wrapper

* Further extract modal component and handle reset action

* Make ComboBox selection callback more flexible

* Add rudimentary loading state to dialog

* Make form unaware of being put inside a modal

* Make modal a live component and completely reset contents on open server-side

* Try to avoid race condition

* Fix race condition

* Remove unnecessary conditional on socket assigns

* Add typespecs and fix formatting

* Make goals form high latency friendly

* Fix tests to account for goal settings form becoming live component

* Fix goal settings form live component declaration

* Add documentation for modal

* Fix small build test

* Fix typo

Co-authored-by: hq1 <hq@mtod.org>

* Revert no longer necessary test changes from 46f65d9

* Fix and clean up modal styling

* Keep focus on dialog when open and show only spinner on backdrop when loading

* Adjust corners and shadows and implement open/close transitions

* Lock body scroll when modal is open

* Make modal top-aligned again to avoid jumping around on variable content height

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
Adrian Gruntkowski 954b70c79b
Run SiteLocker right after site ownership transfer (#3684)
* Run SiteLocker right after site ownership transfer

* Put site unlocking behind full build flag

* Tag regression tests as full build only
3 years ago
hq1 32456d4348
Upgrade stack (deps, elixir 1.16, OTP 26.2.1) (#3678)
* Bump asdf erlang & elixir

* Bump erlang/elixir in the dockerfile

* Remove Oban.Stager config as per https://github.com/sorentwo/oban/blob/main/guides/upgrading/v2.14.md

* Configure Oban for tests as per https://github.com/sorentwo/oban/blob/main/guides/upgrading/v2.14.md

* Mark sampling hint clause with unsafe fragment

Any other/recommended way to do that? cc @ruslandoga

* Address String.slice/2 deprecation

* Update deps

* Address 0.0 matching warning

* Make funnel settings work

although this is probably not the best way to do it.
Needs revisiting, not sure what broke it - could not find
any breaking changes in related dependencies.

* Migrate oban as per https://github.com/sorentwo/oban/blob/main/guides/upgrading/v2.17.md

* Update credo

* Bump CI cache

* Use `Bypass.pass` to avoid exit shutdown message
3 years ago
hq1 c1a1d697a4
Partially revert #3661 - just keep the real errors wrapped, but don't display anything to the user (#3677)
* Revert "Remove unused RocketIcon"

This reverts commit c5e8d0c172.

* Revert "Display either hash or actual error message"

This reverts commit 0c091ab35f.

* Revert "Use ApiErrorNotice in funnels"

This reverts commit 5929de248e.

* Revert "Don't render "No data yet" when there's a NetworkError for example"

This reverts commit 70bee07632.

* Revert "Show the sinking shuttle notice whenever an API error occurs"

This reverts commit 9a62c8af2b.

* Revert "Add Hahash dependency"

This reverts commit b94207ea0a.

* Remove support hash
3 years ago
RobertJoonas 403f559b35
Limit Custom Properties access in Stats API (#3670)
* add new function and doc for allowed_props

* limit props access in Stats API queries

* use dot syntax instead
3 years ago
hq1 24a8aa2821
Reapply sentry context (#3675)
* Reapply "Sentry context in live views (#3672)"

This reverts commit 5449fead160064b8a0081c458cc5dcd34399eb0b.

* Make sure `:selection_made` is handled in `GoalSettings.Form`

That was a bit unepexcted.. normally `handle_info` is injected
by the LiveView use macro and it discards any message gracefully.
After switching to `use PlausibleWeb, :live_view` we're also
using `PlausibleWeb.Live.Flash` that happens to inject its own receive
clause for closing the flash. Which then renders the original,
overridable, `handle_info` catch-all obsolete.

* Update LV SentryContext only on connected sockets

(first mount already has the right context coming from Sentry plug)

* Make sure Live.ChoosePlan passes `current_user_id` session key
3 years ago
hq1 c87b165aef
Revert "Sentry context in live views (#3672)" (#3673)
This reverts commit 9bb2dc00d0.
3 years ago
hq1 9bb2dc00d0
Sentry context in live views (#3672)
* Add common LiveView macro to PlausibleWeb

* Keep peer data, URI and UA in /live websocket metadata

* Use new PlausibleWeb macro in existing LiveViews

* Implement adding some basic Sentry context `on_mount`

* Format

* Use macro in Live.FunnelSettings

* Update FunnelSettings.Form
3 years ago
hq1 dc8210dd84
Use tags rather than context to annotate support hashes in Sentry (#3668) 3 years ago
hq1 9cb44291f7
Rate limit e-mail changes (#3667) 3 years ago
hq1 5c8f39ac4c
Prevent crashes on ambiguity when casting timezones (#3663)
* Fall back to UTC when timezone gap found

* bugfix 1

* bugfix 2

* Apply safe tz conversion to all cases where it's done on a ts other than now

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
hq1 72b4e05bbf
Always enable e-mail verification on full build (#3666)
* Always enable e-mail verification on full build

* s/change/set

* Update lib/plausible/auth/user.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
hq1 4569ac09c1
Don't leak internal server errors, use support hash (#3661)
* Add Hahash dependency

* Don't leak internal server error details to the user

* Show the sinking shuttle notice whenever an API error occurs

* Don't render "No data yet" when there's a NetworkError for example

* Use ApiErrorNotice in funnels

* Display either hash or actual error message

The reason "internal-server-error" doesn't work well as a fallback
hash is that e.g. `NetworkError when attempting to fetch resource`
might be completely at client's fault. In such cases it's better
to display the whole thing still.

* Remove unused RocketIcon
3 years ago