Commit Graph

101 Commits (master)

Author SHA1 Message Date
hq1 59afa20955
Reapply #3878 + bugfix hit rate tracking (#3891)
* Reapply "Replace caching engine (#3878)" (#3883)

This reverts commit c5881cdc6d.

* Ensure hit rate is tracked on `get_or_store`

* Remove :wx and :observer

* Remove unused deps

* Use `:set` table type
3 years ago
hq1 c5881cdc6d
Revert "Replace caching engine (#3878)" (#3883)
This reverts commit 437a3350ff.
3 years ago
hq1 437a3350ff
Replace caching engine (#3878)
* Dependencies: swap Cachex for ConCache

* Implement Cache adapter wrapping ConCache

* Implement cache stats tracker, for metrics

* Use Cache.Adapter in Plausible.Cache

Marking the test as not slow anymore

* Use Cache Adapter when tracking sessions

* Use Cache Adapter for UA parsing

* Rename child identifiers - cachex is obsolete now

* Test stats tracking

* Update grafana metrics

* Put all caches under common child specification

* Try less

* Shorten the function delegation path
3 years ago
hq1 eceac8afd5
Allow inviting users who are members already (#3797)
* Allow e-mail exclusion in team members quota

* Exclude invitee from quota on invitation create

* Enable invitation submission but report errors on quota violation

* Use a single interface for team members quota

* Check the `Keyword.validate/2` result

* Update test/plausible_web/controllers/site/membership_controller_test.exs

Co-authored-by: Uku Taht <Uku.taht@gmail.com>

---------

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
3 years ago
Adrian Gruntkowski f8b4d5066a
Add multiple imports per site (#3724)
* Clean up references to no longer active `google_analytics_imports` Oban queue

* Stub CSV importer

* Add SiteImport schema

* Rename `Plausible.Imported` module file to match module name

* Add `import_id` column to `Imported.*` CH schemas

* Implement Importer behavior and manage imports state using new entities

* Implement importer callbacks and maintain site.imported_data for UA

* Keep imports in sync when forgetting all imports

* Scope imported data queries to completed import IDs

* Mark newly imported data with respective import ID

* Clean up Importer implementation a bit

* Test querying legacy and new imported data

* Send Oban notifications on import worker failure too

* Fix checking for forgettable imports and remove redundant function

* Fix UA integration test

* Change site import source to atom enum and add source label

* Add typespecs and reduce repetition in `Plausible.Imported`

* Improve documentation and typespecs

* Add test for purging particular import

* Switch email notification templates depending on import source

* Document running import synchronously

* Fix UA importer args parsing and ensure it's covered by tests

* Clear `site.stats_start_date` on complete import to force recalculation

* Test Oban notifications (h/t @ruslandoga)

* Purge stats on import failure right away to reduce a chance of leaving debris behind

* Fix typos

Co-authored-by: hq1 <hq@mtod.org>

* Fix another typo

* Refactor fetching earliest import and earliest stats start date

* Use `Date.after?` instead of `Timex.after?`

* Cache import data in site virtual fields and limit queried imports to 5

* Ensure always current `stats_start_date` is used

* Work around broken typespec in Timex

* Make `SiteController.forget_imported` action idempotent

* Discard irrecoverably failed import tasks

* Use macros for site import statuses

There's also a fix ensuring only complete imports are considered
where relevant - couldn't isolate it as it was in a common hunk

* Use `import_id` as worker job uniqueness criterion

* Do not load imported stats data in plugins API context

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
hq1 9fb4ea0c3e
Bugfix: available features for expired trials, no subscriptions (#3740)
* Reorganize how subscriptions/trials are evaluated

* Bugfix: expired trial+no subscriptions should not have access to extra features

* Make self-hosted users always on trial

* Seed secondary user with password

* Format

* Fix docs

* Fix small_test run

* Run the test only on full_build

* More tweaks to small builds

* Allow [Goals] for expired trials with no subscription
3 years ago
hq1 931161f693
Generalize Cache/Warmers - extract only specifics to `Site.Cache` (#3716)
* Simplify caches config

The intervals were proven to never change really,
and previous shape made them not very discoverable either.
We'll look into a more declarative setup.

* Aim for more declarative supervision tree setup

* Generalize `Plausible.Cache`

This makes the `Site.Cache` module implement the specifics,
leaving the common bits reusable for upcoming cache processes.

* Generalize `Plausible.Cache` warmers

* Fix typos
3 years ago
hq1 18ddf1e7a0
Re enable accept traffic until notifications (#3737)
* Revert "Remove 0-sized queue"

This reverts commit 6d229b3f70.

* Revert "Temporarily remove notification cron worker (#3732)"

This reverts commit 447e3929cb.

* Revert "Temporarily disable accept_traffic_until actions (#3730)"

This reverts commit 58afe3376f.

* Send accept_traffic_until notifications at 8 UTC rather than 5

* Add option for dry-run

* Expose dry_run in a dedicated function
3 years ago
hq1 58afe3376f
Temporarily disable accept_traffic_until actions (#3730)
* Temporarily disable accept_traffic_until actions

* Remove commented code
3 years ago
Adrian Gruntkowski 822483c37c
Extract Universal Analytics import logic (#3700)
* Move imported tables schemas to separate modules outside Google ns

* Move buffer for imports to Imported ns

* fix schema newlines

* Extract UA import processing and persistence

* Decouple analytics worker implementation from UA

* Rename env variable for import buffer size

* Preserve old import queue until release
3 years ago
Adrian Gruntkowski 954b70c79b
Run SiteLocker right after site ownership transfer (#3684)
* Run SiteLocker right after site ownership transfer

* Put site unlocking behind full build flag

* Tag regression tests as full build only
3 years ago
hq1 f755b20569
Lock traffic notifications (#3641)
* Update communication

* Remove an unreachable function (mistyped)

* [migration] Make accept_traffic_until a date

* Fix typo

* Set `accept_traffic_until` when creating a site

* Update sites `accept_traffic_until` on subscription change

* Add a note to yearly cancellation notification

* Rephrase annual e-mail for clarity

* Pass the small build test

* Add email notifications

* Fixup

* Implement `accept_traffic_until` notification worker

* Fixup - no need to test this for small build

* Update moduledoc

* Move moduletag

* s/sent_at/sent_on

* Use WHERE NOT EXISTS instead of LEFT JOIN

* Use upsert when tracking notifications sent

* Store sent marker before actually sending notification

* Prefer to keep `accept_traffic_until` on the user record

This gives us a single source of truth, addresses cases like
ownership transparently, simplifies the code and enables CRM toggles.
The only downside is that there's another join performed in the
Sites.Cache full refresh - in this case, small refreshes are
skipped - but this is fine, since the traffic will be let in
anyway.

* Expose `accepted_traffic_until` in the CRM

* Update lib/plausible/auth/user.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Preload owner in CRM

* Use the offset parameter in trial over e-mail contents

* Format

* Harden cache test

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
Adrian Gruntkowski 9d97dc1912
Move limit enforcement to accepting site ownership transfer (#3612)
* Move limit enforcement to accepting site ownerhsip transfer

* enforce pageview limit on ownership transfer accept

* Refactor plan limit check logic

* Extract `ensure_can_take_ownership` to `Invitations` context and refactor

* Improve styling of exceeded limits notice in invitation dialog and disable button

* styling improvements to notice

* make transfer_ownership return transfer to self error

* do not allow transferring to user without active subscription WIP

* Add missing typespec and improve existing ones

* Fix formatting

* Explicitly label direct match on function argument for clarity

* Slightly refactor `CreateInvitation.bulk_transfer_ownership_direct`

* Exclude quota enforcement tests from small build test suite

* Remove unused return type from `invite_error()` union type

* Do not block plan upgrade when there's pending ownership transfer

* Don't block and only warn about missing features on transfer

* Remove `x-init` attribute used for debugging

* Add tests for `Quota.monthly_pageview_usage/2`

* Test and improve site admin ownership transfer actions

* Extend tests for `AcceptInvitation.transfer_ownership`

* Test transfer ownership controller level accept action error cases

* Test choosing plan by user without sites but with a pending ownership transfer

* Test invitation x-data in sites LV

* Remove sitelocker trigger in invitation acceptance code and simplify logic

* Add Quota test for `user.allow_next_upgrade_override` being set

* ignore pageview limit only when subscribing to plan

* Use sandbox Paddle instance for staging

* Use sandbox paddle key for staging and dev

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
3 years ago
hq1 8d85c38047
Add `:payment_required` policy to `GateKeeper` (#3628)
* Migration: Add `accept_traffic_until` to "sites"

* Drop traffic from sites where `accept_traffic_until` is passed
3 years ago
ruslandoga 0eedf9aa98
rm Hammer (#3571) 3 years ago
hq1 a4b9c3b8ba
Remove custom domains support + update build options (#3559)
* Disable super-admin checks on small build

* Mute a test writing to stdout

* Move sampling outside of small build

* Convert waiting_first_pageview to heex and stop relying on env vars

* Set site limit unlimited on small build

* Stop relying on app env to get trial expiry

* Remove custom domains - including migration

* Remove is_selfhosted from layout view

* Quota fixup

* Stop relying on app env for self hosted registration

* Stop relying on app env for pass reset success

* Apply on_trial? check only on full build

* Update templates relying on app env

* Adjusts auth controller tests for small build

* Trial fixup

* Fixup

* Stop relying on app env

* Rest of the fsckn owl

* Update typespecs

* Fix dialyzer warning

* Remove unused module

* Credo + format

* GeoIP is not, for full build

* Use `small_build?()` where applicable

* Implement bypassing FirstLaunchPlug without insertions

* Get Marko's patch de58a18a85

* Test is-dbip=false presence

* Fix typespec

* Remove future hardcodes

* Handle `nil` from `Plausible.Geo.database_type()`

* Remove XXX marker

* Use one typespec for two clauses

* Introduce `MIX_ENV=small_dev`

* Revert "Use one typespec for two clauses"

This reverts commit 8d8cd21764.
3 years ago
hq1 b9ec38038c
Add small build option (#3536)
* Update applications

* Clone community config

* Move modules to experimental dir

* Update runtime config

* Apply first set of compile-time conditionals

* Move funnel schemas to experimental

* Make funnel schema-less build compile

* Use experimental/lib for elixir code

* Move JS funnels to experimental

* Clean up conditional rendering

* Tidy up the pipeline

* Make two builds pass tests without warnings

* Reuse existing dotenvs

* Do a bunch of renames

* Clean up naming

* Run secondary CI

* Update router

* Remove RewriteFunnelDupes migration

Tests were disabled already and it was a one-off shot

* Fixup quota mixins

* Add moduledoc

* Change MIX_ENV for seconary test run

* Skip crm on small

* !fixup

* Exclude flags pipeline

* Update lib/plausible_web/controllers/stats_controller.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
RobertJoonas 02a1271ee4
Bugfix site transfers (#3531)
* Bugfix: allow ownership transfers when premium features enabled but not used

Fields like `props_enabled` and `funnels_enabled` are true by default,
and these fields do not indicate whether the user/site is actually using
these features or not.

* allow site transfers if they will be at limit after transfer

* small refactor
3 years ago
Adrian Gruntkowski f464ceae88
Implement pinned sites (#3469)
* Revert "Remove site pins for now"

This reverts commit 5eccf4eaf6.

* Implement basic site pin schema level logic within user specific preferences

* Add vertical ellipsis menu markup

* Implement basic changesets for user preferences

* Implement pin toggling

* Try to fix pin sorting

* Implement pin toggling in LV

* Adjust moduledocs for new schema(s)

* Remove unnecessary `distinct` from query

* Use `button` for pin/unpin action

* Generalize preference setting

* Rename schema and fields for clarity

* Rename `list_type` -> `entry_type`

* Safeguard setting options

* Test `set_option/4` and `toggle_pin/2`

* Add test for listing pinned sites via `Sites.list`

* Disallow pinning sites outside page explicitly

* Test pinning in LV

* Test conditional rendering of site settings in /sites

* Remove unnecessary TODO comment

* Safeguard `Sites.set_option/4` against invalid user/site combo

* Handle pinned sites in dashboard site picker

* Clear flashes upon (un)pinning sites

* Update CHANGELOG

* Prevent blinking of hamburger menu items on first paint

* Highlight hamburger handle on hover in /sites

* Start showing hotkeys in site picker again

* Sort pinned sites in the order they were pinned

* Update sites list order immediately after pin/unpin toggle

* Refactor and split `Sites.list/3`, extracting `Sites.list_with_invitations/3`

* Cap number of pinned sites at 9 per user

* First pass on visual indication of site cards (dis)appearing

* Apply ellipsis gradient+shadow on card hover

* Fix responsive padding of site cards

* Sort by invitations first, pinned sites second and then the rest

* Revert "Apply ellipsis gradient+shadow on card hover"

This reverts commit 0608796612639030ccbb12df639709f78edc1434.

* Apply more subtle hover effect on the ellipsis menu

* Make error and success flash LV boxes use separate component containers

* Promote `pinned_at` in table migration to a column

* Switch logic to using `pinned_at` as a standard schema field

* Refactor `Sites.list*` getting rid of subquery (h/t @ukutaht)

* Remove migration which is already merged upstream

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
3 years ago
RobertJoonas f977351ae2
Fix upgrade page for legacy trials + bug fixes (#3486)
* allow using Stats API and Props for free_10k subscriptions

* return v3 plans for legacy trials

* do not display grandfathering notice for legacy trials

* set a more accurate BT release date

* fix bug on dev env

Allow the `find/1` function to find sandbox plans

* add error handling and tests for change_plan_preview

* fix feature warning bug

* fix credo warnings

* fix tests

* set BT release date further into the future

* rename function and some vars

* bugfix with limit exceeding

* fix test
3 years ago
Adrian Gruntkowski 07cab27fef
Implement new sites view (#3463)
* Implement complete basics of LV sites

* Reimplement everything in LV except pagination

* Implement basic search capability

* PoC: plot visitors on sites index

* Add rudimentary clipped gradient in minicharts

* Fix clipping gradient, define once

* Format

* Add moduledoc to visitors component

* Move paginator helpers to the top core namespace

* Fix typespec of `Plausible.Sites.list`

* Split sites component into subcomponents

* Add function to uniformly calculate 24h intervals
and visitor totals across multiple sites.

* Integrate batch 24h interval query with plots on sites view

* Don't confuse heex compiler with alpine @ shorthands

* Make linear gradient svg definition truly invisible

* Implement basic pagination

* Extract `site_stats` from site and invitation cards

* Improve pagination

* Tweak css

* Improve filtering on pagination and make WSS fail graceful

* Test `last_24h_visitors_hourly_intervals/2`

* Replace /sites with LV implementation

* Add debounce to search filter

* Fix typespecs

* Fix styling

* Fix mini graph scaling factor calculation

* Fix search consuming itself

* Minimal tweaks to the plots

* Fixup

* Remove magic numbers from the plot

* Create `site_pins` table

* Add `SitePin` schema

* Implement listing invitations, sites and pins in a single query

* Add FIXME note

* Remove site pins for now

* Add tests for `Plausible.Sites.list/3`

* Add a couple more tests to sites dead view

* Remove unnecessary FIXME

* Add LV tests for Sites

* Calculate and display 24h visitors change

* Render the change in bold

* Add clarfying comment on virtual field in `Site` schema

* Remove unnecessary function from Invitations API

* Remove unused list opt from type definition in `Sites`

* Improve joins in list query slightly

* Add comment on manually computing sites list total

* Start searching from a singly character in domain field

* Add typespec to `last_24h_visitors_hourly_intervals`

* Extend moduledoc in visitors component

* Simplify loading sites in LV

* Simplify assigns in LV

* Add missing group for shadow under site card

* Make invitation modal render

* Make HTML in sites LV semantically correct

* Remove autofocus and focus search on `/`

* Remove shadow from search input

* Make search cancel on escape

* Fix tests relying on outdated HTML structure

* Make visitor chart color scheme consistent with dashboard chart

* Update styling of trend labels

* Fix empty state and improve search blur/focus handling

* Use live navigation for pagination

* Implement spinner on load from search

* Remove unused `Plausible.Stats.Clickhouse.last_24h_visitors/1`

* Calculate uniques correctly across hour boundaries

* Swap inlined svg for Heroicons component in invitation modal

* Add order by to base query in 24h hourly intervals

* Revert "Add order by to base query in 24h hourly intervals"

This reverts commit a6be5e3026.

* Query clickhouse 24h visitors only on second mount

* Remove redundant sign from percentage change when negative

* Switch to offset-based pagination

  - offset seems easier to deal with for when actions on
    paginated list will be performed such as site pinning;
    tracking cursor data makes some entries disappear in
    edge cases. The data set is still fairly small and
    static, even for large customers.
  - we're removing Phoenix.Pagination as it doesn't really
    fir any use case, and it was only used to limit the number
    of sites in the site picker
  - site picker is now limited to 9 sites (future: pinned
    sites will be prioritized there)
  - no need to re-query for total count any more
  - BTW, the old /sites template was removed

* Refine the plot queries; Tests pass snapshot

* Add PromEx plugin for LiveView

* Fix tiny plot cut-off at the top

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
3 years ago
Vini Brasil ad05af08a4
Check for limits and features used by site before transferring ownership (#3445)
* Simplify team_member_usage query

* Check limits before transferring ownership

* Extract invite creation to dedicated service module

* Simplify team member usage query

* Remove unnecessary distinct clause

* Delegate CreateInvitation via Memberships
3 years ago
hq1 117eef000d
Upgrade Erlang/Elixir stack (#3454)
* Bump deps

* Bump stack

* Fix deprecation warnings

* Fix VCR cassettes mismatch due to OTP-18414

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Format & fix flaky tests

* Handle raw IPv4 hostnames; test public suffix TLD

* Configure locus db cache_dir

So that maxmind unavailability doesn't affect
application startup. PERSISTENT_CACHE_DIR env var is used
to point locus at the GeoIP DB file.

* WIP: Remove ExVCR

* Fix test env config

* Fixup exvcr

* Remove exvcr from deps

* Add convert script

* Remove exvcr cassettes

* Remove convert script

* Rename test

* Update moduledoc

* Update dockerfile

* Bump CI cache

* Tag more slow tests, why not?

* Use charlist for locus cache option

* Pin nodejs

* Merge google tests, make them async

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
Adrian Gruntkowski 70c001099d
Improve and simplify email verification codes generation (#3407)
* Refactor email verification codes generation to avoid predictability

* Improve `Site.Memberships.any?` slightly

* Update tests

* Fix seeds

* Use `expired?` predicate for checking verification code validity in tests

* Store verification code as string in database to avoid unnecessary int casting
3 years ago
Adrian Gruntkowski f0c749ad70
Implement direct site ownership transfer without invite via CRM (#3416) 3 years ago
Adrian Gruntkowski 439c5014d4
Trigger email reverification on change (#3388)
* Implement PoC for email reverification flow on update

* Improve user settings form and email change validation

* Expose `previous_email` in Kaffy CRM

* Improve plugs setup and remove dead action from AuthController

* Fix seeds

* Extract predicate query functions from AuthController

* Add tests

* Update CHANGELOG.md

* Rename `has_any_sites?` to `Memberships.any?` and `has_any_memberships?`

* Improve flash message on cancelling email change

* Cover one more test case for email update
3 years ago
Adrian Gruntkowski 842bbb7995
Fix invalid subscription preload in SiteLocker email submission routine (#3400) 3 years ago
hq1 0007c0c108
Plugins API: 2nd pass with Goals resource + SharedLinks schema changes (#3396)
* Remove "Context" namespace level

* Change Goal string representation

* Alias Schemas in Plugin API Test Case template

* Update schema & tests for SharedLink resource

* Update Goals interface

- make it possible to create revenue goals
- extract "for site" query to a standalone function

* Fixup typespecs

* Alias Errors module in OpenAPI controllers

* Add missing goals test

* Implement Goals Plugins API resource

* Add extra test to confirm changeset error propagation

* Mute credo

* Fix typos

* Handle changeset traversal in `Errors`

* Use upserts in `Goals.find_or_create`

* Extract touch_site! to Site.Cache, address credo, improve code docs

* Apply formatting

* Remove unused inner join

* Update test/plausible_web/plugins/api/controllers/goals_test.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update test/plausible_web/plugins/api/controllers/goals_test.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update test/plausible_web/plugins/api/controllers/goals_test.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update test/plausible_web/plugins/api/controllers/goals_test.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update test/plausible_web/plugins/api/controllers/goals_test.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update error message on revenue goal currency clash

* Remove unused code

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
Adrian Gruntkowski e67850c11d
Fix and refactor invitation logic (#3376)
* Make membership creation and role updates more explicit in terms of changesets

* Extract invitation accept flow logic and refactor it slightly

* Improve acceptation logic

* Update moduledoc

* Improve SiteLocker API and add typespecs

* Stop naming function not returning a boolean like a predicate

* Refactor rest of invitation actions and safeguard against rogue requests

* Update code docs slightly

* Extend `Billing.check_needs_update/1` tests

* Parametrize selfhost flag and toggle SiteLocker logic on it

* Add tests for newly extracted services

* Add test case and a fix for locking site on grace period ended

* Make invitation controller tests async as there's no more env patching

* Add test cases for self-invites and fix one bug

* Add and refactor tests for rejecting and removing invitations

* Prevent issuing ownership transfer to existing owner

* Improve name of the test

* Improve `Billing.check_needs_to_upgrade/1` return value

* Improve `Billing.SiteLocker.update_sites_for/1` and its tests

* Fix typos

Co-authored-by: hq1 <hq@mtod.org>

* Make invitation removal and rejection resilient to races

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
hq1 082ec91c63
OpenAPI: first pass on Plugins API - Shared Links (#3378)
* Update depenedencies: OpenAPISpex + cursor based pagination

* Update formatter config

* Add internal server error implementation

* Test errors

* Implement pagination interface

* Implement Plugins API module macros

* Implement Public API base URI

(to be used with path helpers once called from within
forwarded router's scope)

* Implement OpenAPI specs + schemas

* Implement Shared Links context module

* Add pagination and error views

* Add Shared Link view

* Implement Shared Link controller

* Expose SharedLink.t() spec

* Implement separate router for the Plugins API

* Update moduledocs

* Always wrap resource objects with `data`

* Update moduledoc

* Use https://github.com/open-api-spex/open_api_spex/pull/425

due to https://github.com/open-api-spex/open_api_spex/issues/92

* Rely on BASE_URL for swagger-ui server definition

* Fixup goals migration

* Migrate broken goals before deleting dupes

* Remove bypassing test rate limiting for which there's none anyway

* Move the context module under `Plausible.` namespace

* Bring back conn assignment to PluginsAPICase template

* Update test/plausible_web/plugins/api/controllers/shared_links_test.exs

Co-authored-by: Uku Taht <Uku.taht@gmail.com>

* Update renamed aliases

* Seed static token for development purposes

* Delegate Plugins API 500s to a familiar shape

* Simplify with statement

---------

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
3 years ago
Uku Taht 7cb2c6bfa3
Allow admins to initiate ownership transfer from the CRM (#3340)
* Allow admins to initiate ownership transfer from the CRM

* Add stronger assertion for bulk invite action

* Fix compile warning

* Move bulk transfer logic to Sites module

* Replaces unused variables with _

* Add typespec for `bulk_transfer_ownership`

* Extract from keywordlist options instead of matching

* Fix and extend bulk transfer tests

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
RobertJoonas 5ba21c04fe
Remove Props FF (#3242)
* stop returning prop_names from /conversions

* remove the old prop-breakdown API route and CSV export

* remove allowed_event_props from CRM

* remove DeprecatedConversions

* remove props FF

* update changelog
3 years ago
Vini Brasil 8c3b541971
Allow internal prop keys in queries (#3231)
* Remove `Plausible.Sites.set_allowed_event_props/2` function

This commit removes the `Plausible.Sites.set_allowed_event_props/2`
function in favor of `Plausible.Props.allow/2`.

* Allow internal prop keys in queries

This commit allows internal prop keys by default in queries. I decided
not to include internal keys in the database, because they might change
and we'd need to migrate it again.

* remove redundant 'props.' from behaviours/index.js

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
3 years ago
hq1 2f9103fe1b
Fix unique constraint validation on weekly/monthly reports (#3200)
* Fix unique constraint validation on weekly/monthly reports

* Undo this annoying lsp formatter race condition
3 years ago
RobertJoonas 51cad4c0ec
Props into CSV export (#3167)
* move set_allowed_event_props into context module

* extract parse_csv fn

* add custom_props.csv to CSV export (under FF)

* add test for custom prop export with a prop filter

* add test for goal filtered scenario

* mix format

* fix flaky test

* extract breakdown_custom_prop_values function

* Revert "fix flaky test"

This reverts commit 256b9bb2e7. The flaky
test was already fixed in 86557b1878

* make set_allowed_event_props a bang function

* test that all files are included in the CSV export
3 years ago
Vini Brasil 8834486a19
Pass cached site struct down the ingestion pipeline (#3027)
* Pass cached site struct down the ingestion pipeline

Revenue goals need the cached site struct during ingestion to get the
goals name and currency. This cache lookup is not necessary as
`GateKeeper.check/1`, which is called first in the ingestion pipeline,
could already return the site struct from the cache.

This commit changes `GateKeeper.check/1` to return the site struct
instead of the site ID. Moreover, this commit changes the ingestion
pipeline to avoid calling the sites cache twice.

Related: https://github.com/plausible/analytics/pull/2957#discussion_r1203921549

* Remove revenue_goals unnecessary fallback

* Change duplicate child_id in cache test

* Remove revenue goal condition from cache query

* Remove Plausible.DataCase.reload/1
3 years ago
Vini Brasil 10d9e3b083
Revenue tracking: Add currency field to goal creation (#2948)
* Add revenue goal option to goal creation

This commit adds a currency field to the goals form. Goals that have a
currency set are now revenue goals, and are cached with sites to later
be used during ingestion.

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

* Enable feature flag in tests

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
4 years ago
RobertJoonas db359c2a6f
Filter out garbage props by `site.allowed_event_props` (#2907)
* add migration

* add not null constraint

* remove not null constraint and default value

* add site schema change

* filter garbage props for prop_key suggestions

* filter garbage props returned by StatsController.conversions/2

* add crm action to set prop allowlist

* use nullable allowed_event_props field instead

* filter garbage props in db query instead

* implement suggestion dealing with string input

* refactor the allowlist condition in db query

* improve test
4 years ago
hq1 825a754976
Make ingest threshold configurable (#2845)
* Make ingest threshold configurable

* Credo
4 years ago
hq1 154ce3a44c
Split clickhouse repos - making the main one read only (#2826)
* Split clickhouse pools into readonly/import deletions

* Remove CRM site transfers

* Initialize ImportDeletionRepo

* Put ImportDeletionRepo to use
4 years ago
hq1 1d01328287
Allow domain change (#2803)
* Migration (PR: https://github.com/plausible/analytics/pull/2802)

* Implement Site.Domain interface allowing change and expiry

* Fixup seeds so they work with V2_MIGRATION_DONE=1

* Update Sites.Cache so it's capable of multi-keyed lookups

* Implement worker handling domain change expiration

* Implement domain change UI

* Implement transition period for public APIs

* Exclude v2 tests in primary test run

* Update lib/plausible_web/controllers/site_controller.ex

Co-authored-by: Vini Brasil <vini@hey.com>

* Update lib/plausible_web/controllers/site_controller.ex

Co-authored-by: Vini Brasil <vini@hey.com>

* Update moduledoc

* Update changelog

* Remove remnant from previous implementation attempt

* !fixup

* !fixup

* Implement domain change via Sites API

cc @ukutaht

* Update CHANGELOG

* Credo

* !fixup commit missing tests

* Allow continuous domain change within the same site

---------

Co-authored-by: Vini Brasil <vini@hey.com>
4 years ago
hq1 d2f2c69387
Conditionally support switching between v1 and v2 clickhouse schemas (#2780)
* Remove ClickhouseSetup module

This has been an implicit point of contact to many
tests. From now on the goal is for each test to maintain
its own, isolated setup so that no accidental clashes
and implicit assumptions are relied upon.

* Implement v2 schema check

An environment variable V2_MIGRATION_DONE acts like
a feature flag, switching plausible from using old events/sessions
schemas to v2 schemas introduced by NumericIDs migration.

* Run both test suites sequentially

While the code for v1 and v2 schemas must be kept still,
we will from now on run tests against both code paths.
Secondary test run will set V2_MIGRATION_DONE=1 variable,
thus making all `Plausible.v2?()` checks return `true'.

* Remove unused function

This is a remnant from the short period when
we would check for existing events before allowing
creating a new site.

* Update test setups/factories with v2 migration check

* Make GateKeeper return site id along with :allow

* Make Billing module check for v2 schema

* Make ingestion aware of v2 schema

* Disable site transfers for when v2 is live

In a separate changeset we will implement simplified
site transfer for when v2 migration is complete.
The new transfer will only rename the site domain in postgres
and keep track of the original site prior to the transfer
so we keep an ingestion grace period until the customers
redeploy their scripting.

* Make Stats base queries aware of v2 schema switch

* Update breakdown with v2 conditionals

* Update pageview local start with v2 check

* Update current visitoris with v2 check

* Update stats controller with v2 checks

* Update external controller with v2 checks

* Update remaining tests with proper fixtures

* Rewrite redundant assignment

* Remove unused alias

* Mute credo, this is not the right time

* Add test_helper prompt

* Fetch priv dir so it works with a release

* Fetch distinct partitions only

* Don't limit inspect output for partitions

* Ensure SQL is printed to IO

* Remove redundant domain fixture
4 years ago
Adam 736e6e385c
Capture no_cache sentry messages together (#2772) 4 years ago
Uku Taht f26ca7da9f
Update native stats start date when transferring (#2719) 4 years ago
Adam 8f86036e57
Keep track of native stats start timestamp when retrieving data (#2715)
* Stats boundary/PoC?

* Delete stats removal

* Drop events check on site creation

* Update seeds script

* Use native_stats_start_at

* Don't rely on native stats pointer in imported stats queries

* Reset site

* Export reset/1

* Remove unnecessary inserted_at settings

* Update seeds

* Remove unnecessary inserted_at setting
4 years ago
Adam Rutkowski 867dad6da7
Implement ingest counters (#2693)
* Clickhouse migration: add ingest_counters table

* Configure ingest counters per MIX_ENV

* Emit telemetry for ingest events with rich metadata

* Allow building Request.t() with fake now() - for testing purposes

* Use clickhousex branch where session_id is assigned to each connection

* Add helper function for getting site id via cache

* Add Ecto schema for `ingest_counters` table

* Implement metrics buffer

* Implement buffering handler for `Plausible.Ingestion.Event` telemetry

* Implement periodic metrics aggregation

* Update counters docs

* Add toStartOfMinute() to ordering key

* Reset the sync connection state in `after` clause

* Flush counters on app termination

* Use separate Repo with async settings enabled at config level

* Switch to clickhouse_settings repo root config key

* Add AsyncInsertRepo module
4 years ago
Adam Rutkowski ecb552479f
Upgrade erlang/elixir/alpine stack (#2695)
* Dockerfile: pin elixir-1.14.3, erlang-25.2.3, alpine-3.17.0

* asdf: pin erlang 25.2.3, elixir 1.14.3-otp-25

* Remove nonsense

* Remove erlang dupe

* Remove erlang, it's in the base image already

* Remove empty file

* Truncate seconds in NaiveDateTime instances

* Bump cache key

* Mitigate Error:error:0308010C:digital envelope routines::unsupported

See: https://www.newline.co/@kchan/how-to-fix-the-error-errorerror0308010cdigital-envelope-routinesunsupported--0f8d3f17

* Upgrade to latest alpine image and pin to an exact digest

* Revert "Upgrade to latest alpine image and pin to an exact digest"

This reverts commit cdf6624efa.

* Pin alpine image to an exact digest

---------

Co-authored-by: Cenk Kücük <c@cenk.me>
4 years ago
Adam Rutkowski 8f9f032968
Delay stats deletions (#2632)
* Implement Site removal transaction

* Implement Stats removal Oban worker

* Configure site removal queue

* Call Site.Removal.run() instead of Purge.delete_site!

* Test site/stats removal

* Remove FIXME - filed a ticket

* Over-communicate lenghty deletion process to the users
4 years ago
Adam Rutkowski 467317f24e
Fix: application startup, event timestamps (#2479)
### Changes

This PR:
- pushes PromEx to the bottom of supervision stack to avoid Endpoint
instrumentation failure
- ensures the site cache is ready by exposing it through the health
check endpoint
- fixes event timestamps being calculated at compile time, with
regression unit and integration tests

### Tests
- [x] Automated tests have been added
- [ ] This PR does not require tests

### Changelog
- [ ] Entry has been added to changelog
- [x] This PR does not make a user-facing change

### Documentation
- [ ] [Docs](https://github.com/plausible/docs) have been updated
- [x] This change does not need a documentation update

### Dark mode
- [ ] The UI has been tested both in dark and light mode
- [x] This PR does not change the UI
4 years ago
Adam Rutkowski 356575ef78
Gatekeep ingestion pipeline (#2472)
* Update Sites.Cache

So it's now capable of refreshing most recent sites.
Refreshing a single site is no longer wanted.

* Introduce Warmer.RecentlyUpdated

This is Sites Cache warmer that runs only for
most recently updated sites every 30s.

* Validate Request creation early

* Rename RateLimiter to GateKeeper and introduce detailed policies

* Update events API tests - a provisioned site is now required

* Update events ingestion tests

* Make limits visible in CRM Sites index

* Hard-deprecate DOMAIN_BLACKLIST

* Remove unnecessary clause

* Fix typo

* Explicitly delegate Warmer.All

* GateKeeper.allwoance => GateKeeper.check

* Instrument Sites.Cache measurments

* Update send_pageview task to output response headers

* Instrument ingestion pipeline

* Credo

* Make event telemetry test a sync case

* Simplify Request.uri/hostname handling

* Use embedded schema, apply action and rely on get_field
4 years ago