Commit Graph

176 Commits (db5488efa677563cc3fce874f72ccb4e7c939708)

Author SHA1 Message Date
Vini Brasil b97290b5cf
Add revenue metrics to top stats (#3059)
This commit adds revenue data to top stats. Average and total are displayed when filtering by a revenue goal (or many if they have the same currency set).
3 years ago
hq1 a356009af8
Implement backend core for funnels (#3023)
* Implement Funnel schema

* Implement Funnels context module

lib/plausible/funnels.ex - restored from:

dca93621 Mark Vini's suggestion applied
7d1eec75 Update lib/plausible/funnels.ex
028036ad Review comments
b694ccd5 Express the guard nicely
acd9c4f2 Prepare ephemeral funnel definitions so that users can test funnels
aea4ebc4 Access Funnel min/max steps via the __using__/1 macro
51f0397d Implement deleting funnels
cf228630 wip
1f6fe25d Add number of steps to funnels list
89f10ecb wip
950a18d9 Dirty funnel save
e62bfce5 Refactor funnel result evaluation
88e38a33 Update funnel tests
bf47f654 Calculate conversion rate/dropoff in the backend
9e62ffb1 Make existing tests pass
7a88fe44 Outline basic error handling
c8ae3eaf Move Funnels to StatsController and use base query
2929bfd4 Minor rework
b490403b !ifxup
ae44db77 Add Funnels.list/1 function
4f3216b3 Unique steps
6e0c3efc Ensure step ordering works
cea93ba1 It works I think
c50ac0c2 WIP
a466ec4c Dirty wip
fbd14e23 Absolute dirty wip
3ddd15ad wip
34ceb3f5 dirty wip

test/plausible_web/controllers/api/stats_controller/funnels_test.exs - restored from:

9b532273 Test funnel stats controller

Co-authored-by: Uku Taht <uku.taht@gmail.com>

* Update Goals implementation

lib/plausible/goal/schema.ex - restored from:

028036ad Review comments
acd9c4f2 Prepare ephemeral funnel definitions so that users can test funnels
4a4ddbdc Optionally preload funnels for goals and stub funnel-goal deletion
fbd14e23 Absolute dirty wip

lib/plausible/goals.ex - restored from:

aea4ebc4 Access Funnel min/max steps via the __using__/1 macro
a418c039 Implement funnel reduction logic on goal deletion
4a4ddbdc Optionally preload funnels for goals and stub funnel-goal deletion
ebdc4333 Extend the prompt in case of funnel-goal deletion
950a18d9 Dirty funnel save
f06c3fcf wip

test/plausible/goals_test.exs - restored from:

a418c039 Implement funnel reduction logic on goal deletion
4a4ddbdc Optionally preload funnels for goals and stub funnel-goal deletion
dee5bc9e Fixup tests

* Update seeds

* Split funnel schema modules

* Add moduledoc to Funnel schema

* Update Funnel.Step moduledoc

* Add typespecs to Plausible.Funnels

* Implement Phoenix.HTML.Safe for Funnel schema

* Fixup a test after ordering goals by desc:id

test/plausible_web/controllers/api/stats_controller/conversions_test.exs - restored from:

26a493cb Fixup a test after ordering goals by desc:id

* Annotate Goal schema with type t()

* Reword schema module docs

* Fix up typespecs

* Stop tagging Funnels.get/2 result

* Nuke Goals.by_id!/2

* Test Funnels module

test/plausible/funnels_test.exs - restored from:

b7a6d3ad ya, rly
028036ad Review comments
acd9c4f2 Prepare ephemeral funnel definitions so that users can test funnels
51f0397d Implement deleting funnels
1f6fe25d Add number of steps to funnels list
dee5bc9e Fixup tests
950a18d9 Dirty funnel save
88e38a33 Update funnel tests
bf47f654 Calculate conversion rate/dropoff in the backend
9e62ffb1 Make existing tests pass
2929bfd4 Minor rework
b490403b !ifxup
ae44db77 Add Funnels.list/1 function
4f3216b3 Unique steps
6e0c3efc Ensure step ordering works
cea93ba1 It works I think
c50ac0c2 WIP
a466ec4c Dirty wip
34ceb3f5 dirty wip

Co-authored-by: Uku Taht <uku.taht@gmail.com>

* Apply changes related to 242081e5

* Evaluate funnels through `Stats.funnel/3` entry point

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
3 years ago
hq1 3dce3142bf
Migration: add funnel related tables (#3022)
priv/repo/migrations/20230417095029_init_funnels.exs - restored from:

aa59adeb wip
4d039ba2 Make funnel names unique
4f3216b3 Unique steps
e55edf8e Rename funnel_goals->funnel_steps
34ceb3f5 dirty wip
3 years ago
Vini Brasil d98242895b
Add revenue fields to events_v2 (#3018) 3 years ago
RobertJoonas 5162da473c
add migration (#2988) 3 years ago
ruslandoga f489d96251
skip v1 table drop in self-host (#2945) 4 years ago
Vini Brasil 3a07487c05
Add currency field to goals table (#2955) 4 years ago
Uku Taht ca3e2e5121
Add unique constraint to api keys (#2931)
* Add unique constraint to api keys

* Fix test

* Remove `user_id` from api key index
4 years ago
hq1 57af1f19ec
Drop events/sessions tables after V2 migration (#2908) 4 years ago
RobertJoonas 412bf1959e
Postgres migration to add `allowed_event_props` field to `site` (#2894)
* add migration

* add not null constraint

* remove not null constraint and default value
4 years ago
Vini Brasil fecbc8b455
Fix conversions top stats bug when comparing (#2893)
This commit fixes a bug where unique conversions and the unique visitors
of the previous period in comparison view were always the same.

First reported at: https://github.com/plausible/analytics/discussions/532#discussioncomment-5760187
4 years ago
Vini Brasil 8dc8423319
Prepare Comparisons for release (#2873)
This pull request removes the comparisons feature flag, making it visible to all users. It also removes percentages from top stats in default view, leaving those for the comparison view only.
4 years ago
Vini Brasil 8ec3dd402f
Prevent overwriting with_imported when comparing (#2871)
This commit fixes a bug where comparisons would not work with imported data. This is because comparisons copies the source query and modifies the dates only, where it should reevaluate if there is imported data for the new comparison date range.

Closes #2870

Co-authored-by: Adam <hq@mtod.org>
4 years ago
hq1 71ef0bd043
Clean up after V2 migration (#2868)
* Clean up after V2 migration

This PR removes all the leftovers and alternative code
branching after v2 migration.

The self-hosted release is being drafted at:

https://github.com/plausible/hosting/issues/68

Refs:
  - https://github.com/plausible/analytics/pull/2865
  - https://github.com/plausible/analytics/pull/2825
  - https://github.com/plausible/analytics/pull/2780

* !fixup
4 years ago
ruslandoga adcce15632
Make self-hosted data migration easier (#2865)
* default to v2

* allow N defaults in data migration prompt and custom messages

* join domains lookup

* remove duplicate test runs from ci (both are v2)
4 years ago
hq1 b9c2110472
V2 migration tweaks for self hosted release (#2825)
* Get rid of PASS_V2_SCHEMA_MIGRATION

* Use in-memory domain lookup + regular table settings

* Remove faulty date arithmetic + prev part calculation

* Set V2_MIGRATION_DONE in Mix.env == :dev

* Mute credo
4 years ago
hq1 c04e9286dd
Associate goals with sites, not domains (#2828)
* Revert "Rephrase error message"

This reverts commit f624443a96.

* Revert "Temporarily disable goal creation"

This reverts commit a091635b9d.

* Update ecto schema

* Make sure goal operations are per site

* Update tests

* Split postgres migrations
4 years ago
hq1 3cb089eab4
Migrate and freeze goals creation (#2833)
* Full migration (to be submitted separately)

* Do not remove `Goal.domain` just yet

* Do not make Goal.site not nullable just yet

* Temporarily disable goal creation

* Rephrase error message

* Add down migration
4 years ago
hq1 509c549c01
Update ua_inspector database (#2827)
* Update `ua_inspector` database

* Add changelog entry
4 years ago
hq1 1d01328287
Allow domain change (#2803)
* Migration (PR: https://github.com/plausible/analytics/pull/2802)

* Implement Site.Domain interface allowing change and expiry

* Fixup seeds so they work with V2_MIGRATION_DONE=1

* Update Sites.Cache so it's capable of multi-keyed lookups

* Implement worker handling domain change expiration

* Implement domain change UI

* Implement transition period for public APIs

* Exclude v2 tests in primary test run

* Update lib/plausible_web/controllers/site_controller.ex

Co-authored-by: Vini Brasil <vini@hey.com>

* Update lib/plausible_web/controllers/site_controller.ex

Co-authored-by: Vini Brasil <vini@hey.com>

* Update moduledoc

* Update changelog

* Remove remnant from previous implementation attempt

* !fixup

* !fixup

* Implement domain change via Sites API

cc @ukutaht

* Update CHANGELOG

* Credo

* !fixup commit missing tests

* Allow continuous domain change within the same site

---------

Co-authored-by: Vini Brasil <vini@hey.com>
4 years ago
hq1 94a86a17eb
Migration: enable domain changes (upcoming feature) (#2802)
* Migration: enable domain changes (upcoming feature)

* Pass domain check within the same site
4 years ago
hq1 d2f2c69387
Conditionally support switching between v1 and v2 clickhouse schemas (#2780)
* Remove ClickhouseSetup module

This has been an implicit point of contact to many
tests. From now on the goal is for each test to maintain
its own, isolated setup so that no accidental clashes
and implicit assumptions are relied upon.

* Implement v2 schema check

An environment variable V2_MIGRATION_DONE acts like
a feature flag, switching plausible from using old events/sessions
schemas to v2 schemas introduced by NumericIDs migration.

* Run both test suites sequentially

While the code for v1 and v2 schemas must be kept still,
we will from now on run tests against both code paths.
Secondary test run will set V2_MIGRATION_DONE=1 variable,
thus making all `Plausible.v2?()` checks return `true'.

* Remove unused function

This is a remnant from the short period when
we would check for existing events before allowing
creating a new site.

* Update test setups/factories with v2 migration check

* Make GateKeeper return site id along with :allow

* Make Billing module check for v2 schema

* Make ingestion aware of v2 schema

* Disable site transfers for when v2 is live

In a separate changeset we will implement simplified
site transfer for when v2 migration is complete.
The new transfer will only rename the site domain in postgres
and keep track of the original site prior to the transfer
so we keep an ingestion grace period until the customers
redeploy their scripting.

* Make Stats base queries aware of v2 schema switch

* Update breakdown with v2 conditionals

* Update pageview local start with v2 check

* Update current visitoris with v2 check

* Update stats controller with v2 checks

* Update external controller with v2 checks

* Update remaining tests with proper fixtures

* Rewrite redundant assignment

* Remove unused alias

* Mute credo, this is not the right time

* Add test_helper prompt

* Fetch priv dir so it works with a release

* Fetch distinct partitions only

* Don't limit inspect output for partitions

* Ensure SQL is printed to IO

* Remove redundant domain fixture
4 years ago
Adam 6637751a5e
Implement Numeric IDs migration (#2762)
* Implement Numeric IDs migration

* Fix typo

* Mute credo for now

* Improve configurability and add stop_t

* Adjust to Ch/Chto only

* Fix opts key for dictionary password

* Add regular ecto migration with numeric ids v2 schemas (#2768)

* Add regular ecto migration

* Fix typo

* Update priv/ingest_repo/migrations/20230320094327_create_v2_schemas.exs

Co-authored-by: Vini Brasil <vini@hey.com>

* Implement v2 events/sessions schema modules (#2777)

* Implement v2 events/sessions schema modules

* Clean up session schemas

---------

Co-authored-by: Vini Brasil <vini@hey.com>

* Update moduledocs

---------

Co-authored-by: Vini Brasil <vini@hey.com>
4 years ago
Adam baa04be191
Remove obsolete migration config (#2769)
The compilation warning advises to use `Application.compile_env/3`
however there is no `:url` environment setting under `:plausible` key
anyway, so we might as well get rid of it completely.
4 years ago
Adam 6d79ca5093
Switch to new clickhouse adapter (ch/chto) (#2733)
* another clickhouse adapter

* don't restore stats_removal.ex

* fix events main-graph error (#2746)

* update ch, chto

* update chto again (#2759)

* Stop treating page filter as an entry_page filter (#2752)

* remove dead code

* stop treating page filter as entry page filter in breakdown queries

* stop treating page filter as entry page filter in aggregate queries

* stop treating page filter as entry page filter in timeseries queries

* mix format

* update changelog

* break code down to smaller functions to keep credo happy

* remove unused functions

* make CSV export return only conversions with goal filter (#2760)

* make CSV export return only conversions with goal filter

* update changelog

* update elixir version in mix.exs (#2742)

* revert admin.ex changes (#2776)

---------

Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
Co-authored-by: ruslandoga <rusl@n-do.ga>
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
4 years ago
Adam 8f86036e57
Keep track of native stats start timestamp when retrieving data (#2715)
* Stats boundary/PoC?

* Delete stats removal

* Drop events check on site creation

* Update seeds script

* Use native_stats_start_at

* Don't rely on native stats pointer in imported stats queries

* Reset site

* Export reset/1

* Remove unnecessary inserted_at settings

* Update seeds

* Remove unnecessary inserted_at setting
4 years ago
Adam 05e7f93da2
Add a migration for native_stats_start_at (#2716) 4 years ago
Adam Rutkowski 043e3ed572
Clickhouse migration: add ingest_counters table (#2692)
* Clickhouse migration: add ingest_counters table

* Add toStartOfMinute() to ordering key

* Explicitly include column to be summarized
4 years ago
Vini Brasil ce03b5ebd7
Move CH migration from clickhouse_repo/ to ingest_repo/ (#2683)
This commit moves a migration that was created in `clickhouse_repo/` to
its correct folder, `ingest_repo/`. The migration was created in
1cb07efe6d prior the read/write
separation.
4 years ago
Vini Brasil 1cb07efe6d
Save city name when importing from GA (#2608)
This commit adds city data to imported records from Google Analytics. The
current implementation sets city to 0 because GA does not use the GeoNames
database.

Google Analytics Reporting API uses [Geographical IDs](https://developers.google.com/analytics/devguides/collection/protocol/v1/geoid)
to identify cities and countries. Plausible uses
[GeoNames](https://geonames.org/) and I couldn't find databases corelating the
two.

Fortunately, GA also returns the city name and this commit uses the city name
and the country ISO code to find the Geoname ID. To avoid making expensive ETS
searches, I created another ETS table in the Location library that uses
{country, city} as a key.

Related PR: https://github.com/plausible/location/pull/3
4 years ago
Adam Rutkowski 8f85b110aa
Split Clickhouse pools into Read-Only and Read/Write (dedicated to writes) (#2661)
* Configure ingest repo access/pool size

If I'm not mistaken 3 is a sane default, the only
inserts we're doing are:

  - session buffer dump
  - events buffer dump
  - GA import dump

And all are serializable within their scopes?

* Add IngestRepo

* Start IngestRepo

* Use IngestRepo for inserts

* Annotate ClickhouseRepo as read_only

So no insert* functions are expanded

* Update moduledoc

* rename alias

* Fix default env var value so it can be casted

* Use IngestRepo for migrations

* Set default ingest pool size from 3 to 5

in case conns are restarting or else...

* Ensure all Repo prometheus metrics are collected
4 years ago
Vini Brasil a730763838
Add city name to imported_locations table (#2633)
See also #2608
4 years ago
Vini Brasil 1b9e6d9ae5
Add city geolocation data to seeds (#2626) 4 years ago
Vini Brasil b6d30019ef
Cascade delete sent_renewal_notifications table when user is deleted (#2549)
* Cascade delete sent_renewal_notifications table when user is deleted

This commit fixes a bug when deleting a user would trigger a constraint
error.

* Update CHANGELOG.md
4 years ago
Vini Brasil b9367941f0
Add more data to pageview seeds (#2471) 4 years ago
Adam Rutkowski 8b5ae9baaa
Migration: index Sites.updated_at (#2467) 4 years ago
Vini Brasil 3bedf9281c
Seed database with pageviews (#2449)
* Seed database with pageviews

This commit adds basic support for database seeding useful for testing,
especially dashboard changes, like intervals.

It creates two years of pageviews with random timestamps. There is lot
of room for improvement, such as adding sources, entry pages,
geolocation, devices, custom events, but this already helps us with
testing.

* Update CONTRIBUTING.md file
4 years ago
Adam Rutkowski aa248dde2e
Add sites metadata rate limit migration (#2428)
* Generate ecto migration for rate limiting sites

* Add ingest rate limit meta-data migration

* Make scale limit seconds not nullable
4 years ago
RobertJoonas c0da024b23
Remove static tracker files (#2116)
* remove tracker files from git index

* generate tracker files on npm test

* generate tracker files for elixir tests/dev/CI

* update tracker/package-lock.json

* exclude npm run deploy from mix test + some docs
4 years ago
RobertJoonas f75d5106f0
Rework outbound links and file downloads (#2208)
* moved custom event code to the bottom + fix indentation

* add handlebars helper fn + extract getLinkEl fn

* extract isOutboundLink function

* extract shouldFollowLink function

* remove middle and click variables

* use only one click handler for outbounds and downloads

* extract sendLinkClickEvent function

* add error handling when script compilation fails

* use callback instead of fixed timeout

* do not prevent default if externally prevented + test

* add more tests

* generate tracker files in priv/tracker/js

* update changelog

* requested changes after review

* regenerate tracker files

* use return instead of else if

* move middleMouseButton outside the function
4 years ago
Uku Taht d104abb53d
Add fallback for favicon (#2279)
* Add fallback for favicon

* Add Favicon tests

* Changelog

* Move placeholder icon to priv folder
4 years ago
RobertJoonas 39ce850f18
exclude pages by hash when using `script.exclusions.hash.js` (#2172)
* account for hash part of the URL when excluding pages and using hash extension

* changelog update
4 years ago
Uku Taht 99fd101135
Add basic test harness for browserstack/playwright (#1961)
* Add basic test harness for browserstack/playwright

* Refactor the tests

* added the first test for outbound links

* tests for outbound-links and file-downloads

* added more browser versions to test on

* Lint tracker test files

* Update harness.js with BrowserStack example

* Fix Playwright request mocks

* Add test harness to CI

* Remove Safari on Windows from browsers list

Co-authored-by: Robert <robertjoonas16@gmail.com>
Co-authored-by: Vinicius Brasil <vini@hey.com>
4 years ago
Uku Taht 7e658210ba Remove db-ip city lite 5 years ago
RobertJoonas 11654ddc07
Script extension additions (#1915)
* added data-include attribute to plausible.exclusions.js

* reorder extensions in filename when serving the plausible script

* fix formatting

* tweaks after review

* changelog update
5 years ago
Uku Taht 18e2711556 Package new db-ip library in the git repo 5 years ago
RobertJoonas 199206babc
Dimensions continued (#1847)
* added the first version of dimensions extension

* finished dimensions script extension + updated tracking to use it

* script variants build
5 years ago
Uku Taht 5e415c2420 Add entry_props back in 5 years ago
Uku Taht 8fb4f3f886 Revert entry props 5 years ago
RobertJoonas 40275b64d4
Pageview custom dimensions (#1816)
* added custom dimension filtering tests for pages

* first filter UI in place

* pages, entry pages and exit pages can be filtered by pageview props

* added tests for expected filtering behaviour

* fix dimension filter for sources + tests

* added is_not filtering functionality

* fixed formatting

* fixed admin_test

* added (none) as filter value + is_not filter type in UI

* added prefilling applied filter values and some UI tweaks

* added fetch options

* Make prop suggestions work with `props` filter

* Fix test

* Track login state internally

* Add CHANGELOG entry

Co-authored-by: Uku Taht <uku.taht@gmail.com>
5 years ago