Commit Graph

1621 Commits (fe0edb2e1e58a40841796f5616ff528bfaa8a196)

Author SHA1 Message Date
Karl-Aksel Puulmann 66b7d383d2 Unify behavior of entry/exit page hostnames with rest 3 years ago
Karl-Aksel Puulmann 54867fb05d Merge remote-tracking branch 'origin/master' into new-table-querying 3 years ago
hq1 6af80dd246
Filter by hostnames (#3963)
* CH Migration: exit/entry hostnames in sessions_v2

* Leave only exit_page_hostname, we already record hostnames

* Use ClickHouse DDL in favour of ecto so that cluster is included

* Compress with ZSTD(3)

* Expose Hostname filter in the dashboard dropdown

* Add `exit_page_hostname` to ClickHouse `sessions_v2` schema

* Start tracking hostname changes in sessions

* Implement hostname filter suggestions

* Enable filtering by `event:hostname`

* Add tests for filtering by hostnames

* Ensure filter suggestions work for exit pages too

* Allow overriding hostnames with `send_pageview` mix task

* Remove `:window_time_on_page` flag

It seems that we can remove it after all?

* Initialize `experimental_hostname_filter` query parameter

* Rewrite cache store behaviour with regards to session hostnames

* Work around inconsistent session merging

So that `populate_stats` can get closer to actual ingestion

* Improve top stats test

* Make it possible to filter sessions by entry/exit hostnames

* Update pages tests

* Expose `experimental_hostname_filtering` temporarily in the UI

* Untested yet: also apply experimental filtering to sources

* Introduce `hostname_filter` feature flag

* Format

* Test top sources with hostname filter + experimental flag
3 years ago
Adrian Gruntkowski e6d83e946f
Populate new columns in imports and exports (#3969)
* Extend `Imported*` schemas with newly added columns

* Populate newly added `Imported*` fields in GA4 imports

* Extend exports with newly added fields

* Extend CSV importer to ingest new fields

* Fix alias shadowing error

* Add more extensive GA4 import fixtures

* Apply rounding and casting to sampled visits
3 years ago
Karl-Aksel Puulmann 8e9b4a1840 Breakdowns retain old behavior if getting metric visitors 3 years ago
Karl-Aksel Puulmann 98012c2e31 Simplify code 3 years ago
Karl-Aksel Puulmann 6bb2740e99 Improve naming 3 years ago
Karl-Aksel Puulmann 3d70b4d077 both => sample_percent 3 years ago
ruslandoga 2ba4988c95
specify insert columns in import from s3 (#3968)
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
3 years ago
Karl-Aksel Puulmann 17a5a5e2c8 TableDecider tests 3 years ago
Karl-Aksel Puulmann cb97cc096c Remove debug code 3 years ago
Karl-Aksel Puulmann fba617c6f4 Handle percentage separately from other metrics 3 years ago
Karl-Aksel Puulmann a9a404ed09 moduledocs for TableDecider 3 years ago
Karl-Aksel Puulmann d3e8965c78 Add support for (plain) breakdowns deciding which table to use 3 years ago
Karl-Aksel Puulmann eed662fa40 events, pageviews are strictly event metrics 3 years ago
Karl-Aksel Puulmann eb8f1e2543 both -> either 3 years ago
Karl-Aksel Puulmann f5b3c7e026 Handle visits, pageviews, events and visitors metrics from other table 3 years ago
Karl-Aksel Puulmann 4ffb25dcaf Dont filter by session properties on events table if querying sessions and joining in events 3 years ago
Karl-Aksel Puulmann 83a6e244ef Make query_events private 3 years ago
Karl-Aksel Puulmann 7b61eaf047 Handle entry/exit page visit props separately from props stored in events table 3 years ago
Karl-Aksel Puulmann 3cb85d6b8f Fix assignment 3 years ago
Karl-Aksel Puulmann ffb4133cd6 Add some TODOs 3 years ago
Karl-Aksel Puulmann 450c63dd26 Simplify events_join_sessions? 3 years ago
Karl-Aksel Puulmann 2130bc2f3a Make default sessions join more conditional 3 years ago
Karl-Aksel Puulmann 6e415d118a Allow filtering by visit props on event queries if flag is on 3 years ago
Karl-Aksel Puulmann 38880e1401 Remove a hanging TODO 3 years ago
Karl-Aksel Puulmann d2a999b728 Include sample_percent in both tables 3 years ago
Karl-Aksel Puulmann 563da1a382 both -> either 3 years ago
Karl-Aksel Puulmann 960e15e25b WIP new querying system for deciding what tables to query 3 years ago
Karl-Aksel Puulmann 08211661cb Move experimental_session_count? logic to within query object 3 years ago
Adrian Gruntkowski 9f27fa303c
Fix dry run mode in `DataMigration.SiteImports` (#3965) 3 years ago
Adrian Gruntkowski 23a3699dd7
Improve import stats toggle and `with_imported` flag computation (#3960)
* Check import presence across all imports and not just the first one

Also, simplify imported data toggle rendering to not explicitly
refer to the earliest import source.

* Change imported stats toggle icon in dashboard

* Test `Imported.get_imports_date_range/1`

* Simplify failed UA/GA import email copy
3 years ago
Adrian Gruntkowski 71fe541359
Implement script for backfilling legacy site import entries and adjusting end dates of site imports (#3954)
* Always select and clear import ID 0 when referring to legacy imports

* Implement script for adding site import entries and adjusting end dates

* Log cases where end date computation is using fallback

* Don't log queries when running the migration to reduce noise
3 years ago
Adrian Gruntkowski 5bf59d1d8a
Implement adjusting imported date range to actual and existing stats (#3943)
* Implement adjusting imported date range to actual and existing stats

* Drop redundant prefix from import list entries

* Make pageview numbers in imports list formatted for readability

* Test and improve date range cropping

* DRY UA and GA4 stats start and end date API calls

* Extend UA/GA import controller tests and improve error handling

* refactor finding longest open range without existing data

* Fix typo in test description

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>

* Rename `open_ranges` to `free_ranges`

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
3 years ago
ruslandoga c263df5805
CSV imports (UI) (#3845)
* add basic ui

* remove TODO

* credo

* allow folder upload

* redirect external

* mention folder, use folder icon for file picker

* back to multiple file upload

* mention zip

* escape dots in archive filename
3 years ago
hq1 b31433a7bf
Ensure all the react container attributes are strings (#3948) 3 years ago
hq1 edf70d14b6
Use sessionStorage for "dashboard first launch" banner tracking (#3892)
* Use sessionStorage for offer e-mail report banner tracking

Keeping it within the cookie is problematic, as the banners don't
expire and overflow the cookie with data when enough new sites
are added.

Ref https://github.com/plausible/analytics/issues/3762

* Update changelog

* Extract a component

* Make is_dbip evaluate to quoted boolean
3 years ago
hq1 7523abe93e
Add metrics to ingestion pipeline (#3927)
* Add metrics to ingestion pipeline

* Format

* Format

* Update buckets

* Credo
3 years ago
Karl-Aksel Puulmann 4af7019011
Ignore sessions without entry/exit pages when breaking down entry/exit pages (#3933)
* Ignore sessions without entry/exit pages when breaking down entry/exit pages

* Update stats controller tests to have more realistic test data (pageview followed by event)
3 years ago
hq1 2fae0146a4
Reapply 3918 (#3940)
* Reapply "Pages shield (#3918)"

This reverts commit 33b5c10654.

* Make the FF check work against the site actor
3 years ago
hq1 9989ce6927
Migration for 3918 (#3939)
* Revert "Pages shield (#3918)"

This reverts commit 53f94a9f82.

* Migration: Shield page rules
3 years ago
hq1 53f94a9f82
Pages shield (#3918)
* Migration: Shield page rules

* Add Ecto schema for Page Rules

* Add Page Rule cache

* Fix typo

* BTW: Use already imported function

* Extend Shields context interface + split existing tests

* Ingestion: filter matching patches + refactor shield actions

* Add LV section for adding Page Rules

* Validate max page path length

* Put Pages Shield behind a feature flag

* Update CHANGELOG

* Update docs link anchor

As per https://github.com/plausible/docs/pull/477

* Update lib/plausible/shields.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible_web/live/shields/page_rules.ex

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Update lib/plausible_web/live/shields/page_rules.ex

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
3 years ago
Adrian Gruntkowski ba5b80a8c0
Add label to site imports and populate it (#3914) 3 years ago
RobertJoonas d6e1e8bebd
Put total conversions on the graph + goal-filtered CSV export improvements (#3929)
* Add validation for the events metric in main_graph

* Test the already existing events metric support in main-graph

* Put total conversions on the graph

* extract main_graph_csv function (refactor only)

* add total_conversions and conversion_rate to goal-filtered visitors.csv

* update changelog
3 years ago
Uku Taht fd879eeb16
Store referrers from android apps (#3715)
* Store referrers from android apps

* Add test for unknown referrer protocol

* Store android referrer protocol
3 years ago
RobertJoonas c32779a3e5
Timeseries for conversion rate (#3919)
* add conversion rate to Stats API timeseries

* make sure CR can be queried as the only metric

* add a test asserting zeros are returned

* add tests for filtering by other properties at the same time

* Remove unnecessary validation of params

1. It doesn't make to validate `interval` (and its granularity) in all
   endpoints. It's only relevant for the main graph.

2. The plug (renamed to `date_validation_plug`) already makes sure that
   the dates are validated. No need to call the same function again in
   Top Stats and Funnel endpoints.

* add metric validation to main graph

* Add tests for main graph API

* put conversion rate on the graph

* update changelog

* Add revenue metrics into metrics.ex

* make fn private

* avoid setting graph metric to visitors in goal-filtered view
3 years ago
Adrian Gruntkowski d6e81670e4
Unify UA and GA4 import flow into one (#3888)
* Unify GA4 and UA import flow into one

* Clean up property and view data retrieval via Google HTTP APIs

* Turn `Map.get` into `Map.fetch!` in API response processing code

* Bump list account summaries page size limit to max of 200

* Show only views in legacy flow and fix legacy redirect after import start

* Move google analytics import actions tests to a separate module

* Extend Google Analytics controller tests

* DRY up `property?` predicate (h/t @RobertJoonas)
3 years ago
ruslandoga 5f9465614b
Include domain and dates in zip archive filename (#3921)
* include domain and dates in zip archive filename

* adapt to comments
3 years ago
Karl-Aksel Puulmann 32ab138301
Fix issue with name clash (#3925)
Unexpectedly, table.name caused a name clash after CR refactor, so using a unique name
for the output column

Sentry issue: https://sentry.plausible.io/organizations/sentry/issues/5612
3 years ago
Karl-Aksel Puulmann c219652dae
Re-apply Move conversion_rate logic from elixir to clickhouse (#3924)
* Revert "Revert "Move conversion_rate logic from elixir to clickhouse (#3887)"…"

This reverts commit 253fb5d67d.

* Fix issue with missing columns

The issue came from refactoring event:goal UNION ALL logic and trying to move
name select from first to last. If any other tables were joined, the incorrect
item would be used as an array index, causing this issue.

Added a relevant test.
3 years ago