Commit Graph

48 Commits (db5488efa677563cc3fce874f72ccb4e7c939708)

Author SHA1 Message Date
Uku Taht 98055d2f1c
Fix queries with wildcard goals (#3015)
* Fix queries with wildcard goals

* Changelog
3 years ago
Vini Brasil e4d4f7d954
Revenue tracking: Ingestion and breakdown queries (#2957)
* Add revenue fields to ClickHouse events

This commit adds 4 fields to the ClickHouse events_v2 table:

* `revenue_source_amount` and `revenue_source_currency` store revenue in
  the original currency sent during ingestion

* `revenue_reporting_amount` and `revenue_reporting_currency` store
  revenue in a common currency to perform calculations, and this
  currency is defined by the user when setting up the goal

The type of amount fields is `Nullable(Decimal64(3))`. That covers all
fiat currencies and allows us to store huge amounts. Even though
ClickHouse does not suggest using `Nullable`, this is a good use case,
because otherwise additional work would have to be done to
differentiate missing values from real zeroes.

I ran a benchmark with the data pattern we expect in production, where
we have more missing values than real decimals. I created 100 million
records where 90% of decimals are missing. The difference between the
tables in storage is just 0.4Mb.

* Add revenue parameter to Events API

This commit adds support for sending revenue data in ingestion using the
`revenue` parameter - aliased to `$`.

* Add revenue parameter to mix send_pageview

* Add average and total revenue to breakdown queries
3 years ago
RobertJoonas dafb60164a
Allow setting sample_threshold in dashboard API queries (#2958)
* refactor Query.from/2

* allow specifying sample_threshold number in query
4 years ago
Uku Taht 6be67bc6c2
Carry _sample_factor from right side to left in join (#2962) 4 years ago
RobertJoonas 7d935b79bf
Treat page filter as entry page filter only for bounce rate (#2874)
* add entry page filter condition to bounce rate

* Fix select_merge with dynamic

* fix tests and add one test

* generalize page_filter_condition function

* use dynamic_filter_condition for session filters too

* disable views_per_visit with a page filter

* update changelog

* disable credo for complex function

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
4 years ago
RobertJoonas caeff41932
Prop filter modal (#2841)
* Add PropFilterModal (only UI)

* small variable refactor

* allow selecting prop value filter type

* allow selecting only one prop_key

* allow selecting many prop_values only when prop_key selected

* handle submitting filter

* get applied filters from query + remove option

* change prop filter label format

* support member and not_member filter types for pageview props

* show (none) value in filter suggestions

* refactor zip_results/4 and remove unused code

* fix displaying (none) values in goals section prop breakdown

* remove unnecessary functionality

* fix bug: returning prop names for goal :member filter

* fix bug: submitting regular filter modal with Enter key

* bugfix: disallow opening prop filter modal when feature flag disabled

* mix format

* break selected combobox values into multiple lines

* fix useEffect behavior for focusing on prop_key field

* support submitting prop filter with Enter key

* refactor getFormState in PropFilterModal

* separate fetchPropKey and fetchPropValue functions

* Allow querying props for pageview goals

* Make the internal props API only return a list of props (not map)
* Separate function for fetching all props in Stats API goal breakdown (this returns a map as before)

* ditch state for keeping search bar visible

* group by event_name in db query
4 years ago
hq1 71ef0bd043
Clean up after V2 migration (#2868)
* Clean up after V2 migration

This PR removes all the leftovers and alternative code
branching after v2 migration.

The self-hosted release is being drafted at:

https://github.com/plausible/hosting/issues/68

Refs:
  - https://github.com/plausible/analytics/pull/2865
  - https://github.com/plausible/analytics/pull/2825
  - https://github.com/plausible/analytics/pull/2780

* !fixup
4 years ago
Uku Taht 46048e50f7
Support multiple filters - frontend (#2773)
* Wrap Plausible.Stats.Filters with unit tests

* Parse `member` filter type

* Support for `member` filter in `aggregate_time_on_page`

* Support `not_member` filter type

* Support `matches_member` and `not_matches_member` filters

* Extract util module for React filters

* Implement Combobox from scratch with no libs

* Support multple filter clauses in combobox

* Don't use browser / os in version label

* Show highlighted option in combobox

* WIP

* Fix location filters outside filter modal

* Align open/close behaviour with react-select

* Styling updates for combobox

* Add support for wildcards in Combobox

* Implement keybindings for combobox

* Allow free choice inputs in combobox

* Rename 'Save filter' -> Apply filter

* Remove TODO comment

* Clean up some rebase mistakes

* Rename `allowWildcard` -> `freeChoice`

* Dark mode fixes

* Remove hint from filter modal

* Escape pipe character in filter modal

* Do not allow selecting duplicate options in combobox

* Escape brackets in `page_regex/1`

* Fix disabled style in dark mode

* Add regex fallback for safari

* Show no matches found when visibleOptions is empty

* Disable enter key when no visible options

* Do not submit empty form fields

* Remove unnecessary setOpen(true)
4 years ago
hq1 d2f2c69387
Conditionally support switching between v1 and v2 clickhouse schemas (#2780)
* Remove ClickhouseSetup module

This has been an implicit point of contact to many
tests. From now on the goal is for each test to maintain
its own, isolated setup so that no accidental clashes
and implicit assumptions are relied upon.

* Implement v2 schema check

An environment variable V2_MIGRATION_DONE acts like
a feature flag, switching plausible from using old events/sessions
schemas to v2 schemas introduced by NumericIDs migration.

* Run both test suites sequentially

While the code for v1 and v2 schemas must be kept still,
we will from now on run tests against both code paths.
Secondary test run will set V2_MIGRATION_DONE=1 variable,
thus making all `Plausible.v2?()` checks return `true'.

* Remove unused function

This is a remnant from the short period when
we would check for existing events before allowing
creating a new site.

* Update test setups/factories with v2 migration check

* Make GateKeeper return site id along with :allow

* Make Billing module check for v2 schema

* Make ingestion aware of v2 schema

* Disable site transfers for when v2 is live

In a separate changeset we will implement simplified
site transfer for when v2 migration is complete.
The new transfer will only rename the site domain in postgres
and keep track of the original site prior to the transfer
so we keep an ingestion grace period until the customers
redeploy their scripting.

* Make Stats base queries aware of v2 schema switch

* Update breakdown with v2 conditionals

* Update pageview local start with v2 check

* Update current visitoris with v2 check

* Update stats controller with v2 checks

* Update external controller with v2 checks

* Update remaining tests with proper fixtures

* Rewrite redundant assignment

* Remove unused alias

* Mute credo, this is not the right time

* Add test_helper prompt

* Fetch priv dir so it works with a release

* Fetch distinct partitions only

* Don't limit inspect output for partitions

* Ensure SQL is printed to IO

* Remove redundant domain fixture
4 years ago
Uku Taht a154a4214b
Support multiple filters - backend (#2749)
* Wrap Plausible.Stats.Filters with unit tests

* Parse `member` filter type

* Support escaped | in member filter

* Support for `member` filter in `aggregate_time_on_page`

* Add support for `member` filter type on goals

* Disable Credo warning

* Support `not_member` filter type

* Disable credo for `query_sessions`

* Support `matches_member` and `not_matches_member` filters

* Disable Credo for `Filters.filter_value/2`

* Support `matches_member` and `not_matches_member` for goal filter

* Support for contains_member and friends

* Updates for new chto driver

* make top_stats_test.exs:203 pass (#2779)

---------

Co-authored-by: ruslandoga <rusl@n-do.ga>
4 years ago
Adam 6d79ca5093
Switch to new clickhouse adapter (ch/chto) (#2733)
* another clickhouse adapter

* don't restore stats_removal.ex

* fix events main-graph error (#2746)

* update ch, chto

* update chto again (#2759)

* Stop treating page filter as an entry_page filter (#2752)

* remove dead code

* stop treating page filter as entry page filter in breakdown queries

* stop treating page filter as entry page filter in aggregate queries

* stop treating page filter as entry page filter in timeseries queries

* mix format

* update changelog

* break code down to smaller functions to keep credo happy

* remove unused functions

* make CSV export return only conversions with goal filter (#2760)

* make CSV export return only conversions with goal filter

* update changelog

* update elixir version in mix.exs (#2742)

* revert admin.ex changes (#2776)

---------

Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
Co-authored-by: ruslandoga <rusl@n-do.ga>
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
4 years ago
RobertJoonas d0895efbdd
Stop treating page filter as an entry_page filter (#2752)
* remove dead code

* stop treating page filter as entry page filter in breakdown queries

* stop treating page filter as entry page filter in aggregate queries

* stop treating page filter as entry page filter in timeseries queries

* mix format

* update changelog

* break code down to smaller functions to keep credo happy

* remove unused functions
4 years ago
RobertJoonas 874d664521
Add the new `Views per Visit` metric to dashboard Top Stats and CSV export (#2728)
* globally rename 'pages_per_visit' to 'views_per_visit'

* change the order of top stats

* rename 'Visits' to 'Total visits' in the UI

* add views_per_visit to UI

* put the new metric under a feature flag

* add new metric to CSV export under feature flag

* mix format

* use only one feature flag
4 years ago
RobertJoonas 0d6c72d50f
Add new `Pages / Visit` metric to Stats API Aggregate and Timeseries (#2714)
* refactor metric validation

* link to the correct docs section

* add the new metric to aggregate API

* explicitly remove __internal_visits

The overall metric selection is well defined by
`Plausible.Stats.Timeseries.empty_row/2`. The only metric that needs
to be removed from the timeseries response is __internal_visits.

This commit also moves the `remove_internal_visits_metric` function to a
new Util module to be used by both breakdown and timeseries.

* add the new metric to timeseries API

* mix format

* add moduledoc to keep credo happy

* convert pages_per_visit to string straight away

* do rounding in db query

* query # of visits with sum(sign) instead

* stop converting pages_per_visit to string
4 years ago
Adam 8f86036e57
Keep track of native stats start timestamp when retrieving data (#2715)
* Stats boundary/PoC?

* Delete stats removal

* Drop events check on site creation

* Update seeds script

* Use native_stats_start_at

* Don't rely on native stats pointer in imported stats queries

* Reset site

* Export reset/1

* Remove unnecessary inserted_at settings

* Update seeds

* Remove unnecessary inserted_at setting
4 years ago
Uku Taht 3ed5e9d27d
Refactor visits metric internals (#2691)
* Stop relying on implicitly selected `visits` metric

* Make the `visits` metric more explicit in imported data queries

* Rename internally used `visits` to `__internal_visits`
4 years ago
Adam Rutkowski 8a2df294f5
Replace empty OS/screen size/Browser with `(not set)` in UI/API/exports (#2646)
* Make Device section components aware of (not set)

So that no extra sub-filters are possible when the unset
top item is selected.

* Support '(not set)' in breakdown/filters

* Update expectations for export tests

* Add extra tests for returning/filtering by '(not set)'

* Add changelog entry

* Remove ListReport conditional render

* Prevent redundant sub-filters

* Fix filter text rendering

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
4 years ago
RobertJoonas bd0de97521
Fix returning more pageviews with a visit property filter (#2612)
* fix subquery for sessions in base_event_query/2

As the 'sessions' table is using the CollapsingMergeTree engine, we have
to select session_id's distinctively. Otherwise we will get multiple rows
(with sign -1 and 1) as long as the background merge hasn't happened.

* update changelog

* use GROUP BY instead of SELECT DISTINCT

* remove comma
4 years ago
Uku Taht 8bdbabbdad
Enable Dialyzer and Credo in CI (#2315) 4 years ago
Uku Taht 5e415c2420 Add entry_props back in 5 years ago
RobertJoonas 40275b64d4
Pageview custom dimensions (#1816)
* added custom dimension filtering tests for pages

* first filter UI in place

* pages, entry pages and exit pages can be filtered by pageview props

* added tests for expected filtering behaviour

* fix dimension filter for sources + tests

* added is_not filtering functionality

* fixed formatting

* fixed admin_test

* added (none) as filter value + is_not filter type in UI

* added prefilling applied filter values and some UI tweaks

* added fetch options

* Make prop suggestions work with `props` filter

* Fix test

* Track login state internally

* Add CHANGELOG entry

Co-authored-by: Uku Taht <uku.taht@gmail.com>
5 years ago
Uku Taht 66aceda9e9 Fix main graph selection with imported data 5 years ago
RobertJoonas e5cf800d4a
Improve stats api (#1759)
* can now query events metric when breaking down on a visit prop

* updated changelog

* fixed formatting

* fixed a bug - querying for bounce rate does not return visits anymore

* removed the changelog update
5 years ago
Uku Taht e27734ed79
[Continued] Google Analytics import (#1753)
* Add has_imported_stats boolean to Site

* Add Google Analytics import panel to general settings

* Get GA profiles to display in import settings panel

* Add import_from_google method as entrypoint to import data

* Add imported_visitors table

* Remove conflicting code from migration

* Import visitors data into clickhouse database

* Pass another dataset to main graph for rendering in red

This adds another entry to the JSON data returned via the main graph API
called `imported_plot`, which is similar to `plot` in form but will be
completed with previously imported data.  Currently it simply returns
the values from `plot` / 2. The data is rendered in the main graph in
red without fill, and without an indicator for the present. Rationale:
imported data will not continue to grow so there is no projection
forward, only backwards.

* Hook imported GA data to dashboard timeseries plot

* Add settings option to forget imported data

* Import sources from google analytics

* Merge imported sources when queried

* Merge imported source data native data when querying sources

* Start converting metrics to atoms so they can be subqueried

This changes "visitors" and in some places "sources" to atoms. This does
not change the behaviour of the functions - the tests all pass unchanged
following this commit. This is necessary as joining subqueries requires
that the keys in `select` statements be atoms and not strings.

* Convery GA (direct) source to empty string

* Import utm campaign and utm medium from GA

* format

* Import all data types from GA into new tables

* Handle large amounts of more data more safely

* Fix some mistakes in tables

* Make GA requests in chunks of 5 queries

* Only display imported timeseries when there is no filter

* Correctly show last 30 minutes timeseries when 'realtime'

* Add with_imported key to Query struct

* Account for injected :is_not filter on sources from dashboard

* Also add tentative imported_utm_sources table

This needs a bit more work on the google import side, as GA do not
report sources and utm sources as distinct things.

* Return imported data to dashboard for rest of Sources panel

This extends the merge_imported function definition for sources to
utm_sources, utm_mediums and utm_campaigns too. This appears to be
working on the DB side but something is incomplete on the client side.

* Clear imported stats from all tables when requested

* Merge entry pages and exit pages from imported data into unfiltered dashboard view

This requires converting the `"visits"` and `"visit_duration"` metrics
to atoms so that they can be used in ecto subqueries.

* Display imported devices, browsers and OSs on dashboard

* Display imported country data on dashboard

* Add more metrics to entries/exits for modals

* make sure data is returned via API with correct keys

* Import regions and cities from GA

* Capitalize device upon import to match native data

* Leave query limits/offsets until after possibly joining with imported data

* Also import timeOnPage and pageviews for pages from GA

* imported_countries -> imported_locations

* Get timeOnPage and pageviews for pages from GA

These are needed for the pages modal, and for calculating exit rates for
exit pages.

* Add indicator to dashboard when imported data is being used

* Don't show imported data as separately line on main graph

* "bounce_rate" -> :bounce_rate, so it works in subqueries

* Drop imported browser and OS versions

These are not needed.

* Toggle displaying imported data by clicking indicator

* Parse referrers with RefInspector

- Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual
  referrer host + path, whereas 'ga:source' includes utm_mediums and
  other values when relevant.
- 'ga:fullReferror' does however include search engine names directly,
  so they are manually checked for as RefInspector won't pick up on
  these.

* Keep imported data indicator on dashboard and strikethrough when hidden

* Add unlink google button to import panel

* Rename some GA browsers and OSes to plausible versions

* Get main top pages and exit pages panels working correctly with imported data

* mix format

* Fetch time_on_pages for imported data when needed

* entry pages need to fetch bounces from GA

* "sample_percent" -> :sample_percent as only atoms can be used in subqueries

* Calculate bounce_rate for joined native and imported data for top pages modal

* Flip some query bindings around to be less misleading

* Fixup entry page modal visit durations

* mix format

* Fetch bounces and visit_duration for sources from GA

* add more source metrics used for data in modals

* Make sources modals display correct values

* imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration

* Merge imported data into aggregate stats

* Reformat top graph side icons

* Ensure sample_percent is yielded from aggregate data

* filter event_props should be strings

* Hide imported data from frontend when using filter

* Fix existing tests

* fix tests

* Fix imported indicator appearing when filtering

* comma needed, lost when rebasing

* Import utm_terms and utm_content from GA

* Merge imported utm_term and utm_content

* Rename imported Countries data as Locations

* Set imported city schema field to int

* Remove utm_terms and utm_content when clearing imported

* Clean locations import from Google Analytics

- Country and region should be set to "" when GA provides "(not set)"
- City should be set to 0 for "unknown", as we cannot reliably import
  city data from GA.

* Display imported region and city in dashboard

* os -> operating_system in some parts of code

The inconsistency of using os in some places and operating_system in
others causes trouble with subqueries and joins for the native and
imported data, which would require additional logic to account for. The
simplest solution is the just use a consistent word for all uses. This
doesn't make any user-facing or database changes.

* to_atom -> to_existing_atom

* format

* "events" metric -> :events

* ignore imported data when "events" in metrics

* update "bounce_rate"

* atomise some more metrics from new city and region api

* atomise some more metrics for email handlers

* "conversion_rate" -> :conversion_rate during csv export

* Move imported data stats code to own module

* Move imported timeseries function to Stats.Imported

* Use Timex.parse to import dates from GA

* has_imported_stats -> imported_source

* "time_on_page" -> :time_on_page

* Convert imported GA data to UTC

* Clean up GA request code a bit

There was some weird logic here with two separate lists that really
ought to be together, so this merges those.

* Fail sooner if GA timezone can't be identified

* Link imported tables to site by id

* imported_utm_content -> imported_utm_contents

* Imported GA from all of time

* Reorganise GA data fetch logic

- Fetch data from the start of time (2005)
- Check whether no data was fetched, and if so, inform user and don't
  consider data to be imported.

* Clarify removal of "visits" data when it isn't in metrics

* Apply location filters from API

This makes it consistent with the sources etc which filter out 'Direct /
None' on the API side. These filters are used by both the native and
imported data handling code, which would otherwise both duplicate the
filters in their `where` clauses.

* Do not use changeset for setting site.imported_source

* Add all metrics to all dimensions

* Run GA import in the background

* Send email when GA import completes

* Add handler to insert imported data into tests and imported_browsers_factory

* Add remaining import data test factories

* Add imported location data to test

* Test main graph with imported data

* Add imported data to operating systems tests

* Add imported data to pages tests

* Add imported data to entry pages tests

* Add imported data to exit pages tests

* Add imported data to devices tests

* Add imported data to sources tests

* Add imported data to UTM tests

* Add new test module for the data import step

* Test import of sources GA data

* Test import of utm_mediums GA data

* Test import of utm_campaigns GA data

* Add tests for UTM terms

* Add tests for UTM contents

* Add test for importing pages and entry pages data from GA

* Add test for importing exit page data

* Fix module file name typo

* Add test for importing location data from GA

* Add test for importing devices data from GA

* Add test for importing browsers data from GA

* Add test for importing OS data from GA

* Paginate GA requests to download all data

* Bump clickhouse_ecto version

* Move RefInspector wrapper function into module

* Drop timezone transform on import

* Order imported by side_id then date

* More strings -> atoms

Also changes a conditional to be a bit nicer

* Remove parallelisation of data import

* Split sources and UTM sources from fetched GA data

GA has only a "source" dimension and no "UTM source" dimension. Instead
it returns these combined. The logic herein to tease these apart is:

1. "(direct)" -> it's a direct source
2. if the source is a domain -> it's a source
3. "google" -> it's from adwords; let's make this a UTM source "adwords"
4. else -> just a UTM source

* Keep prop names in queries as strings

* fix typo

* Fix import

* Insert data to clickhouse in batches

* Fix link when removing imported data

* Merge source tables

* Import hostname as well as pathname

* Record start and end time of imported data

* Track import progress

* Fix month interval with imported data

* Do not JOIN when imported date range has no overlap

* Fix time on page using exits

Co-authored-by: mcol <mcol@posteo.net>
5 years ago
Uku Taht ebb9476b56 Fix visit duration stat 5 years ago
Uku Taht 48ad7485c8
PR 1393 continued (#1542)
* Add `utm_content` and `utm_term`.
Support `utm_content` and `utm_term` as requested in #515.

* Add dropdown for UTM options

* Remove utm_content and term from filter modal for now

Co-authored-by: Blender Defender <defenderblender@gmail.com>
5 years ago
Uku Taht 05bf43c1be
City level location data (#1449)
* Merge branch 'plausible_master'

* Add City level details

* Add City level details

* Use ISO codes instead of geoname_id for subdivisions

* Add easier way to configure geolocation database

* Add workflow for dev branch

* Correct clickhouse migration

* Translate subdivision names

* Translate city names

* WIP

* Region and country filters

* Fix region filter

* Remove region_name when removing region filter

* Add modals for regions and cities

* Remove dead code

* WIP

* Revert "WIP"

This reverts commit 3202bf2fe9.

* Feature flag to hide cities when deployed

* Add changelog entry

* Remove unused code

* Remove unused variables

* Fix test

Co-authored-by: AymanTerra <aymanterra@yahoo.com>
5 years ago
Uku Taht 941a91ec22 Make better use of the index on sessions table 5 years ago
Uku Taht 7364733faf Ensure that question marks are encoded properly for the database 5 years ago
Uku Taht bd7de59a9c Show total visitors when filtered for goal 5 years ago
Uku Taht ebc59313a5 Do not match trailing slash by default in page regex 5 years ago
Uku Taht bea26f0707 Fix custom prop breakdown when filtering with visit prop 5 years ago
Uku Taht d3ecbd4e17 Fix sampling 5 years ago
Uku Taht 63b7abad58 Disable sampling in the API 5 years ago
Uku Taht a9b16fa6c6 Fix exit rate when filtering for page 5 years ago
Uku Taht 3d3601c332 Support negated glob filters everywhere 5 years ago
Uku Taht 49214eaff8 Fix exit pages when using a goal filter 5 years ago
Uku Taht d9ac8ccdb9 Fix samplign in main graph and top pages 5 years ago
Uku Taht 2f98da8f3a Filter for pageviews when no other filter is specified in base.ex 5 years ago
Uku Taht 669866a16b
Finish stats module (#1248)
* Fix small inconsitencies in stats module

* Format
5 years ago
Uku Taht 4fef567623 Fix goal filter 5 years ago
Uku Taht ecbc68f05b Fix entry pages 5 years ago
Uku Taht 79f6756ba2 Make is not filter work 5 years ago
Uku Taht 39f5bc6afb Make sure Direct / None is not showed in small reports 5 years ago
Uku Taht 7d1b1214e2 Implement dashboard with new stats module 5 years ago
Uku Taht ad6faece87
Stats API: Support IN filter expression (#1090)
* Support IN filter expression

* Update clickhouse_ecto
6 years ago
Uku Taht 432ad2a606
Stats API improvements (#826)
* Show hourly data for a day

* Fix aggregations with long periods and comaprisons enabled

* Return Direct / None instead of empty string when missing source

* Fix/Align breakdown query with current queries in the UI

* Show correct number of days for 7d and 30d

* Wrap results of aggregate call in `results`

* Allow breakdown by event:name

* Ensure event:name filter works in all contexts

* WIP

* Add support for custom event props
6 years ago
Uku Taht a29ac44434
Api improvements (#723)
* Require date parameter with custom period

* Validate format of the `date` param with custom period

* Return proper status codes for auth failures

* Add breakdown endpoint

* Fix pagination

* Add API validations for breakdown call

* Change the breakdown endpoint to return an object instead of an array

* Remove change to aggregate call

* Change timeseries call
6 years ago