Compare commits

...

105 Commits

Author SHA1 Message Date
Adrian Gruntkowski a7603c9e49
Improve import procedure to ensure no time range overlaps (#3970)
* Always scope import ID by site as well

* Do not schedule new import job if there are any site imports in progress

* Disable import buttons when any import is in progress

* Simplify `schedule_job/4` (h/t @RobertJoonas)
3 years ago
Adrian Gruntkowski cffff0340c
Handle Google API timeouts gracefully during imports (#3975) 3 years ago
Adrian Gruntkowski 33eed9d7db
Delete imports which have no stats (#3972) 3 years ago
Marko Saric e5b7f1afd0
Changing the copy of the locked screen (#3967)
changing the copy here as I think that in some situations the "we're still counting stats" message is now shown even to those dashboards where we've stopped counting stats so best to avoid that
3 years ago
Uku Taht f966419a4a
Update ua_inspector (#3957)
Co-authored-by: hq1 <hq@mtod.org>
3 years ago
hq1 f9f0407d68
Remove `experimtnal_hostname_filter` and keep it on by default (#3973)
* Remove `experimental_hostname_filter` and keep it on by default

* Catch up with changes done via e5b56dbe6
3 years ago
RobertJoonas e5b56dbe62
Refactor VisitorGraph (#3936)
* Give a more semantic name to a function

* Make the LineGraph component thinner

* Move LineGraph into a separate file

* Move interval logic into interval-picker.js

This commit also fixes a bug where the interval name displayed inside
the picker component flickers the default interval when the graph is
loading.

The problem was that we were counting on graphData for returning us the
current interval: `let currentInterval = graphData?.interval`

We should always know the default interval before making the main-graph
request. Sending graphData to IntervalPicker component does not make
sense anyway.

* extract data fetching functions out of VisitorGraph component

* Return graph_metric key from Top Stats API

This commit introduces no behavioral changes - only starts returning an
additional field, allowing us to avoid the following logic in React:

1. Finding the metric names, given a stat display name. E.g.
   `Unique visitors (last 30 min) -> visitors`

2. Checking if a metric is graphable or not

* Move metric state into localStorage

This commit gets rid of the internal `metric` state in the VisitorGraph
component and starts using localStorage for that instead.

This commit also chains the main-graph request into the top-stats request
callback - meaning that we'll always fetch new graph data after top stats
are updated. And we do it all in a single function.

Doing so simplifies the loading state significantly, and also helps to
make it clear, that at all times, existing top stats are required before
we can fetch the graph. That's because the metric is determined by which
Top stats are returned (for example, we can't be sure whether revenue
metrics will be returned or not).

* Make sure graph tooltip says "Converted Visitors"

* Extract a StatsExport function component

Again, instead of relying on `graphData?.interval` we can read it from
localStorage, or default to the largest interval available. The export
should not be dependant on the graph.

* Extract SamplingNotice function component

* Extract WithImportedSwitch function component

* Stop "lazy-loading" the graph and top stats

Since the container is always on top on the page, it will be visible on
the first render in any case - no matter the screen size.

* Turn VisitorGraph into a function component

* Display empty container until everything has loaded

* Do not display loading spinner on realtime ticks

* Turn Top Stats into a fn component

* fetch top stats and graph async

* Make sure revenue metrics can remain on the graph

* Add an extra check to canMetricBeGraphed

* fix typo

* remove redundant double negation
3 years ago
Karl-Aksel Puulmann 3115c6e7a8
Reducing JOINs in queries (#3966)
* Move experimental_session_count? logic to within query object

* WIP new querying system for deciding what tables to query

* both -> either

* Include sample_percent in both tables

* Remove a hanging TODO

* Allow filtering by visit props on event queries if flag is on

* Make default sessions join more conditional

* Simplify events_join_sessions?

* Add some TODOs

* Fix assignment

* Handle entry/exit page visit props separately from props stored in events table

* Update test which created sessions/events differently from everyone else

* Make query_events private

* Dont filter by session properties on events table if querying sessions and joining in events

* Handle visits, pageviews, events and visitors metrics from other table

* both -> either

* events, pageviews are strictly event metrics

* Add support for (plain) breakdowns deciding which table to use

* Run tests with experimental_reduced_joins as a separate job

Also refactor which tests are run with postgres:15 to reduce number of jobs

* moduledocs for TableDecider

* Fix matrix

* Custom build name

* Move TEST_EXPERIMENTAL_REDUCED_JOINS check

* Handle percentage separately from other metrics

* Remove debug code

* TableDecider tests

* both => sample_percent

* Improve naming

* Simplify code

* Breakdowns retain old behavior if getting metric visitors

* Unify behavior of entry/exit page hostnames with rest

* Fix test naming
3 years ago
hq1 6af80dd246
Filter by hostnames (#3963)
* CH Migration: exit/entry hostnames in sessions_v2

* Leave only exit_page_hostname, we already record hostnames

* Use ClickHouse DDL in favour of ecto so that cluster is included

* Compress with ZSTD(3)

* Expose Hostname filter in the dashboard dropdown

* Add `exit_page_hostname` to ClickHouse `sessions_v2` schema

* Start tracking hostname changes in sessions

* Implement hostname filter suggestions

* Enable filtering by `event:hostname`

* Add tests for filtering by hostnames

* Ensure filter suggestions work for exit pages too

* Allow overriding hostnames with `send_pageview` mix task

* Remove `:window_time_on_page` flag

It seems that we can remove it after all?

* Initialize `experimental_hostname_filter` query parameter

* Rewrite cache store behaviour with regards to session hostnames

* Work around inconsistent session merging

So that `populate_stats` can get closer to actual ingestion

* Improve top stats test

* Make it possible to filter sessions by entry/exit hostnames

* Update pages tests

* Expose `experimental_hostname_filtering` temporarily in the UI

* Untested yet: also apply experimental filtering to sources

* Introduce `hostname_filter` feature flag

* Format

* Test top sources with hostname filter + experimental flag
3 years ago
Adrian Gruntkowski e6d83e946f
Populate new columns in imports and exports (#3969)
* Extend `Imported*` schemas with newly added columns

* Populate newly added `Imported*` fields in GA4 imports

* Extend exports with newly added fields

* Extend CSV importer to ingest new fields

* Fix alias shadowing error

* Add more extensive GA4 import fixtures

* Apply rounding and casting to sampled visits
3 years ago
RobertJoonas bd73fc8266
CH Migration - add more imported metrics and properties (#3949)
* add migration

* add utm_source to imported_sources

* quickfix

* satisfy credo

* Revert "satisfy credo"

This reverts commit bb0b228164.

* Revert "quickfix"

This reverts commit ab6f70c79e.

---------

Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
3 years ago
ruslandoga 2ba4988c95
specify insert columns in import from s3 (#3968)
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
3 years ago
hq1 1f778e0c11
CH Migration: exit page hostname on sessions_v2 (#3953)
* CH Migration: exit/entry hostnames in sessions_v2

* Leave only exit_page_hostname, we already record hostnames

* Use ClickHouse DDL in favour of ecto so that cluster is included

* Compress with ZSTD(3)
3 years ago
Adrian Gruntkowski 9f27fa303c
Fix dry run mode in `DataMigration.SiteImports` (#3965) 3 years ago
Adrian Gruntkowski 23a3699dd7
Improve import stats toggle and `with_imported` flag computation (#3960)
* Check import presence across all imports and not just the first one

Also, simplify imported data toggle rendering to not explicitly
refer to the earliest import source.

* Change imported stats toggle icon in dashboard

* Test `Imported.get_imports_date_range/1`

* Simplify failed UA/GA import email copy
3 years ago
Adrian Gruntkowski 71fe541359
Implement script for backfilling legacy site import entries and adjusting end dates of site imports (#3954)
* Always select and clear import ID 0 when referring to legacy imports

* Implement script for adding site import entries and adjusting end dates

* Log cases where end date computation is using fallback

* Don't log queries when running the migration to reduce noise
3 years ago
Adrian Gruntkowski 5bf59d1d8a
Implement adjusting imported date range to actual and existing stats (#3943)
* Implement adjusting imported date range to actual and existing stats

* Drop redundant prefix from import list entries

* Make pageview numbers in imports list formatted for readability

* Test and improve date range cropping

* DRY UA and GA4 stats start and end date API calls

* Extend UA/GA import controller tests and improve error handling

* refactor finding longest open range without existing data

* Fix typo in test description

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>

* Rename `open_ranges` to `free_ranges`

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
3 years ago
ruslandoga c263df5805
CSV imports (UI) (#3845)
* add basic ui

* remove TODO

* credo

* allow folder upload

* redirect external

* mention folder, use folder icon for file picker

* back to multiple file upload

* mention zip

* escape dots in archive filename
3 years ago
hq1 b31433a7bf
Ensure all the react container attributes are strings (#3948) 3 years ago
hq1 edf70d14b6
Use sessionStorage for "dashboard first launch" banner tracking (#3892)
* Use sessionStorage for offer e-mail report banner tracking

Keeping it within the cookie is problematic, as the banners don't
expire and overflow the cookie with data when enough new sites
are added.

Ref https://github.com/plausible/analytics/issues/3762

* Update changelog

* Extract a component

* Make is_dbip evaluate to quoted boolean
3 years ago
hq1 7523abe93e
Add metrics to ingestion pipeline (#3927)
* Add metrics to ingestion pipeline

* Format

* Format

* Update buckets

* Credo
3 years ago
Karl-Aksel Puulmann 604bf88451
Add github action to validate whether migrations and app change at the same time (#3945) 3 years ago
Karl-Aksel Puulmann 4af7019011
Ignore sessions without entry/exit pages when breaking down entry/exit pages (#3933)
* Ignore sessions without entry/exit pages when breaking down entry/exit pages

* Update stats controller tests to have more realistic test data (pageview followed by event)
3 years ago
hq1 2fae0146a4
Reapply 3918 (#3940)
* Reapply "Pages shield (#3918)"

This reverts commit 33b5c10654.

* Make the FF check work against the site actor
3 years ago
hq1 9989ce6927
Migration for 3918 (#3939)
* Revert "Pages shield (#3918)"

This reverts commit 53f94a9f82.

* Migration: Shield page rules
3 years ago
hq1 53f94a9f82
Pages shield (#3918)
* Migration: Shield page rules

* Add Ecto schema for Page Rules

* Add Page Rule cache

* Fix typo

* BTW: Use already imported function

* Extend Shields context interface + split existing tests

* Ingestion: filter matching patches + refactor shield actions

* Add LV section for adding Page Rules

* Validate max page path length

* Put Pages Shield behind a feature flag

* Update CHANGELOG

* Update docs link anchor

As per https://github.com/plausible/docs/pull/477

* Update lib/plausible/shields.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible_web/live/shields/page_rules.ex

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Update lib/plausible_web/live/shields/page_rules.ex

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
3 years ago
Adrian Gruntkowski ba5b80a8c0
Add label to site imports and populate it (#3914) 3 years ago
Uku Taht 1d017e86a1
Fix escaping of source filters (#3930)
* Fix escaping of source filters

* CHANGELOG

* Fix typo

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
3 years ago
Adrian Gruntkowski 52c226c428
Add `label` column to `site_imports` schema (#3926) 3 years ago
RobertJoonas fb61f0b425
Capitalize Total Conversions in graph tooltip (#3934) 3 years ago
RobertJoonas d6e1e8bebd
Put total conversions on the graph + goal-filtered CSV export improvements (#3929)
* Add validation for the events metric in main_graph

* Test the already existing events metric support in main-graph

* Put total conversions on the graph

* extract main_graph_csv function (refactor only)

* add total_conversions and conversion_rate to goal-filtered visitors.csv

* update changelog
3 years ago
Uku Taht 561dcd821e
Mask pathanme in filter menu event (#3932) 3 years ago
Uku Taht fd879eeb16
Store referrers from android apps (#3715)
* Store referrers from android apps

* Add test for unknown referrer protocol

* Store android referrer protocol
3 years ago
Uku Taht 8992c8ee07
Add tracking to filter button (#3928) 3 years ago
RobertJoonas c32779a3e5
Timeseries for conversion rate (#3919)
* add conversion rate to Stats API timeseries

* make sure CR can be queried as the only metric

* add a test asserting zeros are returned

* add tests for filtering by other properties at the same time

* Remove unnecessary validation of params

1. It doesn't make to validate `interval` (and its granularity) in all
   endpoints. It's only relevant for the main graph.

2. The plug (renamed to `date_validation_plug`) already makes sure that
   the dates are validated. No need to call the same function again in
   Top Stats and Funnel endpoints.

* add metric validation to main graph

* Add tests for main graph API

* put conversion rate on the graph

* update changelog

* Add revenue metrics into metrics.ex

* make fn private

* avoid setting graph metric to visitors in goal-filtered view
3 years ago
Adrian Gruntkowski d6e81670e4
Unify UA and GA4 import flow into one (#3888)
* Unify GA4 and UA import flow into one

* Clean up property and view data retrieval via Google HTTP APIs

* Turn `Map.get` into `Map.fetch!` in API response processing code

* Bump list account summaries page size limit to max of 200

* Show only views in legacy flow and fix legacy redirect after import start

* Move google analytics import actions tests to a separate module

* Extend Google Analytics controller tests

* DRY up `property?` predicate (h/t @RobertJoonas)
3 years ago
ruslandoga 5f9465614b
Include domain and dates in zip archive filename (#3921)
* include domain and dates in zip archive filename

* adapt to comments
3 years ago
Karl-Aksel Puulmann 32ab138301
Fix issue with name clash (#3925)
Unexpectedly, table.name caused a name clash after CR refactor, so using a unique name
for the output column

Sentry issue: https://sentry.plausible.io/organizations/sentry/issues/5612
3 years ago
Karl-Aksel Puulmann c219652dae
Re-apply Move conversion_rate logic from elixir to clickhouse (#3924)
* Revert "Revert "Move conversion_rate logic from elixir to clickhouse (#3887)"…"

This reverts commit 253fb5d67d.

* Fix issue with missing columns

The issue came from refactoring event:goal UNION ALL logic and trying to move
name select from first to last. If any other tables were joined, the incorrect
item would be used as an array index, causing this issue.

Added a relevant test.
3 years ago
Karl-Aksel Puulmann 253fb5d67d
Revert "Move conversion_rate logic from elixir to clickhouse (#3887)" (#3923)
This reverts commit 1909743b90.
3 years ago
Karl-Aksel Puulmann 1909743b90
Move conversion_rate logic from elixir to clickhouse (#3887)
* Separate out query building from pagination/execution logic.

* Refactor pageview_goals breakdown query, removing index column from results

* Remove zip_columns logic

* Use common pagination util

* Do everything in a single query for breakdowns for goals

* Order in DB

* Make sure column order is identical

* Calculate CR within the goal breakdown query

* Calculate CR for property breakdowns

* WIP: Calculate group CR

* CR with order_by

* Compatibility fix

* Import Ecto.Query and cleanup

* handle total_visitors the same way as add_percentage

* Handle conversion_rate in aggregate.ex

* Solve rebase fail

* Simplify maybe_add_group_conversion_rate

* Add conversion_rate defaults to 0 test

* Add test for conversion_rate should not be calculated with imported data (failing here and on master)

* Dont include imported data when breakdown by prop or goal

* Remove revenue_nils
3 years ago
ruslandoga 64f35bf8f6
make robots.txt restrictive by default (#3905)
* make robots.txt restrictive by default

* add note about cloud robots.txt
3 years ago
ruslandoga 279e89c693
CSV imports (no UI) (#3895)
* encode/decode date range in filenames

* Update lib/plausible/imported/csv_importer.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible/imported/csv_importer.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* drop unused functions

* send failure email if there is no data to export

* use PlausibleWeb.Email.mailer_email_from()

* ensure we get dates from minmax date query

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
ruslandoga 4242b52be4
Allow importing extra config (#3906)
* allow importing extra config

* changelog

* fix typo

* add test
3 years ago
ruslandoga dfcc8d794a
Add space after logo on the index page (#3825) 3 years ago
Karl-Aksel Puulmann 02d2256483
Set exit_page only on pageviews (#3870)
* Set exit_page only on pageviews

* Update tests

* Update entry_page on first pageview

* Update CHANGELOG.md
3 years ago
ruslandoga 07b714a143
Update Sentry (#3843)
* update Sentry

* Sentry.HTTPClient.child_spec is now optional

* Sentry.EventFilter is deprecated

* update sentry to 10.2.0

* fix dialyzer warnings
3 years ago
ruslandoga c08aa29ad8
Remove DISABLE_AUTH deprecation warning (#3904)
* remove DISABLE_AUTH deprecation warning

* changelog
3 years ago
ruslandoga 0a0c7ce165
Remove init-admin.sh (#3903)
* remove init-admin.sh

* changelog
3 years ago
ruslandoga 8c0e94f72c
Fix manual embed width (#3910)
* fix manual embed width

* changelog
3 years ago
ruslandoga 4ae00540dc
Split Elixir CI jobs (#3898)
* split elixir tests

* more understandable matrix

* add unused deps in mix.lock check

* get all deps for static checks
3 years ago
ruslandoga 5e74b1cf74
CSV exports (UI) (#3875)
* ui

* fix redirect link

* improve make minio

* use implicit button form for csv export

* add exports_bucket helper

* read S3_EXPORTS_BUCKET

* supply s3_bucket in export_csv job args

* make plausible_minio use unprotected port

* move s3_csv_export queue to base queues

* Update lib/plausible_web/controllers/site_controller.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
Karl-Aksel Puulmann d3586a81e3
UX improvement: don't autofocus on Custom properties on dashboard (#3872)
* UX improvement: don't autofocus on Custom properties on dashboard

Currently when you visit the dashboard with a certain configuration,
it will automatically scroll to the bottom, which is annoying.

To reproduce:
1. localStorage.clear() in console
2. Refresh, open properties at the very bottom
3. Refresh again

Expected behavior:
1. Dashboard shows

Actual behavior:
1. Dashboard scrolls to the bottom

* Changelog entry
3 years ago
hq1 59afa20955
Reapply #3878 + bugfix hit rate tracking (#3891)
* Reapply "Replace caching engine (#3878)" (#3883)

This reverts commit c5881cdc6d.

* Ensure hit rate is tracked on `get_or_store`

* Remove :wx and :observer

* Remove unused deps

* Use `:set` table type
3 years ago
RobertJoonas e8f3946dde
Fix division by zero in imported queries (#3890)
* prevent division by 0 in merge_imported queries

* Revert "fix bounce_rate change bug (#3886)"

This reverts commit 6eef32a8ff.

After 02aa0b2, we can keep on assuming that bounce rate is always numeric.
3 years ago
Adrian Gruntkowski 4d7d88cfec
Implement basics of GA4 import (#3851)
* Implement LV date input using flatpickr

* Implement basics of GA4 import (very dirty WIP)

* Split Google HTTP API into UA and GA4 specific parts

* Add a quick way to record GA4 API responses

* Add first GA4 import fixtures with GA4 Data API responses

* Extract GA4 and UA specific logic form Google API

* Extract UA and GA4 specific actions to distinct controllers

* Add integration test for GA4 importer

* Update GA4 fixtures

* Test GA4 API

* Add debug logging and fix paginating through API results in in GA4 import

* Revert "Implement LV date input using flatpickr"

This reverts commit c696f8ee39d5702f27015c09a4f079ca124cc7bb.

* Fix note
3 years ago
ruslandoga f2350b5165
Add `/tmp/` to .gitignore and simplify s3 cleanups in tests (#3889) 3 years ago
ruslandoga 5a3072ca21
CSV exports (no UI) (#3836)
* csv exports

* use ex_unit's tmp_dir
3 years ago
RobertJoonas 6eef32a8ff
fix bounce_rate change bug (#3886) 3 years ago
RobertJoonas 7641c66a2b
Stats api time on page (#3858)
* add metric validation + support in aggregate

* add a test ensuring comparison works

* disallow time_on_page with a goal filter

* Return time_on_page as `nil` from aggregate API

In case time_on_page cannot be calculated, we'll return it as `nil` from
the Stats API.

This is to make the behaviour consistent between breakdown and aggregate
endpoints. As for the UI, we'll still continue to report time_on_page as
0 - not changing any UI behaviour as discussed with Marko.

* add tests for time_on_page in event:page breakdown

* update changelog

* invalidate time_on_page with event:name filter

* add the ability to only query time_on_page in page breakdown

We'll need the visitors metric to get the list of pages to calculate the
time_on_page for.
3 years ago
Cenk Kücük dea9287d74
Revert "Temporarily disable tzdata auto-updating (#3873)" (#3885)
This reverts commit 11368e5143.
3 years ago
hq1 c5881cdc6d
Revert "Replace caching engine (#3878)" (#3883)
This reverts commit 437a3350ff.
3 years ago
Marko Saric ca7edd47cb
removing target="_blank" from blog link (#3882) 3 years ago
hq1 437a3350ff
Replace caching engine (#3878)
* Dependencies: swap Cachex for ConCache

* Implement Cache adapter wrapping ConCache

* Implement cache stats tracker, for metrics

* Use Cache.Adapter in Plausible.Cache

Marking the test as not slow anymore

* Use Cache Adapter when tracking sessions

* Use Cache Adapter for UA parsing

* Rename child identifiers - cachex is obsolete now

* Test stats tracking

* Update grafana metrics

* Put all caches under common child specification

* Try less

* Shorten the function delegation path
3 years ago
hq1 bb8a272a36
Fix margins in help guides section (#3881) 3 years ago
hq1 b68e3aa6a2
changes to the in-app footer (clean) (#3880)
* changes to the in-app footer

changes to the in-app footer to focus on product usage links rather than marketing pages

* Format

---------

Co-authored-by: Marko Saric <34340819+metmarkosaric@users.noreply.github.com>
3 years ago
ruslandoga 83a46fb47e
Dockerfile and co updates (#3811)
* wip

* more env, setup user before packages to avoid them picking our uid

* make entrypoint.sh executable by all again

* apparently it's a best practice to have executables be owned by root

* make entrypoint executable in COPY

* stop writing to /app, write to /tmp if needed

* fewer changes

* system user

* keep same style for multiline commands

* fewer changes

* add changelog entry

* fix group assignment for plausible user

* use gid=999

* no home

* no home

* add gecos

* add plausible user to nogroup instead of creating a custom one

* eh

* fewer changes

* fewer changes

* fewer changes

* use PERSISTENT_CACHE_DIR instead of STORAGE_DIR

* ignore more

* cleanup

* remove hex timeout env var

* use ERL_FLAGS=+JMsingle true in public builds

* fallback to /tmp and nest under /tzdata_data/ for tzdata

---------

Co-authored-by: Cenk Kücük <cenk@plausible.io>
3 years ago
Karl-Aksel Puulmann 11368e5143
Temporarily disable tzdata auto-updating (#3873)
This can cause issues as 1 day after a deploy the tzdata list is updated
from 2021 to 2024. Selecting a timezone thats not supported in 2021 can
then break dashboards temporarily following a deploy.

This can be reverted after https://github.com/plausible/analytics/pull/3811
3 years ago
Karl-Aksel Puulmann a9d3c03782
Validate the same metric isnt queried multiple times in external stats API (#3871)
* Validate the same metric isnt queried multiple times in external stats API

Issue: https://3.basecamp.com/5308029/buckets/35611491/card_tables/cards/7161347855

* Changelog entry

* Make credo happy
3 years ago
Karl-Aksel Puulmann 26d41ddbb9
Mutation to populate event session columns (#3844)
* WIP mutation to populate event session columns

* Remove duplication

* report errors, allow_nondeterministic_updates

* use right columns

* Update existing columns instead of session_* ones

* Make dialyzer happy

* Fix issue with passing pre-existing params in

* Logger -> IO.puts

* Use IngestRepo.config for connection settings

* Make dictionary options configurable

* Move allow_nondeterministic_mutations to within the migration

* Solve credo warning about too deep nesting

* Missed logger call

* Pattern matching in function head
3 years ago
Karl-Aksel Puulmann 0cdba7d407
Fix broken tests (#3867) 3 years ago
Karl-Aksel Puulmann c6d98397a8
Move `add_percentage` logic into clickhouse (#3854)
* Remove `add_percentage`, calculate percentages in clickhouse queries

This simplifies querying logic and avoids doing extra queries and avoids
race conditions.

* Remove special none handling from breakdowns, handling percentages correctly

* Add (failing) test showing expected add_percentage behavior for user making multiple sessions

* Update add_percentage behavior to use separate subqueries
3 years ago
Karl-Aksel Puulmann c60a2faee4
Write event table session columns (#3865)
* Write event table session columns

* Update testing factory rig
3 years ago
Karl-Aksel Puulmann 70d21f1b69
Improve compression of event table session columns (#3864) 3 years ago
Karl-Aksel Puulmann 8d977e0f76
Tests: session properties without the prefixes (#3863)
* Undo event session attributes renaming

* Rename session_ attributes in tests
3 years ago
Cenk Kücük fecd665c19
Push private images to GHCR (#3859) 3 years ago
Karl-Aksel Puulmann d5048fd6b4
Stop writing session properties into events table (#3800)
* Refactor: Explicitly add field names to INSERT

This avoids issues when code schema is out of sync with real schema

* Dont write session parameters to events

These would only be stored on first event anyways. Work remains to be done
on tests which have their own helper

* Remove writes to country_code in a test

* Remove old columns from being accessible in elixir code

* Update most tests to use new way of adding session props to events

* Update testing harness

* Update stats controller test

* Update for shield rules

* update breakdown tests

* Fix typing of state for dialyzer

* Drop support for old session attributes code

* Update remaining tests

* cond -> if
3 years ago
hq1 5eb9d724e5
Plugins API: mark data_domain as nullable in capabilities schema (#3840)
* Plugins API: mark data_domain as nullable in capabilities schema

* fixup
3 years ago
Karl-Aksel Puulmann ea38b45685
Ecto migration to move sessions_v2 to VersionedCollapsingMergeTree (#3809)
This migration will noop in staging/production as it already has been run. It also leaves
behind a backup table that initially takes no extra space but will need to be cleaned up
manually
3 years ago
Adrian Gruntkowski 39aa81a16f
Implement UI for multiple imports (#3727)
* Create a stub of site settings section for imports and exports

* Use legacy site import indication to determine UA import handling

* Add provisional logos for upcoming import sources

* Stub basics of import page

* Add very rudimentary support for multiple UA imports

* Implement imports list as live view

* Add support for opening LV modal from backend and closing from frontend

* Introduce notion of themes to `button` and `button_link` components

* Add confirmation modal on deleting import

* Swap GA4 logo

* Implement disabled state support for `button_link` component

* Disable export and non-implemented import sources

* Use native starts start date for upper boundary of import time range

* Ensure integrations view uses legacy UA import flow

* Remove unnecessary preload in SiteController

* Remove unnecessary exception for legacy imports

* Move API controller stats tests under PlausibleWeb

* Test listing imports

* Add test for explicit listener setup

* Add tests for legacy flag state in UA importer

* Add test for purging legacy import data

* Add tests for `Sites.native_stats_start_date`

* Test forgetting imports

* Add `Stats.Clickhouse.imported_pageview_counts/1` and fix test flakiness

* Show page view counts on imports list

* Add tests for static imports and exports view

* Adjust button look slightly

* Use `case` instead of `cond`

* Make feature flag customisable per site

* Fix buttons and empty state styling

* Add another import to seeds

* Use JS confirm dialog instead of modal for deletion confirmations

* Revert "Add support for opening LV modal from backend and closing from frontend"

This reverts commit 260e6c753032b451542e24be9edc2118790b5a00.

* Default `legacy` to false when inserting new import jobs

* Drop `method` attribute from `button_link` and `unstyled_link` components
3 years ago
Adrian Gruntkowski fdbe4cc0a4
Add `legacy` flag column to `site_imports` schema (#3842) 3 years ago
Marko Saric 0fe5682f91
Change the CTA to upgrade now rather than contact (#3837) 3 years ago
hq1 316148e788
Fix settings options mobile (#3838)
* Fix settings options on mobile view

* Fix settings options selected item on mobile

* Format
3 years ago
ruslandoga f3423aefec
Add csv importer (#3795)
* add csv importer

* make table validation explicit

* update some docs

* improve docs

* add minio container to ci

* more tests

* eh

* continue

* add passing test

* add failing test

* add config test

* add minio to Makefile

* testcontainers

* remove extra whitespace

* explain the implementation a bit

* account for async deletes in tests

* bounces is UInt32

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
ruslandoga 31cf3e54f8
Add Bamboo.Mua (#3654) 3 years ago
hq1 f1b6a672d4
Fix `added_by` not saving on adding country rules (#3835)
* Fix `added_by` not saving on adding country rules

* Format

* Remove dupe attr
3 years ago
hq1 518cdb3307
Shield: Country Rules (#3828)
* Migration: add country rules

* Add CountryRule schema

* Implement CountryRule cache

* Add country rules context interface

* Start country rules cache

* Lookup country rules on ingestion

* Remove :shields feature flag from test helpers

* Add nested sidebar menu for Shields

* Fix typo

* IP Rules: hide description on mobile view

* Prepare SiteController to handle multiple shield types

* Seed some country shield

* Implement LV for country rules

* Remove "YOU" indicator from country rules

* Fix small build

* Format

* Update typespecs

* Make docs link point at /countries

* Fix flash on top of modal for Safari

* Build the rule struct with site_id provided up-front

* Clarify why we're messaging the ComboBox component

* Re-open combobox suggestions after pressing Escape

* Update changelog

* Fix font size in country table cells

* Pass `added_by` via rule add options

* Display site's timezone timestamps in rule tooltips

* Display formatted timestamps in site's timezone

And simplify+test Timezone module; an input timestamp converted
to UTC can never be ambiguous.

* Remove no-op atom

* Display the maximum number of rules when reached

* Improve readability of remove button tests

* Credo

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
hq1 b7b5dcf4c9
Migration: add country rules (#3834)
* Migration: add country rules

* Make `country_code` not nullable
3 years ago
Cenk Kücük 8a8df685a7
Update country code for anonymising VPN services (#3766)
* Update country code for anonymising VPN services

* Update to latest
3 years ago
RobertJoonas dd428430f5
Query imported data for views_per_visit (#3830)
* query imported data for views_per_visit

* changelog update
3 years ago
Cenk Kücük cc867396b2
Update elixir.yml (#3831)
Only test against the last 2 major versions
3 years ago
ruslandoga bea8ceeffb
use newer postgres versions in the ci (#3821) 3 years ago
hq1 c822f8de1e
Hide user name overlay on mobile (#3829) 3 years ago
Karl-Aksel Puulmann c6e06359c6
Avoid changing Custom properties tab automatically every time filter changes (#3777)
Follow-up to https://github.com/plausible/analytics/pull/3719

The previous behavior was predicated on:
- Allowing a single custom property filter
- Allowing breaking down only by the chosen custom property filter

Now these restrictions are removed the previous auto-switching just becomes annoying
3 years ago
Marko Saric 19ce62db61
Update README.md 3 years ago
Uku Taht a1fedb751b Update license hash 3 years ago
Cenk Kücük 0f0f604d2e
Branding stuff (#3816)
* Replace footer text

* add COPYING.txt file

* Add new logos

* Use new logos in all layouts

* New logos

* Check license key on startup

* Bypass license check when Mix.env == :dev

* Use new logos with smaller wordmarks

* Add generic logo_path/1 function

* Use new favicons everywhere

* Bypass license check in test env

* Use sha256 for license key hash

* Mix.env -> config_env()

* Use Mix.evn at compile time rather than runtime

* Mix format

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
3 years ago
Cenk Kücük bdaf80c05d
Publish community edition under plausible/community-edition on GHCR and dockerhub (#3815) 3 years ago
Karl-Aksel Puulmann acd4f33c8e
Migration to improve compression options of sessions_v2 and events_v2 tables (#3803)
Part of https://3.basecamp.com/5308029/buckets/35611491/messages/7061875211
Based on work in https://3.basecamp.com/5308029/buckets/35611491/messages/6949640880#__recording_7015576045
3 years ago
hq1 bca617d0bf
`String.to_atom/1` is safe to do at compile-time (#3813)
otherwise random compile-time errors may occur
3 years ago
RobertJoonas 52f584efa9
Group `os_version` by `os` (#3806)
* improve test

* add os to os_version breakdown

...and add operating_system_versions.csv to the CSV export

* fix conversion rate for os_version breakdown

* update changelog

* fix existing CSV tests

* use case instead of cond
3 years ago
Karl-Aksel Puulmann 8ba2b934b3
Add data migration for moving sessions_v2 table to VersionedCollapsingMergeTree (#3802)
* Add data migration for moving to VersionedCollapsingMergeTree

This has been tested locally and partially on staging. Still requires a bit of work to verify.

Verification query:

```
SELECT main._partition_id, tmp.count, main.count
FROM (
SELECT _partition_id, count() AS count
FROM sessions_v2_tmp_versioned
GROUP BY _partition_id
) AS tmp
FULL OUTER JOIN (
SELECT _partition_id, count() AS count
FROM sessions_v2
GROUP BY _partition_id
) AS main
ON (tmp._partition_id == main._partition_id)
ORDER BY main._partition_id
```

* Add an early exit to migration

* cluster? extract common code
3 years ago
RobertJoonas d74b1d5e60
Reapply conversion rate into Stats API + bugfixes (#3805)
* Revert "Revert api conversion rate (#3789)"

This reverts commit 8e8790dd30.

* fix browser_version CR breakdown bug

* changelog bugfix

* inspect data structures before sending to sentry
3 years ago
hq1 6035618213
Add `GET /capabilities` to Plugins API (#3808)
* Add `GET /capabilities` to Plugins API

It aims to:

 - help the client verify the data-domain the token is associated with
 - list all the features available for the site's owner
   (and therefore determine availability of the subset of those for the current
   Plugins API caller)

The endpoint does not require authentication, in the sense that it'll
always respond with 200 OK. However when the token is provided,
a verification lookup is made.

* Remove IO.inspect() call

* Credo

* Aesthetics

* s/send_resp/send_error/

* Call preload just once
3 years ago
hq1 23b9032148
Allow site deletion with IP Rules stored (#3804)
* Allow site deletion with IP Rules stored

* drop_if_exists
3 years ago

@ -0,0 +1,73 @@
# This file excludes paths from the Docker build context.
#
# By default, Docker's build context includes all files (and folders) in the
# current directory. Even if a file isn't copied into the container it is still sent to
# the Docker daemon.
#
# There are multiple reasons to exclude files from the build context:
#
# 1. Prevent nested folders from being copied into the container (ex: exclude
# /assets/node_modules when copying /assets)
# 2. Reduce the size of the build context and improve build time (ex. /build, /deps, /doc)
# 3. Avoid sending files containing sensitive information
#
# More information on using .dockerignore is available here:
# https://docs.docker.com/engine/reference/builder/#dockerignore-file
.dockerignore
# Ignore git, but keep git HEAD and refs to access current commit hash if needed:
#
# $ cat .git/HEAD | awk '{print ".git/"$2}' | xargs cat
# d0b8727759e1e0e7aa3d41707d12376e373d5ecc
.git
!.git/HEAD
!.git/refs
# Common development/test artifacts
/cover/
/doc/
/test/
/tmp/
.elixir_ls
plausible-report.xml
.env
.idea
*.iml
*.log
*.code-workspace
.vscode
# Mix artifacts
/_build/
/deps/
*.ez
# Generated on crash by the VM
erl_crash.dump
# If NPM crashes, it generates a log, let's ignore it too.
npm-debug.log
# Static artifacts - These should be fetched and built inside the Docker image
/assets/node_modules/
/tracker/node_modules/
/priv/static/cache_manifest.json
/priv/static/css
/priv/static/js
/priv/version.json
# Auto-generated tracker files
/priv/tracker/js/*.js
# Dializer
/priv/plts/*.plt
/priv/plts/*.plt.hash
# Geolocation databases
/priv/geodb/*.mmdb
/priv/geodb/*.mmdb.gz
# Docker volumes
.clickhouse_db_vol*
plausible_db*

@ -0,0 +1,67 @@
name: Build Private Images GHCR
on:
push:
branches: [master, stable]
tags: ['r*']
pull_request:
types: [synchronize, labeled]
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
if: ${{ github.event_name == 'push' || contains(github.event.pull_request.labels.*.name, 'deploy-to-staging') }}
runs-on: buildjet-16vcpu-ubuntu-2204
permissions:
contents: read
packages: write
steps:
- name: Docker meta
id: meta
uses: docker/metadata-action@v5.0.0
with:
images: ghcr.io/plausible/analytics/ee
tags: |
type=ref,event=pr
type=ref,event=branch
type=ref,event=tag
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push
id: docker_build
uses: docker/build-push-action@v5
with:
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
build-args: |
MIX_ENV=prod
BUILD_METADATA=${{ steps.meta.outputs.json }}
- name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}
- name: Notify team on failure
if: ${{ failure() }}
uses: fjogeleit/http-request-action@v1
with:
url: ${{ secrets.BUILD_NOTIFICATION_URL }}
method: 'POST'
customHeaders: '{"Content-Type": "application/json"}'
data: '{"content": "<a href=\"https://github.com/plausible/analytics/actions/workflows/build-private-images.yml\">Build failed</a>"}'

@ -0,0 +1,68 @@
name: Build Public Images GHCR
on:
push:
tags: ['v*']
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
runs-on: buildjet-16vcpu-ubuntu-2204
permissions:
contents: read
packages: write
steps:
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/plausible/community-edition
tags: |
type=semver,pattern={{version}},prefix=v
type=semver,pattern={{major}}.{{minor}},prefix=v
type=semver,pattern={{major}},prefix=v
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push
id: docker_build
uses: docker/build-push-action@v5
with:
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
platforms: linux/amd64, linux/arm64
cache-from: type=gha
cache-to: type=gha,mode=max
build-args: |
MIX_ENV=small
BUILD_METADATA=${{ steps.meta.outputs.json }}
ERL_FLAGS=+JPperf true
- name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}
- name: Notify team on failure
if: ${{ failure() }}
uses: fjogeleit/http-request-action@v1
with:
url: ${{ secrets.BUILD_NOTIFICATION_URL }}
method: 'POST'
customHeaders: '{"Content-Type": "application/json"}'
data: '{"content": "<a href=\"https://github.com/plausible/analytics/actions/workflows/build-public-images.yml\">Build failed</a>"}'

@ -17,7 +17,7 @@ jobs:
id: meta id: meta
uses: docker/metadata-action@v5 uses: docker/metadata-action@v5
with: with:
images: plausible/analytics images: plausible/community-edition
tags: | tags: |
type=semver,pattern={{version}},prefix=v type=semver,pattern={{version}},prefix=v
type=semver,pattern={{major}}.{{minor}},prefix=v type=semver,pattern={{major}}.{{minor}},prefix=v
@ -48,7 +48,7 @@ jobs:
build-args: | build-args: |
MIX_ENV=small MIX_ENV=small
BUILD_METADATA=${{ steps.meta.outputs.json }} BUILD_METADATA=${{ steps.meta.outputs.json }}
ERL_FLAGS=+JPperf true ERL_FLAGS=+JMsingle true
- name: Image digest - name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }} run: echo ${{ steps.docker_build.outputs.digest }}

@ -1,26 +1,42 @@
name: Elixir CI name: Elixir CI
on: on:
pull_request:
push: push:
branches: [master, stable] branches: [master, stable]
pull_request:
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.ref }} group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true cancel-in-progress: true
env:
CACHE_VERSION: v7
PERSISTENT_CACHE_DIR: cached
jobs: jobs:
build: build:
name: Build and test name: "Build and test (${{ matrix.mix_env }}, ${{ matrix.postgres_image }}${{ matrix.test_experimental_reduced_joins == '1' && ', experimental_reduced_joins' || '' }})"
runs-on: ubuntu-latest runs-on: ubuntu-latest
strategy: strategy:
matrix: matrix:
postgres_version: [13, 14, 15] mix_env: ['test', 'small_test']
postgres_image: ['postgres:16']
test_experimental_reduced_joins: ['0']
include:
- mix_env: 'test'
postgres_image: 'postgres:15'
test_experimental_reduced_joins: '0'
- mix_env: 'test'
postgres_image: 'postgres:16'
test_experimental_reduced_joins: '1'
env:
MIX_ENV: ${{ matrix.mix_env }}
TEST_EXPERIMENTAL_REDUCED_JOINS: ${{ matrix.test_experimental_reduced_joins }}
services: services:
postgres: postgres:
image: postgres:${{ matrix.postgres_version }} image: ${{ matrix.postgres_image }}
ports: ports:
- 5432:5432 - 5432:5432
env: env:
@ -30,7 +46,6 @@ jobs:
--health-interval 10s --health-interval 10s
--health-timeout 5s --health-timeout 5s
--health-retries 5 --health-retries 5
clickhouse: clickhouse:
image: clickhouse/clickhouse-server:23.3.7.5-alpine image: clickhouse/clickhouse-server:23.3.7.5-alpine
ports: ports:
@ -41,43 +56,91 @@ jobs:
--health-interval 10s --health-interval 10s
--health-timeout 5s --health-timeout 5s
--health-retries 5 --health-retries 5
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: marocchino/tool-versions-action@v1
id: versions
- uses: erlef/setup-beam@v1
with:
elixir-version: ${{ steps.versions.outputs.elixir }}
otp-version: ${{ steps.versions.outputs.erlang }}
- uses: actions/cache@v4
with:
path: |
deps
_build
tracker/node_modules
priv/tracker/js
${{ env.PERSISTENT_CACHE_DIR }}
key: ${{ env.MIX_ENV }}-${{ env.CACHE_VERSION }}-${{ github.head_ref || github.ref }}-${{ hashFiles('**/mix.lock') }}
restore-keys: |
${{ env.MIX_ENV }}-${{ env.CACHE_VERSION }}-${{ github.head_ref || github.ref }}-
${{ env.MIX_ENV }}-${{ env.CACHE_VERSION }}-refs/heads/master-
- name: Check for changes in tracker/**
uses: dorny/paths-filter@v3
id: changes
with:
filters: |
tracker:
- 'tracker/**'
- name: Check if priv/tracker/js/plausible.js exists
run: |
if [ -f priv/tracker/js/plausible.js ]; then
echo "HAS_BUILT_TRACKER=true" >> $GITHUB_ENV
else
echo "HAS_BUILT_TRACKER=false" >> $GITHUB_ENV
fi
- run: npm install --prefix ./tracker
if: steps.changes.outputs.tracker == 'true' || env.HAS_BUILT_TRACKER == 'false'
- run: npm run deploy --prefix ./tracker
if: steps.changes.outputs.tracker == 'true' || env.HAS_BUILT_TRACKER == 'false'
- run: mix deps.get --only $MIX_ENV
- run: mix compile --warnings-as-errors --all-warnings
- run: mix do ecto.create, ecto.migrate
- run: mix run -e "Tzdata.ReleaseUpdater.poll_for_update"
- run: mix test --include slow --include minio --max-failures 1 --warnings-as-errors
if: env.MIX_ENV == 'test'
- run: mix test --include slow --max-failures 1 --warnings-as-errors
if: env.MIX_ENV == 'small_test'
static:
name: Static checks (format, credo, dialyzer)
env:
MIX_ENV: test
runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with: with:
fetch-depth: 0 fetch-depth: 0
- name: Read .tool-versions
uses: marocchino/tool-versions-action@v1 - uses: marocchino/tool-versions-action@v1
id: versions id: versions
- name: Set up Elixir - uses: erlef/setup-beam@v1
uses: erlef/setup-beam@v1
with: with:
elixir-version: ${{steps.versions.outputs.elixir}} elixir-version: ${{ steps.versions.outputs.elixir }}
otp-version: ${{ steps.versions.outputs.erlang}} otp-version: ${{ steps.versions.outputs.erlang }}
- name: Restore dependencies cache
uses: buildjet/cache@v4 - uses: actions/cache@v4
with: with:
path: | path: |
deps deps
_build _build
priv/plts priv/plts
key: ${{ runner.os }}-mix-v7-${{ hashFiles('**/mix.lock') }} key: static-${{ env.MIX_ENV }}-${{ env.CACHE_VERSION }}-${{ github.head_ref || github.ref }}-${{ hashFiles('**/mix.lock') }}
restore-keys: ${{ runner.os }}-mix-v7- restore-keys: |
- name: Install dependencies static-${{ env.MIX_ENV }}-${{ env.CACHE_VERSION }}-${{ github.head_ref || github.ref }}-
run: mix deps.get && npm install --prefix ./tracker static-${{ env.MIX_ENV }}-${{ env.CACHE_VERSION }}-refs/heads/master-
- name: Check Formatting
run: mix format --check-formatted - run: mix deps.get
- name: Check Compile Warnings - run: mix compile --warnings-as-errors --all-warnings
run: mix compile --warnings-as-errors --all-warnings - run: mix format --check-formatted
- name: Generate tracker files - run: mix deps.unlock --check-unused
run: npm run deploy --prefix ./tracker - run: mix credo diff --from-git-merge-base origin/master
- name: Check Credo Warnings - run: mix dialyzer
run: mix credo diff --from-git-merge-base origin/master
- name: Run tests
run: mix test --include slow --max-failures 1 --warnings-as-errors
- name: Run tests (small build)
run: MIX_ENV=small_test mix test --include slow --max-failures 1 --warnings-as-errors
- name: Check Dialyzer
run: mix dialyzer
env:
MIX_ENV: test

@ -0,0 +1,28 @@
name: Validate migrations
on:
pull_request:
paths:
- 'priv/repo/migrations/**'
- 'priv/ingest_repo/migrations/**'
jobs:
validate:
name: App code does not change at the same time
runs-on: ubuntu-latest
steps:
- uses: dorny/paths-filter@v3
id: changes
with:
list-files: json
filters: |
lib:
- 'lib/**'
- 'extra/**'
- 'config/**'
- if: steps.changes.outputs.lib == 'true'
run: |
echo "::error file=${{ fromJSON(steps.changes.outputs.lib_files)[0] }}::Code and migrations shouldn't be changed at the same time"
exit 1

3
.gitignore vendored

@ -19,6 +19,9 @@ erl_crash.dump
# Also ignore archive artifacts (built via "mix archive.build"). # Also ignore archive artifacts (built via "mix archive.build").
*.ez *.ez
# Temporary files, for example, from tests.
/tmp/
# Ignore package tarball (built via "mix hex.build"). # Ignore package tarball (built via "mix hex.build").
plausible-*.tar plausible-*.tar

@ -2,6 +2,19 @@
All notable changes to this project will be documented in this file. All notable changes to this project will be documented in this file.
### Added ### Added
- Pages Block List in Site Settings
- Add `conversion_rate` to Stats API Timeseries and on the main graph
- Add `total_conversions` and `conversion_rate` to `visitors.csv` in a goal-filtered CSV export
- Ability to display total conversions (with a goal filter) on the main graph
- Add `conversion_rate` to Stats API Timeseries and on the main graph
- Add `time_on_page` metric into the Stats API
- County Block List in Site Settings
- Query the `views_per_visit` metric based on imported data as well if possible
- Group `operating_system_versions` by `operating_system` in Stats API breakdown
- Add `operating_system_versions.csv` into the CSV export
- Display `Total visitors`, `Conversions`, and `CR` in the "Details" views of Countries, Regions and Cities (when filtering by a goal)
- Add `conversion_rate` to Regions and Cities reports (when filtering by a goal)
- Add the `conversion_rate` metric to Stats API Breakdown and Aggregate endpoints
- IP Block List in Site Settings - IP Block List in Site Settings
- Allow filtering with `contains`/`matches` operator for Sources, Browsers and Operating Systems. - Allow filtering with `contains`/`matches` operator for Sources, Browsers and Operating Systems.
- Allow filtering by multiple custom properties - Allow filtering by multiple custom properties
@ -22,21 +35,34 @@ All notable changes to this project will be documented in this file.
- Add support for 2FA authentication - Add support for 2FA authentication
- Add 'browser_versions.csv' to CSV export - Add 'browser_versions.csv' to CSV export
- Add `CLICKHOUSE_MAX_BUFFER_SIZE_BYTES` env var which defaults to `100000` (100KB) - Add `CLICKHOUSE_MAX_BUFFER_SIZE_BYTES` env var which defaults to `100000` (100KB)
- Add alternative SMTP adapter plausible/analytics#3654
- Add `EXTRA_CONFIG_PATH` env var to specify extra Elixir config plausible/analytics#3906
- Add restrictive `robots.txt` for self-hosted plausible/analytics#3905
### Removed ### Removed
- Removed the nested custom event property breakdown UI when filtering by a goal in Goal Conversions - Removed the nested custom event property breakdown UI when filtering by a goal in Goal Conversions
- Removed the `prop_names` returned in the Stats API `event:goal` breakdown response - Removed the `prop_names` returned in the Stats API `event:goal` breakdown response
- Removed the `prop-breakdown.csv` file from CSV export - Removed the `prop-breakdown.csv` file from CSV export
- Deprecated `CLICKHOUSE_MAX_BUFFER_SIZE` - Deprecated `CLICKHOUSE_MAX_BUFFER_SIZE`
- Removed `/app/init-admin.sh` that was deprecated in v2.0.0 plausible/analytics#3903
- Remove `DISABLE_AUTH` deprecation warning plausible/analytics#3904
### Changed ### Changed
- A visits `entry_page` and `exit_page` is only set and updated for pageviews, not custom events
- Limit the number of Goal Conversions shown on the dashboard and render a "Details" link when there are more entries to show - Limit the number of Goal Conversions shown on the dashboard and render a "Details" link when there are more entries to show
- Show Outbound Links / File Downloads / 404 Pages / Cloaked Links instead of Goal Conversions when filtering by the corresponding goal - Show Outbound Links / File Downloads / 404 Pages / Cloaked Links instead of Goal Conversions when filtering by the corresponding goal
- Require custom properties to be explicitly added from Site Settings > Custom Properties in order for them to show up on the dashboard - Require custom properties to be explicitly added from Site Settings > Custom Properties in order for them to show up on the dashboard
- GA/SC sections moved to new settings: Integrations - GA/SC sections moved to new settings: Integrations
- Replace `CLICKHOUSE_MAX_BUFFER_SIZE` with `CLICKHOUSE_MAX_BUFFER_SIZE_BYTES` - Replace `CLICKHOUSE_MAX_BUFFER_SIZE` with `CLICKHOUSE_MAX_BUFFER_SIZE_BYTES`
- Validate metric isn't queried multiple times
### Fixed ### Fixed
- Creating many sites no longer leads to cookie overflow
- Ignore sessions without pageviews for `entry_page` and `exit_page` breakdowns
- Using `VersionedCollapsingMergeTree` to store visit data to avoid rare race conditions that led to wrong visit data being shown
- Fix `conversion_rate` metric in a `browser_versions` breakdown
- Calculate `conversion_rate` percentage change in the same way like `bounce_rate` (subtraction instead of division)
- Calculate `bounce_rate` percentage change in the Stats API in the same way as it's done in the dashboard
- Stop returning custom events in goal breakdown with a pageview goal filter and vice versa - Stop returning custom events in goal breakdown with a pageview goal filter and vice versa
- Only return `(none)` values in custom property breakdown for the first page (pagination) of results - Only return `(none)` values in custom property breakdown for the first page (pagination) of results
- Fixed weekly/monthly e-mail report [rendering issues](https://github.com/plausible/analytics/issues/284) - Fixed weekly/monthly e-mail report [rendering issues](https://github.com/plausible/analytics/issues/284)
@ -51,6 +77,10 @@ All notable changes to this project will be documented in this file.
- Allow custom event timeseries in stats API plausible/analytics#3505 - Allow custom event timeseries in stats API plausible/analytics#3505
- Fixes for sites with UTF characters in domain plausible/analytics#3560 - Fixes for sites with UTF characters in domain plausible/analytics#3560
- Fix crash when using special characters in filter plausible/analytics#3634 - Fix crash when using special characters in filter plausible/analytics#3634
- Fix automatic scrolling to the bottom on the dashboard if previously selected properties tab plausible/analytics#3872
- Allow running the container with arbitrary UID plausible/analytics#2986
- Fix `width=manual` in embedded dashboards plausible/analytics#3910
- Fix URL escaping when pipes are used in UTM tags plausible/analytics#3930
## v2.0.0 - 2023-07-12 ## v2.0.0 - 2023-07-12

@ -47,8 +47,7 @@ RUN npm run deploy --prefix ./tracker && \
mix assets.deploy && \ mix assets.deploy && \
mix phx.digest priv/static && \ mix phx.digest priv/static && \
mix download_country_database && \ mix download_country_database && \
# https://hexdocs.pm/sentry/Sentry.Sources.html#module-source-code-storage mix sentry.package_source_code
mix sentry_recompile
WORKDIR /app WORKDIR /app
COPY rel rel COPY rel rel
@ -64,18 +63,15 @@ ENV LANG=C.UTF-8
ARG MIX_ENV=small ARG MIX_ENV=small
ENV MIX_ENV=$MIX_ENV ENV MIX_ENV=$MIX_ENV
RUN apk upgrade --no-cache RUN adduser -S -H -u 999 -G nogroup plausible -g 'Plausible Analytics'
RUN apk upgrade --no-cache
RUN apk add --no-cache openssl ncurses libstdc++ libgcc ca-certificates RUN apk add --no-cache openssl ncurses libstdc++ libgcc ca-certificates
COPY ./rel/docker-entrypoint.sh /entrypoint.sh COPY --from=buildcontainer --chmod=a+rX /app/_build/${MIX_ENV}/rel/plausible /app
COPY --chmod=755 ./rel/docker-entrypoint.sh /entrypoint.sh
RUN chmod a+x /entrypoint.sh && \
adduser -h /app -u 1000 -s /bin/sh -D plausibleuser
COPY --from=buildcontainer /app/_build/${MIX_ENV}/rel/plausible /app USER 999
RUN chown -R plausibleuser:plausibleuser /app
USER plausibleuser
WORKDIR /app WORKDIR /app
ENV LISTEN_IP=0.0.0.0 ENV LISTEN_IP=0.0.0.0
ENTRYPOINT ["/entrypoint.sh"] ENTRYPOINT ["/entrypoint.sh"]

@ -36,3 +36,12 @@ postgres-prod: ## Start a container with the same version of postgres as the one
postgres-stop: ## Stop and remove the postgres container postgres-stop: ## Stop and remove the postgres container
docker stop plausible_db && docker rm plausible_db docker stop plausible_db && docker rm plausible_db
minio: ## Start a transient container with a recent version of minio (s3)
docker run -d --rm -p 10000:10000 -p 10001:10001 --name plausible_minio minio/minio server /data --address ":10000" --console-address ":10001"
while ! docker exec plausible_minio mc alias set local http://localhost:10000 minioadmin minioadmin; do sleep 1; done
docker exec plausible_minio mc mb local/dev-exports
docker exec plausible_minio mc mb local/dev-imports
minio-stop: ## Stop and remove the minio container
docker stop plausible_minio

@ -15,7 +15,7 @@
<br /><br /> <br /><br />
</p> </p>
[Plausible Analytics](https://plausible.io/) is an easy to use, lightweight (< 1 KB), open source and privacy-friendly alternative to Google Analytics. It doesn’t use cookies and is fully compliant with GDPR, CCPA and PECR. You can self-host or have us run Plausible for you in the cloud. Here's [the live demo of our own website stats](https://plausible.io/plausible.io). Made and hosted in the EU 🇪🇺 [Plausible Analytics](https://plausible.io/) is an easy to use, lightweight (< 1 KB), open source and privacy-friendly alternative to Google Analytics. It doesn’t use cookies and is fully compliant with GDPR, CCPA and PECR. You can self-host Plausible Community Edition or have us manage Plausible Analytics for you in the cloud. Here's [the live demo of our own website stats](https://plausible.io/plausible.io). Made and hosted in the EU 🇪🇺
We are dedicated to making web analytics more privacy-friendly. Our mission is to reduce corporate surveillance by providing an alternative web analytics tool which doesn’t come from the AdTech world. We are completely independent and solely funded by our subscribers. We are dedicated to making web analytics more privacy-friendly. Our mission is to reduce corporate surveillance by providing an alternative web analytics tool which doesn’t come from the AdTech world. We are completely independent and solely funded by our subscribers.
@ -23,7 +23,7 @@ We are dedicated to making web analytics more privacy-friendly. Our mission is t
## Why Plausible? ## Why Plausible?
Here's what makes Plausible a great Google Analytics alternative and why we're trusted by 11,000+ paying subscribers to deliver their website and business insights: Here's what makes Plausible a great Google Analytics alternative and why we're trusted by 12,000+ paying subscribers to deliver their website and business insights:
- **Clutter Free**: Plausible Analytics provides [simple web analytics](https://plausible.io/simple-web-analytics) and it cuts through the noise. No layers of menus, no need for custom reports. Get all the important insights on one single page. No training necessary. - **Clutter Free**: Plausible Analytics provides [simple web analytics](https://plausible.io/simple-web-analytics) and it cuts through the noise. No layers of menus, no need for custom reports. Get all the important insights on one single page. No training necessary.
- **GDPR/CCPA/PECR compliant**: Measure traffic, not individuals. No personal data or IP addresses are ever stored in our database. We don't use cookies or any other persistent identifiers. [Read more about our data policy](https://plausible.io/data-policy) - **GDPR/CCPA/PECR compliant**: Measure traffic, not individuals. No personal data or IP addresses are ever stored in our database. We don't use cookies or any other persistent identifiers. [Read more about our data policy](https://plausible.io/data-policy)
@ -49,32 +49,32 @@ We choose the subscription business model rather than the business model of surv
## Getting started with Plausible ## Getting started with Plausible
The easiest way to get started with Plausible is with [our official managed service in the cloud](https://plausible.io/#pricing). It takes 2 minutes to start counting your stats with a worldwide CDN, high availability, backups, security and maintenance all done for you by us. The easiest way to get started with Plausible Analytics is with [our official managed service in the cloud](https://plausible.io/#pricing). It takes 2 minutes to start counting your stats with a worldwide CDN, high availability, backups, security and maintenance all done for you by us.
In order to be compliant with the GDPR and the Schrems II ruling, all visitor data for our managed service in the cloud is exclusively processed on servers and cloud infrastructure owned and operated by European providers. Your website data never leaves the EU. In order to be compliant with the GDPR and the Schrems II ruling, all visitor data for our managed service in the cloud is exclusively processed on servers and cloud infrastructure owned and operated by European providers. Your website data never leaves the EU.
Our managed hosting can save a substantial amount of developer time and resources. For most sites this ends up being the best value option and the revenue goes to funding the maintenance and further development of Plausible. So you’ll be supporting open source software and getting a great service! Our managed hosting can save a substantial amount of developer time and resources. For most sites this ends up being the best value option and the revenue goes to funding the maintenance and further development of Plausible. So you’ll be supporting open source software and getting a great service!
### Can Plausible Analytics be self-hosted? ### Can Plausible be self-hosted?
Plausible is [open source web analytics](https://plausible.io/open-source-website-analytics) and we have a free as in beer [self-hosted solution](https://plausible.io/self-hosted-web-analytics). Here are the differences between Plausible Analytics managed hosting in the cloud and the self-hosting: Plausible is [open source web analytics](https://plausible.io/open-source-website-analytics) and we have a free as in beer and self-hosted solution called [Plausible Community Edition (CE)](https://plausible.io/self-hosted-web-analytics). Here are the differences between Plausible Analytics managed hosting in the cloud and the Plausible CE:
| | Plausible Cloud | Self-Hosting | | | Plausible Analytics Cloud | Plausible Community Edition |
| ------------- | ------------- | ------------- | | ------------- | ------------- | ------------- |
| **Infrastructure management** | Easy and convenient. It takes 2 minutes to start counting your stats with a worldwide CDN, high availability, backups, security and maintenance all done for you by us. We manage everything so you don’t have to worry about anything and can focus on your stats. | You do it all yourself. You need to get a server and you need to manage your infrastructure. You are responsible for installation, maintenance, upgrades, server capacity, uptime, backup, security, stability, consistency, loading time and so on.| | **Infrastructure management** | Easy and convenient. It takes 2 minutes to start counting your stats with a worldwide CDN, high availability, backups, security and maintenance all done for you by us. We manage everything so you don’t have to worry about anything and can focus on your stats. | You do it all yourself. You need to get a server and you need to manage your infrastructure. You are responsible for installation, maintenance, upgrades, server capacity, uptime, backup, security, stability, consistency, loading time and so on.|
| **Release schedule** | Continuously developed and improved with new features and updates multiple times per week. | [It's a long term release](https://plausible.io/blog/building-open-source) published twice per year so latest features and improvements won't be immediately available.| | **Release schedule** | Continuously developed and improved with new features and updates multiple times per week. | [It's a long term release](https://plausible.io/blog/building-open-source) published twice per year so latest features and improvements won't be immediately available.|
| **Premium features** | All features available as listed in [our pricing plans](https://plausible.io/#pricing). | Selected premium features such as funnels and ecommerce revenue goals are not available as we aim to ensure a protective barrier around our cloud offering.| | **Premium features** | All features available as listed in [our pricing plans](https://plausible.io/#pricing). | Selected premium features such as funnels and ecommerce revenue goals are not available as we aim to ensure a [protective barrier around our cloud offering](https://plausible.io/blog/community-edition).|
| **Bot filtering** | Advanced bot filtering for more accurate stats. Our algorithm detects and excludes non-human traffic patterns. We also exclude known bots by the User-Agent header and filter out traffic from data centers and referrer spam domains. | Basic bot filtering that targets the most common non-human traffic based on the User-Agent header and referrer spam domains.| | **Bot filtering** | Advanced bot filtering for more accurate stats. Our algorithm detects and excludes non-human traffic patterns. We also exclude known bots by the User-Agent header and filter out traffic from data centers and referrer spam domains. | Basic bot filtering that targets the most common non-human traffic based on the User-Agent header and referrer spam domains.|
| **Server location** | All visitor data is exclusively processed on EU-owned cloud infrastructure. We keep your site data on a secure, encrypted and green energy powered server in Germany. This ensures that your site data is protected by the strict European Union data privacy laws and ensures compliance with GDPR. Your website data never leaves the EU. | You have full control and can host your instance on any server in any country that you wish. Host it on a server in your basement or host it with any cloud provider wherever you want, even those that are not GDPR compliant.| | **Server location** | All visitor data is exclusively processed on EU-owned cloud infrastructure. We keep your site data on a secure, encrypted and green energy powered server in Germany. This ensures that your site data is protected by the strict European Union data privacy laws and ensures compliance with GDPR. Your website data never leaves the EU. | You have full control and can host your instance on any server in any country that you wish. Host it on a server in your basement or host it with any cloud provider wherever you want, even those that are not GDPR compliant.|
| **Data portability** | You see all your site stats and metrics on our modern-looking, simple to use and fast loading dashboard. You can only see the stats aggregated in the dashboard. You can download the stats using the [CSV export](https://plausible.io/docs/export-stats), [stats API](https://plausible.io/docs/stats-api) or tools such as the [Data Studio Connector](https://plausible.io/docs/integration-guides#google-data-studio). | Do you want access to the raw data? Self-hosting gives you that option. You can take the data directly from the ClickHouse database. | | **Data portability** | You see all your site stats and metrics on our modern-looking, simple to use and fast loading dashboard. You can only see the stats aggregated in the dashboard. You can download the stats using the [CSV export](https://plausible.io/docs/export-stats), [stats API](https://plausible.io/docs/stats-api) or tools such as the [Data Studio Connector](https://plausible.io/docs/integration-guides#google-data-studio). | Do you want access to the raw data? Self-hosting gives you that option. You can take the data directly from the ClickHouse database. |
| **Premium support** | Real support delivered by real human beings who build and maintain Plausible. | Premium support is not included. Self-hosting is community supported only.| | **Premium support** | Real support delivered by real human beings who build and maintain Plausible. | Premium support is not included. CE is community supported only.|
| **Costs** | There's a cost associated with providing an analytics service so we charge a subscription fee. We choose the subscription business model rather than the business model of surveillance capitalism. Your money funds further development of Plausible. | You need to pay for your server, CDN, backups and whatever other cost there is associated with running the infrastructure. You never have to pay any fees to us. Your money goes to 3rd party companies with no connection to us.| | **Costs** | There's a cost associated with providing an analytics service so we charge a subscription fee. We choose the subscription business model rather than the business model of surveillance capitalism. Your money funds further development of Plausible. | You need to pay for your server, CDN, backups and whatever other cost there is associated with running the infrastructure. You never have to pay any fees to us. Your money goes to 3rd party companies with no connection to us.|
Interested in self-hosting Plausible on your server? Take a look at our [self-hosting installation instructions](https://plausible.io/docs/self-hosting). Interested in self-hosting Plausible CE on your server? Take a look at our [Plausible CE installation instructions](https://github.com/plausible/community-edition/).
Plausible Self-Hosted is a community supported project and there are no guarantees that you will get support from the creators of Plausible to troubleshoot your self-hosting issues. There is a [community supported forum](https://github.com/plausible/analytics/discussions/categories/self-hosted-support) where you can ask for help. Plausible CE is a community supported project and there are no guarantees that you will get support from the creators of Plausible to troubleshoot your self-hosting issues. There is a [community supported forum](https://github.com/plausible/analytics/discussions/categories/self-hosted-support) where you can ask for help.
Our only source of funding is our premium, managed service for running Plausible in the cloud. If you're looking for an alternative way to support the project, we've put together some sponsorship packages. If you choose to self-host Plausible you can [become a sponsor](https://github.com/sponsors/plausible) which is a great way to give back to the community and to contribute to the long-term sustainability of the project. Our only source of funding is our premium, managed service for running Plausible in the cloud.
## Technology ## Technology
@ -95,7 +95,7 @@ To stay up to date with all the latest news and product updates, make sure to fo
## License & Trademarks ## License & Trademarks
Plausible is open source under the GNU Affero General Public License Version 3 (AGPLv3) or any later version. You can [find it here](https://github.com/plausible/analytics/blob/master/LICENSE.md). Plausible CE is open source under the GNU Affero General Public License Version 3 (AGPLv3) or any later version. You can [find it here](https://github.com/plausible/analytics/blob/master/LICENSE.md).
To avoid issues with AGPL virality, we've released the JavaScript tracker which gets included on your website under the MIT license. You can [find it here](https://github.com/plausible/analytics/blob/master/tracker/LICENSE.md). To avoid issues with AGPL virality, we've released the JavaScript tracker which gets included on your website under the MIT license. You can [find it here](https://github.com/plausible/analytics/blob/master/tracker/LICENSE.md).

@ -22,6 +22,7 @@ if (container) {
conversionsOptedOut: container.dataset.conversionsOptedOut === 'true', conversionsOptedOut: container.dataset.conversionsOptedOut === 'true',
funnelsOptedOut: container.dataset.funnelsOptedOut === 'true', funnelsOptedOut: container.dataset.funnelsOptedOut === 'true',
propsOptedOut: container.dataset.propsOptedOut === 'true', propsOptedOut: container.dataset.propsOptedOut === 'true',
revenueGoals: JSON.parse(container.dataset.revenueGoals),
funnels: JSON.parse(container.dataset.funnels), funnels: JSON.parse(container.dataset.funnels),
statsBegin: container.dataset.statsBegin, statsBegin: container.dataset.statsBegin,
nativeStatsBegin: container.dataset.nativeStatsBegin, nativeStatsBegin: container.dataset.nativeStatsBegin,

@ -159,10 +159,10 @@ export default function PlausibleCombobox(props) {
}, []) }, [])
useEffect(() => { useEffect(() => {
if (props.singleOption && props.values.length === 0) { if (props.singleOption && props.values.length === 0 && props.autoFocus) {
searchRef.current.focus() searchRef.current.focus()
} }
}, [props.values.length === 0]) }, [props.values.length === 0, props.singleOption, props.autoFocus])
const searchBoxClass = 'border-none py-1 px-0 w-full inline-block rounded-md focus:outline-none focus:ring-0 text-sm' const searchBoxClass = 'border-none py-1 px-0 w-full inline-block rounded-md focus:outline-none focus:ring-0 text-sm'

@ -0,0 +1,5 @@
Plausible copyright notice
This software is Copyright 2024 Plausible Insights OÜ
This repository is provided for informational purposes only, and no rights to use, distribute or otherwise exploit this software are granted to you. Our other repositories include software that is offered under the Affero GPL, but this repository is not. If you wish to discuss a license, or our hosted service that uses the software, please contact us at hello@plausible.io

@ -258,6 +258,10 @@ class Filters extends React.Component {
) )
} }
trackFilterMenu() {
window.plausible && window.plausible('Filter Menu: Open', {u: `${window.location.protocol}//${window.location.hostname}/:dashboard`})
}
renderDropDown() { renderDropDown() {
const { history, query, site } = this.props; const { history, query, site } = this.props;
@ -266,7 +270,7 @@ class Filters extends React.Component {
{({ open }) => ( {({ open }) => (
<> <>
<div> <div>
<Menu.Button className="flex items-center text-xs md:text-sm font-medium leading-tight px-3 py-2 cursor-pointer ml-auto text-gray-500 dark:text-gray-200 hover:bg-gray-200 dark:hover:bg-gray-900 rounded"> <Menu.Button onClick={this.trackFilterMenu} className="flex items-center text-xs md:text-sm font-medium leading-tight px-3 py-2 cursor-pointer ml-auto text-gray-500 dark:text-gray-200 hover:bg-gray-200 dark:hover:bg-gray-900 rounded">
{this.renderDropdownButton()} {this.renderDropdownButton()}
</Menu.Button> </Menu.Button>
</div> </div>

@ -59,6 +59,7 @@ export function parseQuery(querystring, site) {
'region': q.get('region'), 'region': q.get('region'),
'city': q.get('city'), 'city': q.get('city'),
'page': q.get('page'), 'page': q.get('page'),
'hostname': q.get('hostname'),
'entry_page': q.get('entry_page'), 'entry_page': q.get('entry_page'),
'exit_page': q.get('exit_page') 'exit_page': q.get('exit_page')
} }

@ -52,7 +52,7 @@ export default function Router({ site, loggedIn, currentUserRole }) {
<ExitPagesModal site={site} /> <ExitPagesModal site={site} />
</Route> </Route>
<Route path="/:domain/countries"> <Route path="/:domain/countries">
<ModalTable title="Top countries" site={site} endpoint={url.apiPath(site, '/countries')} filter={{ country: 'code', country_labels: 'name' }} keyLabel="Country" renderIcon={renderCountryIcon} /> <ModalTable title="Top countries" site={site} endpoint={url.apiPath(site, '/countries')} filter={{ country: 'code', country_labels: 'name' }} keyLabel="Country" renderIcon={renderCountryIcon} showPercentage={true}/>
</Route> </Route>
<Route path="/:domain/regions"> <Route path="/:domain/regions">
<ModalTable title="Top regions" site={site} endpoint={url.apiPath(site, '/regions')} filter={{ region: 'code', region_labels: 'name' }} keyLabel="Region" renderIcon={renderRegionIcon} /> <ModalTable title="Top regions" site={site} endpoint={url.apiPath(site, '/regions')} filter={{ region: 'code', region_labels: 'name' }} keyLabel="Region" renderIcon={renderRegionIcon} />

@ -1,4 +1,4 @@
import React, { useCallback, useState, useEffect } from "react"; import React, { useCallback, useState } from "react"
import ListReport from "../reports/list"; import ListReport from "../reports/list";
import Combobox from '../../components/combobox' import Combobox from '../../components/combobox'
import * as api from '../../api' import * as api from '../../api'
@ -15,10 +15,6 @@ export default function Properties(props) {
const [propKey, setPropKey] = useState(choosePropKey()) const [propKey, setPropKey] = useState(choosePropKey())
useEffect(() => {
setPropKey(choosePropKey())
}, [query.filters.goal, query.filters.props])
function singleGoalFilterApplied() { function singleGoalFilterApplied() {
const goalFilter = query.filters.goal const goalFilter = query.filters.goal
if (goalFilter) { if (goalFilter) {

@ -1,27 +1,53 @@
import numberFormatter, {durationFormatter} from '../../util/number-formatter' import numberFormatter, {durationFormatter} from '../../util/number-formatter'
import { parsePrefix } from '../../util/filters'
export const METRIC_MAPPING = { export function getGraphableMetrics(query, site) {
'Unique visitors (last 30 min)': 'visitors', const isRealtime = query.period === 'realtime'
'Pageviews (last 30 min)': 'pageviews', const goalFilter = query.filters.goal
'Unique visitors': 'visitors', const pageFilter = query.filters.page
'Visit duration': 'visit_duration',
'Total pageviews': 'pageviews', if (isRealtime && goalFilter) {
'Views per visit': 'views_per_visit', return ["visitors"]
'Total visits': 'visits', } else if (isRealtime) {
'Bounce rate': 'bounce_rate', return ["visitors", "pageviews"]
'Unique conversions': 'conversions', } else if (goalFilter && canGraphRevenueMetrics(goalFilter, site)) {
'Average revenue': 'average_revenue', return ["visitors", "events", "average_revenue", "total_revenue", "conversion_rate"]
'Total revenue': 'total_revenue', } else if (goalFilter) {
return ["visitors", "events", "conversion_rate"]
} else if (pageFilter) {
return ["visitors", "visits", "pageviews", "bounce_rate", "time_on_page"]
} else {
return ["visitors", "visits", "pageviews", "views_per_visit", "bounce_rate", "visit_duration"]
}
}
// Revenue metrics can only be graphed if:
// * The query is filtered by at least one revenue goal
// * All revenue goals in filter have the same currency
function canGraphRevenueMetrics(goalFilter, site) {
const goalsInFilter = parsePrefix(goalFilter).values
const revenueGoalsInFilter = site.revenueGoals.filter((rg) => {
return goalsInFilter.includes(rg.event_name)
})
const singleCurrency = revenueGoalsInFilter.every((rg) => {
return rg.currency === revenueGoalsInFilter[0].currency
})
return revenueGoalsInFilter.length > 0 && singleCurrency
} }
export const METRIC_LABELS = { export const METRIC_LABELS = {
'visitors': 'Visitors', 'visitors': 'Visitors',
'pageviews': 'Pageviews', 'pageviews': 'Pageviews',
'events': 'Total Conversions',
'views_per_visit': 'Views per Visit', 'views_per_visit': 'Views per Visit',
'visits': 'Visits', 'visits': 'Visits',
'bounce_rate': 'Bounce Rate', 'bounce_rate': 'Bounce Rate',
'visit_duration': 'Visit Duration', 'visit_duration': 'Visit Duration',
'conversions': 'Converted Visitors', 'conversions': 'Converted Visitors',
'conversion_rate': 'Conversion Rate',
'average_revenue': 'Average Revenue', 'average_revenue': 'Average Revenue',
'total_revenue': 'Total Revenue', 'total_revenue': 'Total Revenue',
} }
@ -29,23 +55,17 @@ export const METRIC_LABELS = {
export const METRIC_FORMATTER = { export const METRIC_FORMATTER = {
'visitors': numberFormatter, 'visitors': numberFormatter,
'pageviews': numberFormatter, 'pageviews': numberFormatter,
'events': numberFormatter,
'visits': numberFormatter, 'visits': numberFormatter,
'views_per_visit': (number) => (number), 'views_per_visit': (number) => (number),
'bounce_rate': (number) => (`${number}%`), 'bounce_rate': (number) => (`${number}%`),
'visit_duration': durationFormatter, 'visit_duration': durationFormatter,
'conversions': numberFormatter, 'conversions': numberFormatter,
'conversion_rate': (number) => (`${number}%`),
'total_revenue': numberFormatter, 'total_revenue': numberFormatter,
'average_revenue': numberFormatter, 'average_revenue': numberFormatter,
} }
export const LoadingState = {
loading: 'loading',
refreshing: 'refreshing',
loaded: 'loaded',
isLoadingOrRefreshing: function (state) { return [this.loading, this.refreshing].includes(state) },
isLoadedOrRefreshing: function (state) { return [this.loaded, this.refreshing].includes(state) }
}
const buildComparisonDataset = function(comparisonPlot) { const buildComparisonDataset = function(comparisonPlot) {
if (!comparisonPlot) return [] if (!comparisonPlot) return []

@ -6,7 +6,7 @@ import * as storage from '../../util/storage'
import { isKeyPressed } from '../../keybinding.js' import { isKeyPressed } from '../../keybinding.js'
import { monthsBetweenDates } from '../../util/date.js' import { monthsBetweenDates } from '../../util/date.js'
export const INTERVAL_LABELS = { const INTERVAL_LABELS = {
'minute': 'Minutes', 'minute': 'Minutes',
'hour': 'Hours', 'hour': 'Hours',
'date': 'Days', 'date': 'Days',
@ -14,11 +14,19 @@ export const INTERVAL_LABELS = {
'month': 'Months' 'month': 'Months'
} }
export const getStoredInterval = function(period, domain) { function validIntervals(site, query) {
if (query.period === "custom" && monthsBetweenDates(query.from, query.to) > 12) {
return ["week", "month"]
} else {
return site.validIntervalsByPeriod[query.period]
}
}
function getStoredInterval(period, domain) {
return storage.getItem(`interval__${period}__${domain}`) return storage.getItem(`interval__${period}__${domain}`)
} }
export const storeInterval = function(period, domain, interval) { function storeInterval(period, domain, interval) {
storage.setItem(`interval__${period}__${domain}`, interval) storage.setItem(`interval__${period}__${domain}`, interval)
} }
@ -33,37 +41,47 @@ function subscribeKeybinding(element) {
}, [handleKeyPress]) }, [handleKeyPress])
} }
function DropdownItem({ option, currentInterval, updateInterval }) { export const getCurrentInterval = function(site, query) {
return ( const options = validIntervals(site, query)
<Menu.Item onClick={() => updateInterval(option)} key={option} disabled={option == currentInterval}>
{({ active }) => ( const storedInterval = getStoredInterval(query.period, site.domain)
<span className={classNames({ const defaultInterval = [...options].pop()
'bg-gray-100 dark:bg-gray-900 text-gray-900 dark:text-gray-200 cursor-pointer': active,
'text-gray-700 dark:text-gray-200': !active, if (storedInterval && options.includes(storedInterval)) {
'font-bold cursor-none select-none': option == currentInterval, return storedInterval
}, 'block px-4 py-2 text-sm')}> } else {
{INTERVAL_LABELS[option]} return defaultInterval
</span> }
)}
</Menu.Item>
)
} }
export function IntervalPicker({ graphData, query, site, updateInterval }) { export function IntervalPicker({ query, site, onIntervalUpdate }) {
if (query.period == 'realtime') return null if (query.period == 'realtime') return null
const menuElement = React.useRef(null) const menuElement = React.useRef(null)
subscribeKeybinding(menuElement) const options = validIntervals(site, query)
const currentInterval = getCurrentInterval(site, query)
let currentInterval = graphData?.interval subscribeKeybinding(menuElement)
let options = site.validIntervalsByPeriod[query.period] function updateInterval(interval) {
if (query.period === "custom" && monthsBetweenDates(query.from, query.to) > 12) { storeInterval(query.period, site.domain, interval)
options = ["week", "month"] onIntervalUpdate(interval)
} }
if (!options.includes(currentInterval)) { function renderDropdownItem(option) {
currentInterval = [...options].pop() return (
<Menu.Item onClick={() => updateInterval(option)} key={option} disabled={option == currentInterval}>
{({ active }) => (
<span className={classNames({
'bg-gray-100 dark:bg-gray-900 text-gray-900 dark:text-gray-200 cursor-pointer': active,
'text-gray-700 dark:text-gray-200': !active,
'font-bold cursor-none select-none': option == currentInterval,
}, 'block px-4 py-2 text-sm')}>
{INTERVAL_LABELS[option]}
</span>
)}
</Menu.Item>
)
} }
return ( return (
@ -85,7 +103,7 @@ export function IntervalPicker({ graphData, query, site, updateInterval }) {
leaveFrom="opacity-100 scale-100" leaveFrom="opacity-100 scale-100"
leaveTo="opacity-0 scale-95"> leaveTo="opacity-0 scale-95">
<Menu.Items className="py-1 text-left origin-top-right absolute right-0 mt-2 w-56 rounded-md shadow-lg bg-white dark:bg-gray-800 ring-1 ring-black ring-opacity-5 focus:outline-none z-10" static> <Menu.Items className="py-1 text-left origin-top-right absolute right-0 mt-2 w-56 rounded-md shadow-lg bg-white dark:bg-gray-800 ring-1 ring-black ring-opacity-5 focus:outline-none z-10" static>
{options.map((option) => DropdownItem({ option, currentInterval, updateInterval }))} {options.map(renderDropdownItem)}
</Menu.Items> </Menu.Items>
</Transition> </Transition>
</> </>

@ -0,0 +1,247 @@
import React from 'react';
import { withRouter } from 'react-router-dom'
import Chart from 'chart.js/auto';
import { navigateToQuery } from '../../query'
import GraphTooltip from './graph-tooltip'
import { buildDataSet, METRIC_LABELS, METRIC_FORMATTER } from './graph-util'
import dateFormatter from './date-formatter';
import FadeIn from '../../fade-in';
import classNames from 'classnames';
const calculateMaximumY = function(dataset) {
const yAxisValues = dataset
.flatMap((item) => item.data)
.map((item) => item || 0)
if (yAxisValues) {
return Math.max(...yAxisValues)
} else {
return 1
}
}
class LineGraph extends React.Component {
constructor(props) {
super(props);
this.regenerateChart = this.regenerateChart.bind(this);
this.updateWindowDimensions = this.updateWindowDimensions.bind(this);
}
getGraphMetric() {
let metric = this.props.graphData.metric
if (metric == 'visitors' && this.props.query.filters.goal) {
return 'conversions'
} else {
return metric
}
}
regenerateChart() {
const { graphData, query } = this.props
const metric = this.getGraphMetric()
const graphEl = document.getElementById("main-graph-canvas")
this.ctx = graphEl.getContext('2d');
const dataSet = buildDataSet(graphData.plot, graphData.comparison_plot, graphData.present_index, this.ctx, METRIC_LABELS[metric])
return new Chart(this.ctx, {
type: 'line',
data: {
labels: graphData.labels,
datasets: dataSet
},
options: {
animation: false,
plugins: {
legend: { display: false },
tooltip: {
enabled: false,
mode: 'index',
intersect: false,
position: 'average',
external: GraphTooltip(graphData, metric, query)
},
},
responsive: true,
maintainAspectRatio: false,
onResize: this.updateWindowDimensions,
elements: { line: { tension: 0 }, point: { radius: 0 } },
onClick: this.maybeHopToHoveredPeriod.bind(this),
scale: {
ticks: { precision: 0, maxTicksLimit: 8 }
},
scales: {
y: {
min: 0,
suggestedMax: calculateMaximumY(dataSet),
ticks: {
callback: METRIC_FORMATTER[metric],
color: this.props.darkTheme ? 'rgb(243, 244, 246)' : undefined
},
grid: {
zeroLineColor: 'transparent',
drawBorder: false,
}
},
yComparison: {
min: 0,
suggestedMax: calculateMaximumY(dataSet),
display: false,
grid: { display: false },
},
x: {
grid: { display: false },
ticks: {
callback: function(val, _index, _ticks) {
if (this.getLabelForValue(val) == "__blank__") return ""
const hasMultipleYears =
graphData.labels
.filter((date) => typeof date === 'string')
.map(date => date.split('-')[0])
.filter((value, index, list) => list.indexOf(value) === index)
.length > 1
if (graphData.interval === 'hour' && query.period !== 'day') {
const date = dateFormatter({
interval: "date",
longForm: false,
period: query.period,
shouldShowYear: hasMultipleYears,
})(this.getLabelForValue(val))
const hour = dateFormatter({
interval: graphData.interval,
longForm: false,
period: query.period,
shouldShowYear: hasMultipleYears,
})(this.getLabelForValue(val))
// Returns a combination of date and hour. This is because
// small intervals like hour may return multiple days
// depending on the query period.
return `${date}, ${hour}`
}
if (graphData.interval === 'minute' && query.period !== 'realtime') {
return dateFormatter({
interval: "hour", longForm: false, period: query.period,
})(this.getLabelForValue(val))
}
return dateFormatter({
interval: graphData.interval, longForm: false, period: query.period, shouldShowYear: hasMultipleYears,
})(this.getLabelForValue(val))
},
color: this.props.darkTheme ? 'rgb(243, 244, 246)' : undefined
}
}
},
interaction: {
mode: 'index',
intersect: false,
}
}
});
}
repositionTooltip(e) {
const tooltipEl = document.getElementById('chartjs-tooltip');
if (tooltipEl && window.innerWidth >= 768) {
if (e.clientX > 0.66 * window.innerWidth) {
tooltipEl.style.right = (window.innerWidth - e.clientX) + window.pageXOffset + 'px'
tooltipEl.style.left = null;
} else {
tooltipEl.style.right = null;
tooltipEl.style.left = e.clientX + window.pageXOffset + 'px'
}
tooltipEl.style.top = e.clientY + window.pageYOffset + 'px'
tooltipEl.style.opacity = 1;
}
}
componentDidMount() {
if (this.props.graphData) {
this.chart = this.regenerateChart();
}
window.addEventListener('mousemove', this.repositionTooltip);
}
componentDidUpdate(prevProps) {
const { graphData, darkTheme } = this.props;
const tooltip = document.getElementById('chartjs-tooltip');
if (
graphData !== prevProps.graphData ||
darkTheme !== prevProps.darkTheme
) {
if (graphData) {
if (this.chart) {
this.chart.destroy();
}
this.chart = this.regenerateChart();
this.chart.update();
}
if (tooltip) {
tooltip.style.display = 'none';
}
}
if (!graphData) {
if (this.chart) {
this.chart.destroy();
}
if (tooltip) {
tooltip.style.display = 'none';
}
}
}
componentWillUnmount() {
// Ensure that the tooltip doesn't hang around when we are loading more data
const tooltip = document.getElementById('chartjs-tooltip');
if (tooltip) {
tooltip.style.opacity = 0;
tooltip.style.display = 'none';
}
window.removeEventListener('mousemove', this.repositionTooltip)
}
/**
* The current ticks' limits are set to treat iPad (regular/Mini/Pro) as a regular screen.
* @param {*} chart - The chart instance.
* @param {*} dimensions - An object containing the new dimensions *of the chart.*
*/
updateWindowDimensions(chart, dimensions) {
chart.options.scales.x.ticks.maxTicksLimit = dimensions.width < 720 ? 5 : 8
}
maybeHopToHoveredPeriod(e) {
const element = this.chart.getElementsAtEventForMode(e, 'index', { intersect: false })[0]
const date = this.props.graphData.labels[element.index] || this.props.graphData.comparison_labels[element.index]
if (this.props.graphData.interval === 'month') {
navigateToQuery(this.props.history, this.props.query, { period: 'month', date })
} else if (this.props.graphData.interval === 'date') {
navigateToQuery(this.props.history, this.props.query, { period: 'day', date })
}
}
render() {
const { graphData } = this.props
const canvasClass = classNames('mt-4 select-none', { 'cursor-pointer': !['minute', 'hour'].includes(graphData?.interval) })
return (
<FadeIn show={graphData}>
<div className="relative h-96 print:h-auto print:pb-8 w-full z-0">
<canvas id="main-graph-canvas" className={canvasClass}></canvas>
</div>
</FadeIn>
)
}
}
export default withRouter(LineGraph)

@ -0,0 +1,17 @@
import React from "react"
export default function SamplingNotice({topStatData}) {
const samplePercent = topStatData?.samplePercent
if (samplePercent && samplePercent < 100) {
return (
<div tooltip={`Stats based on a ${samplePercent}% sample of all visitors`} className="cursor-pointer w-4 h-4 mx-2">
<svg className="absolute w-4 h-4 dark:text-gray-300 text-gray-700" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor">
<path strokeLinecap="round" strokeLinejoin="round" strokeWidth="2" d="M13 16h-1v-4h-1m1-4h.01M21 12a9 9 0 11-18 0 9 9 0 0118 0z" />
</svg>
</div>
)
} else {
return null
}
}

@ -0,0 +1,53 @@
import React, { useState } from "react"
import * as api from '../../api'
import { getCurrentInterval } from "./interval-picker"
export default function StatsExport({site, query}) {
const [exporting, setExporting] = useState(false)
function startExport() {
setExporting(true)
document.cookie = "exporting="
pollExportReady()
}
function pollExportReady() {
if (document.cookie.includes('exporting')) {
setTimeout(pollExportReady, 1000)
} else {
setExporting(false)
}
}
function renderLoading() {
return (
<svg className="animate-spin h-4 w-4 text-indigo-500" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24">
<circle className="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" strokeWidth="4"></circle>
<path className="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z"></path>
</svg>
)
}
function renderExportLink() {
const interval = getCurrentInterval(site, query)
const queryParams = api.serializeQuery(query, [{ interval }])
const endpoint = `/${encodeURIComponent(site.domain)}/export${queryParams}`
return (
<a href={endpoint} download onClick={startExport}>
<svg className="absolute text-gray-700 feather dark:text-gray-300 hover:text-indigo-600 dark:hover:text-indigo-600" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4"></path>
<polyline points="7 10 12 15 17 10"></polyline>
<line x1="12" y1="15" x2="12" y2="3"></line>
</svg>
</a>
)
}
return (
<div className="w-4 h-4 mx-2">
{exporting && renderLoading()}
{!exporting && renderExportLink()}
</div>
)
}

@ -3,8 +3,9 @@ import { Tooltip } from '../../util/tooltip'
import { SecondsSinceLastLoad } from '../../util/seconds-since-last-load' import { SecondsSinceLastLoad } from '../../util/seconds-since-last-load'
import classNames from "classnames"; import classNames from "classnames";
import numberFormatter, { durationFormatter } from '../../util/number-formatter' import numberFormatter, { durationFormatter } from '../../util/number-formatter'
import { METRIC_MAPPING } from './graph-util' import * as storage from '../../util/storage'
import { formatDateRange } from '../../util/date.js' import { formatDateRange } from '../../util/date.js'
import { getGraphableMetrics } from "./graph-util.js";
function Maybe({condition, children}) { function Maybe({condition, children}) {
if (condition) { if (condition) {
@ -14,103 +15,103 @@ function Maybe({condition, children}) {
} }
} }
export default class TopStats extends React.Component { function renderPercentageComparison(name, comparison, forceDarkBg = false) {
renderPercentageComparison(name, comparison, forceDarkBg = false) { const formattedComparison = numberFormatter(Math.abs(comparison))
const formattedComparison = numberFormatter(Math.abs(comparison))
const defaultClassName = classNames({
const defaultClassName = classNames({ "pl-2 text-xs dark:text-gray-100": !forceDarkBg,
"pl-2 text-xs dark:text-gray-100": !forceDarkBg, "pl-2 text-xs text-gray-100": forceDarkBg
"pl-2 text-xs text-gray-100": forceDarkBg })
})
const noChangeClassName = classNames({
const noChangeClassName = classNames({ "pl-2 text-xs text-gray-700 dark:text-gray-300": !forceDarkBg,
"pl-2 text-xs text-gray-700 dark:text-gray-300": !forceDarkBg, "pl-2 text-xs text-gray-300": forceDarkBg
"pl-2 text-xs text-gray-300": forceDarkBg })
})
if (comparison > 0) {
if (comparison > 0) { const color = name === 'Bounce rate' ? 'text-red-400' : 'text-green-500'
const color = name === 'Bounce rate' ? 'text-red-400' : 'text-green-500' return <span className={defaultClassName}><span className={color + ' font-bold'}>&uarr;</span> {formattedComparison}%</span>
return <span className={defaultClassName}><span className={color + ' font-bold'}>&uarr;</span> {formattedComparison}%</span> } else if (comparison < 0) {
} else if (comparison < 0) { const color = name === 'Bounce rate' ? 'text-green-500' : 'text-red-400'
const color = name === 'Bounce rate' ? 'text-green-500' : 'text-red-400' return <span className={defaultClassName}><span className={color + ' font-bold'}>&darr;</span> {formattedComparison}%</span>
return <span className={defaultClassName}><span className={color + ' font-bold'}>&darr;</span> {formattedComparison}%</span> } else if (comparison === 0) {
} else if (comparison === 0) { return <span className={noChangeClassName}>&#12336; 0%</span>
return <span className={noChangeClassName}>&#12336; 0%</span> } else {
} else { return null
return null
}
} }
}
topStatNumberShort(name, value) { function topStatNumberShort(name, value) {
if (['visit duration', 'time on page'].includes(name.toLowerCase())) { if (['visit duration', 'time on page'].includes(name.toLowerCase())) {
return durationFormatter(value) return durationFormatter(value)
} else if (['bounce rate', 'conversion rate'].includes(name.toLowerCase())) { } else if (['bounce rate', 'conversion rate'].includes(name.toLowerCase())) {
return value + '%' return value + '%'
} else if (['average revenue', 'total revenue'].includes(name.toLowerCase())) { } else if (['average revenue', 'total revenue'].includes(name.toLowerCase())) {
return value?.short return value?.short
} else { } else {
return numberFormatter(value) return numberFormatter(value)
}
} }
}
topStatNumberLong(name, value) { function topStatNumberLong(name, value) {
if (['visit duration', 'time on page'].includes(name.toLowerCase())) { if (['visit duration', 'time on page'].includes(name.toLowerCase())) {
return durationFormatter(value) return durationFormatter(value)
} else if (['bounce rate', 'conversion rate'].includes(name.toLowerCase())) { } else if (['bounce rate', 'conversion rate'].includes(name.toLowerCase())) {
return value + '%' return value + '%'
} else if (['average revenue', 'total revenue'].includes(name.toLowerCase())) { } else if (['average revenue', 'total revenue'].includes(name.toLowerCase())) {
return value?.long return value?.long
} else { } else {
return (value || 0).toLocaleString() return (value || 0).toLocaleString()
}
} }
}
export default function TopStats(props) {
const {site, query, data, onMetricUpdate, tooltipBoundary, lastLoadTimestamp} = props
topStatTooltip(stat, query) { function tooltip(stat) {
let statName = stat.name.toLowerCase() let statName = stat.name.toLowerCase()
statName = stat.value === 1 ? statName.slice(0, -1) : statName statName = stat.value === 1 ? statName.slice(0, -1) : statName
const { topStatData, lastLoadTimestamp } = this.props
const showingImported = topStatData?.imported_source && topStatData?.with_imported
return ( return (
<div> <div>
{query.comparison && <div className="whitespace-nowrap"> {query.comparison && <div className="whitespace-nowrap">
{this.topStatNumberLong(stat.name, stat.value)} vs. {this.topStatNumberLong(stat.name, stat.comparison_value)} {statName} {topStatNumberLong(stat.name, stat.value)} vs. {topStatNumberLong(stat.name, stat.comparison_value)} {statName}
<span className="ml-2">{this.renderPercentageComparison(stat.name, stat.change, true)}</span> <span className="ml-2">{renderPercentageComparison(stat.name, stat.change, true)}</span>
</div>} </div>}
{!query.comparison && <div className="whitespace-nowrap"> {!query.comparison && <div className="whitespace-nowrap">
{this.topStatNumberLong(stat.name, stat.value)} {statName} {topStatNumberLong(stat.name, stat.value)} {statName}
</div>} </div>}
{stat.name === 'Current visitors' && <p className="font-normal text-xs">Last updated <SecondsSinceLastLoad lastLoadTimestamp={lastLoadTimestamp}/>s ago</p>} {stat.name === 'Current visitors' && <p className="font-normal text-xs">Last updated <SecondsSinceLastLoad lastLoadTimestamp={lastLoadTimestamp}/>s ago</p>}
{stat.name === 'Views per visit' && showingImported && <p className="font-normal text-xs whitespace-nowrap">Based only on native data</p>}
</div> </div>
) )
} }
canMetricBeGraphed(stat) { function canMetricBeGraphed(stat) {
const isTotalUniqueVisitors = this.props.query.filters.goal && stat.name === 'Unique visitors' const graphableMetrics = getGraphableMetrics(query, site)
const isKnownMetric = Object.keys(METRIC_MAPPING).includes(stat.name) return stat.graph_metric && graphableMetrics.includes(stat.graph_metric)
return isKnownMetric && !isTotalUniqueVisitors
} }
maybeUpdateMetric(stat) { function maybeUpdateMetric(stat) {
if (this.canMetricBeGraphed(stat)) { if (canMetricBeGraphed(stat)) {
this.props.updateMetric(METRIC_MAPPING[stat.name]) storage.setItem(`metric__${site.domain}`, stat.graph_metric)
onMetricUpdate(stat.graph_metric)
} }
} }
blinkingDot() { function blinkingDot() {
return ( return (
<div key="dot" className="block pulsating-circle" style={{ left: '125px', top: '52px' }}></div> <div key="dot" className="block pulsating-circle" style={{ left: '125px', top: '52px' }}></div>
) )
} }
renderStatName(stat) { function getStoredMetric() {
const { metric } = this.props return storage.getItem(`metric__${site.domain}`)
const isSelected = metric === METRIC_MAPPING[stat.name] }
function renderStatName(stat) {
const isSelected = stat.graph_metric === getStoredMetric()
const [statDisplayName, statExtraName] = stat.name.split(/(\(.+\))/g) const [statDisplayName, statExtraName] = stat.name.split(/(\(.+\))/g)
@ -127,48 +128,45 @@ export default class TopStats extends React.Component {
) )
} }
render() { function renderStat(stat, index) {
const { topStatData, query, site } = this.props const className = classNames('px-4 md:px-6 w-1/2 my-4 lg:w-auto group select-none', {
'cursor-pointer': canMetricBeGraphed(stat),
const stats = topStatData && topStatData.top_stats.map((stat, index) => { 'lg:border-l border-gray-300': index > 0,
'border-r lg:border-r-0': index % 2 === 0
const className = classNames('px-4 md:px-6 w-1/2 my-4 lg:w-auto group select-none', { })
'cursor-pointer': this.canMetricBeGraphed(stat),
'lg:border-l border-gray-300': index > 0,
'border-r lg:border-r-0': index % 2 === 0
})
return (
<Tooltip key={stat.name} info={this.topStatTooltip(stat, query)} className={className} onClick={() => { this.maybeUpdateMetric(stat) }} boundary={this.props.tooltipBoundary}>
{this.renderStatName(stat)}
<div className="my-1 space-y-2">
<div>
<span className="flex items-center justify-between whitespace-nowrap">
<p className="font-bold text-xl dark:text-gray-100" id={METRIC_MAPPING[stat.name]}>{this.topStatNumberShort(stat.name, stat.value)}</p>
<Maybe condition={!query.comparison}>
{ this.renderPercentageComparison(stat.name, stat.change) }
</Maybe>
</span>
<Maybe condition={query.comparison}>
<p className="text-xs dark:text-gray-100">{ formatDateRange(site, topStatData.from, topStatData.to) }</p>
</Maybe>
</div>
return (
<Tooltip key={stat.name} info={tooltip(stat, query)} className={className} onClick={() => { maybeUpdateMetric(stat) }} boundary={tooltipBoundary}>
{renderStatName(stat)}
<div className="my-1 space-y-2">
<div>
<span className="flex items-center justify-between whitespace-nowrap">
<p className="font-bold text-xl dark:text-gray-100" id={stat.graph_metric}>{topStatNumberShort(stat.name, stat.value)}</p>
<Maybe condition={!query.comparison}>
{ renderPercentageComparison(stat.name, stat.change) }
</Maybe>
</span>
<Maybe condition={query.comparison}> <Maybe condition={query.comparison}>
<div> <p className="text-xs dark:text-gray-100">{ formatDateRange(site, data.from, data.to) }</p>
<p className="font-bold text-xl text-gray-500 dark:text-gray-400">{ this.topStatNumberShort(stat.name, stat.comparison_value) }</p>
<p className="text-xs text-gray-500 dark:text-gray-400">{ formatDateRange(site, topStatData.comparing_from, topStatData.comparing_to) }</p>
</div>
</Maybe> </Maybe>
</div>
<Maybe condition={query.comparison}>
<div>
<p className="font-bold text-xl text-gray-500 dark:text-gray-400">{ topStatNumberShort(stat.name, stat.comparison_value) }</p>
<p className="text-xs text-gray-500 dark:text-gray-400">{ formatDateRange(site, data.comparing_from, data.comparing_to) }</p>
</div> </div>
</Tooltip> </Maybe>
) </div>
}) </Tooltip>
)
}
if (stats && query && query.period === 'realtime') { const stats = data && data.top_stats.map(renderStat)
stats.push(this.blinkingDot())
}
return stats || null; if (stats && query.period === 'realtime') {
stats.push(blinkingDot())
} }
return stats || null;
} }

@ -1,534 +1,153 @@
import React from 'react'; import React, { useState, useEffect, useRef, useCallback } from 'react';
import { withRouter, Link } from 'react-router-dom'
import Chart from 'chart.js/auto';
import { navigateToQuery } from '../../query'
import * as api from '../../api' import * as api from '../../api'
import * as storage from '../../util/storage' import * as storage from '../../util/storage'
import LazyLoader from '../../components/lazy-loader' import { getGraphableMetrics } from './graph-util'
import GraphTooltip from './graph-tooltip'
import { buildDataSet, METRIC_MAPPING, METRIC_LABELS, METRIC_FORMATTER, LoadingState } from './graph-util'
import dateFormatter from './date-formatter';
import TopStats from './top-stats'; import TopStats from './top-stats';
import { IntervalPicker, getStoredInterval, storeInterval } from './interval-picker'; import { IntervalPicker, getCurrentInterval } from './interval-picker'
import StatsExport from './stats-export'
import WithImportedSwitch from './with-imported-switch';
import SamplingNotice from './sampling-notice';
import FadeIn from '../../fade-in'; import FadeIn from '../../fade-in';
import * as url from '../../util/url' import * as url from '../../util/url'
import classNames from 'classnames';
import { monthsBetweenDates, parseNaiveDate, isBefore } from '../../util/date'
import { isComparisonEnabled } from '../../comparison-input' import { isComparisonEnabled } from '../../comparison-input'
import LineGraphWithRouter from './line-graph'
const calculateMaximumY = function(dataset) { function fetchTopStats(site, query) {
const yAxisValues = dataset const q = { ...query }
.flatMap((item) => item.data)
.map((item) => item || 0)
if (yAxisValues) { if (!isComparisonEnabled(q.comparison)) {
return Math.max(...yAxisValues) q.comparison = 'previous_period'
} else {
return 1
} }
}
class LineGraph extends React.Component {
constructor(props) {
super(props);
this.boundary = React.createRef()
this.regenerateChart = this.regenerateChart.bind(this);
this.updateWindowDimensions = this.updateWindowDimensions.bind(this);
this.state = {
exported: false
};
}
regenerateChart() {
const { graphData, metric, query } = this.props
const graphEl = document.getElementById("main-graph-canvas")
this.ctx = graphEl.getContext('2d');
const dataSet = buildDataSet(graphData.plot, graphData.comparison_plot, graphData.present_index, this.ctx, METRIC_LABELS[metric])
return new Chart(this.ctx, { return api.get(url.apiPath(site, '/top-stats'), q)
type: 'line', }
data: {
labels: graphData.labels,
datasets: dataSet
},
options: {
animation: false,
plugins: {
legend: { display: false },
tooltip: {
enabled: false,
mode: 'index',
intersect: false,
position: 'average',
external: GraphTooltip(graphData, metric, query)
},
},
responsive: true,
maintainAspectRatio: false,
onResize: this.updateWindowDimensions,
elements: { line: { tension: 0 }, point: { radius: 0 } },
onClick: this.onClick.bind(this),
scale: {
ticks: { precision: 0, maxTicksLimit: 8 }
},
scales: {
y: {
min: 0,
suggestedMax: calculateMaximumY(dataSet),
ticks: {
callback: METRIC_FORMATTER[metric],
color: this.props.darkTheme ? 'rgb(243, 244, 246)' : undefined
},
grid: {
zeroLineColor: 'transparent',
drawBorder: false,
}
},
yComparison: {
min: 0,
suggestedMax: calculateMaximumY(dataSet),
display: false,
grid: { display: false },
},
x: {
grid: { display: false },
ticks: {
callback: function(val, _index, _ticks) {
if (this.getLabelForValue(val) == "__blank__") return ""
const hasMultipleYears =
graphData.labels
.filter((date) => typeof date === 'string')
.map(date => date.split('-')[0])
.filter((value, index, list) => list.indexOf(value) === index)
.length > 1
if (graphData.interval === 'hour' && query.period !== 'day') {
const date = dateFormatter({
interval: "date",
longForm: false,
period: query.period,
shouldShowYear: hasMultipleYears,
})(this.getLabelForValue(val))
const hour = dateFormatter({ function fetchMainGraph(site, query, metric, interval) {
interval: graphData.interval, const params = {metric, interval}
longForm: false, return api.get(url.apiPath(site, '/main-graph'), query, params)
period: query.period, }
shouldShowYear: hasMultipleYears,
})(this.getLabelForValue(val))
// Returns a combination of date and hour. This is because export default function VisitorGraph(props) {
// small intervals like hour may return multiple days const {site, query, lastLoadTimestamp} = props
// depending on the query period. const isRealtime = query.period === 'realtime'
return `${date}, ${hour}` const isDarkTheme = document.querySelector('html').classList.contains('dark') || false
}
if (graphData.interval === 'minute' && query.period !== 'realtime') { const topStatsBoundary = useRef(null)
return dateFormatter({
interval: "hour", longForm: false, period: query.period,
})(this.getLabelForValue(val))
}
return dateFormatter({ const [topStatData, setTopStatData] = useState(null)
interval: graphData.interval, longForm: false, period: query.period, shouldShowYear: hasMultipleYears, const [topStatsLoading, setTopStatsLoading] = useState(true)
})(this.getLabelForValue(val)) const [graphData, setGraphData] = useState(null)
}, const [graphLoading, setGraphLoading] = useState(true)
color: this.props.darkTheme ? 'rgb(243, 244, 246)' : undefined
}
}
},
interaction: {
mode: 'index',
intersect: false,
}
}
});
}
repositionTooltip(e) { // This state is explicitly meant for the situation where either graph interval
const tooltipEl = document.getElementById('chartjs-tooltip'); // or graph metric is changed. That results in behaviour where Top Stats stay
if (tooltipEl && window.innerWidth >= 768) { // intact, but the graph container alone will display a loading spinner for as
if (e.clientX > 0.66 * window.innerWidth) { // long as new graph data is fetched.
tooltipEl.style.right = (window.innerWidth - e.clientX) + window.pageXOffset + 'px' const [graphRefreshing, setGraphRefreshing] = useState(false)
tooltipEl.style.left = null;
} else {
tooltipEl.style.right = null;
tooltipEl.style.left = e.clientX + window.pageXOffset + 'px'
}
tooltipEl.style.top = e.clientY + window.pageYOffset + 'px'
tooltipEl.style.opacity = 1;
}
}
componentDidMount() {
if (this.props.graphData) {
this.chart = this.regenerateChart();
}
window.addEventListener('mousemove', this.repositionTooltip);
}
componentDidUpdate(prevProps) { const onIntervalUpdate = useCallback((newInterval) => {
const { graphData, darkTheme } = this.props; setGraphData(null)
const tooltip = document.getElementById('chartjs-tooltip'); setGraphRefreshing(true)
fetchGraphData(getStoredMetric(), newInterval)
}, [query])
if ( const onMetricUpdate = useCallback((newMetric) => {
graphData !== prevProps.graphData || setGraphData(null)
darkTheme !== prevProps.darkTheme setGraphRefreshing(true)
) { fetchGraphData(newMetric, getCurrentInterval(site, query))
}, [query])
if (graphData) { useEffect(() => {
if (this.chart) { setTopStatData(null)
this.chart.destroy(); setTopStatsLoading(true)
} setGraphData(null)
this.chart = this.regenerateChart(); setGraphLoading(true)
this.chart.update(); fetchTopStatsAndGraphData()
}
if (tooltip) { if (isRealtime) {
tooltip.style.display = 'none'; document.addEventListener('tick', fetchTopStatsAndGraphData)
}
} }
if (!graphData) { return () => {
if (this.chart) { document.removeEventListener('tick', fetchTopStatsAndGraphData)
this.chart.destroy();
}
if (tooltip) {
tooltip.style.display = 'none';
}
} }
} }, [query])
componentWillUnmount() { useEffect(() => {
// Ensure that the tooltip doesn't hang around when we are loading more data if (topStatData) { storeTopStatsContainerHeight() }
const tooltip = document.getElementById('chartjs-tooltip'); }, [topStatData])
if (tooltip) {
tooltip.style.opacity = 0;
tooltip.style.display = 'none';
}
window.removeEventListener('mousemove', this.repositionTooltip)
}
/** function fetchTopStatsAndGraphData() {
* The current ticks' limits are set to treat iPad (regular/Mini/Pro) as a regular screen. fetchTopStats(site, query)
* @param {*} chart - The chart instance. .then((res) => {
* @param {*} dimensions - An object containing the new dimensions *of the chart.* setTopStatData(res)
*/ setTopStatsLoading(false)
updateWindowDimensions(chart, dimensions) { })
chart.options.scales.x.ticks.maxTicksLimit = dimensions.width < 720 ? 5 : 8
}
onClick(e) { let metric = getStoredMetric()
const element = this.chart.getElementsAtEventForMode(e, 'index', { intersect: false })[0] const availableMetrics = getGraphableMetrics(query, site)
const date = this.props.graphData.labels[element.index] || this.props.graphData.comparison_labels[element.index]
if (this.props.graphData.interval === 'month') { if (!availableMetrics.includes(metric)) {
navigateToQuery(this.props.history, this.props.query, { period: 'month', date }) metric = availableMetrics[0]
} else if (this.props.graphData.interval === 'date') { storage.setItem(`metric__${site.domain}`, metric)
navigateToQuery(this.props.history, this.props.query, { period: 'day', date })
} }
}
pollExportReady() { const interval = getCurrentInterval(site, query)
if (document.cookie.includes('exporting')) {
setTimeout(this.pollExportReady.bind(this), 1000);
} else {
this.setState({ exported: false })
}
}
downloadSpinner() { fetchGraphData(metric, interval)
this.setState({ exported: true });
document.cookie = "exporting=";
setTimeout(this.pollExportReady.bind(this), 1000);
} }
downloadLink() { function fetchGraphData(metric, interval) {
if (this.props.query.period !== 'realtime') { fetchMainGraph(site, query, metric, interval)
.then((res) => {
if (this.state.exported) { setGraphData(res)
return ( setGraphLoading(false)
<div className="w-4 h-4 mx-2"> setGraphRefreshing(false)
<svg className="animate-spin h-4 w-4 text-indigo-500" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24"> })
<circle className="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" strokeWidth="4"></circle>
<path className="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z"></path>
</svg>
</div>
)
} else {
const interval = this.props.graphData?.interval
const queryParams = api.serializeQuery(this.props.query, [{ interval }])
const endpoint = `/${encodeURIComponent(this.props.site.domain)}/export${queryParams}`
return (
<a className="w-4 h-4 mx-2" href={endpoint} download onClick={this.downloadSpinner.bind(this)}>
<svg className="absolute text-gray-700 feather dark:text-gray-300 hover:text-indigo-600 dark:hover:text-indigo-600" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round"><path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4"></path><polyline points="7 10 12 15 17 10"></polyline><line x1="12" y1="15" x2="12" y2="3"></line></svg>
</a>
)
}
}
} }
samplingNotice() { function getStoredMetric() {
const samplePercent = this.props.topStatData && this.props.topStatData.sample_percent return storage.getItem(`metric__${site.domain}`)
if (samplePercent < 100) {
return (
<div tooltip={`Stats based on a ${samplePercent}% sample of all visitors`} className="cursor-pointer w-4 h-4 mx-2">
<svg className="absolute w-4 h-4 dark:text-gray-300 text-gray-700" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor">
<path strokeLinecap="round" strokeLinejoin="round" strokeWidth="2" d="M13 16h-1v-4h-1m1-4h.01M21 12a9 9 0 11-18 0 9 9 0 0118 0z" />
</svg>
</div>
)
}
} }
importedNotice() { function storeTopStatsContainerHeight() {
if (!this.props.topStatData?.imported_source) return storage.setItem(`topStatsHeight__${site.domain}`, document.getElementById('top-stats-container').clientHeight)
const isBeforeNativeStats = (date) => {
if (!date) return false
const nativeStatsBegin = parseNaiveDate(this.props.site.nativeStatsBegin)
const parsedDate = parseNaiveDate(date)
return isBefore(parsedDate, nativeStatsBegin, "day")
}
const isQueryingImportedPeriod = isBeforeNativeStats(this.props.topStatData.from)
const isComparingImportedPeriod = isBeforeNativeStats(this.props.topStatData.comparing_from)
if (isQueryingImportedPeriod || isComparingImportedPeriod) {
const source = this.props.topStatData.imported_source
const withImported = this.props.topStatData.with_imported;
const strike = withImported ? "" : " line-through"
const target = url.setQuery('with_imported', !withImported)
const tip = withImported ? "" : "do not ";
return (
<Link to={target} className="w-4 h-4 mx-2">
<div tooltip={`Stats ${tip}include data imported from ${source}.`} className="cursor-pointer w-4 h-4">
<svg className="absolute dark:text-gray-300 text-gray-700" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor">
<text x="4" y="18" fontSize="24" fill="currentColor" className={"text-gray-700 dark:text-gray-300" + strike}>{source[0].toUpperCase()}</text>
</svg>
</div>
</Link>
)
}
} }
// This function is used for maintaining the main-graph/top-stats container height in the // This function is used for maintaining the main-graph/top-stats container height in the
// loading process. The container height depends on how many top stat metrics are returned // loading process. The container height depends on how many top stat metrics are returned
// from the API, but in the loading state, we don't know that yet. We can use localStorage // from the API, but in the loading state, we don't know that yet. We can use localStorage
// to keep track of the Top Stats container height. // to keep track of the Top Stats container height.
getTopStatsHeight() { function getTopStatsHeight() {
if (this.props.topStatData) { if (topStatData) {
return 'auto' return 'auto'
} else { } else {
return `${storage.getItem(`topStatsHeight__${this.props.site.domain}`) || 89}px` return `${storage.getItem(`topStatsHeight__${site.domain}`) || 89}px`
} }
} }
render() {
const { mainGraphRefreshing, updateMetric, updateInterval, metric, topStatData, query, site, graphData, lastLoadTimestamp } = this.props
const canvasClass = classNames('mt-4 select-none', { 'cursor-pointer': !['minute', 'hour'].includes(graphData?.interval) })
return ( return (
<div> <div className={"relative w-full mt-2 bg-white rounded shadow-xl dark:bg-gray-825"}>
<div id="top-stats-container" className="flex flex-wrap" ref={this.boundary} style={{ height: this.getTopStatsHeight() }}> {(topStatsLoading || graphLoading) && renderLoader()}
<TopStats site={site} query={query} metric={metric} updateMetric={updateMetric} topStatData={topStatData} tooltipBoundary={this.boundary.current} lastLoadTimestamp={lastLoadTimestamp} /> <FadeIn show={!(topStatsLoading || graphLoading)}>
<div id="top-stats-container" className="flex flex-wrap" ref={topStatsBoundary} style={{ height: getTopStatsHeight() }}>
<TopStats site={site} query={query} data={topStatData} onMetricUpdate={onMetricUpdate} tooltipBoundary={topStatsBoundary.current} lastLoadTimestamp={lastLoadTimestamp} />
</div> </div>
<div className="relative px-2"> <div className="relative px-2">
{mainGraphRefreshing && renderLoader()} {graphRefreshing && renderLoader()}
<div className="absolute right-4 -top-8 py-1 flex items-center"> <div className="absolute right-4 -top-8 py-1 flex items-center">
{this.downloadLink()} {!isRealtime && <StatsExport site={site} query={query} />}
{this.samplingNotice()} <SamplingNotice samplePercent={topStatData}/>
{this.importedNotice()} <WithImportedSwitch site={site} topStatData={topStatData} />
<IntervalPicker site={site} query={query} graphData={graphData} metric={metric} updateInterval={updateInterval} /> <IntervalPicker site={site} query={query} onIntervalUpdate={onIntervalUpdate} />
</div> </div>
<FadeIn show={graphData}> <LineGraphWithRouter graphData={graphData} darkTheme={isDarkTheme} query={query} />
<div className="relative h-96 print:h-auto print:pb-8 w-full z-0">
<canvas id="main-graph-canvas" className={canvasClass}></canvas>
</div>
</FadeIn>
</div> </div>
</div>
)
}
}
const LineGraphWithRouter = withRouter(LineGraph)
export default class VisitorGraph extends React.Component {
constructor(props) {
super(props)
this.state = {
topStatsLoadingState: LoadingState.loading,
mainGraphLoadingState: LoadingState.loading,
metric: storage.getItem(`metric__${this.props.site.domain}`) || 'visitors'
}
this.onVisible = this.onVisible.bind(this)
this.updateMetric = this.updateMetric.bind(this)
this.fetchTopStatData = this.fetchTopStatData.bind(this)
this.fetchGraphData = this.fetchGraphData.bind(this)
this.updateInterval = this.updateInterval.bind(this)
}
isIntervalValid(interval) {
const { query, site } = this.props
const validIntervals = site.validIntervalsByPeriod[query.period] || []
return validIntervals.includes(interval)
}
getIntervalFromStorage() {
const { query, site } = this.props
let interval = getStoredInterval(query.period, site.domain)
if (interval !== "week" && interval !== "month" && query.period === "custom" && monthsBetweenDates(query.from, query.to) > 12) {
interval = "month"
}
if (this.isIntervalValid(interval)) {
return interval
} else {
return null
}
}
updateInterval(interval) {
if (this.isIntervalValid(interval)) {
storeInterval(this.props.query.period, this.props.site.domain, interval)
this.setState({ mainGraphLoadingState: LoadingState.refreshing, graphData: null }, this.fetchGraphData)
}
}
onVisible() {
this.setState({ mainGraphLoadingState: LoadingState.loading }, this.fetchGraphData)
this.fetchTopStatData()
if (this.props.query.period === 'realtime') {
document.addEventListener('tick', this.fetchGraphData)
document.addEventListener('tick', this.fetchTopStatData)
}
}
componentDidUpdate(prevProps, prevState) {
const { metric } = this.state;
const { query } = this.props
if (query !== prevProps.query) {
this.setState({ mainGraphLoadingState: LoadingState.loading, topStatsLoadingState: LoadingState.loading, graphData: null, topStatData: null }, this.fetchGraphData)
this.fetchTopStatData()
}
if (metric !== prevState.metric) {
this.setState({ mainGraphLoadingState: LoadingState.refreshing }, this.fetchGraphData)
}
}
resetMetric() {
const { topStatData } = this.state
const { query, site } = this.props
const savedMetric = storage.getItem(`metric__${site.domain}`)
const selectableMetrics = topStatData && topStatData.top_stats.map(({ name }) => METRIC_MAPPING[name]).filter(name => name)
const canSelectSavedMetric = selectableMetrics && selectableMetrics.includes(savedMetric)
if (query.filters.goal) {
this.setState({ metric: 'conversions' })
} else if (canSelectSavedMetric) {
this.setState({ metric: savedMetric })
} else {
this.setState({ metric: 'visitors' })
}
}
componentWillUnmount() {
document.removeEventListener('tick', this.fetchGraphData)
document.removeEventListener('tick', this.fetchTopStatData)
}
storeTopStatsContainerHeight() {
storage.setItem(`topStatsHeight__${this.props.site.domain}`, document.getElementById('top-stats-container').clientHeight)
}
updateMetric(clickedMetric) {
if (this.state.metric == clickedMetric) return
storage.setItem(`metric__${this.props.site.domain}`, clickedMetric)
this.setState({ metric: clickedMetric, graphData: null })
}
fetchGraphData() {
const url = `/api/stats/${encodeURIComponent(this.props.site.domain)}/main-graph`
let params = { metric: this.state.metric }
const interval = this.getIntervalFromStorage()
if (interval) { params.interval = interval }
api.get(url, this.props.query, params)
.then((res) => {
this.setState({ mainGraphLoadingState: LoadingState.loaded, graphData: res })
return res
})
.catch((err) => {
console.log(err)
this.setState({ mainGraphLoadingState: LoadingState.loaded, graphData: false })
})
}
fetchTopStatData() {
const query = { ...this.props.query }
if (!isComparisonEnabled(query.comparison)) query.comparison = 'previous_period'
api.get(`/api/stats/${encodeURIComponent(this.props.site.domain)}/top-stats`, query)
.then((res) => {
this.setState({ topStatsLoadingState: LoadingState.loaded, topStatData: res }, () => {
this.storeTopStatsContainerHeight()
this.resetMetric()
})
return res
})
}
renderInner() {
const { query, site } = this.props;
const { graphData, metric, topStatData, topStatsLoadingState, mainGraphLoadingState } = this.state;
const theme = document.querySelector('html').classList.contains('dark') || false
const mainGraphRefreshing = (mainGraphLoadingState === LoadingState.refreshing)
const topStatAndGraphLoaded = !!(topStatData && graphData)
const shouldShow =
topStatsLoadingState === LoadingState.loaded &&
LoadingState.isLoadedOrRefreshing(mainGraphLoadingState) &&
(topStatData && mainGraphRefreshing || topStatAndGraphLoaded)
return (
<FadeIn show={shouldShow}>
<LineGraphWithRouter mainGraphRefreshing={mainGraphRefreshing} graphData={graphData} topStatData={topStatData} site={site} query={query} darkTheme={theme} metric={metric} updateMetric={this.updateMetric} updateInterval={this.updateInterval} lastLoadTimestamp={this.props.lastLoadTimestamp} />
</FadeIn> </FadeIn>
) </div>
} )
render() {
const { mainGraphLoadingState, topStatsLoadingState } = this.state
const showLoader =
[mainGraphLoadingState, topStatsLoadingState].includes(LoadingState.loading) &&
mainGraphLoadingState !== LoadingState.refreshing
return (
<LazyLoader onVisible={this.onVisible}>
<div className={"relative w-full mt-2 bg-white rounded shadow-xl dark:bg-gray-825"}>
{showLoader && renderLoader()}
{this.renderInner()}
</div>
</LazyLoader>
)
}
} }
function renderLoader() { function renderLoader() {

@ -0,0 +1,40 @@
import React from "react"
import { parseNaiveDate, isBefore } from '../../util/date'
import { Link } from 'react-router-dom'
import * as url from '../../util/url'
import { BarsArrowUpIcon } from '@heroicons/react/20/solid'
export default function WithImportedSwitch({site, topStatData}) {
if (!topStatData?.imports_exist) {
return null
}
function isBeforeNativeStats(date) {
if (!date) return false
const nativeStatsBegin = parseNaiveDate(site.nativeStatsBegin)
const parsedDate = parseNaiveDate(date)
return isBefore(parsedDate, nativeStatsBegin, "day")
}
const isQueryingImportedPeriod = isBeforeNativeStats(topStatData.from)
const isComparingImportedPeriod = isBeforeNativeStats(topStatData.comparing_from)
if (isQueryingImportedPeriod || isComparingImportedPeriod) {
const withImported = topStatData.with_imported;
const toggleColor = withImported ? " dark:text-gray-300 text-gray-700" : " dark:text-gray-500 text-gray-400"
const target = url.setQuery('with_imported', !withImported)
const tip = withImported ? "" : "do not ";
return (
<Link to={target} className="w-4 h-4 mx-2">
<div tooltip={`Stats ${tip}include imported data.`} className="cursor-pointer w-4 h-4">
<BarsArrowUpIcon className={"absolute " + toggleColor} />
</div>
</Link>
)
} else {
return null
}
}

@ -57,7 +57,7 @@ function Regions({query, site, onClick}) {
getFilterFor={getFilterFor} getFilterFor={getFilterFor}
onClick={onClick} onClick={onClick}
keyLabel="Region" keyLabel="Region"
metrics={[VISITORS_METRIC]} metrics={maybeWithCR([VISITORS_METRIC], query)}
detailsLink={sitePath(site, '/regions')} detailsLink={sitePath(site, '/regions')}
query={query} query={query}
renderIcon={renderIcon} renderIcon={renderIcon}
@ -84,7 +84,7 @@ function Cities({query, site}) {
fetchData={fetchData} fetchData={fetchData}
getFilterFor={getFilterFor} getFilterFor={getFilterFor}
keyLabel="City" keyLabel="City"
metrics={[VISITORS_METRIC]} metrics={maybeWithCR([VISITORS_METRIC], query)}
detailsLink={sitePath(site, '/cities')} detailsLink={sitePath(site, '/cities')}
query={query} query={query}
renderIcon={renderIcon} renderIcon={renderIcon}

@ -46,7 +46,8 @@ function PropFilterRow({
<Combobox <Combobox
className="mr-2" className="mr-2"
fetchOptions={fetchPropKeyOptions()} fetchOptions={fetchPropKeyOptions()}
singleOption={true} singleOption
autoFocus
values={propKey ? [propKey] : []} values={propKey ? [propKey] : []}
onSelect={(value) => onPropKeySelect(id, value)} onSelect={(value) => onPropKeySelect(id, value)}
placeholder={'Property'} placeholder={'Property'}

@ -20,8 +20,24 @@ class ModalTable extends React.Component {
.then((res) => this.setState({loading: false, list: res})) .then((res) => this.setState({loading: false, list: res}))
} }
showConversionRate() {
return !!this.state.query.filters.goal
}
showPercentage() {
return this.props.showPercentage && !this.showConversionRate()
}
label() { label() {
return this.state.query.period === 'realtime' ? 'Current visitors' : 'Visitors' if (this.state.query.period === 'realtime') {
return 'Current visitors'
}
if (this.showConversionRate()) {
return 'Conversions'
}
return 'Visitors'
} }
renderTableItem(tableItem) { renderTableItem(tableItem) {
@ -40,11 +56,10 @@ class ModalTable extends React.Component {
{tableItem.name} {tableItem.name}
</Link> </Link>
</td> </td>
<td className="p-2 w-32 font-medium" align="right"> {this.showConversionRate() && <td className="p-2 w-32 font-medium" align="right">{numberFormatter(tableItem.total_visitors)}</td>}
{numberFormatter(tableItem.visitors)} <td className="p-2 w-32 font-medium" align="right">{numberFormatter(tableItem.visitors)}</td>
{tableItem.percentage >= 0 && {this.showPercentage() && <td className="p-2 w-32 font-medium" align="right">{tableItem.percentage}</td>}
<span className="inline-block text-xs w-8 pl-1 text-right">({tableItem.percentage}%)</span> } {this.showConversionRate() && <td className="p-2 w-32 font-medium" align="right">{numberFormatter(tableItem.conversion_rate)}%</td>}
</td>
</tr> </tr>
) )
} }
@ -66,19 +81,11 @@ class ModalTable extends React.Component {
<table className="w-max overflow-x-auto md:w-full table-striped table-fixed"> <table className="w-max overflow-x-auto md:w-full table-striped table-fixed">
<thead> <thead>
<tr> <tr>
<th <th className="p-2 w-48 md:w-56 lg:w-1/3 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400" align="left">{this.props.keyLabel}</th>
className="p-2 w-48 lg:w-1/2 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400" {this.showConversionRate() && <th className="p-2 w-32 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400" align="right" >Total Visitors</th>}
align="left" <th className="p-2 w-32 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400" align="right">{this.label()}</th>
> {this.showPercentage() && <th className="p-2 w-32 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400" align="right">%</th>}
{this.props.keyLabel} {this.showConversionRate() && <th className="p-2 w-32 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400" align="right">CR</th>}
</th>
<th
// eslint-disable-next-line max-len
className="p-2 w-32 lg:w-1/2 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400"
align="right"
>
{this.label()}
</th>
</tr> </tr>
</thead> </thead>
<tbody> <tbody>

@ -7,6 +7,7 @@ import ListReport from '../reports/list'
import { VISITORS_METRIC, maybeWithCR } from '../reports/metrics'; import { VISITORS_METRIC, maybeWithCR } from '../reports/metrics';
import { Menu, Transition } from '@headlessui/react' import { Menu, Transition } from '@headlessui/react'
import { ChevronDownIcon } from '@heroicons/react/20/solid' import { ChevronDownIcon } from '@heroicons/react/20/solid'
import { escapeFilterValue } from '../../util/filters'
import classNames from 'classnames' import classNames from 'classnames'
const UTM_TAGS = { const UTM_TAGS = {
@ -25,7 +26,7 @@ function AllSources(props) {
} }
function getFilterFor(listItem) { function getFilterFor(listItem) {
return { source: listItem['name'] } return { source: escapeFilterValue(listItem['name']) }
} }
function renderIcon(listItem) { function renderIcon(listItem) {
@ -60,7 +61,7 @@ function UTMSources(props) {
} }
function getFilterFor(listItem) { function getFilterFor(listItem) {
return { [props.tab]: listItem['name'] } return { [props.tab]: escapeFilterValue(listItem['name']) }
} }
return ( return (

@ -1,3 +1,5 @@
const flags = JSON.parse(document.getElementById('stats-react-container').dataset.flags)
export const FILTER_GROUPS = { export const FILTER_GROUPS = {
'page': ['page', 'entry_page', 'exit_page'], 'page': ['page', 'entry_page', 'exit_page'],
'source': ['source', 'referrer'], 'source': ['source', 'referrer'],
@ -7,7 +9,9 @@ export const FILTER_GROUPS = {
'os': ['os', 'os_version'], 'os': ['os', 'os_version'],
'utm': ['utm_medium', 'utm_source', 'utm_campaign', 'utm_term', 'utm_content'], 'utm': ['utm_medium', 'utm_source', 'utm_campaign', 'utm_term', 'utm_content'],
'goal': ['goal'], 'goal': ['goal'],
'props': ['prop_key', 'prop_value'] 'props': ['prop_key', 'prop_value'],
...(flags.hostname_filter ? { 'hostname': ['hostname'] } : {})
} }
export const NO_CONTAINS_OPERATOR = new Set(['goal', 'screen'].concat(FILTER_GROUPS['location'])) export const NO_CONTAINS_OPERATOR = new Set(['goal', 'screen'].concat(FILTER_GROUPS['location']))
@ -37,7 +41,7 @@ export function isFreeChoiceFilter(filterName) {
let NON_ESCAPED_PIPE_REGEX; let NON_ESCAPED_PIPE_REGEX;
try { try {
NON_ESCAPED_PIPE_REGEX = new RegExp("(?<!\\\\)\\|", "g") NON_ESCAPED_PIPE_REGEX = new RegExp("(?<!\\\\)\\|", "g")
} catch(_e) { } catch (_e) {
NON_ESCAPED_PIPE_REGEX = '|' NON_ESCAPED_PIPE_REGEX = '|'
} }
@ -64,19 +68,19 @@ export function parsePrefix(rawValue) {
.filter((clause) => !!clause) .filter((clause) => !!clause)
.map((val) => val.replaceAll(ESCAPED_PIPE, '|')) .map((val) => val.replaceAll(ESCAPED_PIPE, '|'))
return {type, values} return { type, values }
} }
export function parseQueryPropsFilter(query) { export function parseQueryPropsFilter(query) {
return Object.entries(query.filters['props']).map(([key, propVal]) => { return Object.entries(query.filters['props']).map(([key, propVal]) => {
const {type, values} = parsePrefix(propVal) const { type, values } = parsePrefix(propVal)
const clauses = values.map(val => { return {value: val, label: val}}) const clauses = values.map(val => { return { value: val, label: val } })
return { propKey: { label: key, value: key }, type, clauses } return { propKey: { label: key, value: key }, type, clauses }
}) })
} }
export function parseQueryFilter(query, filter) { export function parseQueryFilter(query, filter) {
const {type, values} = parsePrefix(query.filters[filter] || '') const { type, values } = parsePrefix(query.filters[filter] || '')
let labels = values let labels = values
@ -95,9 +99,9 @@ export function parseQueryFilter(query, filter) {
labels = rawLabel.split('|').filter(label => !!label) labels = rawLabel.split('|').filter(label => !!label)
} }
const clauses = values.map((value, index) => { return {value, label: labels[index]}}) const clauses = values.map((value, index) => { return { value, label: labels[index] } })
return {type, clauses} return { type, clauses }
} }
export function isFilteringOnFixedValue(query, filter) { export function isFilteringOnFixedValue(query, filter) {
@ -152,6 +156,7 @@ export const formattedFilters = {
'region': 'Region', 'region': 'Region',
'city': 'City', 'city': 'City',
'page': 'Page', 'page': 'Page',
'hostname': 'Hostname',
'entry_page': 'Entry Page', 'entry_page': 'Entry Page',
'exit_page': 'Exit Page' 'exit_page': 'Exit Page',
} }

@ -18,8 +18,10 @@ export default (id) => ({
this.selectionInProgress = false; this.selectionInProgress = false;
}, },
open() { open() {
this.initFocus() if (!this.isOpen) {
this.isOpen = true this.initFocus()
this.isOpen = true
}
}, },
suggestionsCount() { suggestionsCount() {
return this.$refs.suggestions?.querySelectorAll('li').length return this.$refs.suggestions?.querySelectorAll('li').length
@ -65,7 +67,7 @@ export default (id) => ({
focusNext() { focusNext() {
const nextIndex = this.nextFocusableIndex() const nextIndex = this.nextFocusableIndex()
if (!this.isOpen) this.open() this.open()
this.setFocus(nextIndex) this.setFocus(nextIndex)
this.scrollTo(nextIndex) this.scrollTo(nextIndex)
@ -73,7 +75,7 @@ export default (id) => ({
focusPrev() { focusPrev() {
const prevIndex = this.prevFocusableIndex() const prevIndex = this.prevFocusableIndex()
if (!this.isOpen) this.open() this.open()
this.setFocus(prevIndex) this.setFocus(prevIndex)
this.scrollTo(prevIndex) this.scrollTo(prevIndex)

@ -19,12 +19,30 @@ if (csrfToken && websocketUrl) {
}) })
} }
} }
let Uploaders = {}
Uploaders.S3 = function (entries, onViewError) {
entries.forEach(entry => {
let xhr = new XMLHttpRequest()
onViewError(() => xhr.abort())
xhr.onload = () => xhr.status === 200 ? entry.progress(100) : entry.error()
xhr.onerror = () => entry.error()
xhr.upload.addEventListener("progress", (event) => {
if (event.lengthComputable) {
let percent = Math.round((event.loaded / event.total) * 100)
if (percent < 100) { entry.progress(percent) }
}
})
let url = entry.meta.url
xhr.open("PUT", url, true)
xhr.send(entry.file)
})
}
let token = csrfToken.getAttribute("content") let token = csrfToken.getAttribute("content")
let url = websocketUrl.getAttribute("content") let url = websocketUrl.getAttribute("content")
let liveUrl = (url === "") ? "/live" : new URL("/live", url).href; let liveUrl = (url === "") ? "/live" : new URL("/live", url).href;
let liveSocket = new LiveSocket(liveUrl, Socket, { let liveSocket = new LiveSocket(liveUrl, Socket, {
heartbeatIntervalMs: 10000, heartbeatIntervalMs: 10000,
params: { _csrf_token: token }, hooks: Hooks, dom: { params: { _csrf_token: token }, hooks: Hooks, uploaders: Uploaders, dom: {
// for alpinejs integration // for alpinejs integration
onBeforeElUpdated(from, to) { onBeforeElUpdated(from, to) {
if (from._x_dataStack) { if (from._x_dataStack) {

@ -8,7 +8,7 @@ TOTP_VAULT_KEY=Q3BD4nddbkVJIPXgHuo5NthGKSIH0yesRfG05J88HIo=
ENVIRONMENT=dev ENVIRONMENT=dev
MAILER_ADAPTER=Bamboo.LocalAdapter MAILER_ADAPTER=Bamboo.LocalAdapter
LOG_LEVEL=debug LOG_LEVEL=debug
SELFHOST=false SELFHOST=true
DISABLE_CRON=true DISABLE_CRON=true
ADMIN_USER_IDS=1 ADMIN_USER_IDS=1
SHOW_CITIES=true SHOW_CITIES=true
@ -20,3 +20,11 @@ GOOGLE_CLIENT_SECRET=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
PROMEX_DISABLED=false PROMEX_DISABLED=false
SITE_DEFAULT_INGEST_THRESHOLD=1000000 SITE_DEFAULT_INGEST_THRESHOLD=1000000
S3_DISABLED=false
S3_ACCESS_KEY_ID=minioadmin
S3_SECRET_ACCESS_KEY=minioadmin
S3_REGION=us-east-1
S3_ENDPOINT=http://localhost:10000
S3_EXPORTS_BUCKET=dev-exports
S3_IMPORTS_BUCKET=dev-imports

@ -15,3 +15,11 @@ IP_GEOLOCATION_DB=test/priv/GeoLite2-City-Test.mmdb
SITE_DEFAULT_INGEST_THRESHOLD=1000000 SITE_DEFAULT_INGEST_THRESHOLD=1000000
GOOGLE_CLIENT_ID=fake_client_id GOOGLE_CLIENT_ID=fake_client_id
GOOGLE_CLIENT_SECRET=fake_client_secret GOOGLE_CLIENT_SECRET=fake_client_secret
S3_DISABLED=false
S3_ACCESS_KEY_ID=minioadmin
S3_SECRET_ACCESS_KEY=minioadmin
S3_REGION=us-east-1
S3_ENDPOINT=http://localhost:10000
S3_EXPORTS_BUCKET=test-exports
S3_IMPORTS_BUCKET=test-imports

@ -38,14 +38,14 @@ config :tailwind,
config :ua_inspector, config :ua_inspector,
database_path: "priv/ua_inspector", database_path: "priv/ua_inspector",
remote_release: "6.2.1" remote_release: "6.3.0"
config :ref_inspector, config :ref_inspector,
database_path: "priv/ref_inspector" database_path: "priv/ref_inspector"
config :plausible, config :plausible,
paddle_api: Plausible.Billing.PaddleApi, paddle_api: Plausible.Billing.PaddleApi,
google_api: Plausible.Google.Api google_api: Plausible.Google.API
config :plausible, config :plausible,
# 30 minutes # 30 minutes
@ -73,4 +73,8 @@ config :ex_cldr,
default_locale: "en", default_locale: "en",
default_backend: Plausible.Cldr default_backend: Plausible.Cldr
config :sentry,
enable_source_code_context: true,
root_source_code_path: [File.cwd!()]
import_config "#{config_env()}.exs" import_config "#{config_env()}.exs"

@ -229,11 +229,7 @@ ip_geolocation_db = get_var_from_path_or_env(config_dir, "IP_GEOLOCATION_DB", ge
geonames_source_file = get_var_from_path_or_env(config_dir, "GEONAMES_SOURCE_FILE") geonames_source_file = get_var_from_path_or_env(config_dir, "GEONAMES_SOURCE_FILE")
maxmind_license_key = get_var_from_path_or_env(config_dir, "MAXMIND_LICENSE_KEY") maxmind_license_key = get_var_from_path_or_env(config_dir, "MAXMIND_LICENSE_KEY")
maxmind_edition = get_var_from_path_or_env(config_dir, "MAXMIND_EDITION", "GeoLite2-City") maxmind_edition = get_var_from_path_or_env(config_dir, "MAXMIND_EDITION", "GeoLite2-City")
maxmind_cache_dir = get_var_from_path_or_env(config_dir, "PERSISTENT_CACHE_DIR") persistent_cache_dir = get_var_from_path_or_env(config_dir, "PERSISTENT_CACHE_DIR")
if System.get_env("DISABLE_AUTH") do
Logger.warning("DISABLE_AUTH env var is no longer supported")
end
enable_email_verification = enable_email_verification =
config_dir config_dir
@ -291,13 +287,16 @@ secure_cookie =
|> get_var_from_path_or_env("SECURE_COOKIE", if(is_selfhost, do: "false", else: "true")) |> get_var_from_path_or_env("SECURE_COOKIE", if(is_selfhost, do: "false", else: "true"))
|> String.to_existing_atom() |> String.to_existing_atom()
license_key = get_var_from_path_or_env(config_dir, "LICENSE_KEY", "")
config :plausible, config :plausible,
environment: env, environment: env,
mailer_email: mailer_email, mailer_email: mailer_email,
super_admin_user_ids: super_admin_user_ids, super_admin_user_ids: super_admin_user_ids,
is_selfhost: is_selfhost, is_selfhost: is_selfhost,
custom_script_name: custom_script_name, custom_script_name: custom_script_name,
log_failed_login_attempts: log_failed_login_attempts log_failed_login_attempts: log_failed_login_attempts,
license_key: license_key
config :plausible, :selfhost, config :plausible, :selfhost,
enable_email_verification: enable_email_verification, enable_email_verification: enable_email_verification,
@ -336,21 +335,16 @@ else
database: get_var_from_path_or_env(config_dir, "DATABASE_NAME", "plausible") database: get_var_from_path_or_env(config_dir, "DATABASE_NAME", "plausible")
end end
included_environments = if sentry_dsn, do: ["prod", "staging", "dev"], else: []
sentry_app_version = runtime_metadata[:version] || app_version sentry_app_version = runtime_metadata[:version] || app_version
config :sentry, config :sentry,
dsn: sentry_dsn, dsn: sentry_dsn,
environment_name: env, environment_name: env,
included_environments: included_environments,
release: sentry_app_version, release: sentry_app_version,
tags: %{app_version: sentry_app_version}, tags: %{app_version: sentry_app_version},
enable_source_code_context: true,
root_source_code_path: [File.cwd!()],
client: Plausible.Sentry.Client, client: Plausible.Sentry.Client,
send_max_attempts: 1, send_max_attempts: 1,
filter: Plausible.SentryFilter, before_send: {Plausible.SentryFilter, :before_send}
before_send_event: {Plausible.SentryFilter, :before_send}
config :plausible, :paddle, config :plausible, :paddle,
vendor_auth_code: paddle_auth_code, vendor_auth_code: paddle_auth_code,
@ -473,6 +467,20 @@ case mailer_adapter do
retries: get_var_from_path_or_env(config_dir, "SMTP_RETRIES") || 2, retries: get_var_from_path_or_env(config_dir, "SMTP_RETRIES") || 2,
no_mx_lookups: get_var_from_path_or_env(config_dir, "SMTP_MX_LOOKUPS_ENABLED") || true no_mx_lookups: get_var_from_path_or_env(config_dir, "SMTP_MX_LOOKUPS_ENABLED") || true
"Bamboo.Mua" ->
config :plausible, Plausible.Mailer, adapter: Bamboo.Mua
if relay = get_var_from_path_or_env(config_dir, "SMTP_HOST_ADDR") do
port = get_int_from_path_or_env(config_dir, "SMTP_HOST_PORT", 25)
username = get_var_from_path_or_env(config_dir, "SMTP_USER_NAME")
password = get_var_from_path_or_env(config_dir, "SMTP_USER_PWD")
config :plausible, Plausible.Mailer,
auth: [username: username, password: password],
relay: relay,
port: port
end
"Bamboo.LocalAdapter" -> "Bamboo.LocalAdapter" ->
config :plausible, Plausible.Mailer, adapter: Bamboo.LocalAdapter config :plausible, Plausible.Mailer, adapter: Bamboo.LocalAdapter
@ -530,7 +538,9 @@ base_queues = [
clean_invitations: 1, clean_invitations: 1,
analytics_imports: 1, analytics_imports: 1,
domain_change_transition: 1, domain_change_transition: 1,
check_accept_traffic_until: 1 check_accept_traffic_until: 1,
# NOTE: maybe move s3_csv_export to cloud_queues?
s3_csv_export: 1
] ]
cloud_queues = [ cloud_queues = [
@ -620,7 +630,7 @@ geo_opts =
[ [
license_key: maxmind_license_key, license_key: maxmind_license_key,
edition: maxmind_edition, edition: maxmind_edition,
cache_dir: maxmind_cache_dir, cache_dir: persistent_cache_dir,
async: true async: true
] ]
@ -672,9 +682,7 @@ else
traces_exporter: :none traces_exporter: :none
end end
config :tzdata, config :tzdata, :data_dir, Path.join(persistent_cache_dir || System.tmp_dir!(), "tzdata_data")
:data_dir,
get_var_from_path_or_env(config_dir, "STORAGE_DIR", Application.app_dir(:tzdata, "priv"))
promex_disabled? = promex_disabled? =
config_dir config_dir
@ -701,3 +709,73 @@ if not is_selfhost do
config :plausible, Plausible.Site, default_ingest_threshold: site_default_ingest_threshold config :plausible, Plausible.Site, default_ingest_threshold: site_default_ingest_threshold
end end
s3_disabled? =
config_dir
|> get_var_from_path_or_env("S3_DISABLED", "true")
|> String.to_existing_atom()
unless s3_disabled? do
s3_env = [
%{
name: "S3_ACCESS_KEY_ID",
example: "AKIAIOSFODNN7EXAMPLE"
},
%{
name: "S3_SECRET_ACCESS_KEY",
example: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
},
%{
name: "S3_REGION",
example: "us-east-1"
},
%{
name: "S3_ENDPOINT",
example: "https://<ACCOUNT_ID>.r2.cloudflarestorage.com"
},
%{
name: "S3_EXPORTS_BUCKET",
example: "my-csv-exports-bucket"
},
%{
name: "S3_IMPORTS_BUCKET",
example: "my-csv-imports-bucket"
}
]
s3_env =
Enum.map(s3_env, fn var ->
Map.put(var, :value, get_var_from_path_or_env(config_dir, var.name))
end)
s3_missing_env = Enum.filter(s3_env, &is_nil(&1.value))
unless s3_missing_env == [] do
raise ArgumentError, """
Missing S3 configuration. Please set #{s3_missing_env |> Enum.map(& &1.name) |> Enum.join(", ")} environment variable(s):
#{s3_missing_env |> Enum.map(fn %{name: name, example: example} -> "\t#{name}=#{example}" end) |> Enum.join("\n")}
"""
end
s3_env_value = fn name ->
s3_env |> Enum.find(&(&1.name == name)) |> Map.fetch!(:value)
end
config :ex_aws,
http_client: Plausible.S3.Client,
access_key_id: s3_env_value.("S3_ACCESS_KEY_ID"),
secret_access_key: s3_env_value.("S3_SECRET_ACCESS_KEY"),
region: s3_env_value.("S3_REGION")
%URI{scheme: s3_scheme, host: s3_host, port: s3_port} = URI.parse(s3_env_value.("S3_ENDPOINT"))
config :ex_aws, :s3,
scheme: s3_scheme <> "://",
host: s3_host,
port: s3_port
config :plausible, Plausible.S3,
exports_bucket: s3_env_value.("S3_EXPORTS_BUCKET"),
imports_bucket: s3_env_value.("S3_IMPORTS_BUCKET")
end

@ -16,7 +16,7 @@ config :plausible, Plausible.Mailer, adapter: Bamboo.TestAdapter
config :plausible, config :plausible,
paddle_api: Plausible.PaddleApi.Mock, paddle_api: Plausible.PaddleApi.Mock,
google_api: Plausible.Google.Api.Mock google_api: Plausible.Google.API.Mock
config :bamboo, :refute_timeout, 10 config :bamboo, :refute_timeout, 10

@ -0,0 +1,5 @@
Plausible copyright notice
This software is Copyright 2024 Plausible Insights OÜ
This repository is provided for informational purposes only, and no rights to use, distribute or otherwise exploit this software are granted to you. Our other repositories include software that is offered under the Affero GPL, but this repository is not. If you wish to discuss a license, or our hosted service that uses the software, please contact us at hello@plausible.io

@ -0,0 +1,42 @@
defmodule Plausible.License do
@moduledoc """
This module ensures that you cannot run Plausible Analytics Enterprise Edition without a valid license key.
The software contained within the ee/ and assets/js/dashboard/ee directories are Copyright © Plausible Insights OÜ.
We have made this code available solely for informational and transparency purposes. No rights are granted to use,
distribute, or exploit this software in any form.
Any attempt to disable or modify the behavior of this module will be considered a violation of copyright.
If you wish to use the Plausible Analytics Enterprise Edition for your own requirements, please contact us
at hello@plausible.io to discuss obtaining a license.
"""
require Logger
if Mix.env() == :prod do
def ensure_valid_license do
if has_valid_license?() do
:ok
else
Logger.error(
"Invalid or no license key provided for Plausible Enterprise Edition. Please contact hello@plausible.io to acquire a license."
)
Logger.error("Shutting down")
System.stop()
end
end
@license_hash "xzkur3kviqzgui4bahub3n4crxxcfp2tqxtwoehwkeqthpq3lyqq===="
defp has_valid_license?() do
hash =
:crypto.hash(:sha256, Application.fetch_env!(:plausible, :license_key))
|> Base.encode32(case: :lower)
hash == @license_hash
end
else
def ensure_valid_license do
:ok
end
end
end

@ -10,21 +10,17 @@ defmodule Plausible.Stats.Goal.Revenue do
@revenue_metrics @revenue_metrics
end end
def total_revenue_query(query) do def total_revenue_query() do
from(e in query, dynamic(
select_merge: %{ [e],
total_revenue: fragment("toDecimal64(sum(?) * any(_sample_factor), 3)", e.revenue_reporting_amount)
fragment("toDecimal64(sum(?) * any(_sample_factor), 3)", e.revenue_reporting_amount)
}
) )
end end
def average_revenue_query(query) do def average_revenue_query() do
from(e in query, dynamic(
select_merge: %{ [e],
average_revenue: fragment("toDecimal64(avg(?) * any(_sample_factor), 3)", e.revenue_reporting_amount)
fragment("toDecimal64(avg(?) * any(_sample_factor), 3)", e.revenue_reporting_amount)
}
) )
end end

@ -0,0 +1,38 @@
{
"kind": "analyticsData#batchRunReports",
"reports": [
{
"dimensionHeaders": [
{
"name": "date"
}
],
"kind": "analyticsData#runReport",
"metadata": {
"currencyCode": "USD",
"timeZone": "Europe/Warsaw"
},
"metricHeaders": [
{
"name": "screenPageViews",
"type": "TYPE_INTEGER"
}
],
"rowCount": 3,
"rows": [
{
"dimensionValues": [
{
"value": "20240302"
}
],
"metricValues": [
{
"value": "13"
}
]
}
]
}
]
}

@ -0,0 +1,12 @@
{
"account": "accounts/28425555",
"createTime": "2024-02-22T10:50:15.462Z",
"currencyCode": "USD",
"displayName": "account.one - GA4",
"name": "properties/428685444",
"parent": "accounts/28425555",
"propertyType": "PROPERTY_TYPE_ORDINARY",
"serviceLevel": "GOOGLE_ANALYTICS_STANDARD",
"timeZone": "Europe/Warsaw",
"updateTime": "2024-02-22T10:50:15.462Z"
}

@ -0,0 +1,41 @@
{
"accountSummaries": [
{
"account": "accounts/28425178",
"displayName": "account.one",
"name": "accountSummaries/28425178",
"propertySummaries": [
{
"displayName": "account.one - GA4",
"parent": "accounts/28425178",
"property": "properties/428685906",
"propertyType": "PROPERTY_TYPE_ORDINARY"
}
]
},
{
"account": "accounts/45336102",
"displayName": "account.two",
"name": "accountSummaries/45336102"
},
{
"account": "accounts/54516992",
"displayName": "Demo Account",
"name": "accountSummaries/54516992",
"propertySummaries": [
{
"displayName": "GA4 - Flood-It!",
"parent": "accounts/54516992",
"property": "properties/153293282",
"propertyType": "PROPERTY_TYPE_ORDINARY"
},
{
"displayName": "GA4 - Google Merch Shop",
"parent": "accounts/54516992",
"property": "properties/213025502",
"propertyType": "PROPERTY_TYPE_ORDINARY"
}
]
}
]
}

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

@ -0,0 +1,786 @@
{
"kind": "analyticsData#batchRunReports",
"reports": [
{
"dimensionHeaders": [
{
"name": "date"
}
],
"kind": "analyticsData#runReport",
"metadata": {
"currencyCode": "USD",
"timeZone": "Europe/Amsterdam"
},
"metricHeaders": [
{
"name": "totalUsers",
"type": "TYPE_INTEGER"
},
{
"name": "screenPageViews",
"type": "TYPE_INTEGER"
},
{
"name": "bounces",
"type": "TYPE_INTEGER"
},
{
"name": "sessions",
"type": "TYPE_INTEGER"
},
{
"name": "userEngagementDuration",
"type": "TYPE_SECONDS"
}
],
"rowCount": 31,
"rows": [
{
"dimensionValues": [
{
"value": "20240131"
}
],
"metricValues": [
{
"value": "141"
},
{
"value": "195"
},
{
"value": "56"
},
{
"value": "146"
},
{
"value": "4815"
}
]
},
{
"dimensionValues": [
{
"value": "20240130"
}
],
"metricValues": [
{
"value": "115"
},
{
"value": "171"
},
{
"value": "42"
},
{
"value": "115"
},
{
"value": "3481"
}
]
},
{
"dimensionValues": [
{
"value": "20240129"
}
],
"metricValues": [
{
"value": "105"
},
{
"value": "153"
},
{
"value": "54"
},
{
"value": "113"
},
{
"value": "2895"
}
]
},
{
"dimensionValues": [
{
"value": "20240128"
}
],
"metricValues": [
{
"value": "184"
},
{
"value": "259"
},
{
"value": "74"
},
{
"value": "190"
},
{
"value": "6861"
}
]
},
{
"dimensionValues": [
{
"value": "20240127"
}
],
"metricValues": [
{
"value": "155"
},
{
"value": "202"
},
{
"value": "67"
},
{
"value": "158"
},
{
"value": "5259"
}
]
},
{
"dimensionValues": [
{
"value": "20240126"
}
],
"metricValues": [
{
"value": "132"
},
{
"value": "189"
},
{
"value": "57"
},
{
"value": "142"
},
{
"value": "5545"
}
]
},
{
"dimensionValues": [
{
"value": "20240125"
}
],
"metricValues": [
{
"value": "151"
},
{
"value": "229"
},
{
"value": "71"
},
{
"value": "155"
},
{
"value": "5365"
}
]
},
{
"dimensionValues": [
{
"value": "20240124"
}
],
"metricValues": [
{
"value": "137"
},
{
"value": "221"
},
{
"value": "64"
},
{
"value": "148"
},
{
"value": "4454"
}
]
},
{
"dimensionValues": [
{
"value": "20240123"
}
],
"metricValues": [
{
"value": "166"
},
{
"value": "267"
},
{
"value": "61"
},
{
"value": "171"
},
{
"value": "4850"
}
]
},
{
"dimensionValues": [
{
"value": "20240122"
}
],
"metricValues": [
{
"value": "161"
},
{
"value": "233"
},
{
"value": "73"
},
{
"value": "165"
},
{
"value": "4871"
}
]
},
{
"dimensionValues": [
{
"value": "20240121"
}
],
"metricValues": [
{
"value": "254"
},
{
"value": "397"
},
{
"value": "114"
},
{
"value": "267"
},
{
"value": "8845"
}
]
},
{
"dimensionValues": [
{
"value": "20240120"
}
],
"metricValues": [
{
"value": "214"
},
{
"value": "383"
},
{
"value": "100"
},
{
"value": "223"
},
{
"value": "6914"
}
]
},
{
"dimensionValues": [
{
"value": "20240119"
}
],
"metricValues": [
{
"value": "381"
},
{
"value": "5571"
},
{
"value": "100"
},
{
"value": "228"
},
{
"value": "98549"
}
]
},
{
"dimensionValues": [
{
"value": "20240118"
}
],
"metricValues": [
{
"value": "177"
},
{
"value": "278"
},
{
"value": "86"
},
{
"value": "186"
},
{
"value": "5553"
}
]
},
{
"dimensionValues": [
{
"value": "20240117"
}
],
"metricValues": [
{
"value": "204"
},
{
"value": "271"
},
{
"value": "97"
},
{
"value": "216"
},
{
"value": "6777"
}
]
},
{
"dimensionValues": [
{
"value": "20240116"
}
],
"metricValues": [
{
"value": "200"
},
{
"value": "280"
},
{
"value": "65"
},
{
"value": "203"
},
{
"value": "9381"
}
]
},
{
"dimensionValues": [
{
"value": "20240115"
}
],
"metricValues": [
{
"value": "255"
},
{
"value": "311"
},
{
"value": "86"
},
{
"value": "269"
},
{
"value": "12365"
}
]
},
{
"dimensionValues": [
{
"value": "20240114"
}
],
"metricValues": [
{
"value": "243"
},
{
"value": "302"
},
{
"value": "86"
},
{
"value": "257"
},
{
"value": "11073"
}
]
},
{
"dimensionValues": [
{
"value": "20240113"
}
],
"metricValues": [
{
"value": "231"
},
{
"value": "276"
},
{
"value": "92"
},
{
"value": "244"
},
{
"value": "10084"
}
]
},
{
"dimensionValues": [
{
"value": "20240112"
}
],
"metricValues": [
{
"value": "215"
},
{
"value": "265"
},
{
"value": "75"
},
{
"value": "225"
},
{
"value": "10195"
}
]
},
{
"dimensionValues": [
{
"value": "20240111"
}
],
"metricValues": [
{
"value": "183"
},
{
"value": "235"
},
{
"value": "64"
},
{
"value": "191"
},
{
"value": "8200"
}
]
},
{
"dimensionValues": [
{
"value": "20240110"
}
],
"metricValues": [
{
"value": "190"
},
{
"value": "244"
},
{
"value": "61"
},
{
"value": "203"
},
{
"value": "9056"
}
]
},
{
"dimensionValues": [
{
"value": "20240109"
}
],
"metricValues": [
{
"value": "201"
},
{
"value": "290"
},
{
"value": "63"
},
{
"value": "213"
},
{
"value": "11651"
}
]
},
{
"dimensionValues": [
{
"value": "20240108"
}
],
"metricValues": [
{
"value": "195"
},
{
"value": "235"
},
{
"value": "71"
},
{
"value": "209"
},
{
"value": "7946"
}
]
},
{
"dimensionValues": [
{
"value": "20240107"
}
],
"metricValues": [
{
"value": "205"
},
{
"value": "274"
},
{
"value": "80"
},
{
"value": "218"
},
{
"value": "10048"
}
]
},
{
"dimensionValues": [
{
"value": "20240106"
}
],
"metricValues": [
{
"value": "256"
},
{
"value": "303"
},
{
"value": "119"
},
{
"value": "281"
},
{
"value": "8440"
}
]
},
{
"dimensionValues": [
{
"value": "20240105"
}
],
"metricValues": [
{
"value": "226"
},
{
"value": "270"
},
{
"value": "83"
},
{
"value": "237"
},
{
"value": "9311"
}
]
},
{
"dimensionValues": [
{
"value": "20240104"
}
],
"metricValues": [
{
"value": "223"
},
{
"value": "274"
},
{
"value": "81"
},
{
"value": "236"
},
{
"value": "12051"
}
]
},
{
"dimensionValues": [
{
"value": "20240103"
}
],
"metricValues": [
{
"value": "233"
},
{
"value": "283"
},
{
"value": "79"
},
{
"value": "245"
},
{
"value": "10628"
}
]
},
{
"dimensionValues": [
{
"value": "20240102"
}
],
"metricValues": [
{
"value": "203"
},
{
"value": "269"
},
{
"value": "60"
},
{
"value": "210"
},
{
"value": "11168"
}
]
},
{
"dimensionValues": [
{
"value": "20240101"
}
],
"metricValues": [
{
"value": "191"
},
{
"value": "224"
},
{
"value": "71"
},
{
"value": "197"
},
{
"value": "8080"
}
]
}
]
}
]
}

@ -0,0 +1,38 @@
{
"kind": "analyticsData#batchRunReports",
"reports": [
{
"dimensionHeaders": [
{
"name": "date"
}
],
"kind": "analyticsData#runReport",
"metadata": {
"currencyCode": "USD",
"timeZone": "Europe/Warsaw"
},
"metricHeaders": [
{
"name": "screenPageViews",
"type": "TYPE_INTEGER"
}
],
"rowCount": 3,
"rows": [
{
"dimensionValues": [
{
"value": "20240222"
}
],
"metricValues": [
{
"value": "13"
}
]
}
]
}
]
}

@ -0,0 +1,38 @@
{
"reports": [
{
"columnHeader": {
"dimensions": [
"ga:date"
],
"metricHeader": {
"metricHeaderEntries": [
{
"name": "ga:pageviews",
"type": "INTEGER"
}
]
}
},
"data": {
"isDataGolden": true,
"rowCount": 849,
"rows": [
{
"dimensions": [
"20200421"
],
"metrics": [
{
"values": [
"37"
]
}
]
}
]
},
"nextPageToken": "1"
}
]
}

@ -0,0 +1,38 @@
{
"reports": [
{
"columnHeader": {
"dimensions": [
"ga:date"
],
"metricHeader": {
"metricHeaderEntries": [
{
"name": "ga:pageviews",
"type": "INTEGER"
}
]
}
},
"data": {
"isDataGolden": true,
"rowCount": 849,
"rows": [
{
"dimensions": [
"20120118"
],
"metrics": [
{
"values": [
"37"
]
}
]
}
]
},
"nextPageToken": "1"
}
]
}

@ -0,0 +1,38 @@
{
"reports": [
{
"columnHeader": {
"dimensions": [
"ga:date"
],
"metricHeader": {
"metricHeaderEntries": [
{
"name": "ga:pageviews",
"type": "INTEGER"
}
]
}
},
"data": {
"isDataGolden": true,
"rowCount": 849,
"rows": [
{
"dimensions": [
"20170118"
],
"metrics": [
{
"values": [
"37"
]
}
]
}
]
},
"nextPageToken": "1"
}
]
}

@ -24,6 +24,7 @@ defmodule Mix.Tasks.SendPageview do
page: :string, page: :string,
referrer: :string, referrer: :string,
host: :string, host: :string,
hostname: :string,
event: :string, event: :string,
props: :string, props: :string,
revenue_currency: :string, revenue_currency: :string,
@ -86,6 +87,7 @@ defmodule Mix.Tasks.SendPageview do
referrer = Keyword.get(opts, :referrer, @default_referrer) referrer = Keyword.get(opts, :referrer, @default_referrer)
event = Keyword.get(opts, :event, @default_event) event = Keyword.get(opts, :event, @default_event)
props = Keyword.get(opts, :props, @default_props) props = Keyword.get(opts, :props, @default_props)
hostname = Keyword.get(opts, :hostname, domain)
revenue = revenue =
if Keyword.get(opts, :revenue_currency) do if Keyword.get(opts, :revenue_currency) do
@ -97,7 +99,7 @@ defmodule Mix.Tasks.SendPageview do
%{ %{
name: event, name: event,
url: "http://#{domain}#{page}", url: "http://#{hostname}#{page}",
domain: domain, domain: domain,
referrer: referrer, referrer: referrer,
props: props, props: props,

@ -2,11 +2,15 @@ defmodule Plausible.Application do
@moduledoc false @moduledoc false
use Application use Application
use Plausible
require Logger require Logger
def start(_type, _args) do def start(_type, _args) do
on_full_build(do: Plausible.License.ensure_valid_license())
children = [ children = [
Plausible.Cache.Stats,
Plausible.Repo, Plausible.Repo,
Plausible.ClickhouseRepo, Plausible.ClickhouseRepo,
Plausible.IngestRepo, Plausible.IngestRepo,
@ -20,13 +24,15 @@ defmodule Plausible.Application do
Supervisor.child_spec(Plausible.Event.WriteBuffer, id: Plausible.Event.WriteBuffer), Supervisor.child_spec(Plausible.Event.WriteBuffer, id: Plausible.Event.WriteBuffer),
Supervisor.child_spec(Plausible.Session.WriteBuffer, id: Plausible.Session.WriteBuffer), Supervisor.child_spec(Plausible.Session.WriteBuffer, id: Plausible.Session.WriteBuffer),
ReferrerBlocklist, ReferrerBlocklist,
Supervisor.child_spec({Cachex, name: :user_agents, limit: 10_000, stats: true}, Plausible.Cache.Adapter.child_spec(:user_agents, :cache_user_agents,
id: :cachex_user_agents ttl_check_interval: :timer.seconds(5),
global_ttl: :timer.minutes(60)
), ),
Supervisor.child_spec({Cachex, name: :sessions, limit: nil, stats: true}, Plausible.Cache.Adapter.child_spec(:sessions, :cache_sessions,
id: :cachex_sessions ttl_check_interval: :timer.seconds(1),
global_ttl: :timer.minutes(30)
), ),
{Plausible.Site.Cache, []}, {Plausible.Site.Cache, ttl_check_interval: false},
{Plausible.Cache.Warmer, {Plausible.Cache.Warmer,
[ [
child_name: Plausible.Site.Cache.All, child_name: Plausible.Site.Cache.All,
@ -41,7 +47,7 @@ defmodule Plausible.Application do
interval: :timer.seconds(30), interval: :timer.seconds(30),
warmer_fn: :refresh_updated_recently warmer_fn: :refresh_updated_recently
]}, ]},
{Plausible.Shield.IPRuleCache, []}, {Plausible.Shield.IPRuleCache, ttl_check_interval: false},
{Plausible.Cache.Warmer, {Plausible.Cache.Warmer,
[ [
child_name: Plausible.Shield.IPRuleCache.All, child_name: Plausible.Shield.IPRuleCache.All,
@ -56,6 +62,36 @@ defmodule Plausible.Application do
interval: :timer.seconds(35), interval: :timer.seconds(35),
warmer_fn: :refresh_updated_recently warmer_fn: :refresh_updated_recently
]}, ]},
{Plausible.Shield.CountryRuleCache, ttl_check_interval: false},
{Plausible.Cache.Warmer,
[
child_name: Plausible.Shield.CountryRuleCache.All,
cache_impl: Plausible.Shield.CountryRuleCache,
interval: :timer.minutes(3) + Enum.random(1..:timer.seconds(10)),
warmer_fn: :refresh_all
]},
{Plausible.Cache.Warmer,
[
child_name: Plausible.Shield.CountryRuleCache.RecentlyUpdated,
cache_impl: Plausible.Shield.CountryRuleCache,
interval: :timer.seconds(35),
warmer_fn: :refresh_updated_recently
]},
{Plausible.Shield.PageRuleCache, ttl_check_interval: false, ets_options: [:bag]},
{Plausible.Cache.Warmer,
[
child_name: Plausible.Shield.PageRuleCache.All,
cache_impl: Plausible.Shield.PageRuleCache,
interval: :timer.minutes(3) + Enum.random(1..:timer.seconds(10)),
warmer_fn: :refresh_all
]},
{Plausible.Cache.Warmer,
[
child_name: Plausible.Shield.PageRuleCache.RecentlyUpdated,
cache_impl: Plausible.Shield.PageRuleCache,
interval: :timer.seconds(35),
warmer_fn: :refresh_updated_recently
]},
{Plausible.Auth.TOTP.Vault, key: totp_vault_key()}, {Plausible.Auth.TOTP.Vault, key: totp_vault_key()},
PlausibleWeb.Endpoint, PlausibleWeb.Endpoint,
{Oban, Application.get_env(:plausible, Oban)}, {Oban, Application.get_env(:plausible, Oban)},
@ -101,10 +137,10 @@ defmodule Plausible.Application do
defp maybe_add_sentry_pool(pool_config) do defp maybe_add_sentry_pool(pool_config) do
case Sentry.Config.dsn() do case Sentry.Config.dsn() do
dsn when is_binary(dsn) -> {"http" <> _rest = url, _, _} ->
Map.put(pool_config, dsn, size: 50) Map.put(pool_config, url, size: 50)
_ -> nil ->
pool_config pool_config
end end
end end
@ -150,16 +186,6 @@ defmodule Plausible.Application do
) )
end end
def report_cache_stats() do
case Cachex.stats(:user_agents) do
{:ok, stats} ->
Logger.info("User agent cache stats: #{inspect(stats)}")
e ->
IO.puts("Unable to show cache stats: #{inspect(e)}")
end
end
defp setup_opentelemetry() do defp setup_opentelemetry() do
OpentelemetryPhoenix.setup() OpentelemetryPhoenix.setup()
OpentelemetryEcto.setup([:plausible, :repo]) OpentelemetryEcto.setup([:plausible, :repo])

@ -87,6 +87,18 @@ defmodule Plausible.Billing.Feature do
@features @features
end end
@doc """
Lists all the feature short names, e.g. RevenueGoals
"""
defmacro list_short_names() do
@features
|> Enum.map(fn mod ->
Module.split(mod)
|> List.last()
|> String.to_atom()
end)
end
@doc false @doc false
defmacro __using__(opts \\ []) do defmacro __using__(opts \\ []) do
quote location: :keep do quote location: :keep do

@ -8,10 +8,10 @@ defmodule Plausible.Cache do
# - Optionally override `unwrap_cache_keys/1` # - Optionally override `unwrap_cache_keys/1`
# - Populate the cache with `Plausible.Cache.Warmer` # - Populate the cache with `Plausible.Cache.Warmer`
Serves as a thin wrapper around Cachex, but the underlying Serves as a wrapper around `Plausible.Cache.Adapter`, where the underlying
implementation can be transparently swapped. implementation can be transparently swapped.
Even though the Cachex process is started, cache access is disabled Even though normally the relevant Adapter processes are started, cache access is disabled
during tests via the `:plausible, #{__MODULE__}, enabled: bool()` application env key. during tests via the `:plausible, #{__MODULE__}, enabled: bool()` application env key.
This can be overridden on case by case basis, using the child specs options. This can be overridden on case by case basis, using the child specs options.
@ -61,26 +61,30 @@ defmodule Plausible.Cache do
@behaviour Plausible.Cache @behaviour Plausible.Cache
@modes [:all, :updated_recently] @modes [:all, :updated_recently]
alias Plausible.Cache.Adapter
@spec get(any(), Keyword.t()) :: any() | nil @spec get(any(), Keyword.t()) :: any() | nil
def get(key, opts \\ []) do def get(key, opts \\ []) when is_list(opts) do
cache_name = Keyword.get(opts, :cache_name, name()) cache_name = Keyword.get(opts, :cache_name, name())
force? = Keyword.get(opts, :force?, false) force? = Keyword.get(opts, :force?, false)
if Plausible.Cache.enabled?() or force? do if Plausible.Cache.enabled?() or force? do
case Cachex.get(cache_name, key) do Adapter.get(cache_name, key)
{:ok, nil} -> else
nil get_from_source(key)
end
{:ok, item} -> end
item
{:error, e} -> @spec get_or_store(any(), (-> any()), Keyword.t()) :: any() | nil
Logger.error("Error retrieving key from '#{inspect(cache_name)}': #{inspect(e)}") def get_or_store(key, fallback_fn, opts \\ [])
when is_function(fallback_fn, 0) and is_list(opts) do
cache_name = Keyword.get(opts, :cache_name, name())
force? = Keyword.get(opts, :force?, false)
nil if Plausible.Cache.enabled?() or force? do
end Adapter.get(cache_name, key, fallback_fn)
else else
get_from_source(key) get_from_source(key) || fallback_fn.()
end end
end end
@ -117,10 +121,10 @@ defmodule Plausible.Cache do
def merge_items(new_items, opts) do def merge_items(new_items, opts) do
new_items = unwrap_cache_keys(new_items) new_items = unwrap_cache_keys(new_items)
cache_name = Keyword.get(opts, :cache_name, name()) cache_name = Keyword.get(opts, :cache_name, name())
true = Cachex.put_many!(cache_name, new_items) :ok = Adapter.put_many(cache_name, new_items)
if Keyword.get(opts, :delete_stale_items?, true) do if Keyword.get(opts, :delete_stale_items?, true) do
{:ok, old_keys} = Cachex.keys(cache_name) old_keys = Adapter.keys(cache_name)
new = MapSet.new(Enum.into(new_items, [], fn {k, _} -> k end)) new = MapSet.new(Enum.into(new_items, [], fn {k, _} -> k end))
old = MapSet.new(old_keys) old = MapSet.new(old_keys)
@ -128,7 +132,7 @@ defmodule Plausible.Cache do
old old
|> MapSet.difference(new) |> MapSet.difference(new)
|> Enum.each(fn k -> |> Enum.each(fn k ->
Cachex.del(cache_name, k) Adapter.delete(cache_name, k)
end) end)
end end
@ -139,11 +143,7 @@ defmodule Plausible.Cache do
def child_spec(opts) do def child_spec(opts) do
cache_name = Keyword.get(opts, :cache_name, name()) cache_name = Keyword.get(opts, :cache_name, name())
child_id = Keyword.get(opts, :child_id, child_id()) child_id = Keyword.get(opts, :child_id, child_id())
Adapter.child_spec(cache_name, child_id, opts)
Supervisor.child_spec(
{Cachex, name: cache_name, limit: nil, stats: true},
id: child_id
)
end end
@doc """ @doc """
@ -154,7 +154,7 @@ defmodule Plausible.Cache do
@spec ready?(atom()) :: boolean @spec ready?(atom()) :: boolean
def ready?(cache_name \\ name()) do def ready?(cache_name \\ name()) do
case size(cache_name) do case size(cache_name) do
n when n > 0 -> n when is_integer(n) and n > 0 ->
true true
0 -> 0 ->
@ -165,21 +165,8 @@ defmodule Plausible.Cache do
end end
end end
@spec size() :: non_neg_integer() defdelegate size(cache_name \\ name()), to: Plausible.Cache.Adapter
def size(cache_name \\ name()) do defdelegate hit_rate(cache_name \\ name()), to: Plausible.Cache.Stats
case Cachex.size(cache_name) do
{:ok, size} -> size
_ -> 0
end
end
@spec hit_rate() :: number()
def hit_rate(cache_name \\ name()) do
case Cachex.stats(cache_name) do
{:ok, stats} -> Map.get(stats, :hit_rate, 0)
_ -> 0
end
end
@spec telemetry_event_refresh(atom(), atom()) :: list(atom()) @spec telemetry_event_refresh(atom(), atom()) :: list(atom())
def telemetry_event_refresh(cache_name \\ name(), mode) when mode in @modes do def telemetry_event_refresh(cache_name \\ name(), mode) when mode in @modes do
@ -189,26 +176,16 @@ defmodule Plausible.Cache do
defp refresh(mode, query, opts) when mode in @modes do defp refresh(mode, query, opts) when mode in @modes do
cache_name = Keyword.get(opts, :cache_name, name()) cache_name = Keyword.get(opts, :cache_name, name())
measure_duration(telemetry_event_refresh(cache_name, mode), fn -> Plausible.PromEx.Plugins.PlausibleMetrics.measure_duration(
items = Plausible.Repo.all(query) telemetry_event_refresh(cache_name, mode),
:ok = merge_items(items, opts) fn ->
end) items = Plausible.Repo.all(query)
:ok = merge_items(items, opts)
end
)
:ok :ok
end end
defp measure_duration(event, fun) when is_function(fun, 0) do
{duration, result} = time_it(fun)
:telemetry.execute(event, %{duration: duration}, %{})
result
end
defp time_it(fun) do
start = System.monotonic_time()
result = fun.()
stop = System.monotonic_time()
{stop - start, result}
end
end end
end end
end end

@ -0,0 +1,136 @@
defmodule Plausible.Cache.Adapter do
@moduledoc """
Interface for the underlying cache implementation.
Currently: ConCache
Using the Adapter module directly, the user must ensure that the relevant
processes are available to use, which is normally done via the child specification.
"""
require Logger
@spec child_spec(atom(), atom(), Keyword.t()) :: Supervisor.child_spec()
def child_spec(name, child_id, opts \\ [])
when is_atom(name) and is_atom(child_id) and is_list(opts) do
cache_name = Keyword.get(opts, :cache_name, name)
child_id = Keyword.get(opts, :child_id, child_id)
ttl_check_interval = Keyword.get(opts, :ttl_check_interval, false)
opts =
opts
|> Keyword.put(:name, cache_name)
|> Keyword.put(:ttl_check_interval, ttl_check_interval)
Supervisor.child_spec(
{ConCache, opts},
id: child_id
)
end
@spec size(atom()) :: non_neg_integer() | nil
def size(cache_name) do
ConCache.size(cache_name)
catch
:exit, _ -> nil
end
@spec get(atom(), any()) :: any()
def get(cache_name, key) do
result = ConCache.get(cache_name, key)
if is_nil(result), do: miss(cache_name), else: hit(cache_name)
result
catch
:exit, _ ->
Logger.error("Error retrieving key from '#{inspect(cache_name)}'")
nil
end
@spec get(atom(), any(), (-> any())) :: any()
def get(cache_name, key, fallback_fn) do
cache = ConCache.Owner.cache(cache_name)
case ConCache.Operations.get(cache, key) do
nil ->
get_or_store_isolated(cache, cache_name, key, fallback_fn)
value ->
hit(cache_name)
value
end
catch
:exit, _ ->
Logger.error("Error retrieving key from '#{inspect(cache_name)}'")
nil
end
@spec put(atom(), any(), any()) :: any()
def put(cache_name, key, value) do
:ok = ConCache.put(cache_name, key, value)
value
catch
:exit, _ ->
Logger.error("Error putting a key to '#{cache_name}'")
nil
end
@spec put_many(atom(), [any()]) :: :ok
def put_many(cache_name, items) when is_list(items) do
true = :ets.insert(ConCache.ets(cache_name), items)
:ok
catch
:exit, _ ->
Logger.error("Error putting keys to '#{cache_name}'")
:ok
end
@spec delete(atom(), any()) :: :ok
def delete(cache_name, key) do
ConCache.dirty_delete(cache_name, key)
catch
:exit, _ ->
Logger.error("Error deleting a key in '#{cache_name}'")
:ok
end
@spec keys(atom()) :: Enumerable.t()
def keys(cache_name) do
ets = ConCache.ets(cache_name)
Stream.resource(
fn -> :ets.first(ets) end,
fn
:"$end_of_table" -> {:halt, nil}
prev_key -> {[prev_key], :ets.next(ets, prev_key)}
end,
fn _ -> :ok end
)
catch
:exit, _ ->
Logger.error("Error retrieving key from '#{inspect(cache_name)}'")
[]
end
defp hit(cache_name) do
Plausible.Cache.Stats.record_hit(cache_name)
end
defp miss(cache_name) do
Plausible.Cache.Stats.record_miss(cache_name)
end
defp get_or_store_isolated(cache, cache_name, key, fun) do
ConCache.isolated(cache_name, key, fn ->
case ConCache.Operations.get(cache, key) do
nil ->
new_value = fun.()
ConCache.Operations.dirty_put(cache, key, new_value)
miss(cache_name)
new_value
existing ->
hit(cache_name)
existing
end
end)
end
end

@ -0,0 +1,84 @@
defmodule Plausible.Cache.Stats do
@moduledoc """
Keeps track of hit/miss ratio for various caches.
"""
use GenServer
@hit :hit
@miss :miss
@telemetry_hit [:plausible, :cache, :adapter, @hit]
@telemetry_miss [:plausible, :cache, :adapter, @miss]
@telemetry_events [@telemetry_hit, @telemetry_miss]
def start_link(_opts) do
GenServer.start_link(__MODULE__, nil)
end
def record_hit(cache_name) do
:telemetry.execute(@telemetry_hit, %{}, %{cache_name: cache_name})
end
def record_miss(cache_name) do
:telemetry.execute(@telemetry_miss, %{}, %{cache_name: cache_name})
end
def init(nil) do
__MODULE__ =
:ets.new(__MODULE__, [
:public,
:named_table,
:set,
read_concurrency: true,
write_concurrency: true
])
:telemetry.attach_many(
"plausible-cache-stats",
@telemetry_events,
&__MODULE__.handle_telemetry_event/4,
nil
)
{:ok, nil}
end
def handle_telemetry_event(@telemetry_hit, _measurments, %{cache_name: cache_name}, _) do
bump(cache_name, @hit)
end
def handle_telemetry_event(@telemetry_miss, _measurments, %{cache_name: cache_name}, _) do
bump(cache_name, @miss)
end
def gather(cache_name) do
{:ok,
%{
hit_rate: hit_rate(cache_name),
count: size(cache_name) || 0
}}
end
defdelegate size(cache_name), to: Plausible.Cache.Adapter
def bump(cache_name, type) do
:ets.update_counter(
__MODULE__,
{cache_name, type},
1,
{{cache_name, type}, 0}
)
end
def hit_rate(cache_name) do
hit = :ets.lookup_element(__MODULE__, {cache_name, @hit}, 2, 0)
miss = :ets.lookup_element(__MODULE__, {cache_name, @miss}, 2, 0)
hit_miss = hit + miss
if hit_miss == 0 do
0.0
else
hit / hit_miss * 100
end
end
end

@ -15,6 +15,15 @@ defmodule Plausible.ClickhouseEventV2 do
field :session_id, Ch, type: "UInt64" field :session_id, Ch, type: "UInt64"
field :timestamp, :naive_datetime field :timestamp, :naive_datetime
field :"meta.key", {:array, :string}
field :"meta.value", {:array, :string}
field :revenue_source_amount, Ch, type: "Nullable(Decimal64(3))"
field :revenue_source_currency, Ch, type: "FixedString(3)"
field :revenue_reporting_amount, Ch, type: "Nullable(Decimal64(3))"
field :revenue_reporting_currency, Ch, type: "FixedString(3)"
# Session attributes
field :referrer, :string field :referrer, :string
field :referrer_source, :string field :referrer_source, :string
field :utm_medium, :string field :utm_medium, :string
@ -33,16 +42,6 @@ defmodule Plausible.ClickhouseEventV2 do
field :operating_system_version, Ch, type: "LowCardinality(String)" field :operating_system_version, Ch, type: "LowCardinality(String)"
field :browser, Ch, type: "LowCardinality(String)" field :browser, Ch, type: "LowCardinality(String)"
field :browser_version, Ch, type: "LowCardinality(String)" field :browser_version, Ch, type: "LowCardinality(String)"
field :"meta.key", {:array, :string}
field :"meta.value", {:array, :string}
field :revenue_source_amount, Ch, type: "Nullable(Decimal64(3))"
field :revenue_source_currency, Ch, type: "FixedString(3)"
field :revenue_reporting_amount, Ch, type: "Nullable(Decimal64(3))"
field :revenue_reporting_currency, Ch, type: "FixedString(3)"
field :transferred_from, :string
end end
def new(attrs) do def new(attrs) do
@ -56,22 +55,6 @@ defmodule Plausible.ClickhouseEventV2 do
:pathname, :pathname,
:user_id, :user_id,
:timestamp, :timestamp,
:operating_system,
:operating_system_version,
:browser,
:browser_version,
:referrer,
:referrer_source,
:utm_medium,
:utm_source,
:utm_campaign,
:utm_content,
:utm_term,
:country_code,
:subdivision1_code,
:subdivision2_code,
:city_geoname_id,
:screen_size,
:"meta.key", :"meta.key",
:"meta.value", :"meta.value",
:revenue_source_amount, :revenue_source_amount,
@ -82,4 +65,28 @@ defmodule Plausible.ClickhouseEventV2 do
) )
|> validate_required([:name, :site_id, :hostname, :pathname, :user_id, :timestamp]) |> validate_required([:name, :site_id, :hostname, :pathname, :user_id, :timestamp])
end end
@session_properties [
:session_id,
:referrer,
:referrer_source,
:utm_medium,
:utm_source,
:utm_campaign,
:utm_content,
:utm_term,
:country_code,
:subdivision1_code,
:subdivision2_code,
:city_geoname_id,
:screen_size,
:operating_system,
:operating_system_version,
:browser,
:browser_version
]
def merge_session(%__MODULE__{} = event, session) do
Map.merge(event, Map.take(session, @session_properties))
end
end end

@ -43,6 +43,7 @@ defmodule Plausible.ClickhouseSessionV2 do
field :is_bounce, BoolUInt8 field :is_bounce, BoolUInt8
field :entry_page, :string field :entry_page, :string
field :exit_page, :string field :exit_page, :string
field :exit_page_hostname, :string
field :pageviews, Ch, type: "Int32" field :pageviews, Ch, type: "Int32"
field :events, Ch, type: "Int32" field :events, Ch, type: "Int32"
field :sign, Ch, type: "Int8" field :sign, Ch, type: "Int8"

@ -62,16 +62,21 @@ defmodule Plausible.DataMigration do
|> EEx.eval_file(assigns: assigns) |> EEx.eval_file(assigns: assigns)
end end
def run_sql(name, assigns \\ []) do def run_sql(name, assigns \\ [], options \\ []) do
query = unwrap(name, assigns) query = unwrap(name, assigns)
do_run(name, query) do_run(name, query, options)
end end
defp do_run(name, query) do defp do_run(name, query, options \\ []) do
{:ok, res} = @repo.query(query, [], timeout: :infinity) case @repo.query(query, [], [timeout: :infinity] ++ options) do
IO.puts(" #{IO.ANSI.yellow()}#{name} #{IO.ANSI.green()}Done!#{IO.ANSI.reset()}\n") {:ok, res} ->
IO.puts(String.duplicate("-", 78)) IO.puts(" #{IO.ANSI.yellow()}#{name} #{IO.ANSI.green()}Done!#{IO.ANSI.reset()}\n")
{:ok, res} IO.puts(String.duplicate("-", 78))
{:ok, res}
result ->
result
end
end end
defp unwrap_with_io(name, assigns) do defp unwrap_with_io(name, assigns) do

@ -0,0 +1,166 @@
defmodule Plausible.DataMigration.PopulateEventSessionColumns do
@moduledoc """
Populates event session columns with data from sessions table.
Run via: ./bin/plausible rpc "Plausible.DataMigration.PopulateEventSessionColumns.run"
Kill via: ./bin/plausible rpc "Plausible.DataMigration.PopulateEventSessionColumns.kill"
Monitor via ./bin/plausible rpc "Plausible.DataMigration.PopulateEventSessionColumns.report_progress"
Suggested to run in a screen/tmux session to be able to easily monitor
SQL files available at: priv/data_migrations/PopulateEventSessionColumns/sql
"""
use Plausible.DataMigration, dir: "PopulateEventSessionColumns", repo: Plausible.IngestRepo
require Logger
# See https://clickhouse.com/docs/en/sql-reference/dictionaries#cache for meaning of these defaults
@default_dictionary_config %{
lifetime: 600_000,
size_in_cells: 1_000_000,
max_threads_for_updates: 6
}
def run(opts \\ []) do
cluster? = Plausible.MigrationUtils.clustered_table?("sessions_v2")
{:ok, _} =
run_sql("create-sessions-dictionary",
cluster?: cluster?,
dictionary_connection_params:
Keyword.get(opts, :dictionary_connection_string, dictionary_connection_params()),
dictionary_config: dictionary_config(opts)
)
{partitions, _, _, _} = get_partitions(opts)
IO.puts("Starting mutation on #{length(partitions)} partition(s)")
for partition <- partitions do
{:ok, _} =
run_sql("update-table", [cluster?: cluster?, partition: partition],
settings: [allow_nondeterministic_mutations: 1]
)
end
wait_until_mutations_complete(opts)
IO.puts("Mutations seem done, cleaning up!")
{:ok, _} = run_sql("drop-sessions-dictionary", cluster?: cluster?)
end
def kill(opts \\ []) do
cluster? = Plausible.MigrationUtils.clustered_table?("events_v2")
report_progress(opts)
IO.puts("Killing running mutations")
{:ok, _} = run_sql("kill-running-mutations", cluster?: cluster?)
end
def wait_until_mutations_complete(opts \\ []) do
Process.sleep(5_000)
in_progress? = report_progress(opts)
if in_progress? do
wait_until_mutations_complete(opts)
end
end
def report_progress(opts \\ []) do
{partitions, parts, min_partition, max_partition} = get_partitions(opts)
{:ok, %{rows: mutation_results}} =
run_sql("get-mutations-progress",
min_partition: min_partition,
max_partition: max_partition
)
[
[
mutations,
parts_to_do,
running_for,
total_size,
todo_size,
progress,
latest_fail_reason,
_,
_
]
] =
mutation_results
{:ok, %{rows: [[merges]]}} = run_sql("get-merges-progress")
{:ok, %{rows: disks}} = run_sql("get-disks")
IO.puts("\n\n#{Timex.now() |> Timex.format!("{ISO:Extended}")}")
# List partitions that need to run
IO.puts(
"Progress report for partitions #{Enum.min(partitions)}-#{Enum.max(partitions)} (parts: #{length(parts)})"
)
IO.puts("Disks overview:")
for [name, path, full_space, total_space, full_percentage] <- disks do
IO.puts(
" #{name} at #{path} is at #{full_space}/#{total_space} (#{full_percentage}% full)"
)
end
IO.puts("Currently #{mutations} mutation(s) are running.")
if mutations > 0 do
IO.puts(" To do #{parts_to_do} parts, #{todo_size}")
IO.puts(" Out of #{length(parts)} parts, #{total_size}")
IO.puts(" Running for #{format_duration(running_for)}")
if progress > 0 do
estimated_time_left = round(running_for / progress / 100 - running_for)
IO.puts(" Estimated #{progress}% done, #{format_duration(estimated_time_left)} left")
end
if latest_fail_reason do
IO.puts(" Some mutations might be failing. ClickHouse report: #{latest_fail_reason}")
end
end
IO.puts("Currently #{merges} merge(s) are running relating to mutations.")
mutations > 0
end
defp dictionary_config(opts) do
@default_dictionary_config
|> Map.merge(Keyword.get(opts, :dictionary_config, %{}))
end
# See https://clickhouse.com/docs/en/sql-reference/dictionaries#clickhouse for context
defp dictionary_connection_params() do
Plausible.IngestRepo.config()
|> Enum.map(fn
{:database, database} -> "DB '#{database}'"
{:username, username} -> "USER '#{username}'"
{:password, password} -> "PASSWORD '#{password}'"
_ -> nil
end)
|> Enum.reject(&is_nil/1)
|> Enum.join(" ")
end
defp get_partitions(opts) do
[min_partition, max_partition] = Keyword.get(opts, :partition_range, ["0", "999999"])
{:ok, %{rows: [[partitions, parts]]}} =
run_sql("list-partitions", min_partition: min_partition, max_partition: max_partition)
{partitions, parts, min_partition, max_partition}
end
defp format_duration(seconds) do
seconds
|> Timex.Duration.from_seconds()
|> Timex.format_duration(Timex.Format.Duration.Formatters.Humanized)
end
end

@ -0,0 +1,188 @@
defmodule Plausible.DataMigration.SiteImports do
@moduledoc """
Site imports migration backfilling SiteImport entries for old imports
and alters import end dates to match actual end date of respective import stats.
"""
import Ecto.Query
alias Plausible.ClickhouseRepo
alias Plausible.Imported
alias Plausible.Repo
alias Plausible.Site
def run(opts \\ []) do
dry_run? = Keyword.get(opts, :dry_run?, true)
site_import_query =
from(i in Imported.SiteImport, where: i.site_id == parent_as(:site).id, select: 1)
sites_with_imports =
from(s in Site, as: :site, where: not is_nil(s.imported_data) or exists(site_import_query))
|> Repo.all(log: false)
sites_count = length(sites_with_imports)
IO.puts("Processing #{sites_count} sites with imports (DRY RUN: #{dry_run?})...")
for {site, idx} <- Enum.with_index(sites_with_imports) do
site_imports =
from(i in Imported.SiteImport, where: i.site_id == ^site.id)
|> Repo.all(log: false)
IO.puts(
"Processing site ID #{site.id} (#{idx + 1} / #{sites_count}) (imported_data: #{is_struct(site.imported_data)}, site_imports: #{length(site_imports)})"
)
site_imports =
if site.imported_data && not Enum.any?(site_imports, & &1.legacy) do
IO.puts("Creating legacy site import entry for site ID #{site.id}")
# create legacy entry if there's not one yet
params =
site.imported_data
|> Imported.SiteImport.from_legacy()
|> Map.put(:site_id, site.id)
|> Map.take([:legacy, :start_date, :end_date, :source, :status, :site_id])
legacy_site_import =
%Imported.SiteImport{}
|> Ecto.Changeset.change(params)
|> insert!(dry_run?)
[legacy_site_import | site_imports]
else
IO.puts("Legacy site import entry for site ID #{site.id} already exists")
site_imports
end
# adjust end date for each site import
for site_import <- site_imports do
IO.puts(
"Adjusting end date for site import #{site_import.id} (site ID #{site.id}, start date: #{site_import.start_date}, end date: #{site_import.end_date})"
)
import_ids =
if site_import.legacy do
[0, site_import.id]
else
[site_import.id]
end
end_date = imported_stats_end_date(site.id, import_ids)
if !end_date do
IO.puts(
"Site import #{site_import.id} (site ID #{site.id}) does not have any recorded stats. Removing it."
)
if site_import.legacy do
clear_imported_data(site, dry_run?)
end
delete!(site_import, dry_run?)
else
end_date =
if Date.compare(end_date, site_import.end_date) in [:lt, :eq] do
end_date
else
IO.puts(
"Site import #{site_import.id} (site ID #{site.id}) computed end date is later than the current one. Skipping."
)
site_import.end_date
end
site_import
|> Ecto.Changeset.change(end_date: end_date)
|> update!(dry_run?)
IO.puts(
"End date of site import #{site_import.id} (site ID #{site.id}) adjusted to #{end_date}"
)
end
end
IO.puts("Done processing site ID #{site.id}")
end
IO.puts("Finished")
end
# Exposed for testing purposes
@doc false
def imported_stats_end_date(site_id, import_ids) do
[first_schema | schemas] = Imported.schemas()
query =
Enum.reduce(schemas, max_date_query(first_schema, site_id, import_ids), fn schema, query ->
from(s in subquery(union_all(query, ^max_date_query(schema, site_id, import_ids))))
end)
dates = ClickhouseRepo.all(from(q in query, select: q.max_date), log: false)
if dates != [] do
case Enum.max(dates, Date) do
# no stats for this domain yet
~D[1970-01-01] ->
nil
date ->
date
end
else
nil
end
end
defp insert!(changeset, false = _dry_run?) do
Repo.insert!(changeset)
end
defp insert!(changeset, true = _dry_run?) do
if changeset.valid? do
changeset
|> Ecto.Changeset.change(id: 0)
|> Ecto.Changeset.apply_changes()
else
raise "Invalid insert: #{inspect(changeset)}"
end
end
defp clear_imported_data(site, false = _dry_run?) do
Repo.update_all(from(s in Site, where: s.id == ^site.id), set: [imported_data: nil])
end
defp clear_imported_data(site, true = _dry_run?) do
%{site | imported_data: nil}
end
defp update!(changeset, false = _dry_run?) do
Repo.update!(changeset)
end
defp update!(changeset, true = _dry_run?) do
if changeset.valid? do
Ecto.Changeset.apply_changes(changeset)
else
raise "Invalid update: #{inspect(changeset)}"
end
end
defp delete!(entity, false = _dry_run?) do
Repo.delete!(entity)
end
defp delete!(entity, true = _dry_run?) do
entity
end
defp max_date_query(schema, site_id, import_ids) do
from(q in schema,
where: q.site_id == ^site_id,
where: q.import_id in ^import_ids,
select: %{max_date: fragment("max(?)", q.date)}
)
end
end

@ -0,0 +1,75 @@
defmodule Plausible.DataMigration.VersionedSessions do
@moduledoc """
Sessions CollapsingMergeTree -> VersionedCollapsingMergeTree migration, SQL files available at:
priv/data_migrations/VersionedSessions/sql
"""
use Plausible.DataMigration, dir: "VersionedSessions", repo: Plausible.IngestRepo
@suffix_format "{YYYY}{0M}{0D}{h24}{m}{s}"
@versioned_table_engines [
"ReplicatedVersionedCollapsingMergeTree",
"VersionedCollapsingMergeTree"
]
def run(opts \\ []) do
run_exchange? = Keyword.get(opts, :run_exchange?, true)
unique_suffix = Timex.now() |> Timex.format!(@suffix_format)
cluster? = Plausible.MigrationUtils.clustered_table?("sessions_v2")
{:ok, %{rows: partitions}} = run_sql("list-partitions")
partitions = Enum.map(partitions, fn [part] -> part end)
{:ok, %{rows: [[current_table_engine, table_settings]]}} =
run_sql("get-sessions-table-settings")
if Enum.member?(@versioned_table_engines, current_table_engine) do
IO.puts("sessions_v2 table is already versioned, no migration needed")
else
{:ok, _} = run_sql("drop-sessions-tmp-table", cluster?: cluster?)
{:ok, _} =
run_sql("create-sessions-tmp-table",
cluster?: cluster?,
table_settings: table_settings,
unique_suffix: unique_suffix
)
for partition <- partitions do
{:ok, _} = run_sql("attach-partition", partition: partition)
end
if run_exchange? do
run_exchange(cluster?)
end
IO.puts("Migration done!")
end
end
defp run_exchange(cluster?) do
case run_sql("exchange-sessions-tables", cluster?: cluster?) do
{:ok, _} ->
nil
# Docker containers don't seem to support EXCHANGE TABLE, hack around this with a non-atomic swap
{:error, %Ch.Error{code: 1}} ->
IO.puts("Exchanging sessions_v2 and sessions_v2_tmp_versioned non-atomically")
{:ok, _} =
run_sql("rename-table",
from: "sessions_v2",
to: "sessions_v2_backup",
cluster?: cluster?
)
{:ok, _} =
run_sql("rename-table",
from: "sessions_v2_tmp_versioned",
to: "sessions_v2",
cluster?: cluster?
)
end
end
end

@ -0,0 +1,371 @@
defmodule Plausible.Exports do
@moduledoc """
Contains functions to export data for events and sessions as Zip archives.
"""
require Plausible
import Ecto.Query
@doc """
Renders filename for the Zip archive containing the exported CSV files.
Examples:
iex> archive_filename("plausible.io", ~D[2021-01-01], ~D[2024-12-31])
"plausible_io_20210101_20241231.zip"
iex> archive_filename("Bücher.example", ~D[2021-01-01], ~D[2024-12-31])
"Bücher_example_20210101_20241231.zip"
"""
def archive_filename(domain, min_date, max_date) do
name =
Enum.join(
[
String.replace(domain, ".", "_"),
Calendar.strftime(min_date, "%Y%m%d"),
Calendar.strftime(max_date, "%Y%m%d")
],
"_"
)
name <> ".zip"
end
@doc """
Builds Ecto queries to export data from `events_v2` and `sessions_v2`
tables into the format of `imported_*` tables for a website.
"""
@spec export_queries(pos_integer, extname: String.t(), date_range: Date.Range.t()) ::
%{String.t() => Ecto.Query.t()}
def export_queries(site_id, opts \\ []) do
extname = opts[:extname] || ".csv"
date_range = opts[:date_range]
filename = fn table ->
name =
if date_range do
first_date = Timex.format!(date_range.first, "{YYYY}{0M}{0D}")
last_date = Timex.format!(date_range.last, "{YYYY}{0M}{0D}")
"#{table}_#{first_date}_#{last_date}"
else
table
end
name <> extname
end
%{
filename.("imported_visitors") => export_visitors_q(site_id),
filename.("imported_sources") => export_sources_q(site_id),
# NOTE: this query can result in `MEMORY_LIMIT_EXCEEDED` error
filename.("imported_pages") => export_pages_q(site_id),
filename.("imported_entry_pages") => export_entry_pages_q(site_id),
filename.("imported_exit_pages") => export_exit_pages_q(site_id),
filename.("imported_locations") => export_locations_q(site_id),
filename.("imported_devices") => export_devices_q(site_id),
filename.("imported_browsers") => export_browsers_q(site_id),
filename.("imported_operating_systems") => export_operating_systems_q(site_id)
}
end
Plausible.on_full_build do
defp sampled(table) do
Plausible.Stats.Sampling.add_query_hint(from(table))
end
else
defp sampled(table) do
table
end
end
defmacrop date(timestamp) do
quote do
selected_as(fragment("toDate(?)", unquote(timestamp)), :date)
end
end
defmacrop visit_duration(t) do
quote do
selected_as(
fragment("greatest(sum(?*?),0)", unquote(t).sign, unquote(t).duration),
:visit_duration
)
end
end
defmacrop visitors(t) do
quote do
selected_as(
fragment("toUInt64(round(uniq(?)*any(_sample_factor)))", unquote(t).user_id),
:visitors
)
end
end
defmacrop visits(t) do
quote do
selected_as(sum(unquote(t).sign), :visits)
end
end
defmacrop bounces(t) do
quote do
selected_as(
fragment("greatest(sum(?*?),0)", unquote(t).sign, unquote(t).is_bounce),
:bounces
)
end
end
defmacrop pageviews(t) do
quote do
selected_as(
fragment("greatest(sum(?*?),0)", unquote(t).sign, unquote(t).pageviews),
:pageviews
)
end
end
@spec export_visitors_q(pos_integer) :: Ecto.Query.t()
def export_visitors_q(site_id) do
from s in sampled("sessions_v2"),
where: s.site_id == ^site_id,
group_by: selected_as(:date),
select: [
date(s.start),
visitors(s),
pageviews(s),
bounces(s),
visits(s),
visit_duration(s)
]
end
@spec export_sources_q(pos_integer) :: Ecto.Query.t()
def export_sources_q(site_id) do
from s in sampled("sessions_v2"),
where: s.site_id == ^site_id,
group_by: [
selected_as(:date),
selected_as(:source),
s.referrer,
s.utm_source,
s.utm_medium,
s.utm_campaign,
s.utm_content,
s.utm_term
],
order_by: selected_as(:date),
select: [
date(s.start),
selected_as(s.referrer_source, :source),
s.referrer,
s.utm_source,
s.utm_medium,
s.utm_campaign,
s.utm_content,
s.utm_term,
pageviews(s),
visitors(s),
visits(s),
visit_duration(s),
bounces(s)
]
end
@spec export_pages_q(pos_integer) :: Ecto.Query.t()
def export_pages_q(site_id) do
window_q =
from e in sampled("events_v2"),
where: e.site_id == ^site_id,
select: %{
timestamp: e.timestamp,
next_timestamp:
over(fragment("leadInFrame(?)", e.timestamp),
partition_by: e.session_id,
order_by: e.timestamp,
frame: fragment("ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING")
),
pathname: e.pathname,
hostname: e.hostname,
name: e.name,
user_id: e.user_id,
session_id: e.session_id,
_sample_factor: fragment("_sample_factor")
}
from e in subquery(window_q),
group_by: [selected_as(:date), e.pathname],
order_by: selected_as(:date),
select: [
date(e.timestamp),
selected_as(fragment("any(?)", e.hostname), :hostname),
selected_as(e.pathname, :page),
selected_as(
fragment("toUInt64(round(uniq(?)*any(_sample_factor)))", e.session_id),
:visits
),
visitors(e),
selected_as(
fragment("toUInt64(round(countIf(?='pageview')*any(_sample_factor)))", e.name),
:pageviews
),
# NOTE: are exits pageviews or any events?
selected_as(
fragment("toUInt64(round(countIf(?=0)*any(_sample_factor)))", e.next_timestamp),
:exits
),
selected_as(
fragment("sum(greatest(?,0))", e.next_timestamp - e.timestamp),
:time_on_page
)
]
end
@spec export_entry_pages_q(pos_integer) :: Ecto.Query.t()
def export_entry_pages_q(site_id) do
from s in sampled("sessions_v2"),
where: s.site_id == ^site_id,
group_by: [selected_as(:date), s.entry_page],
order_by: selected_as(:date),
select: [
date(s.start),
s.entry_page,
visitors(s),
selected_as(
fragment("toUInt64(round(sum(?)*any(_sample_factor)))", s.sign),
:entrances
),
visit_duration(s),
bounces(s),
pageviews(s)
]
end
@spec export_exit_pages_q(pos_integer) :: Ecto.Query.t()
def export_exit_pages_q(site_id) do
from s in sampled("sessions_v2"),
where: s.site_id == ^site_id,
group_by: [selected_as(:date), s.exit_page],
order_by: selected_as(:date),
select: [
date(s.start),
s.exit_page,
visitors(s),
visit_duration(s),
selected_as(
fragment("toUInt64(round(sum(?)*any(_sample_factor)))", s.sign),
:exits
),
bounces(s),
pageviews(s)
]
end
@spec export_locations_q(pos_integer) :: Ecto.Query.t()
def export_locations_q(site_id) do
from s in sampled("sessions_v2"),
where: s.site_id == ^site_id,
where: s.city_geoname_id != 0 and s.country_code != "\0\0" and s.country_code != "ZZ",
group_by: [selected_as(:date), s.country_code, selected_as(:region), s.city_geoname_id],
order_by: selected_as(:date),
select: [
date(s.start),
selected_as(s.country_code, :country),
selected_as(s.subdivision1_code, :region),
selected_as(s.city_geoname_id, :city),
visitors(s),
visits(s),
visit_duration(s),
bounces(s),
pageviews(s)
]
end
@spec export_devices_q(pos_integer) :: Ecto.Query.t()
def export_devices_q(site_id) do
from s in sampled("sessions_v2"),
where: s.site_id == ^site_id,
group_by: [selected_as(:date), s.screen_size],
order_by: selected_as(:date),
select: [
date(s.start),
selected_as(s.screen_size, :device),
visitors(s),
visits(s),
visit_duration(s),
bounces(s),
pageviews(s)
]
end
@spec export_browsers_q(pos_integer) :: Ecto.Query.t()
def export_browsers_q(site_id) do
from s in sampled("sessions_v2"),
where: s.site_id == ^site_id,
group_by: [selected_as(:date), s.browser, s.browser_version],
order_by: selected_as(:date),
select: [
date(s.start),
s.browser,
s.browser_version,
visitors(s),
visits(s),
visit_duration(s),
bounces(s),
pageviews(s)
]
end
@spec export_operating_systems_q(pos_integer) :: Ecto.Query.t()
def export_operating_systems_q(site_id) do
from s in sampled("sessions_v2"),
where: s.site_id == ^site_id,
group_by: [selected_as(:date), s.operating_system, s.operating_system_version],
order_by: selected_as(:date),
select: [
date(s.start),
s.operating_system,
s.operating_system_version,
visitors(s),
visits(s),
visit_duration(s),
bounces(s),
pageviews(s)
]
end
@doc """
Creates a streamable Zip archive from the provided (named) Ecto queries.
Example usage:
{:ok, pool} = Ch.start_link(pool_size: 1)
DBConnection.run(pool, fn conn ->
conn
|> stream_archive(export_queries(_site_id = 1), format: "CSVWithNames")
|> Stream.into(File.stream!("export.zip"))
|> Stream.run()
end)
"""
@spec stream_archive(DBConnection.t(), %{String.t() => Ecto.Query.t()}, [Ch.query_option()]) ::
Enumerable.t()
def stream_archive(conn, named_queries, opts \\ []) do
entries =
Enum.map(named_queries, fn {name, query} ->
{sql, params} = Plausible.ClickhouseRepo.to_sql(:all, query)
datastream =
conn
|> Ch.stream(sql, params, opts)
|> Stream.map(fn %Ch.Result{data: data} -> data end)
Zstream.entry(name, datastream, coder: Zstream.Coder.Stored)
end)
Zstream.zip(entries)
end
end

@ -22,6 +22,9 @@ defmodule Plausible.Geo do
* `:edition` - the name of the MaxMind database to be downloaded from MaxMind * `:edition` - the name of the MaxMind database to be downloaded from MaxMind
servers. Defaults to `GeoLite2-City`. servers. Defaults to `GeoLite2-City`.
* `:cache_dir` - if set, the downloaded .mmdb files are cached there across
restarts.
* `:async` - when used, configures the database loading to run * `:async` - when used, configures the database loading to run
asynchronously. asynchronously.
@ -42,17 +45,19 @@ defmodule Plausible.Geo do
cond do cond do
license_key = opts[:license_key] -> license_key = opts[:license_key] ->
edition = opts[:edition] || "GeoLite2-City" edition = opts[:edition] || "GeoLite2-City"
maxmind_opts = [license_key: license_key]
if is_binary(opts[:cache_dir]) do loader_opts =
:ok = if is_binary(opts[:cache_dir]) do
:locus.start_loader(@db, {:maxmind, edition}, [
license_key: license_key,
database_cache_file: database_cache_file:
String.to_charlist(Path.join(opts[:cache_dir], edition <> ".mmdb.gz")) String.to_charlist(Path.join(opts[:cache_dir], edition <> ".mmdb.gz"))
) ]
else else
:ok = :locus.start_loader(@db, {:maxmind, edition}, license_key: license_key) [:no_cache]
end end
:ok = :locus.start_loader(@db, {:maxmind, edition}, maxmind_opts ++ loader_opts)
path = opts[:path] -> path = opts[:path] ->
:ok = :locus.start_loader(@db, path) :ok = :locus.start_loader(@db, path)

@ -80,6 +80,14 @@ defmodule Plausible.Goals do
def find_or_create(_, %{"goal_type" => "page"}), do: {:missing, "page_path"} def find_or_create(_, %{"goal_type" => "page"}), do: {:missing, "page_path"}
def list_revenue_goals(site) do
from(g in Plausible.Goal,
where: g.site_id == ^site.id and not is_nil(g.currency),
select: %{event_name: g.event_name, currency: g.currency}
)
|> Plausible.Repo.all()
end
def for_site(site, opts \\ []) do def for_site(site, opts \\ []) do
site site
|> for_site_query(opts) |> for_site_query(opts)

@ -1,9 +1,13 @@
defmodule Plausible.Google.Api do defmodule Plausible.Google.API do
alias Plausible.Google.{ReportRequest, HTTP} @moduledoc """
API to Google services.
"""
use Timex use Timex
require Logger
@type google_analytics_view() :: {view_name :: String.t(), view_id :: String.t()} alias Plausible.Google.HTTP
require Logger
@search_console_scope URI.encode_www_form( @search_console_scope URI.encode_www_form(
"email https://www.googleapis.com/auth/webmasters.readonly" "email https://www.googleapis.com/auth/webmasters.readonly"
@ -17,9 +21,46 @@ defmodule Plausible.Google.Api do
Jason.encode!([site_id, redirect_to]) Jason.encode!([site_id, redirect_to])
end end
def import_authorize_url(site_id, redirect_to) do def import_authorize_url(site_id, redirect_to, opts \\ []) do
legacy = Keyword.get(opts, :legacy, true)
"https://accounts.google.com/o/oauth2/v2/auth?client_id=#{client_id()}&redirect_uri=#{redirect_uri()}&prompt=consent&response_type=code&access_type=offline&scope=#{@import_scope}&state=" <> "https://accounts.google.com/o/oauth2/v2/auth?client_id=#{client_id()}&redirect_uri=#{redirect_uri()}&prompt=consent&response_type=code&access_type=offline&scope=#{@import_scope}&state=" <>
Jason.encode!([site_id, redirect_to]) Jason.encode!([site_id, redirect_to, legacy])
end
def fetch_access_token!(code) do
HTTP.fetch_access_token!(code)
end
def list_properties_and_views(access_token) do
with {:ok, properties} <- Plausible.Google.GA4.API.list_properties(access_token),
{:ok, views} <- Plausible.Google.UA.API.list_views(access_token) do
{:ok, properties ++ views}
end
end
def get_property_or_view(access_token, property_or_view) do
if property?(property_or_view) do
Plausible.Google.GA4.API.get_property(access_token, property_or_view)
else
Plausible.Google.UA.API.get_view(access_token, property_or_view)
end
end
def get_analytics_start_date(access_token, property_or_view) do
if property?(property_or_view) do
Plausible.Google.GA4.API.get_analytics_start_date(access_token, property_or_view)
else
Plausible.Google.UA.API.get_analytics_start_date(access_token, property_or_view)
end
end
def get_analytics_end_date(access_token, property_or_view) do
if property?(property_or_view) do
Plausible.Google.GA4.API.get_analytics_end_date(access_token, property_or_view)
else
Plausible.Google.UA.API.get_analytics_end_date(access_token, property_or_view)
end
end end
def fetch_verified_properties(auth) do def fetch_verified_properties(auth) do
@ -53,138 +94,7 @@ defmodule Plausible.Google.Api do
end end
end end
@spec list_views(access_token :: String.t()) :: def maybe_refresh_token(%Plausible.Site.GoogleAuth{} = auth) do
{:ok, %{(hostname :: String.t()) => [google_analytics_view()]}} | {:error, term()}
@doc """
Lists Google Analytics views grouped by hostname.
"""
def list_views(access_token) do
case HTTP.list_views_for_user(access_token) do
{:ok, %{"items" => views}} ->
views = Enum.group_by(views, &view_hostname/1, &view_names/1)
{:ok, views}
error ->
error
end
end
defp view_hostname(view) do
case view do
%{"websiteUrl" => url} when is_binary(url) -> url |> URI.parse() |> Map.get(:host)
_any -> "Others"
end
end
defp view_names(%{"name" => name, "id" => id}) do
{"#{id} - #{name}", id}
end
@spec get_view(access_token :: String.t(), lookup_id :: String.t()) ::
{:ok, google_analytics_view()} | {:ok, nil} | {:error, term()}
@doc """
Returns a single Google Analytics view if the user has access to it.
"""
def get_view(access_token, lookup_id) do
case list_views(access_token) do
{:ok, views} ->
view =
views
|> Map.values()
|> List.flatten()
|> Enum.find(fn {_name, id} -> id == lookup_id end)
{:ok, view}
{:error, cause} ->
{:error, cause}
end
end
@type import_auth :: {
access_token :: String.t(),
refresh_token :: String.t(),
expires_at :: String.t()
}
@per_page 7_500
@backoff_factor :timer.seconds(10)
@max_attempts 5
@spec import_analytics(Date.Range.t(), String.t(), import_auth(), (String.t(), [map()] -> :ok)) ::
:ok | {:error, term()}
@doc """
Imports stats from a Google Analytics UA view to a Plausible site.
This function fetches Google Analytics reports in batches of #{@per_page} per
request. The batches are then passed to persist callback.
Requests to Google Analytics can fail, and are retried at most
#{@max_attempts} times with an exponential backoff. Returns `:ok` when
importing has finished or `{:error, term()}` when a request to GA failed too
many times.
Useful links:
- [Feature documentation](https://plausible.io/docs/google-analytics-import)
- [GA API reference](https://developers.google.com/analytics/devguides/reporting/core/v4/rest/v4/reports/batchGet#ReportRequest)
- [GA Dimensions reference](https://ga-dev-tools.web.app/dimensions-metrics-explorer)
"""
def import_analytics(date_range, view_id, auth, persist_fn) do
with {:ok, access_token} <- maybe_refresh_token(auth) do
do_import_analytics(date_range, view_id, access_token, persist_fn)
end
end
defp do_import_analytics(date_range, view_id, access_token, persist_fn) do
Enum.reduce_while(ReportRequest.full_report(), :ok, fn report_request, :ok ->
report_request = %ReportRequest{
report_request
| date_range: date_range,
view_id: view_id,
access_token: access_token,
page_token: nil,
page_size: @per_page
}
case fetch_and_persist(report_request, persist_fn: persist_fn) do
:ok -> {:cont, :ok}
{:error, _} = error -> {:halt, error}
end
end)
end
@spec fetch_and_persist(ReportRequest.t(), Keyword.t()) ::
:ok | {:error, term()}
def fetch_and_persist(%ReportRequest{} = report_request, opts \\ []) do
persist_fn = Keyword.fetch!(opts, :persist_fn)
attempt = Keyword.get(opts, :attempt, 1)
sleep_time = Keyword.get(opts, :sleep_time, @backoff_factor)
case HTTP.get_report(report_request) do
{:ok, {rows, next_page_token}} ->
:ok = persist_fn.(report_request.dataset, rows)
if next_page_token do
fetch_and_persist(
%ReportRequest{report_request | page_token: next_page_token},
opts
)
else
:ok
end
{:error, cause} ->
if attempt >= @max_attempts do
{:error, cause}
else
Process.sleep(attempt * sleep_time)
fetch_and_persist(report_request, Keyword.merge(opts, attempt: attempt + 1))
end
end
end
defp maybe_refresh_token(%Plausible.Site.GoogleAuth{} = auth) do
with true <- needs_to_refresh_token?(auth.expires), with true <- needs_to_refresh_token?(auth.expires),
{:ok, {new_access_token, expires_at}} <- do_refresh_token(auth.refresh_token), {:ok, {new_access_token, expires_at}} <- do_refresh_token(auth.refresh_token),
changeset <- changeset <-
@ -200,7 +110,7 @@ defmodule Plausible.Google.Api do
end end
end end
defp maybe_refresh_token({access_token, refresh_token, expires_at}) do def maybe_refresh_token({access_token, refresh_token, expires_at}) do
with true <- needs_to_refresh_token?(expires_at), with true <- needs_to_refresh_token?(expires_at),
{:ok, {new_access_token, _expires_at}} <- do_refresh_token(refresh_token) do {:ok, {new_access_token, _expires_at}} <- do_refresh_token(refresh_token) do
{:ok, new_access_token} {:ok, new_access_token}
@ -210,6 +120,8 @@ defmodule Plausible.Google.Api do
end end
end end
def property?(value), do: String.starts_with?(value, "properties/")
defp do_refresh_token(refresh_token) do defp do_refresh_token(refresh_token) do
case HTTP.refresh_auth_token(refresh_token) do case HTTP.refresh_auth_token(refresh_token) do
{:ok, %{"access_token" => new_access_token, "expires_in" => expires_in}} -> {:ok, %{"access_token" => new_access_token, "expires_in" => expires_in}} ->

@ -0,0 +1,152 @@
defmodule Plausible.Google.GA4.API do
@moduledoc """
API for Google Analytics 4.
"""
alias Plausible.Google
alias Plausible.Google.GA4
require Logger
@type import_auth :: {
access_token :: String.t(),
refresh_token :: String.t(),
expires_at :: String.t()
}
@per_page 50_000
@backoff_factor :timer.seconds(10)
@max_attempts 5
def list_properties(access_token) do
case GA4.HTTP.list_accounts_for_user(access_token) do
{:ok, %{"accountSummaries" => accounts}} ->
accounts =
accounts
|> Enum.filter(& &1["propertySummaries"])
|> Enum.map(fn account ->
%{"account" => account_id, "displayName" => account_name} = account
{"#{account_name} (#{account_id})",
Enum.map(account["propertySummaries"], fn property ->
%{"displayName" => property_name, "property" => property_id} = property
{"#{property_name} (#{property_id})", property_id}
end)}
end)
{:ok, accounts}
{:error, cause} ->
{:error, cause}
end
end
def get_property(access_token, lookup_property) do
case GA4.HTTP.get_property(access_token, lookup_property) do
{:ok, property} ->
%{"displayName" => property_name, "name" => property_id, "account" => account_id} =
property
{:ok,
%{
id: property_id,
name: "#{property_name} (#{property_id})",
account_id: account_id
}}
{:error, cause} ->
{:error, cause}
end
end
def get_analytics_start_date(access_token, property) do
GA4.HTTP.get_analytics_start_date(access_token, property)
end
def get_analytics_end_date(access_token, property) do
GA4.HTTP.get_analytics_end_date(access_token, property)
end
def import_analytics(date_range, property, auth, persist_fn) do
Logger.debug(
"[#{inspect(__MODULE__)}:#{property}] Starting import from #{date_range.first} to #{date_range.last}"
)
with {:ok, access_token} <- Google.API.maybe_refresh_token(auth) do
do_import_analytics(date_range, property, access_token, persist_fn)
end
end
defp do_import_analytics(date_range, property, access_token, persist_fn) do
Enum.reduce_while(GA4.ReportRequest.full_report(), :ok, fn report_request, :ok ->
Logger.debug(
"[#{inspect(__MODULE__)}:#{property}] Starting to import #{report_request.dataset}"
)
report_request = prepare_request(report_request, date_range, property, access_token)
case fetch_and_persist(report_request, persist_fn: persist_fn) do
:ok -> {:cont, :ok}
{:error, _} = error -> {:halt, error}
end
end)
end
@spec fetch_and_persist(GA4.ReportRequest.t(), Keyword.t()) ::
:ok | {:error, term()}
def fetch_and_persist(%GA4.ReportRequest{} = report_request, opts \\ []) do
persist_fn = Keyword.fetch!(opts, :persist_fn)
attempt = Keyword.get(opts, :attempt, 1)
sleep_time = Keyword.get(opts, :sleep_time, @backoff_factor)
case GA4.HTTP.get_report(report_request) do
{:ok, {rows, row_count}} ->
Logger.debug(
"[#{inspect(__MODULE__)}:#{report_request.property}] Fetched #{length(rows)} rows of total #{row_count} with offset #{report_request.offset} for #{report_request.dataset}"
)
:ok = persist_fn.(report_request.dataset, rows)
Logger.debug(
"[#{inspect(__MODULE__)}:#{report_request.property}] Persisted #{length(rows)} for #{report_request.dataset}"
)
if report_request.offset + @per_page < row_count do
fetch_and_persist(
%GA4.ReportRequest{report_request | offset: report_request.offset + @per_page},
opts
)
else
:ok
end
{:error, cause} ->
if attempt >= @max_attempts do
Logger.debug(
"[#{inspect(__MODULE__)}:#{report_request.property}] Request failed for #{report_request.dataset}. Terminating."
)
{:error, cause}
else
Logger.debug(
"[#{inspect(__MODULE__)}:#{report_request.property}] Request failed for #{report_request.dataset}. Will retry."
)
Process.sleep(attempt * sleep_time)
fetch_and_persist(report_request, Keyword.merge(opts, attempt: attempt + 1))
end
end
end
defp prepare_request(report_request, date_range, property, access_token) do
%GA4.ReportRequest{
report_request
| date_range: date_range,
property: property,
access_token: access_token,
offset: 0,
limit: @per_page
}
end
end

@ -0,0 +1,244 @@
defmodule Plausible.Google.GA4.HTTP do
@moduledoc """
HTTP client implementation for Google Analytics 4 API.
"""
alias Plausible.HTTPClient
require Logger
@spec get_report(Plausible.Google.GA4.ReportRequest.t()) ::
{:ok, {[map()], non_neg_integer()}} | {:error, any()}
def get_report(%Plausible.Google.GA4.ReportRequest{} = report_request) do
params = %{
requests: [
%{
property: report_request.property,
dateRanges: [
%{
startDate: report_request.date_range.first,
endDate: report_request.date_range.last
}
],
dimensions: Enum.map(report_request.dimensions, &%{name: &1}),
metrics: Enum.map(report_request.metrics, &build_metric/1),
orderBys: [
%{
dimension: %{
dimensionName: "date",
orderType: "ALPHANUMERIC"
},
desc: true
}
],
limit: report_request.limit,
offset: report_request.offset
}
]
}
url =
"#{reporting_api_url()}/v1beta/#{report_request.property}:batchRunReports"
response =
HTTPClient.impl().post(
url,
[{"Authorization", "Bearer #{report_request.access_token}"}],
params,
receive_timeout: 60_000
)
with {:ok, %{body: body}} <- response,
{:ok, report} <- parse_report_from_response(body),
row_count <- Map.fetch!(report, "rowCount"),
{:ok, report} <- convert_to_maps(report) do
{:ok, {report, row_count}}
else
{:error, %{reason: %{status: status, body: body}}} ->
Logger.debug(
"[#{inspect(__MODULE__)}:#{report_request.property}] Request failed for #{report_request.dataset} with code #{status}: #{inspect(body)}"
)
Sentry.Context.set_extra_context(%{ga_response: %{body: body, status: status}})
{:error, :request_failed}
{:error, reason} ->
Logger.debug(
"[#{inspect(__MODULE__)}:#{report_request.property}] Request failed for #{report_request.dataset}: #{inspect(reason)}"
)
{:error, :request_failed}
end
end
defp build_metric(expression) do
case String.split(expression, " = ") do
[name, expression] ->
%{
name: name,
expression: expression
}
[name] ->
%{name: name}
end
end
defp parse_report_from_response(%{"reports" => [report | _]}) do
{:ok, report}
end
defp parse_report_from_response(body) do
Sentry.Context.set_extra_context(%{google_analytics4_response: body})
Logger.error(
"Google Analytics 4: Failed to find report in response. Reason: #{inspect(body)}"
)
{:error, {:invalid_response, body}}
end
defp convert_to_maps(%{
"rows" => rows,
"dimensionHeaders" => dimension_headers,
"metricHeaders" => metric_headers
})
when is_list(rows) do
dimension_headers = Enum.map(dimension_headers, & &1["name"])
metric_headers = Enum.map(metric_headers, & &1["name"])
report =
Enum.map(rows, fn %{"dimensionValues" => dimensions, "metricValues" => metrics} ->
dimension_values = Enum.map(dimensions, & &1["value"])
metric_values = Enum.map(metrics, & &1["value"])
metrics = Enum.zip(metric_headers, metric_values)
dimensions = Enum.zip(dimension_headers, dimension_values)
%{metrics: Map.new(metrics), dimensions: Map.new(dimensions)}
end)
{:ok, report}
end
defp convert_to_maps(response) do
Logger.error(
"Google Analytics 4: Failed to read report in response. Reason: #{inspect(response)}"
)
Sentry.Context.set_extra_context(%{google_analytics4_response: response})
{:error, {:invalid_response, response}}
end
def list_accounts_for_user(access_token) do
url = "#{admin_api_url()}/v1beta/accountSummaries?pageSize=200"
headers = [{"Authorization", "Bearer #{access_token}"}]
case HTTPClient.impl().get(url, headers) do
{:ok, %Finch.Response{body: body, status: 200}} ->
{:ok, body}
{:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [401, 403] ->
{:error, :authentication_failed}
{:error, %{reason: :timeout}} ->
{:error, :timeout}
{:error, error} ->
Sentry.capture_message("Error listing GA4 accounts for user", extra: %{error: error})
{:error, :unknown}
end
end
def get_property(access_token, property) do
url = "#{admin_api_url()}/v1beta/#{property}"
headers = [{"Authorization", "Bearer #{access_token}"}]
case HTTPClient.impl().get(url, headers) do
{:ok, %Finch.Response{body: body, status: 200}} ->
{:ok, body}
{:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [401, 403] ->
{:error, :authentication_failed}
{:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [404] ->
{:error, :not_found}
{:error, %{reason: :timeout}} ->
{:error, :timeout}
{:error, error} ->
Sentry.capture_message("Error retrieving GA4 property #{property}",
extra: %{error: error}
)
{:error, :unknown}
end
end
@earliest_valid_date "2015-08-14"
def get_analytics_start_date(access_token, property) do
get_analytics_boundary_date(access_token, property, :start)
end
def get_analytics_end_date(access_token, property) do
get_analytics_boundary_date(access_token, property, :end)
end
defp get_analytics_boundary_date(access_token, property, edge) do
descending? = edge == :end
params = %{
requests: [
%{
property: "#{property}",
dateRanges: [
%{startDate: @earliest_valid_date, endDate: Date.to_iso8601(Timex.today())}
],
dimensions: [%{name: "date"}],
metrics: [%{name: "screenPageViews"}],
orderBys: [
%{dimension: %{dimensionName: "date", orderType: "ALPHANUMERIC"}, desc: descending?}
],
limit: 1
}
]
}
url = "#{reporting_api_url()}/v1beta/#{property}:batchRunReports"
headers = [{"Authorization", "Bearer #{access_token}"}]
case HTTPClient.impl().post(url, headers, params) do
{:ok, %Finch.Response{body: body, status: 200}} ->
report = List.first(body["reports"])
date =
case report["rows"] do
[%{"dimensionValues" => [%{"value" => date_str}]}] ->
Timex.parse!(date_str, "%Y%m%d", :strftime) |> NaiveDateTime.to_date()
_ ->
nil
end
{:ok, date}
{:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [401, 403] ->
{:error, :authentication_failed}
{:error, %{reason: :timeout}} ->
{:error, :timeout}
{:error, error} ->
Sentry.capture_message("Error retrieving GA4 #{edge} date",
extra: %{error: error}
)
{:error, :unknown}
end
end
defp reporting_api_url, do: "https://analyticsdata.googleapis.com"
defp admin_api_url, do: "https://analyticsadmin.googleapis.com"
end

@ -0,0 +1,135 @@
defmodule Plausible.Google.GA4.ReportRequest do
@moduledoc """
Report request struct for Google Analytics 4 API
"""
defstruct [
:dataset,
:dimensions,
:metrics,
:date_range,
:property,
:access_token,
:offset,
:limit
]
@type t() :: %__MODULE__{
dataset: String.t(),
dimensions: [String.t()],
metrics: [String.t()],
date_range: Date.Range.t(),
property: term(),
access_token: String.t(),
offset: non_neg_integer(),
limit: non_neg_integer()
}
def full_report do
[
%__MODULE__{
dataset: "imported_visitors",
dimensions: ["date"],
metrics: [
"totalUsers",
"screenPageViews",
"bounces = sessions - engagedSessions",
"sessions",
"userEngagementDuration"
]
},
%__MODULE__{
dataset: "imported_sources",
dimensions: [
"date",
"sessionSource",
"sessionMedium",
"sessionCampaignName",
"sessionManualAdContent",
"sessionGoogleAdsKeyword",
"pageReferrer"
],
metrics: [
"screenPageViews",
"totalUsers",
"sessions",
"bounces = sessions - engagedSessions",
"userEngagementDuration"
]
},
%__MODULE__{
dataset: "imported_pages",
dimensions: ["date", "hostName", "pagePath"],
# NOTE: no exits as GA4 DATA API does not provide that metric
metrics: ["totalUsers", "screenPageViews", "sessions", "userEngagementDuration"]
},
%__MODULE__{
dataset: "imported_entry_pages",
dimensions: ["date", "landingPage"],
metrics: [
"screenPageViews",
"totalUsers",
"sessions",
"userEngagementDuration",
"bounces = sessions - engagedSessions"
]
},
# NOTE: Skipping for now as there's no dimension directly mapping to exit page path
# %__MODULE__{
# dataset: "imported_exit_pages",
# dimensions: ["date", "ga:exitPagePath"],
# metrics: [
# "totalUsers",
# "sessions",
# "screenPageViews",
# "userEngagementDuration",
# "bounces = sessions - engagedSessions"
# ]
# },
%__MODULE__{
dataset: "imported_locations",
dimensions: ["date", "countryId", "region", "city"],
metrics: [
"screenPageViews",
"totalUsers",
"sessions",
"bounces = sessions - engagedSessions",
"userEngagementDuration"
]
},
%__MODULE__{
dataset: "imported_devices",
dimensions: ["date", "deviceCategory"],
metrics: [
"screenPageViews",
"totalUsers",
"sessions",
"bounces = sessions - engagedSessions",
"userEngagementDuration"
]
},
%__MODULE__{
dataset: "imported_browsers",
dimensions: ["date", "browser"],
metrics: [
"screenPageViews",
"totalUsers",
"sessions",
"bounces = sessions - engagedSessions",
"userEngagementDuration"
]
},
%__MODULE__{
dataset: "imported_operating_systems",
dimensions: ["date", "operatingSystem", "operatingSystemVersion"],
metrics: [
"screenPageViews",
"totalUsers",
"sessions",
"bounces = sessions - engagedSessions",
"userEngagementDuration"
]
}
]
end
end

@ -2,97 +2,6 @@ defmodule Plausible.Google.HTTP do
require Logger require Logger
alias Plausible.HTTPClient alias Plausible.HTTPClient
@spec get_report(Plausible.Google.ReportRequest.t()) ::
{:ok, {[map()], String.t() | nil}} | {:error, any()}
def get_report(%Plausible.Google.ReportRequest{} = report_request) do
params = %{
reportRequests: [
%{
viewId: report_request.view_id,
dateRanges: [
%{
startDate: report_request.date_range.first,
endDate: report_request.date_range.last
}
],
dimensions: Enum.map(report_request.dimensions, &%{name: &1, histogramBuckets: []}),
metrics: Enum.map(report_request.metrics, &%{expression: &1}),
hideTotals: true,
hideValueRanges: true,
orderBys: [%{fieldName: "ga:date", sortOrder: "DESCENDING"}],
pageSize: report_request.page_size,
pageToken: report_request.page_token
}
]
}
response =
HTTPClient.impl().post(
"#{reporting_api_url()}/v4/reports:batchGet",
[{"Authorization", "Bearer #{report_request.access_token}"}],
params,
receive_timeout: 60_000
)
with {:ok, %{body: body}} <- response,
{:ok, report} <- parse_report_from_response(body),
token <- Map.get(report, "nextPageToken"),
{:ok, report} <- convert_to_maps(report) do
{:ok, {report, token}}
else
{:error, %{reason: %{status: status, body: body}}} ->
Sentry.Context.set_extra_context(%{ga_response: %{body: body, status: status}})
{:error, :request_failed}
{:error, _reason} ->
{:error, :request_failed}
end
end
defp parse_report_from_response(body) do
with %{"reports" => [report | _]} <- body do
{:ok, report}
else
_ ->
Sentry.Context.set_extra_context(%{google_analytics_response: body})
Logger.error(
"Google Analytics: Failed to find report in response. Reason: #{inspect(body)}"
)
{:error, {:invalid_response, body}}
end
end
defp convert_to_maps(%{
"data" => %{} = data,
"columnHeader" => %{
"dimensions" => dimension_headers,
"metricHeader" => %{"metricHeaderEntries" => metric_headers}
}
}) do
metric_headers = Enum.map(metric_headers, & &1["name"])
rows = Map.get(data, "rows", [])
report =
Enum.map(rows, fn %{"dimensions" => dimensions, "metrics" => [%{"values" => metrics}]} ->
metrics = Enum.zip(metric_headers, metrics)
dimensions = Enum.zip(dimension_headers, dimensions)
%{metrics: Map.new(metrics), dimensions: Map.new(dimensions)}
end)
{:ok, report}
end
defp convert_to_maps(response) do
Logger.error(
"Google Analytics: Failed to read report in response. Reason: #{inspect(response)}"
)
Sentry.Context.set_extra_context(%{google_analytics_response: response})
{:error, {:invalid_response, response}}
end
def list_sites(access_token) do def list_sites(access_token) do
url = "#{api_url()}/webmasters/v3/sites" url = "#{api_url()}/webmasters/v3/sites"
headers = [{"Content-Type", "application/json"}, {"Authorization", "Bearer #{access_token}"}] headers = [{"Content-Type", "application/json"}, {"Authorization", "Bearer #{access_token}"}]
@ -113,7 +22,7 @@ defmodule Plausible.Google.HTTP do
end end
end end
def fetch_access_token(code) do def fetch_access_token!(code) do
url = "#{api_url()}/oauth2/v4/token" url = "#{api_url()}/oauth2/v4/token"
headers = [{"Content-Type", "application/x-www-form-urlencoded"}] headers = [{"Content-Type", "application/x-www-form-urlencoded"}]
@ -130,24 +39,6 @@ defmodule Plausible.Google.HTTP do
response.body response.body
end end
def list_views_for_user(access_token) do
url = "#{api_url()}/analytics/v3/management/accounts/~all/webproperties/~all/profiles"
headers = [{"Authorization", "Bearer #{access_token}"}]
case HTTPClient.impl().get(url, headers) do
{:ok, %Finch.Response{body: body, status: 200}} ->
{:ok, body}
{:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [401, 403] ->
{:error, :authentication_failed}
{:error, %HTTPClient.Non200Error{} = error} ->
Sentry.capture_message("Error listing GA views for user", extra: %{error: error})
{:error, :unknown}
end
end
def list_stats(access_token, property, date_range, limit, page \\ nil) do def list_stats(access_token, property, date_range, limit, page \\ nil) do
property = URI.encode_www_form(property) property = URI.encode_www_form(property)
@ -215,57 +106,9 @@ defmodule Plausible.Google.HTTP do
end end
end end
@earliest_valid_date "2005-01-01"
def get_analytics_start_date(view_id, access_token) do
params = %{
reportRequests: [
%{
viewId: view_id,
dateRanges: [
%{startDate: @earliest_valid_date, endDate: Date.to_iso8601(Timex.today())}
],
dimensions: [%{name: "ga:date", histogramBuckets: []}],
metrics: [%{expression: "ga:pageviews"}],
hideTotals: true,
hideValueRanges: true,
orderBys: [%{fieldName: "ga:date", sortOrder: "ASCENDING"}],
pageSize: 1
}
]
}
url = "#{reporting_api_url()}/v4/reports:batchGet"
headers = [{"Authorization", "Bearer #{access_token}"}]
case HTTPClient.post(url, headers, params) do
{:ok, %Finch.Response{body: body, status: 200}} ->
report = List.first(body["reports"])
date =
case report["data"]["rows"] do
[%{"dimensions" => [date_str]}] ->
Timex.parse!(date_str, "%Y%m%d", :strftime) |> NaiveDateTime.to_date()
_ ->
nil
end
{:ok, date}
{:error, %{reason: %Finch.Response{body: body}}} ->
Sentry.capture_message("Error fetching Google view ID", extra: %{body: inspect(body)})
{:error, body}
{:error, %{reason: reason} = e} ->
Sentry.capture_message("Error fetching Google view ID", extra: %{error: inspect(e)})
{:error, reason}
end
end
defp config, do: Application.get_env(:plausible, :google) defp config, do: Application.get_env(:plausible, :google)
defp client_id, do: Keyword.fetch!(config(), :client_id) defp client_id, do: Keyword.fetch!(config(), :client_id)
defp client_secret, do: Keyword.fetch!(config(), :client_secret) defp client_secret, do: Keyword.fetch!(config(), :client_secret)
defp reporting_api_url, do: Keyword.fetch!(config(), :reporting_api_url)
defp api_url, do: Keyword.fetch!(config(), :api_url) defp api_url, do: Keyword.fetch!(config(), :api_url)
defp redirect_uri, do: PlausibleWeb.Endpoint.url() <> "/auth/google/callback" defp redirect_uri, do: PlausibleWeb.Endpoint.url() <> "/auth/google/callback"
end end

@ -0,0 +1,155 @@
defmodule Plausible.Google.UA.API do
@moduledoc """
API for Universal Analytics
"""
alias Plausible.Google
alias Plausible.Google.UA
@type google_analytics_view() :: {view_name :: String.t(), view_id :: String.t()}
@type import_auth :: {
access_token :: String.t(),
refresh_token :: String.t(),
expires_at :: String.t()
}
@per_page 7_500
@backoff_factor :timer.seconds(10)
@max_attempts 5
@spec list_views(access_token :: String.t()) ::
{:ok, %{(hostname :: String.t()) => [google_analytics_view()]}} | {:error, term()}
@doc """
Lists Google Analytics views grouped by hostname.
"""
def list_views(access_token) do
case UA.HTTP.list_views_for_user(access_token) do
{:ok, %{"items" => views}} ->
views =
views
|> Enum.group_by(&view_hostname/1, &view_names/1)
|> Enum.sort_by(fn {key, _} -> key end)
{:ok, views}
error ->
error
end
end
@spec get_view(access_token :: String.t(), lookup_id :: String.t()) ::
{:ok, google_analytics_view()} | {:ok, nil} | {:error, term()}
@doc """
Returns a single Google Analytics view if the user has access to it.
"""
def get_view(access_token, lookup_id) do
with {:ok, views} <- list_views(access_token) do
views =
views
|> Enum.map(&elem(&1, 1))
|> List.flatten()
case Enum.find(views, fn {_name, id} -> id == lookup_id end) do
{view_name, view_id} ->
{:ok, %{id: view_id, name: "#{view_name}"}}
nil ->
{:error, :not_found}
end
end
end
def get_analytics_start_date(access_token, view_id) do
UA.HTTP.get_analytics_start_date(access_token, view_id)
end
def get_analytics_end_date(access_token, view_id) do
UA.HTTP.get_analytics_end_date(access_token, view_id)
end
@spec import_analytics(Date.Range.t(), String.t(), import_auth(), (String.t(), [map()] -> :ok)) ::
:ok | {:error, term()}
@doc """
Imports stats from a Google Analytics UA view to a Plausible site.
This function fetches Google Analytics reports in batches of #{@per_page} per
request. The batches are then passed to persist callback.
Requests to Google Analytics can fail, and are retried at most
#{@max_attempts} times with an exponential backoff. Returns `:ok` when
importing has finished or `{:error, term()}` when a request to GA failed too
many times.
Useful links:
- [Feature documentation](https://plausible.io/docs/google-analytics-import)
- [GA API reference](https://developers.google.com/analytics/devguides/reporting/core/v4/rest/v4/reports/batchGet#ReportRequest)
- [GA Dimensions reference](https://ga-dev-tools.web.app/dimensions-metrics-explorer)
"""
def import_analytics(date_range, view_id, auth, persist_fn) do
with {:ok, access_token} <- Google.API.maybe_refresh_token(auth) do
do_import_analytics(date_range, view_id, access_token, persist_fn)
end
end
defp do_import_analytics(date_range, view_id, access_token, persist_fn) do
Enum.reduce_while(UA.ReportRequest.full_report(), :ok, fn report_request, :ok ->
report_request = %UA.ReportRequest{
report_request
| date_range: date_range,
view_id: view_id,
access_token: access_token,
page_token: nil,
page_size: @per_page
}
case fetch_and_persist(report_request, persist_fn: persist_fn) do
:ok -> {:cont, :ok}
{:error, _} = error -> {:halt, error}
end
end)
end
@spec fetch_and_persist(UA.ReportRequest.t(), Keyword.t()) ::
:ok | {:error, term()}
def fetch_and_persist(%UA.ReportRequest{} = report_request, opts \\ []) do
persist_fn = Keyword.fetch!(opts, :persist_fn)
attempt = Keyword.get(opts, :attempt, 1)
sleep_time = Keyword.get(opts, :sleep_time, @backoff_factor)
case UA.HTTP.get_report(report_request) do
{:ok, {rows, next_page_token}} ->
:ok = persist_fn.(report_request.dataset, rows)
if next_page_token do
fetch_and_persist(
%UA.ReportRequest{report_request | page_token: next_page_token},
opts
)
else
:ok
end
{:error, cause} ->
if attempt >= @max_attempts do
{:error, cause}
else
Process.sleep(attempt * sleep_time)
fetch_and_persist(report_request, Keyword.merge(opts, attempt: attempt + 1))
end
end
end
defp view_hostname(view) do
case view do
%{"websiteUrl" => url} when is_binary(url) -> url |> URI.parse() |> Map.get(:host)
_any -> "Others"
end
end
defp view_names(%{"name" => name, "id" => id}) do
{"#{id} - #{name}", id}
end
end

@ -0,0 +1,191 @@
defmodule Plausible.Google.UA.HTTP do
@moduledoc """
HTTP client implementation for Universal Analytics API.
"""
require Logger
alias Plausible.HTTPClient
@spec get_report(Plausible.Google.UA.ReportRequest.t()) ::
{:ok, {[map()], String.t() | nil}} | {:error, any()}
def get_report(%Plausible.Google.UA.ReportRequest{} = report_request) do
params = %{
reportRequests: [
%{
viewId: report_request.view_id,
dateRanges: [
%{
startDate: report_request.date_range.first,
endDate: report_request.date_range.last
}
],
dimensions: Enum.map(report_request.dimensions, &%{name: &1, histogramBuckets: []}),
metrics: Enum.map(report_request.metrics, &%{expression: &1}),
hideTotals: true,
hideValueRanges: true,
orderBys: [%{fieldName: "ga:date", sortOrder: "DESCENDING"}],
pageSize: report_request.page_size,
pageToken: report_request.page_token
}
]
}
response =
HTTPClient.impl().post(
"#{reporting_api_url()}/v4/reports:batchGet",
[{"Authorization", "Bearer #{report_request.access_token}"}],
params,
receive_timeout: 60_000
)
with {:ok, %{body: body}} <- response,
{:ok, report} <- parse_report_from_response(body),
token <- Map.get(report, "nextPageToken"),
{:ok, report} <- convert_to_maps(report) do
{:ok, {report, token}}
else
{:error, %{reason: %{status: status, body: body}}} ->
Sentry.Context.set_extra_context(%{ga_response: %{body: body, status: status}})
{:error, :request_failed}
{:error, _reason} ->
{:error, :request_failed}
end
end
defp parse_report_from_response(%{"reports" => [report | _]}) do
{:ok, report}
end
defp parse_report_from_response(body) do
Sentry.Context.set_extra_context(%{universal_analytics_response: body})
Logger.error(
"Universal Analytics: Failed to find report in response. Reason: #{inspect(body)}"
)
{:error, {:invalid_response, body}}
end
defp convert_to_maps(%{
"data" => %{} = data,
"columnHeader" => %{
"dimensions" => dimension_headers,
"metricHeader" => %{"metricHeaderEntries" => metric_headers}
}
}) do
metric_headers = Enum.map(metric_headers, & &1["name"])
rows = Map.get(data, "rows", [])
report =
Enum.map(rows, fn %{"dimensions" => dimensions, "metrics" => [%{"values" => metrics}]} ->
metrics = Enum.zip(metric_headers, metrics)
dimensions = Enum.zip(dimension_headers, dimensions)
%{metrics: Map.new(metrics), dimensions: Map.new(dimensions)}
end)
{:ok, report}
end
defp convert_to_maps(response) do
Logger.error(
"Universal Analytics: Failed to read report in response. Reason: #{inspect(response)}"
)
Sentry.Context.set_extra_context(%{universal_analytics_response: response})
{:error, {:invalid_response, response}}
end
def list_views_for_user(access_token) do
url = "#{api_url()}/analytics/v3/management/accounts/~all/webproperties/~all/profiles"
headers = [{"Authorization", "Bearer #{access_token}"}]
case HTTPClient.impl().get(url, headers) do
{:ok, %Finch.Response{body: body, status: 200}} ->
{:ok, body}
{:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [401, 403] ->
{:error, :authentication_failed}
{:error, %{reason: :timeout}} ->
{:error, :timeout}
{:error, error} ->
Sentry.capture_message("Error listing UA views for user", extra: %{error: error})
{:error, :unknown}
end
end
@earliest_valid_date "2005-01-01"
def get_analytics_start_date(access_token, view_id) do
get_analytics_boundary_date(access_token, view_id, :start)
end
def get_analytics_end_date(access_token, view_id) do
get_analytics_boundary_date(access_token, view_id, :end)
end
defp get_analytics_boundary_date(access_token, view_id, edge) do
sort_order =
if edge == :start do
"ASCENDING"
else
"DESCENDING"
end
params = %{
reportRequests: [
%{
viewId: view_id,
dateRanges: [
%{startDate: @earliest_valid_date, endDate: Date.to_iso8601(Timex.today())}
],
dimensions: [%{name: "ga:date", histogramBuckets: []}],
metrics: [%{expression: "ga:pageviews"}],
hideTotals: true,
hideValueRanges: true,
orderBys: [%{fieldName: "ga:date", sortOrder: sort_order}],
pageSize: 1
}
]
}
url = "#{reporting_api_url()}/v4/reports:batchGet"
headers = [{"Authorization", "Bearer #{access_token}"}]
case HTTPClient.impl().post(url, headers, params) do
{:ok, %Finch.Response{body: body, status: 200}} ->
report = List.first(body["reports"])
date =
case report["data"]["rows"] do
[%{"dimensions" => [date_str]}] ->
Timex.parse!(date_str, "%Y%m%d", :strftime) |> NaiveDateTime.to_date()
_ ->
nil
end
{:ok, date}
{:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [401, 403] ->
{:error, :authentication_failed}
{:error, %{reason: :timeout}} ->
{:error, :timeout}
{:error, error} ->
Sentry.capture_message("Error retrieving UA #{edge} date",
extra: %{error: error}
)
{:error, :unknown}
end
end
defp config, do: Application.get_env(:plausible, :google)
defp reporting_api_url, do: Keyword.fetch!(config(), :reporting_api_url)
defp api_url, do: Keyword.fetch!(config(), :api_url)
end

@ -1,4 +1,8 @@
defmodule Plausible.Google.ReportRequest do defmodule Plausible.Google.UA.ReportRequest do
@moduledoc """
Report request struct for Universal Analytics API
"""
defstruct [ defstruct [
:dataset, :dataset,
:dimensions, :dimensions,

@ -7,8 +7,7 @@ defmodule Plausible.Imported do
* `Plausible.Imported.UniversalAnalytics` - existing mechanism, for legacy Google * `Plausible.Imported.UniversalAnalytics` - existing mechanism, for legacy Google
analytics formerly known as "Google Analytics" analytics formerly known as "Google Analytics"
* `Plausible.Imported.NoopImporter` - importer stub, used mainly for testing purposes * `Plausible.Imported.NoopImporter` - importer stub, used mainly for testing purposes
* `Plausible.Imported.CSVImporter` - a placeholder stub for CSV importer that will * `Plausible.Imported.CSVImporter` - CSV importer from S3
be added soon
For more information on implementing importers, see `Plausible.Imported.Importer`. For more information on implementing importers, see `Plausible.Imported.Importer`.
""" """
@ -38,29 +37,66 @@ defmodule Plausible.Imported do
# Maximum number of complete imports to account for when querying stats # Maximum number of complete imports to account for when querying stats
@max_complete_imports 5 @max_complete_imports 5
@spec schemas() :: [module()]
def schemas, do: @tables
@spec tables() :: [String.t()] @spec tables() :: [String.t()]
def tables, do: @table_names def tables, do: @table_names
@spec max_complete_imports() :: non_neg_integer()
def max_complete_imports(), do: @max_complete_imports
@spec load_import_data(Site.t()) :: Site.t() @spec load_import_data(Site.t()) :: Site.t()
def load_import_data(%{import_data_loaded: true} = site), do: site def load_import_data(%{import_data_loaded: true} = site), do: site
def load_import_data(site) do def load_import_data(site) do
complete_import_ids = list_complete_import_ids(site) complete_import_ids = list_complete_import_ids(site)
earliest_import = get_earliest_import(site) || %{} dates = get_imports_date_range(site)
%{ %{
site site
| import_data_loaded: true, | import_data_loaded: true,
earliest_import_start_date: Map.get(earliest_import, :start_date), earliest_import_start_date: dates.start_date,
earliest_import_end_date: Map.get(earliest_import, :end_date), latest_import_end_date: dates.end_date,
complete_import_ids: complete_import_ids complete_import_ids: complete_import_ids
} }
end end
@spec list_all_imports(Site.t()) :: [SiteImport.t()] @spec get_import(Site.t(), non_neg_integer()) :: SiteImport.t() | nil
def list_all_imports(site) do def get_import(site, import_id) do
from(i in SiteImport, where: i.site_id == ^site.id) Repo.get_by(SiteImport, id: import_id, site_id: site.id)
|> Repo.all() end
defdelegate listen(), to: Imported.Importer
@spec list_all_imports(Site.t(), atom()) :: [SiteImport.t()]
def list_all_imports(site, status \\ nil) do
imports =
from(i in SiteImport, where: i.site_id == ^site.id, order_by: [desc: i.inserted_at])
|> maybe_filter_by_status(status)
|> Repo.all()
if site.imported_data && not Enum.any?(imports, & &1.legacy) do
imports ++ [SiteImport.from_legacy(site.imported_data)]
else
imports
end
end
@spec other_imports_in_progress?(SiteImport.t()) :: boolean()
def other_imports_in_progress?(site_import) do
Repo.exists?(
from(i in SiteImport,
where: i.site_id == ^site_import.site_id and i.id != ^site_import.id,
where: i.status in ^[SiteImport.pending(), SiteImport.importing()]
)
)
end
defp maybe_filter_by_status(query, nil), do: query
defp maybe_filter_by_status(query, status) do
where(query, [i], i.status == ^status)
end end
@spec list_complete_import_ids(Site.t()) :: [non_neg_integer()] @spec list_complete_import_ids(Site.t()) :: [non_neg_integer()]
@ -68,37 +104,51 @@ defmodule Plausible.Imported do
ids = ids =
from(i in SiteImport, from(i in SiteImport,
where: i.site_id == ^site.id and i.status == ^SiteImport.completed(), where: i.site_id == ^site.id and i.status == ^SiteImport.completed(),
select: i.id, select: {i.legacy, i.id},
limit: @max_complete_imports limit: @max_complete_imports
) )
|> Repo.all() |> Repo.all()
has_legacy? = Enum.any?(ids, fn {legacy?, _} -> legacy? end)
ids = Enum.map(ids, &elem(&1, 1))
# account for legacy imports as well # account for legacy imports as well
if site.imported_data && site.imported_data.status == "ok" do if has_legacy? || (site.imported_data && site.imported_data.status == "ok") do
[0 | ids] [0 | ids]
else else
ids ids
end end
end end
@spec get_earliest_import(Site.t()) :: SiteImport.t() | nil @spec get_imports_date_range(Site.t()) :: %{
def get_earliest_import(site) do start_date: Date.t() | nil,
first_import = end_date: Date.t() | nil
}
def get_imports_date_range(site) do
dates =
from(i in SiteImport, from(i in SiteImport,
where: i.site_id == ^site.id and i.status == ^SiteImport.completed(), where: i.site_id == ^site.id and i.status == ^SiteImport.completed(),
order_by: i.start_date, select: %{start_date: min(i.start_date), end_date: max(i.end_date)}
limit: 1
) )
|> Repo.one() |> Repo.one()
legacy_import = dates = dates || %{start_date: nil, end_date: nil}
if site.imported_data && site.imported_data.status == "ok" do
site.imported_data if site.imported_data && site.imported_data.status == "ok" do
end start_date =
[dates.start_date, site.imported_data.start_date]
|> Enum.reject(&is_nil/1)
|> Enum.min(Date, fn -> nil end)
end_date =
[dates.end_date, site.imported_data.end_date]
|> Enum.reject(&is_nil/1)
|> Enum.max(Date, fn -> nil end)
[legacy_import, first_import] %{start_date: start_date, end_date: end_date}
|> Enum.reject(&is_nil/1) else
|> Enum.min_by(& &1.start_date, Date, fn -> nil end) dates
end
end end
@spec delete_imports_for_site(Site.t()) :: :ok @spec delete_imports_for_site(Site.t()) :: :ok
@ -107,4 +157,66 @@ defmodule Plausible.Imported do
:ok :ok
end end
@spec check_dates(Site.t(), Date.t() | nil, Date.t() | nil) ::
{:ok, Date.t(), Date.t()} | {:error, :no_data | :no_time_window}
def check_dates(_site, nil, _end_date), do: {:error, :no_data}
def check_dates(site, start_date, end_date) do
cutoff_date = Plausible.Sites.native_stats_start_date(site) || Timex.today(site.timezone)
end_date = Enum.min([end_date, cutoff_date], Date)
with true <- Date.diff(end_date, start_date) >= 2,
[_ | _] = free_ranges <- find_free_ranges(start_date, end_date, site) do
longest = Enum.max_by(free_ranges, &Date.diff(&1.last, &1.first))
{:ok, longest.first, longest.last}
else
_ -> {:error, :no_time_window}
end
end
defp find_free_ranges(start_date, end_date, site) do
occupied_ranges =
site
|> Imported.list_all_imports(Imported.SiteImport.completed())
|> Enum.reject(&(Date.diff(&1.end_date, &1.start_date) < 2))
|> Enum.map(&Date.range(&1.start_date, &1.end_date))
Date.range(start_date, end_date)
|> free_ranges(start_date, occupied_ranges, [])
end
# This function recursively finds open ranges that are not yet occupied
# by existing imported data. The idea is that we keep moving a dynamic
# date index `d` from start until the end of `imported_range`, hopping
# over each occupied range, and capturing the open ranges step-by-step
# in the `result` array.
defp free_ranges(import_range, d, [occupied_range | rest_of_occupied_ranges], result) do
cond do
Date.diff(occupied_range.last, d) <= 0 ->
free_ranges(import_range, d, rest_of_occupied_ranges, result)
in_range?(d, occupied_range) || Date.diff(occupied_range.first, d) < 2 ->
d = occupied_range.last
free_ranges(import_range, d, rest_of_occupied_ranges, result)
true ->
free_range = Date.range(d, occupied_range.first)
result = result ++ [free_range]
d = occupied_range.last
free_ranges(import_range, d, rest_of_occupied_ranges, result)
end
end
defp free_ranges(import_range, d, [], result) do
if Date.diff(import_range.last, d) < 2 do
result
else
result ++ [Date.range(d, import_range.last)]
end
end
defp in_range?(date, range) do
Date.before?(range.first, date) && Date.after?(range.last, date)
end
end end

@ -8,9 +8,11 @@ defmodule Plausible.Imported.Browser do
field :import_id, Ch, type: "UInt64" field :import_id, Ch, type: "UInt64"
field :date, :date field :date, :date
field :browser, :string field :browser, :string
field :browser_version, :string
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :visits, Ch, type: "UInt64" field :visits, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64" field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32" field :bounces, Ch, type: "UInt32"
end end
end end

@ -8,12 +8,13 @@ defmodule Plausible.Imported.Buffer do
use GenServer use GenServer
require Logger require Logger
def start_link do def start_link(opts \\ []) do
GenServer.start_link(__MODULE__, nil) GenServer.start_link(__MODULE__, opts)
end end
def init(_opts) do def init(opts) do
{:ok, %{buffers: %{}}} flush_interval = Keyword.get(opts, :flush_interval_ms, 1000)
{:ok, %{flush_interval: flush_interval, buffers: %{}}}
end end
@spec insert_many(pid(), term(), [map()]) :: :ok @spec insert_many(pid(), term(), [map()]) :: :ok
@ -68,14 +69,14 @@ defmodule Plausible.Imported.Buffer do
def handle_call(:flush_all_buffers, _from, state) do def handle_call(:flush_all_buffers, _from, state) do
Enum.each(state.buffers, fn {table_name, records} -> Enum.each(state.buffers, fn {table_name, records} ->
flush_buffer(records, table_name) flush_buffer(records, table_name, state.flush_interval)
end) end)
{:reply, :ok, put_in(state.buffers, %{})} {:reply, :ok, put_in(state.buffers, %{})}
end end
def handle_continue({:flush, table_name}, state) do def handle_continue({:flush, table_name}, state) do
flush_buffer(state.buffers[table_name], table_name) flush_buffer(state.buffers[table_name], table_name, state.flush_interval)
{:noreply, put_in(state.buffers[table_name], [])} {:noreply, put_in(state.buffers[table_name], [])}
end end
@ -85,10 +86,10 @@ defmodule Plausible.Imported.Buffer do
|> Keyword.fetch!(:max_buffer_size) |> Keyword.fetch!(:max_buffer_size)
end end
defp flush_buffer(records, table_name) do defp flush_buffer(records, table_name, flush_interval) do
# Clickhouse does not recommend sending more than 1 INSERT operation per second, and this # Clickhouse does not recommend sending more than 1 INSERT operation per second, and this
# sleep call slows down the flushing # sleep call slows down the flushing
Process.sleep(1000) Process.sleep(flush_interval)
Logger.info("Import: Flushing #{length(records)} from #{table_name} buffer") Logger.info("Import: Flushing #{length(records)} from #{table_name} buffer")
insert_all(table_name, records) insert_all(table_name, records)

@ -1,6 +1,6 @@
defmodule Plausible.Imported.CSVImporter do defmodule Plausible.Imported.CSVImporter do
@moduledoc """ @moduledoc """
CSV importer stub. CSV importer from S3 that uses ClickHouse [s3 table function.](https://clickhouse.com/docs/en/sql-reference/table-functions/s3)
""" """
use Plausible.Imported.Importer use Plausible.Imported.Importer
@ -16,10 +16,214 @@ defmodule Plausible.Imported.CSVImporter do
def email_template(), do: "google_analytics_import.html" def email_template(), do: "google_analytics_import.html"
@impl true @impl true
def parse_args(%{"s3_path" => s3_path}), do: [s3_path: s3_path] def parse_args(%{"uploads" => uploads}), do: [uploads: uploads]
@impl true @impl true
def import_data(_site_import, _opts) do def import_data(site_import, opts) do
:ok %{
id: import_id,
site_id: site_id,
start_date: start_date,
end_date: end_date
} = site_import
uploads = Keyword.fetch!(opts, :uploads)
%{access_key_id: s3_access_key_id, secret_access_key: s3_secret_access_key} =
Plausible.S3.import_clickhouse_credentials()
{:ok, ch} =
Plausible.IngestRepo.config()
|> Keyword.replace!(:pool_size, 1)
|> Ch.start_link()
Enum.each(uploads, fn upload ->
%{"filename" => filename, "s3_url" => s3_url} = upload
{table, _, _} = parse_filename!(filename)
s3_structure = input_structure!(table)
s3_structure_cols_expr =
s3_structure
|> String.split(",", trim: true)
|> Enum.map_join(", ", fn kv ->
[col, _type] = String.split(kv)
col
end)
statement =
"""
INSERT INTO {table:Identifier}(site_id, #{s3_structure_cols_expr}, import_id) \
SELECT {site_id:UInt64} AS site_id, *, {import_id:UInt64} AS import_id \
FROM s3({s3_url:String},{s3_access_key_id:String},{s3_secret_access_key:String},{s3_format:String},{s3_structure:String}) \
WHERE date >= {start_date:Date} AND date <= {end_date:Date}\
"""
params =
%{
"table" => table,
"site_id" => site_id,
"import_id" => import_id,
"s3_url" => s3_url,
"s3_access_key_id" => s3_access_key_id,
"s3_secret_access_key" => s3_secret_access_key,
"s3_format" => "CSVWithNames",
"s3_structure" => s3_structure,
"start_date" => start_date,
"end_date" => end_date
}
Ch.query!(ch, statement, params, timeout: :infinity)
end)
rescue
# we are cancelling on any argument or ClickHouse errors
e in [ArgumentError, Ch.Error] ->
{:error, Exception.message(e)}
end
input_structures = %{
"imported_browsers" =>
"date Date, browser String, browser_version String, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32, pageviews UInt64",
"imported_devices" =>
"date Date, device String, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32, pageviews UInt64",
"imported_entry_pages" =>
"date Date, entry_page String, visitors UInt64, entrances UInt64, visit_duration UInt64, bounces UInt32, pageviews UInt64",
"imported_exit_pages" =>
"date Date, exit_page String, visitors UInt64, visit_duration UInt64, exits UInt64, bounces UInt32, pageviews UInt64",
"imported_locations" =>
"date Date, country String, region String, city UInt64, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32, pageviews UInt64",
"imported_operating_systems" =>
"date Date, operating_system String, operating_system_version String, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32, pageviews UInt64",
"imported_pages" =>
"date Date, hostname String, page String, visits UInt64, visitors UInt64, pageviews UInt64, exits UInt64, time_on_page UInt64",
"imported_sources" =>
"date Date, source String, referrer String, utm_source String, utm_medium String, utm_campaign String, utm_content String, utm_term String, pageviews UInt64, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32",
"imported_visitors" =>
"date Date, visitors UInt64, pageviews UInt64, bounces UInt64, visits UInt64, visit_duration UInt64"
}
@doc """
Extracts min/max date range from a list of uploads.
Examples:
iex> date_range([
...> %{"filename" => "imported_devices_20190101_20210101.csv"},
...> "imported_pages_20200101_20220101.csv"
...> ])
Date.range(~D[2019-01-01], ~D[2022-01-01])
iex> date_range([])
nil
"""
@spec date_range([String.t() | %{String.t() => String.t()}, ...]) :: Date.Range.t() | nil
def date_range([_ | _] = uploads), do: date_range(uploads, _start_date = nil, _end_date = nil)
def date_range([]), do: nil
defp date_range([upload | uploads], prev_start_date, prev_end_date) do
filename =
case upload do
%{"filename" => filename} -> filename
filename when is_binary(filename) -> filename
end
{_table, start_date, end_date} = parse_filename!(filename)
start_date =
if prev_start_date do
Enum.min([start_date, prev_start_date], Date)
else
start_date
end
end_date =
if prev_end_date do
Enum.max([end_date, prev_end_date], Date)
else
end_date
end
date_range(uploads, start_date, end_date)
end
defp date_range([], first, last), do: Date.range(first, last)
@spec parse_date!(String.t()) :: Date.t()
defp parse_date!(date) do
date |> Timex.parse!("{YYYY}{0M}{0D}") |> NaiveDateTime.to_date()
end
@doc """
Extracts table name and min/max dates from the filename.
Examples:
iex> parse_filename!("my_data.csv")
** (ArgumentError) invalid filename
iex> parse_filename!("imported_devices_00010101_20250101.csv")
{"imported_devices", ~D[0001-01-01], ~D[2025-01-01]}
"""
@spec parse_filename!(String.t()) ::
{table :: String.t(), start_date :: Date.t(), end_date :: Date.t()}
def parse_filename!(filename)
for {table, input_structure} <- input_structures do
defp input_structure!(unquote(table)), do: unquote(input_structure)
def parse_filename!(
<<unquote(table)::bytes, ?_, start_date::8-bytes, ?_, end_date::8-bytes, ".csv">>
) do
{unquote(table), parse_date!(start_date), parse_date!(end_date)}
end
end
def parse_filename!(_filename) do
raise ArgumentError, "invalid filename"
end
@doc """
Checks if the provided filename conforms to the expected format.
Examples:
iex> valid_filename?("my_data.csv")
false
iex> valid_filename?("imported_devices_00010101_20250101.csv")
true
"""
@spec valid_filename?(String.t()) :: boolean
def valid_filename?(filename) do
try do
parse_filename!(filename)
else
_ -> true
rescue
_ -> false
end
end
@doc """
Extracts the table name from the provided filename.
Raises if the filename doesn't conform to the expected format.
Examples:
iex> extract_table("my_data.csv")
** (ArgumentError) invalid filename
iex> extract_table("imported_devices_00010101_20250101.csv")
"imported_devices"
"""
@spec extract_table(String.t()) :: String.t()
def extract_table(filename) do
{table, _start_date, _end_date} = parse_filename!(filename)
table
end end
end end

@ -11,6 +11,7 @@ defmodule Plausible.Imported.Device do
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :visits, Ch, type: "UInt64" field :visits, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64" field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32" field :bounces, Ch, type: "UInt32"
end end
end end

@ -11,6 +11,7 @@ defmodule Plausible.Imported.EntryPage do
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :entrances, Ch, type: "UInt64" field :entrances, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64" field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32" field :bounces, Ch, type: "UInt32"
end end
end end

@ -8,7 +8,10 @@ defmodule Plausible.Imported.ExitPage do
field :import_id, Ch, type: "UInt64" field :import_id, Ch, type: "UInt64"
field :date, :date field :date, :date
field :exit_page, :string field :exit_page, :string
field :visitors, Ch, type: "UInt64"
field :exits, Ch, type: "UInt64" field :exits, Ch, type: "UInt64"
field :visitors, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32"
end end
end end

@ -0,0 +1,276 @@
defmodule Plausible.Imported.GoogleAnalytics4 do
@moduledoc """
Import implementation for Google Analytics 4.
"""
use Plausible.Imported.Importer
@missing_values ["(none)", "(not set)", "(not provided)", "(other)"]
@impl true
def name(), do: :google_analytics_4
@impl true
def label(), do: "Google Analytics 4"
@impl true
def email_template(), do: "google_analytics_import.html"
@impl true
def parse_args(
%{"property" => property, "start_date" => start_date, "end_date" => end_date} = args
) do
start_date = Date.from_iso8601!(start_date)
end_date = Date.from_iso8601!(end_date)
date_range = Date.range(start_date, end_date)
auth = {
Map.fetch!(args, "access_token"),
Map.fetch!(args, "refresh_token"),
Map.fetch!(args, "token_expires_at")
}
[
property: property,
date_range: date_range,
auth: auth
]
end
@doc """
Imports stats from a Google Analytics 4 property to a Plausible site.
This function fetches Google Analytics 4 reports which are then passed in batches
to Clickhouse by the `Plausible.Imported.Buffer` process.
"""
@impl true
def import_data(site_import, opts) do
date_range = Keyword.fetch!(opts, :date_range)
property = Keyword.fetch!(opts, :property)
auth = Keyword.fetch!(opts, :auth)
flush_interval_ms = Keyword.get(opts, :flush_interval_ms, 1000)
{:ok, buffer} = Plausible.Imported.Buffer.start_link(flush_interval_ms: flush_interval_ms)
persist_fn = fn table, rows ->
records = from_report(rows, site_import.site_id, site_import.id, table)
Plausible.Imported.Buffer.insert_many(buffer, table, records)
end
try do
Plausible.Google.GA4.API.import_analytics(date_range, property, auth, persist_fn)
after
Plausible.Imported.Buffer.flush(buffer)
Plausible.Imported.Buffer.stop(buffer)
end
end
def from_report(nil, _site_id, _import_id, _metric), do: nil
def from_report(data, site_id, import_id, table) do
Enum.reduce(data, [], fn row, acc ->
if Map.get(row.dimensions, "date") in @missing_values do
acc
else
[new_from_report(site_id, import_id, table, row) | acc]
end
end)
end
defp parse_number(nr) do
{float, ""} = Float.parse(nr)
round(float)
end
defp new_from_report(site_id, import_id, "imported_visitors", row) do
%{
site_id: site_id,
import_id: import_id,
date: get_date(row),
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(),
visits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number()
}
end
defp new_from_report(site_id, import_id, "imported_sources", row) do
referrer_uri = row.dimensions |> Map.fetch!("pageReferrer") |> URI.parse()
referrer =
if PlausibleWeb.RefInspector.right_uri?(referrer_uri) do
PlausibleWeb.RefInspector.format_referrer(referrer_uri)
end
%{
site_id: site_id,
import_id: import_id,
date: get_date(row),
source: row.dimensions |> Map.fetch!("sessionSource") |> parse_referrer(),
referrer: referrer,
# Only `source` exists in GA4 API
utm_source: nil,
utm_medium: row.dimensions |> Map.fetch!("sessionMedium") |> default_if_missing(),
utm_campaign: row.dimensions |> Map.fetch!("sessionCampaignName") |> default_if_missing(),
utm_content: row.dimensions |> Map.fetch!("sessionManualAdContent") |> default_if_missing(),
utm_term: row.dimensions |> Map.fetch!("sessionGoogleAdsKeyword") |> default_if_missing(),
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
visits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(),
visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number()
}
end
defp new_from_report(site_id, import_id, "imported_pages", row) do
%{
site_id: site_id,
import_id: import_id,
date: get_date(row),
hostname: row.dimensions |> Map.fetch!("hostName") |> String.replace_prefix("www.", ""),
page: row.dimensions |> Map.fetch!("pagePath") |> URI.parse() |> Map.get(:path),
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
visits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
# NOTE: no exits metric in GA4 API currently
exits: 0,
time_on_page: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number()
}
end
defp new_from_report(site_id, import_id, "imported_entry_pages", row) do
%{
site_id: site_id,
import_id: import_id,
date: get_date(row),
entry_page: row.dimensions |> Map.fetch!("landingPage"),
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
entrances: row.metrics |> Map.fetch!("sessions") |> parse_number(),
visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
bounces: row.metrics |> Map.fetch!("bounces") |> parse_number()
}
end
# NOTE: no exit pages metrics in GA4 API available for now
# defp new_from_report(site_id, import_id, "imported_exit_pages", row) do
# %{
# site_id: site_id,
# import_id: import_id,
# date: get_date(row),
# exit_page: Map.fetch!(row.dimensions, "exitPage"),
# visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
# exits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
# visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number(),
# pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
# bounces: row.metrics |> Map.fetch!("bounces") |> parse_number()
# }
# end
defp new_from_report(site_id, import_id, "imported_locations", row) do
country_code = row.dimensions |> Map.fetch!("countryId") |> default_if_missing("")
city_name = row.dimensions |> Map.fetch!("city") |> default_if_missing("")
city_data = Location.get_city(city_name, country_code)
%{
site_id: site_id,
import_id: import_id,
date: get_date(row),
country: country_code,
region: row.dimensions |> Map.fetch!("region") |> default_if_missing(""),
city: city_data && city_data.id,
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
visits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(),
visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number()
}
end
defp new_from_report(site_id, import_id, "imported_devices", row) do
%{
site_id: site_id,
import_id: import_id,
date: get_date(row),
device: row.dimensions |> Map.fetch!("deviceCategory") |> String.capitalize(),
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
visits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(),
visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number()
}
end
@browser_google_to_plausible %{
"User-Agent:Opera" => "Opera",
"Mozilla Compatible Agent" => "Mobile App",
"Android Webview" => "Mobile App",
"Android Browser" => "Mobile App",
"Safari (in-app)" => "Mobile App",
"User-Agent: Mozilla" => "Firefox",
"(not set)" => ""
}
defp new_from_report(site_id, import_id, "imported_browsers", row) do
browser = Map.fetch!(row.dimensions, "browser")
%{
site_id: site_id,
import_id: import_id,
date: get_date(row),
browser: Map.get(@browser_google_to_plausible, browser, browser),
# Does not exist in GA4 API
browser_version: nil,
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
visits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(),
visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number()
}
end
@os_google_to_plausible %{
"Macintosh" => "Mac",
"Linux" => "GNU/Linux",
"(not set)" => ""
}
defp new_from_report(site_id, import_id, "imported_operating_systems", row) do
os = Map.fetch!(row.dimensions, "operatingSystem")
%{
site_id: site_id,
import_id: import_id,
date: get_date(row),
operating_system: Map.get(@os_google_to_plausible, os, os),
operating_system_version: row.dimensions |> Map.fetch!("operatingSystemVersion"),
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
visits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(),
visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number()
}
end
defp get_date(%{dimensions: %{"date" => date}}) do
date
|> Timex.parse!("%Y%m%d", :strftime)
|> NaiveDateTime.to_date()
end
defp default_if_missing(value, default \\ nil)
defp default_if_missing(value, default) when value in @missing_values, do: default
defp default_if_missing(value, _default), do: value
defp parse_referrer(nil), do: nil
defp parse_referrer("(direct)"), do: nil
defp parse_referrer("google"), do: "Google"
defp parse_referrer("bing"), do: "Bing"
defp parse_referrer("duckduckgo"), do: "DuckDuckGo"
defp parse_referrer(ref) do
RefInspector.parse("https://" <> ref)
|> PlausibleWeb.RefInspector.parse()
end
end

@ -4,6 +4,7 @@ defmodule Plausible.Imported.ImportSources do
""" """
@sources [ @sources [
Plausible.Imported.GoogleAnalytics4,
Plausible.Imported.UniversalAnalytics, Plausible.Imported.UniversalAnalytics,
Plausible.Imported.NoopImporter, Plausible.Imported.NoopImporter,
Plausible.Imported.CSVImporter Plausible.Imported.CSVImporter

@ -91,6 +91,7 @@ defmodule Plausible.Imported.Importer do
like LiveView, where notifications can be listened for via `handle_info/2`. like LiveView, where notifications can be listened for via `handle_info/2`.
""" """
alias Plausible.Imported
alias Plausible.Imported.SiteImport alias Plausible.Imported.SiteImport
alias Plausible.Repo alias Plausible.Repo
@ -98,7 +99,7 @@ defmodule Plausible.Imported.Importer do
@callback label() :: String.t() @callback label() :: String.t()
@callback email_template() :: String.t() @callback email_template() :: String.t()
@callback parse_args(map()) :: Keyword.t() @callback parse_args(map()) :: Keyword.t()
@callback import_data(Plausible.Imported.SiteImport.t(), Keyword.t()) :: :ok | {:error, any()} @callback import_data(SiteImport.t(), Keyword.t()) :: :ok | {:error, any()}
@callback before_start(SiteImport.t()) :: :ok | {:ok, map()} | {:error, any()} @callback before_start(SiteImport.t()) :: :ok | {:ok, map()} | {:error, any()}
@callback on_success(SiteImport.t(), map()) :: :ok @callback on_success(SiteImport.t(), map()) :: :ok
@callback on_failure(SiteImport.t()) :: :ok @callback on_failure(SiteImport.t()) :: :ok
@ -108,7 +109,7 @@ defmodule Plausible.Imported.Importer do
@behaviour Plausible.Imported.Importer @behaviour Plausible.Imported.Importer
@spec new_import(Plausible.Site.t(), Plausible.Auth.User.t(), Keyword.t()) :: @spec new_import(Plausible.Site.t(), Plausible.Auth.User.t(), Keyword.t()) ::
{:ok, Oban.Job.t()} | {:error, Ecto.Changeset.t()} {:ok, Oban.Job.t()} | {:error, Ecto.Changeset.t() | :import_in_progress}
def new_import(site, user, opts) do def new_import(site, user, opts) do
Plausible.Imported.Importer.new_import(name(), site, user, opts, &before_start/1) Plausible.Imported.Importer.new_import(name(), site, user, opts, &before_start/1)
end end
@ -155,7 +156,7 @@ defmodule Plausible.Imported.Importer do
def new_import(source, site, user, opts, before_start_fun) do def new_import(source, site, user, opts, before_start_fun) do
import_params = import_params =
opts opts
|> Keyword.take([:start_date, :end_date]) |> Keyword.take([:label, :start_date, :end_date, :legacy])
|> Keyword.put(:source, source) |> Keyword.put(:source, source)
|> Map.new() |> Map.new()
@ -205,18 +206,32 @@ defmodule Plausible.Imported.Importer do
Oban.Notifier.notify(Oban, @oban_channel, %{event => site_import.id}) Oban.Notifier.notify(Oban, @oban_channel, %{event => site_import.id})
end end
@doc """
Allows to explicitly start listening for importer job notifications.
Listener must explicitly filter out a subset of imports that apply to the given context.
"""
@spec listen() :: :ok
def listen() do
:ok = Oban.Notifier.listen([@oban_channel])
end
defp schedule_job(site_import, opts) do defp schedule_job(site_import, opts) do
{listen?, opts} = Keyword.pop(opts, :listen?, false) {listen?, opts} = Keyword.pop(opts, :listen?, false)
if listen? do if listen? do
:ok = Oban.Notifier.listen([@oban_channel]) :ok = listen()
end end
opts if not Imported.other_imports_in_progress?(site_import) do
|> Keyword.put(:import_id, site_import.id) opts
|> Map.new() |> Keyword.put(:import_id, site_import.id)
|> Plausible.Workers.ImportAnalytics.new() |> Map.new()
|> Oban.insert() |> Plausible.Workers.ImportAnalytics.new()
|> Oban.insert()
else
{:error, :import_in_progress}
end
end end
defp mark_complete(site_import, extra_data, on_success_fun) do defp mark_complete(site_import, extra_data, on_success_fun) do

@ -13,6 +13,7 @@ defmodule Plausible.Imported.Location do
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :visits, Ch, type: "UInt64" field :visits, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64" field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32" field :bounces, Ch, type: "UInt32"
end end
end end

@ -8,9 +8,11 @@ defmodule Plausible.Imported.OperatingSystem do
field :import_id, Ch, type: "UInt64" field :import_id, Ch, type: "UInt64"
field :date, :date field :date, :date
field :operating_system, :string field :operating_system, :string
field :operating_system_version, :string
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :visits, Ch, type: "UInt64" field :visits, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64" field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32" field :bounces, Ch, type: "UInt32"
end end
end end

@ -9,6 +9,7 @@ defmodule Plausible.Imported.Page do
field :date, :date field :date, :date
field :hostname, :string field :hostname, :string
field :page, :string field :page, :string
field :visits, Ch, type: "UInt64"
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64" field :pageviews, Ch, type: "UInt64"
field :exits, Ch, type: "UInt64" field :exits, Ch, type: "UInt64"

@ -18,8 +18,10 @@ defmodule Plausible.Imported.SiteImport do
schema "site_imports" do schema "site_imports" do
field :start_date, :date field :start_date, :date
field :end_date, :date field :end_date, :date
field :label, :string
field :source, Ecto.Enum, values: ImportSources.names() field :source, Ecto.Enum, values: ImportSources.names()
field :status, Ecto.Enum, values: @statuses field :status, Ecto.Enum, values: @statuses
field :legacy, :boolean, default: false
belongs_to :site, Site belongs_to :site, Site
belongs_to :imported_by, User belongs_to :imported_by, User
@ -31,16 +33,38 @@ defmodule Plausible.Imported.SiteImport do
defmacro unquote(status)(), do: unquote(status) defmacro unquote(status)(), do: unquote(status)
end end
@spec label(t() | Site.ImportedData.t()) :: String.t()
def label(%__MODULE__{source: source, label: label}) do
build_label(ImportSources.by_name(source).label(), label)
end
# NOTE: this is necessary for backwards compatibility # NOTE: this is necessary for backwards compatibility
# with legacy imports # with legacy imports
@spec label(t() | Site.ImportedData.t()) :: String.t() def label(%Site.ImportedData{source: source}), do: build_label(source, nil)
def label(%__MODULE__{source: source}), do: ImportSources.by_name(source).label()
def label(%Site.ImportedData{source: source}), do: source @spec from_legacy(Site.ImportedData.t()) :: t()
def from_legacy(%Site.ImportedData{} = data) do
status =
case data.status do
"ok" -> completed()
"error" -> failed()
_ -> importing()
end
%__MODULE__{
id: 0,
legacy: true,
start_date: data.start_date,
end_date: data.end_date,
source: :universal_analytics,
status: status
}
end
@spec create_changeset(Site.t(), User.t(), map()) :: Ecto.Changeset.t() @spec create_changeset(Site.t(), User.t(), map()) :: Ecto.Changeset.t()
def create_changeset(site, user, params) do def create_changeset(site, user, params) do
%__MODULE__{} %__MODULE__{}
|> cast(params, [:source, :start_date, :end_date]) |> cast(params, [:label, :source, :start_date, :end_date, :legacy])
|> validate_required([:source]) |> validate_required([:source])
|> put_assoc(:site, site) |> put_assoc(:site, site)
|> put_assoc(:imported_by, user) |> put_assoc(:imported_by, user)
@ -65,4 +89,7 @@ defmodule Plausible.Imported.SiteImport do
def fail_changeset(site_import) do def fail_changeset(site_import) do
change(site_import, status: failed()) change(site_import, status: failed())
end end
defp build_label(source, nil), do: source
defp build_label(source, label), do: "#{source} (#{label})"
end end

@ -8,6 +8,8 @@ defmodule Plausible.Imported.Source do
field :import_id, Ch, type: "UInt64" field :import_id, Ch, type: "UInt64"
field :date, :date field :date, :date
field :source, :string field :source, :string
field :referrer, :string
field :utm_source, :string
field :utm_medium, :string field :utm_medium, :string
field :utm_campaign, :string field :utm_campaign, :string
field :utm_content, :string field :utm_content, :string
@ -15,6 +17,7 @@ defmodule Plausible.Imported.Source do
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :visits, Ch, type: "UInt64" field :visits, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64" field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32" field :bounces, Ch, type: "UInt32"
end end
end end

@ -20,37 +20,43 @@ defmodule Plausible.Imported.UniversalAnalytics do
@impl true @impl true
def before_start(site_import) do def before_start(site_import) do
site = Repo.preload(site_import, :site).site if site_import.legacy do
site = Repo.preload(site_import, :site).site
site
|> Plausible.Site.start_import( site
site_import.start_date, |> Plausible.Site.start_import(
site_import.end_date, site_import.start_date,
label() site_import.end_date,
) label()
|> Repo.update!() )
|> Repo.update!()
end
:ok :ok
end end
@impl true @impl true
def on_success(site_import, _extra_data) do def on_success(site_import, _extra_data) do
site = Repo.preload(site_import, :site).site if site_import.legacy do
site = Repo.preload(site_import, :site).site
site site
|> Plausible.Site.import_success() |> Plausible.Site.import_success()
|> Repo.update!() |> Repo.update!()
end
:ok :ok
end end
@impl true @impl true
def on_failure(site_import) do def on_failure(site_import) do
site = Repo.preload(site_import, :site).site if site_import.legacy do
site = Repo.preload(site_import, :site).site
site site
|> Plausible.Site.import_failure() |> Plausible.Site.import_failure()
|> Repo.update!() |> Repo.update!()
end
:ok :ok
end end
@ -96,7 +102,7 @@ defmodule Plausible.Imported.UniversalAnalytics do
end end
try do try do
Plausible.Google.Api.import_analytics(date_range, view_id, auth, persist_fn) Plausible.Google.UA.API.import_analytics(date_range, view_id, auth, persist_fn)
after after
Plausible.Imported.Buffer.flush(buffer) Plausible.Imported.Buffer.flush(buffer)
Plausible.Imported.Buffer.stop(buffer) Plausible.Imported.Buffer.stop(buffer)

@ -13,6 +13,7 @@ defmodule Plausible.Ingestion.Event do
defstruct domain: nil, defstruct domain: nil,
site: nil, site: nil,
clickhouse_event_attrs: %{}, clickhouse_event_attrs: %{},
clickhouse_session_attrs: %{},
clickhouse_event: nil, clickhouse_event: nil,
dropped?: false, dropped?: false,
drop_reason: nil, drop_reason: nil,
@ -27,11 +28,14 @@ defmodule Plausible.Ingestion.Event do
| :invalid | :invalid
| :dc_ip | :dc_ip
| :site_ip_blocklist | :site_ip_blocklist
| :site_country_blocklist
| :site_page_blocklist
@type t() :: %__MODULE__{ @type t() :: %__MODULE__{
domain: String.t() | nil, domain: String.t() | nil,
site: %Plausible.Site{} | nil, site: %Plausible.Site{} | nil,
clickhouse_event_attrs: map(), clickhouse_event_attrs: map(),
clickhouse_session_attrs: map(),
clickhouse_event: %ClickhouseEventV2{} | nil, clickhouse_event: %ClickhouseEventV2{} | nil,
dropped?: boolean(), dropped?: boolean(),
drop_reason: drop_reason(), drop_reason: drop_reason(),
@ -76,6 +80,10 @@ defmodule Plausible.Ingestion.Event do
[:plausible, :ingest, :event, :dropped] [:plausible, :ingest, :event, :dropped]
end end
def telemetry_pipeline_step_duration() do
[:plausible, :ingest, :pipeline, :step]
end
@spec emit_telemetry_buffered(t()) :: :ok @spec emit_telemetry_buffered(t()) :: :ok
def emit_telemetry_buffered(event) do def emit_telemetry_buffered(event) do
:telemetry.execute(telemetry_event_buffered(), %{}, %{ :telemetry.execute(telemetry_event_buffered(), %{}, %{
@ -95,32 +103,42 @@ defmodule Plausible.Ingestion.Event do
defp pipeline() do defp pipeline() do
[ [
&drop_datacenter_ip/1, drop_datacenter_ip: &drop_datacenter_ip/1,
&drop_shield_rule_ip/1, drop_shield_rule_page: &drop_shield_rule_page/1,
&put_user_agent/1, drop_shield_rule_ip: &drop_shield_rule_ip/1,
&put_basic_info/1, put_geolocation: &put_geolocation/1,
&put_referrer/1, drop_shield_rule_country: &drop_shield_rule_country/1,
&put_utm_tags/1, put_user_agent: &put_user_agent/1,
&put_geolocation/1, put_basic_info: &put_basic_info/1,
&put_props/1, put_referrer: &put_referrer/1,
&put_revenue/1, put_utm_tags: &put_utm_tags/1,
&put_salts/1, put_props: &put_props/1,
&put_user_id/1, put_revenue: &put_revenue/1,
&validate_clickhouse_event/1, put_salts: &put_salts/1,
&register_session/1, put_user_id: &put_user_id/1,
&write_to_buffer/1 validate_clickhouse_event: &validate_clickhouse_event/1,
register_session: &register_session/1,
write_to_buffer: &write_to_buffer/1
] ]
end end
defp process_unless_dropped(%__MODULE__{} = initial_event, pipeline) do defp process_unless_dropped(%__MODULE__{} = initial_event, pipeline) do
Enum.reduce_while(pipeline, initial_event, fn pipeline_step, acc_event -> Enum.reduce_while(pipeline, initial_event, fn {step_name, step_fn}, acc_event ->
case pipeline_step.(acc_event) do Plausible.PromEx.Plugins.PlausibleMetrics.measure_duration(
%__MODULE__{dropped?: true} = dropped -> {:halt, dropped} telemetry_pipeline_step_duration(),
%__MODULE__{dropped?: false} = event -> {:cont, event} fn -> execute_step(step_fn, acc_event) end,
end %{step: "#{step_name}"}
)
end) end)
end end
defp execute_step(step_fn, acc_event) do
case step_fn.(acc_event) do
%__MODULE__{dropped?: true} = dropped -> {:halt, dropped}
%__MODULE__{dropped?: false} = event -> {:cont, event}
end
end
defp new(domain, request) do defp new(domain, request) do
struct!(__MODULE__, domain: domain, request: request) struct!(__MODULE__, domain: domain, request: request)
end end
@ -139,10 +157,14 @@ defmodule Plausible.Ingestion.Event do
struct!(event, fields) struct!(event, fields)
end end
defp update_attrs(%__MODULE__{} = event, %{} = attrs) do defp update_event_attrs(%__MODULE__{} = event, %{} = attrs) do
struct!(event, clickhouse_event_attrs: Map.merge(event.clickhouse_event_attrs, attrs)) struct!(event, clickhouse_event_attrs: Map.merge(event.clickhouse_event_attrs, attrs))
end end
defp update_session_attrs(%__MODULE__{} = event, %{} = attrs) do
struct!(event, clickhouse_session_attrs: Map.merge(event.clickhouse_session_attrs, attrs))
end
defp drop_datacenter_ip(%__MODULE__{} = event) do defp drop_datacenter_ip(%__MODULE__{} = event) do
case event.request.ip_classification do case event.request.ip_classification do
"dc_ip" -> "dc_ip" ->
@ -154,15 +176,18 @@ defmodule Plausible.Ingestion.Event do
end end
defp drop_shield_rule_ip(%__MODULE__{} = event) do defp drop_shield_rule_ip(%__MODULE__{} = event) do
domain = event.domain if Plausible.Shields.ip_blocked?(event.domain, event.request.remote_ip) do
address = event.request.remote_ip drop(event, :site_ip_blocklist)
else
case Plausible.Shield.IPRuleCache.get({domain, address}) do event
%Plausible.Shield.IPRule{action: :deny} -> end
drop(event, :site_ip_blocklist) end
_ -> defp drop_shield_rule_page(%__MODULE__{} = event) do
event if Plausible.Shields.page_blocked?(event.domain, event.request.pathname) do
drop(event, :site_page_blocklist)
else
event
end end
end end
@ -175,7 +200,7 @@ defmodule Plausible.Ingestion.Event do
drop(event, :bot) drop(event, :bot)
%UAInspector.Result{} = user_agent -> %UAInspector.Result{} = user_agent ->
update_attrs(event, %{ update_session_attrs(event, %{
operating_system: os_name(user_agent), operating_system: os_name(user_agent),
operating_system_version: os_version(user_agent), operating_system_version: os_version(user_agent),
browser: browser_name(user_agent), browser: browser_name(user_agent),
@ -189,7 +214,7 @@ defmodule Plausible.Ingestion.Event do
end end
defp put_basic_info(%__MODULE__{} = event) do defp put_basic_info(%__MODULE__{} = event) do
update_attrs(event, %{ update_event_attrs(event, %{
domain: event.domain, domain: event.domain,
site_id: event.site.id, site_id: event.site.id,
timestamp: event.request.timestamp, timestamp: event.request.timestamp,
@ -202,7 +227,7 @@ defmodule Plausible.Ingestion.Event do
defp put_referrer(%__MODULE__{} = event) do defp put_referrer(%__MODULE__{} = event) do
ref = parse_referrer(event.request.uri, event.request.referrer) ref = parse_referrer(event.request.uri, event.request.referrer)
update_attrs(event, %{ update_session_attrs(event, %{
referrer_source: get_referrer_source(event.request, ref), referrer_source: get_referrer_source(event.request, ref),
referrer: clean_referrer(ref) referrer: clean_referrer(ref)
}) })
@ -211,7 +236,7 @@ defmodule Plausible.Ingestion.Event do
defp put_utm_tags(%__MODULE__{} = event) do defp put_utm_tags(%__MODULE__{} = event) do
query_params = event.request.query_params query_params = event.request.query_params
update_attrs(event, %{ update_session_attrs(event, %{
utm_medium: query_params["utm_medium"], utm_medium: query_params["utm_medium"],
utm_source: query_params["utm_source"], utm_source: query_params["utm_source"],
utm_campaign: query_params["utm_campaign"], utm_campaign: query_params["utm_campaign"],
@ -221,16 +246,34 @@ defmodule Plausible.Ingestion.Event do
end end
defp put_geolocation(%__MODULE__{} = event) do defp put_geolocation(%__MODULE__{} = event) do
result = Plausible.Ingestion.Geolocation.lookup(event.request.remote_ip) || %{} case event.request.ip_classification do
"anonymous_vpn_ip" ->
update_session_attrs(event, %{country_code: "A1"})
update_attrs(event, result) _any ->
result = Plausible.Ingestion.Geolocation.lookup(event.request.remote_ip) || %{}
update_session_attrs(event, result)
end
end end
defp drop_shield_rule_country(
%__MODULE__{domain: domain, clickhouse_session_attrs: %{country_code: cc}} = event
)
when is_binary(domain) and is_binary(cc) do
if Plausible.Shields.country_blocked?(domain, cc) do
drop(event, :site_country_blocklist)
else
event
end
end
defp drop_shield_rule_country(%__MODULE__{} = event), do: event
defp put_props(%__MODULE__{request: %{props: %{} = props}} = event) do defp put_props(%__MODULE__{request: %{props: %{} = props}} = event) do
# defensive: ensuring the keys/values are always in the same order # defensive: ensuring the keys/values are always in the same order
{keys, values} = Enum.unzip(props) {keys, values} = Enum.unzip(props)
update_attrs(event, %{ update_event_attrs(event, %{
"meta.key": keys, "meta.key": keys,
"meta.value": values "meta.value": values
}) })
@ -241,7 +284,7 @@ defmodule Plausible.Ingestion.Event do
defp put_revenue(event) do defp put_revenue(event) do
on_full_build do on_full_build do
attrs = Plausible.Ingestion.Event.Revenue.get_revenue_attrs(event) attrs = Plausible.Ingestion.Event.Revenue.get_revenue_attrs(event)
update_attrs(event, attrs) update_event_attrs(event, attrs)
else else
event event
end end
@ -252,7 +295,7 @@ defmodule Plausible.Ingestion.Event do
end end
defp put_user_id(%__MODULE__{} = event) do defp put_user_id(%__MODULE__{} = event) do
update_attrs(event, %{ update_event_attrs(event, %{
user_id: user_id:
generate_user_id( generate_user_id(
event.request, event.request,
@ -287,10 +330,17 @@ defmodule Plausible.Ingestion.Event do
event.salts.previous event.salts.previous
) )
session_id = Plausible.Session.CacheStore.on_event(event.clickhouse_event, previous_user_id) session =
Plausible.Session.CacheStore.on_event(
event.clickhouse_event,
event.clickhouse_session_attrs,
previous_user_id
)
clickhouse_event = Map.put(event.clickhouse_event, :session_id, session_id) %{
%{event | clickhouse_event: clickhouse_event} event
| clickhouse_event: ClickhouseEventV2.merge_session(event.clickhouse_event, session)
}
end end
defp write_to_buffer(%__MODULE__{clickhouse_event: clickhouse_event} = event) do defp write_to_buffer(%__MODULE__{clickhouse_event: clickhouse_event} = event) do
@ -325,18 +375,14 @@ defmodule Plausible.Ingestion.Event do
uri = URI.parse(ref.referer) uri = URI.parse(ref.referer)
if PlausibleWeb.RefInspector.right_uri?(uri) do if PlausibleWeb.RefInspector.right_uri?(uri) do
host = String.replace_prefix(uri.host, "www.", "") PlausibleWeb.RefInspector.format_referrer(uri)
path = uri.path || ""
host <> String.trim_trailing(path, "/")
end end
end end
defp parse_user_agent(%Request{user_agent: user_agent}) when is_binary(user_agent) do defp parse_user_agent(%Request{user_agent: user_agent}) when is_binary(user_agent) do
case Cachex.fetch(:user_agents, user_agent, &UAInspector.parse/1) do Plausible.Cache.Adapter.get(:user_agents, user_agent, fn ->
{:ok, user_agent} -> user_agent UAInspector.parse(user_agent)
{:commit, user_agent} -> user_agent end)
_ -> nil
end
end end
defp parse_user_agent(request), do: request defp parse_user_agent(request), do: request

@ -130,7 +130,8 @@ defmodule Plausible.Ingestion.WriteBuffer do
|> Ch.RowBinary.encode_names_and_types(types) |> Ch.RowBinary.encode_names_and_types(types)
|> IO.iodata_to_binary() |> IO.iodata_to_binary()
insert_sql = "INSERT INTO #{schema.__schema__(:source)} FORMAT RowBinaryWithNamesAndTypes" insert_sql =
"INSERT INTO #{schema.__schema__(:source)} (#{Enum.join(fields, ", ")}) FORMAT RowBinaryWithNamesAndTypes"
%{ %{
fields: fields, fields: fields,

@ -4,9 +4,13 @@ defmodule Plausible.MigrationUtils do
""" """
def on_cluster_statement(table) do def on_cluster_statement(table) do
if(clustered_table?(table), do: "ON CLUSTER '{cluster}'", else: "")
end
def clustered_table?(table) do
case Plausible.IngestRepo.query("SELECT 1 FROM system.replicas WHERE table = '#{table}'") do case Plausible.IngestRepo.query("SELECT 1 FROM system.replicas WHERE table = '#{table}'") do
{:ok, %{rows: []}} -> "" {:ok, %{rows: []}} -> false
{:ok, _} -> "ON CLUSTER '{cluster}'" {:ok, _} -> true
end end
end end
end end

@ -0,0 +1,35 @@
defmodule Plausible.Plugins.API.Capabilities do
@moduledoc """
Context module for querying API capabilities
"""
require Plausible.Billing.Feature
alias Plausible.Billing.Feature
@spec get(Plug.Conn.t()) :: {:ok, map()}
def get(conn) do
conn = PlausibleWeb.Plugs.AuthorizePluginsAPI.call(conn, send_error?: false)
site = conn.assigns[:authorized_site]
features =
if site do
site = Plausible.Repo.preload(site, :owner)
Feature.list()
|> Enum.map(fn mod ->
result = mod.check_availability(site.owner)
feature = mod |> Module.split() |> List.last()
{feature, result == :ok}
end)
else
Enum.map(Feature.list_short_names(), &{&1, false})
end
{:ok,
%{
authorized: not is_nil(site),
data_domain: site && site.domain,
features: Enum.into(features, %{})
}}
end
end

@ -58,17 +58,19 @@ defmodule Plausible.Purge do
def delete_imported_stats!(%Plausible.Imported.SiteImport{} = site_import) do def delete_imported_stats!(%Plausible.Imported.SiteImport{} = site_import) do
site_import = Repo.preload(site_import, :site) site_import = Repo.preload(site_import, :site)
delete_imported_stats!(site_import.site, site_import.id)
:ok
end
def delete_imported_stats!(%Plausible.Site{} = site, import_id) do
Enum.each(Plausible.Imported.tables(), fn table -> Enum.each(Plausible.Imported.tables(), fn table ->
sql = "ALTER TABLE #{table} DELETE WHERE site_id = {$0:UInt64} AND import_id = {$1:UInt64}" sql = "ALTER TABLE #{table} DELETE WHERE site_id = {$0:UInt64} AND import_id = {$1:UInt64}"
Ecto.Adapters.SQL.query!(Plausible.ImportDeletionRepo, sql, [ Ecto.Adapters.SQL.query!(Plausible.ImportDeletionRepo, sql, [site.id, import_id])
site_import.site_id,
site_import.id
])
end) end)
Plausible.Sites.clear_stats_start_date!(site_import.site) Plausible.Sites.clear_stats_start_date!(site)
:ok :ok
end end

Some files were not shown because too many files have changed in this diff Show More

Loading…
Cancel
Save