* default to source_query.include_imported before true
This fixes a bug where a Stats API aggregate query was trying
to query current period *without imported data* but previous
period *with imported data*.
* remove __internal_visits at a better time
As soon as we have made the db query, we don't need it anymore
* disallow events metric with imported data
* update changelog
* keep default in a better place
* remove unused option
* update test description
Co-authored-by: Vini Brasil <vini@hey.com>
---------
Co-authored-by: Vini Brasil <vini@hey.com>
* add migration
* add not null constraint
* remove not null constraint and default value
* add site schema change
* filter garbage props for prop_key suggestions
* filter garbage props returned by StatsController.conversions/2
* add crm action to set prop allowlist
* use nullable allowed_event_props field instead
* filter garbage props in db query instead
* implement suggestion dealing with string input
* refactor the allowlist condition in db query
* improve test
* Reject invalid country codes early
* Validate all date parameters early
* Ensure request JSON parses to a map
* Re-arrange function calls
* Extend malicious filter validation
* add entry page filter condition to bounce rate
* Fix select_merge with dynamic
* fix tests and add one test
* generalize page_filter_condition function
* use dynamic_filter_condition for session filters too
* disable views_per_visit with a page filter
* update changelog
* disable credo for complex function
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* Add PropFilterModal (only UI)
* small variable refactor
* allow selecting prop value filter type
* allow selecting only one prop_key
* allow selecting many prop_values only when prop_key selected
* handle submitting filter
* get applied filters from query + remove option
* change prop filter label format
* support member and not_member filter types for pageview props
* show (none) value in filter suggestions
* refactor zip_results/4 and remove unused code
* fix displaying (none) values in goals section prop breakdown
* remove unnecessary functionality
* fix bug: returning prop names for goal :member filter
* fix bug: submitting regular filter modal with Enter key
* bugfix: disallow opening prop filter modal when feature flag disabled
* mix format
* break selected combobox values into multiple lines
* fix useEffect behavior for focusing on prop_key field
* support submitting prop filter with Enter key
* refactor getFormState in PropFilterModal
* separate fetchPropKey and fetchPropValue functions
* Allow querying props for pageview goals
* Make the internal props API only return a list of props (not map)
* Separate function for fetching all props in Stats API goal breakdown (this returns a map as before)
* ditch state for keeping search bar visible
* group by event_name in db query
This commit fixes a bug where comparisons would not work with imported data. This is because comparisons copies the source query and modifies the dates only, where it should reevaluate if there is imported data for the new comparison date range.
Closes#2870
Co-authored-by: Adam <hq@mtod.org>
* Escape pipes in goal names
* Apply suggestions from code review
Use ~S sigil for better readability
Co-authored-by: Vini Brasil <vini@hey.com>
---------
Co-authored-by: Vini Brasil <vini@hey.com>
* Wrap Plausible.Stats.Filters with unit tests
* Parse `member` filter type
* Support for `member` filter in `aggregate_time_on_page`
* Support `not_member` filter type
* Support `matches_member` and `not_matches_member` filters
* Extract util module for React filters
* Implement Combobox from scratch with no libs
* Support multple filter clauses in combobox
* Don't use browser / os in version label
* Show highlighted option in combobox
* WIP
* Fix location filters outside filter modal
* Align open/close behaviour with react-select
* Styling updates for combobox
* Add support for wildcards in Combobox
* Implement keybindings for combobox
* Allow free choice inputs in combobox
* Rename 'Save filter' -> Apply filter
* Remove TODO comment
* Clean up some rebase mistakes
* Rename `allowWildcard` -> `freeChoice`
* Dark mode fixes
* Remove hint from filter modal
* Escape pipe character in filter modal
* Do not allow selecting duplicate options in combobox
* Escape brackets in `page_regex/1`
* Fix disabled style in dark mode
* Add regex fallback for safari
* Show no matches found when visibleOptions is empty
* Disable enter key when no visible options
* Do not submit empty form fields
* Remove unnecessary setOpen(true)
* Remove ClickhouseSetup module
This has been an implicit point of contact to many
tests. From now on the goal is for each test to maintain
its own, isolated setup so that no accidental clashes
and implicit assumptions are relied upon.
* Implement v2 schema check
An environment variable V2_MIGRATION_DONE acts like
a feature flag, switching plausible from using old events/sessions
schemas to v2 schemas introduced by NumericIDs migration.
* Run both test suites sequentially
While the code for v1 and v2 schemas must be kept still,
we will from now on run tests against both code paths.
Secondary test run will set V2_MIGRATION_DONE=1 variable,
thus making all `Plausible.v2?()` checks return `true'.
* Remove unused function
This is a remnant from the short period when
we would check for existing events before allowing
creating a new site.
* Update test setups/factories with v2 migration check
* Make GateKeeper return site id along with :allow
* Make Billing module check for v2 schema
* Make ingestion aware of v2 schema
* Disable site transfers for when v2 is live
In a separate changeset we will implement simplified
site transfer for when v2 migration is complete.
The new transfer will only rename the site domain in postgres
and keep track of the original site prior to the transfer
so we keep an ingestion grace period until the customers
redeploy their scripting.
* Make Stats base queries aware of v2 schema switch
* Update breakdown with v2 conditionals
* Update pageview local start with v2 check
* Update current visitoris with v2 check
* Update stats controller with v2 checks
* Update external controller with v2 checks
* Update remaining tests with proper fixtures
* Rewrite redundant assignment
* Remove unused alias
* Mute credo, this is not the right time
* Add test_helper prompt
* Fetch priv dir so it works with a release
* Fetch distinct partitions only
* Don't limit inspect output for partitions
* Ensure SQL is printed to IO
* Remove redundant domain fixture
* Wrap Plausible.Stats.Filters with unit tests
* Parse `member` filter type
* Support escaped | in member filter
* Support for `member` filter in `aggregate_time_on_page`
* Add support for `member` filter type on goals
* Disable Credo warning
* Support `not_member` filter type
* Disable credo for `query_sessions`
* Support `matches_member` and `not_matches_member` filters
* Disable Credo for `Filters.filter_value/2`
* Support `matches_member` and `not_matches_member` for goal filter
* Support for contains_member and friends
* Updates for new chto driver
* make top_stats_test.exs:203 pass (#2779)
---------
Co-authored-by: ruslandoga <rusl@n-do.ga>
This pull request adds support for multiple comparison modes, changes the comparison checkbox to a combobox, and implements the year over year comparison mode. The feature is still behind a feature flag.
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* globally rename 'pages_per_visit' to 'views_per_visit'
* change the order of top stats
* rename 'Visits' to 'Total visits' in the UI
* add views_per_visit to UI
* put the new metric under a feature flag
* add new metric to CSV export under feature flag
* mix format
* use only one feature flag
* refactor metric validation
* link to the correct docs section
* add the new metric to aggregate API
* explicitly remove __internal_visits
The overall metric selection is well defined by
`Plausible.Stats.Timeseries.empty_row/2`. The only metric that needs
to be removed from the timeseries response is __internal_visits.
This commit also moves the `remove_internal_visits_metric` function to a
new Util module to be used by both breakdown and timeseries.
* add the new metric to timeseries API
* mix format
* add moduledoc to keep credo happy
* convert pages_per_visit to string straight away
* do rounding in db query
* query # of visits with sum(sign) instead
* stop converting pages_per_visit to string
* Stop relying on implicitly selected `visits` metric
* Make the `visits` metric more explicit in imported data queries
* Rename internally used `visits` to `__internal_visits`
* Use HeadlessUI for search select box
* Remove downshift from package.json
* More consistent API for Combobox component
* Combine toFilterType and valueWithoutPrefix into a single function
* Rename MyCombobox -> PlausibleCombobox
* Update webpack-cli
* Disable cache for build
* Revert "Disable cache for build"
This reverts commit aa130541f8.
* Disable cache for build
* Update webpack dependencies
* Remove glob from webpack config
* Webpack is required by package.json
* Require autoprefixer in postcss config
* Revert build changes
* Fix styling for dark mode
* Reject unknown imported cities from queries
This commit fixes a bug where the city report returned `N/A` entries.
The functions that build imported data queries were using SQL
`COALESCE`, assuming city data is `NULL` when unknown, when actually its
unknown value is `0`.
This commit addresses the problem using SQL `NULLIF` combined with the
previous `COALESCE` call. With this change both `0` and `NULL` are
treated as unknown.
Since 1cb07efe6d cities can be `NULL`, but
previously we saved `0` as unknown.
Closes#1960
* Add entry to CHANGELOG
* Ignore cyclomatic complexity Credo check
* Make Device section components aware of (not set)
So that no extra sub-filters are possible when the unset
top item is selected.
* Support '(not set)' in breakdown/filters
* Update expectations for export tests
* Add extra tests for returning/filtering by '(not set)'
* Add changelog entry
* Remove ListReport conditional render
* Prevent redundant sub-filters
* Fix filter text rendering
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* fix subquery for sessions in base_event_query/2
As the 'sessions' table is using the CollapsingMergeTree engine, we have
to select session_id's distinctively. Otherwise we will get multiple rows
(with sign -1 and 1) as long as the background merge hasn't happened.
* update changelog
* use GROUP BY instead of SELECT DISTINCT
* remove comma
* Consolidate task timeouts, incrase to 15s
* Use Task.async_stream for parallel Clickhouse queries
* Propagate Opentelemetry context to child Task process
* Fix breakdown API pagination when using event metrics
This commit fixes a bug where the subsequent breakdown API pages had
the same items as the first page. The fix sorts the underlying
ClickHouse query by timestamp, keeping the same order between requests,
as we use OFFSET/LIMIT pagination.
* Fix repeated results assertion
* Add different ORDER BY to each breakdown property
* Return empty list when breaking down by event:page without events
This commit fixes a bug with pagination where breaking down by event:page
would always return results despite pagination.
Closes#2255
* Update CHANGELOG.md
People are likely to enter (copy/paste) goals from external sources
which can lead to whitespace characters appended by accident.
That renders the goal unusable and hard to distinct visually.
Normally to fix up existing goals we would use a data migration,
but this should be good enough to check if the problem
with never appearing goals resurfaces.
* Refactor Plausible.Sites.stats_start_date/1 function
This commit adds documentation, typespecs and better pattern matching to
the Plausible.Sites.stats_start_date/1 function.
* Use memoized stats_start_date instead of querying CH
* Prevent domain from being updated from CRM
* Add stats_start_date to CRM site form
This commit adds stats_start_date to CRM site form, and creates a
separate Ecto changeset for CRM changes.
* Alias Plausible.Site calls
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* List all timezones in CRM form
* Require sites.public in CRM changeset
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* add separate module for filter parsing
* add tests for filter parser
* allow escaping pipe character in filter value
* add documentation and doctests
* do not remove escape chars from wildcard values
* changelog update
* change the parse_filters/1 function argument
This commit fixes a bug where location filters were filtering stats but
not the locations list. This was caused by a `Map.put/3` call that
overrides the user filter. This commit rollbacks 5b57143273
changes and removes the `Map.put/3` call.
Closes#1982
* added custom dimension filtering tests for pages
* first filter UI in place
* pages, entry pages and exit pages can be filtered by pageview props
* added tests for expected filtering behaviour
* fix dimension filter for sources + tests
* added is_not filtering functionality
* fixed formatting
* fixed admin_test
* added (none) as filter value + is_not filter type in UI
* added prefilling applied filter values and some UI tweaks
* added fetch options
* Make prop suggestions work with `props` filter
* Fix test
* Track login state internally
* Add CHANGELOG entry
Co-authored-by: Uku Taht <uku.taht@gmail.com>