Compare commits

..

17 Commits

Author SHA1 Message Date
Adrian Gruntkowski a7603c9e49
Improve import procedure to ensure no time range overlaps (#3970)
* Always scope import ID by site as well

* Do not schedule new import job if there are any site imports in progress

* Disable import buttons when any import is in progress

* Simplify `schedule_job/4` (h/t @RobertJoonas)
3 years ago
Adrian Gruntkowski cffff0340c
Handle Google API timeouts gracefully during imports (#3975) 3 years ago
Adrian Gruntkowski 33eed9d7db
Delete imports which have no stats (#3972) 3 years ago
Marko Saric e5b7f1afd0
Changing the copy of the locked screen (#3967)
changing the copy here as I think that in some situations the "we're still counting stats" message is now shown even to those dashboards where we've stopped counting stats so best to avoid that
3 years ago
Uku Taht f966419a4a
Update ua_inspector (#3957)
Co-authored-by: hq1 <hq@mtod.org>
3 years ago
hq1 f9f0407d68
Remove `experimtnal_hostname_filter` and keep it on by default (#3973)
* Remove `experimental_hostname_filter` and keep it on by default

* Catch up with changes done via e5b56dbe6
3 years ago
RobertJoonas e5b56dbe62
Refactor VisitorGraph (#3936)
* Give a more semantic name to a function

* Make the LineGraph component thinner

* Move LineGraph into a separate file

* Move interval logic into interval-picker.js

This commit also fixes a bug where the interval name displayed inside
the picker component flickers the default interval when the graph is
loading.

The problem was that we were counting on graphData for returning us the
current interval: `let currentInterval = graphData?.interval`

We should always know the default interval before making the main-graph
request. Sending graphData to IntervalPicker component does not make
sense anyway.

* extract data fetching functions out of VisitorGraph component

* Return graph_metric key from Top Stats API

This commit introduces no behavioral changes - only starts returning an
additional field, allowing us to avoid the following logic in React:

1. Finding the metric names, given a stat display name. E.g.
   `Unique visitors (last 30 min) -> visitors`

2. Checking if a metric is graphable or not

* Move metric state into localStorage

This commit gets rid of the internal `metric` state in the VisitorGraph
component and starts using localStorage for that instead.

This commit also chains the main-graph request into the top-stats request
callback - meaning that we'll always fetch new graph data after top stats
are updated. And we do it all in a single function.

Doing so simplifies the loading state significantly, and also helps to
make it clear, that at all times, existing top stats are required before
we can fetch the graph. That's because the metric is determined by which
Top stats are returned (for example, we can't be sure whether revenue
metrics will be returned or not).

* Make sure graph tooltip says "Converted Visitors"

* Extract a StatsExport function component

Again, instead of relying on `graphData?.interval` we can read it from
localStorage, or default to the largest interval available. The export
should not be dependant on the graph.

* Extract SamplingNotice function component

* Extract WithImportedSwitch function component

* Stop "lazy-loading" the graph and top stats

Since the container is always on top on the page, it will be visible on
the first render in any case - no matter the screen size.

* Turn VisitorGraph into a function component

* Display empty container until everything has loaded

* Do not display loading spinner on realtime ticks

* Turn Top Stats into a fn component

* fetch top stats and graph async

* Make sure revenue metrics can remain on the graph

* Add an extra check to canMetricBeGraphed

* fix typo

* remove redundant double negation
3 years ago
Karl-Aksel Puulmann 3115c6e7a8
Reducing JOINs in queries (#3966)
* Move experimental_session_count? logic to within query object

* WIP new querying system for deciding what tables to query

* both -> either

* Include sample_percent in both tables

* Remove a hanging TODO

* Allow filtering by visit props on event queries if flag is on

* Make default sessions join more conditional

* Simplify events_join_sessions?

* Add some TODOs

* Fix assignment

* Handle entry/exit page visit props separately from props stored in events table

* Update test which created sessions/events differently from everyone else

* Make query_events private

* Dont filter by session properties on events table if querying sessions and joining in events

* Handle visits, pageviews, events and visitors metrics from other table

* both -> either

* events, pageviews are strictly event metrics

* Add support for (plain) breakdowns deciding which table to use

* Run tests with experimental_reduced_joins as a separate job

Also refactor which tests are run with postgres:15 to reduce number of jobs

* moduledocs for TableDecider

* Fix matrix

* Custom build name

* Move TEST_EXPERIMENTAL_REDUCED_JOINS check

* Handle percentage separately from other metrics

* Remove debug code

* TableDecider tests

* both => sample_percent

* Improve naming

* Simplify code

* Breakdowns retain old behavior if getting metric visitors

* Unify behavior of entry/exit page hostnames with rest

* Fix test naming
3 years ago
hq1 6af80dd246
Filter by hostnames (#3963)
* CH Migration: exit/entry hostnames in sessions_v2

* Leave only exit_page_hostname, we already record hostnames

* Use ClickHouse DDL in favour of ecto so that cluster is included

* Compress with ZSTD(3)

* Expose Hostname filter in the dashboard dropdown

* Add `exit_page_hostname` to ClickHouse `sessions_v2` schema

* Start tracking hostname changes in sessions

* Implement hostname filter suggestions

* Enable filtering by `event:hostname`

* Add tests for filtering by hostnames

* Ensure filter suggestions work for exit pages too

* Allow overriding hostnames with `send_pageview` mix task

* Remove `:window_time_on_page` flag

It seems that we can remove it after all?

* Initialize `experimental_hostname_filter` query parameter

* Rewrite cache store behaviour with regards to session hostnames

* Work around inconsistent session merging

So that `populate_stats` can get closer to actual ingestion

* Improve top stats test

* Make it possible to filter sessions by entry/exit hostnames

* Update pages tests

* Expose `experimental_hostname_filtering` temporarily in the UI

* Untested yet: also apply experimental filtering to sources

* Introduce `hostname_filter` feature flag

* Format

* Test top sources with hostname filter + experimental flag
3 years ago
Adrian Gruntkowski e6d83e946f
Populate new columns in imports and exports (#3969)
* Extend `Imported*` schemas with newly added columns

* Populate newly added `Imported*` fields in GA4 imports

* Extend exports with newly added fields

* Extend CSV importer to ingest new fields

* Fix alias shadowing error

* Add more extensive GA4 import fixtures

* Apply rounding and casting to sampled visits
3 years ago
RobertJoonas bd73fc8266
CH Migration - add more imported metrics and properties (#3949)
* add migration

* add utm_source to imported_sources

* quickfix

* satisfy credo

* Revert "satisfy credo"

This reverts commit bb0b228164.

* Revert "quickfix"

This reverts commit ab6f70c79e.

---------

Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
3 years ago
ruslandoga 2ba4988c95
specify insert columns in import from s3 (#3968)
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
3 years ago
hq1 1f778e0c11
CH Migration: exit page hostname on sessions_v2 (#3953)
* CH Migration: exit/entry hostnames in sessions_v2

* Leave only exit_page_hostname, we already record hostnames

* Use ClickHouse DDL in favour of ecto so that cluster is included

* Compress with ZSTD(3)
3 years ago
Adrian Gruntkowski 9f27fa303c
Fix dry run mode in `DataMigration.SiteImports` (#3965) 3 years ago
Adrian Gruntkowski 23a3699dd7
Improve import stats toggle and `with_imported` flag computation (#3960)
* Check import presence across all imports and not just the first one

Also, simplify imported data toggle rendering to not explicitly
refer to the earliest import source.

* Change imported stats toggle icon in dashboard

* Test `Imported.get_imports_date_range/1`

* Simplify failed UA/GA import email copy
3 years ago
Adrian Gruntkowski 71fe541359
Implement script for backfilling legacy site import entries and adjusting end dates of site imports (#3954)
* Always select and clear import ID 0 when referring to legacy imports

* Implement script for adding site import entries and adjusting end dates

* Log cases where end date computation is using fallback

* Don't log queries when running the migration to reduce noise
3 years ago
Adrian Gruntkowski 5bf59d1d8a
Implement adjusting imported date range to actual and existing stats (#3943)
* Implement adjusting imported date range to actual and existing stats

* Drop redundant prefix from import list entries

* Make pageview numbers in imports list formatted for readability

* Test and improve date range cropping

* DRY UA and GA4 stats start and end date API calls

* Extend UA/GA import controller tests and improve error handling

* refactor finding longest open range without existing data

* Fix typo in test description

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>

* Rename `open_ranges` to `free_ranges`

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
3 years ago

@ -15,14 +15,25 @@ env:
jobs: jobs:
build: build:
name: Build and test name: "Build and test (${{ matrix.mix_env }}, ${{ matrix.postgres_image }}${{ matrix.test_experimental_reduced_joins == '1' && ', experimental_reduced_joins' || '' }})"
runs-on: ubuntu-latest runs-on: ubuntu-latest
strategy: strategy:
matrix: matrix:
mix_env: ['test', 'small_test'] mix_env: ['test', 'small_test']
postgres_image: ['postgres:15', 'postgres:16'] postgres_image: ['postgres:16']
test_experimental_reduced_joins: ['0']
include:
- mix_env: 'test'
postgres_image: 'postgres:15'
test_experimental_reduced_joins: '0'
- mix_env: 'test'
postgres_image: 'postgres:16'
test_experimental_reduced_joins: '1'
env: env:
MIX_ENV: ${{ matrix.mix_env }} MIX_ENV: ${{ matrix.mix_env }}
TEST_EXPERIMENTAL_REDUCED_JOINS: ${{ matrix.test_experimental_reduced_joins }}
services: services:
postgres: postgres:
image: ${{ matrix.postgres_image }} image: ${{ matrix.postgres_image }}
@ -132,4 +143,4 @@ jobs:
- run: mix format --check-formatted - run: mix format --check-formatted
- run: mix deps.unlock --check-unused - run: mix deps.unlock --check-unused
- run: mix credo diff --from-git-merge-base origin/master - run: mix credo diff --from-git-merge-base origin/master
- run: mix dialyzer - run: mix dialyzer

@ -22,6 +22,7 @@ if (container) {
conversionsOptedOut: container.dataset.conversionsOptedOut === 'true', conversionsOptedOut: container.dataset.conversionsOptedOut === 'true',
funnelsOptedOut: container.dataset.funnelsOptedOut === 'true', funnelsOptedOut: container.dataset.funnelsOptedOut === 'true',
propsOptedOut: container.dataset.propsOptedOut === 'true', propsOptedOut: container.dataset.propsOptedOut === 'true',
revenueGoals: JSON.parse(container.dataset.revenueGoals),
funnels: JSON.parse(container.dataset.funnels), funnels: JSON.parse(container.dataset.funnels),
statsBegin: container.dataset.statsBegin, statsBegin: container.dataset.statsBegin,
nativeStatsBegin: container.dataset.nativeStatsBegin, nativeStatsBegin: container.dataset.nativeStatsBegin,

@ -59,6 +59,7 @@ export function parseQuery(querystring, site) {
'region': q.get('region'), 'region': q.get('region'),
'city': q.get('city'), 'city': q.get('city'),
'page': q.get('page'), 'page': q.get('page'),
'hostname': q.get('hostname'),
'entry_page': q.get('entry_page'), 'entry_page': q.get('entry_page'),
'exit_page': q.get('exit_page') 'exit_page': q.get('exit_page')
} }

@ -1,4 +1,42 @@
import numberFormatter, {durationFormatter} from '../../util/number-formatter' import numberFormatter, {durationFormatter} from '../../util/number-formatter'
import { parsePrefix } from '../../util/filters'
export function getGraphableMetrics(query, site) {
const isRealtime = query.period === 'realtime'
const goalFilter = query.filters.goal
const pageFilter = query.filters.page
if (isRealtime && goalFilter) {
return ["visitors"]
} else if (isRealtime) {
return ["visitors", "pageviews"]
} else if (goalFilter && canGraphRevenueMetrics(goalFilter, site)) {
return ["visitors", "events", "average_revenue", "total_revenue", "conversion_rate"]
} else if (goalFilter) {
return ["visitors", "events", "conversion_rate"]
} else if (pageFilter) {
return ["visitors", "visits", "pageviews", "bounce_rate", "time_on_page"]
} else {
return ["visitors", "visits", "pageviews", "views_per_visit", "bounce_rate", "visit_duration"]
}
}
// Revenue metrics can only be graphed if:
// * The query is filtered by at least one revenue goal
// * All revenue goals in filter have the same currency
function canGraphRevenueMetrics(goalFilter, site) {
const goalsInFilter = parsePrefix(goalFilter).values
const revenueGoalsInFilter = site.revenueGoals.filter((rg) => {
return goalsInFilter.includes(rg.event_name)
})
const singleCurrency = revenueGoalsInFilter.every((rg) => {
return rg.currency === revenueGoalsInFilter[0].currency
})
return revenueGoalsInFilter.length > 0 && singleCurrency
}
export const METRIC_LABELS = { export const METRIC_LABELS = {
'visitors': 'Visitors', 'visitors': 'Visitors',
@ -28,12 +66,6 @@ export const METRIC_FORMATTER = {
'average_revenue': numberFormatter, 'average_revenue': numberFormatter,
} }
export const LoadingState = {
loading: 0,
updatingGraph: 1,
loaded: 2
}
const buildComparisonDataset = function(comparisonPlot) { const buildComparisonDataset = function(comparisonPlot) {
if (!comparisonPlot) return [] if (!comparisonPlot) return []

@ -5,6 +5,7 @@ import classNames from "classnames";
import numberFormatter, { durationFormatter } from '../../util/number-formatter' import numberFormatter, { durationFormatter } from '../../util/number-formatter'
import * as storage from '../../util/storage' import * as storage from '../../util/storage'
import { formatDateRange } from '../../util/date.js' import { formatDateRange } from '../../util/date.js'
import { getGraphableMetrics } from "./graph-util.js";
function Maybe({condition, children}) { function Maybe({condition, children}) {
if (condition) { if (condition) {
@ -88,7 +89,8 @@ export default function TopStats(props) {
} }
function canMetricBeGraphed(stat) { function canMetricBeGraphed(stat) {
return !!stat.graph_metric const graphableMetrics = getGraphableMetrics(query, site)
return stat.graph_metric && graphableMetrics.includes(stat.graph_metric)
} }
function maybeUpdateMetric(stat) { function maybeUpdateMetric(stat) {

@ -1,7 +1,7 @@
import React, { useState, useEffect, useRef, useCallback } from 'react'; import React, { useState, useEffect, useRef, useCallback } from 'react';
import * as api from '../../api' import * as api from '../../api'
import * as storage from '../../util/storage' import * as storage from '../../util/storage'
import { LoadingState } from './graph-util' import { getGraphableMetrics } from './graph-util'
import TopStats from './top-stats'; import TopStats from './top-stats';
import { IntervalPicker, getCurrentInterval } from './interval-picker' import { IntervalPicker, getCurrentInterval } from './interval-picker'
import StatsExport from './stats-export' import StatsExport from './stats-export'
@ -14,7 +14,7 @@ import LineGraphWithRouter from './line-graph'
function fetchTopStats(site, query) { function fetchTopStats(site, query) {
const q = { ...query } const q = { ...query }
if (!isComparisonEnabled(q.comparison)) { if (!isComparisonEnabled(q.comparison)) {
q.comparison = 'previous_period' q.comparison = 'previous_period'
} }
@ -34,26 +34,35 @@ export default function VisitorGraph(props) {
const topStatsBoundary = useRef(null) const topStatsBoundary = useRef(null)
const [loading, setLoading] = useState(LoadingState.loading)
const [topStatData, setTopStatData] = useState(null) const [topStatData, setTopStatData] = useState(null)
const [topStatsLoading, setTopStatsLoading] = useState(true)
const [graphData, setGraphData] = useState(null) const [graphData, setGraphData] = useState(null)
const [graphLoading, setGraphLoading] = useState(true)
// This state is explicitly meant for the situation where either graph interval
// or graph metric is changed. That results in behaviour where Top Stats stay
// intact, but the graph container alone will display a loading spinner for as
// long as new graph data is fetched.
const [graphRefreshing, setGraphRefreshing] = useState(false)
const onIntervalUpdate = useCallback((newInterval) => { const onIntervalUpdate = useCallback((newInterval) => {
setGraphData(null) setGraphData(null)
setLoading(LoadingState.updatingGraph) setGraphRefreshing(true)
fetchGraphData(getStoredMetric(), newInterval) fetchGraphData(getStoredMetric(), newInterval)
}, [query]) }, [query])
const onMetricUpdate = useCallback((newMetric) => { const onMetricUpdate = useCallback((newMetric) => {
setGraphData(null) setGraphData(null)
setLoading(LoadingState.updatingGraph) setGraphRefreshing(true)
fetchGraphData(newMetric, getCurrentInterval(site, query)) fetchGraphData(newMetric, getCurrentInterval(site, query))
}, [query]) }, [query])
useEffect(() => { useEffect(() => {
setTopStatData(null) setTopStatData(null)
setTopStatsLoading(true)
setGraphData(null) setGraphData(null)
setLoading(LoadingState.loading) setGraphLoading(true)
fetchTopStatsAndGraphData() fetchTopStatsAndGraphData()
if (isRealtime) { if (isRealtime) {
@ -73,26 +82,28 @@ export default function VisitorGraph(props) {
fetchTopStats(site, query) fetchTopStats(site, query)
.then((res) => { .then((res) => {
setTopStatData(res) setTopStatData(res)
setTopStatsLoading(false)
})
let metric = getStoredMetric()
const availableMetrics = getGraphableMetrics(query, site)
if (!availableMetrics.includes(metric)) {
metric = availableMetrics[0]
storage.setItem(`metric__${site.domain}`, metric)
}
let metric = getStoredMetric() const interval = getCurrentInterval(site, query)
const availableMetrics = res.top_stats.filter(stat => !!stat.graph_metric).map(stat => stat.graph_metric)
if (!availableMetrics.includes(metric)) {
metric = availableMetrics[0]
storage.setItem(`metric__${site.domain}`, metric)
}
const interval = getCurrentInterval(site, query)
fetchGraphData(metric, interval) fetchGraphData(metric, interval)
})
} }
function fetchGraphData(metric, interval) { function fetchGraphData(metric, interval) {
fetchMainGraph(site, query, metric, interval) fetchMainGraph(site, query, metric, interval)
.then((res) => { .then((res) => {
setGraphData({...res, metric}) setGraphData(res)
setLoading(LoadingState.loaded) setGraphLoading(false)
setGraphRefreshing(false)
}) })
} }
@ -119,13 +130,13 @@ export default function VisitorGraph(props) {
return ( return (
<div className={"relative w-full mt-2 bg-white rounded shadow-xl dark:bg-gray-825"}> <div className={"relative w-full mt-2 bg-white rounded shadow-xl dark:bg-gray-825"}>
{loading === LoadingState.loading && renderLoader()} {(topStatsLoading || graphLoading) && renderLoader()}
<FadeIn show={loading !== LoadingState.loading}> <FadeIn show={!(topStatsLoading || graphLoading)}>
<div id="top-stats-container" className="flex flex-wrap" ref={topStatsBoundary} style={{ height: getTopStatsHeight() }}> <div id="top-stats-container" className="flex flex-wrap" ref={topStatsBoundary} style={{ height: getTopStatsHeight() }}>
<TopStats site={site} query={query} data={topStatData} onMetricUpdate={onMetricUpdate} tooltipBoundary={topStatsBoundary.current} lastLoadTimestamp={lastLoadTimestamp} /> <TopStats site={site} query={query} data={topStatData} onMetricUpdate={onMetricUpdate} tooltipBoundary={topStatsBoundary.current} lastLoadTimestamp={lastLoadTimestamp} />
</div> </div>
<div className="relative px-2"> <div className="relative px-2">
{loading === LoadingState.updatingGraph && renderLoader()} {graphRefreshing && renderLoader()}
<div className="absolute right-4 -top-8 py-1 flex items-center"> <div className="absolute right-4 -top-8 py-1 flex items-center">
{!isRealtime && <StatsExport site={site} query={query} />} {!isRealtime && <StatsExport site={site} query={query} />}
<SamplingNotice samplePercent={topStatData}/> <SamplingNotice samplePercent={topStatData}/>

@ -2,9 +2,10 @@ import React from "react"
import { parseNaiveDate, isBefore } from '../../util/date' import { parseNaiveDate, isBefore } from '../../util/date'
import { Link } from 'react-router-dom' import { Link } from 'react-router-dom'
import * as url from '../../util/url' import * as url from '../../util/url'
import { BarsArrowUpIcon } from '@heroicons/react/20/solid'
export default function WithImportedSwitch({site, topStatData}) { export default function WithImportedSwitch({site, topStatData}) {
if (!topStatData?.imported_source) { if (!topStatData?.imports_exist) {
return null return null
} }
@ -21,18 +22,15 @@ export default function WithImportedSwitch({site, topStatData}) {
const isComparingImportedPeriod = isBeforeNativeStats(topStatData.comparing_from) const isComparingImportedPeriod = isBeforeNativeStats(topStatData.comparing_from)
if (isQueryingImportedPeriod || isComparingImportedPeriod) { if (isQueryingImportedPeriod || isComparingImportedPeriod) {
const source = topStatData.imported_source
const withImported = topStatData.with_imported; const withImported = topStatData.with_imported;
const strike = withImported ? "" : " line-through" const toggleColor = withImported ? " dark:text-gray-300 text-gray-700" : " dark:text-gray-500 text-gray-400"
const target = url.setQuery('with_imported', !withImported) const target = url.setQuery('with_imported', !withImported)
const tip = withImported ? "" : "do not "; const tip = withImported ? "" : "do not ";
return ( return (
<Link to={target} className="w-4 h-4 mx-2"> <Link to={target} className="w-4 h-4 mx-2">
<div tooltip={`Stats ${tip}include data imported from ${source}.`} className="cursor-pointer w-4 h-4"> <div tooltip={`Stats ${tip}include imported data.`} className="cursor-pointer w-4 h-4">
<svg className="absolute dark:text-gray-300 text-gray-700" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor"> <BarsArrowUpIcon className={"absolute " + toggleColor} />
<text x="4" y="18" fontSize="24" fill="currentColor" className={"text-gray-700 dark:text-gray-300" + strike}>{source[0].toUpperCase()}</text>
</svg>
</div> </div>
</Link> </Link>
) )

@ -1,3 +1,5 @@
const flags = JSON.parse(document.getElementById('stats-react-container').dataset.flags)
export const FILTER_GROUPS = { export const FILTER_GROUPS = {
'page': ['page', 'entry_page', 'exit_page'], 'page': ['page', 'entry_page', 'exit_page'],
'source': ['source', 'referrer'], 'source': ['source', 'referrer'],
@ -7,7 +9,9 @@ export const FILTER_GROUPS = {
'os': ['os', 'os_version'], 'os': ['os', 'os_version'],
'utm': ['utm_medium', 'utm_source', 'utm_campaign', 'utm_term', 'utm_content'], 'utm': ['utm_medium', 'utm_source', 'utm_campaign', 'utm_term', 'utm_content'],
'goal': ['goal'], 'goal': ['goal'],
'props': ['prop_key', 'prop_value'] 'props': ['prop_key', 'prop_value'],
...(flags.hostname_filter ? { 'hostname': ['hostname'] } : {})
} }
export const NO_CONTAINS_OPERATOR = new Set(['goal', 'screen'].concat(FILTER_GROUPS['location'])) export const NO_CONTAINS_OPERATOR = new Set(['goal', 'screen'].concat(FILTER_GROUPS['location']))
@ -37,7 +41,7 @@ export function isFreeChoiceFilter(filterName) {
let NON_ESCAPED_PIPE_REGEX; let NON_ESCAPED_PIPE_REGEX;
try { try {
NON_ESCAPED_PIPE_REGEX = new RegExp("(?<!\\\\)\\|", "g") NON_ESCAPED_PIPE_REGEX = new RegExp("(?<!\\\\)\\|", "g")
} catch(_e) { } catch (_e) {
NON_ESCAPED_PIPE_REGEX = '|' NON_ESCAPED_PIPE_REGEX = '|'
} }
@ -64,19 +68,19 @@ export function parsePrefix(rawValue) {
.filter((clause) => !!clause) .filter((clause) => !!clause)
.map((val) => val.replaceAll(ESCAPED_PIPE, '|')) .map((val) => val.replaceAll(ESCAPED_PIPE, '|'))
return {type, values} return { type, values }
} }
export function parseQueryPropsFilter(query) { export function parseQueryPropsFilter(query) {
return Object.entries(query.filters['props']).map(([key, propVal]) => { return Object.entries(query.filters['props']).map(([key, propVal]) => {
const {type, values} = parsePrefix(propVal) const { type, values } = parsePrefix(propVal)
const clauses = values.map(val => { return {value: val, label: val}}) const clauses = values.map(val => { return { value: val, label: val } })
return { propKey: { label: key, value: key }, type, clauses } return { propKey: { label: key, value: key }, type, clauses }
}) })
} }
export function parseQueryFilter(query, filter) { export function parseQueryFilter(query, filter) {
const {type, values} = parsePrefix(query.filters[filter] || '') const { type, values } = parsePrefix(query.filters[filter] || '')
let labels = values let labels = values
@ -95,9 +99,9 @@ export function parseQueryFilter(query, filter) {
labels = rawLabel.split('|').filter(label => !!label) labels = rawLabel.split('|').filter(label => !!label)
} }
const clauses = values.map((value, index) => { return {value, label: labels[index]}}) const clauses = values.map((value, index) => { return { value, label: labels[index] } })
return {type, clauses} return { type, clauses }
} }
export function isFilteringOnFixedValue(query, filter) { export function isFilteringOnFixedValue(query, filter) {
@ -152,6 +156,7 @@ export const formattedFilters = {
'region': 'Region', 'region': 'Region',
'city': 'City', 'city': 'City',
'page': 'Page', 'page': 'Page',
'hostname': 'Hostname',
'entry_page': 'Entry Page', 'entry_page': 'Entry Page',
'exit_page': 'Exit Page' 'exit_page': 'Exit Page',
} }

@ -38,7 +38,7 @@ config :tailwind,
config :ua_inspector, config :ua_inspector,
database_path: "priv/ua_inspector", database_path: "priv/ua_inspector",
remote_release: "6.2.1" remote_release: "6.3.0"
config :ref_inspector, config :ref_inspector,
database_path: "priv/ref_inspector" database_path: "priv/ref_inspector"

@ -0,0 +1,38 @@
{
"kind": "analyticsData#batchRunReports",
"reports": [
{
"dimensionHeaders": [
{
"name": "date"
}
],
"kind": "analyticsData#runReport",
"metadata": {
"currencyCode": "USD",
"timeZone": "Europe/Warsaw"
},
"metricHeaders": [
{
"name": "screenPageViews",
"type": "TYPE_INTEGER"
}
],
"rowCount": 3,
"rows": [
{
"dimensionValues": [
{
"value": "20240302"
}
],
"metricValues": [
{
"value": "13"
}
]
}
]
}
]
}

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

@ -10,7 +10,7 @@
"kind": "analyticsData#runReport", "kind": "analyticsData#runReport",
"metadata": { "metadata": {
"currencyCode": "USD", "currencyCode": "USD",
"timeZone": "Europe/Warsaw" "timeZone": "Europe/Amsterdam"
}, },
"metricHeaders": [ "metricHeaders": [
{ {
@ -34,77 +34,749 @@
"type": "TYPE_SECONDS" "type": "TYPE_SECONDS"
} }
], ],
"rowCount": 3, "rowCount": 31,
"rows": [ "rows": [
{ {
"dimensionValues": [ "dimensionValues": [
{ {
"value": "20240226" "value": "20240131"
} }
], ],
"metricValues": [ "metricValues": [
{ {
"value": "1" "value": "141"
}, },
{ {
"value": "1" "value": "195"
}, },
{ {
"value": "1" "value": "56"
}, },
{ {
"value": "1" "value": "146"
}, },
{ {
"value": "0" "value": "4815"
} }
] ]
}, },
{ {
"dimensionValues": [ "dimensionValues": [
{ {
"value": "20240224" "value": "20240130"
} }
], ],
"metricValues": [ "metricValues": [
{ {
"value": "2" "value": "115"
}, },
{ {
"value": "2" "value": "171"
}, },
{ {
"value": "2" "value": "42"
}, },
{ {
"value": "2" "value": "115"
}, },
{ {
"value": "0" "value": "3481"
} }
] ]
}, },
{ {
"dimensionValues": [ "dimensionValues": [
{ {
"value": "20240222" "value": "20240129"
} }
], ],
"metricValues": [ "metricValues": [
{ {
"value": "3" "value": "105"
}, },
{ {
"value": "13" "value": "153"
}, },
{ {
"value": "0" "value": "54"
}, },
{ {
"value": "4" "value": "113"
}, },
{ {
"value": "29" "value": "2895"
}
]
},
{
"dimensionValues": [
{
"value": "20240128"
}
],
"metricValues": [
{
"value": "184"
},
{
"value": "259"
},
{
"value": "74"
},
{
"value": "190"
},
{
"value": "6861"
}
]
},
{
"dimensionValues": [
{
"value": "20240127"
}
],
"metricValues": [
{
"value": "155"
},
{
"value": "202"
},
{
"value": "67"
},
{
"value": "158"
},
{
"value": "5259"
}
]
},
{
"dimensionValues": [
{
"value": "20240126"
}
],
"metricValues": [
{
"value": "132"
},
{
"value": "189"
},
{
"value": "57"
},
{
"value": "142"
},
{
"value": "5545"
}
]
},
{
"dimensionValues": [
{
"value": "20240125"
}
],
"metricValues": [
{
"value": "151"
},
{
"value": "229"
},
{
"value": "71"
},
{
"value": "155"
},
{
"value": "5365"
}
]
},
{
"dimensionValues": [
{
"value": "20240124"
}
],
"metricValues": [
{
"value": "137"
},
{
"value": "221"
},
{
"value": "64"
},
{
"value": "148"
},
{
"value": "4454"
}
]
},
{
"dimensionValues": [
{
"value": "20240123"
}
],
"metricValues": [
{
"value": "166"
},
{
"value": "267"
},
{
"value": "61"
},
{
"value": "171"
},
{
"value": "4850"
}
]
},
{
"dimensionValues": [
{
"value": "20240122"
}
],
"metricValues": [
{
"value": "161"
},
{
"value": "233"
},
{
"value": "73"
},
{
"value": "165"
},
{
"value": "4871"
}
]
},
{
"dimensionValues": [
{
"value": "20240121"
}
],
"metricValues": [
{
"value": "254"
},
{
"value": "397"
},
{
"value": "114"
},
{
"value": "267"
},
{
"value": "8845"
}
]
},
{
"dimensionValues": [
{
"value": "20240120"
}
],
"metricValues": [
{
"value": "214"
},
{
"value": "383"
},
{
"value": "100"
},
{
"value": "223"
},
{
"value": "6914"
}
]
},
{
"dimensionValues": [
{
"value": "20240119"
}
],
"metricValues": [
{
"value": "381"
},
{
"value": "5571"
},
{
"value": "100"
},
{
"value": "228"
},
{
"value": "98549"
}
]
},
{
"dimensionValues": [
{
"value": "20240118"
}
],
"metricValues": [
{
"value": "177"
},
{
"value": "278"
},
{
"value": "86"
},
{
"value": "186"
},
{
"value": "5553"
}
]
},
{
"dimensionValues": [
{
"value": "20240117"
}
],
"metricValues": [
{
"value": "204"
},
{
"value": "271"
},
{
"value": "97"
},
{
"value": "216"
},
{
"value": "6777"
}
]
},
{
"dimensionValues": [
{
"value": "20240116"
}
],
"metricValues": [
{
"value": "200"
},
{
"value": "280"
},
{
"value": "65"
},
{
"value": "203"
},
{
"value": "9381"
}
]
},
{
"dimensionValues": [
{
"value": "20240115"
}
],
"metricValues": [
{
"value": "255"
},
{
"value": "311"
},
{
"value": "86"
},
{
"value": "269"
},
{
"value": "12365"
}
]
},
{
"dimensionValues": [
{
"value": "20240114"
}
],
"metricValues": [
{
"value": "243"
},
{
"value": "302"
},
{
"value": "86"
},
{
"value": "257"
},
{
"value": "11073"
}
]
},
{
"dimensionValues": [
{
"value": "20240113"
}
],
"metricValues": [
{
"value": "231"
},
{
"value": "276"
},
{
"value": "92"
},
{
"value": "244"
},
{
"value": "10084"
}
]
},
{
"dimensionValues": [
{
"value": "20240112"
}
],
"metricValues": [
{
"value": "215"
},
{
"value": "265"
},
{
"value": "75"
},
{
"value": "225"
},
{
"value": "10195"
}
]
},
{
"dimensionValues": [
{
"value": "20240111"
}
],
"metricValues": [
{
"value": "183"
},
{
"value": "235"
},
{
"value": "64"
},
{
"value": "191"
},
{
"value": "8200"
}
]
},
{
"dimensionValues": [
{
"value": "20240110"
}
],
"metricValues": [
{
"value": "190"
},
{
"value": "244"
},
{
"value": "61"
},
{
"value": "203"
},
{
"value": "9056"
}
]
},
{
"dimensionValues": [
{
"value": "20240109"
}
],
"metricValues": [
{
"value": "201"
},
{
"value": "290"
},
{
"value": "63"
},
{
"value": "213"
},
{
"value": "11651"
}
]
},
{
"dimensionValues": [
{
"value": "20240108"
}
],
"metricValues": [
{
"value": "195"
},
{
"value": "235"
},
{
"value": "71"
},
{
"value": "209"
},
{
"value": "7946"
}
]
},
{
"dimensionValues": [
{
"value": "20240107"
}
],
"metricValues": [
{
"value": "205"
},
{
"value": "274"
},
{
"value": "80"
},
{
"value": "218"
},
{
"value": "10048"
}
]
},
{
"dimensionValues": [
{
"value": "20240106"
}
],
"metricValues": [
{
"value": "256"
},
{
"value": "303"
},
{
"value": "119"
},
{
"value": "281"
},
{
"value": "8440"
}
]
},
{
"dimensionValues": [
{
"value": "20240105"
}
],
"metricValues": [
{
"value": "226"
},
{
"value": "270"
},
{
"value": "83"
},
{
"value": "237"
},
{
"value": "9311"
}
]
},
{
"dimensionValues": [
{
"value": "20240104"
}
],
"metricValues": [
{
"value": "223"
},
{
"value": "274"
},
{
"value": "81"
},
{
"value": "236"
},
{
"value": "12051"
}
]
},
{
"dimensionValues": [
{
"value": "20240103"
}
],
"metricValues": [
{
"value": "233"
},
{
"value": "283"
},
{
"value": "79"
},
{
"value": "245"
},
{
"value": "10628"
}
]
},
{
"dimensionValues": [
{
"value": "20240102"
}
],
"metricValues": [
{
"value": "203"
},
{
"value": "269"
},
{
"value": "60"
},
{
"value": "210"
},
{
"value": "11168"
}
]
},
{
"dimensionValues": [
{
"value": "20240101"
}
],
"metricValues": [
{
"value": "191"
},
{
"value": "224"
},
{
"value": "71"
},
{
"value": "197"
},
{
"value": "8080"
} }
] ]
} }

@ -0,0 +1,38 @@
{
"reports": [
{
"columnHeader": {
"dimensions": [
"ga:date"
],
"metricHeader": {
"metricHeaderEntries": [
{
"name": "ga:pageviews",
"type": "INTEGER"
}
]
}
},
"data": {
"isDataGolden": true,
"rowCount": 849,
"rows": [
{
"dimensions": [
"20200421"
],
"metrics": [
{
"values": [
"37"
]
}
]
}
]
},
"nextPageToken": "1"
}
]
}

@ -0,0 +1,38 @@
{
"reports": [
{
"columnHeader": {
"dimensions": [
"ga:date"
],
"metricHeader": {
"metricHeaderEntries": [
{
"name": "ga:pageviews",
"type": "INTEGER"
}
]
}
},
"data": {
"isDataGolden": true,
"rowCount": 849,
"rows": [
{
"dimensions": [
"20170118"
],
"metrics": [
{
"values": [
"37"
]
}
]
}
]
},
"nextPageToken": "1"
}
]
}

@ -24,6 +24,7 @@ defmodule Mix.Tasks.SendPageview do
page: :string, page: :string,
referrer: :string, referrer: :string,
host: :string, host: :string,
hostname: :string,
event: :string, event: :string,
props: :string, props: :string,
revenue_currency: :string, revenue_currency: :string,
@ -86,6 +87,7 @@ defmodule Mix.Tasks.SendPageview do
referrer = Keyword.get(opts, :referrer, @default_referrer) referrer = Keyword.get(opts, :referrer, @default_referrer)
event = Keyword.get(opts, :event, @default_event) event = Keyword.get(opts, :event, @default_event)
props = Keyword.get(opts, :props, @default_props) props = Keyword.get(opts, :props, @default_props)
hostname = Keyword.get(opts, :hostname, domain)
revenue = revenue =
if Keyword.get(opts, :revenue_currency) do if Keyword.get(opts, :revenue_currency) do
@ -97,7 +99,7 @@ defmodule Mix.Tasks.SendPageview do
%{ %{
name: event, name: event,
url: "http://#{domain}#{page}", url: "http://#{hostname}#{page}",
domain: domain, domain: domain,
referrer: referrer, referrer: referrer,
props: props, props: props,

@ -65,4 +65,28 @@ defmodule Plausible.ClickhouseEventV2 do
) )
|> validate_required([:name, :site_id, :hostname, :pathname, :user_id, :timestamp]) |> validate_required([:name, :site_id, :hostname, :pathname, :user_id, :timestamp])
end end
@session_properties [
:session_id,
:referrer,
:referrer_source,
:utm_medium,
:utm_source,
:utm_campaign,
:utm_content,
:utm_term,
:country_code,
:subdivision1_code,
:subdivision2_code,
:city_geoname_id,
:screen_size,
:operating_system,
:operating_system_version,
:browser,
:browser_version
]
def merge_session(%__MODULE__{} = event, session) do
Map.merge(event, Map.take(session, @session_properties))
end
end end

@ -43,6 +43,7 @@ defmodule Plausible.ClickhouseSessionV2 do
field :is_bounce, BoolUInt8 field :is_bounce, BoolUInt8
field :entry_page, :string field :entry_page, :string
field :exit_page, :string field :exit_page, :string
field :exit_page_hostname, :string
field :pageviews, Ch, type: "Int32" field :pageviews, Ch, type: "Int32"
field :events, Ch, type: "Int32" field :events, Ch, type: "Int32"
field :sign, Ch, type: "Int8" field :sign, Ch, type: "Int8"

@ -0,0 +1,188 @@
defmodule Plausible.DataMigration.SiteImports do
@moduledoc """
Site imports migration backfilling SiteImport entries for old imports
and alters import end dates to match actual end date of respective import stats.
"""
import Ecto.Query
alias Plausible.ClickhouseRepo
alias Plausible.Imported
alias Plausible.Repo
alias Plausible.Site
def run(opts \\ []) do
dry_run? = Keyword.get(opts, :dry_run?, true)
site_import_query =
from(i in Imported.SiteImport, where: i.site_id == parent_as(:site).id, select: 1)
sites_with_imports =
from(s in Site, as: :site, where: not is_nil(s.imported_data) or exists(site_import_query))
|> Repo.all(log: false)
sites_count = length(sites_with_imports)
IO.puts("Processing #{sites_count} sites with imports (DRY RUN: #{dry_run?})...")
for {site, idx} <- Enum.with_index(sites_with_imports) do
site_imports =
from(i in Imported.SiteImport, where: i.site_id == ^site.id)
|> Repo.all(log: false)
IO.puts(
"Processing site ID #{site.id} (#{idx + 1} / #{sites_count}) (imported_data: #{is_struct(site.imported_data)}, site_imports: #{length(site_imports)})"
)
site_imports =
if site.imported_data && not Enum.any?(site_imports, & &1.legacy) do
IO.puts("Creating legacy site import entry for site ID #{site.id}")
# create legacy entry if there's not one yet
params =
site.imported_data
|> Imported.SiteImport.from_legacy()
|> Map.put(:site_id, site.id)
|> Map.take([:legacy, :start_date, :end_date, :source, :status, :site_id])
legacy_site_import =
%Imported.SiteImport{}
|> Ecto.Changeset.change(params)
|> insert!(dry_run?)
[legacy_site_import | site_imports]
else
IO.puts("Legacy site import entry for site ID #{site.id} already exists")
site_imports
end
# adjust end date for each site import
for site_import <- site_imports do
IO.puts(
"Adjusting end date for site import #{site_import.id} (site ID #{site.id}, start date: #{site_import.start_date}, end date: #{site_import.end_date})"
)
import_ids =
if site_import.legacy do
[0, site_import.id]
else
[site_import.id]
end
end_date = imported_stats_end_date(site.id, import_ids)
if !end_date do
IO.puts(
"Site import #{site_import.id} (site ID #{site.id}) does not have any recorded stats. Removing it."
)
if site_import.legacy do
clear_imported_data(site, dry_run?)
end
delete!(site_import, dry_run?)
else
end_date =
if Date.compare(end_date, site_import.end_date) in [:lt, :eq] do
end_date
else
IO.puts(
"Site import #{site_import.id} (site ID #{site.id}) computed end date is later than the current one. Skipping."
)
site_import.end_date
end
site_import
|> Ecto.Changeset.change(end_date: end_date)
|> update!(dry_run?)
IO.puts(
"End date of site import #{site_import.id} (site ID #{site.id}) adjusted to #{end_date}"
)
end
end
IO.puts("Done processing site ID #{site.id}")
end
IO.puts("Finished")
end
# Exposed for testing purposes
@doc false
def imported_stats_end_date(site_id, import_ids) do
[first_schema | schemas] = Imported.schemas()
query =
Enum.reduce(schemas, max_date_query(first_schema, site_id, import_ids), fn schema, query ->
from(s in subquery(union_all(query, ^max_date_query(schema, site_id, import_ids))))
end)
dates = ClickhouseRepo.all(from(q in query, select: q.max_date), log: false)
if dates != [] do
case Enum.max(dates, Date) do
# no stats for this domain yet
~D[1970-01-01] ->
nil
date ->
date
end
else
nil
end
end
defp insert!(changeset, false = _dry_run?) do
Repo.insert!(changeset)
end
defp insert!(changeset, true = _dry_run?) do
if changeset.valid? do
changeset
|> Ecto.Changeset.change(id: 0)
|> Ecto.Changeset.apply_changes()
else
raise "Invalid insert: #{inspect(changeset)}"
end
end
defp clear_imported_data(site, false = _dry_run?) do
Repo.update_all(from(s in Site, where: s.id == ^site.id), set: [imported_data: nil])
end
defp clear_imported_data(site, true = _dry_run?) do
%{site | imported_data: nil}
end
defp update!(changeset, false = _dry_run?) do
Repo.update!(changeset)
end
defp update!(changeset, true = _dry_run?) do
if changeset.valid? do
Ecto.Changeset.apply_changes(changeset)
else
raise "Invalid update: #{inspect(changeset)}"
end
end
defp delete!(entity, false = _dry_run?) do
Repo.delete!(entity)
end
defp delete!(entity, true = _dry_run?) do
entity
end
defp max_date_query(schema, site_id, import_ids) do
from(q in schema,
where: q.site_id == ^site_id,
where: q.import_id in ^import_ids,
select: %{max_date: fragment("max(?)", q.date)}
)
end
end

@ -118,51 +118,27 @@ defmodule Plausible.Exports do
end end
end end
defmacrop pageviews(t) do
quote do
selected_as(
fragment("greatest(sum(?*?),0)", unquote(t).sign, unquote(t).pageviews),
:pageviews
)
end
end
@spec export_visitors_q(pos_integer) :: Ecto.Query.t() @spec export_visitors_q(pos_integer) :: Ecto.Query.t()
def export_visitors_q(site_id) do def export_visitors_q(site_id) do
visitors_sessions_q = from s in sampled("sessions_v2"),
from s in sampled("sessions_v2"), where: s.site_id == ^site_id,
where: s.site_id == ^site_id, group_by: selected_as(:date),
group_by: selected_as(:date),
select: %{
date: date(s.start),
bounces: bounces(s),
visits: visits(s),
visit_duration: visit_duration(s)
# NOTE: can we use just sessions_v2 table in this query? sum(pageviews) and visitors(s)?
# visitors: visitors(s)
}
visitors_events_q =
from e in sampled("events_v2"),
where: e.site_id == ^site_id,
group_by: selected_as(:date),
select: %{
date: date(e.timestamp),
visitors: visitors(e),
pageviews:
selected_as(
fragment("toUInt64(round(countIf(?='pageview')*any(_sample_factor)))", e.name),
:pageviews
)
}
visitors_q =
"e"
|> with_cte("e", as: ^visitors_events_q)
|> with_cte("s", as: ^visitors_sessions_q)
from e in visitors_q,
full_join: s in "s",
on: e.date == s.date,
order_by: selected_as(:date),
select: [ select: [
selected_as(fragment("greatest(?,?)", s.date, e.date), :date), date(s.start),
e.visitors, visitors(s),
e.pageviews, pageviews(s),
s.bounces, bounces(s),
s.visits, visits(s),
s.visit_duration visit_duration(s)
] ]
end end
@ -173,6 +149,8 @@ defmodule Plausible.Exports do
group_by: [ group_by: [
selected_as(:date), selected_as(:date),
selected_as(:source), selected_as(:source),
s.referrer,
s.utm_source,
s.utm_medium, s.utm_medium,
s.utm_campaign, s.utm_campaign,
s.utm_content, s.utm_content,
@ -182,10 +160,13 @@ defmodule Plausible.Exports do
select: [ select: [
date(s.start), date(s.start),
selected_as(s.referrer_source, :source), selected_as(s.referrer_source, :source),
s.referrer,
s.utm_source,
s.utm_medium, s.utm_medium,
s.utm_campaign, s.utm_campaign,
s.utm_content, s.utm_content,
s.utm_term, s.utm_term,
pageviews(s),
visitors(s), visitors(s),
visits(s), visits(s),
visit_duration(s), visit_duration(s),
@ -210,6 +191,7 @@ defmodule Plausible.Exports do
hostname: e.hostname, hostname: e.hostname,
name: e.name, name: e.name,
user_id: e.user_id, user_id: e.user_id,
session_id: e.session_id,
_sample_factor: fragment("_sample_factor") _sample_factor: fragment("_sample_factor")
} }
@ -220,6 +202,10 @@ defmodule Plausible.Exports do
date(e.timestamp), date(e.timestamp),
selected_as(fragment("any(?)", e.hostname), :hostname), selected_as(fragment("any(?)", e.hostname), :hostname),
selected_as(e.pathname, :page), selected_as(e.pathname, :page),
selected_as(
fragment("toUInt64(round(uniq(?)*any(_sample_factor)))", e.session_id),
:visits
),
visitors(e), visitors(e),
selected_as( selected_as(
fragment("toUInt64(round(countIf(?='pageview')*any(_sample_factor)))", e.name), fragment("toUInt64(round(countIf(?='pageview')*any(_sample_factor)))", e.name),
@ -252,7 +238,8 @@ defmodule Plausible.Exports do
:entrances :entrances
), ),
visit_duration(s), visit_duration(s),
bounces(s) bounces(s),
pageviews(s)
] ]
end end
@ -266,10 +253,13 @@ defmodule Plausible.Exports do
date(s.start), date(s.start),
s.exit_page, s.exit_page,
visitors(s), visitors(s),
visit_duration(s),
selected_as( selected_as(
fragment("toUInt64(round(sum(?)*any(_sample_factor)))", s.sign), fragment("toUInt64(round(sum(?)*any(_sample_factor)))", s.sign),
:exits :exits
) ),
bounces(s),
pageviews(s)
] ]
end end
@ -288,7 +278,8 @@ defmodule Plausible.Exports do
visitors(s), visitors(s),
visits(s), visits(s),
visit_duration(s), visit_duration(s),
bounces(s) bounces(s),
pageviews(s)
] ]
end end
@ -304,7 +295,8 @@ defmodule Plausible.Exports do
visitors(s), visitors(s),
visits(s), visits(s),
visit_duration(s), visit_duration(s),
bounces(s) bounces(s),
pageviews(s)
] ]
end end
@ -312,15 +304,17 @@ defmodule Plausible.Exports do
def export_browsers_q(site_id) do def export_browsers_q(site_id) do
from s in sampled("sessions_v2"), from s in sampled("sessions_v2"),
where: s.site_id == ^site_id, where: s.site_id == ^site_id,
group_by: [selected_as(:date), s.browser], group_by: [selected_as(:date), s.browser, s.browser_version],
order_by: selected_as(:date), order_by: selected_as(:date),
select: [ select: [
date(s.start), date(s.start),
s.browser, s.browser,
s.browser_version,
visitors(s), visitors(s),
visits(s), visits(s),
visit_duration(s), visit_duration(s),
bounces(s) bounces(s),
pageviews(s)
] ]
end end
@ -328,15 +322,17 @@ defmodule Plausible.Exports do
def export_operating_systems_q(site_id) do def export_operating_systems_q(site_id) do
from s in sampled("sessions_v2"), from s in sampled("sessions_v2"),
where: s.site_id == ^site_id, where: s.site_id == ^site_id,
group_by: [selected_as(:date), s.operating_system], group_by: [selected_as(:date), s.operating_system, s.operating_system_version],
order_by: selected_as(:date), order_by: selected_as(:date),
select: [ select: [
date(s.start), date(s.start),
s.operating_system, s.operating_system,
s.operating_system_version,
visitors(s), visitors(s),
visits(s), visits(s),
visit_duration(s), visit_duration(s),
bounces(s) bounces(s),
pageviews(s)
] ]
end end

@ -80,6 +80,14 @@ defmodule Plausible.Goals do
def find_or_create(_, %{"goal_type" => "page"}), do: {:missing, "page_path"} def find_or_create(_, %{"goal_type" => "page"}), do: {:missing, "page_path"}
def list_revenue_goals(site) do
from(g in Plausible.Goal,
where: g.site_id == ^site.id and not is_nil(g.currency),
select: %{event_name: g.event_name, currency: g.currency}
)
|> Plausible.Repo.all()
end
def for_site(site, opts \\ []) do def for_site(site, opts \\ []) do
site site
|> for_site_query(opts) |> for_site_query(opts)

@ -55,6 +55,14 @@ defmodule Plausible.Google.API do
end end
end end
def get_analytics_end_date(access_token, property_or_view) do
if property?(property_or_view) do
Plausible.Google.GA4.API.get_analytics_end_date(access_token, property_or_view)
else
Plausible.Google.UA.API.get_analytics_end_date(access_token, property_or_view)
end
end
def fetch_verified_properties(auth) do def fetch_verified_properties(auth) do
with {:ok, access_token} <- maybe_refresh_token(auth), with {:ok, access_token} <- maybe_refresh_token(auth),
{:ok, sites} <- Plausible.Google.HTTP.list_sites(access_token) do {:ok, sites} <- Plausible.Google.HTTP.list_sites(access_token) do

@ -64,6 +64,10 @@ defmodule Plausible.Google.GA4.API do
GA4.HTTP.get_analytics_start_date(access_token, property) GA4.HTTP.get_analytics_start_date(access_token, property)
end end
def get_analytics_end_date(access_token, property) do
GA4.HTTP.get_analytics_end_date(access_token, property)
end
def import_analytics(date_range, property, auth, persist_fn) do def import_analytics(date_range, property, auth, persist_fn) do
Logger.debug( Logger.debug(
"[#{inspect(__MODULE__)}:#{property}] Starting import from #{date_range.first} to #{date_range.last}" "[#{inspect(__MODULE__)}:#{property}] Starting import from #{date_range.first} to #{date_range.last}"

@ -140,8 +140,11 @@ defmodule Plausible.Google.GA4.HTTP do
{:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [401, 403] -> {:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [401, 403] ->
{:error, :authentication_failed} {:error, :authentication_failed}
{:error, %HTTPClient.Non200Error{} = error} -> {:error, %{reason: :timeout}} ->
Sentry.capture_message("Error listing Google accounts for user", extra: %{error: error}) {:error, :timeout}
{:error, error} ->
Sentry.capture_message("Error listing GA4 accounts for user", extra: %{error: error})
{:error, :unknown} {:error, :unknown}
end end
end end
@ -161,8 +164,11 @@ defmodule Plausible.Google.GA4.HTTP do
{:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [404] -> {:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [404] ->
{:error, :not_found} {:error, :not_found}
{:error, %HTTPClient.Non200Error{} = error} -> {:error, %{reason: :timeout}} ->
Sentry.capture_message("Error retrieving Google property #{property}", {:error, :timeout}
{:error, error} ->
Sentry.capture_message("Error retrieving GA4 property #{property}",
extra: %{error: error} extra: %{error: error}
) )
@ -171,7 +177,18 @@ defmodule Plausible.Google.GA4.HTTP do
end end
@earliest_valid_date "2015-08-14" @earliest_valid_date "2015-08-14"
def get_analytics_start_date(access_token, property) do def get_analytics_start_date(access_token, property) do
get_analytics_boundary_date(access_token, property, :start)
end
def get_analytics_end_date(access_token, property) do
get_analytics_boundary_date(access_token, property, :end)
end
defp get_analytics_boundary_date(access_token, property, edge) do
descending? = edge == :end
params = %{ params = %{
requests: [ requests: [
%{ %{
@ -182,7 +199,7 @@ defmodule Plausible.Google.GA4.HTTP do
dimensions: [%{name: "date"}], dimensions: [%{name: "date"}],
metrics: [%{name: "screenPageViews"}], metrics: [%{name: "screenPageViews"}],
orderBys: [ orderBys: [
%{dimension: %{dimensionName: "date", orderType: "ALPHANUMERIC"}, desc: false} %{dimension: %{dimensionName: "date", orderType: "ALPHANUMERIC"}, desc: descending?}
], ],
limit: 1 limit: 1
} }
@ -207,13 +224,18 @@ defmodule Plausible.Google.GA4.HTTP do
{:ok, date} {:ok, date}
{:error, %{reason: %Finch.Response{body: body}}} -> {:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [401, 403] ->
Sentry.capture_message("Error fetching GA4 start date", extra: %{body: inspect(body)}) {:error, :authentication_failed}
{:error, body}
{:error, %{reason: reason} = e} -> {:error, %{reason: :timeout}} ->
Sentry.capture_message("Error fetching GA4 start date", extra: %{error: inspect(e)}) {:error, :timeout}
{:error, reason}
{:error, error} ->
Sentry.capture_message("Error retrieving GA4 #{edge} date",
extra: %{error: error}
)
{:error, :unknown}
end end
end end

@ -46,9 +46,11 @@ defmodule Plausible.Google.GA4.ReportRequest do
"sessionMedium", "sessionMedium",
"sessionCampaignName", "sessionCampaignName",
"sessionManualAdContent", "sessionManualAdContent",
"sessionGoogleAdsKeyword" "sessionGoogleAdsKeyword",
"pageReferrer"
], ],
metrics: [ metrics: [
"screenPageViews",
"totalUsers", "totalUsers",
"sessions", "sessions",
"bounces = sessions - engagedSessions", "bounces = sessions - engagedSessions",
@ -59,12 +61,13 @@ defmodule Plausible.Google.GA4.ReportRequest do
dataset: "imported_pages", dataset: "imported_pages",
dimensions: ["date", "hostName", "pagePath"], dimensions: ["date", "hostName", "pagePath"],
# NOTE: no exits as GA4 DATA API does not provide that metric # NOTE: no exits as GA4 DATA API does not provide that metric
metrics: ["totalUsers", "screenPageViews", "userEngagementDuration"] metrics: ["totalUsers", "screenPageViews", "sessions", "userEngagementDuration"]
}, },
%__MODULE__{ %__MODULE__{
dataset: "imported_entry_pages", dataset: "imported_entry_pages",
dimensions: ["date", "landingPage"], dimensions: ["date", "landingPage"],
metrics: [ metrics: [
"screenPageViews",
"totalUsers", "totalUsers",
"sessions", "sessions",
"userEngagementDuration", "userEngagementDuration",
@ -75,12 +78,19 @@ defmodule Plausible.Google.GA4.ReportRequest do
# %__MODULE__{ # %__MODULE__{
# dataset: "imported_exit_pages", # dataset: "imported_exit_pages",
# dimensions: ["date", "ga:exitPagePath"], # dimensions: ["date", "ga:exitPagePath"],
# metrics: ["totalUsers", "sessions"] # metrics: [
# "totalUsers",
# "sessions",
# "screenPageViews",
# "userEngagementDuration",
# "bounces = sessions - engagedSessions"
# ]
# }, # },
%__MODULE__{ %__MODULE__{
dataset: "imported_locations", dataset: "imported_locations",
dimensions: ["date", "countryId", "region", "city"], dimensions: ["date", "countryId", "region", "city"],
metrics: [ metrics: [
"screenPageViews",
"totalUsers", "totalUsers",
"sessions", "sessions",
"bounces = sessions - engagedSessions", "bounces = sessions - engagedSessions",
@ -91,6 +101,7 @@ defmodule Plausible.Google.GA4.ReportRequest do
dataset: "imported_devices", dataset: "imported_devices",
dimensions: ["date", "deviceCategory"], dimensions: ["date", "deviceCategory"],
metrics: [ metrics: [
"screenPageViews",
"totalUsers", "totalUsers",
"sessions", "sessions",
"bounces = sessions - engagedSessions", "bounces = sessions - engagedSessions",
@ -101,6 +112,7 @@ defmodule Plausible.Google.GA4.ReportRequest do
dataset: "imported_browsers", dataset: "imported_browsers",
dimensions: ["date", "browser"], dimensions: ["date", "browser"],
metrics: [ metrics: [
"screenPageViews",
"totalUsers", "totalUsers",
"sessions", "sessions",
"bounces = sessions - engagedSessions", "bounces = sessions - engagedSessions",
@ -109,8 +121,9 @@ defmodule Plausible.Google.GA4.ReportRequest do
}, },
%__MODULE__{ %__MODULE__{
dataset: "imported_operating_systems", dataset: "imported_operating_systems",
dimensions: ["date", "operatingSystem"], dimensions: ["date", "operatingSystem", "operatingSystemVersion"],
metrics: [ metrics: [
"screenPageViews",
"totalUsers", "totalUsers",
"sessions", "sessions",
"bounces = sessions - engagedSessions", "bounces = sessions - engagedSessions",

@ -64,6 +64,10 @@ defmodule Plausible.Google.UA.API do
UA.HTTP.get_analytics_start_date(access_token, view_id) UA.HTTP.get_analytics_start_date(access_token, view_id)
end end
def get_analytics_end_date(access_token, view_id) do
UA.HTTP.get_analytics_end_date(access_token, view_id)
end
@spec import_analytics(Date.Range.t(), String.t(), import_auth(), (String.t(), [map()] -> :ok)) :: @spec import_analytics(Date.Range.t(), String.t(), import_auth(), (String.t(), [map()] -> :ok)) ::
:ok | {:error, term()} :ok | {:error, term()}
@doc """ @doc """

@ -108,14 +108,33 @@ defmodule Plausible.Google.UA.HTTP do
{:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [401, 403] -> {:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [401, 403] ->
{:error, :authentication_failed} {:error, :authentication_failed}
{:error, %HTTPClient.Non200Error{} = error} -> {:error, %{reason: :timeout}} ->
Sentry.capture_message("Error listing GA views for user", extra: %{error: error}) {:error, :timeout}
{:error, error} ->
Sentry.capture_message("Error listing UA views for user", extra: %{error: error})
{:error, :unknown} {:error, :unknown}
end end
end end
@earliest_valid_date "2005-01-01" @earliest_valid_date "2005-01-01"
def get_analytics_start_date(access_token, view_id) do def get_analytics_start_date(access_token, view_id) do
get_analytics_boundary_date(access_token, view_id, :start)
end
def get_analytics_end_date(access_token, view_id) do
get_analytics_boundary_date(access_token, view_id, :end)
end
defp get_analytics_boundary_date(access_token, view_id, edge) do
sort_order =
if edge == :start do
"ASCENDING"
else
"DESCENDING"
end
params = %{ params = %{
reportRequests: [ reportRequests: [
%{ %{
@ -127,7 +146,7 @@ defmodule Plausible.Google.UA.HTTP do
metrics: [%{expression: "ga:pageviews"}], metrics: [%{expression: "ga:pageviews"}],
hideTotals: true, hideTotals: true,
hideValueRanges: true, hideValueRanges: true,
orderBys: [%{fieldName: "ga:date", sortOrder: "ASCENDING"}], orderBys: [%{fieldName: "ga:date", sortOrder: sort_order}],
pageSize: 1 pageSize: 1
} }
] ]
@ -151,13 +170,18 @@ defmodule Plausible.Google.UA.HTTP do
{:ok, date} {:ok, date}
{:error, %{reason: %Finch.Response{body: body}}} -> {:error, %HTTPClient.Non200Error{} = error} when error.reason.status in [401, 403] ->
Sentry.capture_message("Error fetching UA start date", extra: %{body: inspect(body)}) {:error, :authentication_failed}
{:error, body}
{:error, %{reason: :timeout}} ->
{:error, :timeout}
{:error, error} ->
Sentry.capture_message("Error retrieving UA #{edge} date",
extra: %{error: error}
)
{:error, %{reason: reason} = e} -> {:error, :unknown}
Sentry.capture_message("Error fetching UA start date", extra: %{error: inspect(e)})
{:error, reason}
end end
end end

@ -37,6 +37,9 @@ defmodule Plausible.Imported do
# Maximum number of complete imports to account for when querying stats # Maximum number of complete imports to account for when querying stats
@max_complete_imports 5 @max_complete_imports 5
@spec schemas() :: [module()]
def schemas, do: @tables
@spec tables() :: [String.t()] @spec tables() :: [String.t()]
def tables, do: @table_names def tables, do: @table_names
@ -48,28 +51,29 @@ defmodule Plausible.Imported do
def load_import_data(site) do def load_import_data(site) do
complete_import_ids = list_complete_import_ids(site) complete_import_ids = list_complete_import_ids(site)
earliest_import = get_earliest_import(site) || %{} dates = get_imports_date_range(site)
%{ %{
site site
| import_data_loaded: true, | import_data_loaded: true,
earliest_import_start_date: Map.get(earliest_import, :start_date), earliest_import_start_date: dates.start_date,
earliest_import_end_date: Map.get(earliest_import, :end_date), latest_import_end_date: dates.end_date,
complete_import_ids: complete_import_ids complete_import_ids: complete_import_ids
} }
end end
@spec get_import(non_neg_integer()) :: SiteImport.t() | nil @spec get_import(Site.t(), non_neg_integer()) :: SiteImport.t() | nil
def get_import(import_id) do def get_import(site, import_id) do
Repo.get(SiteImport, import_id) Repo.get_by(SiteImport, id: import_id, site_id: site.id)
end end
defdelegate listen(), to: Imported.Importer defdelegate listen(), to: Imported.Importer
@spec list_all_imports(Site.t()) :: [SiteImport.t()] @spec list_all_imports(Site.t(), atom()) :: [SiteImport.t()]
def list_all_imports(site) do def list_all_imports(site, status \\ nil) do
imports = imports =
from(i in SiteImport, where: i.site_id == ^site.id, order_by: [desc: i.inserted_at]) from(i in SiteImport, where: i.site_id == ^site.id, order_by: [desc: i.inserted_at])
|> maybe_filter_by_status(status)
|> Repo.all() |> Repo.all()
if site.imported_data && not Enum.any?(imports, & &1.legacy) do if site.imported_data && not Enum.any?(imports, & &1.legacy) do
@ -79,42 +83,72 @@ defmodule Plausible.Imported do
end end
end end
@spec other_imports_in_progress?(SiteImport.t()) :: boolean()
def other_imports_in_progress?(site_import) do
Repo.exists?(
from(i in SiteImport,
where: i.site_id == ^site_import.site_id and i.id != ^site_import.id,
where: i.status in ^[SiteImport.pending(), SiteImport.importing()]
)
)
end
defp maybe_filter_by_status(query, nil), do: query
defp maybe_filter_by_status(query, status) do
where(query, [i], i.status == ^status)
end
@spec list_complete_import_ids(Site.t()) :: [non_neg_integer()] @spec list_complete_import_ids(Site.t()) :: [non_neg_integer()]
def list_complete_import_ids(site) do def list_complete_import_ids(site) do
ids = ids =
from(i in SiteImport, from(i in SiteImport,
where: i.site_id == ^site.id and i.status == ^SiteImport.completed(), where: i.site_id == ^site.id and i.status == ^SiteImport.completed(),
select: i.id, select: {i.legacy, i.id},
limit: @max_complete_imports limit: @max_complete_imports
) )
|> Repo.all() |> Repo.all()
has_legacy? = Enum.any?(ids, fn {legacy?, _} -> legacy? end)
ids = Enum.map(ids, &elem(&1, 1))
# account for legacy imports as well # account for legacy imports as well
if site.imported_data && site.imported_data.status == "ok" do if has_legacy? || (site.imported_data && site.imported_data.status == "ok") do
[0 | ids] [0 | ids]
else else
ids ids
end end
end end
@spec get_earliest_import(Site.t()) :: SiteImport.t() | nil @spec get_imports_date_range(Site.t()) :: %{
def get_earliest_import(site) do start_date: Date.t() | nil,
first_import = end_date: Date.t() | nil
}
def get_imports_date_range(site) do
dates =
from(i in SiteImport, from(i in SiteImport,
where: i.site_id == ^site.id and i.status == ^SiteImport.completed(), where: i.site_id == ^site.id and i.status == ^SiteImport.completed(),
order_by: i.start_date, select: %{start_date: min(i.start_date), end_date: max(i.end_date)}
limit: 1
) )
|> Repo.one() |> Repo.one()
legacy_import = dates = dates || %{start_date: nil, end_date: nil}
if site.imported_data && site.imported_data.status == "ok" do
site.imported_data if site.imported_data && site.imported_data.status == "ok" do
end start_date =
[dates.start_date, site.imported_data.start_date]
|> Enum.reject(&is_nil/1)
|> Enum.min(Date, fn -> nil end)
end_date =
[dates.end_date, site.imported_data.end_date]
|> Enum.reject(&is_nil/1)
|> Enum.max(Date, fn -> nil end)
[legacy_import, first_import] %{start_date: start_date, end_date: end_date}
|> Enum.reject(&is_nil/1) else
|> Enum.min_by(& &1.start_date, Date, fn -> nil end) dates
end
end end
@spec delete_imports_for_site(Site.t()) :: :ok @spec delete_imports_for_site(Site.t()) :: :ok
@ -123,4 +157,66 @@ defmodule Plausible.Imported do
:ok :ok
end end
@spec check_dates(Site.t(), Date.t() | nil, Date.t() | nil) ::
{:ok, Date.t(), Date.t()} | {:error, :no_data | :no_time_window}
def check_dates(_site, nil, _end_date), do: {:error, :no_data}
def check_dates(site, start_date, end_date) do
cutoff_date = Plausible.Sites.native_stats_start_date(site) || Timex.today(site.timezone)
end_date = Enum.min([end_date, cutoff_date], Date)
with true <- Date.diff(end_date, start_date) >= 2,
[_ | _] = free_ranges <- find_free_ranges(start_date, end_date, site) do
longest = Enum.max_by(free_ranges, &Date.diff(&1.last, &1.first))
{:ok, longest.first, longest.last}
else
_ -> {:error, :no_time_window}
end
end
defp find_free_ranges(start_date, end_date, site) do
occupied_ranges =
site
|> Imported.list_all_imports(Imported.SiteImport.completed())
|> Enum.reject(&(Date.diff(&1.end_date, &1.start_date) < 2))
|> Enum.map(&Date.range(&1.start_date, &1.end_date))
Date.range(start_date, end_date)
|> free_ranges(start_date, occupied_ranges, [])
end
# This function recursively finds open ranges that are not yet occupied
# by existing imported data. The idea is that we keep moving a dynamic
# date index `d` from start until the end of `imported_range`, hopping
# over each occupied range, and capturing the open ranges step-by-step
# in the `result` array.
defp free_ranges(import_range, d, [occupied_range | rest_of_occupied_ranges], result) do
cond do
Date.diff(occupied_range.last, d) <= 0 ->
free_ranges(import_range, d, rest_of_occupied_ranges, result)
in_range?(d, occupied_range) || Date.diff(occupied_range.first, d) < 2 ->
d = occupied_range.last
free_ranges(import_range, d, rest_of_occupied_ranges, result)
true ->
free_range = Date.range(d, occupied_range.first)
result = result ++ [free_range]
d = occupied_range.last
free_ranges(import_range, d, rest_of_occupied_ranges, result)
end
end
defp free_ranges(import_range, d, [], result) do
if Date.diff(import_range.last, d) < 2 do
result
else
result ++ [Date.range(d, import_range.last)]
end
end
defp in_range?(date, range) do
Date.before?(range.first, date) && Date.after?(range.last, date)
end
end end

@ -8,9 +8,11 @@ defmodule Plausible.Imported.Browser do
field :import_id, Ch, type: "UInt64" field :import_id, Ch, type: "UInt64"
field :date, :date field :date, :date
field :browser, :string field :browser, :string
field :browser_version, :string
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :visits, Ch, type: "UInt64" field :visits, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64" field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32" field :bounces, Ch, type: "UInt32"
end end
end end

@ -43,9 +43,17 @@ defmodule Plausible.Imported.CSVImporter do
{table, _, _} = parse_filename!(filename) {table, _, _} = parse_filename!(filename)
s3_structure = input_structure!(table) s3_structure = input_structure!(table)
s3_structure_cols_expr =
s3_structure
|> String.split(",", trim: true)
|> Enum.map_join(", ", fn kv ->
[col, _type] = String.split(kv)
col
end)
statement = statement =
""" """
INSERT INTO {table:Identifier} \ INSERT INTO {table:Identifier}(site_id, #{s3_structure_cols_expr}, import_id) \
SELECT {site_id:UInt64} AS site_id, *, {import_id:UInt64} AS import_id \ SELECT {site_id:UInt64} AS site_id, *, {import_id:UInt64} AS import_id \
FROM s3({s3_url:String},{s3_access_key_id:String},{s3_secret_access_key:String},{s3_format:String},{s3_structure:String}) \ FROM s3({s3_url:String},{s3_access_key_id:String},{s3_secret_access_key:String},{s3_format:String},{s3_structure:String}) \
WHERE date >= {start_date:Date} AND date <= {end_date:Date}\ WHERE date >= {start_date:Date} AND date <= {end_date:Date}\
@ -75,20 +83,21 @@ defmodule Plausible.Imported.CSVImporter do
input_structures = %{ input_structures = %{
"imported_browsers" => "imported_browsers" =>
"date Date, browser String, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32", "date Date, browser String, browser_version String, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32, pageviews UInt64",
"imported_devices" => "imported_devices" =>
"date Date, device String, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32", "date Date, device String, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32, pageviews UInt64",
"imported_entry_pages" => "imported_entry_pages" =>
"date Date, entry_page String, visitors UInt64, entrances UInt64, visit_duration UInt64, bounces UInt32", "date Date, entry_page String, visitors UInt64, entrances UInt64, visit_duration UInt64, bounces UInt32, pageviews UInt64",
"imported_exit_pages" => "date Date, exit_page String, visitors UInt64, exits UInt64", "imported_exit_pages" =>
"date Date, exit_page String, visitors UInt64, visit_duration UInt64, exits UInt64, bounces UInt32, pageviews UInt64",
"imported_locations" => "imported_locations" =>
"date Date, country String, region String, city UInt64, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32", "date Date, country String, region String, city UInt64, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32, pageviews UInt64",
"imported_operating_systems" => "imported_operating_systems" =>
"date Date, operating_system String, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32", "date Date, operating_system String, operating_system_version String, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32, pageviews UInt64",
"imported_pages" => "imported_pages" =>
"date Date, hostname String, page String, visitors UInt64, pageviews UInt64, exits UInt64, time_on_page UInt64", "date Date, hostname String, page String, visits UInt64, visitors UInt64, pageviews UInt64, exits UInt64, time_on_page UInt64",
"imported_sources" => "imported_sources" =>
"date Date, source String, utm_medium String, utm_campaign String, utm_content String, utm_term String, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32", "date Date, source String, referrer String, utm_source String, utm_medium String, utm_campaign String, utm_content String, utm_term String, pageviews UInt64, visitors UInt64, visits UInt64, visit_duration UInt64, bounces UInt32",
"imported_visitors" => "imported_visitors" =>
"date Date, visitors UInt64, pageviews UInt64, bounces UInt64, visits UInt64, visit_duration UInt64" "date Date, visitors UInt64, pageviews UInt64, bounces UInt64, visits UInt64, visit_duration UInt64"
} }

@ -11,6 +11,7 @@ defmodule Plausible.Imported.Device do
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :visits, Ch, type: "UInt64" field :visits, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64" field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32" field :bounces, Ch, type: "UInt32"
end end
end end

@ -11,6 +11,7 @@ defmodule Plausible.Imported.EntryPage do
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :entrances, Ch, type: "UInt64" field :entrances, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64" field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32" field :bounces, Ch, type: "UInt32"
end end
end end

@ -8,7 +8,10 @@ defmodule Plausible.Imported.ExitPage do
field :import_id, Ch, type: "UInt64" field :import_id, Ch, type: "UInt64"
field :date, :date field :date, :date
field :exit_page, :string field :exit_page, :string
field :visitors, Ch, type: "UInt64"
field :exits, Ch, type: "UInt64" field :exits, Ch, type: "UInt64"
field :visitors, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32"
end end
end end

@ -96,17 +96,28 @@ defmodule Plausible.Imported.GoogleAnalytics4 do
end end
defp new_from_report(site_id, import_id, "imported_sources", row) do defp new_from_report(site_id, import_id, "imported_sources", row) do
referrer_uri = row.dimensions |> Map.fetch!("pageReferrer") |> URI.parse()
referrer =
if PlausibleWeb.RefInspector.right_uri?(referrer_uri) do
PlausibleWeb.RefInspector.format_referrer(referrer_uri)
end
%{ %{
site_id: site_id, site_id: site_id,
import_id: import_id, import_id: import_id,
date: get_date(row), date: get_date(row),
source: row.dimensions |> Map.fetch!("sessionSource") |> parse_referrer(), source: row.dimensions |> Map.fetch!("sessionSource") |> parse_referrer(),
referrer: referrer,
# Only `source` exists in GA4 API
utm_source: nil,
utm_medium: row.dimensions |> Map.fetch!("sessionMedium") |> default_if_missing(), utm_medium: row.dimensions |> Map.fetch!("sessionMedium") |> default_if_missing(),
utm_campaign: row.dimensions |> Map.fetch!("sessionCampaignName") |> default_if_missing(), utm_campaign: row.dimensions |> Map.fetch!("sessionCampaignName") |> default_if_missing(),
utm_content: row.dimensions |> Map.fetch!("sessionManualAdContent") |> default_if_missing(), utm_content: row.dimensions |> Map.fetch!("sessionManualAdContent") |> default_if_missing(),
utm_term: row.dimensions |> Map.fetch!("sessionGoogleAdsKeyword") |> default_if_missing(), utm_term: row.dimensions |> Map.fetch!("sessionGoogleAdsKeyword") |> default_if_missing(),
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(), visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
visits: row.metrics |> Map.fetch!("sessions") |> parse_number(), visits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(), bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(),
visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number() visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number()
} }
@ -120,6 +131,7 @@ defmodule Plausible.Imported.GoogleAnalytics4 do
hostname: row.dimensions |> Map.fetch!("hostName") |> String.replace_prefix("www.", ""), hostname: row.dimensions |> Map.fetch!("hostName") |> String.replace_prefix("www.", ""),
page: row.dimensions |> Map.fetch!("pagePath") |> URI.parse() |> Map.get(:path), page: row.dimensions |> Map.fetch!("pagePath") |> URI.parse() |> Map.get(:path),
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(), visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
visits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(), pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
# NOTE: no exits metric in GA4 API currently # NOTE: no exits metric in GA4 API currently
exits: 0, exits: 0,
@ -136,6 +148,7 @@ defmodule Plausible.Imported.GoogleAnalytics4 do
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(), visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
entrances: row.metrics |> Map.fetch!("sessions") |> parse_number(), entrances: row.metrics |> Map.fetch!("sessions") |> parse_number(),
visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number(), visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
bounces: row.metrics |> Map.fetch!("bounces") |> parse_number() bounces: row.metrics |> Map.fetch!("bounces") |> parse_number()
} }
end end
@ -148,7 +161,10 @@ defmodule Plausible.Imported.GoogleAnalytics4 do
# date: get_date(row), # date: get_date(row),
# exit_page: Map.fetch!(row.dimensions, "exitPage"), # exit_page: Map.fetch!(row.dimensions, "exitPage"),
# visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(), # visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
# exits: row.metrics |> Map.fetch!("sessions") |> parse_number() # exits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
# visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number(),
# pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
# bounces: row.metrics |> Map.fetch!("bounces") |> parse_number()
# } # }
# end # end
@ -166,6 +182,7 @@ defmodule Plausible.Imported.GoogleAnalytics4 do
city: city_data && city_data.id, city: city_data && city_data.id,
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(), visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
visits: row.metrics |> Map.fetch!("sessions") |> parse_number(), visits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(), bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(),
visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number() visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number()
} }
@ -179,6 +196,7 @@ defmodule Plausible.Imported.GoogleAnalytics4 do
device: row.dimensions |> Map.fetch!("deviceCategory") |> String.capitalize(), device: row.dimensions |> Map.fetch!("deviceCategory") |> String.capitalize(),
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(), visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
visits: row.metrics |> Map.fetch!("sessions") |> parse_number(), visits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(), bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(),
visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number() visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number()
} }
@ -202,8 +220,11 @@ defmodule Plausible.Imported.GoogleAnalytics4 do
import_id: import_id, import_id: import_id,
date: get_date(row), date: get_date(row),
browser: Map.get(@browser_google_to_plausible, browser, browser), browser: Map.get(@browser_google_to_plausible, browser, browser),
# Does not exist in GA4 API
browser_version: nil,
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(), visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
visits: row.metrics |> Map.fetch!("sessions") |> parse_number(), visits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(), bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(),
visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number() visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number()
} }
@ -223,8 +244,10 @@ defmodule Plausible.Imported.GoogleAnalytics4 do
import_id: import_id, import_id: import_id,
date: get_date(row), date: get_date(row),
operating_system: Map.get(@os_google_to_plausible, os, os), operating_system: Map.get(@os_google_to_plausible, os, os),
operating_system_version: row.dimensions |> Map.fetch!("operatingSystemVersion"),
visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(), visitors: row.metrics |> Map.fetch!("totalUsers") |> parse_number(),
visits: row.metrics |> Map.fetch!("sessions") |> parse_number(), visits: row.metrics |> Map.fetch!("sessions") |> parse_number(),
pageviews: row.metrics |> Map.fetch!("screenPageViews") |> parse_number(),
bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(), bounces: row.metrics |> Map.fetch!("bounces") |> parse_number(),
visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number() visit_duration: row.metrics |> Map.fetch!("userEngagementDuration") |> parse_number()
} }

@ -91,6 +91,7 @@ defmodule Plausible.Imported.Importer do
like LiveView, where notifications can be listened for via `handle_info/2`. like LiveView, where notifications can be listened for via `handle_info/2`.
""" """
alias Plausible.Imported
alias Plausible.Imported.SiteImport alias Plausible.Imported.SiteImport
alias Plausible.Repo alias Plausible.Repo
@ -98,7 +99,7 @@ defmodule Plausible.Imported.Importer do
@callback label() :: String.t() @callback label() :: String.t()
@callback email_template() :: String.t() @callback email_template() :: String.t()
@callback parse_args(map()) :: Keyword.t() @callback parse_args(map()) :: Keyword.t()
@callback import_data(Plausible.Imported.SiteImport.t(), Keyword.t()) :: :ok | {:error, any()} @callback import_data(SiteImport.t(), Keyword.t()) :: :ok | {:error, any()}
@callback before_start(SiteImport.t()) :: :ok | {:ok, map()} | {:error, any()} @callback before_start(SiteImport.t()) :: :ok | {:ok, map()} | {:error, any()}
@callback on_success(SiteImport.t(), map()) :: :ok @callback on_success(SiteImport.t(), map()) :: :ok
@callback on_failure(SiteImport.t()) :: :ok @callback on_failure(SiteImport.t()) :: :ok
@ -108,7 +109,7 @@ defmodule Plausible.Imported.Importer do
@behaviour Plausible.Imported.Importer @behaviour Plausible.Imported.Importer
@spec new_import(Plausible.Site.t(), Plausible.Auth.User.t(), Keyword.t()) :: @spec new_import(Plausible.Site.t(), Plausible.Auth.User.t(), Keyword.t()) ::
{:ok, Oban.Job.t()} | {:error, Ecto.Changeset.t()} {:ok, Oban.Job.t()} | {:error, Ecto.Changeset.t() | :import_in_progress}
def new_import(site, user, opts) do def new_import(site, user, opts) do
Plausible.Imported.Importer.new_import(name(), site, user, opts, &before_start/1) Plausible.Imported.Importer.new_import(name(), site, user, opts, &before_start/1)
end end
@ -222,11 +223,15 @@ defmodule Plausible.Imported.Importer do
:ok = listen() :ok = listen()
end end
opts if not Imported.other_imports_in_progress?(site_import) do
|> Keyword.put(:import_id, site_import.id) opts
|> Map.new() |> Keyword.put(:import_id, site_import.id)
|> Plausible.Workers.ImportAnalytics.new() |> Map.new()
|> Oban.insert() |> Plausible.Workers.ImportAnalytics.new()
|> Oban.insert()
else
{:error, :import_in_progress}
end
end end
defp mark_complete(site_import, extra_data, on_success_fun) do defp mark_complete(site_import, extra_data, on_success_fun) do

@ -13,6 +13,7 @@ defmodule Plausible.Imported.Location do
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :visits, Ch, type: "UInt64" field :visits, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64" field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32" field :bounces, Ch, type: "UInt32"
end end
end end

@ -8,9 +8,11 @@ defmodule Plausible.Imported.OperatingSystem do
field :import_id, Ch, type: "UInt64" field :import_id, Ch, type: "UInt64"
field :date, :date field :date, :date
field :operating_system, :string field :operating_system, :string
field :operating_system_version, :string
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :visits, Ch, type: "UInt64" field :visits, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64" field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32" field :bounces, Ch, type: "UInt32"
end end
end end

@ -9,6 +9,7 @@ defmodule Plausible.Imported.Page do
field :date, :date field :date, :date
field :hostname, :string field :hostname, :string
field :page, :string field :page, :string
field :visits, Ch, type: "UInt64"
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64" field :pageviews, Ch, type: "UInt64"
field :exits, Ch, type: "UInt64" field :exits, Ch, type: "UInt64"

@ -8,6 +8,8 @@ defmodule Plausible.Imported.Source do
field :import_id, Ch, type: "UInt64" field :import_id, Ch, type: "UInt64"
field :date, :date field :date, :date
field :source, :string field :source, :string
field :referrer, :string
field :utm_source, :string
field :utm_medium, :string field :utm_medium, :string
field :utm_campaign, :string field :utm_campaign, :string
field :utm_content, :string field :utm_content, :string
@ -15,6 +17,7 @@ defmodule Plausible.Imported.Source do
field :visitors, Ch, type: "UInt64" field :visitors, Ch, type: "UInt64"
field :visits, Ch, type: "UInt64" field :visits, Ch, type: "UInt64"
field :visit_duration, Ch, type: "UInt64" field :visit_duration, Ch, type: "UInt64"
field :pageviews, Ch, type: "UInt64"
field :bounces, Ch, type: "UInt32" field :bounces, Ch, type: "UInt32"
end end
end end

@ -44,26 +44,6 @@ defmodule Plausible.Ingestion.Event do
changeset: %Ecto.Changeset{} changeset: %Ecto.Changeset{}
} }
@session_properties [
:session_id,
:referrer,
:referrer_source,
:utm_medium,
:utm_source,
:utm_campaign,
:utm_content,
:utm_term,
:country_code,
:subdivision1_code,
:subdivision2_code,
:city_geoname_id,
:screen_size,
:operating_system,
:operating_system_version,
:browser,
:browser_version
]
@spec build_and_buffer(Request.t()) :: {:ok, %{buffered: [t()], dropped: [t()]}} @spec build_and_buffer(Request.t()) :: {:ok, %{buffered: [t()], dropped: [t()]}}
def build_and_buffer(%Request{domains: domains} = request) do def build_and_buffer(%Request{domains: domains} = request) do
processed_events = processed_events =
@ -359,8 +339,7 @@ defmodule Plausible.Ingestion.Event do
%{ %{
event event
| clickhouse_event: | clickhouse_event: ClickhouseEventV2.merge_session(event.clickhouse_event, session)
Map.merge(event.clickhouse_event, Map.take(session, @session_properties))
} }
end end

@ -47,7 +47,14 @@ defmodule Plausible.Session.CacheStore do
do: event.pathname, do: event.pathname,
else: session.entry_page else: session.entry_page
), ),
hostname:
if(event.name == "pageview" and session.hostname == "",
do: event.hostname,
else: session.hostname
),
exit_page: if(event.name == "pageview", do: event.pathname, else: session.exit_page), exit_page: if(event.name == "pageview", do: event.pathname, else: session.exit_page),
exit_page_hostname:
if(event.name == "pageview", do: event.hostname, else: session.exit_page_hostname),
is_bounce: false, is_bounce: false,
duration: Timex.diff(event.timestamp, session.start, :second) |> abs, duration: Timex.diff(event.timestamp, session.start, :second) |> abs,
pageviews: pageviews:
@ -60,11 +67,12 @@ defmodule Plausible.Session.CacheStore do
%Plausible.ClickhouseSessionV2{ %Plausible.ClickhouseSessionV2{
sign: 1, sign: 1,
session_id: Plausible.ClickhouseSessionV2.random_uint64(), session_id: Plausible.ClickhouseSessionV2.random_uint64(),
hostname: event.hostname, hostname: if(event.name == "pageview", do: event.hostname, else: ""),
site_id: event.site_id, site_id: event.site_id,
user_id: event.user_id, user_id: event.user_id,
entry_page: if(event.name == "pageview", do: event.pathname, else: ""), entry_page: if(event.name == "pageview", do: event.pathname, else: ""),
exit_page: if(event.name == "pageview", do: event.pathname, else: ""), exit_page: if(event.name == "pageview", do: event.pathname, else: ""),
exit_page_hostname: if(event.name == "pageview", do: event.hostname, else: ""),
is_bounce: true, is_bounce: true,
duration: 0, duration: 0,
pageviews: if(event.name == "pageview", do: 1, else: 0), pageviews: if(event.name == "pageview", do: 1, else: 0),

@ -59,7 +59,7 @@ defmodule Plausible.Site do
# `site` in `assigns`. # `site` in `assigns`.
field :import_data_loaded, :boolean, default: false, virtual: true field :import_data_loaded, :boolean, default: false, virtual: true
field :earliest_import_start_date, :date, virtual: true field :earliest_import_start_date, :date, virtual: true
field :earliest_import_end_date, :date, virtual: true field :latest_import_end_date, :date, virtual: true
field :complete_import_ids, {:array, :integer}, default: [], virtual: true field :complete_import_ids, {:array, :integer}, default: [], virtual: true
timestamps() timestamps()

@ -5,19 +5,6 @@ defmodule Plausible.Stats.Aggregate do
import Ecto.Query import Ecto.Query
alias Plausible.Stats.{Query, Util} alias Plausible.Stats.{Query, Util}
@revenue_metrics on_full_build(do: Plausible.Stats.Goal.Revenue.revenue_metrics(), else: [])
@event_metrics [
:visitors,
:pageviews,
:events,
:sample_percent,
:conversion_rate,
:total_visitors
] ++ @revenue_metrics
@session_metrics [:visits, :bounce_rate, :visit_duration, :views_per_visit, :sample_percent]
def aggregate(site, query, metrics) do def aggregate(site, query, metrics) do
{currency, metrics} = {currency, metrics} =
on_full_build do on_full_build do
@ -28,18 +15,17 @@ defmodule Plausible.Stats.Aggregate do
Query.trace(query, metrics) Query.trace(query, metrics)
event_metrics = {event_metrics, session_metrics, other_metrics} =
metrics metrics
|> Util.maybe_add_visitors_metric() |> Util.maybe_add_visitors_metric()
|> Enum.filter(&(&1 in @event_metrics)) |> Plausible.Stats.TableDecider.partition_metrics(query)
event_task = fn -> aggregate_events(site, query, event_metrics) end event_task = fn -> aggregate_events(site, query, event_metrics) end
session_metrics = Enum.filter(metrics, &(&1 in @session_metrics))
session_task = fn -> aggregate_sessions(site, query, session_metrics) end session_task = fn -> aggregate_sessions(site, query, session_metrics) end
time_on_page_task = time_on_page_task =
if :time_on_page in metrics do if :time_on_page in other_metrics do
fn -> aggregate_time_on_page(site, query) end fn -> aggregate_time_on_page(site, query) end
else else
fn -> %{} end fn -> %{} end

@ -3,7 +3,7 @@ defmodule Plausible.Stats.Base do
use Plausible use Plausible
use Plausible.Stats.Fragments use Plausible.Stats.Fragments
alias Plausible.Stats.{Query, Filters} alias Plausible.Stats.{Query, Filters, TableDecider}
alias Plausible.Timezones alias Plausible.Timezones
import Ecto.Query import Ecto.Query
@ -15,7 +15,7 @@ defmodule Plausible.Stats.Base do
def base_event_query(site, query) do def base_event_query(site, query) do
events_q = query_events(site, query) events_q = query_events(site, query)
if Enum.any?(Filters.visit_props(), &query.filters["visit:" <> &1]) do if TableDecider.events_join_sessions?(query) do
sessions_q = sessions_q =
from( from(
s in query_sessions(site, query), s in query_sessions(site, query),
@ -30,11 +30,15 @@ defmodule Plausible.Stats.Base do
on: e.session_id == sq.session_id on: e.session_id == sq.session_id
) )
else else
events_q if query.experimental_reduced_joins? do
events_q |> filter_by_visit_props(Filters.event_table_visit_props(), query)
else
events_q
end
end end
end end
def query_events(site, query) do defp query_events(site, query) do
{first_datetime, last_datetime} = utc_boundaries(query, site) {first_datetime, last_datetime} = utc_boundaries(query, site)
q = q =
@ -48,7 +52,10 @@ defmodule Plausible.Stats.Base do
q = Plausible.Stats.Sampling.add_query_hint(q, query) q = Plausible.Stats.Sampling.add_query_hint(q, query)
end end
q = from(e in q, where: ^dynamic_filter_condition(query, "event:page", :pathname)) q =
q
|> where([e], ^dynamic_filter_condition(query, "event:page", :pathname))
|> where([e], ^dynamic_filter_condition(query, "event:hostname", :hostname))
q = q =
case query.filters["event:name"] do case query.filters["event:name"] do
@ -122,17 +129,6 @@ defmodule Plausible.Stats.Base do
q q
end end
@api_prop_name_to_db %{
"source" => "referrer_source",
"device" => "screen_size",
"screen" => "screen_size",
"os" => "operating_system",
"os_version" => "operating_system_version",
"country" => "country_code",
"region" => "subdivision1_code",
"city" => "city_geoname_id"
}
def query_sessions(site, query) do def query_sessions(site, query) do
{first_datetime, last_datetime} = {first_datetime, last_datetime} =
utc_boundaries(query, site) utc_boundaries(query, site)
@ -140,8 +136,7 @@ defmodule Plausible.Stats.Base do
q = from(s in "sessions_v2", where: s.site_id == ^site.id) q = from(s in "sessions_v2", where: s.site_id == ^site.id)
sessions_q = sessions_q =
if FunWithFlags.enabled?(:experimental_session_count, for: site) or if query.experimental_session_count? do
query.experimental_session_count? do
from s in q, where: s.timestamp >= ^first_datetime and s.start < ^last_datetime from s in q, where: s.timestamp >= ^first_datetime and s.start < ^last_datetime
else else
from s in q, where: s.start >= ^first_datetime and s.start < ^last_datetime from s in q, where: s.start >= ^first_datetime and s.start < ^last_datetime
@ -151,9 +146,24 @@ defmodule Plausible.Stats.Base do
sessions_q = Plausible.Stats.Sampling.add_query_hint(sessions_q, query) sessions_q = Plausible.Stats.Sampling.add_query_hint(sessions_q, query)
end end
sessions_q = filter_by_entry_props(sessions_q, query) filter_by_entry_props(sessions_q, query)
|> filter_by_visit_props(Filters.visit_props(), query)
end
Enum.reduce(Filters.visit_props(), sessions_q, fn prop_name, sessions_q -> @api_prop_name_to_db %{
"source" => "referrer_source",
"device" => "screen_size",
"screen" => "screen_size",
"os" => "operating_system",
"os_version" => "operating_system_version",
"country" => "country_code",
"region" => "subdivision1_code",
"city" => "city_geoname_id",
"entry_page_hostname" => "hostname"
}
defp filter_by_visit_props(q, visit_props, query) do
Enum.reduce(visit_props, q, fn prop_name, sessions_q ->
filter_key = "visit:" <> prop_name filter_key = "visit:" <> prop_name
db_field = db_field =
@ -237,6 +247,13 @@ defmodule Plausible.Stats.Base do
} }
end end
defp select_event_metric(:visits) do
%{
visits:
dynamic([e], fragment("toUInt64(round(uniq(?) * any(_sample_factor)))", e.session_id))
}
end
on_full_build do on_full_build do
defp select_event_metric(:total_revenue) do defp select_event_metric(:total_revenue) do
%{total_revenue: Plausible.Stats.Goal.Revenue.total_revenue_query()} %{total_revenue: Plausible.Stats.Goal.Revenue.total_revenue_query()}

@ -6,7 +6,7 @@ defmodule Plausible.Stats.Breakdown do
import Plausible.Stats.{Base, Imported} import Plausible.Stats.{Base, Imported}
import Ecto.Query import Ecto.Query
require OpenTelemetry.Tracer, as: Tracer require OpenTelemetry.Tracer, as: Tracer
alias Plausible.Stats.{Query, Util} alias Plausible.Stats.{Query, Util, TableDecider}
@no_ref "Direct / None" @no_ref "Direct / None"
@not_set "(not set)" @not_set "(not set)"
@ -103,7 +103,6 @@ defmodule Plausible.Stats.Breakdown do
{event_q, page_q} -> {event_q, page_q} ->
from( from(
e in subquery(union_all(event_q, ^page_q)), e in subquery(union_all(event_q, ^page_q)),
# :TODO: Handle other orderings
order_by: [desc: e.visitors] order_by: [desc: e.visitors]
) )
|> apply_pagination(pagination) |> apply_pagination(pagination)
@ -200,9 +199,35 @@ defmodule Plausible.Stats.Breakdown do
metrics_to_select = Util.maybe_add_visitors_metric(metrics) -- @computed_metrics metrics_to_select = Util.maybe_add_visitors_metric(metrics) -- @computed_metrics
breakdown_sessions(site, query, property, metrics_to_select) case breakdown_table(query, metrics, property) do
|> maybe_add_group_conversion_rate(&breakdown_sessions/4, site, query, property, metrics) :session ->
|> paginate_and_execute(metrics, pagination) breakdown_sessions(site, query, property, metrics_to_select)
|> maybe_add_group_conversion_rate(&breakdown_sessions/4, site, query, property, metrics)
|> paginate_and_execute(metrics, pagination)
:event ->
breakdown_events(site, query, property, metrics_to_select)
|> maybe_add_group_conversion_rate(&breakdown_events/4, site, query, property, metrics)
|> paginate_and_execute(metrics, pagination)
end
end
# Backwards compatibility
# defp breakdown_table(%Query{experimental_reduced_joins?: false}, _, _), do: :session
defp breakdown_table(_query, _metrics, "visit:entry_page"), do: :session
defp breakdown_table(_query, _metrics, "visit:entry_page_hostname"), do: :session
defp breakdown_table(_query, _metrics, "visit:exit_page"), do: :session
defp breakdown_table(_query, _metrics, "visit:exit_page_hostname"), do: :session
defp breakdown_table(query, metrics, property) do
{_, session_metrics, _} = TableDecider.partition_metrics(metrics, query, property)
if not Enum.empty?(session_metrics) do
:session
else
:event
end
end end
defp zip_results(event_result, session_result, property, metrics) do defp zip_results(event_result, session_result, property, metrics) do
@ -273,94 +298,6 @@ defmodule Plausible.Stats.Breakdown do
end end
defp breakdown_time_on_page(site, query, pages) do defp breakdown_time_on_page(site, query, pages) do
if FunWithFlags.enabled?(:window_time_on_page) do
window_breakdown_time_on_page(site, query, pages)
else
neighbor_breakdown_time_on_page(site, query, pages)
end
end
defp neighbor_breakdown_time_on_page(site, query, pages) do
q =
from(
e in base_event_query(site, Query.remove_event_filters(query, [:page, :props])),
select: {
fragment("? as p", e.pathname),
fragment("? as t", e.timestamp),
fragment("? as s", e.session_id)
},
order_by: [e.session_id, e.timestamp]
)
{base_query_raw, base_query_raw_params} = ClickhouseRepo.to_sql(:all, q)
select =
if query.include_imported do
"sum(td), count(case when p2 != p then 1 end)"
else
"round(sum(td)/count(case when p2 != p then 1 end))"
end
pages_idx = length(base_query_raw_params)
params = base_query_raw_params ++ [pages]
time_query = "
SELECT
p,
#{select}
FROM
(SELECT
p,
p2,
sum(t2-t) as td
FROM
(SELECT
*,
neighbor(t, 1) as t2,
neighbor(p, 1) as p2,
neighbor(s, 1) as s2
FROM (#{base_query_raw}))
WHERE s=s2 AND p IN {$#{pages_idx}:Array(String)}
GROUP BY p,p2,s)
GROUP BY p"
{:ok, res} = ClickhouseRepo.query(time_query, params)
if query.include_imported do
# Imported page views have pre-calculated values
res =
res.rows
|> Enum.map(fn [page, time, visits] -> {page, {time, visits}} end)
|> Enum.into(%{})
from(
i in "imported_pages",
group_by: i.page,
where: i.site_id == ^site.id,
where: i.date >= ^query.date_range.first and i.date <= ^query.date_range.last,
where: i.page in ^pages,
select: %{
page: i.page,
pageviews: fragment("sum(?) - sum(?)", i.pageviews, i.exits),
time_on_page: sum(i.time_on_page)
}
)
|> ClickhouseRepo.all()
|> Enum.reduce(res, fn %{page: page, pageviews: pageviews, time_on_page: time}, res ->
{restime, resviews} = Map.get(res, page, {0, 0})
Map.put(res, page, {restime + time, resviews + pageviews})
end)
|> Enum.map(fn
{page, {_, 0}} -> {page, nil}
{page, {time, pageviews}} -> {page, time / pageviews}
end)
|> Enum.into(%{})
else
res.rows |> Enum.map(fn [page, time] -> {page, time} end) |> Enum.into(%{})
end
end
defp window_breakdown_time_on_page(site, query, pages) do
import Ecto.Query import Ecto.Query
windowed_pages_q = windowed_pages_q =

@ -208,11 +208,12 @@ defmodule Plausible.Stats.FilterSuggestions do
"operating_system" -> :operating_system "operating_system" -> :operating_system
"operating_system_version" -> :operating_system_version "operating_system_version" -> :operating_system_version
"screen_size" -> :screen_size "screen_size" -> :screen_size
"hostname" -> :hostname
_ -> :unknown _ -> :unknown
end end
q = q =
if(filter_name == :pathname, if(filter_name == :pathname or filter_name == :hostname,
do: base_event_query(site, query), do: base_event_query(site, query),
else: query_sessions(site, query) else: query_sessions(site, query)
) )
@ -230,6 +231,12 @@ defmodule Plausible.Stats.FilterSuggestions do
where: fragment("? ilike ?", e.pathname, ^filter_query) where: fragment("? ilike ?", e.pathname, ^filter_query)
) )
:hostname ->
from(e in q,
select: e.hostname,
where: fragment("? ilike ?", e.hostname, ^filter_query)
)
:entry_page -> :entry_page ->
from(e in q, from(e in q,
select: e.entry_page, select: e.entry_page,

@ -31,7 +31,7 @@ defmodule Plausible.Stats.Filters.DashboardFilterParser do
{is_negated, val} = parse_negated_prefix(val) {is_negated, val} = parse_negated_prefix(val)
{is_contains, val} = parse_contains_prefix(val) {is_contains, val} = parse_contains_prefix(val)
is_list = list_expression?(val) is_list = list_expression?(val)
is_wildcard = String.contains?(key, ["page", "goal"]) && wildcard_expression?(val) is_wildcard = String.contains?(key, ["page", "goal", "hostname"]) && wildcard_expression?(val)
val = if is_list, do: parse_member_list(val), else: remove_escape_chars(val) val = if is_list, do: parse_member_list(val), else: remove_escape_chars(val)
val = if key == "goal", do: wrap_goal_value(val), else: val val = if key == "goal", do: wrap_goal_value(val), else: val

@ -23,11 +23,22 @@ defmodule Plausible.Stats.Filters do
:region, :region,
:city, :city,
:entry_page, :entry_page,
:exit_page :exit_page,
:entry_page_hostname,
:exit_page_hostname
] ]
def visit_props(), do: @visit_props |> Enum.map(&to_string/1) def visit_props(), do: @visit_props |> Enum.map(&to_string/1)
@event_props [:name, :page, :goal] @event_table_visit_props @visit_props --
[
:entry_page,
:exit_page,
:entry_page_hostname,
:exit_page_hostname
]
def event_table_visit_props(), do: @event_table_visit_props |> Enum.map(&to_string/1)
@event_props [:name, :page, :goal, :hostname]
def event_props(), do: @event_props |> Enum.map(&to_string/1) def event_props(), do: @event_props |> Enum.map(&to_string/1)

@ -9,7 +9,8 @@ defmodule Plausible.Stats.Query do
imported_data_requested: false, imported_data_requested: false,
include_imported: false, include_imported: false,
now: nil, now: nil,
experimental_session_count?: false experimental_session_count?: false,
experimental_reduced_joins?: false
require OpenTelemetry.Tracer, as: Tracer require OpenTelemetry.Tracer, as: Tracer
alias Plausible.Stats.{Filters, Interval} alias Plausible.Stats.{Filters, Interval}
@ -22,7 +23,8 @@ defmodule Plausible.Stats.Query do
query = query =
__MODULE__ __MODULE__
|> struct!(now: now) |> struct!(now: now)
|> put_experimental_session_count(params) |> put_experimental_session_count(site, params)
|> put_experimental_reduced_joins(site, params)
|> put_period(site, params) |> put_period(site, params)
|> put_interval(params) |> put_interval(params)
|> put_parsed_filters(params) |> put_parsed_filters(params)
@ -36,11 +38,27 @@ defmodule Plausible.Stats.Query do
query query
end end
defp put_experimental_session_count(query, params) do defp put_experimental_session_count(query, site, params) do
if Map.get(params, "experimental_session_count") == "true" do if Map.has_key?(params, "experimental_session_count") do
struct!(query, experimental_session_count?: true) struct!(query,
experimental_session_count?: Map.get(params, "experimental_session_count") == "true"
)
else else
query struct!(query,
experimental_session_count?: FunWithFlags.enabled?(:experimental_session_count, for: site)
)
end
end
defp put_experimental_reduced_joins(query, site, params) do
if Map.has_key?(params, "experimental_reduced_joins") do
struct!(query,
experimental_reduced_joins?: Map.get(params, "experimental_reduced_joins") == "true"
)
else
struct!(query,
experimental_reduced_joins?: FunWithFlags.enabled?(:experimental_reduced_joins, for: site)
)
end end
end end
@ -261,8 +279,8 @@ defmodule Plausible.Stats.Query do
@spec include_imported?(t(), Plausible.Site.t(), boolean()) :: boolean() @spec include_imported?(t(), Plausible.Site.t(), boolean()) :: boolean()
def include_imported?(query, site, requested?) do def include_imported?(query, site, requested?) do
cond do cond do
is_nil(site.earliest_import_end_date) -> false is_nil(site.latest_import_end_date) -> false
Date.after?(query.date_range.first, site.earliest_import_end_date) -> false Date.after?(query.date_range.first, site.latest_import_end_date) -> false
Enum.any?(query.filters) -> false Enum.any?(query.filters) -> false
query.period == "realtime" -> false query.period == "realtime" -> false
true -> requested? true -> requested?

@ -0,0 +1,111 @@
defmodule Plausible.Stats.TableDecider do
@moduledoc """
This module contains logic for deciding which tables need to be queried given a query
and metrics, with the purpose of reducing the number of queries and JOINs needed to perform.
"""
import Enum, only: [empty?: 1]
alias Plausible.Stats.Query
def events_join_sessions?(query) do
Enum.any?(query.filters, &(filters_partitioner(query, &1) == :session))
end
def partition_metrics(metrics, query, breakdown_property \\ nil) do
%{
event: event_only_metrics,
session: session_only_metrics,
either: either_metrics,
other: other_metrics,
sample_percent: sample_percent
} =
partition(metrics, query, &metric_partitioner/2)
# Treat breakdown property as yet another filter
filters =
if breakdown_property do
Map.put(query.filters, breakdown_property, nil)
else
query.filters
end
%{event: event_only_filters, session: session_only_filters} =
partition(filters, query, &filters_partitioner/2)
cond do
# Only one table needs to be queried
empty?(event_only_metrics) && empty?(event_only_filters) ->
{[], session_only_metrics ++ either_metrics ++ sample_percent, other_metrics}
empty?(session_only_metrics) && empty?(session_only_filters) ->
{event_only_metrics ++ either_metrics ++ sample_percent, [], other_metrics}
# Filters on both events and sessions, but only one kind of metric
empty?(event_only_metrics) ->
{[], session_only_metrics ++ either_metrics ++ sample_percent, other_metrics}
empty?(session_only_metrics) ->
{event_only_metrics ++ either_metrics ++ sample_percent, [], other_metrics}
# Default: prefer sessions
true ->
{event_only_metrics ++ sample_percent,
session_only_metrics ++ either_metrics ++ sample_percent, other_metrics}
end
end
defp metric_partitioner(_, :conversion_rate), do: :event
defp metric_partitioner(_, :average_revenue), do: :event
defp metric_partitioner(_, :total_revenue), do: :event
defp metric_partitioner(_, :pageviews), do: :event
defp metric_partitioner(_, :events), do: :event
defp metric_partitioner(_, :bounce_rate), do: :session
defp metric_partitioner(_, :visit_duration), do: :session
defp metric_partitioner(_, :views_per_visit), do: :session
# Metrics which used to only be queried from one table but can be calculated from either
defp metric_partitioner(%Query{experimental_reduced_joins?: true}, :visits), do: :either
defp metric_partitioner(%Query{experimental_reduced_joins?: true}, :visitors), do: :either
defp metric_partitioner(_, :visits), do: :session
defp metric_partitioner(_, :visitors), do: :event
# Calculated metrics - handled on callsite separately from other metrics.
defp metric_partitioner(_, :time_on_page), do: :other
defp metric_partitioner(_, :total_visitors), do: :other
defp metric_partitioner(_, :percentage), do: :other
# Sample percentage is included in both tables if queried.
defp metric_partitioner(_, :sample_percent), do: :sample_percent
defp metric_partitioner(%Query{experimental_reduced_joins?: false}, unknown) do
raise ArgumentError, "Metric #{unknown} not supported without experimental_reduced_joins?"
end
defp metric_partitioner(_, _), do: :either
defp filters_partitioner(_, {"event:" <> _, _}), do: :event
defp filters_partitioner(_, {"visit:entry_page", _}), do: :session
defp filters_partitioner(_, {"visit:entry_page_hostname", _}), do: :session
defp filters_partitioner(_, {"visit:exit_page", _}), do: :session
defp filters_partitioner(_, {"visit:exit_page_hostname", _}), do: :session
defp filters_partitioner(%Query{experimental_reduced_joins?: true}, {"visit:" <> _, _}),
do: :either
defp filters_partitioner(_, {"visit:" <> _, _}),
do: :session
defp filters_partitioner(%Query{experimental_reduced_joins?: false}, {unknown, _}) do
raise ArgumentError, "Filter #{unknown} not supported without experimental_reduced_joins?"
end
defp filters_partitioner(_, _), do: :either
@default %{event: [], session: [], either: [], other: [], sample_percent: []}
defp partition(values, query, partitioner) do
Enum.reduce(values, @default, fn value, acc ->
key = partitioner.(query, value)
Map.put(acc, key, Map.fetch!(acc, key) ++ [value])
end)
end
end

@ -18,15 +18,11 @@ defmodule Plausible.Stats.Timeseries do
@typep value :: nil | integer() | float() @typep value :: nil | integer() | float()
@type results :: nonempty_list(%{required(:date) => Date.t(), required(metric()) => value()}) @type results :: nonempty_list(%{required(:date) => Date.t(), required(metric()) => value()})
@revenue_metrics on_full_build(do: Plausible.Stats.Goal.Revenue.revenue_metrics(), else: [])
@event_metrics [:visitors, :pageviews, :events, :conversion_rate] ++ @revenue_metrics
@session_metrics [:visits, :bounce_rate, :visit_duration, :views_per_visit]
def timeseries(site, query, metrics) do def timeseries(site, query, metrics) do
steps = buckets(query) steps = buckets(query)
event_metrics = Enum.filter(metrics, &(&1 in @event_metrics)) {event_metrics, session_metrics, _} =
session_metrics = Enum.filter(metrics, &(&1 in @session_metrics)) Plausible.Stats.TableDecider.partition_metrics(metrics, query)
{currency, event_metrics} = {currency, event_metrics} =
on_full_build do on_full_build do

@ -4,7 +4,6 @@ defmodule PlausibleWeb.Api.StatsController do
use Plausible.Repo use Plausible.Repo
use PlausibleWeb.Plugs.ErrorHandler use PlausibleWeb.Plugs.ErrorHandler
alias Plausible.Imported.SiteImport
alias Plausible.Stats alias Plausible.Stats
alias Plausible.Stats.{Query, Comparisons} alias Plausible.Stats.{Query, Comparisons}
alias PlausibleWeb.Api.Helpers, as: H alias PlausibleWeb.Api.Helpers, as: H
@ -68,8 +67,8 @@ defmodule PlausibleWeb.Api.StatsController do
* `with_imported` - boolean indicating whether the Google Analytics data * `with_imported` - boolean indicating whether the Google Analytics data
was queried or not. was queried or not.
* `imported_source` - the source of the imported data, when applicable. * `imports_exist` - boolean indicating whether there are any completed
Currently only Google Analytics is supported. imports for a given site or not.
* `full_intervals` - map of dates indicating whether the interval has been * `full_intervals` - map of dates indicating whether the interval has been
cut off by the requested date range or not. For example, if looking at a cut off by the requested date range or not. For example, if looking at a
@ -87,7 +86,7 @@ defmodule PlausibleWeb.Api.StatsController do
"2021-11-01" => true, "2021-11-01" => true,
"2021-12-01" => false "2021-12-01" => false
}, },
"imported_source" => nil, "imports_exist" => false,
"interval" => "month", "interval" => "month",
"labels" => ["2021-09-01", "2021-10-01", "2021-11-01", "2021-12-01"], "labels" => ["2021-09-01", "2021-10-01", "2021-11-01", "2021-12-01"],
"plot" => [0, 0, 0, 0], "plot" => [0, 0, 0, 0],
@ -129,9 +128,8 @@ defmodule PlausibleWeb.Api.StatsController do
present_index = present_index_for(site, query, labels) present_index = present_index_for(site, query, labels)
full_intervals = build_full_intervals(query, labels) full_intervals = build_full_intervals(query, labels)
site_import = Plausible.Imported.get_earliest_import(site)
json(conn, %{ json(conn, %{
metric: metric,
plot: plot_timeseries(timeseries_result, metric), plot: plot_timeseries(timeseries_result, metric),
labels: labels, labels: labels,
comparison_plot: comparison_result && plot_timeseries(comparison_result, metric), comparison_plot: comparison_result && plot_timeseries(comparison_result, metric),
@ -139,7 +137,7 @@ defmodule PlausibleWeb.Api.StatsController do
present_index: present_index, present_index: present_index,
interval: query.interval, interval: query.interval,
with_imported: with_imported?(query, comparison_query), with_imported: with_imported?(query, comparison_query),
imported_source: site_import && SiteImport.label(site_import), imports_exist: site.complete_import_ids != [],
full_intervals: full_intervals full_intervals: full_intervals
}) })
else else
@ -214,14 +212,12 @@ defmodule PlausibleWeb.Api.StatsController do
{top_stats, sample_percent} = fetch_top_stats(site, query, comparison_query) {top_stats, sample_percent} = fetch_top_stats(site, query, comparison_query)
site_import = Plausible.Imported.get_earliest_import(site)
json(conn, %{ json(conn, %{
top_stats: top_stats, top_stats: top_stats,
interval: query.interval, interval: query.interval,
sample_percent: sample_percent, sample_percent: sample_percent,
with_imported: with_imported?(query, comparison_query), with_imported: with_imported?(query, comparison_query),
imported_source: site_import && SiteImport.label(site_import), imports_exist: site.complete_import_ids != [],
comparing_from: comparison_query && comparison_query.date_range.first, comparing_from: comparison_query && comparison_query.date_range.first,
comparing_to: comparison_query && comparison_query.date_range.last, comparing_to: comparison_query && comparison_query.date_range.last,
from: query.date_range.first, from: query.date_range.first,
@ -462,6 +458,13 @@ defmodule PlausibleWeb.Api.StatsController do
query = Query.from(site, params) query = Query.from(site, params)
pagination = parse_pagination(params) pagination = parse_pagination(params)
query =
if query.filters["event:hostname"] do
Query.put_filter(query, "visit:entry_page_hostname", query.filters["event:hostname"])
else
query
end
extra_metrics = extra_metrics =
if params["detailed"], do: [:bounce_rate, :visit_duration], else: [] if params["detailed"], do: [:bounce_rate, :visit_duration], else: []
@ -767,6 +770,13 @@ defmodule PlausibleWeb.Api.StatsController do
pagination = parse_pagination(params) pagination = parse_pagination(params)
metrics = breakdown_metrics(query, [:visits, :visit_duration]) metrics = breakdown_metrics(query, [:visits, :visit_duration])
query =
if query.filters["event:hostname"] do
Query.put_filter(query, "visit:entry_page_hostname", query.filters["event:hostname"])
else
query
end
entry_pages = entry_pages =
Stats.breakdown(site, query, "visit:entry_page", metrics, pagination) Stats.breakdown(site, query, "visit:entry_page", metrics, pagination)
|> transform_keys(%{entry_page: :name}) |> transform_keys(%{entry_page: :name})
@ -797,6 +807,13 @@ defmodule PlausibleWeb.Api.StatsController do
{limit, page} = parse_pagination(params) {limit, page} = parse_pagination(params)
metrics = breakdown_metrics(query, [:visits]) metrics = breakdown_metrics(query, [:visits])
query =
if query.filters["event:hostname"] do
Query.put_filter(query, "visit:exit_page_hostname", query.filters["event:hostname"])
else
query
end
exit_pages = exit_pages =
Stats.breakdown(site, query, "visit:exit_page", metrics, {limit, page}) Stats.breakdown(site, query, "visit:exit_page", metrics, {limit, page})
|> add_exit_rate(site, query, limit) |> add_exit_rate(site, query, limit)
@ -1343,6 +1360,7 @@ defmodule PlausibleWeb.Api.StatsController do
metric = metric =
case params["metric"] do case params["metric"] do
nil -> :visitors nil -> :visitors
"conversions" -> :visitors
m -> Plausible.Stats.Metrics.from_string!(m) m -> Plausible.Stats.Metrics.from_string!(m)
end end

@ -1,6 +1,11 @@
defmodule PlausibleWeb.GoogleAnalyticsController do defmodule PlausibleWeb.GoogleAnalyticsController do
use PlausibleWeb, :controller use PlausibleWeb, :controller
alias Plausible.Google
alias Plausible.Imported
require Plausible.Imported.SiteImport
plug(PlausibleWeb.RequireAccountPlug) plug(PlausibleWeb.RequireAccountPlug)
plug(PlausibleWeb.AuthorizeSiteAccess, [:owner, :admin, :super_admin]) plug(PlausibleWeb.AuthorizeSiteAccess, [:owner, :admin, :super_admin])
@ -10,6 +15,8 @@ defmodule PlausibleWeb.GoogleAnalyticsController do
"access_token" => access_token, "access_token" => access_token,
"refresh_token" => refresh_token, "refresh_token" => refresh_token,
"expires_at" => expires_at, "expires_at" => expires_at,
"start_date" => start_date,
"end_date" => end_date,
"legacy" => legacy "legacy" => legacy
}) do }) do
site = conn.assigns.site site = conn.assigns.site
@ -22,17 +29,22 @@ defmodule PlausibleWeb.GoogleAnalyticsController do
access_token: access_token, access_token: access_token,
refresh_token: refresh_token, refresh_token: refresh_token,
expires_at: expires_at, expires_at: expires_at,
start_date: start_date,
end_date: end_date,
legacy: legacy, legacy: legacy,
layout: {PlausibleWeb.LayoutView, "focus.html"} layout: {PlausibleWeb.LayoutView, "focus.html"}
) )
end end
def property_or_view_form(conn, %{ def property_or_view_form(
"access_token" => access_token, conn,
"refresh_token" => refresh_token, %{
"expires_at" => expires_at, "access_token" => access_token,
"legacy" => legacy "refresh_token" => refresh_token,
}) do "expires_at" => expires_at,
"legacy" => legacy
} = params
) do
site = conn.assigns.site site = conn.assigns.site
redirect_route = redirect_route =
@ -44,9 +56,21 @@ defmodule PlausibleWeb.GoogleAnalyticsController do
result = result =
if legacy == "true" do if legacy == "true" do
Plausible.Google.UA.API.list_views(access_token) Google.UA.API.list_views(access_token)
else else
Plausible.Google.API.list_properties_and_views(access_token) Google.API.list_properties_and_views(access_token)
end
error =
case params["error"] do
"no_data" ->
"No data found. Nothing to import."
"no_time_window" ->
"Imported data time range is completely overlapping with existing data. Nothing to import."
_ ->
nil
end end
case result do case result do
@ -59,6 +83,7 @@ defmodule PlausibleWeb.GoogleAnalyticsController do
expires_at: expires_at, expires_at: expires_at,
site: conn.assigns.site, site: conn.assigns.site,
properties_and_views: properties_and_views, properties_and_views: properties_and_views,
selected_property_or_view_error: error,
legacy: legacy, legacy: legacy,
layout: {PlausibleWeb.LayoutView, "focus.html"} layout: {PlausibleWeb.LayoutView, "focus.html"}
) )
@ -71,103 +96,175 @@ defmodule PlausibleWeb.GoogleAnalyticsController do
) )
|> redirect(external: redirect_route) |> redirect(external: redirect_route)
{:error, :timeout} ->
conn
|> put_flash(
:error,
"Google Analytics API has timed out. Please try again."
)
|> redirect(external: redirect_route)
{:error, _any} -> {:error, _any} ->
conn conn
|> put_flash( |> put_flash(
:error, :error,
"We were unable to list your Google Analytics properties. If the problem persists, please contact support for assistance." "We were unable to list your Google Analytics properties and views. If the problem persists, please contact support for assistance."
) )
|> redirect(external: redirect_route) |> redirect(external: redirect_route)
end end
end end
# see https://stackoverflow.com/a/57416769 # see https://stackoverflow.com/a/57416769
@google_analytics_new_user_metric_date ~D[2016-08-24] @universal_analytics_new_user_metric_date ~D[2016-08-24]
def property_or_view(
conn,
%{
"property_or_view" => property_or_view,
"access_token" => access_token,
"refresh_token" => refresh_token,
"expires_at" => expires_at,
"legacy" => legacy
} = params
) do
site = conn.assigns.site
redirect_route =
if legacy == "true" do
Routes.site_path(conn, :settings_integrations, site.domain)
else
Routes.site_path(conn, :settings_imports_exports, site.domain)
end
with {:ok, api_start_date} <-
Google.API.get_analytics_start_date(access_token, property_or_view),
{:ok, api_end_date} <- Google.API.get_analytics_end_date(access_token, property_or_view),
{:ok, start_date, end_date} <- Imported.check_dates(site, api_start_date, api_end_date) do
action =
if Timex.before?(api_start_date, @universal_analytics_new_user_metric_date) do
:user_metric_notice
else
:confirm
end
redirect(conn,
external:
Routes.google_analytics_path(conn, action, site.domain,
property_or_view: property_or_view,
access_token: access_token,
refresh_token: refresh_token,
expires_at: expires_at,
start_date: Date.to_iso8601(start_date),
end_date: Date.to_iso8601(end_date),
legacy: legacy
)
)
else
{:error, error} when error in [:no_data, :no_time_window] ->
params =
params
|> Map.take(["access_token", "refresh_token", "expires_at", "legacy"])
|> Map.put("error", Atom.to_string(error))
property_or_view_form(conn, params)
{:error, :authentication_failed} ->
conn
|> put_flash(
:error,
"Google Analytics authentication seems to have expired. Please try again."
)
|> redirect(external: redirect_route)
{:error, :timeout} ->
conn
|> put_flash(
:error,
"Google Analytics API has timed out. Please try again."
)
|> redirect(external: redirect_route)
{:error, _any} ->
conn
|> put_flash(
:error,
"We were unable to retrieve information from Google Analytics. If the problem persists, please contact support for assistance."
)
|> redirect(external: redirect_route)
end
end
def property_or_view(conn, %{ def confirm(conn, %{
"property_or_view" => property_or_view, "property_or_view" => property_or_view,
"access_token" => access_token, "access_token" => access_token,
"refresh_token" => refresh_token, "refresh_token" => refresh_token,
"expires_at" => expires_at, "expires_at" => expires_at,
"start_date" => start_date,
"end_date" => end_date,
"legacy" => legacy "legacy" => legacy
}) do }) do
site = conn.assigns.site site = conn.assigns.site
start_date = Plausible.Google.API.get_analytics_start_date(access_token, property_or_view)
case start_date do start_date = Date.from_iso8601!(start_date)
{:ok, nil} -> end_date = Date.from_iso8601!(end_date)
{:ok, properties_and_views} =
if legacy == "true" do redirect_route =
Plausible.Google.UA.API.list_views(access_token) if legacy == "true" do
else Routes.site_path(conn, :settings_integrations, site.domain)
Plausible.Google.API.list_properties_and_views(access_token) else
end Routes.site_path(conn, :settings_imports_exports, site.domain)
end
case Google.API.get_property_or_view(access_token, property_or_view) do
{:ok, %{name: property_or_view_name, id: property_or_view}} ->
conn conn
|> assign(:skip_plausible_tracking, true) |> assign(:skip_plausible_tracking, true)
|> render("property_or_view_form.html", |> render("confirm.html",
access_token: access_token, access_token: access_token,
refresh_token: refresh_token, refresh_token: refresh_token,
expires_at: expires_at, expires_at: expires_at,
site: site, site: site,
properties_and_views: properties_and_views, selected_property_or_view: property_or_view,
selected_property_or_view_error: "No data found. Nothing to import", selected_property_or_view_name: property_or_view_name,
start_date: start_date,
end_date: end_date,
property?: Google.API.property?(property_or_view),
legacy: legacy, legacy: legacy,
layout: {PlausibleWeb.LayoutView, "focus.html"} layout: {PlausibleWeb.LayoutView, "focus.html"}
) )
{:ok, date} -> {:error, :authentication_failed} ->
action = conn
if Timex.before?(date, @google_analytics_new_user_metric_date) do |> put_flash(
:user_metric_notice :error,
else "Google Analytics authentication seems to have expired. Please try again."
:confirm
end
redirect(conn,
to:
Routes.google_analytics_path(conn, action, site.domain,
property_or_view: property_or_view,
access_token: access_token,
refresh_token: refresh_token,
expires_at: expires_at,
legacy: legacy
)
) )
end |> redirect(external: redirect_route)
end
def confirm(conn, %{
"property_or_view" => property_or_view,
"access_token" => access_token,
"refresh_token" => refresh_token,
"expires_at" => expires_at,
"legacy" => legacy
}) do
site = conn.assigns.site
start_date = Plausible.Google.API.get_analytics_start_date(access_token, property_or_view)
end_date = Plausible.Sites.native_stats_start_date(site) || Timex.today(site.timezone) {:error, :timeout} ->
conn
|> put_flash(
:error,
"Google Analytics API has timed out. Please try again."
)
|> redirect(external: redirect_route)
{:ok, %{name: property_or_view_name, id: property_or_view}} = {:error, :not_found} ->
Plausible.Google.API.get_property_or_view(access_token, property_or_view) conn
|> put_flash(
:error,
"Google Analytics property not found. Please try again."
)
|> redirect(external: redirect_route)
conn {:error, _any} ->
|> assign(:skip_plausible_tracking, true) conn
|> render("confirm.html", |> put_flash(
access_token: access_token, :error,
refresh_token: refresh_token, "We were unable to retrieve information from Google Analytics. If the problem persists, please contact support for assistance."
expires_at: expires_at, )
site: site, |> redirect(external: redirect_route)
selected_property_or_view: property_or_view, end
selected_property_or_view_name: property_or_view_name,
start_date: start_date,
end_date: end_date,
property?: Plausible.Google.API.property?(property_or_view),
legacy: legacy,
layout: {PlausibleWeb.LayoutView, "focus.html"}
)
end end
def import(conn, %{ def import(conn, %{
@ -182,6 +279,9 @@ defmodule PlausibleWeb.GoogleAnalyticsController do
site = conn.assigns.site site = conn.assigns.site
current_user = conn.assigns.current_user current_user = conn.assigns.current_user
start_date = Date.from_iso8601!(start_date)
end_date = Date.from_iso8601!(end_date)
redirect_route = redirect_route =
if legacy == "true" do if legacy == "true" do
Routes.site_path(conn, :settings_integrations, site.domain) Routes.site_path(conn, :settings_integrations, site.domain)
@ -189,36 +289,48 @@ defmodule PlausibleWeb.GoogleAnalyticsController do
Routes.site_path(conn, :settings_imports_exports, site.domain) Routes.site_path(conn, :settings_imports_exports, site.domain)
end end
if Plausible.Google.API.property?(property_or_view) do import_opts = [
{:ok, _} = label: property_or_view,
Plausible.Imported.GoogleAnalytics4.new_import( start_date: start_date,
site, end_date: end_date,
current_user, access_token: access_token,
property: property_or_view, refresh_token: refresh_token,
label: property_or_view, token_expires_at: expires_at,
start_date: start_date, legacy: legacy == "true"
end_date: end_date, ]
access_token: access_token,
refresh_token: refresh_token, with {:ok, start_date, end_date} <- Imported.check_dates(site, start_date, end_date),
token_expires_at: expires_at import_opts = [{:start_date, start_date}, {:end_date, end_date} | import_opts],
) {:ok, _} <- schedule_job(site, current_user, property_or_view, import_opts) do
conn
|> put_flash(:success, "Import scheduled. An email will be sent when it completes.")
|> redirect(external: redirect_route)
else else
Plausible.Imported.UniversalAnalytics.new_import( {:error, :no_time_window} ->
site, conn
current_user, |> put_flash(
view_id: property_or_view, :error,
label: property_or_view, "Import failed. No data could be imported because date range overlaps with existing data."
start_date: start_date, )
end_date: end_date, |> redirect(external: redirect_route)
access_token: access_token,
refresh_token: refresh_token, {:error, :import_in_progress} ->
token_expires_at: expires_at, conn
legacy: legacy == "true" |> put_flash(
) :error,
"There's another import still in progress. Please wait until it's completed or cancel it before starting a new one."
)
|> redirect(external: redirect_route)
end end
end
conn defp schedule_job(site, current_user, property_or_view, opts) do
|> put_flash(:success, "Import scheduled. An email will be sent when it completes.") if Google.API.property?(property_or_view) do
|> redirect(external: redirect_route) opts = Keyword.put(opts, :property, property_or_view)
Imported.GoogleAnalytics4.new_import(site, current_user, opts)
else
opts = Keyword.put(opts, :view_id, property_or_view)
Imported.UniversalAnalytics.new_import(site, current_user, opts)
end
end end
end end

@ -652,7 +652,7 @@ defmodule PlausibleWeb.SiteController do
|> Plausible.Site.remove_imported_data() |> Plausible.Site.remove_imported_data()
|> Repo.update!() |> Repo.update!()
site_import = Plausible.Imported.get_import(import_id) -> site_import = Plausible.Imported.get_import(site, import_id) ->
Oban.cancel_all_jobs( Oban.cancel_all_jobs(
from(j in Oban.Job, from(j in Oban.Job,
where: where:
@ -666,6 +666,8 @@ defmodule PlausibleWeb.SiteController do
Plausible.Repo.delete!(site_import) Plausible.Repo.delete!(site_import)
if site_import.legacy do if site_import.legacy do
Plausible.Purge.delete_imported_stats!(site, 0)
site site
|> Plausible.Site.remove_imported_data() |> Plausible.Site.remove_imported_data()
|> Repo.update!() |> Repo.update!()

@ -64,13 +64,14 @@ defmodule PlausibleWeb.StatsController do
|> render("stats.html", |> render("stats.html",
site: site, site: site,
has_goals: Plausible.Sites.has_goals?(site), has_goals: Plausible.Sites.has_goals?(site),
revenue_goals: list_revenue_goals(site),
funnels: list_funnels(site), funnels: list_funnels(site),
has_props: Plausible.Props.configured?(site), has_props: Plausible.Props.configured?(site),
stats_start_date: stats_start_date, stats_start_date: stats_start_date,
native_stats_start_date: NaiveDateTime.to_date(site.native_stats_start_at), native_stats_start_date: NaiveDateTime.to_date(site.native_stats_start_at),
title: title(conn, site), title: title(conn, site),
demo: demo, demo: demo,
flags: get_flags(conn.assigns[:current_user]), flags: get_flags(conn.assigns[:current_user], site),
is_dbip: is_dbip(), is_dbip: is_dbip(),
dogfood_page_path: dogfood_page_path, dogfood_page_path: dogfood_page_path,
load_dashboard_js: true load_dashboard_js: true
@ -92,10 +93,13 @@ defmodule PlausibleWeb.StatsController do
defp list_funnels(site) do defp list_funnels(site) do
Plausible.Funnels.list(site) Plausible.Funnels.list(site)
end end
else
defp list_funnels(_site) do defp list_revenue_goals(site) do
[] Plausible.Goals.list_revenue_goals(site)
end end
else
defp list_funnels(_site), do: []
defp list_revenue_goals(_site), do: []
end end
@doc """ @doc """
@ -319,6 +323,7 @@ defmodule PlausibleWeb.StatsController do
|> render("stats.html", |> render("stats.html",
site: shared_link.site, site: shared_link.site,
has_goals: Sites.has_goals?(shared_link.site), has_goals: Sites.has_goals?(shared_link.site),
revenue_goals: list_revenue_goals(shared_link.site),
funnels: list_funnels(shared_link.site), funnels: list_funnels(shared_link.site),
has_props: Plausible.Props.configured?(shared_link.site), has_props: Plausible.Props.configured?(shared_link.site),
stats_start_date: stats_start_date, stats_start_date: stats_start_date,
@ -330,7 +335,7 @@ defmodule PlausibleWeb.StatsController do
embedded: conn.params["embed"] == "true", embedded: conn.params["embed"] == "true",
background: conn.params["background"], background: conn.params["background"],
theme: conn.params["theme"], theme: conn.params["theme"],
flags: get_flags(conn.assigns[:current_user]), flags: get_flags(conn.assigns[:current_user], shared_link.site),
is_dbip: is_dbip(), is_dbip: is_dbip(),
load_dashboard_js: true load_dashboard_js: true
) )
@ -348,8 +353,12 @@ defmodule PlausibleWeb.StatsController do
defp shared_link_cookie_name(slug), do: "shared-link-" <> slug defp shared_link_cookie_name(slug), do: "shared-link-" <> slug
defp get_flags(_user) do defp get_flags(user, site) do
%{} %{
hostname_filter:
FunWithFlags.enabled?(:hostname_filter, for: user) ||
FunWithFlags.enabled?(:hostname_filter, for: site)
}
end end
defp is_dbip() do defp is_dbip() do

@ -42,7 +42,39 @@ defmodule PlausibleWeb.Live.ImportsExportsSettings do
end end
def render(assigns) do def render(assigns) do
import_in_progress? =
Enum.any?(
assigns.site_imports,
&(&1.live_status in [SiteImport.pending(), SiteImport.importing()])
)
assigns = assign(assigns, :import_in_progress?, import_in_progress?)
~H""" ~H"""
<div class="mt-5 flex gap-x-4">
<.button_link
class="w-36 h-20"
theme="bright"
disabled={@import_in_progress?}
href={Plausible.Google.API.import_authorize_url(@site.id, "import", legacy: false)}
>
<img src="/images/icon/google_analytics_logo.svg" alt="Google Analytics import" />
</.button_link>
<.button_link
class="w-36 h-20"
theme="bright"
disabled={@import_in_progress?}
href={"/#{URI.encode_www_form(@site.domain)}/settings/import"}
>
<img class="h-16" src="/images/icon/csv_logo.svg" alt="New CSV import" />
</.button_link>
</div>
<p :if={@import_in_progress?} class="mt-4 text-red-400 text-sm">
No new imports can be started until the import in progress is completed or cancelled.
</p>
<header class="relative border-b border-gray-200 pb-4"> <header class="relative border-b border-gray-200 pb-4">
<h3 class="mt-8 text-md leading-6 font-medium text-gray-900 dark:text-gray-100"> <h3 class="mt-8 text-md leading-6 font-medium text-gray-900 dark:text-gray-100">
Existing Imports Existing Imports
@ -62,9 +94,11 @@ defmodule PlausibleWeb.Live.ImportsExportsSettings do
<li :for={entry <- @site_imports} class="py-4 flex items-center justify-between space-x-4"> <li :for={entry <- @site_imports} class="py-4 flex items-center justify-between space-x-4">
<div class="flex flex-col"> <div class="flex flex-col">
<p class="text-sm leading-5 font-medium text-gray-900 dark:text-gray-100"> <p class="text-sm leading-5 font-medium text-gray-900 dark:text-gray-100">
Import from <%= Plausible.Imported.SiteImport.label(entry.site_import) %> <%= Plausible.Imported.SiteImport.label(entry.site_import) %>
<span :if={entry.live_status == SiteImport.completed()} class="text-xs font-normal"> <span :if={entry.live_status == SiteImport.completed()} class="text-xs font-normal">
(<%= Map.get(@pageview_counts, entry.site_import.id, 0) %> page views) (<%= PlausibleWeb.StatsView.large_number_format(
Map.get(@pageview_counts, entry.site_import.id, 0)
) %> page views)
</span> </span>
<Heroicons.clock <Heroicons.clock
:if={entry.live_status == SiteImport.pending()} :if={entry.live_status == SiteImport.pending()}

@ -6,22 +6,8 @@
<%= link("Click here", to: @link) %> to view your dashboard. <%= link("Click here", to: @link) %> to view your dashboard.
<% else %> <% else %>
Unfortunately, your Google Analytics import for <%= @site_import.site.domain %> did not complete successfully. Sorry about that! Unfortunately, your Google Analytics import for <%= @site_import.site.domain %> did not complete successfully. Sorry about that!
<br /> <br /> <br /><br />
This may be due to the user metric setting in your Google Analytics account. We found that changing the user metric setting is necessary to get the correct data out of the Google API. Please try to do the import once again. Sometimes the Google Analytics API just randomly returns empty data. It's intermittent and random. Trying to do the import again may return what you need.
<br /> <br />
Please log into your Google Analytics account and go to Admin -> Property Settings -> User Analysis. Then make sure that "Enable User Metric in Reporting" is set to OFF. Then try importing again. You can switch the setting back after the import.
<br /> <br /> If you've already changed that setting, then it is something else: <br />
<ul>
<li>
Do you have any stats in the Google Analytics property you tried to import? We only import the stats from your first Google Analytics visitor until your first Plausible Analytics visitor. The import fails if there's no returned data from Google from before you counted your first Plausible visitor.
</li>
<li>
Did you set strict data retention limits in your Google Analytics account? You can check the data retention limit you have set for your property in your Google Analytics admin settings by visiting the "Tracking Info" section and clicking on "Data Retention".
</li>
</ul>
<br />
If all of the above is fine, please try to do the import once again. Sometimes the Google Analytics API just randomly returns empty data. It's intermittent and random. Trying to do the import again may return what you need.
<%= if full_build?() do %> <%= if full_build?() do %>
<br /> <br /> <br /> <br />
Please reply to this email to let us know if you're still experiencing issues with the import. Please reply to this email to let us know if you're still experiencing issues with the import.

@ -6,55 +6,47 @@
<%= hidden_input(f, :expires_at, value: @expires_at) %> <%= hidden_input(f, :expires_at, value: @expires_at) %>
<%= hidden_input(f, :legacy, value: @legacy) %> <%= hidden_input(f, :legacy, value: @legacy) %>
<%= case @start_date do %> <div class="mt-6 text-sm text-gray-500 dark:text-gray-200">
<% {:ok, start_date} -> %> Stats from this
<div class="mt-6 text-sm text-gray-500 dark:text-gray-200"> <%= if @property? do %>
Stats from this property
<%= if @property? do %> <% else %>
property view
<% else %> <% end %>
view and time period will be imported from your Google Analytics account to your Plausible dashboard
<% end %> </div>
and time period will be imported from your Google Analytics account to your Plausible dashboard
</div>
<div class="mt-6"> <div class="mt-6">
<%= styled_label( <%= styled_label(
f, f,
:property_or_view, :property_or_view,
"Google Analytics #{if @property?, do: "property", else: "view"}" "Google Analytics #{if @property?, do: "property", else: "view"}"
) %> ) %>
<span class="block w-full text-base dark:text-gray-100 sm:text-sm dark:bg-gray-800"> <span class="block w-full text-base dark:text-gray-100 sm:text-sm dark:bg-gray-800">
<%= @selected_property_or_view_name %> <%= @selected_property_or_view_name %>
</span> </span>
<%= hidden_input(f, :property_or_view, <%= hidden_input(f, :property_or_view,
readonly: "true", readonly: "true",
value: @selected_property_or_view value: @selected_property_or_view
) %> ) %>
</div> </div>
<div class="flex justify-between mt-3"> <div class="flex justify-between mt-3">
<div class="w-36"> <div class="w-36">
<%= styled_label(f, :start_date, "From") %> <%= styled_label(f, :start_date, "From") %>
<span class="block w-full text-base dark:text-gray-100 sm:text-sm dark:bg-gray-800"> <span class="block w-full text-base dark:text-gray-100 sm:text-sm dark:bg-gray-800">
<%= PlausibleWeb.EmailView.date_format(start_date) %> <%= PlausibleWeb.EmailView.date_format(@start_date) %>
</span> </span>
<%= hidden_input(f, :start_date, value: start_date, readonly: "true") %> <%= hidden_input(f, :start_date, value: @start_date, readonly: "true") %>
</div> </div>
<div class="align-middle pt-4 dark:text-gray-100">&rarr;</div> <div class="align-middle pt-4 dark:text-gray-100">&rarr;</div>
<div class="w-36"> <div class="w-36">
<%= styled_label(f, :end_date, "To") %> <%= styled_label(f, :end_date, "To") %>
<span class="block w-full text-base dark:text-gray-100 sm:text-sm dark:bg-gray-800"> <span class="block w-full text-base dark:text-gray-100 sm:text-sm dark:bg-gray-800">
<%= PlausibleWeb.EmailView.date_format(@end_date) %> <%= PlausibleWeb.EmailView.date_format(@end_date) %>
</span> </span>
<%= hidden_input(f, :end_date, value: @end_date, readonly: "true") %> <%= hidden_input(f, :end_date, value: @end_date, readonly: "true") %>
</div> </div>
</div> </div>
<% {:error, error} -> %>
<p class="text-gray-700 dark:text-gray-300 mt-6">
The following error occurred when fetching your Google Analytics data.
</p>
<p class="text-red-700 font-medium mt-3"><%= error %></p>
<% end %>
<%= submit("Confirm import", class: "button mt-6") %> <%= submit("Confirm import", class: "button mt-6") %>
<% end %> <% end %>

@ -39,6 +39,8 @@
access_token: @access_token, access_token: @access_token,
refresh_token: @refresh_token, refresh_token: @refresh_token,
expires_at: @expires_at, expires_at: @expires_at,
start_date: @start_date,
end_date: @end_date,
legacy: @legacy legacy: @legacy
), ),
class: "button mt-6" class: "button mt-6"

@ -10,24 +10,6 @@
<PlausibleWeb.Components.Generic.docs_info slug="google-analytics-import" /> <PlausibleWeb.Components.Generic.docs_info slug="google-analytics-import" />
</header> </header>
<div class="mt-5 flex gap-x-4">
<PlausibleWeb.Components.Generic.button_link
class="w-36 h-20"
theme="bright"
href={Plausible.Google.API.import_authorize_url(@site.id, "import", legacy: false)}
>
<img src="/images/icon/google_analytics_logo.svg" alt="Google Analytics import" />
</PlausibleWeb.Components.Generic.button_link>
<PlausibleWeb.Components.Generic.button_link
class="w-36 h-20"
theme="bright"
href={"/#{URI.encode_www_form(@site.domain)}/settings/import"}
>
<img class="h-16" src="/images/icon/csv_logo.svg" alt="New CSV import" />
</PlausibleWeb.Components.Generic.button_link>
</div>
<%= live_render(@conn, PlausibleWeb.Live.ImportsExportsSettings, <%= live_render(@conn, PlausibleWeb.Live.ImportsExportsSettings,
session: %{"domain" => @site.domain} session: %{"domain" => @site.domain}
) %> ) %>

@ -28,11 +28,11 @@
Please subscribe to the appropriate tier with the link below to access the stats again. Please subscribe to the appropriate tier with the link below to access the stats again.
</p> </p>
<p class="mt-6 text-sm text-gray-500"> <p class="mt-6 text-sm text-gray-500">
You can check the You can configure your
<.styled_link href={Routes.site_path(@conn, :settings_general, @site.domain)}> <.styled_link href={Routes.site_path(@conn, :settings_general, @site.domain)}>
site settings site settings
</.styled_link> </.styled_link>
and we're still counting the stats but your access to the dashboard is restricted. but your access to the dashboard is restricted.
</p> </p>
</div> </div>
<div class="mt-6 w-full text-center"> <div class="mt-6 w-full text-center">

@ -27,6 +27,7 @@
data-props-available={ data-props-available={
to_string(Plausible.Billing.Feature.Props.check_availability(@site.owner) == :ok) to_string(Plausible.Billing.Feature.Props.check_availability(@site.owner) == :ok)
} }
data-revenue-goals={Jason.encode!(@revenue_goals)}
data-funnels={Jason.encode!(@funnels)} data-funnels={Jason.encode!(@funnels)}
data-has-props={to_string(@has_props)} data-has-props={to_string(@has_props)}
data-logged-in={to_string(!!@conn.assigns[:current_user])} data-logged-in={to_string(!!@conn.assigns[:current_user])}

@ -147,7 +147,7 @@
"timex": {:hex, :timex, "3.7.11", "bb95cb4eb1d06e27346325de506bcc6c30f9c6dea40d1ebe390b262fad1862d1", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:gettext, "~> 0.20", [hex: :gettext, repo: "hexpm", optional: false]}, {:tzdata, "~> 1.1", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm", "8b9024f7efbabaf9bd7aa04f65cf8dcd7c9818ca5737677c7b76acbc6a94d1aa"}, "timex": {:hex, :timex, "3.7.11", "bb95cb4eb1d06e27346325de506bcc6c30f9c6dea40d1ebe390b262fad1862d1", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:gettext, "~> 0.20", [hex: :gettext, repo: "hexpm", optional: false]}, {:tzdata, "~> 1.1", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm", "8b9024f7efbabaf9bd7aa04f65cf8dcd7c9818ca5737677c7b76acbc6a94d1aa"},
"tls_certificate_check": {:hex, :tls_certificate_check, "1.21.0", "042ab2c0c860652bc5cf69c94e3a31f96676d14682e22ec7813bd173ceff1788", [:rebar3], [{:ssl_verify_fun, "~> 1.1", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}], "hexpm", "6cee6cffc35a390840d48d463541d50746a7b0e421acaadb833cfc7961e490e7"}, "tls_certificate_check": {:hex, :tls_certificate_check, "1.21.0", "042ab2c0c860652bc5cf69c94e3a31f96676d14682e22ec7813bd173ceff1788", [:rebar3], [{:ssl_verify_fun, "~> 1.1", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}], "hexpm", "6cee6cffc35a390840d48d463541d50746a7b0e421acaadb833cfc7961e490e7"},
"tzdata": {:hex, :tzdata, "1.1.1", "20c8043476dfda8504952d00adac41c6eda23912278add38edc140ae0c5bcc46", [:mix], [{:hackney, "~> 1.17", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "a69cec8352eafcd2e198dea28a34113b60fdc6cb57eb5ad65c10292a6ba89787"}, "tzdata": {:hex, :tzdata, "1.1.1", "20c8043476dfda8504952d00adac41c6eda23912278add38edc140ae0c5bcc46", [:mix], [{:hackney, "~> 1.17", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "a69cec8352eafcd2e198dea28a34113b60fdc6cb57eb5ad65c10292a6ba89787"},
"ua_inspector": {:hex, :ua_inspector, "3.8.0", "c0b0d13200a9bd509225f15ea8cf275c0bec27390a21c355746ff8b8a88c3e4d", [:mix], [{:hackney, "~> 1.0", [hex: :hackney, repo: "hexpm", optional: false]}, {:yamerl, "~> 0.7", [hex: :yamerl, repo: "hexpm", optional: false]}], "hexpm", "7c980bae82a4754075b933e0f383935a681e5a2628856ad3ecf6eb80d8139539"}, "ua_inspector": {:hex, :ua_inspector, "3.9.0", "2021bbddb1ee41f202da7f006fb09f5c5617ad579108b7b9bcf1881828462f04", [:mix], [{:hackney, "~> 1.0", [hex: :hackney, repo: "hexpm", optional: false]}, {:yamerl, "~> 0.7", [hex: :yamerl, repo: "hexpm", optional: false]}], "hexpm", "0f2d1b9806e78b14b91b6f20eaaa4a68f8989f4f2a99a376b874c295ac50a30d"},
"unicode_util_compat": {:hex, :unicode_util_compat, "0.7.0", "bc84380c9ab48177092f43ac89e4dfa2c6d62b40b8bd132b1059ecc7232f9a78", [:rebar3], [], "hexpm", "25eee6d67df61960cf6a794239566599b09e17e668d3700247bc498638152521"}, "unicode_util_compat": {:hex, :unicode_util_compat, "0.7.0", "bc84380c9ab48177092f43ac89e4dfa2c6d62b40b8bd132b1059ecc7232f9a78", [:rebar3], [], "hexpm", "25eee6d67df61960cf6a794239566599b09e17e668d3700247bc498638152521"},
"uuid": {:hex, :uuid, "1.1.8", "e22fc04499de0de3ed1116b770c7737779f226ceefa0badb3592e64d5cfb4eb9", [:mix], [], "hexpm", "c790593b4c3b601f5dc2378baae7efaf5b3d73c4c6456ba85759905be792f2ac"}, "uuid": {:hex, :uuid, "1.1.8", "e22fc04499de0de3ed1116b770c7737779f226ceefa0badb3592e64d5cfb4eb9", [:mix], [], "hexpm", "c790593b4c3b601f5dc2378baae7efaf5b3d73c4c6456ba85759905be792f2ac"},
"websock": {:hex, :websock, "0.5.3", "2f69a6ebe810328555b6fe5c831a851f485e303a7c8ce6c5f675abeb20ebdadc", [:mix], [], "hexpm", "6105453d7fac22c712ad66fab1d45abdf049868f253cf719b625151460b8b453"}, "websock": {:hex, :websock, "0.5.3", "2f69a6ebe810328555b6fe5c831a851f485e303a7c8ce6c5f675abeb20ebdadc", [:mix], [], "hexpm", "6105453d7fac22c712ad66fab1d45abdf049868f253cf719b625151460b8b453"},

@ -0,0 +1,35 @@
defmodule Plausible.IngestRepo.Migrations.AddMetricsToImportedTables do
use Ecto.Migration
@add_pageviews_for [
"imported_browsers",
"imported_devices",
"imported_entry_pages",
"imported_exit_pages",
"imported_locations",
"imported_operating_systems",
"imported_sources"
]
def up do
for table <- @add_pageviews_for do
add_column(table, "pageviews", "UInt64")
end
add_column("imported_browsers", "browser_version", "String")
add_column("imported_exit_pages", "bounces", "UInt32")
add_column("imported_exit_pages", "visit_duration", "UInt64")
add_column("imported_operating_systems", "operating_system_version", "String")
add_column("imported_pages", "visits", "UInt64")
add_column("imported_sources", "referrer", "String")
add_column("imported_sources", "utm_source", "String")
end
defp add_column(table, column, type) do
execute """
ALTER TABLE #{table}
#{Plausible.MigrationUtils.on_cluster_statement(table)}
ADD COLUMN #{column} #{type}
"""
end
end

@ -0,0 +1,19 @@
defmodule Plausible.ClickhouseRepo.Migrations.HostnamesInSessions do
use Ecto.Migration
def up do
execute """
ALTER TABLE sessions_v2
#{Plausible.MigrationUtils.on_cluster_statement("sessions_v2")}
ADD COLUMN exit_page_hostname String CODEC(ZSTD(3))
"""
end
def down do
execute """
ALTER TABLE sessions_v2
#{Plausible.MigrationUtils.on_cluster_statement("sessions_v2")}
DROP COLUMN exit_page_hostname
"""
end
end

@ -12,6 +12,7 @@
FunWithFlags.enable(:imports_exports) FunWithFlags.enable(:imports_exports)
FunWithFlags.enable(:shield_pages) FunWithFlags.enable(:shield_pages)
FunWithFlags.enable(:hostname_filter)
user = Plausible.Factory.insert(:user, email: "user@plausible.test", password: "plausible") user = Plausible.Factory.insert(:user, email: "user@plausible.test", password: "plausible")
@ -161,7 +162,7 @@ native_stats_range
[ [
site_id: site.id, site_id: site.id,
hostname: site.domain, hostname: Enum.random(["en.dummy.site", "es.dummy.site", "dummy.site"]),
timestamp: put_random_time.(date, index), timestamp: put_random_time.(date, index),
referrer_source: Enum.random(["", "Facebook", "Twitter", "DuckDuckGo", "Google"]), referrer_source: Enum.random(["", "Facebook", "Twitter", "DuckDuckGo", "Google"]),
browser: Enum.random(["Edge", "Chrome", "Safari", "Firefox", "Vivaldi"]), browser: Enum.random(["Edge", "Chrome", "Safari", "Firefox", "Vivaldi"]),
@ -196,7 +197,7 @@ native_stats_range
[ [
name: goal4.event_name, name: goal4.event_name,
site_id: site.id, site_id: site.id,
hostname: site.domain, hostname: Enum.random(["en.dummy.site", "es.dummy.site", "dummy.site"]),
timestamp: put_random_time.(date, index), timestamp: put_random_time.(date, index),
referrer_source: Enum.random(["", "Facebook", "Twitter", "DuckDuckGo", "Google"]), referrer_source: Enum.random(["", "Facebook", "Twitter", "DuckDuckGo", "Google"]),
browser: Enum.random(["Edge", "Chrome", "Safari", "Firefox", "Vivaldi"]), browser: Enum.random(["Edge", "Chrome", "Safari", "Firefox", "Vivaldi"]),

File diff suppressed because it is too large Load Diff

@ -8,13 +8,13 @@
- regex: 'NetFront' - regex: 'NetFront'
name: 'NetFront' name: 'NetFront'
- regex: 'Edge' - regex: 'Edge/'
name: 'Edge' name: 'Edge'
- regex: 'Trident' - regex: 'Trident'
name: 'Trident' name: 'Trident'
- regex: '(?<!SmartHu)Blink' - regex: 'Chrome/(?!1?\d\.|2[0-7]\.)'
name: 'Blink' name: 'Blink'
- regex: '(?:Apple)?WebKit' - regex: '(?:Apple)?WebKit'

@ -5,6 +5,82 @@
# @license http://www.gnu.org/licenses/lgpl.html LGPL v3 or later # @license http://www.gnu.org/licenses/lgpl.html LGPL v3 or later
############### ###############
# Quick Search TV (https://play.google.com/store/apps/details?id=com.aospstudio.tvsearch)
- regex: 'Quick Search TV(?:/(?:Wild Moon Edition )?(\d+[\.\d]+))?'
name: 'Quick Search TV'
version: '$1'
# Skye (https://innatical.com/)
- regex: 'Skye/(\d+\.[\.\d]+)'
name: 'Skye'
version: '$1'
engine:
default: 'Blink'
# VD Browser (vpn.video.downloader)
- regex: 'VD/\d+'
name: 'VD Browser'
version: ''
engine:
default: 'Blink'
# SecureX (https://play.google.com/store/apps/details?id=com.securex.browser)
- regex: '\[(?:HB/29|PB/(?:66|81))\]'
name: 'SecureX'
version: ''
engine:
default: 'Blink'
# HotBrowser (https://play.google.com/store/apps/details?id=com.hotsurf.browser)
- regex: '\[HS/\d+\]'
name: 'HotBrowser'
version: ''
engine:
default: 'Blink'
# Proxy Browser (https://play.google.com/store/apps/details?id=proxy.browser.unblock.sites.proxybrowser.unblocksites)
- regex: '\[PB/\d+\]'
name: 'Proxy Browser'
version: ''
engine:
default: 'Blink'
# Onion Browser (https://apps.apple.com/us/app/onion-browser/id519296448)
- regex: '^Normalized (?:iPad|iPhone) \(iOS Safari\)'
name: 'Onion Browser'
version: ''
engine:
default: 'WebKit'
# fGet (https://apps.apple.com/us/app/fget-file-manager-browser/id1582654012)
- regex: 'fGet/'
name: 'fGet'
version: ''
# Nuviu (https://www.nuviu-browser.com/)
- regex: 'Nuviu/(?:(\d+[\.\d]+))?'
name: 'Nuviu'
version: '$1'
# DoCoMo (https://play.google.com/store/apps/details?id=com.airfind.browser)
- regex: 'DoCoMo/(?:(\d+[\.\d]+))?'
name: 'DoCoMo'
version: '$1'
# Airfind Secure Browser (https://play.google.com/store/apps/details?id=com.airfind.browser)
- regex: 'com.airfind.browser/(?:(\d+[\.\d]+))?'
name: 'Airfind Secure Browser'
version: '$1'
engine:
default: 'Blink'
# Arc (https://arc.net/)
- regex: 'ArcMobile2/.+CFNetwork'
name: 'Arc'
version: ''
engine:
default: 'WebKit'
# Nuanti Meta (http://meta.nuanti.com/) # Nuanti Meta (http://meta.nuanti.com/)
- regex: 'Nuanti(?:Meta)?/(\d+\.[\.\d]+)' - regex: 'Nuanti(?:Meta)?/(\d+\.[\.\d]+)'
name: 'Nuanti Meta' name: 'Nuanti Meta'
@ -119,6 +195,11 @@
version: '$1' version: '$1'
engine: engine:
default: 'Blink' default: 'Blink'
- regex: 'Lilo/.+CFNetwork'
name: 'Lilo'
version: ''
engine:
default: 'WebKit'
# Lexi Browser (https://play.google.com/store/apps/details?id=com.lexi.browser) # Lexi Browser (https://play.google.com/store/apps/details?id=com.lexi.browser)
- regex: 'lexi/(\d+[\.\d]+)' - regex: 'lexi/(\d+[\.\d]+)'
@ -302,7 +383,7 @@
name: 'Edge WebView' name: 'Edge WebView'
version: '$1' version: '$1'
engine: engine:
default: 'WebKit' default: 'Blink'
# Mandarin (https://mandarin-browser.com/) # Mandarin (https://mandarin-browser.com/)
- regex: 'Mandarin Browser/(\d+[\.\d]+)' - regex: 'Mandarin Browser/(\d+[\.\d]+)'
@ -850,7 +931,7 @@
name: 'Firefox Reality' name: 'Firefox Reality'
version: '' version: ''
# AVG Secure Browser (https://www.avg.com/en-us/secure-browser) # AVG Secure Browser (https://www.avg.com/en-us/secure-browser | https://play.google.com/store/apps/details?id=com.avg.android.secure.browser)
- regex: 'AVG(?:/(\d+[\.\d]+))?' - regex: 'AVG(?:/(\d+[\.\d]+))?'
name: 'AVG Secure Browser' name: 'AVG Secure Browser'
version: '$1' version: '$1'
@ -858,6 +939,11 @@
default: 'WebKit' default: 'WebKit'
versions: versions:
28: 'Blink' 28: 'Blink'
- regex: 'AT/(\d+[\.\d]+)'
name: 'AVG Secure Browser'
version: '$1'
engine:
default: 'Blink'
# START Internet Browser (https://play.google.com/store/apps/details?id=fyi.start.web.browser) # START Internet Browser (https://play.google.com/store/apps/details?id=fyi.start.web.browser)
- regex: 'Start/(?:(\d+[\.\d]+))?' - regex: 'Start/(?:(\d+[\.\d]+))?'
@ -1111,11 +1197,16 @@
version: '$1' version: '$1'
engine: engine:
default: 'WebKit' default: 'WebKit'
- regex: 'Chrome/.*QwantMobile(?:/(\d+[\.\d]+))?'
name: 'Qwant Mobile'
version: '$1'
engine:
default: 'Blink'
- regex: 'QwantMobile(?:/(\d+[\.\d]+))?' - regex: 'QwantMobile(?:/(\d+[\.\d]+))?'
name: 'Qwant Mobile' name: 'Qwant Mobile'
version: '$1' version: '$1'
engine: engine:
default: 'Gecko' default: '' # multiple (Gecko on Android, WebKit on iOS)
- regex: 'Qwant/(\d+[\.\d]+)' - regex: 'Qwant/(\d+[\.\d]+)'
name: 'Qwant Mobile' name: 'Qwant Mobile'
version: '$1' version: '$1'
@ -1186,7 +1277,7 @@
name: 'Sailfish Browser' name: 'Sailfish Browser'
version: '$1' version: '$1'
engine: engine:
default: 'Gecko' default: ''
# IceCat # IceCat
- regex: 'IceCat(?:/(\d+[\.\d]+))?' - regex: 'IceCat(?:/(\d+[\.\d]+))?'
@ -1719,7 +1810,7 @@
default: '' # multi engine default: '' # multi engine
#Comodo Dragon #Comodo Dragon
- regex: '(?:Comodo[ _])?Dragon(?!fruit)(?:/(\d+[\.\d]+))?' - regex: '(?:Comodo[ _])?Dragon/(\d+[\.\d]+)'
name: 'Comodo Dragon' name: 'Comodo Dragon'
version: '$1' version: '$1'
engine: engine:
@ -1832,7 +1923,7 @@
engine: engine:
default: 'WebKit' default: 'WebKit'
versions: versions:
'4.5': 'Blink' '4.2': 'Blink'
- regex: '(?:Maxthon(?:%20Browser)?|MxBrowser(?:-inhouse|-iPhone)?|MXiOS)[ /](\d+[\.\d]+)?' - regex: '(?:Maxthon(?:%20Browser)?|MxBrowser(?:-inhouse|-iPhone)?|MXiOS)[ /](\d+[\.\d]+)?'
name: 'Maxthon' name: 'Maxthon'
version: '$1' version: '$1'
@ -1868,7 +1959,7 @@
versions: versions:
28: 'Blink' 28: 'Blink'
- regex: 'Iron(?:/(\d+[\.\d]+))?' - regex: 'Iron/(\d+[\.\d]+)'
name: 'Iron' name: 'Iron'
version: '$1' version: '$1'
engine: engine:
@ -1976,14 +2067,14 @@
name: 'Streamy' name: 'Streamy'
version: '$1' version: '$1'
engine: engine:
default: 'WebKit' default: ''
#Isivioo (Android version is Chrome based) #Isivioo (Android version is Chrome based)
- regex: 'isivioo' - regex: 'isivioo'
name: 'Isivioo' name: 'Isivioo'
version: '' version: ''
engine: engine:
default: 'WebKit' default: ''
#UC Browser #UC Browser
- regex: 'UC[ ]?Browser(?:[ /]?(\d+[\.\d]+))?' - regex: 'UC[ ]?Browser(?:[ /]?(\d+[\.\d]+))?'
@ -2032,12 +2123,12 @@
name: 'Web Explorer' name: 'Web Explorer'
version: '$1' version: '$1'
engine: engine:
default: 'WebKit' default: 'Blink'
- regex: 'webexplorer/(\d+)' - regex: 'webexplorer/(\d+)'
name: 'Web Explorer' name: 'Web Explorer'
version: '$1' version: '$1'
engine: engine:
default: 'WebKit' default: ''
# Seznam Browser # Seznam Browser
- regex: 'Chrome.+SznProhlizec/(\d+[\.\d]+)' - regex: 'Chrome.+SznProhlizec/(\d+[\.\d]+)'
@ -2193,28 +2284,28 @@
name: 'Quark' name: 'Quark'
version: '$1' version: '$1'
engine: engine:
default: 'WebKit' default: 'Blink'
# Monument Browser # Monument Browser
- regex: 'MonumentBrowser(?:/(\d+[\.\d]+))?' - regex: 'MonumentBrowser(?:/(\d+[\.\d]+))?'
name: 'Monument Browser' name: 'Monument Browser'
version: '$1' version: '$1'
engine: engine:
default: 'WebKit' default: 'Blink'
# Blue Browser # Blue Browser
- regex: 'BlueBrowser(?:/(\d+[\.\d]+))?' - regex: 'BlueBrowser(?:/(\d+[\.\d]+))?'
name: 'Blue Browser' name: 'Blue Browser'
version: '$1' version: '$1'
engine: engine:
default: 'WebKit' default: 'Blink'
# Japan Browser (https://play.google.com/store/apps/details?id=com.japan.browser.anti.blokir) # Japan Browser (https://play.google.com/store/apps/details?id=com.japan.browser.anti.blokir)
- regex: 'JAPAN Browser(?:/(\d+[\.\d]+))?' - regex: 'JAPAN Browser(?:/(\d+[\.\d]+))?'
name: 'Japan Browser' name: 'Japan Browser'
version: '$1' version: '$1'
engine: engine:
default: 'WebKit' default: 'Blink'
- regex: 'OpenFin/(?:(\d+[\.\d]+))' - regex: 'OpenFin/(?:(\d+[\.\d]+))'
name: 'OpenFin' name: 'OpenFin'

@ -198,6 +198,7 @@
- regex: 'Faraday(?: v(\d+[\.\d]+))?' - regex: 'Faraday(?: v(\d+[\.\d]+))?'
name: 'Faraday' name: 'Faraday'
version: '$1' version: '$1'
url: 'https://github.com/lostisland/faraday'
- regex: '(?:Go-http-client|Go )/?(?:(\d+[\.\d]+))?(?: package http)?' - regex: '(?:Go-http-client|Go )/?(?:(\d+[\.\d]+))?(?: package http)?'
name: 'Go-http-client' name: 'Go-http-client'
@ -520,3 +521,48 @@
- regex: 'AppleCoreMedia/1\.0\.0' - regex: 'AppleCoreMedia/1\.0\.0'
name: 'iOS Application' name: 'iOS Application'
version: '' version: ''
- regex: 'cpp-httplib(?:/(\d+[\.\d]+))?'
name: 'cpp-httplib'
version: '$1'
url: 'https://github.com/yhirose/cpp-httplib'
- regex: 'Definitely-Not-Requests'
name: 'Requests'
version: ''
url: 'https://github.com/psf/requests'
- regex: 'Stealer ([\d\.]+)'
name: 'Stealer'
version: '$1'
url: 'https://github.com/hotrush/stealer/'
- regex: 'Mandrill-PHP(?:/(\d+[\.\d]+))?'
name: 'Mandrill PHP'
version: '$1'
url: 'https://bitbucket.org/mailchimp/mandrill-api-php/src/master/'
- regex: 'babashka.http-client(?:/(\d+[\.\d]+))?'
name: 'Babashka HTTP Client'
version: '$1'
url: 'https://github.com/babashka/http-client'
- regex: 'http.rb(?:/(\d+[\.\d]+))?'
name: 'http.rb'
version: '$1'
url: 'https://github.com/httprb/http'
- regex: 'node-superagent(?:/(\d+[\.\d]+))?'
name: 'superagent'
version: '$1'
url: 'https://github.com/ladjs/superagent'
- regex: 'CakePHP'
name: 'CakePHP'
version: ''
url: 'https://www.cakephp.org/'
- regex: 'request.js'
name: 'request'
version: ''
url: 'https://github.com/request/request'

@ -171,11 +171,41 @@
name: '$1' name: '$1'
version: '$2' version: '$2'
# Signal uses WhatsApp user agent
- regex: 'WhatsApp(?:/2)?$'
name: 'Signal'
version: ''
# WhatsApp Business
- regex: 'WhatsApp/([\d\.]+).+CloudAPI'
name: 'WhatsApp Business'
version: '$1'
# WhatsApp # WhatsApp
- regex: 'WhatsApp(?:[ /]([\d\.]+))?' - regex: 'WhatsApp(?:Electron|\-app|\-linux-app)?(?:[ /]([\d\.]+))?'
name: 'WhatsApp' name: 'WhatsApp'
version: '$1' version: '$1'
# WhatsApp+2
- regex: 'YoWhatsApp2Plus(?:/(\d+[\.\d]+))?'
name: 'WhatsApp+2'
version: '$1'
# FM WhatsApp
- regex: 'YoFMWhatsApp(?:/(\d+[\.\d]+))?'
name: 'FM WhatsApp'
version: '$1'
# GBWhatsApp
- regex: '(?:Yo)?GBWhatsApp(?:/(\d+[\.\d]+))?'
name: 'GBWhatsApp'
version: '$1'
# Yo WhatsApp
- regex: 'YoYoWhatsApp(?:/(\d+[\.\d]+))?'
name: 'Yo WhatsApp'
version: '$1'
# Line # Line
- regex: 'Line(?:[ /]([\d\.]+))' - regex: 'Line(?:[ /]([\d\.]+))'
name: 'Line' name: 'Line'
@ -469,7 +499,7 @@
version: '$1' version: '$1'
# TikTok (https://apps.apple.com/app/id835599320 | https://play.google.com/store/apps/details?id=com.zhiliaoapp.musically) # TikTok (https://apps.apple.com/app/id835599320 | https://play.google.com/store/apps/details?id=com.zhiliaoapp.musically)
- regex: '(?:TikTok[/ ]|musical_ly_|AppName/musical_ly app_version/)(\d+\.(?:[\.\d]+))?' - regex: '(?:TikTok[/ ]|com.zhiliaoapp.musically|musical_ly_|AppName/musical_ly app_version/)(\d+\.(?:[\.\d]+))?'
name: 'TikTok' name: 'TikTok'
version: '$1' version: '$1'
@ -1086,11 +1116,6 @@
name: 'Focus Keeper' name: 'Focus Keeper'
version: '$1' version: '$1'
# Quick Search TV (https://play.google.com/store/apps/details?id=com.aospstudio.tvsearch)
- regex: 'Quick Search TV/(\d+\.[\.\d]+)'
name: 'Quick Search TV'
version: '$1'
# Microsoft Store (https://www.microsoft.com/en-us/store/apps/windows) # Microsoft Store (https://www.microsoft.com/en-us/store/apps/windows)
- regex: 'WindowsStoreSDK' - regex: 'WindowsStoreSDK'
name: 'Microsoft Store' name: 'Microsoft Store'
@ -1112,7 +1137,7 @@
version: '$1' version: '$1'
# Microsoft Bing Search (https://play.google.com/store/apps/details?id=com.microsoft.bing | https://apps.apple.com/us/app/microsoft-bing-search/id345323231) # Microsoft Bing Search (https://play.google.com/store/apps/details?id=com.microsoft.bing | https://apps.apple.com/us/app/microsoft-bing-search/id345323231)
- regex: 'BingSapphire/(\d+\.[\.\d]+)' - regex: '(?:Bing)?Sapphire/(\d+\.[\.\d]+)'
name: 'Microsoft Bing Search' name: 'Microsoft Bing Search'
version: '$1' version: '$1'
@ -1132,7 +1157,7 @@
version: '$1' version: '$1'
# F-Secure SAFE (https://apps.apple.com/app/f-secure-safe/id572847748) # F-Secure SAFE (https://apps.apple.com/app/f-secure-safe/id572847748)
- regex: 'FSCDCSafe (\d+\.[\.\d]+)' - regex: 'FSCDCSafe[ /](\d+\.[\.\d]+)'
name: 'F-Secure SAFE' name: 'F-Secure SAFE'
version: '$1' version: '$1'
@ -1383,22 +1408,11 @@
name: 'Amazon Music' name: 'Amazon Music'
version: '$1' version: '$1'
# Klarna (https://apps.apple.com/us/app/klarna-shop-now-pay-later/id1115120118) # Klarna (https://apps.apple.com/us/app/klarna-shop-now-pay-later/id1115120118)
- regex: 'Klarna/(\d+\.[.\d]+)' - regex: 'Klarna/(\d+\.[.\d]+)?'
name: 'Klarna' name: 'Klarna'
version: '$1' version: '$1'
# WhatsApp+2
- regex: 'YoWhatsApp2Plus(?:/(\d+[\.\d]+))?'
name: 'WhatsApp+2'
version: '$1'
# GBWhatsApp
- regex: 'GBWhatsApp(?:/(\d+[\.\d]+))?'
name: 'GBWhatsApp'
version: '$1'
# R (https://www.r-project.org/) # R (https://www.r-project.org/)
- regex: '^R/(\d+[\.\d]+)' - regex: '^R/(\d+[\.\d]+)'
name: 'R' name: 'R'
@ -2181,11 +2195,6 @@
name: 'NPR One' name: 'NPR One'
version: '$1' version: '$1'
# unsure what this is, but is used quite a bit
- regex: 'Trackable/([\d\.]+)'
name: 'Trackable'
version: '$1'
- regex: 'Uforia/([\d\.]+)' - regex: 'Uforia/([\d\.]+)'
name: 'Uforia' name: 'Uforia'
version: '$1' version: '$1'
@ -2194,7 +2203,7 @@
name: 'L.A. Times' name: 'L.A. Times'
version: '' version: ''
- regex: '(?:^NYT(?: |%20)?Audio|nytios)/([\d.]+)' - regex: '(?:^NYT(?: |%20)?Audio(?:-iOS)?|nytios)/([\d.]+)'
name: 'The New York Times' name: 'The New York Times'
version: '$1' version: '$1'
@ -2213,3 +2222,63 @@
- regex: 'IPTV/([\d\.]+)' - regex: 'IPTV/([\d\.]+)'
name: 'IPTV' name: 'IPTV'
version: '$1' version: '$1'
# IPTV Pro (https://play.google.com/store/apps/details?id=ru.iptvremote.android.iptv.pro)
- regex: 'IPTV Pro/([\d\.]+)'
name: 'IPTV Pro'
version: '$1'
# MetaMask (https://apps.apple.com/us/app/metamask-blockchain-wallet/id1438144202)
- regex: 'MetaMask/'
name: 'MetaMask'
version: ''
# +Simple (https://play.google.com/store/apps/details?id=com.massimple.nacion.gcba.es)
- regex: '\+Simple Browser'
name: '+Simple'
version: ''
# Bitwarden (https://bitwarden.com/)
- regex: 'Bitwarden_Mobile/([\d\.]+)'
name: 'Bitwarden'
version: '$1'
# MX Player (https://play.google.com/store/apps/details?id=com.mxtech.videoplayer.ad)
- regex: 'MXPlayer/([\d\.]+)'
name: 'MX Player'
version: '$1'
# HistoryHound
- regex: 'HistoryHound/([\d\.]+)'
name: 'HistoryHound'
version: '$1'
# Quicksilver (https://qsapp.com/)
- regex: 'Quicksilver'
name: 'Quicksilver'
version: ''
# CamScanner (https://play.google.com/store/apps/details?id=com.intsig.camscanner)
- regex: 'CamScanner/([\d\.]+)'
name: 'CamScanner'
version: '$1'
# OBS Studio (https://obsproject.com/)
- regex: 'OBS/(\d+[\.\d]+)'
name: 'OBS Studio'
version: '$1'
# XSplit Broadcaster (https://www.xsplit.com/broadcaster)
- regex: 'XSplitBroadcaster/(\d+[\.\d]+)'
name: 'XSplit Broadcaster'
version: '$1'
# Twitch Studio (https://www.twitch.tv/broadcast/studio)
- regex: 'twitch-desktop-electron-platform.*spotlight/(\d+[\.\d]+)'
name: 'Twitch Studio'
version: '$1'
# Stream Master (https://www.lovense.com/stream-master)
- regex: 'Stream Master$'
name: 'Stream Master'
version: ''

@ -88,6 +88,7 @@
'de.twokit.video.tv.cast.browser.lg': 'TV Cast' 'de.twokit.video.tv.cast.browser.lg': 'TV Cast'
'de.twokit.video.tv.cast.browser.samsung': 'TV Cast' 'de.twokit.video.tv.cast.browser.samsung': 'TV Cast'
'com.cleanmaster.security': 'CM Security' 'com.cleanmaster.security': 'CM Security'
'com.antivirus.master.cmsecurity': 'CM Security'
'idm.video.free': 'IDM Video Download Manager' 'idm.video.free': 'IDM Video Download Manager'
'mobi.deallauncher.coupons.shopping': 'Coupons & Deals' 'mobi.deallauncher.coupons.shopping': 'Coupons & Deals'
'com.wukongtv.wkcast.intl': 'Quick Cast' 'com.wukongtv.wkcast.intl': 'Quick Cast'
@ -95,6 +96,11 @@
'com.fsecure.ms.kpn.veilig': 'KPN Veilig' 'com.fsecure.ms.kpn.veilig': 'KPN Veilig'
'com.fsecure.ms.actshield': 'ACT Shield' 'com.fsecure.ms.actshield': 'ACT Shield'
'com.fsecure.ms.talktalksa': 'TalkTalk SuperSafe' 'com.fsecure.ms.talktalksa': 'TalkTalk SuperSafe'
'com.fsecure.ms.dnafi': 'DNA Digiturva'
'com.fsecure.ms.teliasweden': 'Telia Trygg'
'com.fsecure.ms.upc.ch': 'UPC Internet Security'
'com.fsecure.ms.sfr': 'SFR Sécurité'
'com.fsecure.ms.sonera': 'Telia Turvapaketti'
'com.bifrostwallet.app': 'Bifrost Wallet' 'com.bifrostwallet.app': 'Bifrost Wallet'
'com.anydesk.anydeskandroid': 'AnyDesk Remote Desktop' 'com.anydesk.anydeskandroid': 'AnyDesk Remote Desktop'
'com.google.android.youtube': 'YouTube' 'com.google.android.youtube': 'YouTube'
@ -116,6 +122,10 @@
'com.box.video.downloader': 'BOX Video Downloader' 'com.box.video.downloader': 'BOX Video Downloader'
'com.oxoo.kinogo': 'Kinogo.ge' 'com.oxoo.kinogo': 'Kinogo.ge'
'com.tradron.hdvideodownloader': 'Download Hub' 'com.tradron.hdvideodownloader': 'Download Hub'
'net.daum.android.daum': 'Daum'
'com.massimple.nacion.gcba.es': '+Simple'
'com.massimple.nacion.parana.es': '+Simple'
'com.microsoft.math': 'Microsoft Math Solver'
# Vpns # Vpns
'org.torproject.android': 'Orbot' 'org.torproject.android': 'Orbot'
@ -125,3 +135,6 @@
'com.surfshark.vpnclient.android': 'Surfshark' 'com.surfshark.vpnclient.android': 'Surfshark'
'com.omshyapps.vpn': 'Omshy VPN' 'com.omshyapps.vpn': 'Omshy VPN'
'com.kuto.vpn': 'KUTO VPN' 'com.kuto.vpn': 'KUTO VPN'
'com.ezt.vpn': 'EZVPN'
'com.nocardteam.nocardvpn': 'NoCard VPN'
'com.nocardteam.nocardvpn.lite': 'NoCard VPN Lite'

@ -1,5 +1,8 @@
# Browsers # Browsers
'mark.via.gg': 'Via'
'mark.via.gp': 'Via' 'mark.via.gp': 'Via'
'mark.via.gq': 'Via'
'mark.via.pm': 'Via'
'com.pure.mini.browser': 'Pure Mini Browser' 'com.pure.mini.browser': 'Pure Mini Browser'
'pure.lite.browser': 'Pure Lite Browser' 'pure.lite.browser': 'Pure Lite Browser'
'acr.browser.Hexa': 'Hexa Web Browser' 'acr.browser.Hexa': 'Hexa Web Browser'
@ -95,6 +98,7 @@
'com.zeesitech.bitchutebrowser': 'Bitchute Browser' 'com.zeesitech.bitchutebrowser': 'Bitchute Browser'
'nova.all.video.downloader': 'Nova Video Downloader Pro' 'nova.all.video.downloader': 'Nova Video Downloader Pro'
'tukidev.pronhubbrowser.tanpavpn': 'PronHub Browser' 'tukidev.pronhubbrowser.tanpavpn': 'PronHub Browser'
'com.crowbar.beaverlite': 'Frost'
'com.crowbar.beaverbrowser': 'Frost+' 'com.crowbar.beaverbrowser': 'Frost+'
'com.lenovo.browser': 'Lenovo Browser' 'com.lenovo.browser': 'Lenovo Browser'
'com.transsion.phoenix': 'Phoenix Browser' 'com.transsion.phoenix': 'Phoenix Browser'
@ -140,6 +144,8 @@
'com.security.xvpn.z35kb.amazon': 'X-VPN' 'com.security.xvpn.z35kb.amazon': 'X-VPN'
'com.security.xvpn.z35kb.huawei': 'X-VPN' 'com.security.xvpn.z35kb.huawei': 'X-VPN'
'com.yandex.browser.lite': 'Yandex Browser Lite' 'com.yandex.browser.lite': 'Yandex Browser Lite'
'com.yandex.browser.beta': 'Yandex Browser'
'com.yandex.browser.alpha': 'Yandex Browser'
'cz.seznam.sbrowser': 'Seznam Browser' 'cz.seznam.sbrowser': 'Seznam Browser'
'com.morrisxar.nav88': 'Office Browser' 'com.morrisxar.nav88': 'Office Browser'
'com.rabbit.incognito.browser': 'Rabbit Private Browser' 'com.rabbit.incognito.browser': 'Rabbit Private Browser'
@ -149,7 +155,7 @@
'id.browser.vivid3': 'Vivid Browser Mini' 'id.browser.vivid3': 'Vivid Browser Mini'
'com.browser.yo.indian': 'Yo Browser' 'com.browser.yo.indian': 'Yo Browser'
'com.mercandalli.android.browser': 'G Browser' 'com.mercandalli.android.browser': 'G Browser'
'com.bf.browser': 'BF Browser' 'com.bf.browser': 'BXE Browser'
'com.qihoo.browser': '360 Browser' 'com.qihoo.browser': '360 Browser'
'com.qihoo.haosou': '360 Browser' 'com.qihoo.haosou': '360 Browser'
'com.qihoo.padbrowser': '360 Browser' 'com.qihoo.padbrowser': '360 Browser'
@ -202,6 +208,7 @@
'com.internet.tvbrowser': 'LUJO TV Browser' 'com.internet.tvbrowser': 'LUJO TV Browser'
'com.logicui.tvbrowser2': 'LogicUI TV Browser' 'com.logicui.tvbrowser2': 'LogicUI TV Browser'
'com.opera.browser.afin': 'Opera' 'com.opera.browser.afin': 'Opera'
'com.opera.browser.beta': 'Opera'
'com.quark.browser': 'Quark' 'com.quark.browser': 'Quark'
'jp.co.yahoo.android.ybrowser': 'Yahoo! Japan Browser' 'jp.co.yahoo.android.ybrowser': 'Yahoo! Japan Browser'
'com.tv.browser.open': 'Open TV Browser' 'com.tv.browser.open': 'Open TV Browser'
@ -225,3 +232,21 @@
'com.cavebrowser': 'Cave Browser' 'com.cavebrowser': 'Cave Browser'
'com.zordo.browser': 'Zordo Browser' 'com.zordo.browser': 'Zordo Browser'
'freedom.theanarch.org.freedom': 'Freedom Browser' 'freedom.theanarch.org.freedom': 'Freedom Browser'
'com.lechneralexander.privatebrowser': 'Private Internet Browser'
'com.airfind.browser': 'Airfind Secure Browser'
'com.securex.browser': 'SecureX'
'com.sec.android.app.sbrowser.beta': 'Samsung Browser'
'threads.thor': 'Thor'
'com.androidbull.incognito.browser': 'Incognito Browser'
'com.mosoft.godzilla': 'Godzilla Browser'
'com.oceanbrowser.mobile.android': 'Ocean Browser'
'com.qmamu.browser': 'Qmamu'
'com.techlastudio.bfbrowser': 'BF Browser'
'com.befaster.bfbrowser': 'BF Browser'
'app.nextinnovations.brokeep': 'BroKeep Browser'
'org.lilo.app': 'Lilo'
'proxy.browser.unblock.sites.proxybrowser.unblocksites': 'Proxy Browser'
'com.hotsurf.browser': 'HotBrowser'
'vpn.video.downloader': 'VD Browser'
'com.aospstudio.tvsearch': 'Quick Search TV'
'com.go.browser': 'GO Browser'

@ -5,6 +5,22 @@
# @license http://www.gnu.org/licenses/lgpl.html LGPL v3 or later # @license http://www.gnu.org/licenses/lgpl.html LGPL v3 or later
############### ###############
# BMW (https://www.bmw.com/)
BMW:
regex: 'AFTLBT962E2(?:[);/ ]|$)'
device: 'car browser'
models:
- regex: 'AFTLBT962E2(?:[);/ ]|$)'
model: 'Car (2022)'
# Jeep (https://www.jeep.com/)
Jeep:
regex: 'AFTLFT962X3(?:[);/ ]|$)'
device: 'car browser'
models:
- regex: 'AFTLFT962X3(?:[);/ ]|$)'
model: 'Wagoneer'
# Tesla Model S # Tesla Model S
Tesla: Tesla:
regex: '(?:Tesla/(?:(?:develop|feature|terminal-das-fsd-eap)-)?[0-9.]+|QtCarBrowser)' regex: '(?:Tesla/(?:(?:develop|feature|terminal-das-fsd-eap)-)?[0-9.]+|QtCarBrowser)'

@ -46,7 +46,7 @@ JXD:
model: 'S601 WiFi' model: 'S601 WiFi'
Sony: Sony:
regex: '(?:PlayStation ?(2|3|4 Pro|4|5|Portable|Vita)|sony_tv;ps5;|\(PS3\))' regex: '(?:PlayStation ?(4 Pro|[2-5]|Portable|Vita)|sony_tv;ps5;|\(PS3\))'
device: 'console' device: 'console'
models: models:
- regex: 'sony_tv;ps5;' - regex: 'sony_tv;ps5;'
@ -55,7 +55,7 @@ Sony:
model: 'PlayStation 4 Pro' model: 'PlayStation 4 Pro'
- regex: '\(PS3\)' - regex: '\(PS3\)'
model: 'PlayStation 3' model: 'PlayStation 3'
- regex: 'PlayStation ?(2|3|4 Pro|4|5|Portable|Vita)' - regex: 'PlayStation ?(4 Pro|[2-5]|Portable|Vita)'
model: 'PlayStation $1' model: 'PlayStation $1'
# Retroid Pocket (www.goretroid.com) # Retroid Pocket (www.goretroid.com)

File diff suppressed because it is too large Load Diff

@ -21,7 +21,7 @@ Apple:
model: 'iPod Touch 5' model: 'iPod Touch 5'
- regex: '(?:Apple-)?iPod7[C,_]?1' - regex: '(?:Apple-)?iPod7[C,_]?1'
model: 'iPod Touch 6' model: 'iPod Touch 6'
- regex: '(?:Apple-)?iPod9[C,_]?1' - regex: '(?:Apple-)?iPod9[C,_]?1|iPodTouch7'
model: 'iPod Touch 7' model: 'iPod Touch 7'
- regex: '(?:Apple-)?iPod' - regex: '(?:Apple-)?iPod'
model: 'iPod Touch' model: 'iPod Touch'

@ -952,9 +952,11 @@ Silva Schneider:
# Skyworth # Skyworth
Skyworth: Skyworth:
regex: 'Sky_worth' regex: 'Sky_worth|SWTV[;,]'
device: 'tv' device: 'tv'
models: models:
- regex: 'SKWE20E21'
model: 'SKWE20E21'
- regex: 'Sky_worth;([^);/]+)' - regex: 'Sky_worth;([^);/]+)'
model: '$1' model: '$1'
@ -986,14 +988,6 @@ Sony:
- regex: '(K[DM]L?-?[0-9]{2}[A-Z]{1,2}[0-9]{1,5})' - regex: '(K[DM]L?-?[0-9]{2}[A-Z]{1,2}[0-9]{1,5})'
model: '$1' model: '$1'
# SWTV
SWTV:
regex: 'SWTV[;,]'
device: 'tv'
models:
- regex: 'SKWE20E21'
model: 'SKWE20E21'
# TD Systems # TD Systems
'TD Systems': 'TD Systems':
regex: 'TDSystems[;,]' regex: 'TDSystems[;,]'

@ -5,6 +5,15 @@
# @license http://www.gnu.org/licenses/lgpl.html LGPL v3 or later # @license http://www.gnu.org/licenses/lgpl.html LGPL v3 or later
############### ###############
# BrightSignOS (https://www.brightsign.biz/ecosystem/brightsign-os/)
##########
- regex: 'BrightSign/(?:[A-Z0-9]+)/(\d+[\.\d]+)'
name: 'BrightSignOS'
version: '$1'
- regex: 'BrightSign/(\d+[\.\d]+)'
name: 'BrightSignOS'
version: '$1'
########## ##########
# LuneOS (https://www.webos-ports.org/) # LuneOS (https://www.webos-ports.org/)
########## ##########
@ -426,7 +435,7 @@
name: 'Android' name: 'Android'
version: '$1' version: '$1'
- regex: '(?:(?:Orca-)?(?<!like |/|RadioPublic |Anghami )Android|Adr|AOSP)[ /]?(?:[a-z]+ )?(\d+[\.\d]*)' - regex: '(?:(?:Orca-)?(?<!like |/|RadioPublic |Anghami |Callpod Keeper for )Android|Adr|AOSP)[ /]?(?:[a-z]+ )?(\d+[\.\d]*)'
name: 'Android' name: 'Android'
version: '$1' version: '$1'
@ -450,7 +459,7 @@
name: 'Android' name: 'Android'
version: '$1' version: '$1'
- regex: 'BeyondPod|AntennaPod|Podkicker|DoggCatcher|Player FM|okhttp|Podcatcher Deluxe|Sonos/.+\(ACR_|Linux x86_64; Quest' - regex: 'BeyondPod|AntennaPod|Podkicker|DoggCatcher|Player FM|okhttp|Podcatcher Deluxe|Sonos/.+\(ACR_|Linux x86_64; Quest|.*WhatsApp/.*A$'
name: 'Android' name: 'Android'
version: '' version: ''
@ -714,7 +723,7 @@
name: 'Windows' name: 'Windows'
version: '$1' version: '$1'
- regex: 'mingw32|winhttp' - regex: 'mingw32|winhttp|WhatsApp/.*W$'
name: 'Windows' name: 'Windows'
version: '' version: ''
@ -833,7 +842,7 @@
name: 'iPadOS' name: 'iPadOS'
version: '$1.$2' version: '$1.$2'
- regex: 'iPad(?:; (?:iOS|iPadOS|iPhone OS)|.+CPU OS) ((1[3-7])+(?:[_\.]\d+)*)' - regex: 'iPad(?:; (?:iOS|iPadOS|iPhone OS)|.+CPU (?:iPad |iPhone )?OS) ((1[3-7])+(?:[_\.]\d+)*)'
name: 'iPadOS' name: 'iPadOS'
version: '$1' version: '$1'
@ -868,9 +877,11 @@
name: 'iOS' name: 'iOS'
version: '$1' version: '$1'
- regex: '^(?!com.apple.Safari.SearchHelper|Safari).*CFNetwork/.+ Darwin/(\d+[\.\d]+)(?!.*(?:x86_64|i386|PowerMac|Power%20Macintosh))' - regex: '^(?!com.apple.Safari.SearchHelper|Safari).*(?:CFNetwork|Mana)/.+ Darwin/(\d+[\.\d]+)(?!.*(?:x86_64|i386|PowerMac|Power%20Macintosh))'
name: 'iOS' name: 'iOS'
versions: versions:
- regex: 'Darwin/23.4.0'
version: '17.4'
- regex: 'Darwin/23.3.0' - regex: 'Darwin/23.3.0'
version: '17.3' version: '17.3'
- regex: 'Darwin/23.2.0' - regex: 'Darwin/23.2.0'
@ -1074,7 +1085,7 @@
name: 'iOS' name: 'iOS'
version: '$1' version: '$1'
- regex: '(?:Podcasts/(?:[\d\.]+)|Instacast(?:HD)?/(?:\d\.[\d\.abc]+)|Pocket Casts, iOS|\(iOS\)|iOS; Opera|Overcast|Castro|Podcat|iCatcher|RSSRadio/|MobileSafari/)(?!.*x86_64)' - regex: '(?:Podcasts/(?:[\d\.]+)|Instacast(?:HD)?/(?:\d\.[\d\.abc]+)|Pocket Casts, iOS|\(iOS\)|iOS; Opera|Overcast|Castro|Podcat|iCatcher|RSSRadio/|MobileSafari/|WhatsApp/.*i$)(?!.*x86_64)'
name: 'iOS' name: 'iOS'
version: '' version: ''
@ -1097,9 +1108,11 @@
########## ##########
# Mac # Mac
########## ##########
- regex: '(?:CFNetwork|StudioDisplay)/.+Darwin(?:/|; )(?:[\d\.]+).+(?:x86_64|i386|Power%20Macintosh)|(?:x86_64-apple-)?darwin(?:[\d\.]+)|PowerMac|com.apple.Safari.SearchHelper|^Safari' - regex: '(?:CFNetwork|Mana|StudioDisplay)/.+Darwin(?:/|; )(?:[\d\.]+).+(?:x86_64|i386|Power%20Macintosh)|(?:x86_64-apple-)?darwin(?:[\d\.]+)|PowerMac|com.apple.Safari.SearchHelper|^Safari'
name: 'Mac' name: 'Mac'
versions: versions:
- regex: '(?:x86_64-apple-)?Darwin(?:/|; )?23.4.0'
version: '14.4'
- regex: '(?:x86_64-apple-)?Darwin(?:/|; )?23.3.0' - regex: '(?:x86_64-apple-)?Darwin(?:/|; )?23.3.0'
version: '14.3' version: '14.3'
- regex: '(?:x86_64-apple-)?Darwin(?:/|; )?23.2.0' - regex: '(?:x86_64-apple-)?Darwin(?:/|; )?23.2.0'
@ -1293,7 +1306,7 @@
name: 'Mac' name: 'Mac'
version: '$1' version: '$1'
- regex: 'Darwin|Macintosh|Mac_PowerPC|PPC|Mac PowerPC|iMac|MacBook|macOS|AppleExchangeWebServices|com.apple.trustd|Sonos/.+\(MDCR_' - regex: 'Darwin|Macintosh|Mac[ _]PowerPC|PPC|iMac|MacBook|macOS|AppleExchangeWebServices|com.apple.trustd|Sonos/.+\(MDCR_|WhatsApp/.*N$'
name: 'Mac' name: 'Mac'
version: '' version: ''

@ -229,6 +229,24 @@
- "ZR" - "ZR"
- "D6" - "D6"
- "F6" - "F6"
- "RC"
- "WD"
- "P3"
- "FT"
- "A9"
- "X2"
- "N3"
- "GD"
- "O9"
- "Q3"
- "F7"
- "K2"
- "P5"
- "H5"
- "V3"
- "K3"
- "Q4"
- "G2"
- "" - ""
- "Firefox": - "Firefox":
- "AX" - "AX"
@ -261,7 +279,6 @@
- "PI" - "PI"
- "PX" - "PX"
- "QA" - "QA"
- "QM"
- "S5" - "S5"
- "SX" - "SX"
- "TF" - "TF"

@ -33,6 +33,7 @@
- "AB": "ABrowse" - "AB": "ABrowse"
- "BW": "AdBlock Browser" - "BW": "AdBlock Browser"
- "A7": "Adult Browser" - "A7": "Adult Browser"
- "A9": "Airfind Secure Browser"
- "AF": "ANT Fresco" - "AF": "ANT Fresco"
- "AG": "ANTGalio" - "AG": "ANTGalio"
- "AL": "Aloha Browser" - "AL": "Aloha Browser"
@ -53,6 +54,7 @@
- "AV": "Amiga Voyager" - "AV": "Amiga Voyager"
- "AW": "Amiga Aweb" - "AW": "Amiga Aweb"
- "PN": "APN Browser" - "PN": "APN Browser"
- "RA": "Arc"
- "AI": "Arvin" - "AI": "Arvin"
- "AK": "Ask.com" - "AK": "Ask.com"
- "AU": "Asus Browser" - "AU": "Asus Browser"
@ -65,6 +67,7 @@
- "A1": "AwoX" - "A1": "AwoX"
- "BA": "Beaker Browser" - "BA": "Beaker Browser"
- "BM": "Beamrise" - "BM": "Beamrise"
- "F7": "BF Browser"
- "BB": "BlackBerry Browser" - "BB": "BlackBerry Browser"
- "H1": "BrowseHere" - "H1": "BrowseHere"
- "B8": "Browser Hup Pro" - "B8": "Browser Hup Pro"
@ -87,6 +90,7 @@
- "BN": "Borealis Navigator" - "BN": "Borealis Navigator"
- "BR": "Brave" - "BR": "Brave"
- "BK": "BriskBard" - "BK": "BriskBard"
- "K2": "BroKeep Browser"
- "B3": "Browspeed Browser" - "B3": "Browspeed Browser"
- "BX": "BrowseX" - "BX": "BrowseX"
- "BZ": "Browzar" - "BZ": "Browzar"
@ -94,7 +98,7 @@
- "4B": "BrowsBit" - "4B": "BrowsBit"
- "BY": "Biyubi" - "BY": "Biyubi"
- "BF": "Byffox" - "BF": "Byffox"
- "B4": "BF Browser" - "B4": "BXE Browser"
- "CA": "Camino" - "CA": "Camino"
- "0C": "Cave Browser" - "0C": "Cave Browser"
- "CL": "CCleaner" - "CL": "CCleaner"
@ -129,6 +133,7 @@
- "C5": "Chromium GOST" - "C5": "Chromium GOST"
- "CY": "Cyberfox" - "CY": "Cyberfox"
- "CS": "Cheshire" - "CS": "Cheshire"
- "RC": "Crow Browser"
- "CT": "Crusta" - "CT": "Crusta"
- "CG": "Craving Explorer" - "CG": "Craving Explorer"
- "CZ": "Crazy Browser" - "CZ": "Crazy Browser"
@ -145,6 +150,7 @@
- "DT": "Delta Browser" - "DT": "Delta Browser"
- "D0": "Desi Browser" - "D0": "Desi Browser"
- "DS": "DeskBrowse" - "DS": "DeskBrowse"
- "D2": "DoCoMo"
- "DF": "Dolphin" - "DF": "Dolphin"
- "DZ": "Dolphin Zero" - "DZ": "Dolphin Zero"
- "DO": "Dorado" - "DO": "Dorado"
@ -167,6 +173,7 @@
- "EP": "GNOME Web" - "EP": "GNOME Web"
- "G1": "G Browser" - "G1": "G Browser"
- "ES": "Espial TV Browser" - "ES": "Espial TV Browser"
- "FG": "fGet"
- "FA": "Falkon" - "FA": "Falkon"
- "FX": "Faux Browser" - "FX": "Faux Browser"
- "F4": "Fiery Browser" - "F4": "Fiery Browser"
@ -192,6 +199,7 @@
- "F5": "Flyperlink" - "F5": "Flyperlink"
- "FU": "FreeU" - "FU": "FreeU"
- "F6": "Freedom Browser" - "F6": "Freedom Browser"
- "FT": "Frost"
- "F3": "Frost+" - "F3": "Frost+"
- "FI": "Fulldive" - "FI": "Fulldive"
- "GA": "Galeon" - "GA": "Galeon"
@ -199,10 +207,12 @@
- "GH": "Ghostery Privacy Browser" - "GH": "Ghostery Privacy Browser"
- "GI": "GinxDroid Browser" - "GI": "GinxDroid Browser"
- "GB": "Glass Browser" - "GB": "Glass Browser"
- "GD": "Godzilla Browser"
- "GE": "Google Earth" - "GE": "Google Earth"
- "GP": "Google Earth Pro" - "GP": "Google Earth Pro"
- "GO": "GOG Galaxy" - "GO": "GOG Galaxy"
- "GR": "GoBrowser" - "GR": "GoBrowser"
- "G2": "GO Browser"
- "HB": "Harman Browser" - "HB": "Harman Browser"
- "HS": "HasBrowser" - "HS": "HasBrowser"
- "HA": "Hawk Turbo Browser" - "HA": "Hawk Turbo Browser"
@ -212,6 +222,7 @@
- "HI": "Hi Browser" - "HI": "Hi Browser"
- "HO": "hola! Browser" - "HO": "hola! Browser"
- "H4": "Holla Web Browser" - "H4": "Holla Web Browser"
- "H5": "HotBrowser"
- "HJ": "HotJava" - "HJ": "HotJava"
- "HT": "HTC Browser" - "HT": "HTC Browser"
- "HU": "Huawei Browser Mobile" - "HU": "Huawei Browser Mobile"
@ -230,6 +241,7 @@
- "IV": "Isivioo" - "IV": "Isivioo"
- "I8": "IVVI Browser" - "I8": "IVVI Browser"
- "IW": "Iceweasel" - "IW": "Iceweasel"
- "N3": "Incognito Browser"
- "IN": "Inspect Browser" - "IN": "Inspect Browser"
- "I9": "Insta Browser" - "I9": "Insta Browser"
- "IE": "Internet Explorer" - "IE": "Internet Explorer"
@ -321,6 +333,8 @@
- "WR": "NextWord Browser" - "WR": "NextWord Browser"
- "NT": "NTENT Browser" - "NT": "NTENT Browser"
- "NU": "Nuanti Meta" - "NU": "Nuanti Meta"
- "NI": "Nuviu"
- "O9": "Ocean Browser"
- "OC": "Oculus Browser" - "OC": "Oculus Browser"
- "O6": "Odd Browser" - "O6": "Odd Browser"
- "O1": "Opera Mini iOS" - "O1": "Opera Mini iOS"
@ -333,6 +347,7 @@
- "O5": "Office Browser" - "O5": "Office Browser"
- "HH": "OhHai Browser" - "HH": "OhHai Browser"
- "OE": "ONE Browser" - "OE": "ONE Browser"
- "N4": "Onion Browser"
- "Y1": "Opera Crypto" - "Y1": "Opera Crypto"
- "OX": "Opera GX" - "OX": "Opera GX"
- "OG": "Opera Neon" - "OG": "Opera Neon"
@ -380,12 +395,16 @@
- "LY": "PolyBrowser" - "LY": "PolyBrowser"
- "PI": "PrivacyWall" - "PI": "PrivacyWall"
- "P4": "Privacy Explorer Fast Safe" - "P4": "Privacy Explorer Fast Safe"
- "P3": "Private Internet Browser"
- "P5": "Proxy Browser"
- "P2": "Pi Browser" - "P2": "Pi Browser"
- "P0": "PronHub Browser" - "P0": "PronHub Browser"
- "PC": "PSI Secure Browser" - "PC": "PSI Secure Browser"
- "RW": "Reqwireless WebViewer" - "RW": "Reqwireless WebViewer"
- "PS": "Microsoft Edge" - "PS": "Microsoft Edge"
- "QA": "Qazweb" - "QA": "Qazweb"
- "Q3": "Qmamu"
- "Q4": "Quick Search TV"
- "Q2": "QQ Browser Lite" - "Q2": "QQ Browser Lite"
- "Q1": "QQ Browser Mini" - "Q1": "QQ Browser Mini"
- "QQ": "QQ Browser" - "QQ": "QQ Browser"
@ -422,6 +441,7 @@
- "S1": "SimpleBrowser" - "S1": "SimpleBrowser"
- "3S": "SilverMob US" - "3S": "SilverMob US"
- "SY": "Sizzy" - "SY": "Sizzy"
- "K3": "Skye"
- "SK": "Skyfire" - "SK": "Skyfire"
- "SS": "Seraphic Sraf" - "SS": "Seraphic Sraf"
- "KK": "SiteKiosk" - "KK": "SiteKiosk"
@ -430,6 +450,7 @@
- "S7": "SP Browser" - "S7": "SP Browser"
- "9S": "Sony Small Browser" - "9S": "Sony Small Browser"
- "8S": "Secure Private Browser" - "8S": "Secure Private Browser"
- "X2": "SecureX"
- "T1": "Stampy Browser" - "T1": "Stampy Browser"
- "7S": "7Star" - "7S": "7Star"
- "SQ": "Smart Browser" - "SQ": "Smart Browser"
@ -460,6 +481,7 @@
- "TR": "T-Browser" - "TR": "T-Browser"
- "TO": "t-online.de Browser" - "TO": "t-online.de Browser"
- "TA": "Tao Browser" - "TA": "Tao Browser"
- "TH": "Thor"
- "1T": "Tor Browser" - "1T": "Tor Browser"
- "TF": "TenFourFox" - "TF": "TenFourFox"
- "TB": "Tenta Browser" - "TB": "Tenta Browser"
@ -483,7 +505,9 @@
- "UE": "Ume Browser" - "UE": "Ume Browser"
- "V0": "vBrowser" - "V0": "vBrowser"
- "VA": "Vast Browser" - "VA": "Vast Browser"
- "V3": "VD Browser"
- "VE": "Venus Browser" - "VE": "Venus Browser"
- "WD": "Vewd Browser"
- "N0": "Nova Video Downloader Pro" - "N0": "Nova Video Downloader Pro"
- "VS": "Viasat Browser" - "VS": "Viasat Browser"
- "VI": "Vivaldi" - "VI": "Vivaldi"

@ -1,2 +1,6 @@
- "Chrome": - "Chrome":
- "Google Chrome" - "Google Chrome"
- "Vewd Browser":
- "Vewd Core"
- "DuckDuckGo Privacy Browser":
- "DuckDuckGo"

@ -136,3 +136,21 @@
- "ZR" - "ZR"
- "D6" - "D6"
- "F6" - "F6"
- "P3"
- "FT"
- "A9"
- "X2"
- "NI"
- "FG"
- "TH"
- "N3"
- "GD"
- "O9"
- "Q3"
- "F7"
- "K2"
- "N4"
- "P5"
- "H5"
- "V3"
- "G2"

@ -111,6 +111,7 @@
- "ORA" - "ORA"
- "VID" - "VID"
- "TIV" - "TIV"
- "BSN"
- "" - ""
- "Mac": - "Mac":
- "MAC" - "MAC"

@ -12,6 +12,7 @@
- "QNX": "BlackBerry Tablet OS" - "QNX": "BlackBerry Tablet OS"
- "BOS": "Bliss OS" - "BOS": "Bliss OS"
- "BMP": "Brew" - "BMP": "Brew"
- "BSN": "BrightSignOS"
- "CAI": "Caixa Mágica" - "CAI": "Caixa Mágica"
- "CES": "CentOS" - "CES": "CentOS"
- "CST": "CentOS Stream" - "CST": "CentOS Stream"

@ -0,0 +1,204 @@
defmodule Plausible.DataMigration.SiteImportsTest do
use Plausible.DataCase, async: true
import ExUnit.CaptureIO
alias Plausible.DataMigration.SiteImports
alias Plausible.Imported
alias Plausible.Repo
alias Plausible.Site
describe "run/1" do
test "runs for empty dataset" do
dry_run_output =
capture_io(fn ->
assert :ok = SiteImports.run()
end)
assert dry_run_output =~ "Processing 0 sites"
assert dry_run_output =~ "DRY RUN: true"
real_run_output =
capture_io(fn ->
assert :ok = SiteImports.run(dry_run?: false)
end)
assert real_run_output =~ "Processing 0 sites"
assert real_run_output =~ "DRY RUN: false"
end
test "adds site import entry when it's missing and adjusts end date" do
site =
insert(:site)
|> Site.start_import(~D[2021-01-02], ~D[2021-01-08], "Google Analytics", "ok")
|> Repo.update!()
populate_stats(site, 0, [
build(:imported_visitors, date: ~D[2021-01-07])
])
assert capture_io(fn ->
assert :ok = SiteImports.run(dry_run?: false)
end) =~ "Processing 1 sites"
site = Repo.reload!(site)
assert [%{id: id, legacy: true} = site_import] = Imported.list_all_imports(site)
assert id > 0
assert site_import.start_date == site.imported_data.start_date
assert site_import.end_date == ~D[2021-01-07]
assert site_import.source == :universal_analytics
end
test "runs in dry mode without making any persistent changes" do
site =
insert(:site)
|> Site.start_import(~D[2021-01-02], ~D[2021-01-08], "Google Analytics", "ok")
|> Repo.update!()
populate_stats(site, 0, [
build(:imported_visitors, date: ~D[2021-01-07])
])
assert capture_io(fn ->
assert :ok = SiteImports.run()
end) =~ "Processing 1 sites"
assert [%{id: id, legacy: true}] = Imported.list_all_imports(site)
assert id == 0
end
test "does not set end date to latter than the current one" do
site =
insert(:site)
|> Site.start_import(~D[2021-01-02], ~D[2021-01-08], "Google Analytics", "ok")
|> Repo.update!()
populate_stats(site, 0, [
build(:imported_visitors, date: ~D[2021-01-10])
])
assert capture_io(fn ->
assert :ok = SiteImports.run(dry_run?: false)
end) =~ "Processing 1 sites"
site = Repo.reload!(site)
assert [%{id: id, legacy: true} = site_import] = Imported.list_all_imports(site)
assert id > 0
assert site_import.start_date == site.imported_data.start_date
assert site_import.end_date == ~D[2021-01-08]
assert site_import.source == :universal_analytics
end
test "removes site import when there are no stats" do
site =
insert(:site)
|> Site.start_import(~D[2021-01-02], ~D[2020-02-02], "Google Analytics", "ok")
|> Repo.update!()
_another_site_import = insert(:site_import, site: site)
assert capture_io(fn ->
assert :ok = SiteImports.run(dry_run?: false)
end) =~ "Processing 1 site"
site = Repo.reload!(site)
assert [] = Imported.list_all_imports(site)
end
test "leaves site and imports unchanged if everything fits" do
site =
insert(:site)
|> Site.start_import(~D[2021-01-02], ~D[2021-01-08], "Google Analytics", "ok")
|> Repo.update!()
existing_import =
insert(:site_import,
site: site,
start_date: ~D[2021-01-02],
end_date: ~D[2021-01-08],
status: :completed,
legacy: true
)
populate_stats(site, existing_import.id, [
build(:imported_visitors, date: ~D[2021-01-08])
])
assert capture_io(fn ->
assert :ok = SiteImports.run(dry_run?: false)
end) =~ "Processing 1 sites"
site = Repo.reload!(site)
assert [%{id: id, legacy: true} = site_import] = Imported.list_all_imports(site)
assert id == existing_import.id
assert site_import.start_date == site.imported_data.start_date
assert site_import.end_date == ~D[2021-01-08]
assert site_import.source == :universal_analytics
end
end
describe "imported_stats_end_date/1" do
test "returns nil when there are no stats" do
site_import = insert(:site_import)
assert SiteImports.imported_stats_end_date(site_import.site_id, [site_import.id]) == nil
end
test "returns date when there are stats recorded in one imported table" do
site_import = insert(:site_import)
populate_stats(site_import.site, site_import.id, [
build(:imported_visitors, date: ~D[2021-01-01])
])
assert SiteImports.imported_stats_end_date(site_import.site_id, [site_import.id]) ==
~D[2021-01-01]
end
test "returns max date across all imported tables" do
site_import = insert(:site_import)
populate_stats(site_import.site, site_import.id, [
build(:imported_visitors, date: ~D[2021-01-01]),
build(:imported_visitors, date: ~D[2021-01-07]),
build(:imported_sources, date: ~D[2021-01-01]),
build(:imported_sources, date: ~D[2021-01-08]),
build(:imported_entry_pages, date: ~D[2021-01-02]),
build(:imported_entry_pages, date: ~D[2021-02-11]),
build(:imported_exit_pages, date: ~D[2021-01-01]),
build(:imported_exit_pages, date: ~D[2021-01-08]),
build(:imported_locations, date: ~D[2021-01-01]),
build(:imported_locations, date: ~D[2021-01-08]),
build(:imported_devices, date: ~D[2021-01-01]),
build(:imported_devices, date: ~D[2021-01-08]),
build(:imported_browsers, date: ~D[2021-01-01]),
build(:imported_browsers, date: ~D[2021-01-08]),
build(:imported_operating_systems, date: ~D[2021-01-01]),
build(:imported_operating_systems, date: ~D[2021-01-08])
])
assert SiteImports.imported_stats_end_date(site_import.site_id, [site_import.id]) ==
~D[2021-02-11]
end
test "considers all imported tables" do
date = ~D[2021-01-11]
for {table, idx} <- Enum.with_index(Imported.tables()) do
site_import = insert(:site_import)
end_date = Date.add(date, idx)
populate_stats(site_import.site, site_import.id, [
build(String.to_atom(table), date: date),
build(String.to_atom(table), date: end_date)
])
assert SiteImports.imported_stats_end_date(site_import.site_id, [site_import.id]) ==
end_date
end
end
end
end

@ -92,6 +92,24 @@ defmodule Plausible.GoalsTest do
assert [currency: {"is invalid", _}] = changeset.errors assert [currency: {"is invalid", _}] = changeset.errors
end end
@tag :full_build_only
test "list_revenue_goals/1 lists event_names and currencies for each revenue goal" do
site = insert(:site)
Goals.create(site, %{"event_name" => "One", "currency" => "EUR"})
Goals.create(site, %{"event_name" => "Two", "currency" => "EUR"})
Goals.create(site, %{"event_name" => "Three", "currency" => "USD"})
Goals.create(site, %{"event_name" => "Four"})
Goals.create(site, %{"page_path" => "/some-page"})
revenue_goals = Goals.list_revenue_goals(site)
assert length(revenue_goals) == 3
assert %{event_name: "One", currency: :EUR} in revenue_goals
assert %{event_name: "Two", currency: :EUR} in revenue_goals
assert %{event_name: "Three", currency: :USD} in revenue_goals
end
test "create/2 clears currency for pageview goals" do test "create/2 clears currency for pageview goals" do
site = insert(:site) site = insert(:site)
{:ok, goal} = Goals.create(site, %{"page_path" => "/purchase", "currency" => "EUR"}) {:ok, goal} = Goals.create(site, %{"page_path" => "/purchase", "currency" => "EUR"})

@ -55,4 +55,17 @@ defmodule Plausible.Google.GA4.APITest do
GA4.API.get_analytics_start_date("some_access_token", "properties/153293282") GA4.API.get_analytics_start_date("some_access_token", "properties/153293282")
end end
end end
describe "get_analytics_end_date/2" do
test "returns stats end date for a given property" do
result = Jason.decode!(File.read!("fixture/ga4_end_date.json"))
expect(Plausible.HTTPClient.Mock, :post, fn _url, _headers, _body ->
{:ok, %Finch.Response{status: 200, body: result}}
end)
assert {:ok, ~D[2024-03-02]} =
GA4.API.get_analytics_end_date("some_access_token", "properties/153293282")
end
end
end end

@ -43,7 +43,7 @@ defmodule Plausible.Imported.GoogleAnalytics4Test do
token_expires_at: DateTime.to_iso8601(past) token_expires_at: DateTime.to_iso8601(past)
) )
site_import = Plausible.Imported.get_import(job.args.import_id) site_import = Plausible.Imported.get_import(site, job.args.import_id)
assert site_import.label == "properties/123456" assert site_import.label == "properties/123456"
@ -82,15 +82,15 @@ defmodule Plausible.Imported.GoogleAnalytics4Test do
Enum.each(Plausible.Imported.tables(), fn table -> Enum.each(Plausible.Imported.tables(), fn table ->
count = count =
case table do case table do
"imported_sources" -> 3 "imported_sources" -> 1090
"imported_visitors" -> 3 "imported_visitors" -> 31
"imported_pages" -> 8 "imported_pages" -> 3340
"imported_entry_pages" -> 4 "imported_entry_pages" -> 2934
"imported_exit_pages" -> 0 "imported_exit_pages" -> 0
"imported_locations" -> 4 "imported_locations" -> 2291
"imported_devices" -> 4 "imported_devices" -> 93
"imported_browsers" -> 5 "imported_browsers" -> 233
"imported_operating_systems" -> 4 "imported_operating_systems" -> 1068
end end
query = from(imported in table, where: imported.site_id == ^site.id) query = from(imported in table, where: imported.site_id == ^site.id)

@ -59,4 +59,208 @@ defmodule Plausible.ImportedTest do
] = Imported.list_all_imports(site) ] = Imported.list_all_imports(site)
end end
end end
describe "get_imports_date_range/1" do
test "returns empty when there are no imports" do
site = insert(:site)
assert %{start_date: nil, end_date: nil} = Imported.get_imports_date_range(site)
end
test "returns empty when only incomplete or failed imports are present" do
site =
insert(:site)
|> Plausible.Site.start_import(
~D[2020-04-01],
~D[2022-06-22],
"Google Analytics",
"error"
)
|> Repo.update!()
_import1 = insert(:site_import, site: site, status: :pending)
_import2 = insert(:site_import, site: site, status: :importing)
_import3 = insert(:site_import, site: site, status: :failed)
_rogue_import = insert(:site_import, site: build(:site), status: :completed)
assert %{start_date: nil, end_date: nil} = Imported.get_imports_date_range(site)
end
test "returns start and end dates considering all imports" do
site =
insert(:site)
|> Plausible.Site.start_import(
~D[2020-04-01],
~D[2022-06-22],
"Google Analytics",
"ok"
)
|> Repo.update!()
_import1 =
insert(:site_import,
site: site,
start_date: ~D[2020-04-02],
end_date: ~D[2022-06-22],
status: :completed,
legacy: true
)
_import2 =
insert(:site_import,
site: site,
start_date: ~D[2022-06-22],
end_date: ~D[2024-01-08],
status: :completed
)
assert %{start_date: ~D[2020-04-01], end_date: ~D[2024-01-08]} =
Imported.get_imports_date_range(site)
end
end
describe "check_dates/3" do
test "crops dates from both ends when overlapping with existing import and native stats" do
site = insert(:site)
populate_stats(site, [
build(:pageview, timestamp: ~N[2023-10-25 15:58:00])
])
start_date = ~D[2016-04-03]
end_date = ~D[2021-05-12]
_existing_import =
insert(:site_import,
site: site,
start_date: start_date,
end_date: end_date,
status: :completed
)
assert {:ok, ~D[2021-05-12], ~D[2023-10-25]} =
Imported.check_dates(site, ~D[2021-04-11], ~D[2024-01-12])
end
test "picks longest continuous range when containing existing import" do
site = insert(:site)
populate_stats(site, [
build(:pageview, timestamp: ~N[2023-10-25 15:58:00])
])
start_date = ~D[2019-04-03]
end_date = ~D[2021-05-12]
_existing_import =
insert(:site_import,
site: site,
start_date: start_date,
end_date: end_date,
status: :completed
)
assert {:ok, ~D[2021-05-12], ~D[2023-10-25]} =
Imported.check_dates(site, ~D[2019-03-21], ~D[2024-01-12])
end
test "does not alter the dates when there are no imports and no native stats" do
site = insert(:site)
assert {:ok, ~D[2021-05-12], ~D[2024-01-12]} =
Imported.check_dates(site, ~D[2021-05-12], ~D[2024-01-12])
end
test "ignores input date range difference smaller than 2 days" do
site = insert(:site)
assert {:error, :no_time_window} =
Imported.check_dates(site, ~D[2024-01-12], ~D[2024-01-12])
assert {:error, :no_time_window} =
Imported.check_dates(site, ~D[2024-01-12], ~D[2024-01-13])
assert {:ok, ~D[2024-01-12], ~D[2024-01-14]} =
Imported.check_dates(site, ~D[2024-01-12], ~D[2024-01-14])
end
test "ignores imports with date range difference smaller than 2 days" do
site = insert(:site)
start_date = ~D[2024-01-12]
end_date = ~D[2024-01-13]
_existing_import =
insert(:site_import,
site: site,
start_date: start_date,
end_date: end_date,
status: :completed
)
assert {:ok, ~D[2021-04-22], ~D[2024-03-14]} =
Imported.check_dates(site, ~D[2021-04-22], ~D[2024-03-14])
end
test "returns no time window when input range starts after native stats start date" do
site = insert(:site)
populate_stats(site, [
build(:pageview, timestamp: ~N[2023-10-25 15:58:00])
])
assert {:error, :no_time_window} =
Imported.check_dates(site, ~D[2023-10-28], ~D[2024-01-13])
end
test "returns no time window when input range starts less than 2 days before native stats start date" do
site = insert(:site)
populate_stats(site, [
build(:pageview, timestamp: ~N[2023-10-25 15:58:00])
])
assert {:error, :no_time_window} =
Imported.check_dates(site, ~D[2023-10-24], ~D[2024-01-13])
end
test "crops time range at native stats start date when effective range is 2 days or longer" do
site = insert(:site)
populate_stats(site, [
build(:pageview, timestamp: ~N[2023-10-25 15:58:00])
])
assert {:ok, ~D[2023-10-23], ~D[2023-10-25]} =
Imported.check_dates(site, ~D[2023-10-23], ~D[2024-01-13])
end
test "returns no data error when start date missing" do
site = insert(:site)
assert {:error, :no_data} = Imported.check_dates(site, nil, nil)
end
test "returns no time window error when date range overlaps with existing import and stats completely" do
site = insert(:site)
populate_stats(site, [
build(:pageview, timestamp: ~N[2023-10-25 15:58:00])
])
start_date = ~D[2016-04-03]
end_date = ~D[2023-10-25]
_existing_import =
insert(:site_import,
site: site,
start_date: start_date,
end_date: end_date,
status: :completed
)
assert {:error, :no_time_window} =
Imported.check_dates(site, ~D[2021-04-11], ~D[2024-01-12])
end
end
end end

@ -89,6 +89,131 @@ defmodule Plausible.Session.CacheStoreTest do
assert session.events == 2 assert session.events == 2
end end
describe "hostname-related attributes" do
test "initial for non-pageview" do
site_id = new_site_id()
event =
build(:event,
name: "custom_event",
site_id: site_id,
pathname: "/path/1",
hostname: "example.com"
)
flush([event])
session = get_session(site_id)
assert session.hostname == ""
assert session.exit_page_hostname == ""
end
test "initial for pageview" do
site_id = new_site_id()
event =
build(:event,
name: "pageview",
site_id: site_id,
pathname: "/path/1",
hostname: "example.com"
)
flush([event])
session = get_session(site_id)
assert session.hostname == "example.com"
assert session.exit_page_hostname == "example.com"
end
test "subsequent pageview after custom_event" do
site_id = new_site_id()
events = [
build(:event,
name: "custom_event",
site_id: site_id,
pathname: "/path/1",
hostname: "whatever.example.com",
timestamp: Timex.shift(Timex.now(), seconds: -5),
user_id: 1
),
build(:event,
name: "pageview",
site_id: site_id,
pathname: "/path/2",
hostname: "example.com",
user_id: 1
)
]
flush(events)
session = get_session(site_id)
assert session.hostname == "example.com"
assert session.exit_page_hostname == "example.com"
end
test "hostname change" do
site_id = new_site_id()
events = [
build(:event,
name: "pageview",
site_id: site_id,
pathname: "/landing",
hostname: "example.com",
timestamp: Timex.shift(Timex.now(), seconds: -5),
user_id: 1
),
build(:event,
name: "pageview",
site_id: site_id,
pathname: "/post/1",
hostname: "blog.example.com",
user_id: 1
)
]
flush(events)
session = get_session(site_id)
assert session.hostname == "example.com"
assert session.exit_page_hostname == "blog.example.com"
end
test "hostname change with custom event in the middle" do
site_id = new_site_id()
events = [
build(:event,
name: "pageview",
site_id: site_id,
pathname: "/landing",
hostname: "example.com",
timestamp: Timex.shift(Timex.now(), seconds: -5),
user_id: 1
),
build(:event,
name: "custom_event",
site_id: site_id,
pathname: "/path/1",
hostname: "analytics.example.com",
timestamp: Timex.shift(Timex.now(), seconds: -3),
user_id: 1
),
build(:event,
name: "pageview",
site_id: site_id,
pathname: "/post/1",
hostname: "blog.example.com",
user_id: 1
)
]
flush(events)
session = get_session(site_id)
assert session.hostname == "example.com"
assert session.exit_page_hostname == "blog.example.com"
end
end
test "initial pageview-specific attributes" do test "initial pageview-specific attributes" do
site_id = new_site_id() site_id = new_site_id()
@ -231,7 +356,12 @@ defmodule Plausible.Session.CacheStoreTest do
end end
defp get_session(site_id) do defp get_session(site_id) do
session_q = from s in Plausible.ClickhouseSessionV2, where: s.site_id == ^site_id session_q =
from s in Plausible.ClickhouseSessionV2,
where: s.site_id == ^site_id,
order_by: [desc: :timestamp],
limit: 1
Plausible.ClickhouseRepo.one!(session_q) Plausible.ClickhouseRepo.one!(session_q)
end end
end end

@ -67,6 +67,9 @@ defmodule Plausible.Stats.DashboardFilterParserTest do
%{"props" => %{"cta" => "Top"}} %{"props" => %{"cta" => "Top"}}
|> assert_parsed(%{"event:props:cta" => {:is, "Top"}}) |> assert_parsed(%{"event:props:cta" => {:is, "Top"}})
%{"hostname" => "dummy.site"}
|> assert_parsed(%{"event:hostname" => {:is, "dummy.site"}})
end end
end end

@ -0,0 +1,164 @@
defmodule Plausible.Stats.TableDeciderTest do
use Plausible.DataCase, async: true
alias Plausible.Stats.Query
import Plausible.Stats.TableDecider
test "events_join_sessions? with experimental_reduced_joins disabled" do
assert not events_join_sessions?(make_query(false, %{}))
assert not events_join_sessions?(make_query(false, %{name: "pageview"}))
assert events_join_sessions?(make_query(false, %{source: "Google"}))
assert events_join_sessions?(make_query(false, %{entry_page: "/"}))
assert events_join_sessions?(make_query(false, %{exit_page: "/"}))
end
test "events_join_sessions? with experimental_reduced_joins enabled" do
assert not events_join_sessions?(make_query(true, %{}))
assert not events_join_sessions?(make_query(true, %{name: "pageview"}))
assert not events_join_sessions?(make_query(true, %{source: "Google"}))
assert events_join_sessions?(make_query(true, %{entry_page: "/"}))
assert events_join_sessions?(make_query(true, %{exit_page: "/"}))
end
describe "partition_metrics" do
test "with no metrics or filters" do
query = make_query(false, %{})
assert partition_metrics([], query) == {[], [], []}
end
test "session-only metrics accordingly" do
query = make_query(false, %{})
assert partition_metrics([:bounce_rate, :views_per_visit], query) ==
{[], [:bounce_rate, :views_per_visit], []}
end
test "event-only metrics accordingly" do
query = make_query(false, %{})
assert partition_metrics([:total_revenue, :visitors], query) ==
{[:total_revenue, :visitors], [], []}
end
test "filters from both, event-only metrics" do
query = make_query(false, %{name: "pageview", source: "Google"})
assert partition_metrics([:total_revenue], query) == {[:total_revenue], [], []}
end
test "filters from both, session-only metrics" do
query = make_query(false, %{name: "pageview", source: "Google"})
assert partition_metrics([:bounce_rate], query) == {[], [:bounce_rate], []}
end
test "session filters but no session metrics" do
query = make_query(false, %{source: "Google"})
assert partition_metrics([:total_revenue], query) == {[:total_revenue], [], []}
end
test "sample_percent is added to both types of metrics" do
query = make_query(false, %{})
assert partition_metrics([:total_revenue, :sample_percent], query) ==
{[:total_revenue, :sample_percent], [], []}
assert partition_metrics([:bounce_rate, :sample_percent], query) ==
{[], [:bounce_rate, :sample_percent], []}
assert partition_metrics([:total_revenue, :bounce_rate, :sample_percent], query) ==
{[:total_revenue, :sample_percent], [:bounce_rate, :sample_percent], []}
end
test "other metrics put in its own result" do
query = make_query(false, %{})
assert partition_metrics([:time_on_page, :percentage, :total_visitors], query) ==
{[], [], [:time_on_page, :percentage, :total_visitors]}
end
test "raises if unknown metric" do
query = make_query(false, %{})
assert_raise ArgumentError, fn ->
partition_metrics([:foobar], query)
end
end
end
describe "partition_metrics with experimental_reduced_joins enabled" do
test "metrics that can be calculated on either when event-only metrics" do
query = make_query(true, %{})
assert partition_metrics([:total_revenue, :visitors], query) ==
{[:total_revenue, :visitors], [], []}
assert partition_metrics([:pageviews, :visits], query) == {[:pageviews, :visits], [], []}
end
test "metrics that can be calculated on either when session-only metrics" do
query = make_query(true, %{})
assert partition_metrics([:bounce_rate, :visitors], query) ==
{[], [:bounce_rate, :visitors], []}
assert partition_metrics([:visit_duration, :visits], query) ==
{[], [:visit_duration, :visits], []}
end
test "metrics that can be calculated on either are biased to sessions" do
query = make_query(true, %{})
assert partition_metrics([:bounce_rate, :total_revenue, :visitors], query) ==
{[:total_revenue], [:bounce_rate, :visitors], []}
end
test "sample_percent is handled with either metrics" do
query = make_query(true, %{})
assert partition_metrics([:visitors, :sample_percent], query) ==
{[], [:visitors, :sample_percent], []}
end
test "metric can be calculated on either, but filtering on events" do
query = make_query(true, %{name: "pageview"})
assert partition_metrics([:visitors], query) == {[:visitors], [], []}
end
test "metric can be calculated on either, but filtering on events and sessions" do
query = make_query(true, %{name: "pageview", exit_page: "/"})
assert partition_metrics([:visitors], query) == {[], [:visitors], []}
end
test "metric can be calculated on either, filtering on either" do
query = make_query(true, %{source: "Google"})
assert partition_metrics([:visitors], query) == {[], [:visitors], []}
end
test "metric can be calculated on either, filtering on sessions" do
query = make_query(true, %{exit_page: "/"})
assert partition_metrics([:visitors], query) == {[], [:visitors], []}
end
test "breakdown value leans metric" do
query = make_query(true, %{})
assert partition_metrics([:visitors], query, "event:name") == {[:visitors], [], []}
assert partition_metrics([:visitors], query, "visit:source") == {[], [:visitors], []}
assert partition_metrics([:visitors], query, "visit:exit_page") == {[], [:visitors], []}
end
end
defp make_query(experimental_reduced_joins?, filters) do
Query.from(build(:site), %{
"experimental_reduced_joins" => to_string(experimental_reduced_joins?),
"filters" => Jason.encode!(filters)
})
end
end

@ -1355,50 +1355,14 @@ defmodule PlausibleWeb.Api.ExternalStatsController.AggregateTest do
assert json_response(conn, 200)["results"] == %{"visitors" => %{"value" => 3}} assert json_response(conn, 200)["results"] == %{"visitors" => %{"value" => 3}}
end end
test "joins correctly with the sessions (CollapsingMergeTree) table", %{ test "handles filtering by visit country", %{
conn: conn, conn: conn,
site: site site: site
} do } do
create_sessions([ populate_stats(site, [
%{ build(:pageview, country_code: "EE"),
site_id: site.id, build(:pageview, country_code: "EE"),
session_id: 1000, build(:pageview, country_code: "EE")
country_code: "EE",
sign: 1,
events: 1
},
%{
site_id: site.id,
session_id: 1000,
country_code: "EE",
sign: -1,
events: 1
},
%{
site_id: site.id,
session_id: 1000,
country_code: "EE",
sign: 1,
events: 2
}
])
create_events([
%{
site_id: site.id,
session_id: 1000,
name: "pageview"
},
%{
site_id: site.id,
session_id: 1000,
name: "pageview"
},
%{
site_id: site.id,
session_id: 1000,
name: "pageview"
}
]) ])
conn = conn =

Some files were not shown because too many files have changed in this diff Show More

Loading…
Cancel
Save