[Continued] Google Analytics import (#1753)
* Add has_imported_stats boolean to Site * Add Google Analytics import panel to general settings * Get GA profiles to display in import settings panel * Add import_from_google method as entrypoint to import data * Add imported_visitors table * Remove conflicting code from migration * Import visitors data into clickhouse database * Pass another dataset to main graph for rendering in red This adds another entry to the JSON data returned via the main graph API called `imported_plot`, which is similar to `plot` in form but will be completed with previously imported data. Currently it simply returns the values from `plot` / 2. The data is rendered in the main graph in red without fill, and without an indicator for the present. Rationale: imported data will not continue to grow so there is no projection forward, only backwards. * Hook imported GA data to dashboard timeseries plot * Add settings option to forget imported data * Import sources from google analytics * Merge imported sources when queried * Merge imported source data native data when querying sources * Start converting metrics to atoms so they can be subqueried This changes "visitors" and in some places "sources" to atoms. This does not change the behaviour of the functions - the tests all pass unchanged following this commit. This is necessary as joining subqueries requires that the keys in `select` statements be atoms and not strings. * Convery GA (direct) source to empty string * Import utm campaign and utm medium from GA * format * Import all data types from GA into new tables * Handle large amounts of more data more safely * Fix some mistakes in tables * Make GA requests in chunks of 5 queries * Only display imported timeseries when there is no filter * Correctly show last 30 minutes timeseries when 'realtime' * Add with_imported key to Query struct * Account for injected :is_not filter on sources from dashboard * Also add tentative imported_utm_sources table This needs a bit more work on the google import side, as GA do not report sources and utm sources as distinct things. * Return imported data to dashboard for rest of Sources panel This extends the merge_imported function definition for sources to utm_sources, utm_mediums and utm_campaigns too. This appears to be working on the DB side but something is incomplete on the client side. * Clear imported stats from all tables when requested * Merge entry pages and exit pages from imported data into unfiltered dashboard view This requires converting the `"visits"` and `"visit_duration"` metrics to atoms so that they can be used in ecto subqueries. * Display imported devices, browsers and OSs on dashboard * Display imported country data on dashboard * Add more metrics to entries/exits for modals * make sure data is returned via API with correct keys * Import regions and cities from GA * Capitalize device upon import to match native data * Leave query limits/offsets until after possibly joining with imported data * Also import timeOnPage and pageviews for pages from GA * imported_countries -> imported_locations * Get timeOnPage and pageviews for pages from GA These are needed for the pages modal, and for calculating exit rates for exit pages. * Add indicator to dashboard when imported data is being used * Don't show imported data as separately line on main graph * "bounce_rate" -> :bounce_rate, so it works in subqueries * Drop imported browser and OS versions These are not needed. * Toggle displaying imported data by clicking indicator * Parse referrers with RefInspector - Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual referrer host + path, whereas 'ga:source' includes utm_mediums and other values when relevant. - 'ga:fullReferror' does however include search engine names directly, so they are manually checked for as RefInspector won't pick up on these. * Keep imported data indicator on dashboard and strikethrough when hidden * Add unlink google button to import panel * Rename some GA browsers and OSes to plausible versions * Get main top pages and exit pages panels working correctly with imported data * mix format * Fetch time_on_pages for imported data when needed * entry pages need to fetch bounces from GA * "sample_percent" -> :sample_percent as only atoms can be used in subqueries * Calculate bounce_rate for joined native and imported data for top pages modal * Flip some query bindings around to be less misleading * Fixup entry page modal visit durations * mix format * Fetch bounces and visit_duration for sources from GA * add more source metrics used for data in modals * Make sources modals display correct values * imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration * Merge imported data into aggregate stats * Reformat top graph side icons * Ensure sample_percent is yielded from aggregate data * filter event_props should be strings * Hide imported data from frontend when using filter * Fix existing tests * fix tests * Fix imported indicator appearing when filtering * comma needed, lost when rebasing * Import utm_terms and utm_content from GA * Merge imported utm_term and utm_content * Rename imported Countries data as Locations * Set imported city schema field to int * Remove utm_terms and utm_content when clearing imported * Clean locations import from Google Analytics - Country and region should be set to "" when GA provides "(not set)" - City should be set to 0 for "unknown", as we cannot reliably import city data from GA. * Display imported region and city in dashboard * os -> operating_system in some parts of code The inconsistency of using os in some places and operating_system in others causes trouble with subqueries and joins for the native and imported data, which would require additional logic to account for. The simplest solution is the just use a consistent word for all uses. This doesn't make any user-facing or database changes. * to_atom -> to_existing_atom * format * "events" metric -> :events * ignore imported data when "events" in metrics * update "bounce_rate" * atomise some more metrics from new city and region api * atomise some more metrics for email handlers * "conversion_rate" -> :conversion_rate during csv export * Move imported data stats code to own module * Move imported timeseries function to Stats.Imported * Use Timex.parse to import dates from GA * has_imported_stats -> imported_source * "time_on_page" -> :time_on_page * Convert imported GA data to UTC * Clean up GA request code a bit There was some weird logic here with two separate lists that really ought to be together, so this merges those. * Fail sooner if GA timezone can't be identified * Link imported tables to site by id * imported_utm_content -> imported_utm_contents * Imported GA from all of time * Reorganise GA data fetch logic - Fetch data from the start of time (2005) - Check whether no data was fetched, and if so, inform user and don't consider data to be imported. * Clarify removal of "visits" data when it isn't in metrics * Apply location filters from API This makes it consistent with the sources etc which filter out 'Direct / None' on the API side. These filters are used by both the native and imported data handling code, which would otherwise both duplicate the filters in their `where` clauses. * Do not use changeset for setting site.imported_source * Add all metrics to all dimensions * Run GA import in the background * Send email when GA import completes * Add handler to insert imported data into tests and imported_browsers_factory * Add remaining import data test factories * Add imported location data to test * Test main graph with imported data * Add imported data to operating systems tests * Add imported data to pages tests * Add imported data to entry pages tests * Add imported data to exit pages tests * Add imported data to devices tests * Add imported data to sources tests * Add imported data to UTM tests * Add new test module for the data import step * Test import of sources GA data * Test import of utm_mediums GA data * Test import of utm_campaigns GA data * Add tests for UTM terms * Add tests for UTM contents * Add test for importing pages and entry pages data from GA * Add test for importing exit page data * Fix module file name typo * Add test for importing location data from GA * Add test for importing devices data from GA * Add test for importing browsers data from GA * Add test for importing OS data from GA * Paginate GA requests to download all data * Bump clickhouse_ecto version * Move RefInspector wrapper function into module * Drop timezone transform on import * Order imported by side_id then date * More strings -> atoms Also changes a conditional to be a bit nicer * Remove parallelisation of data import * Split sources and UTM sources from fetched GA data GA has only a "source" dimension and no "UTM source" dimension. Instead it returns these combined. The logic herein to tease these apart is: 1. "(direct)" -> it's a direct source 2. if the source is a domain -> it's a source 3. "google" -> it's from adwords; let's make this a UTM source "adwords" 4. else -> just a UTM source * Keep prop names in queries as strings * fix typo * Fix import * Insert data to clickhouse in batches * Fix link when removing imported data * Merge source tables * Import hostname as well as pathname * Record start and end time of imported data * Track import progress * Fix month interval with imported data * Do not JOIN when imported date range has no overlap * Fix time on page using exits Co-authored-by: mcol <mcol@posteo.net>pull/1762/head
parent
b4992cedc1
commit
e27734ed79
@ -0,0 +1,249 @@
|
||||
defmodule Plausible.Imported do
|
||||
use Plausible.ClickhouseRepo
|
||||
use Timex
|
||||
|
||||
def forget(site) do
|
||||
Plausible.ClickhouseRepo.clear_imported_stats_for(site.id)
|
||||
end
|
||||
|
||||
def from_google_analytics(nil, _site_id, _metric, _timezone), do: {:ok, nil}
|
||||
|
||||
def from_google_analytics(data, site_id, table) do
|
||||
data =
|
||||
Enum.map(data, fn row ->
|
||||
new_from_google_analytics(site_id, table, row)
|
||||
end)
|
||||
|
||||
case ClickhouseRepo.insert_all(table, data) do
|
||||
{n_rows, _} when n_rows > 0 -> :ok
|
||||
error -> error
|
||||
end
|
||||
end
|
||||
|
||||
defp new_from_google_analytics(site_id, "imported_visitors", %{
|
||||
"dimensions" => [date],
|
||||
"metrics" => [%{"values" => values}]
|
||||
}) do
|
||||
[visitors, pageviews, bounces, visits, visit_duration] =
|
||||
values
|
||||
|> Enum.map(&Integer.parse/1)
|
||||
|> Enum.map(&elem(&1, 0))
|
||||
|
||||
%{
|
||||
site_id: site_id,
|
||||
date: format_date(date),
|
||||
visitors: visitors,
|
||||
pageviews: pageviews,
|
||||
bounces: bounces,
|
||||
visits: visits,
|
||||
visit_duration: visit_duration
|
||||
}
|
||||
end
|
||||
|
||||
# Credit: https://github.com/kvesteri/validators
|
||||
@domain ~r/^(([a-zA-Z]{1})|([a-zA-Z]{1}[a-zA-Z]{1})|([a-zA-Z]{1}[0-9]{1})|([0-9]{1}[a-zA-Z]{1})|([a-zA-Z0-9][-_.a-zA-Z0-9]{0,61}[a-zA-Z0-9]))\.([a-zA-Z]{2,13}|[a-zA-Z0-9-]{2,30}.[a-zA-Z]{2,3})$/
|
||||
|
||||
defp new_from_google_analytics(site_id, "imported_sources", %{
|
||||
"dimensions" => [date, source, medium, campaign, content, term],
|
||||
"metrics" => [%{"values" => [visitors, visits, bounces, visit_duration]}]
|
||||
}) do
|
||||
{visitors, ""} = Integer.parse(visitors)
|
||||
{visits, ""} = Integer.parse(visits)
|
||||
{bounces, ""} = Integer.parse(bounces)
|
||||
{visit_duration, _} = Integer.parse(visit_duration)
|
||||
|
||||
source = if source == "(direct)", do: nil, else: source
|
||||
source = if source && String.match?(source, @domain), do: parse_referrer(source), else: source
|
||||
|
||||
%{
|
||||
site_id: site_id,
|
||||
date: format_date(date),
|
||||
source: parse_referrer(source),
|
||||
utm_medium: nil_if_missing(medium),
|
||||
utm_campaign: nil_if_missing(campaign),
|
||||
utm_content: nil_if_missing(content),
|
||||
utm_term: nil_if_missing(term),
|
||||
visitors: visitors,
|
||||
visits: visits,
|
||||
bounces: bounces,
|
||||
visit_duration: visit_duration
|
||||
}
|
||||
end
|
||||
|
||||
defp new_from_google_analytics(site_id, "imported_pages", %{
|
||||
"dimensions" => [date, hostname, page],
|
||||
"metrics" => [%{"values" => [visitors, pageviews, exits, time_on_page]}]
|
||||
}) do
|
||||
page = URI.parse(page).path
|
||||
{visitors, ""} = Integer.parse(visitors)
|
||||
{pageviews, ""} = Integer.parse(pageviews)
|
||||
{exits, ""} = Integer.parse(exits)
|
||||
{time_on_page, _} = Integer.parse(time_on_page)
|
||||
|
||||
%{
|
||||
site_id: site_id,
|
||||
date: format_date(date),
|
||||
hostname: String.replace_prefix(hostname, "www.", ""),
|
||||
page: page,
|
||||
visitors: visitors,
|
||||
pageviews: pageviews,
|
||||
exits: exits,
|
||||
time_on_page: time_on_page
|
||||
}
|
||||
end
|
||||
|
||||
defp new_from_google_analytics(site_id, "imported_entry_pages", %{
|
||||
"dimensions" => [date, entry_page],
|
||||
"metrics" => [%{"values" => [visitors, entrances, visit_duration, bounces]}]
|
||||
}) do
|
||||
{visitors, ""} = Integer.parse(visitors)
|
||||
{entrances, ""} = Integer.parse(entrances)
|
||||
{bounces, ""} = Integer.parse(bounces)
|
||||
{visit_duration, _} = Integer.parse(visit_duration)
|
||||
|
||||
%{
|
||||
site_id: site_id,
|
||||
date: format_date(date),
|
||||
entry_page: entry_page,
|
||||
visitors: visitors,
|
||||
entrances: entrances,
|
||||
visit_duration: visit_duration,
|
||||
bounces: bounces
|
||||
}
|
||||
end
|
||||
|
||||
defp new_from_google_analytics(site_id, "imported_exit_pages", %{
|
||||
"dimensions" => [date, exit_page],
|
||||
"metrics" => [%{"values" => [visitors, exits]}]
|
||||
}) do
|
||||
{visitors, ""} = Integer.parse(visitors)
|
||||
{exits, ""} = Integer.parse(exits)
|
||||
|
||||
%{
|
||||
site_id: site_id,
|
||||
date: format_date(date),
|
||||
exit_page: exit_page,
|
||||
visitors: visitors,
|
||||
exits: exits
|
||||
}
|
||||
end
|
||||
|
||||
defp new_from_google_analytics(site_id, "imported_locations", %{
|
||||
"dimensions" => [date, country, region],
|
||||
"metrics" => [%{"values" => [visitors, visits, bounces, visit_duration]}]
|
||||
}) do
|
||||
country = if country == "(not set)", do: "", else: country
|
||||
region = if region == "(not set)", do: "", else: region
|
||||
{visitors, ""} = Integer.parse(visitors)
|
||||
{visits, ""} = Integer.parse(visits)
|
||||
{bounces, ""} = Integer.parse(bounces)
|
||||
{visit_duration, _} = Integer.parse(visit_duration)
|
||||
|
||||
%{
|
||||
site_id: site_id,
|
||||
date: format_date(date),
|
||||
country: country,
|
||||
region: region,
|
||||
city: 0,
|
||||
visitors: visitors,
|
||||
visits: visits,
|
||||
bounces: bounces,
|
||||
visit_duration: visit_duration
|
||||
}
|
||||
end
|
||||
|
||||
defp new_from_google_analytics(site_id, "imported_devices", %{
|
||||
"dimensions" => [date, device],
|
||||
"metrics" => [%{"values" => [visitors, visits, bounces, visit_duration]}]
|
||||
}) do
|
||||
{visitors, ""} = Integer.parse(visitors)
|
||||
{visits, ""} = Integer.parse(visits)
|
||||
{bounces, ""} = Integer.parse(bounces)
|
||||
{visit_duration, _} = Integer.parse(visit_duration)
|
||||
|
||||
%{
|
||||
site_id: site_id,
|
||||
date: format_date(date),
|
||||
device: String.capitalize(device),
|
||||
visitors: visitors,
|
||||
visits: visits,
|
||||
bounces: bounces,
|
||||
visit_duration: visit_duration
|
||||
}
|
||||
end
|
||||
|
||||
@browser_google_to_plausible %{
|
||||
"User-Agent:Opera" => "Opera",
|
||||
"Mozilla Compatible Agent" => "Mobile App",
|
||||
"Android Webview" => "Mobile App",
|
||||
"Android Browser" => "Mobile App",
|
||||
"Safari (in-app)" => "Mobile App",
|
||||
"User-Agent: Mozilla" => "Firefox",
|
||||
"(not set)" => ""
|
||||
}
|
||||
|
||||
defp new_from_google_analytics(site_id, "imported_browsers", %{
|
||||
"dimensions" => [date, browser],
|
||||
"metrics" => [%{"values" => [visitors, visits, bounces, visit_duration]}]
|
||||
}) do
|
||||
{visitors, ""} = Integer.parse(visitors)
|
||||
{visits, ""} = Integer.parse(visits)
|
||||
{bounces, ""} = Integer.parse(bounces)
|
||||
{visit_duration, _} = Integer.parse(visit_duration)
|
||||
|
||||
%{
|
||||
site_id: site_id,
|
||||
date: format_date(date),
|
||||
browser: Map.get(@browser_google_to_plausible, browser, browser),
|
||||
visitors: visitors,
|
||||
visits: visits,
|
||||
bounces: bounces,
|
||||
visit_duration: visit_duration
|
||||
}
|
||||
end
|
||||
|
||||
@os_google_to_plausible %{
|
||||
"Macintosh" => "Mac",
|
||||
"Linux" => "GNU/Linux",
|
||||
"(not set)" => ""
|
||||
}
|
||||
|
||||
defp new_from_google_analytics(site_id, "imported_operating_systems", %{
|
||||
"dimensions" => [date, operating_system],
|
||||
"metrics" => [%{"values" => [visitors, visits, bounces, visit_duration]}]
|
||||
}) do
|
||||
{visitors, ""} = Integer.parse(visitors)
|
||||
{visits, ""} = Integer.parse(visits)
|
||||
{bounces, ""} = Integer.parse(bounces)
|
||||
{visit_duration, _} = Integer.parse(visit_duration)
|
||||
|
||||
%{
|
||||
site_id: site_id,
|
||||
date: format_date(date),
|
||||
operating_system: Map.get(@os_google_to_plausible, operating_system, operating_system),
|
||||
visitors: visitors,
|
||||
visits: visits,
|
||||
bounces: bounces,
|
||||
visit_duration: visit_duration
|
||||
}
|
||||
end
|
||||
|
||||
defp format_date(date) do
|
||||
Timex.parse!("#{date}", "%Y%m%d", :strftime)
|
||||
|> NaiveDateTime.to_date()
|
||||
end
|
||||
|
||||
@missing_values ["(none)", "(not set)", "(not provided)"]
|
||||
def nil_if_missing(value) when value in @missing_values, do: nil
|
||||
def nil_if_missing(value), do: value
|
||||
|
||||
def parse_referrer(nil), do: nil
|
||||
def parse_referrer("google"), do: "Google"
|
||||
def parse_referrer("bing"), do: "Bing"
|
||||
def parse_referrer("duckduckgo"), do: "DuckDuckGo"
|
||||
|
||||
def parse_referrer(ref) do
|
||||
RefInspector.parse("https://" <> ref)
|
||||
|> PlausibleWeb.RefInspector.parse()
|
||||
end
|
||||
end
|
||||
@ -0,0 +1,434 @@
|
||||
defmodule Plausible.Stats.Imported do
|
||||
use Plausible.ClickhouseRepo
|
||||
alias Plausible.Stats.Query
|
||||
import Ecto.Query
|
||||
|
||||
@no_ref "Direct / None"
|
||||
|
||||
def merge_imported_timeseries(native_q, _, %Plausible.Stats.Query{include_imported: false}, _),
|
||||
do: native_q
|
||||
|
||||
def merge_imported_timeseries(
|
||||
native_q,
|
||||
site,
|
||||
query,
|
||||
metrics
|
||||
) do
|
||||
imported_q =
|
||||
from(v in "imported_visitors",
|
||||
where: v.site_id == ^site.id,
|
||||
where: v.date >= ^query.date_range.first and v.date <= ^query.date_range.last,
|
||||
select: %{visitors: sum(v.visitors)}
|
||||
)
|
||||
|> apply_interval(query)
|
||||
|
||||
from(s in Ecto.Query.subquery(native_q),
|
||||
full_join: i in subquery(imported_q),
|
||||
on: field(s, :date) == field(i, :date)
|
||||
)
|
||||
|> select_joined_metrics(metrics)
|
||||
end
|
||||
|
||||
defp apply_interval(imported_q, %Plausible.Stats.Query{interval: "month"}) do
|
||||
imported_q
|
||||
|> group_by([i], fragment("toStartOfMonth(?)", i.date))
|
||||
|> select_merge([i], %{date: fragment("toStartOfMonth(?)", i.date)})
|
||||
end
|
||||
|
||||
defp apply_interval(imported_q, _query) do
|
||||
imported_q
|
||||
|> group_by([i], i.date)
|
||||
|> select_merge([i], %{date: i.date})
|
||||
end
|
||||
|
||||
def merge_imported(q, _, %Query{include_imported: false}, _, _), do: q
|
||||
def merge_imported(q, _, _, _, [:events | _]), do: q
|
||||
# GA only has 'source'
|
||||
def merge_imported(q, _, _, "utm_source", _), do: q
|
||||
|
||||
def merge_imported(q, site, query, property, metrics)
|
||||
when property in [
|
||||
"visit:source",
|
||||
"visit:utm_medium",
|
||||
"visit:utm_campaign",
|
||||
"visit:utm_term",
|
||||
"visit:utm_content",
|
||||
"visit:entry_page",
|
||||
"visit:exit_page",
|
||||
"visit:country",
|
||||
"visit:region",
|
||||
"visit:city",
|
||||
"visit:device",
|
||||
"visit:browser",
|
||||
"visit:os",
|
||||
"event:page"
|
||||
] do
|
||||
{table, dim} =
|
||||
case property do
|
||||
"visit:country" ->
|
||||
{"imported_locations", :country}
|
||||
|
||||
"visit:region" ->
|
||||
{"imported_locations", :region}
|
||||
|
||||
"visit:city" ->
|
||||
{"imported_locations", :city}
|
||||
|
||||
"visit:utm_medium" ->
|
||||
{"imported_sources", :utm_medium}
|
||||
|
||||
"visit:utm_campaign" ->
|
||||
{"imported_sources", :utm_campaign}
|
||||
|
||||
"visit:utm_term" ->
|
||||
{"imported_sources", :utm_term}
|
||||
|
||||
"visit:utm_content" ->
|
||||
{"imported_sources", :utm_content}
|
||||
|
||||
"visit:os" ->
|
||||
{"imported_operating_systems", :operating_system}
|
||||
|
||||
"event:page" ->
|
||||
{"imported_pages", :page}
|
||||
|
||||
_ ->
|
||||
dim = String.trim_leading(property, "visit:")
|
||||
{"imported_#{dim}s", String.to_existing_atom(dim)}
|
||||
end
|
||||
|
||||
imported_q =
|
||||
from(
|
||||
i in table,
|
||||
group_by: field(i, ^dim),
|
||||
where: i.site_id == ^site.id,
|
||||
where: i.date >= ^query.date_range.first and i.date <= ^query.date_range.last,
|
||||
select: %{}
|
||||
)
|
||||
|> select_imported_metrics(metrics)
|
||||
|
||||
imported_q =
|
||||
case query.filters[property] do
|
||||
{:is_not, value} ->
|
||||
value = if value == @no_ref, do: "", else: value
|
||||
where(imported_q, [i], field(i, ^dim) != ^value)
|
||||
|
||||
{:member, list} ->
|
||||
where(imported_q, [i], field(i, ^dim) in ^list)
|
||||
|
||||
_ ->
|
||||
imported_q
|
||||
end
|
||||
|
||||
imported_q =
|
||||
case dim do
|
||||
:source ->
|
||||
imported_q
|
||||
|> select_merge([i], %{
|
||||
source: fragment("if(empty(?), ?, ?)", i.source, @no_ref, i.source)
|
||||
})
|
||||
|
||||
:utm_medium ->
|
||||
imported_q
|
||||
|> select_merge([i], %{
|
||||
utm_medium: fragment("if(empty(?), ?, ?)", i.utm_medium, @no_ref, i.utm_medium)
|
||||
})
|
||||
|
||||
:utm_source ->
|
||||
imported_q
|
||||
|> select_merge([i], %{
|
||||
utm_source: fragment("if(empty(?), ?, ?)", i.utm_source, @no_ref, i.utm_source)
|
||||
})
|
||||
|
||||
:utm_campaign ->
|
||||
imported_q
|
||||
|> select_merge([i], %{
|
||||
utm_campaign: fragment("if(empty(?), ?, ?)", i.utm_campaign, @no_ref, i.utm_campaign)
|
||||
})
|
||||
|
||||
:utm_term ->
|
||||
imported_q
|
||||
|> select_merge([i], %{
|
||||
utm_term: fragment("if(empty(?), ?, ?)", i.utm_term, @no_ref, i.utm_term)
|
||||
})
|
||||
|
||||
:utm_content ->
|
||||
imported_q
|
||||
|> select_merge([i], %{
|
||||
utm_content: fragment("if(empty(?), ?, ?)", i.utm_content, @no_ref, i.utm_content)
|
||||
})
|
||||
|
||||
:page ->
|
||||
imported_q
|
||||
|> select_merge([i], %{
|
||||
page: i.page,
|
||||
time_on_page: sum(i.time_on_page)
|
||||
})
|
||||
|
||||
:entry_page ->
|
||||
imported_q
|
||||
|> select_merge([i], %{
|
||||
entry_page: i.entry_page,
|
||||
visits: sum(i.entrances)
|
||||
})
|
||||
|
||||
:exit_page ->
|
||||
imported_q
|
||||
|> select_merge([i], %{exit_page: i.exit_page, visits: sum(i.exits)})
|
||||
|
||||
:country ->
|
||||
imported_q |> select_merge([i], %{country: i.country})
|
||||
|
||||
:region ->
|
||||
imported_q |> select_merge([i], %{region: i.region})
|
||||
|
||||
:city ->
|
||||
imported_q |> select_merge([i], %{city: i.city})
|
||||
|
||||
:device ->
|
||||
imported_q |> select_merge([i], %{device: i.device})
|
||||
|
||||
:browser ->
|
||||
imported_q |> select_merge([i], %{browser: i.browser})
|
||||
|
||||
:operating_system ->
|
||||
imported_q |> select_merge([i], %{operating_system: i.operating_system})
|
||||
end
|
||||
|
||||
q =
|
||||
from(s in Ecto.Query.subquery(q),
|
||||
full_join: i in subquery(imported_q),
|
||||
on: field(s, ^dim) == field(i, ^dim)
|
||||
)
|
||||
|> select_joined_metrics(metrics)
|
||||
|> apply_order_by(metrics)
|
||||
|
||||
case dim do
|
||||
:source ->
|
||||
q
|
||||
|> select_merge([s, i], %{
|
||||
source: fragment("if(empty(?), ?, ?)", s.source, i.source, s.source)
|
||||
})
|
||||
|
||||
:utm_medium ->
|
||||
q
|
||||
|> select_merge([s, i], %{
|
||||
utm_medium: fragment("if(empty(?), ?, ?)", s.utm_medium, i.utm_medium, s.utm_medium)
|
||||
})
|
||||
|
||||
:utm_source ->
|
||||
q
|
||||
|> select_merge([s, i], %{
|
||||
utm_source: fragment("if(empty(?), ?, ?)", s.utm_source, i.utm_source, s.utm_source)
|
||||
})
|
||||
|
||||
:utm_campaign ->
|
||||
q
|
||||
|> select_merge([s, i], %{
|
||||
utm_campaign:
|
||||
fragment("if(empty(?), ?, ?)", s.utm_campaign, i.utm_campaign, s.utm_campaign)
|
||||
})
|
||||
|
||||
:utm_term ->
|
||||
q
|
||||
|> select_merge([s, i], %{
|
||||
utm_term: fragment("if(empty(?), ?, ?)", s.utm_term, i.utm_term, s.utm_term)
|
||||
})
|
||||
|
||||
:utm_content ->
|
||||
q
|
||||
|> select_merge([s, i], %{
|
||||
utm_content: fragment("if(empty(?), ?, ?)", s.utm_content, i.utm_content, s.utm_content)
|
||||
})
|
||||
|
||||
:page ->
|
||||
q
|
||||
|> select_merge([s, i], %{
|
||||
page: fragment("if(empty(?), ?, ?)", i.page, s.page, i.page)
|
||||
})
|
||||
|
||||
:entry_page ->
|
||||
q
|
||||
|> select_merge([s, i], %{
|
||||
entry_page: fragment("if(empty(?), ?, ?)", i.entry_page, s.entry_page, i.entry_page),
|
||||
visits: fragment("? + ?", s.visits, i.visits)
|
||||
})
|
||||
|
||||
:exit_page ->
|
||||
q
|
||||
|> select_merge([s, i], %{
|
||||
exit_page: fragment("if(empty(?), ?, ?)", i.exit_page, s.exit_page, i.exit_page),
|
||||
visits: fragment("coalesce(?, 0) + coalesce(?, 0)", s.visits, i.visits)
|
||||
})
|
||||
|
||||
:country ->
|
||||
q
|
||||
|> select_merge([i, s], %{
|
||||
country: fragment("if(empty(?), ?, ?)", s.country, i.country, s.country)
|
||||
})
|
||||
|
||||
:region ->
|
||||
q
|
||||
|> select_merge([i, s], %{
|
||||
region: fragment("if(empty(?), ?, ?)", s.region, i.region, s.region)
|
||||
})
|
||||
|
||||
:city ->
|
||||
q
|
||||
|> select_merge([i, s], %{
|
||||
city: fragment("coalesce(?, ?)", s.city, i.city)
|
||||
})
|
||||
|
||||
:device ->
|
||||
q
|
||||
|> select_merge([i, s], %{
|
||||
device: fragment("if(empty(?), ?, ?)", s.device, i.device, s.device)
|
||||
})
|
||||
|
||||
:browser ->
|
||||
q
|
||||
|> select_merge([i, s], %{
|
||||
browser: fragment("if(empty(?), ?, ?)", s.browser, i.browser, s.browser)
|
||||
})
|
||||
|
||||
:operating_system ->
|
||||
q
|
||||
|> select_merge([i, s], %{
|
||||
operating_system:
|
||||
fragment(
|
||||
"if(empty(?), ?, ?)",
|
||||
s.operating_system,
|
||||
i.operating_system,
|
||||
s.operating_system
|
||||
)
|
||||
})
|
||||
end
|
||||
end
|
||||
|
||||
def merge_imported(q, site, query, :aggregate, metrics) do
|
||||
imported_q =
|
||||
from(
|
||||
i in "imported_visitors",
|
||||
where: i.site_id == ^site.id,
|
||||
where: i.date >= ^query.date_range.first and i.date <= ^query.date_range.last,
|
||||
select: %{}
|
||||
)
|
||||
|> select_imported_metrics(metrics)
|
||||
|
||||
from(
|
||||
s in subquery(q),
|
||||
cross_join: i in subquery(imported_q),
|
||||
select: %{}
|
||||
)
|
||||
|> select_joined_metrics(metrics)
|
||||
end
|
||||
|
||||
def merge_imported(q, _, _, _, _), do: q
|
||||
|
||||
defp select_imported_metrics(q, []), do: q
|
||||
|
||||
defp select_imported_metrics(q, [:visitors | rest]) do
|
||||
q
|
||||
|> select_merge([i], %{visitors: sum(i.visitors)})
|
||||
|> select_imported_metrics(rest)
|
||||
end
|
||||
|
||||
defp select_imported_metrics(q, [:pageviews | rest]) do
|
||||
q
|
||||
|> select_merge([i], %{pageviews: sum(i.pageviews)})
|
||||
|> select_imported_metrics(rest)
|
||||
end
|
||||
|
||||
defp select_imported_metrics(q, [:bounce_rate | rest]) do
|
||||
q
|
||||
|> select_merge([i], %{
|
||||
bounces: sum(i.bounces),
|
||||
visits: sum(i.visits)
|
||||
})
|
||||
|> select_imported_metrics(rest)
|
||||
end
|
||||
|
||||
defp select_imported_metrics(q, [:visit_duration | rest]) do
|
||||
q
|
||||
|> select_merge([i], %{visit_duration: sum(i.visit_duration)})
|
||||
|> select_imported_metrics(rest)
|
||||
end
|
||||
|
||||
defp select_imported_metrics(q, [_ | rest]) do
|
||||
q
|
||||
|> select_imported_metrics(rest)
|
||||
end
|
||||
|
||||
defp select_joined_metrics(q, []), do: q
|
||||
# TODO: Reverse-engineering the native data bounces and total visit
|
||||
# durations to combine with imported data is inefficient. Instead both
|
||||
# queries should fetch bounces/total_visit_duration and visits and be
|
||||
# used as subqueries to a main query that then find the bounce rate/avg
|
||||
# visit_duration.
|
||||
|
||||
defp select_joined_metrics(q, [:visitors | rest]) do
|
||||
q
|
||||
|> select_merge([s, i], %{
|
||||
:visitors => fragment("coalesce(?, 0) + coalesce(?, 0)", s.visitors, i.visitors)
|
||||
})
|
||||
|> select_joined_metrics(rest)
|
||||
end
|
||||
|
||||
defp select_joined_metrics(q, [:pageviews | rest]) do
|
||||
q
|
||||
|> select_merge([s, i], %{
|
||||
pageviews: fragment("coalesce(?, 0) + coalesce(?, 0)", s.pageviews, i.pageviews)
|
||||
})
|
||||
|> select_joined_metrics(rest)
|
||||
end
|
||||
|
||||
defp select_joined_metrics(q, [:bounce_rate | rest]) do
|
||||
q
|
||||
|> select_merge([s, i], %{
|
||||
bounce_rate:
|
||||
fragment(
|
||||
"round(100 * (coalesce(?, 0) + coalesce((? * ? / 100), 0)) / (coalesce(?, 0) + coalesce(?, 0)))",
|
||||
i.bounces,
|
||||
s.bounce_rate,
|
||||
s.visits,
|
||||
i.visits,
|
||||
s.visits
|
||||
)
|
||||
})
|
||||
|> select_joined_metrics(rest)
|
||||
end
|
||||
|
||||
defp select_joined_metrics(q, [:visit_duration | rest]) do
|
||||
q
|
||||
|> select_merge([s, i], %{
|
||||
visit_duration:
|
||||
fragment(
|
||||
"(? + ? * ?) / (? + ?)",
|
||||
i.visit_duration,
|
||||
s.visit_duration,
|
||||
s.visits,
|
||||
s.visits,
|
||||
i.visits
|
||||
)
|
||||
})
|
||||
|> select_joined_metrics(rest)
|
||||
end
|
||||
|
||||
defp select_joined_metrics(q, [:sample_percent | rest]) do
|
||||
q
|
||||
|> select_merge([s, i], %{sample_percent: s.sample_percent})
|
||||
|> select_joined_metrics(rest)
|
||||
end
|
||||
|
||||
defp select_joined_metrics(q, [_ | rest]) do
|
||||
q
|
||||
|> select_joined_metrics(rest)
|
||||
end
|
||||
|
||||
defp apply_order_by(q, [:visitors | rest]) do
|
||||
order_by(q, [s, i], desc: fragment("coalesce(?, 0) + coalesce(?, 0)", s.visitors, i.visitors))
|
||||
|> apply_order_by(rest)
|
||||
end
|
||||
|
||||
defp apply_order_by(q, _), do: q
|
||||
end
|
||||
@ -0,0 +1,25 @@
|
||||
defmodule PlausibleWeb.RefInspector do
|
||||
def parse(nil), do: nil
|
||||
|
||||
def parse(ref) do
|
||||
case ref.source do
|
||||
:unknown ->
|
||||
uri = URI.parse(String.trim(ref.referer))
|
||||
|
||||
if right_uri?(uri) do
|
||||
String.replace_leading(uri.host, "www.", "")
|
||||
end
|
||||
|
||||
source ->
|
||||
source
|
||||
end
|
||||
end
|
||||
|
||||
def right_uri?(%URI{host: nil}), do: false
|
||||
|
||||
def right_uri?(%URI{host: host, scheme: scheme})
|
||||
when scheme in ["http", "https"] and byte_size(host) > 0,
|
||||
do: true
|
||||
|
||||
def right_uri?(_), do: false
|
||||
end
|
||||
@ -0,0 +1,13 @@
|
||||
Hey <%= user_salutation(@user) %>,
|
||||
<br /><br />
|
||||
<%= if @success do %>
|
||||
Your Google Analytics import has completed.
|
||||
<br /><br />
|
||||
View dashboard: @link
|
||||
<% else %>
|
||||
Unfortunately, your Google Analytics import failed.
|
||||
<% end %>
|
||||
<br /><br />
|
||||
--
|
||||
<br /><br />
|
||||
<%= plausible_url() %><br />
|
||||
@ -0,0 +1,46 @@
|
||||
defmodule Plausible.Workers.ImportGoogleAnalytics do
|
||||
use Plausible.Repo
|
||||
|
||||
use Oban.Worker,
|
||||
queue: :google_analytics_imports,
|
||||
max_attempts: 1,
|
||||
unique: [fields: [:args], period: 60]
|
||||
|
||||
@impl Oban.Worker
|
||||
def perform(
|
||||
%Oban.Job{args: %{"site_id" => site_id, "profile" => profile}},
|
||||
google_api \\ Plausible.Google.Api
|
||||
) do
|
||||
site =
|
||||
Repo.get(Plausible.Site, site_id)
|
||||
|> Repo.preload([:google_auth, [memberships: :user]])
|
||||
|
||||
case google_api.import_analytics(site, profile) do
|
||||
{:ok, _} ->
|
||||
Plausible.Site.import_success(site)
|
||||
|> Repo.update!()
|
||||
|
||||
Enum.each(site.memberships, fn membership ->
|
||||
if membership.role in [:owner, :admin] do
|
||||
PlausibleWeb.Email.import_success(membership.user, site)
|
||||
|> Plausible.Mailer.send_email_safe()
|
||||
end
|
||||
end)
|
||||
|
||||
:ok
|
||||
|
||||
{:error, error} ->
|
||||
Plausible.Site.import_failure(site)
|
||||
|> Repo.update!()
|
||||
|
||||
Enum.each(site.memberships, fn membership ->
|
||||
if membership.role in [:owner, :admin] do
|
||||
PlausibleWeb.Email.import_failure(membership.user, site)
|
||||
|> Plausible.Mailer.send_email_safe()
|
||||
end
|
||||
end)
|
||||
|
||||
{:error, error}
|
||||
end
|
||||
end
|
||||
end
|
||||
@ -0,0 +1,118 @@
|
||||
defmodule Plausible.ClickhouseRepo.Migrations.CreateImportedVisitors do
|
||||
use Ecto.Migration
|
||||
|
||||
def change do
|
||||
create_if_not_exists table(:imported_visitors,
|
||||
engine: "MergeTree() ORDER BY (site_id, date)"
|
||||
) do
|
||||
add(:site_id, :UInt64)
|
||||
add(:date, :date)
|
||||
add(:visitors, :UInt64)
|
||||
add(:pageviews, :UInt64)
|
||||
add(:bounces, :UInt64)
|
||||
add(:visits, :UInt64)
|
||||
add(:visit_duration, :UInt64)
|
||||
end
|
||||
|
||||
create_if_not_exists table(:imported_sources,
|
||||
engine: "MergeTree() ORDER BY (site_id, date, source)"
|
||||
) do
|
||||
add(:site_id, :UInt64)
|
||||
add(:date, :date)
|
||||
add(:source, :string)
|
||||
add(:utm_medium, :string)
|
||||
add(:utm_campaign, :string)
|
||||
add(:utm_content, :string)
|
||||
add(:utm_term, :string)
|
||||
add(:visitors, :UInt64)
|
||||
add(:visits, :UInt64)
|
||||
add(:visit_duration, :UInt64)
|
||||
add(:bounces, :UInt32)
|
||||
end
|
||||
|
||||
create_if_not_exists table(:imported_pages,
|
||||
engine: "MergeTree() ORDER BY (site_id, date, hostname, page)"
|
||||
) do
|
||||
add(:site_id, :UInt64)
|
||||
add(:date, :date)
|
||||
add(:hostname, :string)
|
||||
add(:page, :string)
|
||||
add(:visitors, :UInt64)
|
||||
add(:pageviews, :UInt64)
|
||||
add(:exits, :UInt64)
|
||||
add(:time_on_page, :UInt64)
|
||||
end
|
||||
|
||||
create_if_not_exists table(:imported_entry_pages,
|
||||
engine: "MergeTree() ORDER BY (site_id, date, entry_page)"
|
||||
) do
|
||||
add(:site_id, :UInt64)
|
||||
add(:date, :date)
|
||||
add(:entry_page, :string)
|
||||
add(:visitors, :UInt64)
|
||||
add(:entrances, :UInt64)
|
||||
add(:visit_duration, :UInt64)
|
||||
add(:bounces, :UInt32)
|
||||
end
|
||||
|
||||
create_if_not_exists table(:imported_exit_pages,
|
||||
engine: "MergeTree() ORDER BY (site_id, date, exit_page)"
|
||||
) do
|
||||
add(:site_id, :UInt64)
|
||||
add(:date, :date)
|
||||
add(:exit_page, :string)
|
||||
add(:visitors, :UInt64)
|
||||
add(:exits, :UInt64)
|
||||
end
|
||||
|
||||
create_if_not_exists table(:imported_locations,
|
||||
engine: "MergeTree() ORDER BY (site_id, date, country, region, city)"
|
||||
) do
|
||||
add(:site_id, :UInt64)
|
||||
add(:date, :date)
|
||||
add(:country, :string)
|
||||
add(:region, :string)
|
||||
add(:city, :UInt64)
|
||||
add(:visitors, :UInt64)
|
||||
add(:visits, :UInt64)
|
||||
add(:visit_duration, :UInt64)
|
||||
add(:bounces, :UInt32)
|
||||
end
|
||||
|
||||
create_if_not_exists table(:imported_devices,
|
||||
engine: "MergeTree() ORDER BY (site_id, date, device)"
|
||||
) do
|
||||
add(:site_id, :UInt64)
|
||||
add(:date, :date)
|
||||
add(:device, :string)
|
||||
add(:visitors, :UInt64)
|
||||
add(:visits, :UInt64)
|
||||
add(:visit_duration, :UInt64)
|
||||
add(:bounces, :UInt32)
|
||||
end
|
||||
|
||||
create_if_not_exists table(:imported_browsers,
|
||||
engine: "MergeTree() ORDER BY (site_id, date, browser)"
|
||||
) do
|
||||
add(:site_id, :UInt64)
|
||||
add(:date, :date)
|
||||
add(:browser, :string)
|
||||
add(:visitors, :UInt64)
|
||||
add(:visits, :UInt64)
|
||||
add(:visit_duration, :UInt64)
|
||||
add(:bounces, :UInt32)
|
||||
end
|
||||
|
||||
create_if_not_exists table(:imported_operating_systems,
|
||||
engine: "MergeTree() ORDER BY (site_id, date, operating_system)"
|
||||
) do
|
||||
add(:site_id, :UInt64)
|
||||
add(:date, :date)
|
||||
add(:operating_system, :string)
|
||||
add(:visitors, :UInt64)
|
||||
add(:visits, :UInt64)
|
||||
add(:visit_duration, :UInt64)
|
||||
add(:bounces, :UInt32)
|
||||
end
|
||||
end
|
||||
end
|
||||
@ -0,0 +1,9 @@
|
||||
defmodule Plausible.Repo.Migrations.GoogleAuthImportedSource do
|
||||
use Ecto.Migration
|
||||
|
||||
def change do
|
||||
alter table(:sites) do
|
||||
add :imported_data, :map
|
||||
end
|
||||
end
|
||||
end
|
||||
@ -0,0 +1,606 @@
|
||||
defmodule Plausible.ImportedTest do
|
||||
use PlausibleWeb.ConnCase
|
||||
use Timex
|
||||
import Plausible.TestUtils
|
||||
|
||||
@user_id 123
|
||||
|
||||
describe "Parse and import third party data fetched from Google Analytics" do
|
||||
setup [:create_user, :log_in, :create_new_site, :add_imported_data]
|
||||
|
||||
test "Visitors data imported from Google Analytics", %{conn: conn, site: site} do
|
||||
populate_stats(site, [
|
||||
build(:pageview, timestamp: ~N[2021-01-01 00:00:00]),
|
||||
build(:pageview, timestamp: ~N[2021-01-31 00:00:00])
|
||||
])
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101"],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "1", "60"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210131"],
|
||||
"metrics" => [%{"values" => ["1", "1", "1", "1", "60"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_visitors"
|
||||
)
|
||||
|
||||
conn =
|
||||
get(
|
||||
conn,
|
||||
"/api/stats/#{site.domain}/main-graph?period=month&date=2021-01-01&with_imported=true"
|
||||
)
|
||||
|
||||
assert %{"plot" => plot, "imported_source" => "Google Analytics"} = json_response(conn, 200)
|
||||
|
||||
assert Enum.count(plot) == 31
|
||||
assert List.first(plot) == 2
|
||||
assert List.last(plot) == 2
|
||||
assert Enum.sum(plot) == 4
|
||||
end
|
||||
|
||||
test "Sources are imported", %{conn: conn, site: site} do
|
||||
populate_stats(site, [
|
||||
build(:pageview,
|
||||
referrer_source: "Google",
|
||||
referrer: "google.com",
|
||||
timestamp: ~N[2021-01-01 00:00:00]
|
||||
),
|
||||
build(:pageview,
|
||||
referrer_source: "Google",
|
||||
referrer: "google.com",
|
||||
timestamp: ~N[2021-01-01 00:00:00]
|
||||
),
|
||||
build(:pageview,
|
||||
referrer_source: "DuckDuckGo",
|
||||
referrer: "duckduckgo.com",
|
||||
timestamp: ~N[2021-01-01 00:00:00]
|
||||
)
|
||||
])
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101", "duckduckgo.com", "organic", "", "", ""],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "60"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210131", "google.com", "organic", "", "", ""],
|
||||
"metrics" => [%{"values" => ["1", "1", "1", "60"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "google.com", "paid", "", "", ""],
|
||||
"metrics" => [%{"values" => ["1", "1", "1", "60"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "Twitter", "social", "", "", ""],
|
||||
"metrics" => [%{"values" => ["1", "1", "1", "60"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => [
|
||||
"20210131",
|
||||
"A Nice Newsletter",
|
||||
"email",
|
||||
"newsletter",
|
||||
"",
|
||||
""
|
||||
],
|
||||
"metrics" => [%{"values" => ["1", "1", "1", "60"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "(direct)", "(none)", "", "", ""],
|
||||
"metrics" => [%{"values" => ["1", "1", "1", "60"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_sources"
|
||||
)
|
||||
|
||||
conn =
|
||||
get(
|
||||
conn,
|
||||
"/api/stats/#{site.domain}/sources?period=month&date=2021-01-01&with_imported=true"
|
||||
)
|
||||
|
||||
assert json_response(conn, 200) == [
|
||||
%{"name" => "Google", "visitors" => 4},
|
||||
%{"name" => "DuckDuckGo", "visitors" => 2},
|
||||
%{"name" => "A Nice Newsletter", "visitors" => 1},
|
||||
%{"name" => "Twitter", "visitors" => 1}
|
||||
]
|
||||
end
|
||||
|
||||
test "UTM mediums data imported from Google Analytics", %{conn: conn, site: site} do
|
||||
populate_stats(site, [
|
||||
build(:pageview,
|
||||
utm_medium: "social",
|
||||
timestamp: ~N[2021-01-01 00:00:00]
|
||||
),
|
||||
build(:pageview,
|
||||
utm_medium: "social",
|
||||
timestamp: ~N[2021-01-01 12:00:00]
|
||||
)
|
||||
])
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101", "Twitter", "social", "", "", ""],
|
||||
"metrics" => [%{"values" => ["1", "1", "1", "60"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "(direct)", "(none)", "", "", ""],
|
||||
"metrics" => [%{"values" => ["1", "1", "1", "60"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_sources"
|
||||
)
|
||||
|
||||
conn =
|
||||
get(
|
||||
conn,
|
||||
"/api/stats/#{site.domain}/utm_mediums?period=day&date=2021-01-01&with_imported=true"
|
||||
)
|
||||
|
||||
assert json_response(conn, 200) == [
|
||||
%{
|
||||
"bounce_rate" => 100.0,
|
||||
"name" => "social",
|
||||
"visit_duration" => 20,
|
||||
"visitors" => 3
|
||||
}
|
||||
]
|
||||
end
|
||||
|
||||
test "UTM campaigns data imported from Google Analytics", %{conn: conn, site: site} do
|
||||
populate_stats(site, [
|
||||
build(:pageview, utm_campaign: "profile", timestamp: ~N[2021-01-01 00:00:00]),
|
||||
build(:pageview, utm_campaign: "august", timestamp: ~N[2021-01-01 00:00:00])
|
||||
])
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101", "Twitter", "social", "profile", "", ""],
|
||||
"metrics" => [%{"values" => ["1", "1", "1", "100"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "Gmail", "email", "august", "", ""],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "100"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "Gmail", "email", "(not set)", "", ""],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "100"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_sources"
|
||||
)
|
||||
|
||||
conn =
|
||||
get(
|
||||
conn,
|
||||
"/api/stats/#{site.domain}/utm_campaigns?period=day&date=2021-01-01&with_imported=true"
|
||||
)
|
||||
|
||||
assert json_response(conn, 200) == [
|
||||
%{
|
||||
"name" => "august",
|
||||
"visitors" => 2,
|
||||
"bounce_rate" => 50.0,
|
||||
"visit_duration" => 50.0
|
||||
},
|
||||
%{
|
||||
"name" => "profile",
|
||||
"visitors" => 2,
|
||||
"bounce_rate" => 100.0,
|
||||
"visit_duration" => 50.0
|
||||
}
|
||||
]
|
||||
end
|
||||
|
||||
test "UTM terms data imported from Google Analytics", %{conn: conn, site: site} do
|
||||
populate_stats(site, [
|
||||
build(:pageview, utm_term: "oat milk", timestamp: ~N[2021-01-01 00:00:00]),
|
||||
build(:pageview, utm_term: "Sweden", timestamp: ~N[2021-01-01 00:00:00])
|
||||
])
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101", "Google", "paid", "", "", "oat milk"],
|
||||
"metrics" => [%{"values" => ["1", "1", "1", "100"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "Google", "paid", "", "", "Sweden"],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "100"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "Google", "paid", "", "", "(not set)"],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "100"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_sources"
|
||||
)
|
||||
|
||||
conn =
|
||||
get(
|
||||
conn,
|
||||
"/api/stats/#{site.domain}/utm_terms?period=day&date=2021-01-01&with_imported=true"
|
||||
)
|
||||
|
||||
assert json_response(conn, 200) == [
|
||||
%{
|
||||
"name" => "Sweden",
|
||||
"visitors" => 2,
|
||||
"bounce_rate" => 50.0,
|
||||
"visit_duration" => 50.0
|
||||
},
|
||||
%{
|
||||
"name" => "oat milk",
|
||||
"visitors" => 2,
|
||||
"bounce_rate" => 100.0,
|
||||
"visit_duration" => 50.0
|
||||
}
|
||||
]
|
||||
end
|
||||
|
||||
test "UTM contents data imported from Google Analytics", %{conn: conn, site: site} do
|
||||
populate_stats(site, [
|
||||
build(:pageview, utm_content: "ad", timestamp: ~N[2021-01-01 00:00:00]),
|
||||
build(:pageview, utm_content: "blog", timestamp: ~N[2021-01-01 00:00:00])
|
||||
])
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101", "Google", "paid", "", "ad", ""],
|
||||
"metrics" => [%{"values" => ["1", "1", "1", "100"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "Google", "paid", "", "blog", ""],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "100"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "Google", "paid", "", "(not set)", ""],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "100"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_sources"
|
||||
)
|
||||
|
||||
conn =
|
||||
get(
|
||||
conn,
|
||||
"/api/stats/#{site.domain}/utm_contents?period=day&date=2021-01-01&with_imported=true"
|
||||
)
|
||||
|
||||
assert json_response(conn, 200) == [
|
||||
%{
|
||||
"name" => "blog",
|
||||
"visitors" => 2,
|
||||
"bounce_rate" => 50.0,
|
||||
"visit_duration" => 50.0
|
||||
},
|
||||
%{
|
||||
"name" => "ad",
|
||||
"visitors" => 2,
|
||||
"bounce_rate" => 100.0,
|
||||
"visit_duration" => 50.0
|
||||
}
|
||||
]
|
||||
end
|
||||
|
||||
test "Page event data imported from Google Analytics", %{conn: conn, site: site} do
|
||||
populate_stats(site, [
|
||||
build(:pageview,
|
||||
pathname: "/",
|
||||
hostname: "host-a.com",
|
||||
user_id: @user_id,
|
||||
timestamp: ~N[2021-01-01 00:00:00]
|
||||
),
|
||||
build(:pageview,
|
||||
pathname: "/some-other-page",
|
||||
hostname: "host-a.com",
|
||||
user_id: @user_id,
|
||||
timestamp: ~N[2021-01-01 00:15:00]
|
||||
)
|
||||
])
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101", "host-a.com", "/"],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "700"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "host-b.com", "/some-other-page"],
|
||||
"metrics" => [%{"values" => ["1", "2", "1", "60"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "host-b.com", "/some-other-page?wat=wot"],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "60"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_pages"
|
||||
)
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101", "/"],
|
||||
"metrics" => [%{"values" => ["1", "3", "10", "1"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_entry_pages"
|
||||
)
|
||||
|
||||
conn =
|
||||
get(
|
||||
conn,
|
||||
"/api/stats/#{site.domain}/pages?period=day&date=2021-01-01&detailed=true&with_imported=true"
|
||||
)
|
||||
|
||||
assert json_response(conn, 200) == [
|
||||
%{
|
||||
"bounce_rate" => nil,
|
||||
"time_on_page" => 60,
|
||||
"visitors" => 3,
|
||||
"pageviews" => 4,
|
||||
"name" => "/some-other-page"
|
||||
},
|
||||
%{
|
||||
"bounce_rate" => 25.0,
|
||||
"time_on_page" => 800.0,
|
||||
"visitors" => 2,
|
||||
"pageviews" => 2,
|
||||
"name" => "/"
|
||||
}
|
||||
]
|
||||
end
|
||||
|
||||
test "Exit page event data imported from Google Analytics", %{conn: conn, site: site} do
|
||||
populate_stats(site, [
|
||||
build(:pageview,
|
||||
pathname: "/page1",
|
||||
timestamp: ~N[2021-01-01 00:00:00]
|
||||
),
|
||||
build(:pageview,
|
||||
pathname: "/page1",
|
||||
timestamp: ~N[2021-01-01 00:00:00]
|
||||
),
|
||||
build(:pageview,
|
||||
pathname: "/page1",
|
||||
user_id: @user_id,
|
||||
timestamp: ~N[2021-01-01 00:00:00]
|
||||
),
|
||||
build(:pageview,
|
||||
pathname: "/page2",
|
||||
user_id: @user_id,
|
||||
timestamp: ~N[2021-01-01 00:15:00]
|
||||
)
|
||||
])
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101", "host-a.com", "/page2"],
|
||||
"metrics" => [%{"values" => ["2", "4", "0", "10"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_pages"
|
||||
)
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101", "/page2"],
|
||||
"metrics" => [%{"values" => ["2", "3"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_exit_pages"
|
||||
)
|
||||
|
||||
conn =
|
||||
get(
|
||||
conn,
|
||||
"/api/stats/#{site.domain}/exit-pages?period=day&date=2021-01-01&with_imported=true"
|
||||
)
|
||||
|
||||
assert json_response(conn, 200) == [
|
||||
%{
|
||||
"name" => "/page2",
|
||||
"unique_exits" => 3,
|
||||
"total_exits" => 4,
|
||||
"exit_rate" => 80.0
|
||||
},
|
||||
%{"name" => "/page1", "unique_exits" => 2, "total_exits" => 2, "exit_rate" => 66}
|
||||
]
|
||||
end
|
||||
|
||||
test "Location data imported from Google Analytics", %{conn: conn, site: site} do
|
||||
populate_stats(site, [
|
||||
build(:pageview,
|
||||
country_code: "EE",
|
||||
timestamp: ~N[2021-01-01 00:15:00]
|
||||
),
|
||||
build(:pageview,
|
||||
country_code: "EE",
|
||||
timestamp: ~N[2021-01-01 00:15:00]
|
||||
),
|
||||
build(:pageview,
|
||||
country_code: "GB",
|
||||
timestamp: ~N[2021-01-01 00:15:00]
|
||||
)
|
||||
])
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101", "EE", "Tartumaa"],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "10"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "GB", "Midlothian"],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "10"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_locations"
|
||||
)
|
||||
|
||||
conn =
|
||||
get(
|
||||
conn,
|
||||
"/api/stats/#{site.domain}/countries?period=day&date=2021-01-01&with_imported=true"
|
||||
)
|
||||
|
||||
assert json_response(conn, 200) == [
|
||||
%{
|
||||
"code" => "EE",
|
||||
"alpha_3" => "EST",
|
||||
"name" => "Estonia",
|
||||
"flag" => "🇪🇪",
|
||||
"visitors" => 3,
|
||||
"percentage" => 60
|
||||
},
|
||||
%{
|
||||
"code" => "GB",
|
||||
"alpha_3" => "GBR",
|
||||
"name" => "United Kingdom",
|
||||
"flag" => "🇬🇧",
|
||||
"visitors" => 2,
|
||||
"percentage" => 40
|
||||
}
|
||||
]
|
||||
end
|
||||
|
||||
test "Devices data imported from Google Analytics", %{conn: conn, site: site} do
|
||||
populate_stats(site, [
|
||||
build(:pageview, screen_size: "Desktop", timestamp: ~N[2021-01-01 00:15:00]),
|
||||
build(:pageview, screen_size: "Desktop", timestamp: ~N[2021-01-01 00:15:00]),
|
||||
build(:pageview, screen_size: "Laptop", timestamp: ~N[2021-01-01 00:15:00])
|
||||
])
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101", "mobile"],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "10"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "Laptop"],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "10"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_devices"
|
||||
)
|
||||
|
||||
conn =
|
||||
get(
|
||||
conn,
|
||||
"/api/stats/#{site.domain}/screen-sizes?period=day&date=2021-01-01&with_imported=true"
|
||||
)
|
||||
|
||||
assert json_response(conn, 200) == [
|
||||
%{"name" => "Desktop", "visitors" => 2, "percentage" => 40},
|
||||
%{"name" => "Laptop", "visitors" => 2, "percentage" => 40},
|
||||
%{"name" => "Mobile", "visitors" => 1, "percentage" => 20}
|
||||
]
|
||||
end
|
||||
|
||||
test "Browsers data imported from Google Analytics", %{conn: conn, site: site} do
|
||||
populate_stats(site, [
|
||||
build(:pageview, browser: "Chrome", timestamp: ~N[2021-01-01 00:15:00]),
|
||||
build(:pageview, browser: "Firefox", timestamp: ~N[2021-01-01 00:15:00])
|
||||
])
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101", "User-Agent: Mozilla"],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "10"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "Android Browser"],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "10"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_browsers"
|
||||
)
|
||||
|
||||
conn =
|
||||
get(
|
||||
conn,
|
||||
"/api/stats/#{site.domain}/browsers?period=day&date=2021-01-01&with_imported=true"
|
||||
)
|
||||
|
||||
assert json_response(conn, 200) == [
|
||||
%{"name" => "Firefox", "visitors" => 2, "percentage" => 50},
|
||||
%{"name" => "Mobile App", "visitors" => 1, "percentage" => 25},
|
||||
%{"name" => "Chrome", "visitors" => 1, "percentage" => 25}
|
||||
]
|
||||
end
|
||||
|
||||
test "OS data imported from Google Analytics", %{conn: conn, site: site} do
|
||||
populate_stats(site, [
|
||||
build(:pageview, operating_system: "Mac", timestamp: ~N[2021-01-01 00:15:00]),
|
||||
build(:pageview, operating_system: "Mac", timestamp: ~N[2021-01-01 00:15:00]),
|
||||
build(:pageview, operating_system: "GNU/Linux", timestamp: ~N[2021-01-01 00:15:00])
|
||||
])
|
||||
|
||||
assert :ok =
|
||||
Plausible.Imported.from_google_analytics(
|
||||
[
|
||||
%{
|
||||
"dimensions" => ["20210101", "Macintosh"],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "10"]}]
|
||||
},
|
||||
%{
|
||||
"dimensions" => ["20210101", "Linux"],
|
||||
"metrics" => [%{"values" => ["1", "1", "0", "10"]}]
|
||||
}
|
||||
],
|
||||
site.id,
|
||||
"imported_operating_systems"
|
||||
)
|
||||
|
||||
conn =
|
||||
get(
|
||||
conn,
|
||||
"/api/stats/#{site.domain}/operating-systems?period=day&date=2021-01-01&with_imported=true"
|
||||
)
|
||||
|
||||
assert json_response(conn, 200) == [
|
||||
%{"name" => "Mac", "visitors" => 3, "percentage" => 60},
|
||||
%{"name" => "GNU/Linux", "visitors" => 2, "percentage" => 40}
|
||||
]
|
||||
end
|
||||
end
|
||||
end
|
||||
@ -0,0 +1,86 @@
|
||||
defmodule Plausible.Workers.ImportGoogleAnalyticsTest do
|
||||
use Plausible.DataCase
|
||||
use Bamboo.Test
|
||||
import Double
|
||||
alias Plausible.Workers.ImportGoogleAnalytics
|
||||
|
||||
@imported_data %Plausible.Site.ImportedData{
|
||||
end_date: Timex.today(),
|
||||
source: "Google Analytics",
|
||||
status: "importing"
|
||||
}
|
||||
|
||||
test "updates the imported_data field for the site after succesful import" do
|
||||
user = insert(:user, trial_expiry_date: Timex.today() |> Timex.shift(days: 1))
|
||||
site = insert(:site, members: [user], imported_data: @imported_data)
|
||||
|
||||
api_stub =
|
||||
stub(Plausible.Google.Api, :import_analytics, fn _site, _profile ->
|
||||
{:ok, nil}
|
||||
end)
|
||||
|
||||
ImportGoogleAnalytics.perform(
|
||||
%Oban.Job{args: %{"site_id" => site.id, "profile" => "profile"}},
|
||||
api_stub
|
||||
)
|
||||
|
||||
assert Repo.reload!(site).imported_data.status == "ok"
|
||||
end
|
||||
|
||||
test "sends email to owner after succesful import" do
|
||||
user = insert(:user, trial_expiry_date: Timex.today() |> Timex.shift(days: 1))
|
||||
site = insert(:site, members: [user], imported_data: @imported_data)
|
||||
|
||||
api_stub =
|
||||
stub(Plausible.Google.Api, :import_analytics, fn _site, _profile ->
|
||||
{:ok, nil}
|
||||
end)
|
||||
|
||||
ImportGoogleAnalytics.perform(
|
||||
%Oban.Job{args: %{"site_id" => site.id, "profile" => "profile"}},
|
||||
api_stub
|
||||
)
|
||||
|
||||
assert_email_delivered_with(
|
||||
to: [user],
|
||||
subject: "Google Analytics data imported for #{site.domain}"
|
||||
)
|
||||
end
|
||||
|
||||
test "updates site record after failed import" do
|
||||
user = insert(:user, trial_expiry_date: Timex.today() |> Timex.shift(days: 1))
|
||||
site = insert(:site, members: [user], imported_data: @imported_data)
|
||||
|
||||
api_stub =
|
||||
stub(Plausible.Google.Api, :import_analytics, fn _site, _profile ->
|
||||
{:error, "Something went wrong"}
|
||||
end)
|
||||
|
||||
ImportGoogleAnalytics.perform(
|
||||
%Oban.Job{args: %{"site_id" => site.id, "profile" => "profile"}},
|
||||
api_stub
|
||||
)
|
||||
|
||||
assert Repo.reload!(site).imported_data.status == "error"
|
||||
end
|
||||
|
||||
test "sends email to owner after failed import" do
|
||||
user = insert(:user, trial_expiry_date: Timex.today() |> Timex.shift(days: 1))
|
||||
site = insert(:site, members: [user], imported_data: @imported_data)
|
||||
|
||||
api_stub =
|
||||
stub(Plausible.Google.Api, :import_analytics, fn _site, _profile ->
|
||||
{:error, "Something went wrong"}
|
||||
end)
|
||||
|
||||
ImportGoogleAnalytics.perform(
|
||||
%Oban.Job{args: %{"site_id" => site.id, "profile" => "profile"}},
|
||||
api_stub
|
||||
)
|
||||
|
||||
assert_email_delivered_with(
|
||||
to: [user],
|
||||
subject: "Google Analytics import failed for #{site.domain}"
|
||||
)
|
||||
end
|
||||
end
|
||||
Loading…
Reference in New Issue