You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
analytics/lib/plausible/imported.ex

127 lines
3.5 KiB
Elixir

Add multiple imports per site (#3724) * Clean up references to no longer active `google_analytics_imports` Oban queue * Stub CSV importer * Add SiteImport schema * Rename `Plausible.Imported` module file to match module name * Add `import_id` column to `Imported.*` CH schemas * Implement Importer behavior and manage imports state using new entities * Implement importer callbacks and maintain site.imported_data for UA * Keep imports in sync when forgetting all imports * Scope imported data queries to completed import IDs * Mark newly imported data with respective import ID * Clean up Importer implementation a bit * Test querying legacy and new imported data * Send Oban notifications on import worker failure too * Fix checking for forgettable imports and remove redundant function * Fix UA integration test * Change site import source to atom enum and add source label * Add typespecs and reduce repetition in `Plausible.Imported` * Improve documentation and typespecs * Add test for purging particular import * Switch email notification templates depending on import source * Document running import synchronously * Fix UA importer args parsing and ensure it's covered by tests * Clear `site.stats_start_date` on complete import to force recalculation * Test Oban notifications (h/t @ruslandoga) * Purge stats on import failure right away to reduce a chance of leaving debris behind * Fix typos Co-authored-by: hq1 <hq@mtod.org> * Fix another typo * Refactor fetching earliest import and earliest stats start date * Use `Date.after?` instead of `Timex.after?` * Cache import data in site virtual fields and limit queried imports to 5 * Ensure always current `stats_start_date` is used * Work around broken typespec in Timex * Make `SiteController.forget_imported` action idempotent * Discard irrecoverably failed import tasks * Use macros for site import statuses There's also a fix ensuring only complete imports are considered where relevant - couldn't isolate it as it was in a common hunk * Use `import_id` as worker job uniqueness criterion * Do not load imported stats data in plugins API context --------- Co-authored-by: hq1 <hq@mtod.org>
3 years ago
defmodule Plausible.Imported do
@moduledoc """
Context for managing site statistics imports.
Currently following importers are implemented:
* `Plausible.Imported.UniversalAnalytics` - existing mechanism, for legacy Google
analytics formerly known as "Google Analytics"
* `Plausible.Imported.NoopImporter` - importer stub, used mainly for testing purposes
* `Plausible.Imported.CSVImporter` - CSV importer from S3
Add multiple imports per site (#3724) * Clean up references to no longer active `google_analytics_imports` Oban queue * Stub CSV importer * Add SiteImport schema * Rename `Plausible.Imported` module file to match module name * Add `import_id` column to `Imported.*` CH schemas * Implement Importer behavior and manage imports state using new entities * Implement importer callbacks and maintain site.imported_data for UA * Keep imports in sync when forgetting all imports * Scope imported data queries to completed import IDs * Mark newly imported data with respective import ID * Clean up Importer implementation a bit * Test querying legacy and new imported data * Send Oban notifications on import worker failure too * Fix checking for forgettable imports and remove redundant function * Fix UA integration test * Change site import source to atom enum and add source label * Add typespecs and reduce repetition in `Plausible.Imported` * Improve documentation and typespecs * Add test for purging particular import * Switch email notification templates depending on import source * Document running import synchronously * Fix UA importer args parsing and ensure it's covered by tests * Clear `site.stats_start_date` on complete import to force recalculation * Test Oban notifications (h/t @ruslandoga) * Purge stats on import failure right away to reduce a chance of leaving debris behind * Fix typos Co-authored-by: hq1 <hq@mtod.org> * Fix another typo * Refactor fetching earliest import and earliest stats start date * Use `Date.after?` instead of `Timex.after?` * Cache import data in site virtual fields and limit queried imports to 5 * Ensure always current `stats_start_date` is used * Work around broken typespec in Timex * Make `SiteController.forget_imported` action idempotent * Discard irrecoverably failed import tasks * Use macros for site import statuses There's also a fix ensuring only complete imports are considered where relevant - couldn't isolate it as it was in a common hunk * Use `import_id` as worker job uniqueness criterion * Do not load imported stats data in plugins API context --------- Co-authored-by: hq1 <hq@mtod.org>
3 years ago
For more information on implementing importers, see `Plausible.Imported.Importer`.
"""
import Ecto.Query
alias Plausible.Imported
alias Plausible.Imported.SiteImport
alias Plausible.Repo
alias Plausible.Site
require Plausible.Imported.SiteImport
@tables [
Imported.Visitor,
Imported.Source,
Imported.Page,
Imported.EntryPage,
Imported.ExitPage,
Imported.Location,
Imported.Device,
Imported.Browser,
Imported.OperatingSystem
]
@table_names Enum.map(@tables, & &1.__schema__(:source))
# Maximum number of complete imports to account for when querying stats
@max_complete_imports 5
@spec tables() :: [String.t()]
def tables, do: @table_names
Implement UI for multiple imports (#3727) * Create a stub of site settings section for imports and exports * Use legacy site import indication to determine UA import handling * Add provisional logos for upcoming import sources * Stub basics of import page * Add very rudimentary support for multiple UA imports * Implement imports list as live view * Add support for opening LV modal from backend and closing from frontend * Introduce notion of themes to `button` and `button_link` components * Add confirmation modal on deleting import * Swap GA4 logo * Implement disabled state support for `button_link` component * Disable export and non-implemented import sources * Use native starts start date for upper boundary of import time range * Ensure integrations view uses legacy UA import flow * Remove unnecessary preload in SiteController * Remove unnecessary exception for legacy imports * Move API controller stats tests under PlausibleWeb * Test listing imports * Add test for explicit listener setup * Add tests for legacy flag state in UA importer * Add test for purging legacy import data * Add tests for `Sites.native_stats_start_date` * Test forgetting imports * Add `Stats.Clickhouse.imported_pageview_counts/1` and fix test flakiness * Show page view counts on imports list * Add tests for static imports and exports view * Adjust button look slightly * Use `case` instead of `cond` * Make feature flag customisable per site * Fix buttons and empty state styling * Add another import to seeds * Use JS confirm dialog instead of modal for deletion confirmations * Revert "Add support for opening LV modal from backend and closing from frontend" This reverts commit 260e6c753032b451542e24be9edc2118790b5a00. * Default `legacy` to false when inserting new import jobs * Drop `method` attribute from `button_link` and `unstyled_link` components
3 years ago
@spec max_complete_imports() :: non_neg_integer()
def max_complete_imports(), do: @max_complete_imports
Add multiple imports per site (#3724) * Clean up references to no longer active `google_analytics_imports` Oban queue * Stub CSV importer * Add SiteImport schema * Rename `Plausible.Imported` module file to match module name * Add `import_id` column to `Imported.*` CH schemas * Implement Importer behavior and manage imports state using new entities * Implement importer callbacks and maintain site.imported_data for UA * Keep imports in sync when forgetting all imports * Scope imported data queries to completed import IDs * Mark newly imported data with respective import ID * Clean up Importer implementation a bit * Test querying legacy and new imported data * Send Oban notifications on import worker failure too * Fix checking for forgettable imports and remove redundant function * Fix UA integration test * Change site import source to atom enum and add source label * Add typespecs and reduce repetition in `Plausible.Imported` * Improve documentation and typespecs * Add test for purging particular import * Switch email notification templates depending on import source * Document running import synchronously * Fix UA importer args parsing and ensure it's covered by tests * Clear `site.stats_start_date` on complete import to force recalculation * Test Oban notifications (h/t @ruslandoga) * Purge stats on import failure right away to reduce a chance of leaving debris behind * Fix typos Co-authored-by: hq1 <hq@mtod.org> * Fix another typo * Refactor fetching earliest import and earliest stats start date * Use `Date.after?` instead of `Timex.after?` * Cache import data in site virtual fields and limit queried imports to 5 * Ensure always current `stats_start_date` is used * Work around broken typespec in Timex * Make `SiteController.forget_imported` action idempotent * Discard irrecoverably failed import tasks * Use macros for site import statuses There's also a fix ensuring only complete imports are considered where relevant - couldn't isolate it as it was in a common hunk * Use `import_id` as worker job uniqueness criterion * Do not load imported stats data in plugins API context --------- Co-authored-by: hq1 <hq@mtod.org>
3 years ago
@spec load_import_data(Site.t()) :: Site.t()
def load_import_data(%{import_data_loaded: true} = site), do: site
def load_import_data(site) do
complete_import_ids = list_complete_import_ids(site)
earliest_import = get_earliest_import(site) || %{}
%{
site
| import_data_loaded: true,
earliest_import_start_date: Map.get(earliest_import, :start_date),
earliest_import_end_date: Map.get(earliest_import, :end_date),
complete_import_ids: complete_import_ids
}
end
Implement UI for multiple imports (#3727) * Create a stub of site settings section for imports and exports * Use legacy site import indication to determine UA import handling * Add provisional logos for upcoming import sources * Stub basics of import page * Add very rudimentary support for multiple UA imports * Implement imports list as live view * Add support for opening LV modal from backend and closing from frontend * Introduce notion of themes to `button` and `button_link` components * Add confirmation modal on deleting import * Swap GA4 logo * Implement disabled state support for `button_link` component * Disable export and non-implemented import sources * Use native starts start date for upper boundary of import time range * Ensure integrations view uses legacy UA import flow * Remove unnecessary preload in SiteController * Remove unnecessary exception for legacy imports * Move API controller stats tests under PlausibleWeb * Test listing imports * Add test for explicit listener setup * Add tests for legacy flag state in UA importer * Add test for purging legacy import data * Add tests for `Sites.native_stats_start_date` * Test forgetting imports * Add `Stats.Clickhouse.imported_pageview_counts/1` and fix test flakiness * Show page view counts on imports list * Add tests for static imports and exports view * Adjust button look slightly * Use `case` instead of `cond` * Make feature flag customisable per site * Fix buttons and empty state styling * Add another import to seeds * Use JS confirm dialog instead of modal for deletion confirmations * Revert "Add support for opening LV modal from backend and closing from frontend" This reverts commit 260e6c753032b451542e24be9edc2118790b5a00. * Default `legacy` to false when inserting new import jobs * Drop `method` attribute from `button_link` and `unstyled_link` components
3 years ago
@spec get_import(non_neg_integer()) :: SiteImport.t() | nil
def get_import(import_id) do
Repo.get(SiteImport, import_id)
end
defdelegate listen(), to: Imported.Importer
Add multiple imports per site (#3724) * Clean up references to no longer active `google_analytics_imports` Oban queue * Stub CSV importer * Add SiteImport schema * Rename `Plausible.Imported` module file to match module name * Add `import_id` column to `Imported.*` CH schemas * Implement Importer behavior and manage imports state using new entities * Implement importer callbacks and maintain site.imported_data for UA * Keep imports in sync when forgetting all imports * Scope imported data queries to completed import IDs * Mark newly imported data with respective import ID * Clean up Importer implementation a bit * Test querying legacy and new imported data * Send Oban notifications on import worker failure too * Fix checking for forgettable imports and remove redundant function * Fix UA integration test * Change site import source to atom enum and add source label * Add typespecs and reduce repetition in `Plausible.Imported` * Improve documentation and typespecs * Add test for purging particular import * Switch email notification templates depending on import source * Document running import synchronously * Fix UA importer args parsing and ensure it's covered by tests * Clear `site.stats_start_date` on complete import to force recalculation * Test Oban notifications (h/t @ruslandoga) * Purge stats on import failure right away to reduce a chance of leaving debris behind * Fix typos Co-authored-by: hq1 <hq@mtod.org> * Fix another typo * Refactor fetching earliest import and earliest stats start date * Use `Date.after?` instead of `Timex.after?` * Cache import data in site virtual fields and limit queried imports to 5 * Ensure always current `stats_start_date` is used * Work around broken typespec in Timex * Make `SiteController.forget_imported` action idempotent * Discard irrecoverably failed import tasks * Use macros for site import statuses There's also a fix ensuring only complete imports are considered where relevant - couldn't isolate it as it was in a common hunk * Use `import_id` as worker job uniqueness criterion * Do not load imported stats data in plugins API context --------- Co-authored-by: hq1 <hq@mtod.org>
3 years ago
@spec list_all_imports(Site.t()) :: [SiteImport.t()]
def list_all_imports(site) do
Implement UI for multiple imports (#3727) * Create a stub of site settings section for imports and exports * Use legacy site import indication to determine UA import handling * Add provisional logos for upcoming import sources * Stub basics of import page * Add very rudimentary support for multiple UA imports * Implement imports list as live view * Add support for opening LV modal from backend and closing from frontend * Introduce notion of themes to `button` and `button_link` components * Add confirmation modal on deleting import * Swap GA4 logo * Implement disabled state support for `button_link` component * Disable export and non-implemented import sources * Use native starts start date for upper boundary of import time range * Ensure integrations view uses legacy UA import flow * Remove unnecessary preload in SiteController * Remove unnecessary exception for legacy imports * Move API controller stats tests under PlausibleWeb * Test listing imports * Add test for explicit listener setup * Add tests for legacy flag state in UA importer * Add test for purging legacy import data * Add tests for `Sites.native_stats_start_date` * Test forgetting imports * Add `Stats.Clickhouse.imported_pageview_counts/1` and fix test flakiness * Show page view counts on imports list * Add tests for static imports and exports view * Adjust button look slightly * Use `case` instead of `cond` * Make feature flag customisable per site * Fix buttons and empty state styling * Add another import to seeds * Use JS confirm dialog instead of modal for deletion confirmations * Revert "Add support for opening LV modal from backend and closing from frontend" This reverts commit 260e6c753032b451542e24be9edc2118790b5a00. * Default `legacy` to false when inserting new import jobs * Drop `method` attribute from `button_link` and `unstyled_link` components
3 years ago
imports =
from(i in SiteImport, where: i.site_id == ^site.id, order_by: [desc: i.inserted_at])
|> Repo.all()
if site.imported_data && not Enum.any?(imports, & &1.legacy) do
imports ++ [SiteImport.from_legacy(site.imported_data)]
else
imports
end
Add multiple imports per site (#3724) * Clean up references to no longer active `google_analytics_imports` Oban queue * Stub CSV importer * Add SiteImport schema * Rename `Plausible.Imported` module file to match module name * Add `import_id` column to `Imported.*` CH schemas * Implement Importer behavior and manage imports state using new entities * Implement importer callbacks and maintain site.imported_data for UA * Keep imports in sync when forgetting all imports * Scope imported data queries to completed import IDs * Mark newly imported data with respective import ID * Clean up Importer implementation a bit * Test querying legacy and new imported data * Send Oban notifications on import worker failure too * Fix checking for forgettable imports and remove redundant function * Fix UA integration test * Change site import source to atom enum and add source label * Add typespecs and reduce repetition in `Plausible.Imported` * Improve documentation and typespecs * Add test for purging particular import * Switch email notification templates depending on import source * Document running import synchronously * Fix UA importer args parsing and ensure it's covered by tests * Clear `site.stats_start_date` on complete import to force recalculation * Test Oban notifications (h/t @ruslandoga) * Purge stats on import failure right away to reduce a chance of leaving debris behind * Fix typos Co-authored-by: hq1 <hq@mtod.org> * Fix another typo * Refactor fetching earliest import and earliest stats start date * Use `Date.after?` instead of `Timex.after?` * Cache import data in site virtual fields and limit queried imports to 5 * Ensure always current `stats_start_date` is used * Work around broken typespec in Timex * Make `SiteController.forget_imported` action idempotent * Discard irrecoverably failed import tasks * Use macros for site import statuses There's also a fix ensuring only complete imports are considered where relevant - couldn't isolate it as it was in a common hunk * Use `import_id` as worker job uniqueness criterion * Do not load imported stats data in plugins API context --------- Co-authored-by: hq1 <hq@mtod.org>
3 years ago
end
@spec list_complete_import_ids(Site.t()) :: [non_neg_integer()]
def list_complete_import_ids(site) do
ids =
from(i in SiteImport,
where: i.site_id == ^site.id and i.status == ^SiteImport.completed(),
select: i.id,
limit: @max_complete_imports
)
|> Repo.all()
# account for legacy imports as well
if site.imported_data && site.imported_data.status == "ok" do
[0 | ids]
else
ids
end
end
@spec get_earliest_import(Site.t()) :: SiteImport.t() | nil
def get_earliest_import(site) do
first_import =
from(i in SiteImport,
where: i.site_id == ^site.id and i.status == ^SiteImport.completed(),
order_by: i.start_date,
limit: 1
)
|> Repo.one()
legacy_import =
if site.imported_data && site.imported_data.status == "ok" do
site.imported_data
end
[legacy_import, first_import]
|> Enum.reject(&is_nil/1)
|> Enum.min_by(& &1.start_date, Date, fn -> nil end)
end
@spec delete_imports_for_site(Site.t()) :: :ok
def delete_imports_for_site(site) do
Repo.delete_all(from i in SiteImport, where: i.site_id == ^site.id)
:ok
end
end