You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
analytics/lib/plausible
Adrian Gruntkowski f8b4d5066a
Add multiple imports per site (#3724)
* Clean up references to no longer active `google_analytics_imports` Oban queue

* Stub CSV importer

* Add SiteImport schema

* Rename `Plausible.Imported` module file to match module name

* Add `import_id` column to `Imported.*` CH schemas

* Implement Importer behavior and manage imports state using new entities

* Implement importer callbacks and maintain site.imported_data for UA

* Keep imports in sync when forgetting all imports

* Scope imported data queries to completed import IDs

* Mark newly imported data with respective import ID

* Clean up Importer implementation a bit

* Test querying legacy and new imported data

* Send Oban notifications on import worker failure too

* Fix checking for forgettable imports and remove redundant function

* Fix UA integration test

* Change site import source to atom enum and add source label

* Add typespecs and reduce repetition in `Plausible.Imported`

* Improve documentation and typespecs

* Add test for purging particular import

* Switch email notification templates depending on import source

* Document running import synchronously

* Fix UA importer args parsing and ensure it's covered by tests

* Clear `site.stats_start_date` on complete import to force recalculation

* Test Oban notifications (h/t @ruslandoga)

* Purge stats on import failure right away to reduce a chance of leaving debris behind

* Fix typos

Co-authored-by: hq1 <hq@mtod.org>

* Fix another typo

* Refactor fetching earliest import and earliest stats start date

* Use `Date.after?` instead of `Timex.after?`

* Cache import data in site virtual fields and limit queried imports to 5

* Ensure always current `stats_start_date` is used

* Work around broken typespec in Timex

* Make `SiteController.forget_imported` action idempotent

* Discard irrecoverably failed import tasks

* Use macros for site import statuses

There's also a fix ensuring only complete imports are considered
where relevant - couldn't isolate it as it was in a common hunk

* Use `import_id` as worker job uniqueness criterion

* Do not load imported stats data in plugins API context

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
..
auth Bugfix: available features for expired trials, no subscriptions (#3740) 3 years ago
billing Bugfix: available features for expired trials, no subscriptions (#3740) 3 years ago
cache Generalize Cache/Warmers - extract only specifics to `Site.Cache` (#3716) 3 years ago
data_migration Update credo config + extras (#3547) 3 years ago
event Flush Clickhouse buffers faster by encoding early (#3623) 3 years ago
funnel Add small build option (#3536) 3 years ago
goal Small build updates (#3546) 3 years ago
google Extract Universal Analytics import logic (#3700) 3 years ago
helpers Small build updates (#3546) 3 years ago
imported Add multiple imports per site (#3724) 3 years ago
ingestion IP Block List (#3761) 3 years ago
open_telemetry Make OpenTelemetry sampler ratio configurable via env (#3514) 3 years ago
plugins/api Plugins API exentsions (custom props, bulk goal delete, goal creation => ListResponse always) (#3593) 3 years ago
sentry Upgrade Erlang/Elixir stack (#3454) 3 years ago
session Flush Clickhouse buffers faster by encoding early (#3623) 3 years ago
shield IP Block List (#3761) 3 years ago
site Add multiple imports per site (#3724) 3 years ago
stats Add multiple imports per site (#3724) 3 years ago
telemetry Gatekeep ingestion pipeline (#2472) 4 years ago
application.ex IP Block List (#3761) 3 years ago
async_insert_repo.ex Switch to new clickhouse adapter (ch/chto) (#2733) 4 years ago
cache.ex IP Block List (#3761) 3 years ago
cldr.ex Revenue tracking: Add currency field to goal creation (#2948) 4 years ago
clickhouse_event_v2.ex Protect against custom props payloads breaking event insertions (#3617) 3 years ago
clickhouse_repo.ex Switch to new clickhouse adapter (ch/chto) (#2733) 4 years ago
clickhouse_session_v2.ex use ecto_ch (#2890) 4 years ago
data_migration.ex Make self-hosted data migration easier (#2865) 4 years ago
debug_replay_info.ex Add debug_replay_info context to Sentry Stats reports (#3189) 3 years ago
geo.ex Upgrade Erlang/Elixir stack (#3454) 3 years ago
goals.ex Small build updates (#3546) 3 years ago
http_client.ex Use Phoenix LiveView for the upgrade page (#3382) 3 years ago
import_deletion_repo.ex Split clickhouse repos - making the main one read only (#2826) 4 years ago
imported.ex Add multiple imports per site (#3724) 3 years ago
ingest_repo.ex Switch to new clickhouse adapter (ch/chto) (#2733) 4 years ago
mailer.ex Fix and refactor invitation logic (#3376) 3 years ago
migration_utils.ex Add migration for minmax indexes on session timestamps (#3772) 3 years ago
open_telemetry.ex Implement trace ID ratio-based sampling for OpenTelemetry (#3499) 3 years ago
pagination.ex Implement new sites view (#3463) 3 years ago
prom_ex.ex Implement new sites view (#3463) 3 years ago
props.ex Limit Custom Properties access in Stats API (#3670) 3 years ago
purge.ex Add multiple imports per site (#3724) 3 years ago
rate_limit.ex rm Hammer (#3571) 3 years ago
repo.ex Implement new sites view (#3463) 3 years ago
shields.ex IP Block List (#3761) 3 years ago
site.ex Add multiple imports per site (#3724) 3 years ago
sites.ex Add multiple imports per site (#3724) 3 years ago
stats.ex Add small build option (#3536) 3 years ago
themes.ex Adds dark mode to entire dashboard (#467) 6 years ago
timezones.ex Add multiple imports per site (#3724) 3 years ago
users.ex Bugfix: available features for expired trials, no subscriptions (#3740) 3 years ago