Commit Graph

12 Commits (gzip)

Author SHA1 Message Date
Karl-Aksel Puulmann 26d41ddbb9
Mutation to populate event session columns (#3844)
* WIP mutation to populate event session columns

* Remove duplication

* report errors, allow_nondeterministic_updates

* use right columns

* Update existing columns instead of session_* ones

* Make dialyzer happy

* Fix issue with passing pre-existing params in

* Logger -> IO.puts

* Use IngestRepo.config for connection settings

* Make dictionary options configurable

* Move allow_nondeterministic_mutations to within the migration

* Solve credo warning about too deep nesting

* Missed logger call

* Pattern matching in function head
3 years ago
Karl-Aksel Puulmann 8ba2b934b3
Add data migration for moving sessions_v2 table to VersionedCollapsingMergeTree (#3802)
* Add data migration for moving to VersionedCollapsingMergeTree

This has been tested locally and partially on staging. Still requires a bit of work to verify.

Verification query:

```
SELECT main._partition_id, tmp.count, main.count
FROM (
SELECT _partition_id, count() AS count
FROM sessions_v2_tmp_versioned
GROUP BY _partition_id
) AS tmp
FULL OUTER JOIN (
SELECT _partition_id, count() AS count
FROM sessions_v2
GROUP BY _partition_id
) AS main
ON (tmp._partition_id == main._partition_id)
ORDER BY main._partition_id
```

* Add an early exit to migration

* cluster? extract common code
3 years ago
hq1 0f577a998c
Update credo config + extras (#3547)
* Disable CyclomaticComplexity checks

* Move AuhtorizeSitesAPI plug under extra
3 years ago
hq1 b9ec38038c
Add small build option (#3536)
* Update applications

* Clone community config

* Move modules to experimental dir

* Update runtime config

* Apply first set of compile-time conditionals

* Move funnel schemas to experimental

* Make funnel schema-less build compile

* Use experimental/lib for elixir code

* Move JS funnels to experimental

* Clean up conditional rendering

* Tidy up the pipeline

* Make two builds pass tests without warnings

* Reuse existing dotenvs

* Do a bunch of renames

* Clean up naming

* Run secondary CI

* Update router

* Remove RewriteFunnelDupes migration

Tests were disabled already and it was a one-off shot

* Fixup quota mixins

* Add moduledoc

* Change MIX_ENV for seconary test run

* Skip crm on small

* !fixup

* Exclude flags pipeline

* Update lib/plausible_web/controllers/stats_controller.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
hq1 542385feab
Data Migration: rewrite funnels that contain duplicate goals (#3361)
* Add `RewriteFunnelDupes` data fix

To rewrite funnels referencing goals whose names
are the same. This enables us to enforce all goals
within the site unique later on.

* Credo

* Update priv/data_migrations/FunnelDupeGoals/sql/list-funnels-with-dupe-goal-ids.sql.eex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Revert unrelated changes

* Remove dead code

* Update lib/plausible/data_migration/rewrite_funnel_dupes.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible/data_migration/rewrite_funnel_dupes.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
ruslandoga 1c17e89b65
add transport_opts to data migration (#3179)
* add transport_opts to data migration

* add changelog entry

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
ruslandoga 447ed12155
use ecto_ch (#2890)
* use ecto_ch

* use Ch type in BoolUInt8

* update ecto_ch and ch to latest versions
4 years ago
hq1 71ef0bd043
Clean up after V2 migration (#2868)
* Clean up after V2 migration

This PR removes all the leftovers and alternative code
branching after v2 migration.

The self-hosted release is being drafted at:

https://github.com/plausible/hosting/issues/68

Refs:
  - https://github.com/plausible/analytics/pull/2865
  - https://github.com/plausible/analytics/pull/2825
  - https://github.com/plausible/analytics/pull/2780

* !fixup
4 years ago
ruslandoga adcce15632
Make self-hosted data migration easier (#2865)
* default to v2

* allow N defaults in data migration prompt and custom messages

* join domains lookup

* remove duplicate test runs from ci (both are v2)
4 years ago
hq1 b9c2110472
V2 migration tweaks for self hosted release (#2825)
* Get rid of PASS_V2_SCHEMA_MIGRATION

* Use in-memory domain lookup + regular table settings

* Remove faulty date arithmetic + prev part calculation

* Set V2_MIGRATION_DONE in Mix.env == :dev

* Mute credo
4 years ago
hq1 d2f2c69387
Conditionally support switching between v1 and v2 clickhouse schemas (#2780)
* Remove ClickhouseSetup module

This has been an implicit point of contact to many
tests. From now on the goal is for each test to maintain
its own, isolated setup so that no accidental clashes
and implicit assumptions are relied upon.

* Implement v2 schema check

An environment variable V2_MIGRATION_DONE acts like
a feature flag, switching plausible from using old events/sessions
schemas to v2 schemas introduced by NumericIDs migration.

* Run both test suites sequentially

While the code for v1 and v2 schemas must be kept still,
we will from now on run tests against both code paths.
Secondary test run will set V2_MIGRATION_DONE=1 variable,
thus making all `Plausible.v2?()` checks return `true'.

* Remove unused function

This is a remnant from the short period when
we would check for existing events before allowing
creating a new site.

* Update test setups/factories with v2 migration check

* Make GateKeeper return site id along with :allow

* Make Billing module check for v2 schema

* Make ingestion aware of v2 schema

* Disable site transfers for when v2 is live

In a separate changeset we will implement simplified
site transfer for when v2 migration is complete.
The new transfer will only rename the site domain in postgres
and keep track of the original site prior to the transfer
so we keep an ingestion grace period until the customers
redeploy their scripting.

* Make Stats base queries aware of v2 schema switch

* Update breakdown with v2 conditionals

* Update pageview local start with v2 check

* Update current visitoris with v2 check

* Update stats controller with v2 checks

* Update external controller with v2 checks

* Update remaining tests with proper fixtures

* Rewrite redundant assignment

* Remove unused alias

* Mute credo, this is not the right time

* Add test_helper prompt

* Fetch priv dir so it works with a release

* Fetch distinct partitions only

* Don't limit inspect output for partitions

* Ensure SQL is printed to IO

* Remove redundant domain fixture
4 years ago
Adam 6637751a5e
Implement Numeric IDs migration (#2762)
* Implement Numeric IDs migration

* Fix typo

* Mute credo for now

* Improve configurability and add stop_t

* Adjust to Ch/Chto only

* Fix opts key for dictionary password

* Add regular ecto migration with numeric ids v2 schemas (#2768)

* Add regular ecto migration

* Fix typo

* Update priv/ingest_repo/migrations/20230320094327_create_v2_schemas.exs

Co-authored-by: Vini Brasil <vini@hey.com>

* Implement v2 events/sessions schema modules (#2777)

* Implement v2 events/sessions schema modules

* Clean up session schemas

---------

Co-authored-by: Vini Brasil <vini@hey.com>

* Update moduledocs

---------

Co-authored-by: Vini Brasil <vini@hey.com>
4 years ago