Commit Graph

2544 Commits (32ab138301db33aa7b8b06df91c41d8e2bfd00d6)
 

Author SHA1 Message Date
Karl-Aksel Puulmann 32ab138301
Fix issue with name clash (#3925)
Unexpectedly, table.name caused a name clash after CR refactor, so using a unique name
for the output column

Sentry issue: https://sentry.plausible.io/organizations/sentry/issues/5612
3 years ago
Karl-Aksel Puulmann c219652dae
Re-apply Move conversion_rate logic from elixir to clickhouse (#3924)
* Revert "Revert "Move conversion_rate logic from elixir to clickhouse (#3887)"…"

This reverts commit 253fb5d67d.

* Fix issue with missing columns

The issue came from refactoring event:goal UNION ALL logic and trying to move
name select from first to last. If any other tables were joined, the incorrect
item would be used as an array index, causing this issue.

Added a relevant test.
3 years ago
Karl-Aksel Puulmann 253fb5d67d
Revert "Move conversion_rate logic from elixir to clickhouse (#3887)" (#3923)
This reverts commit 1909743b90.
3 years ago
Karl-Aksel Puulmann 1909743b90
Move conversion_rate logic from elixir to clickhouse (#3887)
* Separate out query building from pagination/execution logic.

* Refactor pageview_goals breakdown query, removing index column from results

* Remove zip_columns logic

* Use common pagination util

* Do everything in a single query for breakdowns for goals

* Order in DB

* Make sure column order is identical

* Calculate CR within the goal breakdown query

* Calculate CR for property breakdowns

* WIP: Calculate group CR

* CR with order_by

* Compatibility fix

* Import Ecto.Query and cleanup

* handle total_visitors the same way as add_percentage

* Handle conversion_rate in aggregate.ex

* Solve rebase fail

* Simplify maybe_add_group_conversion_rate

* Add conversion_rate defaults to 0 test

* Add test for conversion_rate should not be calculated with imported data (failing here and on master)

* Dont include imported data when breakdown by prop or goal

* Remove revenue_nils
3 years ago
ruslandoga 64f35bf8f6
make robots.txt restrictive by default (#3905)
* make robots.txt restrictive by default

* add note about cloud robots.txt
3 years ago
ruslandoga 279e89c693
CSV imports (no UI) (#3895)
* encode/decode date range in filenames

* Update lib/plausible/imported/csv_importer.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible/imported/csv_importer.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* drop unused functions

* send failure email if there is no data to export

* use PlausibleWeb.Email.mailer_email_from()

* ensure we get dates from minmax date query

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
ruslandoga 4242b52be4
Allow importing extra config (#3906)
* allow importing extra config

* changelog

* fix typo

* add test
3 years ago
ruslandoga dfcc8d794a
Add space after logo on the index page (#3825) 3 years ago
Karl-Aksel Puulmann 02d2256483
Set exit_page only on pageviews (#3870)
* Set exit_page only on pageviews

* Update tests

* Update entry_page on first pageview

* Update CHANGELOG.md
3 years ago
ruslandoga 07b714a143
Update Sentry (#3843)
* update Sentry

* Sentry.HTTPClient.child_spec is now optional

* Sentry.EventFilter is deprecated

* update sentry to 10.2.0

* fix dialyzer warnings
3 years ago
ruslandoga c08aa29ad8
Remove DISABLE_AUTH deprecation warning (#3904)
* remove DISABLE_AUTH deprecation warning

* changelog
3 years ago
ruslandoga 0a0c7ce165
Remove init-admin.sh (#3903)
* remove init-admin.sh

* changelog
3 years ago
ruslandoga 8c0e94f72c
Fix manual embed width (#3910)
* fix manual embed width

* changelog
3 years ago
ruslandoga 4ae00540dc
Split Elixir CI jobs (#3898)
* split elixir tests

* more understandable matrix

* add unused deps in mix.lock check

* get all deps for static checks
3 years ago
ruslandoga 5e74b1cf74
CSV exports (UI) (#3875)
* ui

* fix redirect link

* improve make minio

* use implicit button form for csv export

* add exports_bucket helper

* read S3_EXPORTS_BUCKET

* supply s3_bucket in export_csv job args

* make plausible_minio use unprotected port

* move s3_csv_export queue to base queues

* Update lib/plausible_web/controllers/site_controller.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
Karl-Aksel Puulmann d3586a81e3
UX improvement: don't autofocus on Custom properties on dashboard (#3872)
* UX improvement: don't autofocus on Custom properties on dashboard

Currently when you visit the dashboard with a certain configuration,
it will automatically scroll to the bottom, which is annoying.

To reproduce:
1. localStorage.clear() in console
2. Refresh, open properties at the very bottom
3. Refresh again

Expected behavior:
1. Dashboard shows

Actual behavior:
1. Dashboard scrolls to the bottom

* Changelog entry
3 years ago
hq1 59afa20955
Reapply #3878 + bugfix hit rate tracking (#3891)
* Reapply "Replace caching engine (#3878)" (#3883)

This reverts commit c5881cdc6d.

* Ensure hit rate is tracked on `get_or_store`

* Remove :wx and :observer

* Remove unused deps

* Use `:set` table type
3 years ago
RobertJoonas e8f3946dde
Fix division by zero in imported queries (#3890)
* prevent division by 0 in merge_imported queries

* Revert "fix bounce_rate change bug (#3886)"

This reverts commit 6eef32a8ff.

After 02aa0b2, we can keep on assuming that bounce rate is always numeric.
3 years ago
Adrian Gruntkowski 4d7d88cfec
Implement basics of GA4 import (#3851)
* Implement LV date input using flatpickr

* Implement basics of GA4 import (very dirty WIP)

* Split Google HTTP API into UA and GA4 specific parts

* Add a quick way to record GA4 API responses

* Add first GA4 import fixtures with GA4 Data API responses

* Extract GA4 and UA specific logic form Google API

* Extract UA and GA4 specific actions to distinct controllers

* Add integration test for GA4 importer

* Update GA4 fixtures

* Test GA4 API

* Add debug logging and fix paginating through API results in in GA4 import

* Revert "Implement LV date input using flatpickr"

This reverts commit c696f8ee39d5702f27015c09a4f079ca124cc7bb.

* Fix note
3 years ago
ruslandoga f2350b5165
Add `/tmp/` to .gitignore and simplify s3 cleanups in tests (#3889) 3 years ago
ruslandoga 5a3072ca21
CSV exports (no UI) (#3836)
* csv exports

* use ex_unit's tmp_dir
3 years ago
RobertJoonas 6eef32a8ff
fix bounce_rate change bug (#3886) 3 years ago
RobertJoonas 7641c66a2b
Stats api time on page (#3858)
* add metric validation + support in aggregate

* add a test ensuring comparison works

* disallow time_on_page with a goal filter

* Return time_on_page as `nil` from aggregate API

In case time_on_page cannot be calculated, we'll return it as `nil` from
the Stats API.

This is to make the behaviour consistent between breakdown and aggregate
endpoints. As for the UI, we'll still continue to report time_on_page as
0 - not changing any UI behaviour as discussed with Marko.

* add tests for time_on_page in event:page breakdown

* update changelog

* invalidate time_on_page with event:name filter

* add the ability to only query time_on_page in page breakdown

We'll need the visitors metric to get the list of pages to calculate the
time_on_page for.
3 years ago
Cenk Kücük dea9287d74
Revert "Temporarily disable tzdata auto-updating (#3873)" (#3885)
This reverts commit 11368e5143.
3 years ago
hq1 c5881cdc6d
Revert "Replace caching engine (#3878)" (#3883)
This reverts commit 437a3350ff.
3 years ago
Marko Saric ca7edd47cb
removing target="_blank" from blog link (#3882) 3 years ago
hq1 437a3350ff
Replace caching engine (#3878)
* Dependencies: swap Cachex for ConCache

* Implement Cache adapter wrapping ConCache

* Implement cache stats tracker, for metrics

* Use Cache.Adapter in Plausible.Cache

Marking the test as not slow anymore

* Use Cache Adapter when tracking sessions

* Use Cache Adapter for UA parsing

* Rename child identifiers - cachex is obsolete now

* Test stats tracking

* Update grafana metrics

* Put all caches under common child specification

* Try less

* Shorten the function delegation path
3 years ago
hq1 bb8a272a36
Fix margins in help guides section (#3881) 3 years ago
hq1 b68e3aa6a2
changes to the in-app footer (clean) (#3880)
* changes to the in-app footer

changes to the in-app footer to focus on product usage links rather than marketing pages

* Format

---------

Co-authored-by: Marko Saric <34340819+metmarkosaric@users.noreply.github.com>
3 years ago
ruslandoga 83a46fb47e
Dockerfile and co updates (#3811)
* wip

* more env, setup user before packages to avoid them picking our uid

* make entrypoint.sh executable by all again

* apparently it's a best practice to have executables be owned by root

* make entrypoint executable in COPY

* stop writing to /app, write to /tmp if needed

* fewer changes

* system user

* keep same style for multiline commands

* fewer changes

* add changelog entry

* fix group assignment for plausible user

* use gid=999

* no home

* no home

* add gecos

* add plausible user to nogroup instead of creating a custom one

* eh

* fewer changes

* fewer changes

* fewer changes

* use PERSISTENT_CACHE_DIR instead of STORAGE_DIR

* ignore more

* cleanup

* remove hex timeout env var

* use ERL_FLAGS=+JMsingle true in public builds

* fallback to /tmp and nest under /tzdata_data/ for tzdata

---------

Co-authored-by: Cenk Kücük <cenk@plausible.io>
3 years ago
Karl-Aksel Puulmann 11368e5143
Temporarily disable tzdata auto-updating (#3873)
This can cause issues as 1 day after a deploy the tzdata list is updated
from 2021 to 2024. Selecting a timezone thats not supported in 2021 can
then break dashboards temporarily following a deploy.

This can be reverted after https://github.com/plausible/analytics/pull/3811
3 years ago
Karl-Aksel Puulmann a9d3c03782
Validate the same metric isnt queried multiple times in external stats API (#3871)
* Validate the same metric isnt queried multiple times in external stats API

Issue: https://3.basecamp.com/5308029/buckets/35611491/card_tables/cards/7161347855

* Changelog entry

* Make credo happy
3 years ago
Karl-Aksel Puulmann 26d41ddbb9
Mutation to populate event session columns (#3844)
* WIP mutation to populate event session columns

* Remove duplication

* report errors, allow_nondeterministic_updates

* use right columns

* Update existing columns instead of session_* ones

* Make dialyzer happy

* Fix issue with passing pre-existing params in

* Logger -> IO.puts

* Use IngestRepo.config for connection settings

* Make dictionary options configurable

* Move allow_nondeterministic_mutations to within the migration

* Solve credo warning about too deep nesting

* Missed logger call

* Pattern matching in function head
3 years ago
Karl-Aksel Puulmann 0cdba7d407
Fix broken tests (#3867) 3 years ago
Karl-Aksel Puulmann c6d98397a8
Move `add_percentage` logic into clickhouse (#3854)
* Remove `add_percentage`, calculate percentages in clickhouse queries

This simplifies querying logic and avoids doing extra queries and avoids
race conditions.

* Remove special none handling from breakdowns, handling percentages correctly

* Add (failing) test showing expected add_percentage behavior for user making multiple sessions

* Update add_percentage behavior to use separate subqueries
3 years ago
Karl-Aksel Puulmann c60a2faee4
Write event table session columns (#3865)
* Write event table session columns

* Update testing factory rig
3 years ago
Karl-Aksel Puulmann 70d21f1b69
Improve compression of event table session columns (#3864) 3 years ago
Karl-Aksel Puulmann 8d977e0f76
Tests: session properties without the prefixes (#3863)
* Undo event session attributes renaming

* Rename session_ attributes in tests
3 years ago
Cenk Kücük fecd665c19
Push private images to GHCR (#3859) 3 years ago
Karl-Aksel Puulmann d5048fd6b4
Stop writing session properties into events table (#3800)
* Refactor: Explicitly add field names to INSERT

This avoids issues when code schema is out of sync with real schema

* Dont write session parameters to events

These would only be stored on first event anyways. Work remains to be done
on tests which have their own helper

* Remove writes to country_code in a test

* Remove old columns from being accessible in elixir code

* Update most tests to use new way of adding session props to events

* Update testing harness

* Update stats controller test

* Update for shield rules

* update breakdown tests

* Fix typing of state for dialyzer

* Drop support for old session attributes code

* Update remaining tests

* cond -> if
3 years ago
hq1 5eb9d724e5
Plugins API: mark data_domain as nullable in capabilities schema (#3840)
* Plugins API: mark data_domain as nullable in capabilities schema

* fixup
3 years ago
Karl-Aksel Puulmann ea38b45685
Ecto migration to move sessions_v2 to VersionedCollapsingMergeTree (#3809)
This migration will noop in staging/production as it already has been run. It also leaves
behind a backup table that initially takes no extra space but will need to be cleaned up
manually
3 years ago
Adrian Gruntkowski 39aa81a16f
Implement UI for multiple imports (#3727)
* Create a stub of site settings section for imports and exports

* Use legacy site import indication to determine UA import handling

* Add provisional logos for upcoming import sources

* Stub basics of import page

* Add very rudimentary support for multiple UA imports

* Implement imports list as live view

* Add support for opening LV modal from backend and closing from frontend

* Introduce notion of themes to `button` and `button_link` components

* Add confirmation modal on deleting import

* Swap GA4 logo

* Implement disabled state support for `button_link` component

* Disable export and non-implemented import sources

* Use native starts start date for upper boundary of import time range

* Ensure integrations view uses legacy UA import flow

* Remove unnecessary preload in SiteController

* Remove unnecessary exception for legacy imports

* Move API controller stats tests under PlausibleWeb

* Test listing imports

* Add test for explicit listener setup

* Add tests for legacy flag state in UA importer

* Add test for purging legacy import data

* Add tests for `Sites.native_stats_start_date`

* Test forgetting imports

* Add `Stats.Clickhouse.imported_pageview_counts/1` and fix test flakiness

* Show page view counts on imports list

* Add tests for static imports and exports view

* Adjust button look slightly

* Use `case` instead of `cond`

* Make feature flag customisable per site

* Fix buttons and empty state styling

* Add another import to seeds

* Use JS confirm dialog instead of modal for deletion confirmations

* Revert "Add support for opening LV modal from backend and closing from frontend"

This reverts commit 260e6c753032b451542e24be9edc2118790b5a00.

* Default `legacy` to false when inserting new import jobs

* Drop `method` attribute from `button_link` and `unstyled_link` components
3 years ago
Adrian Gruntkowski fdbe4cc0a4
Add `legacy` flag column to `site_imports` schema (#3842) 3 years ago
Marko Saric 0fe5682f91
Change the CTA to upgrade now rather than contact (#3837) 3 years ago
hq1 316148e788
Fix settings options mobile (#3838)
* Fix settings options on mobile view

* Fix settings options selected item on mobile

* Format
3 years ago
ruslandoga f3423aefec
Add csv importer (#3795)
* add csv importer

* make table validation explicit

* update some docs

* improve docs

* add minio container to ci

* more tests

* eh

* continue

* add passing test

* add failing test

* add config test

* add minio to Makefile

* testcontainers

* remove extra whitespace

* explain the implementation a bit

* account for async deletes in tests

* bounces is UInt32

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
ruslandoga 31cf3e54f8
Add Bamboo.Mua (#3654) 3 years ago
hq1 f1b6a672d4
Fix `added_by` not saving on adding country rules (#3835)
* Fix `added_by` not saving on adding country rules

* Format

* Remove dupe attr
3 years ago
hq1 518cdb3307
Shield: Country Rules (#3828)
* Migration: add country rules

* Add CountryRule schema

* Implement CountryRule cache

* Add country rules context interface

* Start country rules cache

* Lookup country rules on ingestion

* Remove :shields feature flag from test helpers

* Add nested sidebar menu for Shields

* Fix typo

* IP Rules: hide description on mobile view

* Prepare SiteController to handle multiple shield types

* Seed some country shield

* Implement LV for country rules

* Remove "YOU" indicator from country rules

* Fix small build

* Format

* Update typespecs

* Make docs link point at /countries

* Fix flash on top of modal for Safari

* Build the rule struct with site_id provided up-front

* Clarify why we're messaging the ComboBox component

* Re-open combobox suggestions after pressing Escape

* Update changelog

* Fix font size in country table cells

* Pass `added_by` via rule add options

* Display site's timezone timestamps in rule tooltips

* Display formatted timestamps in site's timezone

And simplify+test Timezone module; an input timestamp converted
to UTC can never be ambiguous.

* Remove no-op atom

* Display the maximum number of rules when reached

* Improve readability of remove button tests

* Credo

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago