Commit Graph

171 Commits (db5488efa677563cc3fce874f72ccb4e7c939708)

Author SHA1 Message Date
hq1 bbedeff683
Implement Funnels view on dashboard (#3066)
* Add Funnel react component

assets/js/dashboard/stats/behaviours/funnel.js - restored from:

98a76cbd Remove console.info calls
d94db99d Convert Funnel class component into a functional one
028036ad Review comments
3067a940 Stop doing maths in react
73407cc3 Fix error handling when local storage gets corrupted
e8c6fc52 Format numbers on funnel labels
c815709f Reorganize component responsibility
7a88fe44 Outline basic error handling
94caed7c Chart styling updates
4514608a Add percentages to funnel
d622c32d Add funnel picker

Co-authored-by: Uku Taht <uku.taht@gmail.com>

* Pass funnels list to react via data-funnels

* Implement Funnels react API

lib/plausible_web/controllers/api/stats_controller.ex - restored from:

f36ad234 Adjust to Plausible.Stats interface
9b532273 Test funnel stats controller
028036ad Review comments
bea3725f Remove IO.inspect
7a88fe44 Outline basic error handling
c8ae3eaf Move Funnels to StatsController and use base query
667cf222 Put private functions at the bottom

* Tweak funnel presentation

* Handle errors at the top

* Do not register DataLabels plugin globally

or else all the existing charts are affected

* Calculate drop-off percentage evaluating funnels

* Tweak dark mode + implement nicer tooltips

* Make currently selected funnel bold in the picker

* Count user_ids not session_ids when evaluating funnels

So if a visitor goes:

1. Start session
2. Complete funnel step 1
3. Inactive for 30 minutes
4. Complete funnel step 2

We would not be able to track this funnel completion because of the session timeout.
We like to o measure this as funnel completion even though the session expired in the middle.

cc @ukutaht

* Add extra properties to the funnels API

cc @ukutaht

* Improve tooltips so that step to data is rendered

* Change tooltip number formatting

* Remove debugging remnants

* Quick & dirty mobile view

* Fix mobile view: tweak dark mode & funnel switching

* Ignore DOMException: the operation was aborted

Otherwise this sometimes flashes the space shuttle
screen when navigating quickly via a keyboard.

* Format percentages on the main chart

* Close missing tag 🙈

* Revert "Close missing tag 🙈"

This reverts commit 9c2f970e22fd7e2980503242b414f42ce8bce1d2.

* Use jsx to render funnel tooltip

To get markup validated via lsp mostly...

* Fixup: s/class/className

* Fix className interpolation

* Add a ruler to the tooltip

* Tweak funnel chart style

* Fix font distortion issue on chart/canvas labels

* s/class/className

* Put "Set up funnels" link behind a feature flag

* Refactor internal selection storage

Getting ready for live funnel evaluation

* Don't try to connect LV socket if there's no CRSF token set up

This is perfectly okay for some of the templates/layouts.

* Fix up funnel creation typespecs

Unfortunately we can't define a type with literal string keys,
hence this must suffice.

* Use uniq over count/distinct

* Revert JSX in tooltips

Ref: https://github.com/plausible/analytics/pull/3066#discussion_r1241891155

* Remove the extra query for counting all visitors

cc @ukutaht

* Add premium notice

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
3 years ago
Vini Brasil 41c88c4fa0
Add unique names to cache supervisors in tests (#3055)
This commit fixes a flaky test where a supervisor with that same name
was already started.
3 years ago
Vini Brasil 8834486a19
Pass cached site struct down the ingestion pipeline (#3027)
* Pass cached site struct down the ingestion pipeline

Revenue goals need the cached site struct during ingestion to get the
goals name and currency. This cache lookup is not necessary as
`GateKeeper.check/1`, which is called first in the ingestion pipeline,
could already return the site struct from the cache.

This commit changes `GateKeeper.check/1` to return the site struct
instead of the site ID. Moreover, this commit changes the ingestion
pipeline to avoid calling the sites cache twice.

Related: https://github.com/plausible/analytics/pull/2957#discussion_r1203921549

* Remove revenue_goals unnecessary fallback

* Change duplicate child_id in cache test

* Remove revenue goal condition from cache query

* Remove Plausible.DataCase.reload/1
3 years ago
hq1 a356009af8
Implement backend core for funnels (#3023)
* Implement Funnel schema

* Implement Funnels context module

lib/plausible/funnels.ex - restored from:

dca93621 Mark Vini's suggestion applied
7d1eec75 Update lib/plausible/funnels.ex
028036ad Review comments
b694ccd5 Express the guard nicely
acd9c4f2 Prepare ephemeral funnel definitions so that users can test funnels
aea4ebc4 Access Funnel min/max steps via the __using__/1 macro
51f0397d Implement deleting funnels
cf228630 wip
1f6fe25d Add number of steps to funnels list
89f10ecb wip
950a18d9 Dirty funnel save
e62bfce5 Refactor funnel result evaluation
88e38a33 Update funnel tests
bf47f654 Calculate conversion rate/dropoff in the backend
9e62ffb1 Make existing tests pass
7a88fe44 Outline basic error handling
c8ae3eaf Move Funnels to StatsController and use base query
2929bfd4 Minor rework
b490403b !ifxup
ae44db77 Add Funnels.list/1 function
4f3216b3 Unique steps
6e0c3efc Ensure step ordering works
cea93ba1 It works I think
c50ac0c2 WIP
a466ec4c Dirty wip
fbd14e23 Absolute dirty wip
3ddd15ad wip
34ceb3f5 dirty wip

test/plausible_web/controllers/api/stats_controller/funnels_test.exs - restored from:

9b532273 Test funnel stats controller

Co-authored-by: Uku Taht <uku.taht@gmail.com>

* Update Goals implementation

lib/plausible/goal/schema.ex - restored from:

028036ad Review comments
acd9c4f2 Prepare ephemeral funnel definitions so that users can test funnels
4a4ddbdc Optionally preload funnels for goals and stub funnel-goal deletion
fbd14e23 Absolute dirty wip

lib/plausible/goals.ex - restored from:

aea4ebc4 Access Funnel min/max steps via the __using__/1 macro
a418c039 Implement funnel reduction logic on goal deletion
4a4ddbdc Optionally preload funnels for goals and stub funnel-goal deletion
ebdc4333 Extend the prompt in case of funnel-goal deletion
950a18d9 Dirty funnel save
f06c3fcf wip

test/plausible/goals_test.exs - restored from:

a418c039 Implement funnel reduction logic on goal deletion
4a4ddbdc Optionally preload funnels for goals and stub funnel-goal deletion
dee5bc9e Fixup tests

* Update seeds

* Split funnel schema modules

* Add moduledoc to Funnel schema

* Update Funnel.Step moduledoc

* Add typespecs to Plausible.Funnels

* Implement Phoenix.HTML.Safe for Funnel schema

* Fixup a test after ordering goals by desc:id

test/plausible_web/controllers/api/stats_controller/conversions_test.exs - restored from:

26a493cb Fixup a test after ordering goals by desc:id

* Annotate Goal schema with type t()

* Reword schema module docs

* Fix up typespecs

* Stop tagging Funnels.get/2 result

* Nuke Goals.by_id!/2

* Test Funnels module

test/plausible/funnels_test.exs - restored from:

b7a6d3ad ya, rly
028036ad Review comments
acd9c4f2 Prepare ephemeral funnel definitions so that users can test funnels
51f0397d Implement deleting funnels
1f6fe25d Add number of steps to funnels list
dee5bc9e Fixup tests
950a18d9 Dirty funnel save
88e38a33 Update funnel tests
bf47f654 Calculate conversion rate/dropoff in the backend
9e62ffb1 Make existing tests pass
2929bfd4 Minor rework
b490403b !ifxup
ae44db77 Add Funnels.list/1 function
4f3216b3 Unique steps
6e0c3efc Ensure step ordering works
cea93ba1 It works I think
c50ac0c2 WIP
a466ec4c Dirty wip
34ceb3f5 dirty wip

Co-authored-by: Uku Taht <uku.taht@gmail.com>

* Apply changes related to 242081e5

* Evaluate funnels through `Stats.funnel/3` entry point

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
3 years ago
Vini Brasil e4d4f7d954
Revenue tracking: Ingestion and breakdown queries (#2957)
* Add revenue fields to ClickHouse events

This commit adds 4 fields to the ClickHouse events_v2 table:

* `revenue_source_amount` and `revenue_source_currency` store revenue in
  the original currency sent during ingestion

* `revenue_reporting_amount` and `revenue_reporting_currency` store
  revenue in a common currency to perform calculations, and this
  currency is defined by the user when setting up the goal

The type of amount fields is `Nullable(Decimal64(3))`. That covers all
fiat currencies and allows us to store huge amounts. Even though
ClickHouse does not suggest using `Nullable`, this is a good use case,
because otherwise additional work would have to be done to
differentiate missing values from real zeroes.

I ran a benchmark with the data pattern we expect in production, where
we have more missing values than real decimals. I created 100 million
records where 90% of decimals are missing. The difference between the
tables in storage is just 0.4Mb.

* Add revenue parameter to Events API

This commit adds support for sending revenue data in ingestion using the
`revenue` parameter - aliased to `$`.

* Add revenue parameter to mix send_pageview

* Add average and total revenue to breakdown queries
3 years ago
Vini Brasil c559370201
Trim prop list to 30 items during ingestion (#3009)
This commit adds an upper bound limit to custom props. It fails silently
and trims the list instead of dropping the event.
3 years ago
Uku Taht fc918081b5
Fix event name validation (#2975)
* Fix event name validation

* Add @moduledoc

* Move tests to Ingestion.RequestTest

* Remove comment
4 years ago
ruslandoga 40e95ffd3d
add LOG_FAILED_LOGIN_ATTEMPTS (#2936)
* add failed login logs

* put failed login attempt logs behind a config option

* add changelog entry

* add config test

* add auth_controller tests

* move tests to separate non-async test module

---------

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
4 years ago
ruslandoga ce7401dd83
add MAILER_NAME (#2937)
* add MAILER_NAME

* add mailer test

---------

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
4 years ago
ruslandoga 7b22aff11c
add MAILGUN_BASE_URI (#2935)
* add MAILGUN_BASE_URI support

* add changelog entry
4 years ago
RobertJoonas dafb60164a
Allow setting sample_threshold in dashboard API queries (#2958)
* refactor Query.from/2

* allow specifying sample_threshold number in query
4 years ago
Vini Brasil 10d9e3b083
Revenue tracking: Add currency field to goal creation (#2948)
* Add revenue goal option to goal creation

This commit adds a currency field to the goals form. Goals that have a
currency set are now revenue goals, and are cached with sites to later
be used during ingestion.

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

* Enable feature flag in tests

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
4 years ago
Uku Taht 2259651b21
Ignore missing subscription on update (#2930) 4 years ago
RobertJoonas fa54efbc6d
Support `with_imported=true` in Stats API aggregate (#2919)
* default to source_query.include_imported before true

This fixes a bug where a Stats API aggregate query was trying
to query current period *without imported data* but previous
period *with imported data*.

* remove __internal_visits at a better time

As soon as we have made the db query, we don't need it anymore

* disallow events metric with imported data

* update changelog

* keep default in a better place

* remove unused option

* update test description

Co-authored-by: Vini Brasil <vini@hey.com>

---------

Co-authored-by: Vini Brasil <vini@hey.com>
4 years ago
Vini Brasil 4ca2e26b9c
Limit ingestion payload size (#2904)
This pull request adds constraints on user input values during ingestion. The purpose of this PR is to filter out unrealistic or invalid inputs. More comments about each limit inline.
4 years ago
hq1 6a831d9898
Harden API input validation (#2869)
* Reject invalid country codes early

* Validate all date parameters early

* Ensure request JSON parses to a map

* Re-arrange function calls

* Extend malicious filter validation
4 years ago
Vini Brasil 8dc8423319
Prepare Comparisons for release (#2873)
This pull request removes the comparisons feature flag, making it visible to all users. It also removes percentages from top stats in default view, leaving those for the comparison view only.
4 years ago
hq1 71ef0bd043
Clean up after V2 migration (#2868)
* Clean up after V2 migration

This PR removes all the leftovers and alternative code
branching after v2 migration.

The self-hosted release is being drafted at:

https://github.com/plausible/hosting/issues/68

Refs:
  - https://github.com/plausible/analytics/pull/2865
  - https://github.com/plausible/analytics/pull/2825
  - https://github.com/plausible/analytics/pull/2780

* !fixup
4 years ago
hq1 b8e96b948f
Stop using Double in cache store tests (#2854) 4 years ago
Vini Brasil 3d656ae95b
Match day of the week feature for comparisons (#2822)
* Add support for `match_day_of_week?` back-end option

* Add match day of week input to comparison input

* Default match day of the week to true
4 years ago
hq1 c04e9286dd
Associate goals with sites, not domains (#2828)
* Revert "Rephrase error message"

This reverts commit f624443a96.

* Revert "Temporarily disable goal creation"

This reverts commit a091635b9d.

* Update ecto schema

* Make sure goal operations are per site

* Update tests

* Split postgres migrations
4 years ago
hq1 3cb089eab4
Migrate and freeze goals creation (#2833)
* Full migration (to be submitted separately)

* Do not remove `Goal.domain` just yet

* Do not make Goal.site not nullable just yet

* Temporarily disable goal creation

* Rephrase error message

* Add down migration
4 years ago
hq1 154ce3a44c
Split clickhouse repos - making the main one read only (#2826)
* Split clickhouse pools into readonly/import deletions

* Remove CRM site transfers

* Initialize ImportDeletionRepo

* Put ImportDeletionRepo to use
4 years ago
Uku Taht 71e4d228c0
Escape pipes in goal names (#2814)
* Escape pipes in goal names

* Apply suggestions from code review

Use ~S sigil for better readability

Co-authored-by: Vini Brasil <vini@hey.com>

---------

Co-authored-by: Vini Brasil <vini@hey.com>
4 years ago
hq1 1d01328287
Allow domain change (#2803)
* Migration (PR: https://github.com/plausible/analytics/pull/2802)

* Implement Site.Domain interface allowing change and expiry

* Fixup seeds so they work with V2_MIGRATION_DONE=1

* Update Sites.Cache so it's capable of multi-keyed lookups

* Implement worker handling domain change expiration

* Implement domain change UI

* Implement transition period for public APIs

* Exclude v2 tests in primary test run

* Update lib/plausible_web/controllers/site_controller.ex

Co-authored-by: Vini Brasil <vini@hey.com>

* Update lib/plausible_web/controllers/site_controller.ex

Co-authored-by: Vini Brasil <vini@hey.com>

* Update moduledoc

* Update changelog

* Remove remnant from previous implementation attempt

* !fixup

* !fixup

* Implement domain change via Sites API

cc @ukutaht

* Update CHANGELOG

* Credo

* !fixup commit missing tests

* Allow continuous domain change within the same site

---------

Co-authored-by: Vini Brasil <vini@hey.com>
4 years ago
hq1 d2f2c69387
Conditionally support switching between v1 and v2 clickhouse schemas (#2780)
* Remove ClickhouseSetup module

This has been an implicit point of contact to many
tests. From now on the goal is for each test to maintain
its own, isolated setup so that no accidental clashes
and implicit assumptions are relied upon.

* Implement v2 schema check

An environment variable V2_MIGRATION_DONE acts like
a feature flag, switching plausible from using old events/sessions
schemas to v2 schemas introduced by NumericIDs migration.

* Run both test suites sequentially

While the code for v1 and v2 schemas must be kept still,
we will from now on run tests against both code paths.
Secondary test run will set V2_MIGRATION_DONE=1 variable,
thus making all `Plausible.v2?()` checks return `true'.

* Remove unused function

This is a remnant from the short period when
we would check for existing events before allowing
creating a new site.

* Update test setups/factories with v2 migration check

* Make GateKeeper return site id along with :allow

* Make Billing module check for v2 schema

* Make ingestion aware of v2 schema

* Disable site transfers for when v2 is live

In a separate changeset we will implement simplified
site transfer for when v2 migration is complete.
The new transfer will only rename the site domain in postgres
and keep track of the original site prior to the transfer
so we keep an ingestion grace period until the customers
redeploy their scripting.

* Make Stats base queries aware of v2 schema switch

* Update breakdown with v2 conditionals

* Update pageview local start with v2 check

* Update current visitoris with v2 check

* Update stats controller with v2 checks

* Update external controller with v2 checks

* Update remaining tests with proper fixtures

* Rewrite redundant assignment

* Remove unused alias

* Mute credo, this is not the right time

* Add test_helper prompt

* Fetch priv dir so it works with a release

* Fetch distinct partitions only

* Don't limit inspect output for partitions

* Ensure SQL is printed to IO

* Remove redundant domain fixture
4 years ago
Uku Taht a75d0b35b0
Fix Timex.total_offset blowing up during clock changes (#2788)
* Fix Timex.total_offset blowing up during clock changes

* Format large numbers with underscore in tests

Co-authored-by: Adam <hq@mtod.org>

---------

Co-authored-by: Adam <hq@mtod.org>
4 years ago
Vini Brasil 87a63fe28a
Custom comparison mode (#2753)
This commit implements the custom comparison mode using a date range.
4 years ago
Uku Taht a154a4214b
Support multiple filters - backend (#2749)
* Wrap Plausible.Stats.Filters with unit tests

* Parse `member` filter type

* Support escaped | in member filter

* Support for `member` filter in `aggregate_time_on_page`

* Add support for `member` filter type on goals

* Disable Credo warning

* Support `not_member` filter type

* Disable credo for `query_sessions`

* Support `matches_member` and `not_matches_member` filters

* Disable Credo for `Filters.filter_value/2`

* Support `matches_member` and `not_matches_member` for goal filter

* Support for contains_member and friends

* Updates for new chto driver

* make top_stats_test.exs:203 pass (#2779)

---------

Co-authored-by: ruslandoga <rusl@n-do.ga>
4 years ago
Adam 6d79ca5093
Switch to new clickhouse adapter (ch/chto) (#2733)
* another clickhouse adapter

* don't restore stats_removal.ex

* fix events main-graph error (#2746)

* update ch, chto

* update chto again (#2759)

* Stop treating page filter as an entry_page filter (#2752)

* remove dead code

* stop treating page filter as entry page filter in breakdown queries

* stop treating page filter as entry page filter in aggregate queries

* stop treating page filter as entry page filter in timeseries queries

* mix format

* update changelog

* break code down to smaller functions to keep credo happy

* remove unused functions

* make CSV export return only conversions with goal filter (#2760)

* make CSV export return only conversions with goal filter

* update changelog

* update elixir version in mix.exs (#2742)

* revert admin.ex changes (#2776)

---------

Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
Co-authored-by: ruslandoga <rusl@n-do.ga>
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
4 years ago
Adam 736e6e385c
Capture no_cache sentry messages together (#2772) 4 years ago
RobertJoonas a490016f6e
fix test to make build green (#2747) 4 years ago
Vini Brasil 8b0f0cabc2
Implement "Year over Year" comparison mode (#2704)
This pull request adds support for multiple comparison modes, changes the comparison checkbox to a combobox, and implements the year over year comparison mode. The feature is still behind a feature flag.

Co-authored-by: Uku Taht <uku.taht@gmail.com>
4 years ago
RobertJoonas 874d664521
Add the new `Views per Visit` metric to dashboard Top Stats and CSV export (#2728)
* globally rename 'pages_per_visit' to 'views_per_visit'

* change the order of top stats

* rename 'Visits' to 'Total visits' in the UI

* add views_per_visit to UI

* put the new metric under a feature flag

* add new metric to CSV export under feature flag

* mix format

* use only one feature flag
4 years ago
Uku Taht 43bf7dd09f
Use user-agent instead of screen_width to get device type (#2711)
* Use user-agent instead of screen_width to get device type

Co-authored-by: eriknakata <erik.nakata5@gmail.com>

* Fix credo

* Log on unhandled UAInspector device type

* Make 'browser' the default tab in devices report

* Remove device tooltip

* Remove screen_width from ingestion completely

* Remove browserstack harness, run playwright directly

* Select meta key based on OS platform

* Run CI tests in parallel

* Improve device match readability

* Add changelog

---------

Co-authored-by: eriknakata <erik.nakata5@gmail.com>
4 years ago
Uku Taht f26ca7da9f
Update native stats start date when transferring (#2719) 4 years ago
Adam 8f86036e57
Keep track of native stats start timestamp when retrieving data (#2715)
* Stats boundary/PoC?

* Delete stats removal

* Drop events check on site creation

* Update seeds script

* Use native_stats_start_at

* Don't rely on native stats pointer in imported stats queries

* Reset site

* Export reset/1

* Remove unnecessary inserted_at settings

* Update seeds

* Remove unnecessary inserted_at setting
4 years ago
Adam Rutkowski 867dad6da7
Implement ingest counters (#2693)
* Clickhouse migration: add ingest_counters table

* Configure ingest counters per MIX_ENV

* Emit telemetry for ingest events with rich metadata

* Allow building Request.t() with fake now() - for testing purposes

* Use clickhousex branch where session_id is assigned to each connection

* Add helper function for getting site id via cache

* Add Ecto schema for `ingest_counters` table

* Implement metrics buffer

* Implement buffering handler for `Plausible.Ingestion.Event` telemetry

* Implement periodic metrics aggregation

* Update counters docs

* Add toStartOfMinute() to ordering key

* Reset the sync connection state in `after` clause

* Flush counters on app termination

* Use separate Repo with async settings enabled at config level

* Switch to clickhouse_settings repo root config key

* Add AsyncInsertRepo module
4 years ago
RobertJoonas bc557b78fd
Add visits metric under a feature flag (#2687)
* Add visits metric and make it graphable

* include visits metric in csv export (visitors.csv)

* put visits under a feature flag (CSV export)

* feature flag for displaying visits on the dashboard

* fix formatting

* add visits metric to top stats (fix)

* fix imported_test to expect visits metric included

* fix formatting
4 years ago
Vini Brasil 1cb07efe6d
Save city name when importing from GA (#2608)
This commit adds city data to imported records from Google Analytics. The
current implementation sets city to 0 because GA does not use the GeoNames
database.

Google Analytics Reporting API uses [Geographical IDs](https://developers.google.com/analytics/devguides/collection/protocol/v1/geoid)
to identify cities and countries. Plausible uses
[GeoNames](https://geonames.org/) and I couldn't find databases corelating the
two.

Fortunately, GA also returns the city name and this commit uses the city name
and the country ISO code to find the Geoname ID. To avoid making expensive ETS
searches, I created another ETS table in the Location library that uses
{country, city} as a key.

Related PR: https://github.com/plausible/location/pull/3
4 years ago
Adam Rutkowski 8f85b110aa
Split Clickhouse pools into Read-Only and Read/Write (dedicated to writes) (#2661)
* Configure ingest repo access/pool size

If I'm not mistaken 3 is a sane default, the only
inserts we're doing are:

  - session buffer dump
  - events buffer dump
  - GA import dump

And all are serializable within their scopes?

* Add IngestRepo

* Start IngestRepo

* Use IngestRepo for inserts

* Annotate ClickhouseRepo as read_only

So no insert* functions are expanded

* Update moduledoc

* rename alias

* Fix default env var value so it can be casted

* Use IngestRepo for migrations

* Set default ingest pool size from 3 to 5

in case conns are restarting or else...

* Ensure all Repo prometheus metrics are collected
4 years ago
ruslandoga 7b2f4c99ee
Support alternative mailing services (Mailgun, Mandrill, Sendgrid) (#2649)
* more bamboo adapters

* add changelog

* add tests
4 years ago
Adam Rutkowski 8f9f032968
Delay stats deletions (#2632)
* Implement Site removal transaction

* Implement Stats removal Oban worker

* Configure site removal queue

* Call Site.Removal.run() instead of Purge.delete_site!

* Test site/stats removal

* Remove FIXME - filed a ticket

* Over-communicate lenghty deletion process to the users
4 years ago
Adam Rutkowski 5de43b758d
Run tests in async mode where applicable (#2542)
* Set pg pool size for MIX_ENV=test

* Include slow tests in CI run

* Exclude slow tests by default

* Mark tests slow/async where applicable

* Restructure captcha mocks

* Revert async where env is relied upon

* Add --max-failures=1 to CI run

* Set warnings as errors

* Disable async where various mocks are used

* Revert "Disable async where various mocks are used"

This reverts commit 2446b72a29.

* Disable async for test using vcr
4 years ago
Vini Brasil 5152e8d416
Reject events with long URIs and data URIs (#2536)
* Reject events with data URIs

* Reject events with URIs longer than 2,000 characters

* Update CHANGELOG.md
4 years ago
Uku Taht a37b3433d7
Always show direct traffic in sources reports (#2531)
* Remove show_noref behaviour

Removes query param show_noref which was used from React to control
whether to show Direct / None traffic or not. The show_noref behaviour
was untested previously.

Closes #2523

* Add changelog entry

* Fix tests

* Removed files I did not mean to check in :)
4 years ago
Vini Brasil 478e0c6990
Handle GA authentication errors (#2505) 4 years ago
Adam Rutkowski 467317f24e
Fix: application startup, event timestamps (#2479)
### Changes

This PR:
- pushes PromEx to the bottom of supervision stack to avoid Endpoint
instrumentation failure
- ensures the site cache is ready by exposing it through the health
check endpoint
- fixes event timestamps being calculated at compile time, with
regression unit and integration tests

### Tests
- [x] Automated tests have been added
- [ ] This PR does not require tests

### Changelog
- [ ] Entry has been added to changelog
- [x] This PR does not make a user-facing change

### Documentation
- [ ] [Docs](https://github.com/plausible/docs) have been updated
- [x] This change does not need a documentation update

### Dark mode
- [ ] The UI has been tested both in dark and light mode
- [x] This PR does not change the UI
4 years ago
Adam Rutkowski 356575ef78
Gatekeep ingestion pipeline (#2472)
* Update Sites.Cache

So it's now capable of refreshing most recent sites.
Refreshing a single site is no longer wanted.

* Introduce Warmer.RecentlyUpdated

This is Sites Cache warmer that runs only for
most recently updated sites every 30s.

* Validate Request creation early

* Rename RateLimiter to GateKeeper and introduce detailed policies

* Update events API tests - a provisioned site is now required

* Update events ingestion tests

* Make limits visible in CRM Sites index

* Hard-deprecate DOMAIN_BLACKLIST

* Remove unnecessary clause

* Fix typo

* Explicitly delegate Warmer.All

* GateKeeper.allwoance => GateKeeper.check

* Instrument Sites.Cache measurments

* Update send_pageview task to output response headers

* Instrument ingestion pipeline

* Credo

* Make event telemetry test a sync case

* Simplify Request.uri/hostname handling

* Use embedded schema, apply action and rely on get_field
4 years ago
Vini Brasil 994e7d09de
Parse event URL and domain in Plausible.Ingestion.Request (#2351)
* Parse event URL in Plausible.Ingestion.Request

* Parse event domain in Plausible.Ingestion.Request

* Rework ingestion pipeline processing (#2462)

* Rework ingestion pipeline processing

So that Request can have multiple domains and
based on that each event is processed uniformly.
The build_and_buffer/1 function now returns an
accumulator with all the dropped/buffered events
for further inspection.

* Reduce function complexity

* Don't chain struct fields to check for an empty host

* Separate referrer and utm tags

* Fix up `with` clause, credo was right cc @vinibrsl

Co-authored-by: Adam Rutkowski <hq@mtod.org>
4 years ago