Commit Graph

206 Commits (modals)

Author SHA1 Message Date
hq1 0007c0c108
Plugins API: 2nd pass with Goals resource + SharedLinks schema changes (#3396)
* Remove "Context" namespace level

* Change Goal string representation

* Alias Schemas in Plugin API Test Case template

* Update schema & tests for SharedLink resource

* Update Goals interface

- make it possible to create revenue goals
- extract "for site" query to a standalone function

* Fixup typespecs

* Alias Errors module in OpenAPI controllers

* Add missing goals test

* Implement Goals Plugins API resource

* Add extra test to confirm changeset error propagation

* Mute credo

* Fix typos

* Handle changeset traversal in `Errors`

* Use upserts in `Goals.find_or_create`

* Extract touch_site! to Site.Cache, address credo, improve code docs

* Apply formatting

* Remove unused inner join

* Update test/plausible_web/plugins/api/controllers/goals_test.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update test/plausible_web/plugins/api/controllers/goals_test.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update test/plausible_web/plugins/api/controllers/goals_test.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update test/plausible_web/plugins/api/controllers/goals_test.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update test/plausible_web/plugins/api/controllers/goals_test.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update error message on revenue goal currency clash

* Remove unused code

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
RobertJoonas 8bc86d165f
Use Phoenix LiveView for the upgrade page (#3382)
* add a new upgrade page liveview behind a FF

* Create plans_v4.json file

* Add the upgrade page UI template and some basic functionalities

* different content based on subscription plan existing or not
* pageview slider
* monthly/yearly switch

* fix tests

* split into 2 separate functions

* rename variables

* implement volume slider + read default interval/volume from plan

* organize choose-plan.ex better

* remove unused vars from tests

* make monthly_cost and yearly_cost nil by default

The actual prices for all plans are stored in Paddle. We don't need to
keep the duplicates in the JSON files.

* add fetch_prices/1 to PaddleApi

* make v4 business ID's differ from growth ones

* render actual price information from plans

...and make the prices in both growth and business plan boxes change
dynamically when the pageview slider or interval is changed.

* highlight current subscription plan box

* add test describe block for business tier subscription

* connect to live socket only on the specific LV page using focus.html

* only wrap the input slider inside the form

* little readability improvement

* add v4 team_member_limits (after rebase with master)

* extract monthly_quota_box function in user_settings

When the business_tier FF is enabled, this section is different and
links to the new upgrade page.

* document subscription statuses

* change _notice.html.eex to .heex

* extract subscription status notice components

* add failed payment notices to upgrade page

* create class_of_element/2 convenience function for testing

* add cancel_subscription mix task

* implement checkout buttons

* mix format

* get all available plans with prices through plans.ex

* use more suitable function for fetching usage

* avoid double db lookups on mount

* rename variable

* separate functions for getting plan by product_id vs subscription

* separate subscription status docs into context module

* consider cancelled subscriptions

* default volume by usage if no subscription plan

* add enterprise-level volume option to slider

* optimize for darkmode

* UI improvements

* display 2 months free notice for yearly billing
* VAT excluded notice
* note about having a business subscription in user settings
* make the page pop and fit plans on screen on first render

* optimize for mobile and remove background containers

* change default price tag to simply 'N/A'

* fix tests

* Change Paddle.js integration to use JavaScript directly
* rename many variables

* allow users on v1 and v2 plan subscribe to 20M and 50M tiers

* add a test for two months free label

* make it work with a free_10k subscription

* small test improvement and formatting

* change other upgrade link in user settings if FF enabled

* dialyzer

* fix typo

* add test for free_10k user

* silence credo

* mix format

* credo - add moduledoc

* credo - another moduledoc

* handle calls to sentry on the api level

* refactor getting regular subscription plan for LiveView

* post review code style tweaks

* remove unused aliases

* credo - add @moduledoc false to Subscriptions

* crash in cancel_subscription task when Repo update fails

* readability improvements (review suggestions)

* add comment about 'external_resource' module attr

---------

Co-authored-by: Vinicius Brasil <vini@hey.com>
3 years ago
Adrian Gruntkowski e67850c11d
Fix and refactor invitation logic (#3376)
* Make membership creation and role updates more explicit in terms of changesets

* Extract invitation accept flow logic and refactor it slightly

* Improve acceptation logic

* Update moduledoc

* Improve SiteLocker API and add typespecs

* Stop naming function not returning a boolean like a predicate

* Refactor rest of invitation actions and safeguard against rogue requests

* Update code docs slightly

* Extend `Billing.check_needs_update/1` tests

* Parametrize selfhost flag and toggle SiteLocker logic on it

* Add tests for newly extracted services

* Add test case and a fix for locking site on grace period ended

* Make invitation controller tests async as there's no more env patching

* Add test cases for self-invites and fix one bug

* Add and refactor tests for rejecting and removing invitations

* Prevent issuing ownership transfer to existing owner

* Improve name of the test

* Improve `Billing.check_needs_to_upgrade/1` return value

* Improve `Billing.SiteLocker.update_sites_for/1` and its tests

* Fix typos

Co-authored-by: hq1 <hq@mtod.org>

* Make invitation removal and rejection resilient to races

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
hq1 082ec91c63
OpenAPI: first pass on Plugins API - Shared Links (#3378)
* Update depenedencies: OpenAPISpex + cursor based pagination

* Update formatter config

* Add internal server error implementation

* Test errors

* Implement pagination interface

* Implement Plugins API module macros

* Implement Public API base URI

(to be used with path helpers once called from within
forwarded router's scope)

* Implement OpenAPI specs + schemas

* Implement Shared Links context module

* Add pagination and error views

* Add Shared Link view

* Implement Shared Link controller

* Expose SharedLink.t() spec

* Implement separate router for the Plugins API

* Update moduledocs

* Always wrap resource objects with `data`

* Update moduledoc

* Use https://github.com/open-api-spex/open_api_spex/pull/425

due to https://github.com/open-api-spex/open_api_spex/issues/92

* Rely on BASE_URL for swagger-ui server definition

* Fixup goals migration

* Migrate broken goals before deleting dupes

* Remove bypassing test rate limiting for which there's none anyway

* Move the context module under `Plausible.` namespace

* Bring back conn assignment to PluginsAPICase template

* Update test/plausible_web/plugins/api/controllers/shared_links_test.exs

Co-authored-by: Uku Taht <Uku.taht@gmail.com>

* Update renamed aliases

* Seed static token for development purposes

* Delegate Plugins API 500s to a familiar shape

* Simplify with statement

---------

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
3 years ago
Adrian Gruntkowski 777b4b3741
Improve forms (#3380)
* Make client-facing user changesets accept only editable fields

* Add controller test
3 years ago
hq1 311d66b341
Implement Plugins API Token schema (#3370)
* Implement Plugins API Token schema

* Work with domain change grace period

* Do not cast internal data, extend schema with hints
3 years ago
Adrian Gruntkowski 51c1138d0d
Implement better user password validation (#3344)
* Add zxcvbn dependency

* Change password length range requirement from 6-64 to 12-128

* Reimplement register form in LV

* Implement server-side check for password strength

* Add rudimentary strength meter

* Make password input with strength a separate component and improve it

* Fix existing tests to provide strong enough password

* Apply formatting

* Replace existing registration form with new one

* Hide built-in label in `.input` component when none provided

* Crop password to first 32 chars for analysis by zxcvbn

* Add tests for new form components

* Integrate hCaptcha into LV

* Fix existing AuthController tests

* Add tests for Live.RegisterForm

* Hide strength meter when password input is empty

* Randomize client IP in headers during tests to avoid hitting rate limit

* Apply auxilliary formatting fixes to AuthController

* Integrate registration from invitation into LV registration logic

* Fix existing password set and reset forms

* Make `password_length_hint` component more customizable

* Optimize `Auth.User.set_password/2`

* Remove unnecessary attribute from registration form

* Move password set and reset forms to LV

* Add tests for SetPasswordForm LV component

* Add tests for password checks in `Auth.User`

* Document code a bit

* Implement simpler approach to hCaptcha integration

* Update CHANGELOG.md

* Improve consistency of color scheme

* Introduce debounce across all text inputs in registration and password forms

* Fix email input background in register form

* Ensure only single error is rendered for empty password confirmation case

* Remove `/password` form entirely in favor of preferred password reset

* Remove unnecessary `router` option from `live_render` calls

* Make expensive assigns in LV with `assign_new` (h/t @aerosol)

* Accept passwords longer than 32 bytes uniformly as very strong

* Avoid displaying blank error side by side with weak password error

* Make register actions handle errors gracefully

* Render only a single piece of feedback to reduce noise

* Make register and password reset forms pw manager friendly (h/t @cnkk)

* Move registration forms to live routes

* Delete no longer used deadviews

* Adjust registration form in accordance to changes in #3290

* Reintroduce dogfood page path for invitation form from #3290

* Use alternative approach to submitting plausible metrics from LV form

* Rename metrics events and extend tests to account for them
3 years ago
hq1 8c605865b6
Enforce goals unique (#3343)
* Enforce goals unique

* Remove unnecessary alias

* Skip tests that can no longer run anymore

To run, make sure the migration from
priv/repo/migrations/20230914071245_goals_unique.exs
is rolled back.

* Use separate transactions for the migration

* Update priv/repo/migrations/20230914071245_goals_unique.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
hq1 542385feab
Data Migration: rewrite funnels that contain duplicate goals (#3361)
* Add `RewriteFunnelDupes` data fix

To rewrite funnels referencing goals whose names
are the same. This enables us to enforce all goals
within the site unique later on.

* Credo

* Update priv/data_migrations/FunnelDupeGoals/sql/list-funnels-with-dupe-goal-ids.sql.eex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Revert unrelated changes

* Remove dead code

* Update lib/plausible/data_migration/rewrite_funnel_dupes.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible/data_migration/rewrite_funnel_dupes.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
hq1 a5bac43981
[migration] Data fix: rewrite goals for which `event_name` and `page_path` both exist (#3357)
* Rewrite goals for which both event_name and page_path exist.

This might've happened due to the old form only hiding form
fields from the viewport, allowing to submit both values in
certain cases when switching tabs. The tabs behaviour has
been changed as of #3293 but no proper constraints existed.

* Add new check constraint to the goals schema

* Use NOT VALID option for adding the CHECK constraint

* Skip wrapping migration in a single transaction
3 years ago
Uku Taht 7cb2c6bfa3
Allow admins to initiate ownership transfer from the CRM (#3340)
* Allow admins to initiate ownership transfer from the CRM

* Add stronger assertion for bulk invite action

* Fix compile warning

* Move bulk transfer logic to Sites module

* Replaces unused variables with _

* Add typespec for `bulk_transfer_ownership`

* Extract from keywordlist options instead of matching

* Fix and extend bulk transfer tests

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
3 years ago
Adrian Gruntkowski 27a11fc5b7
Filter out empty entries when listing stats for UTM props (#3339)
* Filter out empty entries when listing stats for UTM props

* Update test fixtures removing noref entries in UTM CSV stat exports

* Update external API tests to account for lack of noref records for UTM stats

* Filter out entries with empty UTM props from imported GA stats

* Remove unreachable GA utm_source dim clause in imported stats logic
3 years ago
Vini Brasil 9b029c1558
Improve ownership transfers (#3326)
* Move inline functions to module

* Extend invite/4 for ownership transfers

* Verify inviter has sufficient permissions

* Ensure ownership transfers don't count as team member

This commit changes the team member usage query to exclude ownership
transfer invitations. Previously, when an ownership transfer was
pending, the team member usage was incremented.

* Draw attention to payment notice when transferring ownership

* Remove duplicate mail sending from membership_controller

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
3 years ago
hq1 a1be27eea6
s/superadmin/super_admin (#3318)
* s/superadmin/super_admin

* Add test
3 years ago
Vini Brasil d22c011aa3
Implement limits for team members (#3305)
* Refactor MembershipController.invite_member/2

This commit refactors the controller action used for creating new
invitations. It moves the code to Plausible.Sites.invite/4 and replaces
`ifs` and `cases` with `with`.

* Add team_member_limit to plan definition

* Create usage and limits functions for team members

* Apply team member limit when inviting new users

* Add team members to Usage & Limits section

* Change invite function to receive email address instead of %User{}

* Wrap invite function in a DB transaction

* Remove unnecessary joins from team member usage query

* Replace UNION ALL with UNION to remove duplicates
3 years ago
hq1 b3ff695797
Improve goal settings UX (#3293)
* Add Heroicons dependency

* Add name_of/1 html helper

Currently with Floki there's no way to query for
`[name=foo[some]]` selector

* Update changelog

* Make goal deletion possible with only goal id

* Remove stale goal controllers

* Improve ComboBox component

- make sure the list options are always of the parent input width
- allow passing a suggestion function instead of a module

* Stale fixup

* Update routes

* Use the new goals route in funnel settings

* Use a function in the funnel combo

* Use function in the props combo

* Remove old goals form

* Implement new goal settings

* Update moduledoc

* Fix revenue switch in dark mode

* Connect live socket on goal settings page

* Fixup

* Use Heroicons.trash icon

* Tweak goals search input

* Remove unused alias

* Fix search/button alignment

* Fix backspace icon alignment

* Delegate :superadmin check to get_for_user/3

I'll do props settings separately, it's work in progress
in a branch on top of this one already. cc @ukutaht

* Rename socket assigns

* Fixup to 5c9f58e

* Fixup

* Render ComboBox suggestions asynchronously

This commit:
  - prevents redundant work by checking the socket connection
  - allows passing no options to the ComboBox component,
    so that when combined with the `async` option, the options
    are asynchronously initialized post-render
  - allows updating the suggestions asynchronously with the
    `async` option set to `true` - helpful in case of DB
    queries used for suggestions

* Update tests

* Throttle comboboxes

* Update tests

* Dim the search input

* Use debounce=200 in ComboBox component

* Move creatable option to the top

* Ensure there's always a leading slash for goals

* Test pageview goals with leading / missing

* Make the modal scrollable on small viewports
3 years ago
Vini Brasil 5d9d05c1f6
Usage & Limits section (#3291)
* Update user_settings template to HEEX

* Move site_limit function to Billing.Quota

* Create Billing.Quota.site_usage function

* Move monthly_pageview_limit function to Billing.Quota

* Create Billing.Quota.monthly_pageview_usage function

* Add "Usage & Limits" section to user settings page

* Apply suggestions from code review
3 years ago
Uku Taht 8c870f7c6b Increase sampling test delta to reduce flakyness 3 years ago
Uku Taht ab4eb75109
Inflate sampled results in Stats.Funnel.funnel/3 (#3270)
* Inflate sampled results in Stats.Funnel.funnel/3

* Add test for sampling

* Update test/plausible/funnels_test.exs

Co-authored-by: hq1 <hq@mtod.org>

* Ignore unused binding

* Use bang version of hash function

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
Vini Brasil dad787cb4b
Rename allowance to monthly_pageview_limit (#3273)
* Remove unused site_limit_exempt list

* Rename allowance to monthly_pageview_limit
3 years ago
Vini Brasil 34f1ddfc8c
Refactor Plausible.Billing.Plans module (#3268)
This pull request introduces a series of improvements to Plausible.Billing.Plans, including:

* Tag the JSON file with the plan version
* Rename the JSON field limit to monthly_pageview_limit
* Move site_limit function to Billing.Plans
* Refactor subscription_interval, allowance and site_limit functions
* Remove unused AnalyzePlans task
3 years ago
Vini Brasil 059df171bc
Refactor Plausible.Billing.Plans module (#3249)
This commit introduces a series of improvements on the Plans module including function renaming, documentation and readability. This is the groundwork for billing plans.

There should be no actual changes with this commit, therefore no changes in tests either.
3 years ago
hq1 e467324c5a
Disallow small intervals on large ranges (#3245)
* Update interval dropdown based on custom range selection

* Add `monthsBetweenDates` function

* Reassign interval for custom "period" based on local storage

* Generate intervals by period based on stats_start_date

* Account for "custom" and "all" period intervals dynamically
3 years ago
Vini Brasil 1fedaf18c3
Remove allowed props list from dashboard data attribute (#3230)
This commit removes the allowed props list from the data attribute
passed to React. The list was used for checking whether props configured
or not. This commit adds a new `hasProps` attribute, and removes the
list.
3 years ago
Vini Brasil 59a82288e1
Remove url and path prop keys from suggestions (#3228)
* Remove url prop key from suggestions

This commit prevents the `url` prop key from being suggested by the
props settings page. This prop is used internally for file downloads and
outbound link clicks, and doesn't need to be manually allowed.

* Document Props module
3 years ago
hq1 054d0655af
[pg migration] Delete account with multiple subscriptions (#3220)
* Migrate user deletion

So that subscriptions, google

* Make CRM deletion functional

* Preload subscriptions in an uniform manner
3 years ago
RobertJoonas 17b9e36a19
Refactor: Standardize API metrics used internally (#3208)
* extract add_exit_rate function

* Change internal API metric names for /entry-pages & /exit-pages

* `unique_entrances` -> `visitors`,
* `total_entrances` -> `visits`,
* `unique_exits` -> `visitors`,
* `total_exits` -> `visits`,

This is just a consistency improvement - the `visitors` metric always means
one thing and there's no need to call it different for entry pages internally.

This commit does not change any noticable behavior. The UI labels are kept
the same and the column headers in the CSV export will also remain the same.

* Change internal API metric names for /conversions

* `unique_conversions` -> `visitors`,
* `total_conversions` -> `events`,
3 years ago
RobertJoonas 7b39328d6c
Props details view (#3196)
* make (none) value in custom prop breakdown add +1 to pagination limit

This is needed because the (none) value is always added to the
breakdown_results **after** fetching those from ClickHouse.

* only include (none) values on the first page of results

* fix percentage metric calculation for paginated results

Instead of summing up the number of visitors from the breakdown results
to get the total, we have to make a separate query to `Stats.aggregate`.
Otherwise, the percentages for each results page will wrongly add up to 100%.

Since imported data for aggregated visitors and other properties (such as
browsers, OSs, etc) live in different tables, we have to tweak the tests to
also include the same number of visitors in the `imported_visitors` table.

* add details view for props

* changelog

* exclude imported data from total
3 years ago
Kris Buist 7c5ebab2c6
Fix the interval selection in the all time view (#3110)
Previously, the interval was always overwritten with the default value, which is determined based on the age of the site.
It should not be overwritten when the interval is explicitly passed by the user
3 years ago
Vini Brasil d49d68af8f
Create props settings page (#3191) 3 years ago
hq1 82f356f387
Add debug_replay_info context to Sentry Stats reports (#3189)
* Add debug_replay_info context to Sentry Stats reports

* Test limit
3 years ago
ruslandoga f3a1a9d599
derive jason encoder for request (#3182)
* derive jason encoder for request

* test didn't test

---------

Co-authored-by: hq1 <hq@mtod.org>
3 years ago
RobertJoonas 34fbc3d5bc
New private API to return custom prop values (#3111)
* copy prop_breakdown tests into a separate file

* add a new endpoint for custom props

Duplicate existing goal prop breakdown as a building base for the
new endpoint

* stick to original metric names + CR definitions

We currently use CR with two different definitions, which is inconsistent
and should be changed. This commit just documents the difference for the
time being.

* basic prop breakdown without goal filter

* increase % metric precision to one decimal place

* add some tests without goal filter

* silence credo for TODO comment

* use events metric instead of pageviews

* review feedback

* inline add_cr_a instead
3 years ago
hq1 01fba1f34b
Bump funnel max size to 8 steps (#3094)
* Bump max funnel size to 8 steps

* Improve chart rendering for small screens/large funnels
3 years ago
hq1 ade437c085
Evaluate ephemeral funnels before saving (#3097)
* PoC: evaluate funnel on step selection

* Adjust test to actually _select_ steps

* Throttle and test ephemeral funnel evaluation

* Format compact percentages server-side

At this point it's pointless to duplicate this;
all the primitives exist in the API in case the client
wants a different representation.
3 years ago
hq1 bbedeff683
Implement Funnels view on dashboard (#3066)
* Add Funnel react component

assets/js/dashboard/stats/behaviours/funnel.js - restored from:

98a76cbd Remove console.info calls
d94db99d Convert Funnel class component into a functional one
028036ad Review comments
3067a940 Stop doing maths in react
73407cc3 Fix error handling when local storage gets corrupted
e8c6fc52 Format numbers on funnel labels
c815709f Reorganize component responsibility
7a88fe44 Outline basic error handling
94caed7c Chart styling updates
4514608a Add percentages to funnel
d622c32d Add funnel picker

Co-authored-by: Uku Taht <uku.taht@gmail.com>

* Pass funnels list to react via data-funnels

* Implement Funnels react API

lib/plausible_web/controllers/api/stats_controller.ex - restored from:

f36ad234 Adjust to Plausible.Stats interface
9b532273 Test funnel stats controller
028036ad Review comments
bea3725f Remove IO.inspect
7a88fe44 Outline basic error handling
c8ae3eaf Move Funnels to StatsController and use base query
667cf222 Put private functions at the bottom

* Tweak funnel presentation

* Handle errors at the top

* Do not register DataLabels plugin globally

or else all the existing charts are affected

* Calculate drop-off percentage evaluating funnels

* Tweak dark mode + implement nicer tooltips

* Make currently selected funnel bold in the picker

* Count user_ids not session_ids when evaluating funnels

So if a visitor goes:

1. Start session
2. Complete funnel step 1
3. Inactive for 30 minutes
4. Complete funnel step 2

We would not be able to track this funnel completion because of the session timeout.
We like to o measure this as funnel completion even though the session expired in the middle.

cc @ukutaht

* Add extra properties to the funnels API

cc @ukutaht

* Improve tooltips so that step to data is rendered

* Change tooltip number formatting

* Remove debugging remnants

* Quick & dirty mobile view

* Fix mobile view: tweak dark mode & funnel switching

* Ignore DOMException: the operation was aborted

Otherwise this sometimes flashes the space shuttle
screen when navigating quickly via a keyboard.

* Format percentages on the main chart

* Close missing tag 🙈

* Revert "Close missing tag 🙈"

This reverts commit 9c2f970e22fd7e2980503242b414f42ce8bce1d2.

* Use jsx to render funnel tooltip

To get markup validated via lsp mostly...

* Fixup: s/class/className

* Fix className interpolation

* Add a ruler to the tooltip

* Tweak funnel chart style

* Fix font distortion issue on chart/canvas labels

* s/class/className

* Put "Set up funnels" link behind a feature flag

* Refactor internal selection storage

Getting ready for live funnel evaluation

* Don't try to connect LV socket if there's no CRSF token set up

This is perfectly okay for some of the templates/layouts.

* Fix up funnel creation typespecs

Unfortunately we can't define a type with literal string keys,
hence this must suffice.

* Use uniq over count/distinct

* Revert JSX in tooltips

Ref: https://github.com/plausible/analytics/pull/3066#discussion_r1241891155

* Remove the extra query for counting all visitors

cc @ukutaht

* Add premium notice

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
3 years ago
Vini Brasil 41c88c4fa0
Add unique names to cache supervisors in tests (#3055)
This commit fixes a flaky test where a supervisor with that same name
was already started.
3 years ago
Vini Brasil 8834486a19
Pass cached site struct down the ingestion pipeline (#3027)
* Pass cached site struct down the ingestion pipeline

Revenue goals need the cached site struct during ingestion to get the
goals name and currency. This cache lookup is not necessary as
`GateKeeper.check/1`, which is called first in the ingestion pipeline,
could already return the site struct from the cache.

This commit changes `GateKeeper.check/1` to return the site struct
instead of the site ID. Moreover, this commit changes the ingestion
pipeline to avoid calling the sites cache twice.

Related: https://github.com/plausible/analytics/pull/2957#discussion_r1203921549

* Remove revenue_goals unnecessary fallback

* Change duplicate child_id in cache test

* Remove revenue goal condition from cache query

* Remove Plausible.DataCase.reload/1
3 years ago
hq1 a356009af8
Implement backend core for funnels (#3023)
* Implement Funnel schema

* Implement Funnels context module

lib/plausible/funnels.ex - restored from:

dca93621 Mark Vini's suggestion applied
7d1eec75 Update lib/plausible/funnels.ex
028036ad Review comments
b694ccd5 Express the guard nicely
acd9c4f2 Prepare ephemeral funnel definitions so that users can test funnels
aea4ebc4 Access Funnel min/max steps via the __using__/1 macro
51f0397d Implement deleting funnels
cf228630 wip
1f6fe25d Add number of steps to funnels list
89f10ecb wip
950a18d9 Dirty funnel save
e62bfce5 Refactor funnel result evaluation
88e38a33 Update funnel tests
bf47f654 Calculate conversion rate/dropoff in the backend
9e62ffb1 Make existing tests pass
7a88fe44 Outline basic error handling
c8ae3eaf Move Funnels to StatsController and use base query
2929bfd4 Minor rework
b490403b !ifxup
ae44db77 Add Funnels.list/1 function
4f3216b3 Unique steps
6e0c3efc Ensure step ordering works
cea93ba1 It works I think
c50ac0c2 WIP
a466ec4c Dirty wip
fbd14e23 Absolute dirty wip
3ddd15ad wip
34ceb3f5 dirty wip

test/plausible_web/controllers/api/stats_controller/funnels_test.exs - restored from:

9b532273 Test funnel stats controller

Co-authored-by: Uku Taht <uku.taht@gmail.com>

* Update Goals implementation

lib/plausible/goal/schema.ex - restored from:

028036ad Review comments
acd9c4f2 Prepare ephemeral funnel definitions so that users can test funnels
4a4ddbdc Optionally preload funnels for goals and stub funnel-goal deletion
fbd14e23 Absolute dirty wip

lib/plausible/goals.ex - restored from:

aea4ebc4 Access Funnel min/max steps via the __using__/1 macro
a418c039 Implement funnel reduction logic on goal deletion
4a4ddbdc Optionally preload funnels for goals and stub funnel-goal deletion
ebdc4333 Extend the prompt in case of funnel-goal deletion
950a18d9 Dirty funnel save
f06c3fcf wip

test/plausible/goals_test.exs - restored from:

a418c039 Implement funnel reduction logic on goal deletion
4a4ddbdc Optionally preload funnels for goals and stub funnel-goal deletion
dee5bc9e Fixup tests

* Update seeds

* Split funnel schema modules

* Add moduledoc to Funnel schema

* Update Funnel.Step moduledoc

* Add typespecs to Plausible.Funnels

* Implement Phoenix.HTML.Safe for Funnel schema

* Fixup a test after ordering goals by desc:id

test/plausible_web/controllers/api/stats_controller/conversions_test.exs - restored from:

26a493cb Fixup a test after ordering goals by desc:id

* Annotate Goal schema with type t()

* Reword schema module docs

* Fix up typespecs

* Stop tagging Funnels.get/2 result

* Nuke Goals.by_id!/2

* Test Funnels module

test/plausible/funnels_test.exs - restored from:

b7a6d3ad ya, rly
028036ad Review comments
acd9c4f2 Prepare ephemeral funnel definitions so that users can test funnels
51f0397d Implement deleting funnels
1f6fe25d Add number of steps to funnels list
dee5bc9e Fixup tests
950a18d9 Dirty funnel save
88e38a33 Update funnel tests
bf47f654 Calculate conversion rate/dropoff in the backend
9e62ffb1 Make existing tests pass
2929bfd4 Minor rework
b490403b !ifxup
ae44db77 Add Funnels.list/1 function
4f3216b3 Unique steps
6e0c3efc Ensure step ordering works
cea93ba1 It works I think
c50ac0c2 WIP
a466ec4c Dirty wip
34ceb3f5 dirty wip

Co-authored-by: Uku Taht <uku.taht@gmail.com>

* Apply changes related to 242081e5

* Evaluate funnels through `Stats.funnel/3` entry point

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
3 years ago
Vini Brasil e4d4f7d954
Revenue tracking: Ingestion and breakdown queries (#2957)
* Add revenue fields to ClickHouse events

This commit adds 4 fields to the ClickHouse events_v2 table:

* `revenue_source_amount` and `revenue_source_currency` store revenue in
  the original currency sent during ingestion

* `revenue_reporting_amount` and `revenue_reporting_currency` store
  revenue in a common currency to perform calculations, and this
  currency is defined by the user when setting up the goal

The type of amount fields is `Nullable(Decimal64(3))`. That covers all
fiat currencies and allows us to store huge amounts. Even though
ClickHouse does not suggest using `Nullable`, this is a good use case,
because otherwise additional work would have to be done to
differentiate missing values from real zeroes.

I ran a benchmark with the data pattern we expect in production, where
we have more missing values than real decimals. I created 100 million
records where 90% of decimals are missing. The difference between the
tables in storage is just 0.4Mb.

* Add revenue parameter to Events API

This commit adds support for sending revenue data in ingestion using the
`revenue` parameter - aliased to `$`.

* Add revenue parameter to mix send_pageview

* Add average and total revenue to breakdown queries
3 years ago
Vini Brasil c559370201
Trim prop list to 30 items during ingestion (#3009)
This commit adds an upper bound limit to custom props. It fails silently
and trims the list instead of dropping the event.
3 years ago
Uku Taht fc918081b5
Fix event name validation (#2975)
* Fix event name validation

* Add @moduledoc

* Move tests to Ingestion.RequestTest

* Remove comment
4 years ago
ruslandoga 40e95ffd3d
add LOG_FAILED_LOGIN_ATTEMPTS (#2936)
* add failed login logs

* put failed login attempt logs behind a config option

* add changelog entry

* add config test

* add auth_controller tests

* move tests to separate non-async test module

---------

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
4 years ago
ruslandoga ce7401dd83
add MAILER_NAME (#2937)
* add MAILER_NAME

* add mailer test

---------

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
4 years ago
ruslandoga 7b22aff11c
add MAILGUN_BASE_URI (#2935)
* add MAILGUN_BASE_URI support

* add changelog entry
4 years ago
RobertJoonas dafb60164a
Allow setting sample_threshold in dashboard API queries (#2958)
* refactor Query.from/2

* allow specifying sample_threshold number in query
4 years ago
Vini Brasil 10d9e3b083
Revenue tracking: Add currency field to goal creation (#2948)
* Add revenue goal option to goal creation

This commit adds a currency field to the goals form. Goals that have a
currency set are now revenue goals, and are cached with sites to later
be used during ingestion.

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

* Enable feature flag in tests

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
4 years ago
Uku Taht 2259651b21
Ignore missing subscription on update (#2930) 4 years ago
RobertJoonas fa54efbc6d
Support `with_imported=true` in Stats API aggregate (#2919)
* default to source_query.include_imported before true

This fixes a bug where a Stats API aggregate query was trying
to query current period *without imported data* but previous
period *with imported data*.

* remove __internal_visits at a better time

As soon as we have made the db query, we don't need it anymore

* disallow events metric with imported data

* update changelog

* keep default in a better place

* remove unused option

* update test description

Co-authored-by: Vini Brasil <vini@hey.com>

---------

Co-authored-by: Vini Brasil <vini@hey.com>
4 years ago
Vini Brasil 4ca2e26b9c
Limit ingestion payload size (#2904)
This pull request adds constraints on user input values during ingestion. The purpose of this PR is to filter out unrealistic or invalid inputs. More comments about each limit inline.
4 years ago