Commit Graph

1139 Commits (1cb07efe6dd800c906110858e55a6788cc2cdccf)

Author SHA1 Message Date
Vini Brasil 1cb07efe6d
Save city name when importing from GA (#2608)
This commit adds city data to imported records from Google Analytics. The
current implementation sets city to 0 because GA does not use the GeoNames
database.

Google Analytics Reporting API uses [Geographical IDs](https://developers.google.com/analytics/devguides/collection/protocol/v1/geoid)
to identify cities and countries. Plausible uses
[GeoNames](https://geonames.org/) and I couldn't find databases corelating the
two.

Fortunately, GA also returns the city name and this commit uses the city name
and the country ISO code to find the Geoname ID. To avoid making expensive ETS
searches, I created another ETS table in the Location library that uses
{country, city} as a key.

Related PR: https://github.com/plausible/location/pull/3
4 years ago
Uku Taht 8fcf4d3304
Fix 'field key does not exist' error (#2674) 4 years ago
ruslandoga e9ba60c8b4
improve transfer ownership error message (#2651)
* improve transfer ownership error message

* add changelog

* simplify

* revert changeset invitation error message

* more descriptive error message

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
4 years ago
Adam Rutkowski 8f85b110aa
Split Clickhouse pools into Read-Only and Read/Write (dedicated to writes) (#2661)
* Configure ingest repo access/pool size

If I'm not mistaken 3 is a sane default, the only
inserts we're doing are:

  - session buffer dump
  - events buffer dump
  - GA import dump

And all are serializable within their scopes?

* Add IngestRepo

* Start IngestRepo

* Use IngestRepo for inserts

* Annotate ClickhouseRepo as read_only

So no insert* functions are expanded

* Update moduledoc

* rename alias

* Fix default env var value so it can be casted

* Use IngestRepo for migrations

* Set default ingest pool size from 3 to 5

in case conns are restarting or else...

* Ensure all Repo prometheus metrics are collected
4 years ago
Uku Taht 4494252cce
Add data-domain to tracking script (#2663) 4 years ago
Uku Taht 3b73836136
Fix unknown bind meta (#2654)
* Fix 'unknown bind meta' error

* Changelog

* Use one case instead of nested ifs

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
4 years ago
Uku Taht 1cc5ac0b11
Fix bug with multiple ARRAY JOIN query (#2653)
* Fix bug with multiple ARRAY JOIN query

* Add changelog entry

* Remove 'bug' label from test description

* Simplify pattern match

Co-authored-by: Adam Rutkowski <hq@mtod.org>

* Rename has_join_with_table -> joins_table?

Co-authored-by: Adam Rutkowski <hq@mtod.org>

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
4 years ago
ruslandoga c3e736aae6
add flash ttl (#2652) 4 years ago
Vini Brasil 5159e683cc
First pass at Comparisons (#2650)
This commit adds support for comparing the actual showed period on the main graph with the previous one. This is a first pass and it's hidden under a feature flag because it's not feature complete yet as we want to support other comparison modes.
4 years ago
Adam Rutkowski 8a2df294f5
Replace empty OS/screen size/Browser with `(not set)` in UI/API/exports (#2646)
* Make Device section components aware of (not set)

So that no extra sub-filters are possible when the unset
top item is selected.

* Support '(not set)' in breakdown/filters

* Update expectations for export tests

* Add extra tests for returning/filtering by '(not set)'

* Add changelog entry

* Remove ListReport conditional render

* Prevent redundant sub-filters

* Fix filter text rendering

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
4 years ago
Marko Saric 4ea519fd11
Some changes to the "site taken" copy (#2647)
* Some changes to the "site taken" copy

* Update site.ex

* Fix tests

---------

Co-authored-by: Vinicius Brasil <vini@hey.com>
4 years ago
Adam Rutkowski 482249ffba
Better error message on site creation error (#2644)
* Ensure clear error message on site creation error

* Update colons in API error messages
4 years ago
Adam Rutkowski f678cfab25
Prevent error reports from being sent with short feedback (#2635) 4 years ago
Adam Rutkowski 8f9f032968
Delay stats deletions (#2632)
* Implement Site removal transaction

* Implement Stats removal Oban worker

* Configure site removal queue

* Call Site.Removal.run() instead of Purge.delete_site!

* Test site/stats removal

* Remove FIXME - filed a ticket

* Over-communicate lenghty deletion process to the users
4 years ago
Marko Saric 66c65e0071
Changes to the site settings danger zone (#2629)
* Changes to the site settings danger zone

The idea was to:

1. Show a transfer site ownership message to owners of the site only. This will make ownership transfer feature more discoverable. Will also help people to transfer ownership of a site rather than deleting the whole thing when that makes more sense
2. Add a note that site deletion is a delayed operation
3. Make some other tiny copy changes

What did I break? :)

* Fix markup so the long text wraps sooner

* Fix indent

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
4 years ago
Adam Rutkowski ad12e1ef31
Show user feedback form on server errors (#2617)
* Move Endpoint errors setup to common config

* Implement naive Sentry link resolver

* Implement error report e-mail

* Delete static sentry script

* Implement user feedback form on server errors

* Re-arrange pipe

* Use Sentry.Config.dsn() where applicable

* Fix typo

* Use Map.replace/3
4 years ago
Marko Saric c1d174d0fa
Some changes to the "locked site" copy (#2615)
Some changes to be more consistent with the emails we send. Also "valid" subscription rather than "active" subscription fits better for the different cases where this screen is shown
4 years ago
RobertJoonas bd0de97521
Fix returning more pageviews with a visit property filter (#2612)
* fix subquery for sessions in base_event_query/2

As the 'sessions' table is using the CollapsingMergeTree engine, we have
to select session_id's distinctively. Otherwise we will get multiple rows
(with sign -1 and 1) as long as the background merge hasn't happened.

* update changelog

* use GROUP BY instead of SELECT DISTINCT

* remove comma
4 years ago
ruslandoga e3f095ade7
fix class name typo and include it in safelist (#2610) 4 years ago
Adam Rutkowski 1a4b65c36c
Communicate site creation edge cases to the users (#2605)
* Fingerprint DBConnection.ConnectionError in Sentry

* Check events before creating a site

* Enable sites limit screen

* Remove debugging remnant

* Fix buggy assertions

This wasn't doing what expected:

  iex(1)> Repo.exists?(Plausible.Site, domain: "foo")
  [debug] QUERY OK source="sites" db=0.6ms idle=1906.2ms
  SELECT TRUE FROM "sites" AS s0 LIMIT 1 []

* Encapsulate check to satisfy credo

* Use less technically involved error message

* Bring back e-mail to the limit error message
4 years ago
Vini Brasil 86d5098b31
Stop logging invalid_address errors from Locus (#2604)
This commit stops logging `{:invalid_address, "ip"}` errors. This
reduces noise on Sentry, that is capturing error logs. Other errors are
still logged.
4 years ago
Uku Taht 818d4404a8
Support custom events and props in mix send_pageview (#2606) 4 years ago
ruslandoga 166748dcf2
Replace Geolix with Locus (#2362)
This PR replaces geolix with locus to simplify self-hosted setup. locus can auto-update maxmind dbs which are recommended for self-hosters if they want city-level geolocation. locus is also a bit faster.

This PR also uses a test mmdb file from https://github.com/maxmind/MaxMind-DB for e2e geolocation tests without stubs.
4 years ago
Adam Rutkowski 5ba8697c31
Fingerprint DBConnection.ConnectionError in Sentry (#2597) 4 years ago
Vini Brasil cb5d8a0d25
Make warning notices responsive (#2591)
Closes #2528
4 years ago
RobertJoonas 3999f282a5
Last updated tooltip (#2576)
* extract blinkingDot function

* position pulsating-circle with tailwind instead

* remove unused function

* extract renderStatName function

* display seconds since last realtime update

Adds a 'Last updated X seconds ago' label to the Current Visitors tooltip.

* small refactor: avoid duplication of this.props and this.state

* show the 'last updated ...' tooltip in historical

* changelog update

* use className utility function
4 years ago
Adam Rutkowski 61bf624761
Sentry cleanup (#2590)
* Filter DBConnection logs for clickhouse in Sentry

Removing the never matching clause btw

* Exclude Plug.CSRFProtection.InvalidCSRFTokenError from Sentry

* Turn common Sentry captures into logged warnings

* Exclude fonts from static conf and InvalidPathError from Sentry
4 years ago
Adam Rutkowski 97a3ce72e4
Update clickhousex & fingerprint sentry errors (#2587)
* Revert "Revert "Fingerprint Clickhousex.Error exceptions in Sentry (#2578)" (#2586)"

This reverts commit a78e07236a.

* Bump clickhousex to include https://github.com/plausible/clickhousex/pull/20
4 years ago
Adam Rutkowski a78e07236a
Revert "Fingerprint Clickhousex.Error exceptions in Sentry (#2578)" (#2586)
This reverts commit 2428a2bf36.
4 years ago
Adam Rutkowski 2428a2bf36
Fingerprint Clickhousex.Error exceptions in Sentry (#2578)
* Fingerprint Clickhousex.Error exceptions in Sentry

* Update clickhousex
4 years ago
Marko Saric 934c3c323e
Some changes to the 'dashboard locked' email copy (#2584) 4 years ago
Marko Saric 5ba446e87c
Some changes to the 'over limit' email copy (#2583) 4 years ago
Adam Rutkowski 4bde884da8
Self-host Sentry user feedback script (#2579)
* Add static, minified sentry bundle

OG source: https://browser.sentry-cdn.com/5.9.1/bundle.min.js

* On 500s, load our self-hosted sentry bundle
4 years ago
RobertJoonas 13a05b4e1e
Change metric labels for realtime with goal filter (#2560)
* change metric labels for realtime with goal filter

* changelog

* fix formatting
4 years ago
Uku Taht 51fc9c0945
Refine parallel application-side joins (#2564)
* Consolidate task timeouts, incrase to 15s

* Use Task.async_stream for parallel Clickhouse queries

* Propagate Opentelemetry context to child Task process
4 years ago
Vini Brasil 4503895d0a
Fix breakdown API pagination when using event metrics (#2562)
* Fix breakdown API pagination when using event metrics

This commit fixes a bug where the subsequent breakdown API pages had
the same items as the first page. The fix sorts the underlying
ClickHouse query by timestamp, keeping the same order between requests,
as we use OFFSET/LIMIT pagination.

* Fix repeated results assertion

* Add different ORDER BY to each breakdown property
4 years ago
Adam Rutkowski 44afbaa2f9
Wrap paddle actions with DB transactions (#2558)
* Wrap paddle actions with DB transactions

* Bow to credo

* Make the paddle transactions crash when applicable

* s/changeset/details
4 years ago
Uku Taht 1785653b1e
Ignore unknown countries (#2556)
* Ignore XX and T1 countries

* Add fallback if country_code=nil

* Lookup city overrides directly in CityOverrides module

* Changelog

* Add empty moduledoc

* Remove redundant comment
4 years ago
Vini Brasil 22e2ae1844
Add OTEL tracing to breakdown functions (#2544)
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
4 years ago
Vini Brasil 10a9767912
Remove ingestion requests and Oban queries from OTEL (#2543) 4 years ago
Vini Brasil 5152e8d416
Reject events with long URIs and data URIs (#2536)
* Reject events with data URIs

* Reject events with URIs longer than 2,000 characters

* Update CHANGELOG.md
4 years ago
Vini Brasil 0e87b489f6
Return empty list when breaking down by event:page without events (#2530)
* Return empty list when breaking down by event:page without events

This commit fixes a bug with pagination where breaking down by event:page
would always return results despite pagination.

Closes #2255

* Update CHANGELOG.md
4 years ago
Uku Taht a37b3433d7
Always show direct traffic in sources reports (#2531)
* Remove show_noref behaviour

Removes query param show_noref which was used from React to control
whether to show Direct / None traffic or not. The show_noref behaviour
was untested previously.

Closes #2523

* Add changelog entry

* Fix tests

* Removed files I did not mean to check in :)
4 years ago
Vini Brasil 478e0c6990
Handle GA authentication errors (#2505) 4 years ago
Vini Brasil 9b32dace3e
Add Content-Security-Policy to favicon Plug (#2489) 4 years ago
Serafeim Papastefanos fd6ebf7598
Trim the value in case it contains newlines (#2480)
### Changes

Hello friends, I'm trying to self-host plausible on my server and have
observed that if I add a file named (for example) `secrets/LISTEN_IP` to
configure the bind IP I'll get the following error:

```
Compiling 1 file (.ex)
** (RuntimeError) Invalid LISTEN_IP '172.19.130.83
' error: :einval
    /home/serafeim/plausible/config/runtime.exs:20: (file)
```

(notice that I get the same error when configuring the DATABASE_URL or
whatever comes first)

It seems that for whatever reason the File.read! will insert a newline
at the end of the file. Notice that I have double checked the file with
vim and *it does not contain a newline at the end* or at least I don't
know *how* to make the file not have a newline at the end.

Thus I'm adding a `String.trim()` there to fix that thing; even if I am
doing something wrong and my files *do* contain a newline at the end,
adding a `String.trim()` there definite won't hurt :)

Below you'll find a checklist. For each item on the list, check one
option and delete the other.

### Tests
- [ ] Automated tests have been added
- [x] This PR does not require tests

### Changelog
- [ ] Entry has been added to changelog
- [x] This PR does not make a user-facing change

### Documentation
- [ ] [Docs](https://github.com/plausible/docs) have been updated
- [x] This change does not need a documentation update

### Dark mode
- [ ] The UI has been tested both in dark and light mode
- [x] This PR does not change the UI
4 years ago
Vini Brasil 90786cb3bc
Remove intervals feature flag (#2474)
This commit removes the intervals feature flag, as this feature has
already been released to 100% of the users.
4 years ago
Adam Rutkowski 467317f24e
Fix: application startup, event timestamps (#2479)
### Changes

This PR:
- pushes PromEx to the bottom of supervision stack to avoid Endpoint
instrumentation failure
- ensures the site cache is ready by exposing it through the health
check endpoint
- fixes event timestamps being calculated at compile time, with
regression unit and integration tests

### Tests
- [x] Automated tests have been added
- [ ] This PR does not require tests

### Changelog
- [ ] Entry has been added to changelog
- [x] This PR does not make a user-facing change

### Documentation
- [ ] [Docs](https://github.com/plausible/docs) have been updated
- [x] This change does not need a documentation update

### Dark mode
- [ ] The UI has been tested both in dark and light mode
- [x] This PR does not change the UI
4 years ago
Adam Rutkowski 356575ef78
Gatekeep ingestion pipeline (#2472)
* Update Sites.Cache

So it's now capable of refreshing most recent sites.
Refreshing a single site is no longer wanted.

* Introduce Warmer.RecentlyUpdated

This is Sites Cache warmer that runs only for
most recently updated sites every 30s.

* Validate Request creation early

* Rename RateLimiter to GateKeeper and introduce detailed policies

* Update events API tests - a provisioned site is now required

* Update events ingestion tests

* Make limits visible in CRM Sites index

* Hard-deprecate DOMAIN_BLACKLIST

* Remove unnecessary clause

* Fix typo

* Explicitly delegate Warmer.All

* GateKeeper.allwoance => GateKeeper.check

* Instrument Sites.Cache measurments

* Update send_pageview task to output response headers

* Instrument ingestion pipeline

* Credo

* Make event telemetry test a sync case

* Simplify Request.uri/hostname handling

* Use embedded schema, apply action and rely on get_field
4 years ago
ruslandoga 77cca04417
fix ownership transfer when is_selfhost=true (#2455)
* fix ownership transfer when is_selfhost=true

* add changelog entry
4 years ago