Commit Graph

513 Commits (jsx-conditional-rendering)

Author SHA1 Message Date
Adam Rutkowski 523b114682
Allow super-admin Stats API access for locked sites (#2308) 4 years ago
Vinicius Brasil beabef7269
Redirect user after upgrading only when subscription is created (#2285)
* Create API to check for an active subscription

* Redirect after upgrading only when subscription is created

* Change upgrade page copy

* Add timeout notice to upgrade success page
4 years ago
Adam Rutkowski bf2b6880c5
Block Stats API for locked sites (#2302)
* Extract Sites.locked? predicate

* Lock Stats API when dashboard is locked

* Tidy tests

* Don't pollute application env from tests

* Add changelog entry

* Revert "Add changelog entry"

This reverts commit 76346074f9.
4 years ago
Uku Taht 0bc762bfdc
Fix svg favicons (#2295)
* Overrides content-type for SVG favicons

* Organize favicon rendering

Make sure the placeholder icon is always requested from
/favicon/sources/placeholder

* Run prettier on site-switcher.js

* Yak Shave: upgrade Heroicons to 2.0

* Use HeroIcons instead of custom svg

* Update lib/plausible_web/plugs/favicon.ex

Co-authored-by: Adam Rutkowski <hq@mtod.org>

Co-authored-by: Adam Rutkowski <hq@mtod.org>
4 years ago
Uku Taht 12a513f3cd
Remove Mimic dependency (#2280) 4 years ago
Uku Taht 4096d88171
[Sentry APP-189] Fallback to current date in all time parsing (#2277)
* Fallback to current date in all time parsing

* Add changelog
4 years ago
Uku Taht 65f0ce3061
[Sentry APP-5B] Ensure form errors can be rendered on /settings (#2278)
* Ensure form errors can be rendered on /settings

* Changelog
4 years ago
Uku Taht d104abb53d
Add fallback for favicon (#2279)
* Add fallback for favicon

* Add Favicon tests

* Changelog

* Move placeholder icon to priv folder
4 years ago
Vinicius Brasil 7489290d11
International Domain Names (IDN) Support (#2034)
* Accept letters from non-Latin alphabets in domain names

* Replace static URLs with Router functions in settings_visibility

* Beautify dashboard URL in visibility tab

* Add IDN support to CHANGELOG
4 years ago
Vinicius Brasil c07361fbce
Increase GA import request receive_timeout to 30s (#2274)
This commit increases the receive_timeout from 15s to 30s for Google
Analytics import requests. Timeouts have been reported, and I could
replay some requests that took 15-20 seconds to complete when fetching
10,000 records in a single request.
4 years ago
RobertJoonas 155e274150
Stats API: allow escaping `|` literal character with `\|` when filtering (#2266)
* add separate module for filter parsing

* add tests for filter parser

* allow escaping pipe character in filter value

* add documentation and doctests

* do not remove escape chars from wildcard values

* changelog update

* change the parse_filters/1 function argument
4 years ago
Vinicius Brasil db03cc0ac6
Fix timezone list when time changes (#2268)
This commit fixes an exception when calling `Timex.Timezone.get/2` after
there has been a timezone change. Similar issues have already been reported
in Timex (bitwaker/timex#691, bitwalker/timex#555, bitwalker/timex#625).

This skips timezones from the list when they fail to fetch, and use
`Timex.Timezone.get/2` to get the current offset using `DateTime.utc_now/0`
instead of the `NaiveDateTime` default.
4 years ago
Vinicius Brasil 1adda42a75
Refresh Google Analytics token before import (#2254)
* Capture refresh and expires from GA callback

* Pass GA refresh token to import worker

* Refresh GA token before import
4 years ago
Uku Taht e849e03058
Fix favicons (#2257) 4 years ago
RobertJoonas 80e01fedd5
do not show invoices for free subscription (#2249)
* do not show invoices for free subscription

* use refute instead of negative assert

* changed to refute in the other test too
4 years ago
Vinicius Brasil 2b753cdcfd
Improve GA Imports error reporting (#2244)
* Return error tuple instead of exception when GA request fails

* Report Google Analytics failed request body to Sentry

* Improve Google Analytics function @doc

* fixup! Report Google Analytics failed request body to Sentry

* fixup! Improve Google Analytics function @doc
4 years ago
Uku Taht e16e357dd2
Fix shared link with bad auth (#2225)
* Render 404 when shared link cannot be found

* Add documentation for StatsController and shared link rendering

* Refactor shared_link/2 for more clarity

* Add changelog entry

* Use mermaid graph for sequence diagram

* Use more accurate return value in sequence diagram

* Refactor Ecto query to be more idiomatic

* Remove order dependence in test

* Restore backwards compatibility for older shared links

* Add changelog entry
4 years ago
Uku Taht 669091f2ef
Ignore unknown country in imported data (#2247) 4 years ago
Vinicius Brasil 40182f6d6e
Manually lock outgrown enterprise accounts (#2197)
* Rename enterprise_over_limit e-mail template

* Change suggested_plan/2 function to suggest enterprise plans

* Start grace period for enterprise accounts

* Create separate module for GracePeriod schema

* Enable manual locking of enterprise accounts

* Update outgrown account alert for enterprise accounts

* Document account locking feature

* fixup! Enable manual locking of enterprise accounts
4 years ago
Adam Rutkowski 75264f8f1c
Treat non-200 HTTP responses as errors (#2243)
* Tag non-200 HTTP responses as errors

Co-authored-by: RobertJoonas <robertjoonas16@gmail.com>

* Send get_invoices/1 errors to Sentry

Co-authored-by: RobertJoonas <robertjoonas16@gmail.com>

* Keep Google API module matching non-200 responses

Co-authored-by: RobertJoonas <robertjoonas16@gmail.com>

* Make sure HTTPClient.Error.t() doesn't appear in the UI

* Unify get_invoices/1 signature

Co-authored-by: RobertJoonas <robertjoonas16@gmail.com>

Co-authored-by: RobertJoonas <robertjoonas16@gmail.com>
4 years ago
Uku Taht 5d4918b66b
Allow admins to grant admin rights (#2216)
* Make sure admin can grant the admin role

* Use `site` and `current_user_role` from conn.assigns to avoid extra lookups

* Add some missing documentation

* Restrict owners in the `update_role` action

* Move @moduledoc to appropriate location

* Use Ecto.Enum to get role atoms

* Remove unused functionality

* Ensure that owners cannot lock themselves out

* Add question mark to `can_grant_role`

Co-authored-by: Adam Rutkowski <hq@mtod.org>

* Use strict short-circuit operator for booleans

Co-authored-by: Adam Rutkowski <hq@mtod.org>

Co-authored-by: Adam Rutkowski <hq@mtod.org>
4 years ago
Uku Taht b239f73a6d
Ignore unknown country code (#2223)
* Ignore unknown country code

* Add changelog entry
4 years ago
RobertJoonas 0194d57b38
add upper bound to limit parameter (#2226)
* add upper bound to limit parameter

* add more strict validation for limit parameter + 2 tests

* update changelog
4 years ago
Vinicius Brasil 178e34b9e7
Fix timezone input (#2222)
* Update Timex version from 3.7.7 to 3.7.8

* Generate timezone list from Tzdata

This commit fixes a bug where timezone changes weren't updating the
timezone list displayed when editing or creating a site.

Timezones were being pulled from a static list. This commit changes it
to generate the list from Tzdata, that uses a timezone database with
updated information on time changes. Additionally it adds more timezones
with aliases and links to the list.

Closes #1340

* Use timezone name from browser to recommend timezone

This commit matches the timezone name instead of offset to recommend a
timezone when creating a new site. The JavaScript Intl.DateTimeFormat
API is widely supported according to the link. In any case, if the
timezone fails to match by name, it fallbacks to the offset strategy.

https://caniuse.com/mdn-javascript_builtins_intl_datetimeformat_resolvedoptions_computed_timezone

Closes #904
4 years ago
Adam Rutkowski cc4d4bb8e0
Fixup error reporting (ref #2181) (#2204)
* Fixup error reporting (ref #2181)

Fixes two problems:

 - `extra` sent to Sentry is always expected to be a map
 - no `String.Chars` protocol is implemented for `Finch.Error` struct

* Use explicit paths in http client tests

* Make HTTP Client tests sync
4 years ago
Vinicius Brasil d31db86b49
List all Google Analytics views during import (#2184)
* List all Google Analytics views during import

This commit fixes a bug where different Google Analytics views with the
same name and URI were not shown. This was caused because GA views were
stored as a map, that naturally doesn't support duplicate keys.

This change updates the GA views list to display view IDs, making it
clearer to know what is being imported. The dropdown is now grouped by
website URL.

* Put Google Analytics API URLs in app env

* Add controller test to GA view list
4 years ago
Adam Rutkowski e2aa519c31
Test HTTPClient (#2185)
* Sort dependencies

* Add :bypass dependency

* Test HTTPClient

* Rename test module
4 years ago
Adam Rutkowski 9c23e886aa
Merge pull request #2174 from plausible/fix-flaky-test-assuming-order
Do not assume any order in browser data import test
4 years ago
Adam Rutkowski 79b4550bed Do not assume any order in browser data import test 4 years ago
Adam Rutkowski 2275cd8e22 Respond with 400 when no auth supplied to shared link request 4 years ago
Vinicius Brasil bc68235d04
Add fallback for GA reports without rows (#2121) 4 years ago
Vinicius Brasil a10d44a0d7
Refactor event struct creation function (#2098)
* Replace Ingestion.Request headers with user_agent

* Replace generic Ingestion.Request params with specific fields

* Refactor event building function into small functions

* Move Plausible.Ingestion to Plausible.Ingestion.Event

* Add option to override event fields while building

* Rename Ingestion.Request meta to props

* Replace UTM-specific fields with generic query_params

* Remove Map.from_struct/1 call from ingestion pipeline

* Remove stash options from ingestion
4 years ago
Vinicius Brasil 4d20c7ce70
Catch Google Search Console grant error (#2101)
* Remove invalid Jason.decode argument

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

* Add custom message to Google invalid grant error

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

* Test invalid_grant while refreshing Google token

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
4 years ago
Uku Taht a5757bc61e Allow ownership transfers 4 years ago
Vinicius Brasil 6b8ed12567
Fix Google Search Console API call (#2090)
This commit fixes a bug where fetching Google Search Console keywords
raised a FunctionClauseError. This was introduced in #2046. This commit
also adds test coverage.
4 years ago
ruslandoga ce461b5192 Auth updates 4 years ago
RobertJoonas a058cf6240
added a test and changed hashmode condition (#2082) 4 years ago
Vinicius Brasil 4b9032d822
Google Analytics Import Refactor (#2046)
* Create separate module for GA HTTP requests

* Fetch GA data entirely instead of monthly

* Add buffering to GA imports

* Change positional args to maps when serializing from GA

* Create Google Analytics VCR tests
4 years ago
Igor Kapkov 61fe38e0d4
Fix script path for k6 load test (#2068) 4 years ago
Vinicius Brasil b415ebe776
Fix geolocation subdivision pattern matching (#2063)
* Fix geolocation subdivision pattern matching

This commit fixes a bug where regions were not being saved. This was
caused because Geolix response was returning an additional
`:geolocation` map key. It also adds a test case for this.

Closes #2033

* Add geolocation database to .gitignore
4 years ago
Vinicius Brasil b5ea6ae3dc
Keep user filter when listing cities, countries, and regions stats (#2030)
This commit fixes a bug where location filters were filtering stats but
not the locations list. This was caused by a `Map.put/3` call that
overrides the user filter. This commit rollbacks 5b57143273
changes and removes the `Map.put/3` call.

Closes #1982
4 years ago
Sasha Fonseca db3a852f26
Fix missing 'pageviews' column from 'pages.csv' (#1988)
* Fix missing 'pageviews' column from 'pages.csv'

* Add CHANGELOG entry for Issue #1878 fix
4 years ago
Uku Taht 292a419473
Add feature flag to block event ingest (#1991)
Event ingest can be blocked using the flag `block_event_ingest`
4 years ago
Uku Taht 3e5695408a
Use new Session.CacheStore in favour of Session.Store (#1934)
* Remove Session.Store in favour of Session.CacheStore

* Add CHANGELOG entry

* Use appropriate enum function
5 years ago
Kurt McAlpine b92889aaf0
Add API to retrieve site by domain (#1942) 5 years ago
Uku Taht 102ff1885e Trim trailing whitespace from pathname 5 years ago
RobertJoonas 11654ddc07
Script extension additions (#1915)
* added data-include attribute to plausible.exclusions.js

* reorder extensions in filename when serving the plausible script

* fix formatting

* tweaks after review

* changelog update
5 years ago
Uku Taht 948de2b4c9 Mark more tests as async 5 years ago
Uku Taht cd8df23e5e Do not rely on insertion order in tests 5 years ago
Uku Taht 6f3399bb7c Fix annual notification loop 5 years ago
Uku Taht e4816e8f64 Update Elixir version 5 years ago
Uku Taht 02abbda06d Ignore nil custom prop as well 5 years ago
Uku Taht c92c548ca8 Do not error on empty string custom prop value 5 years ago
Uku Taht f18a211dcc
Ingest throughput improvement test setup (#1867)
* Add OTEL and test Cachex for sessions

* Move load test

* Start apps in the appropriate order
5 years ago
Uku Taht ce55dc4ae9
Retries Google Analytics API request if it returns empty rows (#1858)
* Retries Google Analytics API request if it returns empty rows

* Remove HTTPoison.Retry completely
5 years ago
Uku Taht 5e415c2420 Add entry_props back in 5 years ago
Uku Taht 8fb4f3f886 Revert entry props 5 years ago
RobertJoonas 40275b64d4
Pageview custom dimensions (#1816)
* added custom dimension filtering tests for pages

* first filter UI in place

* pages, entry pages and exit pages can be filtered by pageview props

* added tests for expected filtering behaviour

* fix dimension filter for sources + tests

* added is_not filtering functionality

* fixed formatting

* fixed admin_test

* added (none) as filter value + is_not filter type in UI

* added prefilling applied filter values and some UI tweaks

* added fetch options

* Make prop suggestions work with `props` filter

* Fix test

* Track login state internally

* Add CHANGELOG entry

Co-authored-by: Uku Taht <uku.taht@gmail.com>
5 years ago
Uku Taht 66aceda9e9 Fix main graph selection with imported data 5 years ago
Uku Taht 404a5d15a3 Fix tests 5 years ago
Uku Taht 05c210ad85 Enforce site limit 5 years ago
Uku Taht 3d231fc094 Redirect to correct change plan page for enterprise customers 5 years ago
Andrea Mazzarella 1128ff4bfa
Update the internal /sites api to paginate results (#1824)
* Update the internal /sites api to paginate results and adapts site-switcher to it

* Update the Changelog

* Format internal controller

* Remove the `+ Add Site` link from the site-switcher in the dashboard

* Change camel to snake case and replace imports with fully qualified calls

* Remove trailing comma from site-switcher
5 years ago
Uku Taht d2dc06ec96 Fix trial notifications 5 years ago
Vignesh Joglekar 3b97ecdc62
Adds Main Graph Metric Selection (#1364)
* First pass bringing in previous graph improvements, and comparsion context

* Swaps issue template to new issue form syntax

* Indentation update

* Indentation update?

* More indentation

* Intendation is hard

* Finalized indentation?

* Github indentation

* Missing fields

* Formatting changes

* Checkbox changes

* Uses new timeseries API, various UI improvements, descopes conversions, ToP from graphing

* Fixes Mobile UI Issues

* Improves point detection and display on hover

* Fixes & adds tests for updated main-graph API route

* Changelog

* Changes to better metric option declaration & minor UI/default fixes

* Fixes top stat tooltips showing unformatted numbers for special (non-rounded) top stats

* Formatting

* Fixes regression with dashed portion not stopping at present_index

* Removes comparison + lint

* Improves top stat active style

* Removes comparison tests

* Splits out tooltip and top stats

Still needs:
- Tests
- Potentially more cleanup

* Adds/moves tests for top stats

* Formatting

* Updates metric LS key, removes console log

* Various fixes + cleanup

* Makes tooltip position & style more consistent

* Fixes test (returns import status on both main graph & top stats)

* Fixes interaction with month dateFormatter

* Fixes edge case tooltip behavior

It was simpler than I thought :/

* Make the entire top stat clickable

* Minor UI improvements

* Fixes another tooltip visibility edge case + cleans up boolean algebra

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
5 years ago
Kian-Meng Ang 70cb3eed54
Fix typos (#1829) 5 years ago
Uku Taht 06b165eb6d Run GA import in monthly batches 5 years ago
Uku Taht 7c755d7c6d Forget has_stats 5 years ago
Uku Taht a359271af2 Fix more tests 5 years ago
Uku Taht 25baa6bae9 Fix tests 5 years ago
Uku Taht 333de87ceb Add stats_start_date field 5 years ago
Uku Taht 9308092184 Fix test 5 years ago
Uku Taht d40faf6ec7 More datepicker improvements 5 years ago
Uku Taht 7f58e6be4e Add Year to Date option in datepicker 5 years ago
Uku Taht 8467979051 Add All Time option to datepicker 5 years ago
Uku Taht ed871e7a0b Parse scientific notation in GA import 5 years ago
Uku Taht 8719865a41 Redirect user when they've already upgraded 5 years ago
Uku Taht ce649966ed Fix display 5 years ago
Uku Taht ebdd856df7 Fix tests 5 years ago
Andrea Mazzarella 3c93a2d91b
1421 contains filter (#1799)
* small refactors and adds 'contains' to modal

* supports contains filter in the backend

* moves entry and exit page under the page filter

* prettier

* updates the CHANGELOG

* undo package-lock changes

* fixes formatting for elixir

* renames unused parameter to _

* Update changelog

* Use uppercase for constants and update type/prefix lookup
5 years ago
RobertJoonas e5cf800d4a
Improve stats api (#1759)
* can now query events metric when breaking down on a visit prop

* updated changelog

* fixed formatting

* fixed a bug - querying for bounce rate does not return visits anymore

* removed the changelog update
5 years ago
RobertJoonas 492f47ba1e
Crm transfer data (#1749)
* pull from master

* added query generation by struct fields

* ready, improved tests

* fixed a naming mistake
5 years ago
Uku Taht 51cd24bcaf Allow user to cancel ongoing import 5 years ago
Uku Taht 203f87520b
Ga import improvements (#1784)
* Do not link Google account for import

* Record start date

* Fix tests
5 years ago
Uku Taht 59acff02f8 Fix tests 5 years ago
Uku Taht e1d6d119b0 Add end date to GA import 5 years ago
Uku Taht e026148fb0 Show better error message on duplicate domain
Co-authored-by: Marko Saric <me@markosaric.com>
5 years ago
Uku Taht e27734ed79
[Continued] Google Analytics import (#1753)
* Add has_imported_stats boolean to Site

* Add Google Analytics import panel to general settings

* Get GA profiles to display in import settings panel

* Add import_from_google method as entrypoint to import data

* Add imported_visitors table

* Remove conflicting code from migration

* Import visitors data into clickhouse database

* Pass another dataset to main graph for rendering in red

This adds another entry to the JSON data returned via the main graph API
called `imported_plot`, which is similar to `plot` in form but will be
completed with previously imported data.  Currently it simply returns
the values from `plot` / 2. The data is rendered in the main graph in
red without fill, and without an indicator for the present. Rationale:
imported data will not continue to grow so there is no projection
forward, only backwards.

* Hook imported GA data to dashboard timeseries plot

* Add settings option to forget imported data

* Import sources from google analytics

* Merge imported sources when queried

* Merge imported source data native data when querying sources

* Start converting metrics to atoms so they can be subqueried

This changes "visitors" and in some places "sources" to atoms. This does
not change the behaviour of the functions - the tests all pass unchanged
following this commit. This is necessary as joining subqueries requires
that the keys in `select` statements be atoms and not strings.

* Convery GA (direct) source to empty string

* Import utm campaign and utm medium from GA

* format

* Import all data types from GA into new tables

* Handle large amounts of more data more safely

* Fix some mistakes in tables

* Make GA requests in chunks of 5 queries

* Only display imported timeseries when there is no filter

* Correctly show last 30 minutes timeseries when 'realtime'

* Add with_imported key to Query struct

* Account for injected :is_not filter on sources from dashboard

* Also add tentative imported_utm_sources table

This needs a bit more work on the google import side, as GA do not
report sources and utm sources as distinct things.

* Return imported data to dashboard for rest of Sources panel

This extends the merge_imported function definition for sources to
utm_sources, utm_mediums and utm_campaigns too. This appears to be
working on the DB side but something is incomplete on the client side.

* Clear imported stats from all tables when requested

* Merge entry pages and exit pages from imported data into unfiltered dashboard view

This requires converting the `"visits"` and `"visit_duration"` metrics
to atoms so that they can be used in ecto subqueries.

* Display imported devices, browsers and OSs on dashboard

* Display imported country data on dashboard

* Add more metrics to entries/exits for modals

* make sure data is returned via API with correct keys

* Import regions and cities from GA

* Capitalize device upon import to match native data

* Leave query limits/offsets until after possibly joining with imported data

* Also import timeOnPage and pageviews for pages from GA

* imported_countries -> imported_locations

* Get timeOnPage and pageviews for pages from GA

These are needed for the pages modal, and for calculating exit rates for
exit pages.

* Add indicator to dashboard when imported data is being used

* Don't show imported data as separately line on main graph

* "bounce_rate" -> :bounce_rate, so it works in subqueries

* Drop imported browser and OS versions

These are not needed.

* Toggle displaying imported data by clicking indicator

* Parse referrers with RefInspector

- Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual
  referrer host + path, whereas 'ga:source' includes utm_mediums and
  other values when relevant.
- 'ga:fullReferror' does however include search engine names directly,
  so they are manually checked for as RefInspector won't pick up on
  these.

* Keep imported data indicator on dashboard and strikethrough when hidden

* Add unlink google button to import panel

* Rename some GA browsers and OSes to plausible versions

* Get main top pages and exit pages panels working correctly with imported data

* mix format

* Fetch time_on_pages for imported data when needed

* entry pages need to fetch bounces from GA

* "sample_percent" -> :sample_percent as only atoms can be used in subqueries

* Calculate bounce_rate for joined native and imported data for top pages modal

* Flip some query bindings around to be less misleading

* Fixup entry page modal visit durations

* mix format

* Fetch bounces and visit_duration for sources from GA

* add more source metrics used for data in modals

* Make sources modals display correct values

* imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration

* Merge imported data into aggregate stats

* Reformat top graph side icons

* Ensure sample_percent is yielded from aggregate data

* filter event_props should be strings

* Hide imported data from frontend when using filter

* Fix existing tests

* fix tests

* Fix imported indicator appearing when filtering

* comma needed, lost when rebasing

* Import utm_terms and utm_content from GA

* Merge imported utm_term and utm_content

* Rename imported Countries data as Locations

* Set imported city schema field to int

* Remove utm_terms and utm_content when clearing imported

* Clean locations import from Google Analytics

- Country and region should be set to "" when GA provides "(not set)"
- City should be set to 0 for "unknown", as we cannot reliably import
  city data from GA.

* Display imported region and city in dashboard

* os -> operating_system in some parts of code

The inconsistency of using os in some places and operating_system in
others causes trouble with subqueries and joins for the native and
imported data, which would require additional logic to account for. The
simplest solution is the just use a consistent word for all uses. This
doesn't make any user-facing or database changes.

* to_atom -> to_existing_atom

* format

* "events" metric -> :events

* ignore imported data when "events" in metrics

* update "bounce_rate"

* atomise some more metrics from new city and region api

* atomise some more metrics for email handlers

* "conversion_rate" -> :conversion_rate during csv export

* Move imported data stats code to own module

* Move imported timeseries function to Stats.Imported

* Use Timex.parse to import dates from GA

* has_imported_stats -> imported_source

* "time_on_page" -> :time_on_page

* Convert imported GA data to UTC

* Clean up GA request code a bit

There was some weird logic here with two separate lists that really
ought to be together, so this merges those.

* Fail sooner if GA timezone can't be identified

* Link imported tables to site by id

* imported_utm_content -> imported_utm_contents

* Imported GA from all of time

* Reorganise GA data fetch logic

- Fetch data from the start of time (2005)
- Check whether no data was fetched, and if so, inform user and don't
  consider data to be imported.

* Clarify removal of "visits" data when it isn't in metrics

* Apply location filters from API

This makes it consistent with the sources etc which filter out 'Direct /
None' on the API side. These filters are used by both the native and
imported data handling code, which would otherwise both duplicate the
filters in their `where` clauses.

* Do not use changeset for setting site.imported_source

* Add all metrics to all dimensions

* Run GA import in the background

* Send email when GA import completes

* Add handler to insert imported data into tests and imported_browsers_factory

* Add remaining import data test factories

* Add imported location data to test

* Test main graph with imported data

* Add imported data to operating systems tests

* Add imported data to pages tests

* Add imported data to entry pages tests

* Add imported data to exit pages tests

* Add imported data to devices tests

* Add imported data to sources tests

* Add imported data to UTM tests

* Add new test module for the data import step

* Test import of sources GA data

* Test import of utm_mediums GA data

* Test import of utm_campaigns GA data

* Add tests for UTM terms

* Add tests for UTM contents

* Add test for importing pages and entry pages data from GA

* Add test for importing exit page data

* Fix module file name typo

* Add test for importing location data from GA

* Add test for importing devices data from GA

* Add test for importing browsers data from GA

* Add test for importing OS data from GA

* Paginate GA requests to download all data

* Bump clickhouse_ecto version

* Move RefInspector wrapper function into module

* Drop timezone transform on import

* Order imported by side_id then date

* More strings -> atoms

Also changes a conditional to be a bit nicer

* Remove parallelisation of data import

* Split sources and UTM sources from fetched GA data

GA has only a "source" dimension and no "UTM source" dimension. Instead
it returns these combined. The logic herein to tease these apart is:

1. "(direct)" -> it's a direct source
2. if the source is a domain -> it's a source
3. "google" -> it's from adwords; let's make this a UTM source "adwords"
4. else -> just a UTM source

* Keep prop names in queries as strings

* fix typo

* Fix import

* Insert data to clickhouse in batches

* Fix link when removing imported data

* Merge source tables

* Import hostname as well as pathname

* Record start and end time of imported data

* Track import progress

* Fix month interval with imported data

* Do not JOIN when imported date range has no overlap

* Fix time on page using exits

Co-authored-by: mcol <mcol@posteo.net>
5 years ago
RobertJoonas b4992cedc1
Referrer spam blocklist (#1750)
* integrating blocklist library

* loads blocklist dependency from Github
5 years ago
RobertJoonas 6b22806e1f
Allow admin access to locked dashboards (#1710)
* added super-admin access to locked dashboards

* fixed formatting
5 years ago
Uku Taht b0b9b9212b Fixes password authentication issue with shared links 5 years ago
Uku Taht 5b893bde39 Revert "Include 0 length pageviews in time_on_page calculation"
This reverts commit cee79e7875.
5 years ago
Uku Taht cee79e7875 Include 0 length pageviews in time_on_page calculation 5 years ago
Ralf Zimmermann 408d95fe09
Add custom goal props to breakdown endpoint (#1578) 5 years ago
Uku Taht 0505de9b42 Add site limit to enterprise upgrade plan 5 years ago
Uku Taht 52067803df Update session stats from event 5 years ago
ssendev 8077671f8a
Update session screen_size if unknown (#1610) 5 years ago
Uku Taht 0a5f6c5886 Implement wildcard filters for the API 5 years ago
Uku Taht cd40579740 Ignore unknown country code 5 years ago
Uku Taht 35ee2fe3fb Parse bunny header 5 years ago
Matt Colligan 8148d41e42
Add cities and regions to exported CSV data (#1485) 5 years ago
Uku Taht 60f116facd Fix tests 5 years ago
Uku Taht 0b253421ce Fix tests 5 years ago
Uku Taht 711cbdfca2 Remove unused custom domain code 5 years ago
Uku Taht 15931b9346 Add tests for utm_term and content in the API 5 years ago
Uku Taht 4ea0e29620 Add UTM content and term to CSV export 5 years ago
Uku Taht 48ad7485c8
PR 1393 continued (#1542)
* Add `utm_content` and `utm_term`.
Support `utm_content` and `utm_term` as requested in #515.

* Add dropdown for UTM options

* Remove utm_content and term from filter modal for now

Co-authored-by: Blender Defender <defenderblender@gmail.com>
5 years ago
Uku Taht dc8d20496c Show locked message in shared link 5 years ago
Ralf Zimmermann faf4459b15
Add "events" metrics to public breakdown endpoint (#1529) 5 years ago
Uku Taht 66575f497b Add goal type to JSON response 5 years ago
Uku Taht 97cdc4814e Refactor goals API 5 years ago
Uku Taht 1dba113e2f
[Draft] Improve location translations (#1526)
* WIP

* Use location library for search suggestions

* Remove unused code

* Remove Countries completely

* Fix tests
5 years ago
Uku Taht b49fd19934
Pull 1506 - formatted (#1527)
* adding api route PUT /api/v1/sites/goals with form fields "goal_type" and "goal_value" with supported types "event" and "page"

* arm64 docker images

* adding api route DELETE /api/v1/sites/goals/:goal_id with form param "site_id"

* revert makfile + package.json

* return statement hotfix in case site could not be found

* adding api route PUT /api/v1/sites/goals/:goal_id with form params "site_id", "goal_type", and "goal_value"

* update the goal api routes to accept event_name or page_path instead of goal_value

* cleaning goals model

* mix format

Co-authored-by: Ahmed Abbas <a.abbas@ixdc.net>
5 years ago
RobertJoonas 9022234aa6
added invoices section (#1511)
* added invoices section

* fixed formatting

* UI and code style improvements, paddle sandbox setup only on dev environment (fixed)
5 years ago
Uku Taht a2d0244315 Fix site locked email 5 years ago
Uku Taht 01706b7590 Remove dead code 5 years ago
Uku Taht 4d0bc61ffd Remove Twitter stuff 5 years ago
Uku Taht e7e36b994d Skip usage checking for users who are on grace period 5 years ago
Uku Taht effe56b3e4 Use new iso_codes package 5 years ago
Uku Taht 7cfd9f95f1 Delete a site through the API 5 years ago
Uku Taht 07af25ffc3 Fix subscription_interval for enterprise plans 5 years ago
Uku Taht 9c07cf7ea6 Report allowance issues to Sentry 5 years ago
Uku Taht 2463296403 Fetch enterprise plan corretly 5 years ago
Uku Taht 5880678eaf Fix tests 5 years ago
Uku Taht 7e93500834 Allow admins to access the stats API 5 years ago
Uku Taht 2bdfec1cc0
JS refactor: use generic ListReport for country report (#1487)
* Use ListReport for countries

* Fix countries tests

* Replace Browsers with ListReport

* Use Listreport for OS and screen size
5 years ago
Uku Taht 05bf43c1be
City level location data (#1449)
* Merge branch 'plausible_master'

* Add City level details

* Add City level details

* Use ISO codes instead of geoname_id for subdivisions

* Add easier way to configure geolocation database

* Add workflow for dev branch

* Correct clickhouse migration

* Translate subdivision names

* Translate city names

* WIP

* Region and country filters

* Fix region filter

* Remove region_name when removing region filter

* Add modals for regions and cities

* Remove dead code

* WIP

* Revert "WIP"

This reverts commit 3202bf2fe9.

* Feature flag to hide cities when deployed

* Add changelog entry

* Remove unused code

* Remove unused variables

* Fix test

Co-authored-by: AymanTerra <aymanterra@yahoo.com>
5 years ago
Uku Taht 06d0d0eafa Fix over limit email reccommendation 5 years ago
Uku Taht e355828465 Remove pageview filter by default 5 years ago
Uku Taht 6b55ace014 Fix error with JSON parsing 5 years ago
Uku Taht 0edf4765d4 Ignore maps in prop values 5 years ago
Uku Taht d3311898c5 Show message when grace period is over 5 years ago
Uku Taht e9cb8eb4e2 Remove grace period if user upgrades 5 years ago
Uku Taht ec2a4a1a65 Add [Action required] to over limit email 5 years ago
Uku Taht d547d0f996 Send email notification when sites are locked at grace period end 5 years ago
Uku Taht 29cb7462e6 Add grace period to upgrade 5 years ago
Matt Colligan 7faa2f6673
CSV export output conversions and conversion rate when filtering for goal (#1464)
* Only add percentages to dashboard data when not filtering goal

* Correctly name CSV headers when exporting conversion data

* Remove percentages from tests when filtering for goal
5 years ago
Matt Colligan 3380685d40
Fix custom property total conversions value not displayed & Export custom properties (#1456)
* Fix custom property total conversions value not displayed

The custom property conversion metrics are not consistent with the other
metrics resulting in the total conversions not being displayed in the
dashboard. This fixes that.

* Export custom props of current goal when filtering dashboard for goal

This makes the CSV export also output a `prop_breakdown.csv` file which,
for the currently filtered goal, contains the conversion data for each
of its configured properties.

* Add test for goal-filtered CSV export
5 years ago
Uku Taht 7a02aae2a5 Accept custom props in expected format 5 years ago
Matt Colligan d781301886 Apply existing filter to visitors.csv in export (#1454)
Also adds a test to ensure the filter always applies.
5 years ago
Matt Colligan 29e37c8342 More improvements for CSV export (#1446)
* Round time_on_page metric returned by 'pages' API

This rounds the `time_on_page` metric returned as part of the `pages`
API to the nearest second. While this shows no apparent change in the
web UI, it removes the decimal from the exported data in `pages.csv`.

* Tidy up export tests

* Round time_on_page in db query

* Add goal to CSV export tests

* Display CSV export spinner until download is ready

The mechanism used to make the page aware that the download is ready is:

- Client code sets a cookie and requests download.
- Server code handles download and removes cookie when complete.
- Client code polls every 1s to check the cookies, removing spinner when
  the export cookie is removed.
5 years ago
Matt Colligan 945bf82c72
Fix conversions metrics (#1447)
f576fa2 should have updated the conversion metric names so that
`unique_conversions` and `total_conversions` are the two metrics
returned by the conversions API. This updates those so that the CSV
export outputs the correct data.
5 years ago
Matt Colligan f576fa2a2c
Improvements to CSV export (#1427)
* Add details=True to export API parameters

This makes the ZIP export add `%{"details" => "True"}` to the query's
`params` when fetching data internally for packaging in the ZIP.

This adds bounce_rate and time_on_page to the data in pages.csv, and
bounce_rate and visit_duration to sources.csv.

* Make API return data with consistent names

Some of the data types returned via the JSON or CSV API use inconsistent
naming, and some have redundant name changes (i.e. count -> visitors ->
count). This makes these all consistent and removes the redundancy.

This addresses #1426, fixes some of the CSV headers, and unifies the
JSON and CSV return data labels.

* Update changelog

* Test should use Timex.shift, not relative time

* Return full country names in CSV export

This also replaces the " character with ' in two country names, as those
are the characters used in the names, yielding a more predictable and
'correct' output.

* Fetch CSV exported data concurrently

* Use spinner to indicate when export has started

* Use 300 as default number of brekadown entries for export

Higher numbers (e.g. 1000) seem to cause clickhouse errors when there
many pages to request. It is unclear what is causing the error, as
clickhouse returns an "unknown" error code and an empty error message.
5 years ago
Matt Colligan 8138a6e3a5
Start putting together means to export all aggregate data (#1387)
* Export all dashboard data in zip

This packages all data currently visible in the dashboard into
individual CSVs and downloads them together in a zip.

* Also export conversions in zip of CSVs

* Update export test with zip file response

* Add zip file download to changelog
5 years ago
Uku Taht 7622621a1d Adds test 5 years ago
Uku Taht 1b0323f5c1 Use more path helpers 5 years ago
Uku Taht 710f6f568a Ignores invalid query param part 5 years ago
Uku Taht 6dd9b0bb2f Do not error when bad custom prop value is supplied 5 years ago
Uku Taht b9b37cdd34 Show pricing v2 when appropriate 5 years ago
Uku Taht 26e34596c8 Allow enterprise plan users to create sites 5 years ago