csv-export-oban
no-team-in-self-hosted
sync-imported-data-in-site-imports-data-migration
gh-pages
master
request-logging
new-top-report-action
top-report-refactor
con-cache-patch-test
clean-site-ch-data
entry-exit-hostnames-schema
simplify
dependabot/npm_and_yarn/tracker/express-4.19.2
rename-build-names
migration-for-3918
dependabot/npm_and_yarn/tracker/eslint-plugin-playwright-1.5.4
reload-after-config
dependabot/hex/csv-3.2.1
pr-3887-review
fix-timezones
gzip
reapply-cache-change2
consistency-api
journey-rebased
metmarkosaric-patch-1
journey
dependabot/npm_and_yarn/assets/chart.js-4.4.2
pr-3800-refactor-wip
drop-event-columns
not-goals
exclude-in-api
poc-clickhouse-metadata-queries
feat-tag-ch-queries
fix/site-locker-no-trial-with-sub
time-on-page-improvements
feat/2fa-final-bigger-refactor
feat/parse-x-vercel-forwarded-for
fix/live-flash-messages-in-sites
feat/2fa-poc
fix/invite-modal-message
bt-bugs
modals
fix/redirect-to-people-settings-after-removing-pending-invitation
props-settings-ui-keyboard-propagation
login-404
fix-sessions-pageview-query
revenue-tracking-wip
monetary-values
tagged-events-bug-repro
pr-2841
jsx-conditional-rendering
bounce-rate-entry-page-poc
disable-site-creation
support-multiple-filters-frontend
custom-comparison-mode-end-date-spike
site_inserted_at_limit
ua-device-type
chartjs-upgrade
ua-device-type-ecto-query
not-set-patch
sentry-frontend-integration
process-per-ip
v1.4
v1.2
v1.5.0-rc.0
list
r000000-0-test
r2022.10.10-1
r2022.10.10-2
r2022.10.11-3
r2022.10.13-4
r2022.10.13-5
r2022.10.17-6
r2022.10.18-7
r2022.10.19-8
r2022.10.24-10
r2022.10.24-9
r2022.10.25-11
r2022.10.26-12
r2022.10.27-13
r2022.11.02-14
r2022.11.03-15
r2022.11.09-16
r2022.11.10-17
r2022.11.15-18
r2022.11.21-19
r2022.11.22-20
r2022.11.22-21
r2022.11.24-22
r2022.11.25-23
r2022.11.28-24
r2022.11.29-25
r2022.11.30-26
r2022.12.02-27
r2022.12.19-28
r2022.12.22-29
r2022.12.27-30
r2023.01.02-31
r2023.01.02-32
r2023.01.04-33
r2023.01.05-34
r2023.01.11-35
r2023.01.11-36
r2023.01.12-37
r2023.01.12-38
r2023.01.16-39
r2023.01.17-40
r2023.01.19-41
r2023.01.19-42
r2023.01.19-43
r2023.01.23-44
r2023.01.24-45
r2023.01.25-46
r2023.01.31-47
r2023.02.01-48
r2023.02.01-49
r2023.02.02-50
r2023.02.06-51
r2023.02.07-52
r2023.02.07-53
r2023.02.12-54
r2023.02.14-55
r2023.02.15-56
r2023.02.21-57
r2023.02.22-58
r2023.02.22-59
r2023.02.22-60
r2023.02.23-61
r2023.02.23-62
r2023.02.23-63
r2023.02.27-64
r2023.03.01-65
r2023.03.01-66
r2023.03.02-67
r2023.03.06-68
r2023.03.07-69
r2023.03.07-70
r2023.03.08-71
r2023.03.10-72
r2023.03.13-73
r2023.03.14-74
r2023.03.15-75
r2023.03.15-76
r2023.03.20-77
r2023.03.21-78
r2023.03.21-79
r2023.03.22-80
r2023.03.22-81
r2023.03.23-82
r2023.03.24-83
r2023.03.27-84
r2023.03.27-85
r2023.03.28-86
r2023.03.28-87
r2023.03.30-88
r2023.04.04-89
r2023.04.04-90
r2023.04.04-91
r2023.04.04-92
r2023.04.04-93
r2023.04.04-94
r2023.04.05-95
r2023.04.05-96
r2023.04.06-97
r2023.04.07-98
r2023.04.10-100
r2023.04.10-99
r2023.04.11-101
r2023.04.13-102
r2023.04.24-103
r2023.04.26-104
r2023.04.27-105
r2023.05.02-106
r2023.05.03-107
r2023.05.04-108
r2023.05.09-109
r2023.05.10-110
r2023.05.16-111
r2023.05.16-112
r2023.05.17-113
r2023.05.18-114
r2023.05.22-115
r2023.05.23-116
r2023.05.24-117
r2023.05.25-118
r2023.06.01-119
r2023.06.07-120
r2023.06.12-121
r2023.06.12-122
r2023.06.12-123
r2023.06.13-124
r2023.06.13-125
r2023.06.13-126
r2023.06.14-127
r2023.06.14-128
r2023.06.19-129
r2023.06.20-130
r2023.06.23-131
r2023.06.27-132
r2023.06.27-133
r2023.06.27-134
r2023.06.28-135
r2023.06.28-136
r2023.06.28-137
r2023.06.28-138
r2023.07.04-139
r2023.07.06-140
r2023.07.11-141
r2023.07.11-142
r2023.07.11-143
r2023.07.11-144
r2023.07.12-145
r2023.07.13-146
r2023.07.17-147
r2023.07.18-148
r2023.07.19-149
r2023.07.21-150
r2023.07.24-151
r2023.07.24-152
r2023.07.24-153
r2023.07.24-154
r2023.07.26-155
r2023.07.27-156
r2023.07.27-157
r2023.07.31-158
r2023.08.01-159
r2023.08.02-160
r2023.08.03-161
r2023.08.05-162
r2023.08.07-163
r2023.08.08-164
r2023.08.09-165
r2023.08.10-166
r2023.08.14-167
r2023.08.14-168
r2023.08.15-169
r2023.08.16-170
r2023.08.17-171
r2023.08.21-172
r2023.08.21-173
r2023.08.22-174
r2023.08.22-175
r2023.08.22-176
r2023.08.24-177
r2023.08.28-178
r2023.08.30-179
r2023.08.31-180
r2023.09.04-181
r2023.09.04-182
r2023.09.05-183
r2023.09.05-184
r2023.09.05-185
r2023.09.07-186
r2023.09.12-187
r2023.09.12-188
r2023.09.13-189
r2023.09.13-190
r2023.09.18-191
r2023.09.18-192
r2023.09.21-193
r2023.09.25-194
r2023.09.26-195
r2023.09.28-196
r2023.10.02-197
r2023.10.02-198
r2023.10.02-199
r2023.10.03-200
r2023.10.04-201
r2023.10.05-202
r2023.10.09-203
r2023.10.10-204
r2023.10.10-205
r2023.10.10-206
r2023.10.11-207
r2023.10.11-208
r2023.10.11-209
r2023.10.12-210
r2023.10.13-211
r2023.10.16-212
r2023.10.16-213
r2023.10.17-214
r2023.10.17-215
r2023.10.18-216
r2023.10.18-217
r2023.10.23-218
r2023.10.24-219
r2023.10.25-220
r2023.10.25-221
r2023.10.30-222
r2023.10.30-223
r2023.10.31-224
r2023.11.01-225
r2023.11.01-226
r2023.11.02-227
r2023.11.02-228
r2023.11.02-229
r2023.11.02-230
r2023.11.02-231
r2023.11.02-232
r2023.11.02-233
r2023.11.06-234
r2023.11.06-235
r2023.11.08-236
r2023.11.08-237
r2023.11.08-238
r2023.11.08-239
r2023.11.09-240
r2023.11.10-241
r2023.11.10-242
r2023.11.13-243
r2023.11.13-244
r2023.11.13-245
r2023.11.13-246
r2023.11.14-247
r2023.11.14-248
r2023.11.16-249
r2023.11.17-250
r2023.11.17-251
r2023.11.17-252
r2023.11.17-253
r2023.11.20-254
r2023.11.20-255
r2023.11.20-256
r2023.11.20-257
r2023.11.21-258
r2023.11.23-259
r2023.11.28-260
r2023.11.29-261
r2023.11.29-262
r2023.11.30-263
r2023.11.30-264
r2023.11.30-265
r2023.11.30-266
r2023.11.30-267
r2023.12.04-268
r2023.12.04-269
r2023.12.05-270
r2023.12.06-271
r2023.12.06-272
r2023.12.06-273
r2023.12.06-274
r2023.12.06-275
r2023.12.07-276
r2023.12.11-277
r2023.12.11-278
r2023.12.11-279
r2023.12.11-280
r2023.12.12-281
r2023.12.12-282
r2023.12.12-283
r2023.12.13-284
r2023.12.13-285
r2023.12.13-286
r2023.12.13-287
r2023.12.14-288
r2023.12.14-289
r2023.12.14-290
r2023.12.15-291
r2023.12.20-292
r2023.12.20-293
r2023.12.21-294
r2023.12.21-295
r2023.12.28-296
r2024.01.02-297
r2024.01.02-298
r2024.01.02-299
r2024.01.02-300
r2024.01.03-301
r2024.01.03-302
r2024.01.03-303
r2024.01.04-304
r2024.01.04-305
r2024.01.04-306
r2024.01.08-307
r2024.01.08-308
r2024.01.08-309
r2024.01.09-310
r2024.01.09-311
r2024.01.09-312
r2024.01.11-313
r2024.01.11-314
r2024.01.15-315
r2024.01.15-316
r2024.01.15-317
r2024.01.15-318
r2024.01.22-319
r2024.01.22-320
r2024.01.22-321
r2024.01.22-322
r2024.01.22-323
r2024.01.22-324
r2024.01.22-325
r2024.01.23-326
r2024.01.23-327
r2024.01.25-328
r2024.01.27-329
r2024.01.27-330
r2024.01.27-331
r2024.01.29-332
r2024.01.29-333
r2024.01.29-334
r2024.01.29-335
r2024.01.30-336
r2024.02.01-337
r2024.02.01-338
r2024.02.01-339
r2024.02.01-340
r2024.02.01-341
r2024.02.07-342
r2024.02.12-343
r2024.02.12-344
r2024.02.12-345
r2024.02.12-346
r2024.02.12-347
r2024.02.12-348
r2024.02.13-349
r2024.02.13-350
r2024.02.14-351
r2024.02.14-352
r2024.02.14-353
r2024.02.14-354
r2024.02.15-355
r2024.02.15-356
r2024.02.15-357
r2024.02.15-358
r2024.02.15-359
r2024.02.19-360
r2024.02.20-361
r2024.02.21-362
r2024.02.21-363
r2024.02.21-364
r2024.02.22-365
r2024.02.22-366
r2024.02.23-367
r2024.02.23-368
r2024.02.26-369
r2024.02.26-370
r2024.02.26-371
r2024.02.27-372
r2024.02.27-373
r2024.02.27-374
r2024.02.27-375
r2024.02.27-376
r2024.02.27-377
r2024.02.28-378
r2024.02.28-379
r2024.02.28-380
r2024.03.01-381
r2024.03.05-382
r2024.03.06-383
r2024.03.06-384
r2024.03.08-385
r2024.03.08-386
r2024.03.08-387
r2024.03.11-388
r2024.03.12-389
r2024.03.12-390
r2024.03.12-391
r2024.03.12-392
r2024.03.12-393
r2024.03.13-394
r2024.03.14-395
r2024.03.18-396
r2024.03.18-397
r2024.03.18-398
r2024.03.19-399
r2024.03.21-400
r2024.03.21-401
r2024.03.21-402
r2024.03.21-403
r2024.03.21-404
r2024.03.21-405
r2024.03.21-406
r2024.03.22-407
r2024.03.22-408
r2024.03.22-409
r2024.03.22-410
r2024.03.25-411
r2024.03.25-412
r2024.03.25-413
r2024.03.26-414
r2024.03.26-415
r2024.03.26-416
r2024.03.26-417
r2024.03.28-418
r2024.04.02-419
r2024.04.02-420
r2024.04.03-421
r2024.04.04-422
r2024.04.04-423
r2024.04.04-424
r2024.04.04-425
r2024.04.04-426
r2024.04.04-427
r2024.04.04-428
v1.0.0
v1.1.0
v1.1.1
v1.2-rc.0
v1.2-rc.1
v1.2.0
v1.3.0
v1.3.0-rc.0
v1.4.0
v1.4.0-rc.0
v1.4.1
v1.4.2
v1.4.3
v1.4.4
v1.5.0
v1.5.0-rc.1
v1.5.0-rc.2
v1.5.1
v2.0.0
v2.0.0-rc.0
v2.0.0-rc.1
v2.0.0-rc.2
v2.0.0-rc.3
v2.0.0-rc.4
v2.1.0-rc.0
${ noResults }
13 Commits (32116fe2c98480be32d2967fd91ea96efafa7ad2)
| Author | SHA1 | Message | Date |
|---|---|---|---|
|
|
e27734ed79
|
[Continued] Google Analytics import (#1753)
* Add has_imported_stats boolean to Site * Add Google Analytics import panel to general settings * Get GA profiles to display in import settings panel * Add import_from_google method as entrypoint to import data * Add imported_visitors table * Remove conflicting code from migration * Import visitors data into clickhouse database * Pass another dataset to main graph for rendering in red This adds another entry to the JSON data returned via the main graph API called `imported_plot`, which is similar to `plot` in form but will be completed with previously imported data. Currently it simply returns the values from `plot` / 2. The data is rendered in the main graph in red without fill, and without an indicator for the present. Rationale: imported data will not continue to grow so there is no projection forward, only backwards. * Hook imported GA data to dashboard timeseries plot * Add settings option to forget imported data * Import sources from google analytics * Merge imported sources when queried * Merge imported source data native data when querying sources * Start converting metrics to atoms so they can be subqueried This changes "visitors" and in some places "sources" to atoms. This does not change the behaviour of the functions - the tests all pass unchanged following this commit. This is necessary as joining subqueries requires that the keys in `select` statements be atoms and not strings. * Convery GA (direct) source to empty string * Import utm campaign and utm medium from GA * format * Import all data types from GA into new tables * Handle large amounts of more data more safely * Fix some mistakes in tables * Make GA requests in chunks of 5 queries * Only display imported timeseries when there is no filter * Correctly show last 30 minutes timeseries when 'realtime' * Add with_imported key to Query struct * Account for injected :is_not filter on sources from dashboard * Also add tentative imported_utm_sources table This needs a bit more work on the google import side, as GA do not report sources and utm sources as distinct things. * Return imported data to dashboard for rest of Sources panel This extends the merge_imported function definition for sources to utm_sources, utm_mediums and utm_campaigns too. This appears to be working on the DB side but something is incomplete on the client side. * Clear imported stats from all tables when requested * Merge entry pages and exit pages from imported data into unfiltered dashboard view This requires converting the `"visits"` and `"visit_duration"` metrics to atoms so that they can be used in ecto subqueries. * Display imported devices, browsers and OSs on dashboard * Display imported country data on dashboard * Add more metrics to entries/exits for modals * make sure data is returned via API with correct keys * Import regions and cities from GA * Capitalize device upon import to match native data * Leave query limits/offsets until after possibly joining with imported data * Also import timeOnPage and pageviews for pages from GA * imported_countries -> imported_locations * Get timeOnPage and pageviews for pages from GA These are needed for the pages modal, and for calculating exit rates for exit pages. * Add indicator to dashboard when imported data is being used * Don't show imported data as separately line on main graph * "bounce_rate" -> :bounce_rate, so it works in subqueries * Drop imported browser and OS versions These are not needed. * Toggle displaying imported data by clicking indicator * Parse referrers with RefInspector - Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual referrer host + path, whereas 'ga:source' includes utm_mediums and other values when relevant. - 'ga:fullReferror' does however include search engine names directly, so they are manually checked for as RefInspector won't pick up on these. * Keep imported data indicator on dashboard and strikethrough when hidden * Add unlink google button to import panel * Rename some GA browsers and OSes to plausible versions * Get main top pages and exit pages panels working correctly with imported data * mix format * Fetch time_on_pages for imported data when needed * entry pages need to fetch bounces from GA * "sample_percent" -> :sample_percent as only atoms can be used in subqueries * Calculate bounce_rate for joined native and imported data for top pages modal * Flip some query bindings around to be less misleading * Fixup entry page modal visit durations * mix format * Fetch bounces and visit_duration for sources from GA * add more source metrics used for data in modals * Make sources modals display correct values * imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration * Merge imported data into aggregate stats * Reformat top graph side icons * Ensure sample_percent is yielded from aggregate data * filter event_props should be strings * Hide imported data from frontend when using filter * Fix existing tests * fix tests * Fix imported indicator appearing when filtering * comma needed, lost when rebasing * Import utm_terms and utm_content from GA * Merge imported utm_term and utm_content * Rename imported Countries data as Locations * Set imported city schema field to int * Remove utm_terms and utm_content when clearing imported * Clean locations import from Google Analytics - Country and region should be set to "" when GA provides "(not set)" - City should be set to 0 for "unknown", as we cannot reliably import city data from GA. * Display imported region and city in dashboard * os -> operating_system in some parts of code The inconsistency of using os in some places and operating_system in others causes trouble with subqueries and joins for the native and imported data, which would require additional logic to account for. The simplest solution is the just use a consistent word for all uses. This doesn't make any user-facing or database changes. * to_atom -> to_existing_atom * format * "events" metric -> :events * ignore imported data when "events" in metrics * update "bounce_rate" * atomise some more metrics from new city and region api * atomise some more metrics for email handlers * "conversion_rate" -> :conversion_rate during csv export * Move imported data stats code to own module * Move imported timeseries function to Stats.Imported * Use Timex.parse to import dates from GA * has_imported_stats -> imported_source * "time_on_page" -> :time_on_page * Convert imported GA data to UTC * Clean up GA request code a bit There was some weird logic here with two separate lists that really ought to be together, so this merges those. * Fail sooner if GA timezone can't be identified * Link imported tables to site by id * imported_utm_content -> imported_utm_contents * Imported GA from all of time * Reorganise GA data fetch logic - Fetch data from the start of time (2005) - Check whether no data was fetched, and if so, inform user and don't consider data to be imported. * Clarify removal of "visits" data when it isn't in metrics * Apply location filters from API This makes it consistent with the sources etc which filter out 'Direct / None' on the API side. These filters are used by both the native and imported data handling code, which would otherwise both duplicate the filters in their `where` clauses. * Do not use changeset for setting site.imported_source * Add all metrics to all dimensions * Run GA import in the background * Send email when GA import completes * Add handler to insert imported data into tests and imported_browsers_factory * Add remaining import data test factories * Add imported location data to test * Test main graph with imported data * Add imported data to operating systems tests * Add imported data to pages tests * Add imported data to entry pages tests * Add imported data to exit pages tests * Add imported data to devices tests * Add imported data to sources tests * Add imported data to UTM tests * Add new test module for the data import step * Test import of sources GA data * Test import of utm_mediums GA data * Test import of utm_campaigns GA data * Add tests for UTM terms * Add tests for UTM contents * Add test for importing pages and entry pages data from GA * Add test for importing exit page data * Fix module file name typo * Add test for importing location data from GA * Add test for importing devices data from GA * Add test for importing browsers data from GA * Add test for importing OS data from GA * Paginate GA requests to download all data * Bump clickhouse_ecto version * Move RefInspector wrapper function into module * Drop timezone transform on import * Order imported by side_id then date * More strings -> atoms Also changes a conditional to be a bit nicer * Remove parallelisation of data import * Split sources and UTM sources from fetched GA data GA has only a "source" dimension and no "UTM source" dimension. Instead it returns these combined. The logic herein to tease these apart is: 1. "(direct)" -> it's a direct source 2. if the source is a domain -> it's a source 3. "google" -> it's from adwords; let's make this a UTM source "adwords" 4. else -> just a UTM source * Keep prop names in queries as strings * fix typo * Fix import * Insert data to clickhouse in batches * Fix link when removing imported data * Merge source tables * Import hostname as well as pathname * Record start and end time of imported data * Track import progress * Fix month interval with imported data * Do not JOIN when imported date range has no overlap * Fix time on page using exits Co-authored-by: mcol <mcol@posteo.net> |
5 years ago |
|
|
408d95fe09
|
Add custom goal props to breakdown endpoint (#1578) | 5 years ago |
|
|
faf4459b15
|
Add "events" metrics to public breakdown endpoint (#1529) | 5 years ago |
|
|
c68b88c24b
|
Upgrade otp (#1313)
* update tool versions to latest * Upgrades erlang and elixir, format project Co-authored-by: happysalada <raphael@megzari.com> |
5 years ago |
|
|
08afc2ae71 | Timeout db queries in 10 seconds instead of 5 | 5 years ago |
|
|
63b7abad58 | Disable sampling in the API | 5 years ago |
|
|
669866a16b
|
Finish stats module (#1248)
* Fix small inconsitencies in stats module * Format |
5 years ago |
|
|
7d1b1214e2 | Implement dashboard with new stats module | 5 years ago |
|
|
b107717774
|
Additional metrics for timeseries API call and CSV export (#952)
* Add `metrics` option to timeseries call * Add more metrics to csv export * Add changelog entry |
6 years ago |
|
|
432ad2a606
|
Stats API improvements (#826)
* Show hourly data for a day * Fix aggregations with long periods and comaprisons enabled * Return Direct / None instead of empty string when missing source * Fix/Align breakdown query with current queries in the UI * Show correct number of days for 7d and 30d * Wrap results of aggregate call in `results` * Allow breakdown by event:name * Ensure event:name filter works in all contexts * WIP * Add support for custom event props |
6 years ago |
|
|
a29ac44434
|
Api improvements (#723)
* Require date parameter with custom period * Validate format of the `date` param with custom period * Return proper status codes for auth failures * Add breakdown endpoint * Fix pagination * Add API validations for breakdown call * Change the breakdown endpoint to return an object instead of an array * Remove change to aggregate call * Change timeseries call |
6 years ago |
|
|
77628d04c9
|
Validate API query params (#703)
* Add validations for `date`, `period` and `metrics` variables * Validate interval query parameter |
6 years ago |
|
|
5acb5b7039
|
Stats API (#679)
* WIP * Add ability to filter by anything * Add API keys * Add version to api endpoint * Fix API test route * Fix API tests * Allow 'date' parameter in '6mo' and '12mo' * Rename session -> visit in API filters * Filter expressions in the API * Implement filters in aggregate call * Add `compare` option to aggregate call * Add way to manage API keys through the UI * Authenticate with API key * Use API key in tests |
6 years ago |