csv-export-oban
no-team-in-self-hosted
sync-imported-data-in-site-imports-data-migration
gh-pages
master
request-logging
new-top-report-action
top-report-refactor
con-cache-patch-test
clean-site-ch-data
entry-exit-hostnames-schema
simplify
dependabot/npm_and_yarn/tracker/express-4.19.2
rename-build-names
migration-for-3918
dependabot/npm_and_yarn/tracker/eslint-plugin-playwright-1.5.4
reload-after-config
dependabot/hex/csv-3.2.1
pr-3887-review
fix-timezones
gzip
reapply-cache-change2
consistency-api
journey-rebased
metmarkosaric-patch-1
journey
dependabot/npm_and_yarn/assets/chart.js-4.4.2
pr-3800-refactor-wip
drop-event-columns
not-goals
exclude-in-api
poc-clickhouse-metadata-queries
feat-tag-ch-queries
fix/site-locker-no-trial-with-sub
time-on-page-improvements
feat/2fa-final-bigger-refactor
feat/parse-x-vercel-forwarded-for
fix/live-flash-messages-in-sites
feat/2fa-poc
fix/invite-modal-message
bt-bugs
modals
fix/redirect-to-people-settings-after-removing-pending-invitation
props-settings-ui-keyboard-propagation
login-404
fix-sessions-pageview-query
revenue-tracking-wip
monetary-values
tagged-events-bug-repro
pr-2841
jsx-conditional-rendering
bounce-rate-entry-page-poc
disable-site-creation
support-multiple-filters-frontend
custom-comparison-mode-end-date-spike
site_inserted_at_limit
ua-device-type
chartjs-upgrade
ua-device-type-ecto-query
not-set-patch
sentry-frontend-integration
process-per-ip
v1.4
v1.2
v1.5.0-rc.0
list
r000000-0-test
r2022.10.10-1
r2022.10.10-2
r2022.10.11-3
r2022.10.13-4
r2022.10.13-5
r2022.10.17-6
r2022.10.18-7
r2022.10.19-8
r2022.10.24-10
r2022.10.24-9
r2022.10.25-11
r2022.10.26-12
r2022.10.27-13
r2022.11.02-14
r2022.11.03-15
r2022.11.09-16
r2022.11.10-17
r2022.11.15-18
r2022.11.21-19
r2022.11.22-20
r2022.11.22-21
r2022.11.24-22
r2022.11.25-23
r2022.11.28-24
r2022.11.29-25
r2022.11.30-26
r2022.12.02-27
r2022.12.19-28
r2022.12.22-29
r2022.12.27-30
r2023.01.02-31
r2023.01.02-32
r2023.01.04-33
r2023.01.05-34
r2023.01.11-35
r2023.01.11-36
r2023.01.12-37
r2023.01.12-38
r2023.01.16-39
r2023.01.17-40
r2023.01.19-41
r2023.01.19-42
r2023.01.19-43
r2023.01.23-44
r2023.01.24-45
r2023.01.25-46
r2023.01.31-47
r2023.02.01-48
r2023.02.01-49
r2023.02.02-50
r2023.02.06-51
r2023.02.07-52
r2023.02.07-53
r2023.02.12-54
r2023.02.14-55
r2023.02.15-56
r2023.02.21-57
r2023.02.22-58
r2023.02.22-59
r2023.02.22-60
r2023.02.23-61
r2023.02.23-62
r2023.02.23-63
r2023.02.27-64
r2023.03.01-65
r2023.03.01-66
r2023.03.02-67
r2023.03.06-68
r2023.03.07-69
r2023.03.07-70
r2023.03.08-71
r2023.03.10-72
r2023.03.13-73
r2023.03.14-74
r2023.03.15-75
r2023.03.15-76
r2023.03.20-77
r2023.03.21-78
r2023.03.21-79
r2023.03.22-80
r2023.03.22-81
r2023.03.23-82
r2023.03.24-83
r2023.03.27-84
r2023.03.27-85
r2023.03.28-86
r2023.03.28-87
r2023.03.30-88
r2023.04.04-89
r2023.04.04-90
r2023.04.04-91
r2023.04.04-92
r2023.04.04-93
r2023.04.04-94
r2023.04.05-95
r2023.04.05-96
r2023.04.06-97
r2023.04.07-98
r2023.04.10-100
r2023.04.10-99
r2023.04.11-101
r2023.04.13-102
r2023.04.24-103
r2023.04.26-104
r2023.04.27-105
r2023.05.02-106
r2023.05.03-107
r2023.05.04-108
r2023.05.09-109
r2023.05.10-110
r2023.05.16-111
r2023.05.16-112
r2023.05.17-113
r2023.05.18-114
r2023.05.22-115
r2023.05.23-116
r2023.05.24-117
r2023.05.25-118
r2023.06.01-119
r2023.06.07-120
r2023.06.12-121
r2023.06.12-122
r2023.06.12-123
r2023.06.13-124
r2023.06.13-125
r2023.06.13-126
r2023.06.14-127
r2023.06.14-128
r2023.06.19-129
r2023.06.20-130
r2023.06.23-131
r2023.06.27-132
r2023.06.27-133
r2023.06.27-134
r2023.06.28-135
r2023.06.28-136
r2023.06.28-137
r2023.06.28-138
r2023.07.04-139
r2023.07.06-140
r2023.07.11-141
r2023.07.11-142
r2023.07.11-143
r2023.07.11-144
r2023.07.12-145
r2023.07.13-146
r2023.07.17-147
r2023.07.18-148
r2023.07.19-149
r2023.07.21-150
r2023.07.24-151
r2023.07.24-152
r2023.07.24-153
r2023.07.24-154
r2023.07.26-155
r2023.07.27-156
r2023.07.27-157
r2023.07.31-158
r2023.08.01-159
r2023.08.02-160
r2023.08.03-161
r2023.08.05-162
r2023.08.07-163
r2023.08.08-164
r2023.08.09-165
r2023.08.10-166
r2023.08.14-167
r2023.08.14-168
r2023.08.15-169
r2023.08.16-170
r2023.08.17-171
r2023.08.21-172
r2023.08.21-173
r2023.08.22-174
r2023.08.22-175
r2023.08.22-176
r2023.08.24-177
r2023.08.28-178
r2023.08.30-179
r2023.08.31-180
r2023.09.04-181
r2023.09.04-182
r2023.09.05-183
r2023.09.05-184
r2023.09.05-185
r2023.09.07-186
r2023.09.12-187
r2023.09.12-188
r2023.09.13-189
r2023.09.13-190
r2023.09.18-191
r2023.09.18-192
r2023.09.21-193
r2023.09.25-194
r2023.09.26-195
r2023.09.28-196
r2023.10.02-197
r2023.10.02-198
r2023.10.02-199
r2023.10.03-200
r2023.10.04-201
r2023.10.05-202
r2023.10.09-203
r2023.10.10-204
r2023.10.10-205
r2023.10.10-206
r2023.10.11-207
r2023.10.11-208
r2023.10.11-209
r2023.10.12-210
r2023.10.13-211
r2023.10.16-212
r2023.10.16-213
r2023.10.17-214
r2023.10.17-215
r2023.10.18-216
r2023.10.18-217
r2023.10.23-218
r2023.10.24-219
r2023.10.25-220
r2023.10.25-221
r2023.10.30-222
r2023.10.30-223
r2023.10.31-224
r2023.11.01-225
r2023.11.01-226
r2023.11.02-227
r2023.11.02-228
r2023.11.02-229
r2023.11.02-230
r2023.11.02-231
r2023.11.02-232
r2023.11.02-233
r2023.11.06-234
r2023.11.06-235
r2023.11.08-236
r2023.11.08-237
r2023.11.08-238
r2023.11.08-239
r2023.11.09-240
r2023.11.10-241
r2023.11.10-242
r2023.11.13-243
r2023.11.13-244
r2023.11.13-245
r2023.11.13-246
r2023.11.14-247
r2023.11.14-248
r2023.11.16-249
r2023.11.17-250
r2023.11.17-251
r2023.11.17-252
r2023.11.17-253
r2023.11.20-254
r2023.11.20-255
r2023.11.20-256
r2023.11.20-257
r2023.11.21-258
r2023.11.23-259
r2023.11.28-260
r2023.11.29-261
r2023.11.29-262
r2023.11.30-263
r2023.11.30-264
r2023.11.30-265
r2023.11.30-266
r2023.11.30-267
r2023.12.04-268
r2023.12.04-269
r2023.12.05-270
r2023.12.06-271
r2023.12.06-272
r2023.12.06-273
r2023.12.06-274
r2023.12.06-275
r2023.12.07-276
r2023.12.11-277
r2023.12.11-278
r2023.12.11-279
r2023.12.11-280
r2023.12.12-281
r2023.12.12-282
r2023.12.12-283
r2023.12.13-284
r2023.12.13-285
r2023.12.13-286
r2023.12.13-287
r2023.12.14-288
r2023.12.14-289
r2023.12.14-290
r2023.12.15-291
r2023.12.20-292
r2023.12.20-293
r2023.12.21-294
r2023.12.21-295
r2023.12.28-296
r2024.01.02-297
r2024.01.02-298
r2024.01.02-299
r2024.01.02-300
r2024.01.03-301
r2024.01.03-302
r2024.01.03-303
r2024.01.04-304
r2024.01.04-305
r2024.01.04-306
r2024.01.08-307
r2024.01.08-308
r2024.01.08-309
r2024.01.09-310
r2024.01.09-311
r2024.01.09-312
r2024.01.11-313
r2024.01.11-314
r2024.01.15-315
r2024.01.15-316
r2024.01.15-317
r2024.01.15-318
r2024.01.22-319
r2024.01.22-320
r2024.01.22-321
r2024.01.22-322
r2024.01.22-323
r2024.01.22-324
r2024.01.22-325
r2024.01.23-326
r2024.01.23-327
r2024.01.25-328
r2024.01.27-329
r2024.01.27-330
r2024.01.27-331
r2024.01.29-332
r2024.01.29-333
r2024.01.29-334
r2024.01.29-335
r2024.01.30-336
r2024.02.01-337
r2024.02.01-338
r2024.02.01-339
r2024.02.01-340
r2024.02.01-341
r2024.02.07-342
r2024.02.12-343
r2024.02.12-344
r2024.02.12-345
r2024.02.12-346
r2024.02.12-347
r2024.02.12-348
r2024.02.13-349
r2024.02.13-350
r2024.02.14-351
r2024.02.14-352
r2024.02.14-353
r2024.02.14-354
r2024.02.15-355
r2024.02.15-356
r2024.02.15-357
r2024.02.15-358
r2024.02.15-359
r2024.02.19-360
r2024.02.20-361
r2024.02.21-362
r2024.02.21-363
r2024.02.21-364
r2024.02.22-365
r2024.02.22-366
r2024.02.23-367
r2024.02.23-368
r2024.02.26-369
r2024.02.26-370
r2024.02.26-371
r2024.02.27-372
r2024.02.27-373
r2024.02.27-374
r2024.02.27-375
r2024.02.27-376
r2024.02.27-377
r2024.02.28-378
r2024.02.28-379
r2024.02.28-380
r2024.03.01-381
r2024.03.05-382
r2024.03.06-383
r2024.03.06-384
r2024.03.08-385
r2024.03.08-386
r2024.03.08-387
r2024.03.11-388
r2024.03.12-389
r2024.03.12-390
r2024.03.12-391
r2024.03.12-392
r2024.03.12-393
r2024.03.13-394
r2024.03.14-395
r2024.03.18-396
r2024.03.18-397
r2024.03.18-398
r2024.03.19-399
r2024.03.21-400
r2024.03.21-401
r2024.03.21-402
r2024.03.21-403
r2024.03.21-404
r2024.03.21-405
r2024.03.21-406
r2024.03.22-407
r2024.03.22-408
r2024.03.22-409
r2024.03.22-410
r2024.03.25-411
r2024.03.25-412
r2024.03.25-413
r2024.03.26-414
r2024.03.26-415
r2024.03.26-416
r2024.03.26-417
r2024.03.28-418
r2024.04.02-419
r2024.04.02-420
r2024.04.03-421
r2024.04.04-422
r2024.04.04-423
r2024.04.04-424
r2024.04.04-425
r2024.04.04-426
r2024.04.04-427
r2024.04.04-428
v1.0.0
v1.1.0
v1.1.1
v1.2-rc.0
v1.2-rc.1
v1.2.0
v1.3.0
v1.3.0-rc.0
v1.4.0
v1.4.0-rc.0
v1.4.1
v1.4.2
v1.4.3
v1.4.4
v1.5.0
v1.5.0-rc.1
v1.5.0-rc.2
v1.5.1
v2.0.0
v2.0.0-rc.0
v2.0.0-rc.1
v2.0.0-rc.2
v2.0.0-rc.3
v2.0.0-rc.4
v2.1.0-rc.0
${ noResults }
21 Commits (request-logging)
| Author | SHA1 | Message | Date |
|---|---|---|---|
|
|
e6d83e946f
|
Populate new columns in imports and exports (#3969)
* Extend `Imported*` schemas with newly added columns * Populate newly added `Imported*` fields in GA4 imports * Extend exports with newly added fields * Extend CSV importer to ingest new fields * Fix alias shadowing error * Add more extensive GA4 import fixtures * Apply rounding and casting to sampled visits |
3 years ago |
|
|
2ba4988c95
|
specify insert columns in import from s3 (#3968)
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com> |
3 years ago |
|
|
c263df5805
|
CSV imports (UI) (#3845)
* add basic ui * remove TODO * credo * allow folder upload * redirect external * mention folder, use folder icon for file picker * back to multiple file upload * mention zip * escape dots in archive filename |
3 years ago |
|
|
ba5b80a8c0
|
Add label to site imports and populate it (#3914) | 3 years ago |
|
|
279e89c693
|
CSV imports (no UI) (#3895)
* encode/decode date range in filenames * Update lib/plausible/imported/csv_importer.ex Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com> * Update lib/plausible/imported/csv_importer.ex Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com> * drop unused functions * send failure email if there is no data to export * use PlausibleWeb.Email.mailer_email_from() * ensure we get dates from minmax date query --------- Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com> |
3 years ago |
|
|
4d7d88cfec
|
Implement basics of GA4 import (#3851)
* Implement LV date input using flatpickr * Implement basics of GA4 import (very dirty WIP) * Split Google HTTP API into UA and GA4 specific parts * Add a quick way to record GA4 API responses * Add first GA4 import fixtures with GA4 Data API responses * Extract GA4 and UA specific logic form Google API * Extract UA and GA4 specific actions to distinct controllers * Add integration test for GA4 importer * Update GA4 fixtures * Test GA4 API * Add debug logging and fix paginating through API results in in GA4 import * Revert "Implement LV date input using flatpickr" This reverts commit c696f8ee39d5702f27015c09a4f079ca124cc7bb. * Fix note |
3 years ago |
|
|
39aa81a16f
|
Implement UI for multiple imports (#3727)
* Create a stub of site settings section for imports and exports * Use legacy site import indication to determine UA import handling * Add provisional logos for upcoming import sources * Stub basics of import page * Add very rudimentary support for multiple UA imports * Implement imports list as live view * Add support for opening LV modal from backend and closing from frontend * Introduce notion of themes to `button` and `button_link` components * Add confirmation modal on deleting import * Swap GA4 logo * Implement disabled state support for `button_link` component * Disable export and non-implemented import sources * Use native starts start date for upper boundary of import time range * Ensure integrations view uses legacy UA import flow * Remove unnecessary preload in SiteController * Remove unnecessary exception for legacy imports * Move API controller stats tests under PlausibleWeb * Test listing imports * Add test for explicit listener setup * Add tests for legacy flag state in UA importer * Add test for purging legacy import data * Add tests for `Sites.native_stats_start_date` * Test forgetting imports * Add `Stats.Clickhouse.imported_pageview_counts/1` and fix test flakiness * Show page view counts on imports list * Add tests for static imports and exports view * Adjust button look slightly * Use `case` instead of `cond` * Make feature flag customisable per site * Fix buttons and empty state styling * Add another import to seeds * Use JS confirm dialog instead of modal for deletion confirmations * Revert "Add support for opening LV modal from backend and closing from frontend" This reverts commit 260e6c753032b451542e24be9edc2118790b5a00. * Default `legacy` to false when inserting new import jobs * Drop `method` attribute from `button_link` and `unstyled_link` components |
3 years ago |
|
|
f3423aefec
|
Add csv importer (#3795)
* add csv importer * make table validation explicit * update some docs * improve docs * add minio container to ci * more tests * eh * continue * add passing test * add failing test * add config test * add minio to Makefile * testcontainers * remove extra whitespace * explain the implementation a bit * account for async deletes in tests * bounces is UInt32 Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com> --------- Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com> |
3 years ago |
|
|
f8b4d5066a
|
Add multiple imports per site (#3724)
* Clean up references to no longer active `google_analytics_imports` Oban queue * Stub CSV importer * Add SiteImport schema * Rename `Plausible.Imported` module file to match module name * Add `import_id` column to `Imported.*` CH schemas * Implement Importer behavior and manage imports state using new entities * Implement importer callbacks and maintain site.imported_data for UA * Keep imports in sync when forgetting all imports * Scope imported data queries to completed import IDs * Mark newly imported data with respective import ID * Clean up Importer implementation a bit * Test querying legacy and new imported data * Send Oban notifications on import worker failure too * Fix checking for forgettable imports and remove redundant function * Fix UA integration test * Change site import source to atom enum and add source label * Add typespecs and reduce repetition in `Plausible.Imported` * Improve documentation and typespecs * Add test for purging particular import * Switch email notification templates depending on import source * Document running import synchronously * Fix UA importer args parsing and ensure it's covered by tests * Clear `site.stats_start_date` on complete import to force recalculation * Test Oban notifications (h/t @ruslandoga) * Purge stats on import failure right away to reduce a chance of leaving debris behind * Fix typos Co-authored-by: hq1 <hq@mtod.org> * Fix another typo * Refactor fetching earliest import and earliest stats start date * Use `Date.after?` instead of `Timex.after?` * Cache import data in site virtual fields and limit queried imports to 5 * Ensure always current `stats_start_date` is used * Work around broken typespec in Timex * Make `SiteController.forget_imported` action idempotent * Discard irrecoverably failed import tasks * Use macros for site import statuses There's also a fix ensuring only complete imports are considered where relevant - couldn't isolate it as it was in a common hunk * Use `import_id` as worker job uniqueness criterion * Do not load imported stats data in plugins API context --------- Co-authored-by: hq1 <hq@mtod.org> |
3 years ago |
|
|
822483c37c
|
Extract Universal Analytics import logic (#3700)
* Move imported tables schemas to separate modules outside Google ns * Move buffer for imports to Imported ns * fix schema newlines * Extract UA import processing and persistence * Decouple analytics worker implementation from UA * Rename env variable for import buffer size * Preserve old import queue until release |
3 years ago |
|
|
6d79ca5093
|
Switch to new clickhouse adapter (ch/chto) (#2733)
* another clickhouse adapter * don't restore stats_removal.ex * fix events main-graph error (#2746) * update ch, chto * update chto again (#2759) * Stop treating page filter as an entry_page filter (#2752) * remove dead code * stop treating page filter as entry page filter in breakdown queries * stop treating page filter as entry page filter in aggregate queries * stop treating page filter as entry page filter in timeseries queries * mix format * update changelog * break code down to smaller functions to keep credo happy * remove unused functions * make CSV export return only conversions with goal filter (#2760) * make CSV export return only conversions with goal filter * update changelog * update elixir version in mix.exs (#2742) * revert admin.ex changes (#2776) --------- Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com> Co-authored-by: ruslandoga <rusl@n-do.ga> Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com> |
4 years ago |
|
|
1cb07efe6d
|
Save city name when importing from GA (#2608)
This commit adds city data to imported records from Google Analytics. The current implementation sets city to 0 because GA does not use the GeoNames database. Google Analytics Reporting API uses [Geographical IDs](https://developers.google.com/analytics/devguides/collection/protocol/v1/geoid) to identify cities and countries. Plausible uses [GeoNames](https://geonames.org/) and I couldn't find databases corelating the two. Fortunately, GA also returns the city name and this commit uses the city name and the country ISO code to find the Geoname ID. To avoid making expensive ETS searches, I created another ETS table in the Location library that uses {country, city} as a key. Related PR: https://github.com/plausible/location/pull/3 |
4 years ago |
|
|
e48851a9ed
|
Skip rows with empty dates from GA imports (#2359) | 4 years ago |
|
|
0733efa89e
|
Delete stats improvements (#2318)
* Move clear stats functions to Plausible.Purge * Delete both native and imported stats when deleting a site This commit moves the delete site function to the Plausible.Purge module, and fixes a bug where deleted sites could leave dangling imported stats. * Clear sites.stats_start_date after clearing stats This commit fixes a bug where resetting stats left an invalid state of the stats_start_date field, used for GA imports, for example. |
4 years ago |
|
|
4b9032d822
|
Google Analytics Import Refactor (#2046)
* Create separate module for GA HTTP requests * Fetch GA data entirely instead of monthly * Add buffering to GA imports * Change positional args to maps when serializing from GA * Create Google Analytics VCR tests |
4 years ago |
|
|
06b165eb6d | Run GA import in monthly batches | 5 years ago |
|
|
18fd0dbe04 | Always parse GA numbers as floats | 5 years ago |
|
|
ed871e7a0b | Parse scientific notation in GA import | 5 years ago |
|
|
a375789f52 | Deal with nil when importing | 5 years ago |
|
|
3a0ec5481b | Log import issue | 5 years ago |
|
|
e27734ed79
|
[Continued] Google Analytics import (#1753)
* Add has_imported_stats boolean to Site * Add Google Analytics import panel to general settings * Get GA profiles to display in import settings panel * Add import_from_google method as entrypoint to import data * Add imported_visitors table * Remove conflicting code from migration * Import visitors data into clickhouse database * Pass another dataset to main graph for rendering in red This adds another entry to the JSON data returned via the main graph API called `imported_plot`, which is similar to `plot` in form but will be completed with previously imported data. Currently it simply returns the values from `plot` / 2. The data is rendered in the main graph in red without fill, and without an indicator for the present. Rationale: imported data will not continue to grow so there is no projection forward, only backwards. * Hook imported GA data to dashboard timeseries plot * Add settings option to forget imported data * Import sources from google analytics * Merge imported sources when queried * Merge imported source data native data when querying sources * Start converting metrics to atoms so they can be subqueried This changes "visitors" and in some places "sources" to atoms. This does not change the behaviour of the functions - the tests all pass unchanged following this commit. This is necessary as joining subqueries requires that the keys in `select` statements be atoms and not strings. * Convery GA (direct) source to empty string * Import utm campaign and utm medium from GA * format * Import all data types from GA into new tables * Handle large amounts of more data more safely * Fix some mistakes in tables * Make GA requests in chunks of 5 queries * Only display imported timeseries when there is no filter * Correctly show last 30 minutes timeseries when 'realtime' * Add with_imported key to Query struct * Account for injected :is_not filter on sources from dashboard * Also add tentative imported_utm_sources table This needs a bit more work on the google import side, as GA do not report sources and utm sources as distinct things. * Return imported data to dashboard for rest of Sources panel This extends the merge_imported function definition for sources to utm_sources, utm_mediums and utm_campaigns too. This appears to be working on the DB side but something is incomplete on the client side. * Clear imported stats from all tables when requested * Merge entry pages and exit pages from imported data into unfiltered dashboard view This requires converting the `"visits"` and `"visit_duration"` metrics to atoms so that they can be used in ecto subqueries. * Display imported devices, browsers and OSs on dashboard * Display imported country data on dashboard * Add more metrics to entries/exits for modals * make sure data is returned via API with correct keys * Import regions and cities from GA * Capitalize device upon import to match native data * Leave query limits/offsets until after possibly joining with imported data * Also import timeOnPage and pageviews for pages from GA * imported_countries -> imported_locations * Get timeOnPage and pageviews for pages from GA These are needed for the pages modal, and for calculating exit rates for exit pages. * Add indicator to dashboard when imported data is being used * Don't show imported data as separately line on main graph * "bounce_rate" -> :bounce_rate, so it works in subqueries * Drop imported browser and OS versions These are not needed. * Toggle displaying imported data by clicking indicator * Parse referrers with RefInspector - Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual referrer host + path, whereas 'ga:source' includes utm_mediums and other values when relevant. - 'ga:fullReferror' does however include search engine names directly, so they are manually checked for as RefInspector won't pick up on these. * Keep imported data indicator on dashboard and strikethrough when hidden * Add unlink google button to import panel * Rename some GA browsers and OSes to plausible versions * Get main top pages and exit pages panels working correctly with imported data * mix format * Fetch time_on_pages for imported data when needed * entry pages need to fetch bounces from GA * "sample_percent" -> :sample_percent as only atoms can be used in subqueries * Calculate bounce_rate for joined native and imported data for top pages modal * Flip some query bindings around to be less misleading * Fixup entry page modal visit durations * mix format * Fetch bounces and visit_duration for sources from GA * add more source metrics used for data in modals * Make sources modals display correct values * imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration * Merge imported data into aggregate stats * Reformat top graph side icons * Ensure sample_percent is yielded from aggregate data * filter event_props should be strings * Hide imported data from frontend when using filter * Fix existing tests * fix tests * Fix imported indicator appearing when filtering * comma needed, lost when rebasing * Import utm_terms and utm_content from GA * Merge imported utm_term and utm_content * Rename imported Countries data as Locations * Set imported city schema field to int * Remove utm_terms and utm_content when clearing imported * Clean locations import from Google Analytics - Country and region should be set to "" when GA provides "(not set)" - City should be set to 0 for "unknown", as we cannot reliably import city data from GA. * Display imported region and city in dashboard * os -> operating_system in some parts of code The inconsistency of using os in some places and operating_system in others causes trouble with subqueries and joins for the native and imported data, which would require additional logic to account for. The simplest solution is the just use a consistent word for all uses. This doesn't make any user-facing or database changes. * to_atom -> to_existing_atom * format * "events" metric -> :events * ignore imported data when "events" in metrics * update "bounce_rate" * atomise some more metrics from new city and region api * atomise some more metrics for email handlers * "conversion_rate" -> :conversion_rate during csv export * Move imported data stats code to own module * Move imported timeseries function to Stats.Imported * Use Timex.parse to import dates from GA * has_imported_stats -> imported_source * "time_on_page" -> :time_on_page * Convert imported GA data to UTC * Clean up GA request code a bit There was some weird logic here with two separate lists that really ought to be together, so this merges those. * Fail sooner if GA timezone can't be identified * Link imported tables to site by id * imported_utm_content -> imported_utm_contents * Imported GA from all of time * Reorganise GA data fetch logic - Fetch data from the start of time (2005) - Check whether no data was fetched, and if so, inform user and don't consider data to be imported. * Clarify removal of "visits" data when it isn't in metrics * Apply location filters from API This makes it consistent with the sources etc which filter out 'Direct / None' on the API side. These filters are used by both the native and imported data handling code, which would otherwise both duplicate the filters in their `where` clauses. * Do not use changeset for setting site.imported_source * Add all metrics to all dimensions * Run GA import in the background * Send email when GA import completes * Add handler to insert imported data into tests and imported_browsers_factory * Add remaining import data test factories * Add imported location data to test * Test main graph with imported data * Add imported data to operating systems tests * Add imported data to pages tests * Add imported data to entry pages tests * Add imported data to exit pages tests * Add imported data to devices tests * Add imported data to sources tests * Add imported data to UTM tests * Add new test module for the data import step * Test import of sources GA data * Test import of utm_mediums GA data * Test import of utm_campaigns GA data * Add tests for UTM terms * Add tests for UTM contents * Add test for importing pages and entry pages data from GA * Add test for importing exit page data * Fix module file name typo * Add test for importing location data from GA * Add test for importing devices data from GA * Add test for importing browsers data from GA * Add test for importing OS data from GA * Paginate GA requests to download all data * Bump clickhouse_ecto version * Move RefInspector wrapper function into module * Drop timezone transform on import * Order imported by side_id then date * More strings -> atoms Also changes a conditional to be a bit nicer * Remove parallelisation of data import * Split sources and UTM sources from fetched GA data GA has only a "source" dimension and no "UTM source" dimension. Instead it returns these combined. The logic herein to tease these apart is: 1. "(direct)" -> it's a direct source 2. if the source is a domain -> it's a source 3. "google" -> it's from adwords; let's make this a UTM source "adwords" 4. else -> just a UTM source * Keep prop names in queries as strings * fix typo * Fix import * Insert data to clickhouse in batches * Fix link when removing imported data * Merge source tables * Import hostname as well as pathname * Record start and end time of imported data * Track import progress * Fix month interval with imported data * Do not JOIN when imported date range has no overlap * Fix time on page using exits Co-authored-by: mcol <mcol@posteo.net> |
5 years ago |