make robots.txt restrictive by default (#3905)

* make robots.txt restrictive by default

* add note about cloud robots.txt
reload-after-config r2024.03.19-399
ruslandoga 3 years ago committed by GitHub
parent 279e89c693
commit 64f35bf8f6
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

@ -32,6 +32,7 @@ All notable changes to this project will be documented in this file.
- Add `CLICKHOUSE_MAX_BUFFER_SIZE_BYTES` env var which defaults to `100000` (100KB)
- Add alternative SMTP adapter plausible/analytics#3654
- Add `EXTRA_CONFIG_PATH` env var to specify extra Elixir config plausible/analytics#3906
- Add restrictive `robots.txt` for self-hosted plausible/analytics#3905
### Removed
- Removed the nested custom event property breakdown UI when filtering by a goal in Goal Conversions

@ -33,11 +33,21 @@ defmodule PlausibleWeb.Endpoint do
plug(PlausibleWeb.Tracker)
plug(PlausibleWeb.Favicon)
static_paths = ~w(css js images favicon.ico)
static_paths =
on_full_build do
# NOTE: The Cloud uses custom robots.txt from https://github.com/plausible/website: https://plausible.io/robots.txt
static_paths
else
static_paths ++ ["robots.txt"]
end
plug(Plug.Static,
at: "/",
from: :plausible,
gzip: false,
only: ~w(css js images favicon.ico robots.txt)
only: static_paths
)
on_full_build do

@ -1,5 +1,2 @@
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /
User-agent: *
Disallow: /

Loading…
Cancel
Save