Earth Notes: On Website Technicals (2020-11)

Without adding any overhead (eg extra headers) to normal connections, but to gently redirect spiders to the WWW https versions of most files, I have inserted the following early config for www.earth.org.uk.

# Redirect most Referer-less http accesses to https.
# Aim to gently redirect spiders to canonical https for most content.
# Avoid redirecting (top-level) HTML files that contain own rel=canonical,
# so users directly choosing http can stay on http.
# Avoid breaking the LE ACME challenge.
# Use a 302 (temporary) redirect, for now.
RewriteEngine on
RewriteCond %{HTTPS} off
RewriteCond %{HTTP_REFERER} ^$
RewriteCond %{REQUEST_URI} !^/\.well-known/
RewriteCond %{REQUEST_URI} !^/[^/]*\.html$
RewriteCond %{REQUEST_URI} !^$
RewriteCond %{REQUEST_URI} !^/$
RewriteRule ^/(.*)$ https://www.earth.org.uk/$1 [L,R=302]

Currently there are slightly more http than https WWW requests, by ~10%.

Today I am making amp.EOU https only, eg by redirecting http to https.

Since AMP users are already paying the overhead of the AMP JavaScript, etc, the bandwidth- and latency- saving aspects of http are likely less important to them. They may likely already have being pushed to the AMP page via https-based search and AMP cache.

Making the http option disappear should save a little crawl bandwidth but spiders. About one quarter of AMP page hits are currently http.

Then I will be down from serving 6 variants of each page — www, m, amp for each of http and https — to 5. It's a start.

There's a little wrinkle to avoid interfering with Let's Encrypt auto-renew.

RewriteEngine on
RewriteCond %{HTTPS} off
RewriteCond %{REQUEST_URI} !^/\.well-known/
RewriteRule ^/(.*)$ https://amp.earth.org.uk/$1 [L,R=301]

I also aim to add a couple of other tweaks, eg that make my security rating in WebPageTest better than the current "F"!

The biggest complaint from WebPageTest is fixed by adding the header Strict-Transport-Security: max-age=31536000 which should force a browser to use https for amp.EOU for a year. This raises the WPT security rating for the home page from "F" to "E".

Adding the header X-Frame-Options: DENY, which I already use for the desktop site, improves the security score to "D", but seems to stop images loading in Firefox (though not Chrome Canary). The header is apparently effectively obsoleted by the Content-Security-Policy header though. Given all that, it's not staying!

I note that https://www.theguardian.com/uk includes these headers:

• x-frame-options: SAMEORIGIN
• content-security-policy: default-src https:; script-src https: 'unsafe-inline' 'unsafe-eval' blob: 'unsafe-inline'; frame-src https: data:; style-src https: 'unsafe-inline'; img-src https: data: blob:; media-src https: data: blob:; font-src https: data:; connect-src https: wss:; child-src https: blob:; object-src 'none'; base-uri 'none'
• referrer-policy: no-referrer-when-downgrade
• strict-transport-security: max-age=31536000; includeSubDomains; preload
• x-content-type-options: nosniff
• x-xss-protection: 1; mode=block

It seems as if x-xss-protection is also only for older browsers, and I should concentrate my efforts on crafting content-security-policy.

Part may be Referrer-Policy: origin-when-cross-origin, or the equivalent via content-security-policy.

Another part may be script-src https://cdn.ampproject.org:* to let the AMP scripts run, though that may not let Google ads run.

I do use a little inline CSS to keep the header and CRP (Critical Rendering Path) small, which implies something like style-src 'unsafe-inline' which weakens the whole mechanism. Maybe I should wean myself off local CSS in all critical cases instead.

(I removed from the sitemap an XHTML page that is not canonical.)

Lazy loading seems to win in two ways. Reducing bandwidth is the obvious one, but also in reducing initial visible page rendering time even when not.

So, for example, on the home page, Chrome does not avoid loading any images because even the ones below the fold are not far enough below. But it seems that by letting Chrome concentrate on the important bits above the fold, initial layout is faster. Firefox manages to save bandwidth too by avoiding loading several images for the initial view. Those images will never be loaded if the visitor does not scroll down; even if they do, load on the server is spread out.

Here are three simple scenarios, all from WebPageTest instances in London, all over HTTP/2 (ie one TCP connection) to https://www.earth.org.uk.

Note that the bandwidth limits were not identical across all runs, but higher bandwidth does not beat the advantages of lazy loading!

Chrome without Lazy Loading

For this run all loading=lazy attributes were manually removed from the HTML.

Chrome with Lazy Loading

Firefox with Lazy Loading

Amongst ignorable email complaints from Let's Encrypt I received a worrying one that implied that the actual EOU TLS certs were going to expire.

Looking in the Apache logs I could see redirects and errors during the http-01 challenge for the amp. and m. sites. The two sites fairly aggressively redirect to www. anything that does not look like a top-level HTML page (or script or favicon, etc). That breaks the GET /.well-known/acme-challenge/.... So I put in special-case fixes to not rewrite/redirect any such requests.

Having done that, renewal succeeded by manually running:

% sudo certbot renew

With a fair wind behind, auto-renewal should "just work"!

(See previous work storage note and next.)

I have made a few more tweaks, eg to stop almost all DAILY and WEEKLY periodic updates when battery is LOW or below.