Earth Notes: On Website Technicals (2020-09)

Noting the special support for an expiry time of 31536000 seconds (365 days) to mean 'forever' in QPACK: Header Compression for HTTP/3 (H3), I am making that the 'forever' time for EOU (www/static) too.

I have also removed the Firefox-only 'immutable' from the Cache-Control header. (In fact Safari also supports 'immutable', I see.) Most browsers probably won't use it, it takes some space, and prevents use of the built-in H3 static header compression entry.

The magic config line (to exactly match H3 the static table entry) is now:

Header set Cache-Control "public, max-age=31536000"

A 90-day AdSense test finished today, which indicated that having a heavier-than-average ad load did not generate more revenue than a below-average ad load. So I'm back on the less-pushy below-average setting.

I also took the opportunity to turn off the 'anchor' ads that stick to the top of (desktop) pages when viewed on mobile. I find them distracting and a significant waste of screen real-estate.

It's difficult to tell for sure, but I think that traffic from Google is coming off its peak of the last year or so, at least looking at the GSC performance graph. Impressions are good, but actual clickthroughs less convincing...

I tried out HTTP/2 Test: Verify HTTP/2 support. For https://www.earth.org.uk I get HTTP/2 protocol is supported and ALPN extension is supported. (Both are also supported over plain http, but no browser will make use of that in practice! For some reason although http AMP shows the same, http m says that both are unsupported, on https also: odd.)

It turns out there was some stray Let's Encrypt 'auto' config hanging around:

# ls -al /etc/apache2/sites-enabled/
m.earth.org.uk.conf -> ../sites-available/m.earth.org.uk.conf
m.earth.org.uk-le-ssl.conf -> /etc/apache2/sites-available/m.earth.org.uk-le-ssl.conf

Removing the link, and restarting Apache makes all well with the world, or at least well with the HTTP/2 test:

# rm /etc/apache2/sites-enabled/m.earth.org.uk-le-ssl.conf
# /etc/init.d/apache2 restart

% egrep ':443 .* HTTP/1' /var/log/apache2/other_vhosts_access.log | wc -l
5934
% egrep ':443 .* HTTP/2' /var/log/apache2/other_vhosts_access.log | wc -l
1985

As of today the preferred scheme for the AMP pages is https, though http will still be served and should be fully functional.

This means in practice:

• Making the amphtml link in the header and AMP navigation link point to the https version.
• Giving all the header inter-version and canonical and navigation links explicit http / https schemes.
• Making redirects between versions appropriate in terms of schemes, retaining https as appropriate to avoid security snafus (principle of least surprise).
• Using the correct scheme in robots.txt and sitemap.xml as appropriate.

It will take a little while to get all the wrinkles out!

In future, the m/lite preferred version is likely to remain http for speed, and the www/desktop preferred version become https for a small SEO boost.

2020-10-28: HTTPS Doubles Page Crawl Time

GSC's average page download time (now at ~10kB per page) has risen from ~250ms to ~500ms with the switch from http to https preferred.

Note that Google's crawlers are generally visiting from across the Atlantic, multiplying the effects of the extra TLS setup handshake. If the crawlers ever get to use H3 (HTTP/3) including header compression, some of that may come back.

As of today mean page fetch size is a little under 10kB and ~180 pages are crawled per day, as given in GSC's crawl stats for the AMP site version.

I have enabled Brotli static pre-compression for supporting top-level pages, such as the home and sitemap pages. If that does not cause any problems then I shall extend such br content-encoding to main pages also, either side of making the https set canonical for (say) AMP and desktop. (Brotli compression only works over https.)