Earth Notes: On Website Technicals (2020-06)

Updated 2020-07-21 16:42 GMT.
Tech updates: VIDEO/AUDIO style responsive tweaks, AutoAds on again, CSS trim, Ansible, desktop minify, throttle, pop star, HTTPS, HTTP/2, ADC, RPi speed.
tools
Lots of work on the RPi3 server upgrade, previously on hold for something like two years! AutoAds are on again anticipating for a post-covid rebound. Plus small tweaks and optimisations, and learning to Ansible!

2020-06-29: Relative CPU Speed

Doing a bit of single-threaded text processing and graph-generation work on my MacBook Air (1.7GHz Dual-Core Intel Core i7, hyperthreaded):

time make out/monthly/16WW-date-HDD12-kWh-2019.png
...
2.293u 0.315s 0:02.66 97.7%	0+0k 0+0io 0pf+0w

On the RPi3B (armv7l 4x1.2GHz, no thermal or other throttling I think):

time make out/monthly/16WW-date-HDD12-kWh-2019.png
...
19.634u 0.682s 0:20.27 100.1%	0+0k 0+40io 0pf+0w

On the RPi2B (armv7l 4x0.9GHz):

time make out/monthly/16WW-date-HDD12-kWh-2019.png
...
29.160u 0.600s 0:31.21 95.3%	0+0k 0+40io 0pf+0w

All were a second or later run to ensure caches reasonably warm.

2020-06-26: HTTPS and HTTP/2

In the process of starting to migrate services to the RPi3B, I'm learning lots about Ansible, and fixing some long-standing issues. Given that I can apply the Ansible configuration over and over until I get it right, it now seems worth nailing down some things that I had (or would have) forgotten before.

For example, it turns out that by default with Apache 2.4 settings the favicon.ico site icon is not compressed, even though it can be bulky and is highly compressible with gzip. I'd covered that with site-wide configuration on the existing server, and was able to spot the problem with WebPageTest and ensure the fix is forced into the configuration of the new one.

Also, I have started making https versions of some sites available. The certificates are provided by Lets's Encrypt using:

certbot run -a webroot -i apache -w WebRootDirectory -d example.com

Let's Encrypt piggybacks off the http service for example.com, which much be running, and creates an https Apache configuration and an automatic unattended renewal process, which is clever.

Let's Encrypt's setup does trample on a few things, such as log file details. It also assumes that you won't be setting up your own https site configuration explicitly, so in my case I have to disable the Let's Encrypt version to avoid conflict.

I've used this technique (thank you StackOverflow and others!) to avoid duplicate configuration for http and https versions of the same site:

# HTTP
<VirtualHost *:80>
Include /etc/apache2/sites-include-common/www2.exnet.com.inc
</VirtualHost>
# HTTPS
<IfModule mod_ssl.c>
<VirtualHost *:443>
# Allow HTTP/2.
Protocols h2 http/1.1
# Let's Encrypt SSL support.
SSLCertificateFile /etc/letsencrypt/live/www2.exnet.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www2.exnet.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
# Prevent LE changing the log format (use 'other vhost' as would be default).
CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined
# Include site config last to give it a chance to override everything.
Include /etc/apache2/sites-include-common/www2.exnet.com.inc
</VirtualHost>
</IfModule>

Note the Protocols h2 http/1.1 that seems to be necessary, as well as loading the http2 module, to get HTTPS/2 running. But it is! (Next up, HTTP/3!)

Attemting to expand the new www2.exnet.com certificate to cover plain www.exnet.com with:

% sudo certbot --expand -d www2.exnet.com -d www.exnet.com

is not going to succeed until both are pointing at the same machine (which can be done with some (possibly temporary) DNS changes:

...
   Domain: www.exnet.com
   Type:   unauthorized
   Detail: Invalid response from
   http://www.exnet.com/.well-known/acme-challenge/blahrandomblah
   [79.135.97.XX]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"
...

This command works to add www.exnet.com:

sudo certbot -a webroot -i apache -r /rw/docs-public/www --cert-name www2.exnet.com -d www2.exnet.com -d www.exnet.com

Expander PI ADC

Another goal today is to get the Expander Pi HAT board working to sample the 12V-nominal supply voltage, at least initially with the ABElectronics_CPP_Libraries.

The RPi3B setup is already quite convoluted, so I'll break out this bit here, even though with a stronger hardware flavour than usual!

The setup as now on the RPi2B is based on a FSD of 2.048V on the MCP3424. The Expander Pi uses a MCP3208 8-Channel 12-Bit A/D Converter with FSD of 4.096V.

With resistors, I can construct a divider and protect the ADC input. Ideally I'd have ~4:1 divider for FSD ~16V, just higher than the max expected. The best precise resistors to hand are 86.8kΩ and 11.5kΩ, which gives = (86.8+11.5):11.5 = 98.3:11.5 = 8.55:1 ratio, which gives FSD of 35.01V, somewhat higher than necessary, losing about one bit of precision, but OK for now.

Having wired it all up, an initial reading from the ADC suggests 12.39V, vs 12.68V as seen from the RPI2B's ADC, and 12.83V as seen by the solar controller. A good start with ~3% under-reading: some tweaking may be required!

Yesterday I added a flag as shown here to the top-10 popular pages (those with most visits).

I also put a star (as in the flag) next to the various mentions of popular pages around the site, to tie them together...

Today I improved the scheme to work better when there are multiple pages at a given rank (pages with equal numbers of hits), around 10 in particular.

I also added the rank to a new footer.

None of this is done for lite (or error) pages, nor for anything without a captured hit count, eg unvisited.

2020-06-19: Desktop Minify

I've turned on (gentle) HTML minification for desktop pages; a bit less hardcore than for lite (m-dot) pages. In particular, I'm using --preserve-line-breaks so that the HTML remains more easily human-readable and debuggable. I'm also using --no-include-auto-generated-tags so that the minifier doesn't try to helpfully re-insert optional tags that I've artfully omitted in various places...

(The main but not only effect of this gentle minification is to remove unnecessary quotes from around tag attributes.)

Preserving linebreaks reduces the saving before and after gzip from ~2% to ~1%, but that lower figure is still worth having.

The other reason for doing the minification is to get early warning of HTML syntax errors before hitting the slow bulk validation.

Before:

 2026 Jun 16 404.html
  910 Jun 16 404.htmlgz
36418 Jun 15 thermal-imaging-survey-of-house.html
10092 Jun 15 thermal-imaging-survey-of-house.htmlgz

After:

 2024 Jun 19 404.html
  904 Jun 19 404.htmlgz
36244 Jun 19 thermal-imaging-survey-of-house.html
 9994 Jun 19 thermal-imaging-survey-of-house.htmlgz

Extra zopfli supercompression unthrottled

The EOU site build can put extra effort into super-compressing HTML pages as long as no 'low-power' flag is set, based on the system battery charge. That set of flags changes slowly, typically every 10 minutes. So the test is done once at the start of each make run.

Because I can, I tweaked the make to avoid the extra effort if the CPU is detectably being throttled, eg due to heat or undervoltage. That value can change on the order of seconds, so is performed before each compression.

Thus, the extra work (with small extra btye savings) is done only if there is no shortage of energy and the CPUs are running full tilt. Else the system will finish quicker and with less energy used.

On systems where throttling cannot be detected, other than a newish RPi, only the low-power flag has any effect.

% make Wpages
...
Rebuilding GZIP super-compressed (--i30) note-on-site-technicals-1.htmlgz
Rebuilding GZIP super-compressed (--i30) mass-loft-insulation.htmlgz
Rebuilding GZIP super-compressed (--i30) note-on-site-technicals-21.htmlgz
Rebuilding GZIP super-compressed () diarycast-20200513.htmlgz
Rebuilding GZIP super-compressed () towards-a-LZC-home.htmlgz
Rebuilding GZIP super-compressed (--i30) note-on-site-technicals-28.htmlgz
Rebuilding GZIP super-compressed (--i30) note-on-E2-Energy-Solutions-Expo-2007.htmlgz
Rebuilding GZIP super-compressed (--i30) Smart-Systems-and-Heat-Phase-2-learnings.htmlgz
Rebuilding GZIP super-compressed () from-the-inbox-4.htmlgz
Rebuilding GZIP super-compressed (--i30) note-on-site-technicals-20.htmlgz
...

2020-06-17: Ansible

I fear that at some point soon I'm going to have to rebuild my (new) Pi server from scratch, again. And I really don't want to have to do the whole darn thing by hand in full again and again, for the rest of my life, when a hardware or even big OS upgrade is needed.

So I'm going to think very hard about using Ansible so that I can capture at least 90% of the grunt work in a bunch of config files, with only a few very tricky bits (from reformatting, upwards) to be done manually, and/or to flip from 'secondary' to 'primary'.

This configuration will evolve over time as needs change, but there can be an audit trail, with the whole thing likely fairly small and managed in a SVN or git repo. (There may be a need to hand-build a few key utilities, such as zopflipng; maybe the code for them can be held in the config repo too if small.)

To that end, on my MacBoock Air, I am warming up now with:

% brew install ansible
2020-06-19: progress has been quite decent, and indeed has allowed me to fix some persistent npm install -g permissions problems too. (Seemingly an npm bug.)

2020-06-15: Adapter vs Adaptor

I have been using "adaptor" very roughly 2:1 vs "adapter" on EOU.

The "-er" spelling seems to be generally more common.

Also, since I think of "-or" variants as sometimes implying a person, eg the original "computors" who slaved over pencil and paper or adding machines, often women, I've switched to "-er" for power adapaters and the like.

2020-06-14: CSS Simplification

I have been able to simplify the base and AMP CSS to save a bit of space in the header of all pages, still more in all AMP pages. This may help avoid having to drop functionality from otherwise-oversized head/CRP in some cases.

(The number of pages being automatically rebuilt, with -H to force the header small enough for real content to finish the first TCP frame, was 12 across desktop/lite/AMP before this change and 3 after.)

Controversy of the day: is margin: auto actually equivalent to margin:0 auto for a main (centred) container?

I'm also trying to strip out some other unnecessary fluff on the CRP (Critical Rendering Path) before the very first words of content in the H1.

2020-06-08: AUDIO and VIDEO Text 'dopt'

To help AUDIO fit better into small spaces, eg tables, I have made some of the less critical text class=dopt. If that CSS class is provided via EXTCSS and the available space is very narrow, then the text will disappear so as to reduce clutter.

For consistency I have done the same for VIDEO.

2020-06-07: AutoAds On Again

Given that lockdown is starting to ease again, especially for retailers, I thought that it might be time to poke my head above the parapet, and crank ads up a bit. So, AutoAds is back on for EOU, but with minimum ad load, and with the most egregious positons above the fold (and abive any content) blocked.

2020-06-11: revenue is trickling in, and I'm not seeing horrible ads, so I've cranked AutoAds up from "min to a bit below medium.

2020-06-14: I'm now allowing AutoAds on AMP pages again, since revenue on desktop pages seems decent. Not many AMP pages are allowing any ads currently, and AutoAds doesn't seem able to make use of existing Google Ads slots, so I've re-enabled explicit AutoAds code in my AMP pages.

2020-06-06: Responsive AUDIO

For the latest (May) statscast I wanted to be able to fit several AUDIO elements in adjacent columns of a table, and they are rather wide for that. But default, for at least Chrome and Firefox, the audio element does not seem responsive at alli, so unfriendly for narrow spaces in general.

To fix this I added style=display:block;max-width:100%. For consistency, I added it to both audio and video.