Earth Notes: On Website Technicals (2020-06)
Updated 2024-03-21.2020-06-29: Relative CPU Speed
Doing a bit of single-threaded text processing and graph-generation work on my MacBook Air (1.7GHz Dual-Core Intel Core i7, hyperthreaded):
time make out/monthly/16WW-date-HDD12-kWh-2019.png ... 2.293u 0.315s 0:02.66 97.7% 0+0k 0+0io 0pf+0w
On the RPi3B (armv7l 4x1.2GHz, no thermal or other throttling I think):
time make out/monthly/16WW-date-HDD12-kWh-2019.png ... 19.634u 0.682s 0:20.27 100.1% 0+0k 0+40io 0pf+0w
On the RPi2B (armv7l 4x0.9GHz):
time make out/monthly/16WW-date-HDD12-kWh-2019.png ... 29.160u 0.600s 0:31.21 95.3% 0+0k 0+40io 0pf+0w
All were a second or later run to ensure caches reasonably warm.
2020-06-26: HTTPS and HTTP/2
In the process of starting to migrate services to the RPi3B, I am learning lots about Ansible, and fixing some long-standing issues. Given that I can apply the Ansible configuration over and over until I get it right, it now seems worth nailing down some things that I had (or would have) forgotten before.
For example, it turns out that by default with Apache 2.4 settings the favicon.ico site icon is not compressed, even though it can be bulky and is highly compressible with gzip. I'd covered that with site-wide configuration on the existing server, and was able to spot the problem with WebPageTest and ensure the fix is forced into the configuration of the new one.
Also, I have started making https versions of some sites available. The certificates are provided by Lets's Encrypt using:
certbot run -a webroot -i apache -w WebRootDirectory -d example.com
Let's Encrypt piggybacks off the http service for example.com, which much be running, and creates an https Apache configuration and an automatic unattended renewal process, which is clever.
Let's Encrypt's setup does trample on a few things, such as log file details. It also assumes that you won't be setting up your own https site configuration explicitly, so in my case I have to disable the Let's Encrypt version to avoid conflict.
I have used this technique (thank you StackOverflow and others!) to avoid duplicate configuration for http and https versions of the same site:
# HTTP
<VirtualHost *:80>
Include /etc/apache2/sites-include-common/www2.exnet.com.inc
</VirtualHost>
# HTTPS
<IfModule mod_ssl.c>
<VirtualHost *:443>
# Allow HTTP/2.
Protocols h2 http/1.1
# Let's Encrypt SSL support.
SSLCertificateFile /etc/letsencrypt/live/www2.exnet.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www2.exnet.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
# Prevent LE changing the log format (use 'other vhost' as would be default).
CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined
# Include site config last to give it a chance to override everything.
Include /etc/apache2/sites-include-common/www2.exnet.com.inc
</VirtualHost>
</IfModule>
Note the Protocols h2 http/1.1 that seems to be necessary, as well as loading the http2 module, to get HTTPS/2 running. But it is! (Next up, HTTP/3!)
Attempting to expand the new www2.exnet.com certificate to cover plain www.exnet.com with:
% sudo certbot --expand -d www2.exnet.com -d www.exnet.com
is not going to succeed until both are pointing at the same machine (which can be done with some (possibly temporary) DNS changes:
... Domain: www.exnet.com Type: unauthorized Detail: Invalid response from http://www.exnet.com/.well-known/acme-challenge/blahrandomblah [79.135.97.XX]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p" ...
This command works to add www.exnet.com:
sudo certbot -a webroot -i apache -r /rw/docs-public/www --cert-name www2.exnet.com -d www2.exnet.com -d www.exnet.com
Expander PI ADC
Another goal today is to get the Expander Pi HAT board working to sample the 12V-nominal supply voltage, at least initially with the ABElectronics_CPP_Libraries.
The RPi3B setup is already quite convoluted, so I'll break out this bit here, even though with a stronger hardware flavour than usual!
The setup as now on the RPi2B is based on a FSD of 2.048V on the MCP3424. The Expander Pi uses a MCP3208 8-Channel 12-Bit A/D Converter with FSD of 4.096V.
With resistors, I can construct a divider and protect the ADC input. Ideally I'd have ~4:1 divider for FSD ~16V, just higher than the max expected. The best precise resistors to hand are 86.8kΩ and 11.5kΩ, which gives = (86.8+11.5):11.5 = 98.3:11.5 = 8.55:1 ratio, which gives FSD of 35.01V, somewhat higher than necessary, losing about one bit of precision, but OK for now.
Having wired it all up, an initial reading from the ADC suggests 12.39V, vs 12.68V as seen from the RPI2B's ADC, and 12.83V as seen by the solar controller. A good start with ~3% under-reading: some tweaking may be required!
2020-06-22: Popularity Flag
Yesterday I added a flag as shown here to the top-10 popular pages (those with most visits).
I also put a star (as in the flag) next to the various mentions of popular pages around the site, to tie them together.
Today I improved the scheme to work better when there are multiple pages at a given rank (pages with equal numbers of hits), particularly around rank 10 itself.
I also added the rank to a new footer.
None of this is done for lite (or error) pages, nor for anything without a captured hit count, eg unvisited.
2020-06-19: Desktop Minify
I have turned on (gentle) HTML minification for desktop pages; a bit less hardcore than for lite (m-dot) pages. In particular, I am using --preserve-line-breaks so that the HTML remains more easily human-readable and debuggable. I am also using --no-include-auto-generated-tags so that the minifier does not try to helpfully re-insert optional tags that I have artfully omitted in various places...
(The main but not only effect of this gentle minification is to remove unnecessary quotes from around tag attributes.)
Preserving line breaks reduces the saving before and after gzip from ~2% to ~1%, but that lower figure is still worth having.
The other reason for doing the minification is to get early warning of HTML syntax errors before hitting the slow bulk validation.
Before:
2026 Jun 16 404.html 910 Jun 16 404.htmlgz 36418 Jun 15 thermal-imaging-survey-of-house.html 10092 Jun 15 thermal-imaging-survey-of-house.htmlgz
After:
2024 Jun 19 404.html 904 Jun 19 404.htmlgz 36244 Jun 19 thermal-imaging-survey-of-house.html 9994 Jun 19 thermal-imaging-survey-of-house.htmlgz
Extra zopfli supercompression unthrottled
The EOU site build can put extra effort into super-compressing HTML pages as long as no 'low-power' flag is set, based on the system battery charge. That set of flags changes slowly, typically every 10 minutes. So the test is done once at the start of each make run.
Because I can, I tweaked the make to avoid the extra effort if the CPU is detectably being throttled, eg due to heat or undervoltage. That value can change on the order of seconds, so is performed before each compression.
Thus, the extra work (with small extra byte savings) is done only if there is no shortage of energy and the CPUs are running full tilt. Else the system will finish quicker and with less energy used.
On systems where throttling cannot be detected, other than a newish RPi, only the low-power flag has any effect.
% make Wpages ... Rebuilding GZIP super-compressed (--i30) note-on-site-technicals-1.htmlgz Rebuilding GZIP super-compressed (--i30) mass-loft-insulation.htmlgz Rebuilding GZIP super-compressed (--i30) note-on-site-technicals-21.htmlgz Rebuilding GZIP super-compressed () diarycast-20200513.htmlgz Rebuilding GZIP super-compressed () towards-a-LZC-home.htmlgz Rebuilding GZIP super-compressed (--i30) note-on-site-technicals-28.htmlgz Rebuilding GZIP super-compressed (--i30) note-on-E2-Energy-Solutions-Expo-2007.htmlgz Rebuilding GZIP super-compressed (--i30) Smart-Systems-and-Heat-Phase-2-learnings.htmlgz Rebuilding GZIP super-compressed () from-the-inbox-4.htmlgz Rebuilding GZIP super-compressed (--i30) note-on-site-technicals-20.htmlgz ...
2020-06-17: Ansible
I fear that at some point soon I am going to have to rebuild my (new) Pi server from scratch, again. And I really don't want to have to do the whole darn thing by hand in full again and again, for the rest of my life, when a hardware or even big OS upgrade is needed.
So I am going to think very hard about using Ansible so that I can capture at least 90% of the grunt work in a bunch of config files, with only a few very tricky bits (from reformatting, upwards) to be done manually, and/or to flip from 'secondary' to 'primary'.
This configuration will evolve over time as needs change, but there can be an audit trail, with the whole thing likely fairly small and managed in a SVN or git repo. (There may be a need to hand-build a few key utilities, such as zopflipng; maybe the code for them can be held in the config repo too if small.)
To that end, on my MacBoock Air, I am warming up now with:
% brew install ansible2020-06-19: progress has been quite decent, and indeed has allowed me to fix some persistent
npm install -g permissions problems too. (Seemingly an npm bug.) 2020-06-15: Adapter vs Adaptor
I have been using "adaptor" very roughly 2:1 vs "adapter" on EOU.
The "-er" spelling seems to be generally more common.
Also, since I think of "-or" variants as sometimes implying a person, eg the original "computors" who slaved over pencil and paper or adding machines, often women, I have switched to "-er" for power adapaters and the like.
2020-06-14: CSS Simplification
I have been able to simplify the base and AMP CSS to save a bit of space in the header of all pages, still more in all AMP pages. This may help avoid having to drop functionality from an otherwise-oversized head and thus CRP (Critical Rendering Path) in some cases.
(The number of pages being automatically rebuilt, with -H to force the header small enough for real content to finish the first TCP frame, was 12 across desktop/lite/AMP before this change and 3 after.)
Controversy of the day: is margin: auto actually equivalent to margin:0 auto for a main (centred) container?
I am also trying to strip out some other unnecessary fluff on the CRP (Critical Rendering Path) before the very first words of content in the H1.
2020-06-08: AUDIO and VIDEO Text 'dopt'
To help AUDIO fit better into small spaces, eg tables, I have made some of the less critical text class=dopt. If that CSS class is provided via EXTCSS and the available space is very narrow, then the text will disappear so as to reduce clutter.
For consistency I have done the same for VIDEO.
As of , having refreshed the 'downloads' display, I wrapped some of the less vital new information chunks in dopt.
2020-06-07: AutoAds On Again
Given that lockdown is starting to ease again, especially for retailers, I thought that it might be time to poke my head above the parapet, and crank ads up a bit. So, AutoAds is back on for EOU, but with minimum ad load, and with the most egregious positions above the fold (and above any content) blocked.
2020-06-11: revenue is trickling in, and I am not seeing horrible ads, so I have cranked AutoAds up from minimum to a bit below medium.
2020-06-14: I am now allowing AutoAds on AMP pages again, since revenue on desktop pages seems decent. Not many AMP pages are allowing any ads currently, and AutoAds does not seem able to make use of existing Google Ads slots, so I have re-enabled explicit AutoAds code in my AMP pages.
2020-06-06: Responsive AUDIO
For the latest (May) statscast I wanted to be able to fit several AUDIO elements in adjacent columns of a table, and they are rather wide for that. But default, for at least Chrome and Firefox, the audio element does not seem responsive at all, thus unfriendly for narrow spaces in general.
To fix this I added style=display:block;max-width:100%. For consistency, I added it to both audio and video.