Earth Notes: On Website Technicals (2023-08)
Updated 2024-02-29.2023-08-31: ZIP Main Pages Archive
I have provided a simple ZIP version of the archive, since that may be much easier for most people to understand and use, even though it will be a little larger, eg:
8530288 out/monthly/main-pages-archive.tar.xz 10448110 out/monthly/main-pages-archive.zip
I have provided links to both versions in a new home-page section.
2023-08-30: Embargoed Text
To save remembering to do this stuff manually, I have a new mechanism to embargo (hide) text blocks until a specified date.
The first rebuild on or after that date (UTC) will allow the text to be shown.
That may mean in some cases that I have to manually force a page rebuild. But I also automagically, weekly, when energy levels are HIGH, attempt to rebuild any embargoed pages. Which should be soon enough!
This section is wrapped in an (expired) embargo...
2023-08-27: Main Pages Archive
Woot! I have just created another huge time sink: just what I need!
I have started creating a compact (solid LZMA2 tar) archive of the main page at out/monthly/main-pages-archive.tar.xz. This is currently using the mobile pages as-is, but will probably switch to a dedicated 'offline' mobile variant that will for example pick one unique low-weight WebP or 'L' variant of each image for each IMG (no srcset) when available. Those images, plus (rare) out-of-line included CSS stylesheets etc, will be added to the archive. Links back to 'full' versions of images on-line may be provided, as now.
The idea is that this single file can be downloaded and unpacked for offline reading, and can also be archived periodically as-is, eg annually.
At the moment the bare page archive is less than 2MB. (The uncompressed 'XHTML' hand-edited source is ~7MB.) That will swell greatly once images are added.
This is possibly ripe for use as an SPA (Single Page Application) backing store that can download the archive and use elements from it when connectivity goes down. A random-access (eg ZIP) format might be better for this, though it will compress less well.
Adding (available) image sources (each src) to the archive, silently dropping missing dynamic images for example, pushes the archive size up to ~11MB.
Automatically using the .webp version of any image when available reduces the archive size to ~8.5MB. The largest residuals are:
% find img -type f -ls | sort -n ... 24105 29 Oct 2017 img/LiFePO4-initial-integration-sketch.png 24162 20 Aug 2021 img/a/b/zero-carbon-house-solar-heating.l119077.576x445.m.png.webp 24218 1 Sep 2021 img/a/b/20210831-AudioMoth-MW.l255704.576x372.m.png 25133 8 Oct 2020 img/a/b/20201007-32167ba578d32cd3520ee62f470f9809.l321337.576x324.m.png 25202 20 Aug 2021 img/a/b/20200827-5b8874936086c10127f7080a822ef2d4.l516887.576x768.m.jpg.webp 25516 20 Aug 2021 img/a/b/20210406-d108d897c403b30e6b9764ef08a34ef5.l910557.576x768.m.jpg.webp 26415 1 Jan 2023 img/LocalBytes-monitoring-plug/2022-10-07-fridge/LBplug-all.power.svg 26877 18 Apr 2021 img/a/b/20210406-37a4c9ea00e40f7099dee48b9ae4565e.l1337576.576x768.m.jpg 28178 12 Aug 18:31 img/a/b/looped-electricity-supply-JL.l556537.576x768.m.jpg 44683 1 Jan 2023 img/LocalBytes-monitoring-plug/2022-10-10-TV/LBplug-TV-corner-power-2.svg
Looking for outliers
This little script helps look for large manually-captured PNG and JPEG rasters that do not (yet) have a smaller efficient WebP version which could save offline archive space (and other bandwidth too):
% cat .offline/.build/* | awk '/^INFO: IMGSRC: / && ! ( $3 ~ /^img\/a\// ) && $3 ~ /\.((png)|(jpg))$/' | sort -n -k 4 | tail INFO: IMGSRC: img/meet/WE2017/LightFi.jpg 11723 INFO: IMGSRC: img/7W-LED-bayonet-north-light.jpg 11934 INFO: IMGSRC: img/7W-LED-bayonet-north-light.jpg 11934 INFO: IMGSRC: img/solar-panel-on-wall.jpg 11957 INFO: IMGSRC: img/battery-and-controller.jpg 12287 INFO: IMGSRC: img/BREipark/KODA-House-kitchen-from-above.jpg 13005 INFO: IMGSRC: img/V5-7W-GU10-LED.jpg 14380 INFO: IMGSRC: img/eUW2017/Centriforce-tapes-and-board-tn.jpg 15267 INFO: IMGSRC: img/energy-monitors-Efergy-elite-npower-smartpower-1kW-travel-kettle-tn.jpg 16875 INFO: IMGSRC: img/site/icon/touch-icons/apple-touch-icon-120x120.png 22972
Note the duplicate entry for img/7W-LED-bayonet-north-light.jpg, used in more than one page.
2023-08-26: Code Coverage
Code coverage just started working on my old MBA, and from jUnit tests coverage is over 71% for TRVmodel and nearly 87% for statsHouse. Much better than I had expected! Probably I should aim to get and keep both over ~80%.
Minutes later ... I pushed TRVmodel over 86% with one weird trick!
2023-08-04: Static Content URLs
At the moment all static and dynamic content is served from EOU's www URL. (Except for mobile/lite pages under m.)
However there is provision to serve static content (mainly images) from the static URL. One advantage is that if done right it would not be polluted with cookies from HTML pages using Google ads for example. That could save a little bandwidth.
As long as the certificates and DNS are set up right that should not involve extra DNS look-ups and may allow some sharding, eg under HTTP.
At the moment I have to elect whether the link is to be under the 'www' dynamic or 'static' nominal URLs, even though they map to the same thing for now.
This is potentially error-prone, for example missing opportunities to serve images under /img/ from the static site.
So I just added a rewrite rule in my page-generation script that forces use of the nominal static URL if the path is under /img/. Also, I am now only allowing use of the 'static' URL for things there.
I could also force use of the dynamic URL for anything else, for example. I could also consider what to do about /data/ items, all images and CSS by suffix and/or MIME type, etc. In any case the 'static' site could block any new outgoing Set-Cookie to keep the sub-domain clean, eg using the Apache Header feature.
Again, for now this will make zero change to any generated pages, but it feels logically and incrementally better than the current situation.
2023-08-03: INTVKL
In anticipation of the New Interconnector Fuel Type Category: Viking Link going live in three months, and though the data is not yet present in the Elexon feed, INTVKL was added to the EOU live grid-intensity calculations.
Because I do not have a static intensity measure for the Danish grid at the other end of INTVKL, I have copied the Norwegian grid value for now.
2023-12-29: live!
National Grid announces commercial operations of Viking Link: a few days earlier than I expected!