Earth Notes: On Website Technicals (2023-08)

Updated 2023-09-12.
Tech updates: INTVKL, static content URLs, code coverage, main pages archive, embargoes.
Still summer, still quiet, holidays, but a few tweaks such as getting ready for a new electricity interconnector (INTVKL) and creating an offline EOU archive!

2023-08-31: ZIP Main Pages Archive

I have provided a simple ZIP version of the archive, since that may be much easier for most people to understand and use, even though it will be a little larger, eg:

 8530288 out/monthly/main-pages-archive.tar.xz
10448110 out/monthly/

I have provided links to both versions in a new home-page section.

2023-08-30: Embargoed Text

To save remembering to do this stuff manually, I have a new mechanism to embargo (hide) text blocks until a specified date.

The first rebuild on or after that date (UTC) will allow the text to be shown.

That may mean in some cases that I have to manually force a page rebuild. But I also automagically, weekly, when energy levels are HIGH, attempt to rebuild any embargoed pages. Which should be soon enough!

This section is wrapped in an (expired) embargo...

2023-08-27: Main Pages Archive

Woot! I have just created another huge time sink: just what I need!

I have started creating a compact (solid LZMA2 tar) archive of the main page at out/monthly/main-pages-archive.tar.xz. This is currently using the mobile pages as-is, but will probably switch to a dedicated 'offline' mobile variant that will for example pick one unique low-weight WebP or 'L' variant of each image for each IMG (no srcset) when available. Those images, plus (rare) out-of-line included CSS stylesheets etc, will be added to the archive. Links back to 'full' versions of images on-line may be provided, as now.

The idea is that this single file can be downloaded and unpacked for offline reading, and can also be archived periodically as-is, eg annually.

At the moment the bare page archive is less than 2MB. (The uncompressed 'XHTML' hand-edited source is ~7MB.) That will swell greatly once images are added.

This is possibly ripe for use as an SPA (Single Page Application) backing store that can download the archive and use elements from it when connectivity goes down. A random-access (eg ZIP) format might be better for this, though it will compress less well.

Adding (available) image sources (each src) to the archive, silently dropping missing dynamic images for example, pushes the archive size up to ~11MB.

Automatically using the .webp version of any image when available reduces the archive size to ~8.5MB. The largest residuals are:

% find img -type f -ls | sort -n 
24105 29 Oct  2017 img/LiFePO4-initial-integration-sketch.png
24162 20 Aug  2021 img/a/b/zero-carbon-house-solar-heating.l119077.576x445.m.png.webp
24218  1 Sep  2021 img/a/b/20210831-AudioMoth-MW.l255704.576x372.m.png
25133  8 Oct  2020 img/a/b/20201007-32167ba578d32cd3520ee62f470f9809.l321337.576x324.m.png
25202 20 Aug  2021 img/a/b/20200827-5b8874936086c10127f7080a822ef2d4.l516887.576x768.m.jpg.webp
25516 20 Aug  2021 img/a/b/20210406-d108d897c403b30e6b9764ef08a34ef5.l910557.576x768.m.jpg.webp
26415  1 Jan  2023 img/LocalBytes-monitoring-plug/2022-10-07-fridge/LBplug-all.power.svg
26877 18 Apr  2021 img/a/b/20210406-37a4c9ea00e40f7099dee48b9ae4565e.l1337576.576x768.m.jpg
28178 12 Aug 18:31 img/a/b/looped-electricity-supply-JL.l556537.576x768.m.jpg
44683  1 Jan  2023 img/LocalBytes-monitoring-plug/2022-10-10-TV/LBplug-TV-corner-power-2.svg

Looking for outliers

This little script helps look for large manually-captured PNG and JPEG rasters that do not (yet) have a smaller efficient WebP version which could save offline archive space (and other bandwidth too):

% cat .offline/.build/* | awk '/^INFO: IMGSRC: / && ! ( $3 ~ /^img\/a\// ) && $3 ~ /\.((png)|(jpg))$/' | sort -n -k 4 | tail
INFO: IMGSRC: img/meet/WE2017/LightFi.jpg    11723
INFO: IMGSRC: img/7W-LED-bayonet-north-light.jpg    11934
INFO: IMGSRC: img/7W-LED-bayonet-north-light.jpg    11934
INFO: IMGSRC: img/solar-panel-on-wall.jpg    11957
INFO: IMGSRC: img/battery-and-controller.jpg    12287
INFO: IMGSRC: img/BREipark/KODA-House-kitchen-from-above.jpg    13005
INFO: IMGSRC: img/V5-7W-GU10-LED.jpg    14380
INFO: IMGSRC: img/eUW2017/Centriforce-tapes-and-board-tn.jpg    15267
INFO: IMGSRC: img/energy-monitors-Efergy-elite-npower-smartpower-1kW-travel-kettle-tn.jpg    16875
INFO: IMGSRC: img/site/icon/touch-icons/apple-touch-icon-120x120.png    22972

Note the duplicate entry for img/7W-LED-bayonet-north-light.jpg, used in more than one page.

2023-08-26: Code Coverage

Code coverage just started working on my old MBA, and from jUnit tests coverage is over 71% for TRVmodel and nearly 87% for statsHouse. Much better than I had expected! Probably I should aim to get and keep both over ~80%.

Minutes later ... I pushed TRVmodel over 86% with one weird trick!

2023-08-04: Static Content URLs

At the moment all static and dynamic content is served from EOU's www URL. (Except for mobile/lite pages under m.)

However there is provision to serve static content (mainly images) from the static URL. One advantage is that if done right it would not be polluted with cookies from HTML pages using Google ads for example. That could save a little bandwidth.

As long as the certificates and DNS are set up right that should not involve extra DNS look-ups and may allow some sharding, eg under HTTP.

At the moment I have to elect whether the link is to be under the 'www' dynamic or 'static' nominal URLs, even though they map to the same thing for now.

This is potentially error-prone, for example missing opportunities to serve images under /img/ from the static site.

So I just added a rewrite rule in my page-generation script that forces use of the nominal static URL if the path is under /img/. Also, I am now only allowing use of the 'static' URL for things there.

I could also force use of the dynamic URL for anything else, for example. I could also consider what to do about /data/ items, all images and CSS by suffix and/or MIME type, etc. In any case the 'static' site could block any new outgoing Set-Cookie to keep the sub-domain clean, eg using the Apache Header feature.

Again, for now this will make zero change to any generated pages, but it feels logically and incrementally better than the current situation.

2023-08-03: INTVKL

In anticipation of the New Interconnector Fuel Type Category: Viking Link going live in three months, and though the data is not yet present in the Elexon feed, INTVKL was added to the EOU live grid-intensity calculations.

Because I do not have a static intensity measure for the Danish grid at the other end of INTVKL, I have copied the Norwegian grid value for now.