Earth Notes: On Website Technicals (2024-02)
Updated 2024-03-09.2024-02-25: lockfile
This evening I did brew install procmail on my Mac, and so have access to the real lockfile. Which means that I can suddenly run my builds in parallel, eg make -j 8 ..., taking full advantage of the M1's 8 cores!
2024-02-24: A11y: alt Preview Image Text
To improve Mastodon previews of tooted/posted articles I am adding og:image:alt text for each og:image in article page headers.
This can make headers over-long, so is automatically dropped in that case.
I have extended the script/altTextFromFilename.sh automatic alt text-from-image-name extractor to be able to use custom text from an .alt.txt file where extant.
2024-02-25: all but ~6%
After a full site rebuild, fewer than 30 of ~417 desktop pages are missing og:image:alt that could reasonably have it.
(A bit of editorial rejigging of titles and sub-heads and image names and alt text, and a few more pages have come into the fold...)
After the first auto-toot of an updated page, looking at the Web views of two Mastodon accounts, the server to which it was tooted (v4.2.7) does not show the alt text on hover, but a following server (v4.3.0-nightly.2024-02-23) does. Other toots on the first server seem to show the alt text however.
2024-02-23: Series RSS Feed
I have now made it easy to make an RSS feed for any page 'series', though for now I have set it up for Site Technicals.
The RSS feed file is created/updated as needed containing all the series pages, most recently updated first, and a link to the series head. Each series page gets built with a link to that RSS feed auto-inserted in the page header for auto-discovery with tools such as Brief, and in the series navigation for humans to discover/click manually.
I have also tentatively set this up for Saving Electricity.
2024-02-24: Desktop only
Just about as the complete site rebuild was finishing I decided that the auto-insertion of RSS links in series pages should probably be restricted to desktop full-fat pages only (not mobile/lite nor offline). Both to save some page weight but also for relevance: the feeds are of the desktop pages, less appropriate for other audiences.
2024-02-22: gCO2perkWh New! Improved! and Square!
By way of reducing Mastodon preview stampede load further from the regular grid intensity postings/toots, and to improve the preview look, I have created a smaller (by bytes and pixels) square og:image:
I have also added an og:image:alt of grid carbon intensity equation
.
Except actually it is bigger on all counts! Try again!
This results in a (Mastodon v4.2.7) toot thus:
Old vs new:
7535 17 Sep 12:27 gCO2perkWh-1.png 4478 17 Sep 12:27 gCO2perkWh-1.png.webp 3545 29 Jan 18:08 gCO2perkWh-1.pngL 3484 30 Jan 13:06 gCO2perkWh-1.png.webpL 6883 22 Feb 10:47 gCO2perkWh-sq.png 3870 22 Feb 10:50 gCO2perkWh-sq.png.webp 1780 22 Feb 10:47 gCO2perkWh-sq.pngL 1496 22 Feb 10:50 gCO2perkWh-sq.png.webpL
2024-02-18: significantLink
I have annotated the home-page links to featured and popular articles with significantLink as an experiment. (I have manually annotated one other link also.)
To allow this I additionally now have the home page declare itself to be of the schema.org type WebPage.
2024-02-15: Toot Lite When LOW
For once, an idea when falling asleep that turned out to be good... (Though I nearly forgot it entirely.)
When auto-tooting an EOU page to Mastodon, if the battery is LOW then toot the 'lite' (m.) page rather than the full-fat (www.) desktop page. That will use a little less bandwidth etc at the point of tooting and the stampede to read the page, but also subsequently less for anyone actually visiting the page. Plus it shows off that EOU has 'lite' pages.
This will not happen outside of winter months, generally.
The test could be changed to NOTHIGH, at which point some non-daytime toots would be lite too.
2024-02-12: Intensity Data Source Upgrade
Elexon will be turning off the unfashionable BMRS data service at https://www.bmreports.com/bmrs/ later this year.
In its place is a shiny new API at https://data.elexon.co.uk/bmrs/api/v1/ of which the bit I am interested in (FUELINST) has an endpoint [with] optimised JSON payload and is aimed at frequent requests
(datasets/FUELINST/stream) which sounds like the thing that I should use so as to be a good citizen.
After some faffing I have selected the public domain org.json JSON implementation for now, to parse inbound data.
I think that internally I will convert it to the old format for processing and cacheing initially. I can make some quick optimisations such as only requesting data since the last value already held if more recent than 24h.
2023-02-15: JSONed!
By avoiding doing my research and manuscript correction work, I think that I have finished the upgrade to use the new JSON source.
I think that I will keep testing it in the IDE manually, and look for glitches, but it can probably go live in a day or two.
Lots of old commented-out code currently lurks: that should be cleaned up.
I was able to make an optimisation to request far less data from Elexon each time, typically well under an hour's worth even being quite cautious, and this can likely be trimmed further.
2023-02-16: V1.4.0
Initial updated version (not shrunk with Proguard yet) released at V1.4.0.
At the start of the long store the old data nominally arrived in this CSV form from Elexon. At the very end of the long store it is synthesised from JSON.
% gzip -d < _gridCarbonIntensityGB.longstore.csv.gz | head HDR FUELINST,20240209,23,20240209111000,8521,0,956,3793,12456,36,709,100,347,994,0,1055,0,3177,999,998,0,1397,1412 FUELINST,20240209,23,20240209111500,8514,0,958,3798,12320,0,709,99,394,995,0,1056,0,3177,999,998,0,1397,1412 FUELINST,20240209,23,20240209112000,8481,0,957,3792,12350,0,694,100,506,995,0,1055,0,3170,999,998,0,1397,1412 FUELINST,20240209,23,20240209112500,8466,0,946,3789,12415,0,692,100,528,995,0,1055,0,3164,1000,998,0,1397,1412 FUELINST,20240209,23,20240209113000,8446,0,944,3791,12485,0,692,100,540,995,0,1055,0,3166,999,998,0,1397,1412 FUELINST,20240209,24,20240209113500,8363,0,948,3794,12562,0,691,98,555,994,0,1056,0,3171,999,998,0,1397,1412 FUELINST,20240209,24,20240209114000,8285,0,945,3796,12603,0,692,99,562,994,0,1055,0,3172,999,998,0,1397,1412 FUELINST,20240209,24,20240209114500,8249,0,948,3800,12728,0,693,98,558,994,0,1056,0,3174,999,998,0,1397,1412 FUELINST,20240209,24,20240209115000,8129,0,951,3795,12929,0,694,98,530,994,0,1055,0,3179,999,998,0,1397,1412 % gzip -d < _gridCarbonIntensityGB.longstore.csv.gz | tail FUELINST,20240216,21,20240216102000,11645,0,229,3799,6099,231,735,0,233,2000,0,612,0,2260,999,997,0,1399,804 FUELINST,20240216,21,20240216102500,11396,0,233,3796,6204,212,730,6,254,2001,0,612,0,2259,999,997,0,1398,804 FUELINST,20240216,21,20240216103000,11267,0,228,3793,6349,161,729,43,227,2000,0,612,0,2262,999,997,0,1398,804 FUELINST,20240216,22,20240216103500,11096,0,249,3786,6484,60,731,44,305,2000,0,612,0,2266,999,997,0,1398,804 FUELINST,20240216,22,20240216104000,11218,0,254,3777,6456,0,730,44,347,2001,0,612,0,2272,999,997,0,1398,804 FUELINST,20240216,22,20240216104500,11306,0,240,3780,6470,87,731,44,287,2001,0,612,0,2279,999,997,0,1399,804 FUELINST,20240216,22,20240216105000,11368,0,232,3787,6430,102,729,45,249,2001,0,612,0,2256,999,997,0,1398,804 FUELINST,20240216,22,20240216105500,11347,0,239,3788,6406,98,725,44,280,2000,0,524,0,2234,999,997,0,1398,804 FUELINST,20240216,23,20240216110000,11315,0,238,3785,6342,95,708,44,321,2001,0,365,0,2257,999,997,0,1398,804 FTR,2015
2023-02-16: V1.4.1
I fixed the Proguard minified JAR build by fixing the JRE used to run ant and build.xml.
108038 16 Feb 10:44 .work/reutils-1.4.0.jar 66410 16 Feb 15:44 .work/edhMain.reutils-1.4.1.jar
I trimmed still further the data requested from the Elexon server each time. I am now typically requesting 4 intervals of generation data rather than 287, (~70x down) which feels like a win. Technically 2 of those are still redundant, but I am being cautious and allowing for clock skew and other funnies...
I commented out or removed more historic Twitter support code and libraries. (This de-Twitter-ing process started around 2022-11-11.)
2023-02-17: reliable so far
The have been zero drop-outs in gathered FUELINST data in the approximately 24h the new system has been in place (to 17:00Z). There were three drop-outs on the morning of the 16th under the old system.
2024-03-08: glitch
The Elexon system is having a little lie down, in part, at the moment. I can still talk to the new server and re-fetch the last couple of entries, but they are quite old (up to timestamp=20240308145000).
From the log:
Fri 8 Mar 17:36:02 UTC 2024 INFO: generating traffic-light summary [../_gridCarbonIntensityGB.html]... INFO: doTrafficLights(): long store loaded: timestamp: 460ms. INFO: fetched rows: 2 INFO: CHECKPOINT: data fetched etc: timestamp: 3631ms. INFO: CHECKPOINT: 24h summary computed: timestamp: 4157ms. WARNING: will not update intensity log, input data is stale. INFO: 24h summary: FUELINST.CurrentSummary:status=GREEN:recentChange=YELLOW:timestamp=20240308145000:currentMW=29041:currentIntensity=83:currentStorageDrawdownMW=0:histMinIntensity=77:histAveIntensity=113:histMaxIntensity=181:minIntensityRecordTimestamp=20240307235000:maxIntensityRecordTimestamp=20240307164500:histWindowSize=86100000:histSamples=288:lowerThreshold=86:upperThreshold=125:totalGridLosses=0.07:histAveIntensityByHourOfDay=83,84,89,91,92,102,107,119,122,119,103,86,83,79,87,171,179,177,172,162,138,102,92,80:computeTime_ms=307 ...
And so I cannot compute a current intensity, and give up on the last value computed after 2h:
% tail data/heatBattery/log/live/20240308.log 2024-03-08T14:26Z MT 100 GI 85 IT 179 F GS ES 6 2024-03-08T14:31Z MT 100 GI 86 IT 179 F G ES 1 2024-03-08T14:36Z MT 100 GI 87 IT 179 F GX ES 1 2024-03-08T14:41Z MT 100 GI 87 IT 179 F GX ES 1 2024-03-08T14:46Z MT 100 GI 89 IT 179 F G ES 1 2024-03-08T14:51Z MT 100 GI 89 IT 179 F G ES 1 2024-03-08T14:56Z MT 100 GI 89 IT 179 F G ES 1 2024-03-08T15:01Z MT 100 GI 89 IT 179 F G ES 4 2024-03-08T15:06Z MT 100 GI 89 IT 179 F G ES 4 2024-03-08T15:11Z MT 100 GI 89 IT 179 F G ES 1 2024-03-08T15:16Z MT 100 GI 89 IT 179 F GX ES 1 2024-03-08T15:21Z MT 100 GI 89 IT 179 F GX ES 1 2024-03-08T15:26Z MT 100 GI 89 IT 179 F G ES 1 2024-03-08T15:31Z MT 100 GI 89 IT 179 F GX ES 1 2024-03-08T15:36Z MT 100 GI 89 IT 179 F G ES 1 2024-03-08T15:41Z MT 1 GI 89 IT 0 F G ES 1 2024-03-08T15:46Z MT 1 GI 89 IT 0 F G ES 1 2024-03-08T15:51Z MT 1 GI 89 IT 0 F GX ES 1 2024-03-08T15:56Z MT 1 GI 89 IT 0 F G ES 1 2024-03-08T16:01Z MT 0 GI 89 IT 0 F HG ES 1 2024-03-08T16:06Z MT 0 GI 89 IT 0 F HGX ES 1 2024-03-08T16:11Z MT 0 GI 89 IT 0 F HGX ES 1 2024-03-08T16:16Z MT 0 GI 89 IT 0 F HG ES 1 2024-03-08T16:21Z MT 0 GI 89 IT 0 F HG ES 1 2024-03-08T16:26Z MT 0 GI 89 IT 0 F HG ES 1 2024-03-08T16:31Z MT 0 GI 89 IT 0 F HG ES 1 2024-03-08T16:36Z MT 0 GI 89 IT 0 F HGX ES 1 2024-03-08T16:41Z MT 0 GI 89 IT 0 F HG ES 1 2024-03-08T16:46Z MT 0 GI 89 IT 0 F HG ES 1 2024-03-08T16:51Z MT 0 GI - IT 0 F Hr ES 1 2024-03-08T16:56Z MT 0 GI - IT 0 F Hr ES 1 2024-03-08T17:01Z MT 0 GI - IT 0 F Hr ES 1 2024-03-08T17:06Z MT 0 GI - IT 0 F Hr ES 1 2024-03-08T17:11Z MT 0 GI - IT 0 F Hr ES 1 2024-03-08T17:16Z MT 0 GI - IT 0 F Hr ES 1 2024-03-08T17:21Z MT 0 GI - IT 0 F Hr ES 1 2024-03-08T17:26Z MT 0 GI - IT 0 F Hr ES 1
The intensity page is continuing to display, with a warning about the data being stale.
So I think that things are failing 'better' than before!
... And it came back ~12h later ...
... 2024-03-09T05:06Z MT 0 GI - IT 179 F Lr ES 1 2024-03-09T05:11Z MT 0 GI - IT 179 F Lr ES 1 2024-03-09T05:16Z MT 6 GI 91 IT 179 F LG ES 1 2024-03-09T05:21Z MT 100 GI 90 IT 179 F LG ES 1 2024-03-09T05:31Z MT 100 GI 89 IT 179 F LG ES 4 2024-03-09T05:41Z MT 100 GI 89 IT 179 F LG ES 4 ...
2024-02-07: Intensity Page Tweak
I made some tiny tweaks to the GB Grid Intensity page, removing some redundant stuff including Twitterisms, by editing the properties file and waiting for a recalculation/rebuild.
I validated the live HTML with the online Nu Html Checker, as usual, which pointed out some further characters to excise...
(Yes, this was "testing in production", so sue me!)