Earth Notes: On Website Technicals (2025-02)

Updated 2025-03-03.
Tech updates: FOSDEM, archive bandwidth, grid 2026 intensities, GBGI, L hours, BIND died, data purge, storage CoP up, Pagefind, kgCO2-to-date, Thermino.
Considering forming a citizens' assembly to set terms for a Royal Commission to guide the forming of a ministerial task force to consider the scoping and creation of a focus group to sketch terms of reference for a study group to outline the agenda for a pre-meeting to form a steering group to get ready to think about upgrading the RPi server, easy does it... Most heat pump tweaking (other than its noisy water pipe freeze prevention) may be done, but the Thermino has to be reconnected and have its algorithms shaped...

2025-02-28: Heat Battery Back Online

The Sunamp Thermino heat battery has been reconnected to 16WW's DHW (Domestic Hot Water) system.

The pekoe RPi that listens to the Thermino is back on top of it, with a new network cable, which just worked with no crossovers etc this time. The Thermino logs are being recorded again, as of this afternoon.

I very slightly upgraded the heat battery driver main RPi script to now show a 'w' flag at all (not just L) times when a pasteurisation is pending: 

2025-02-28T17:31Z MT 0 GI 242 IT 0 F HRbvxN ES 1
2025-02-28T17:41Z MT 0 GI 243 IT 0 F HRbvxN ES 1
2025-02-28T17:51Z MT 0 GI 243 IT 0 F HRbvxN ES 1
...
2025-02-28T18:01Z MT 0 GI 246 IT 0 F HRbvxwN ES 1

Skype

Given that Skype is apparently to be shut down , and I have not used it in years I think, I have deleted the Skype client on my laptop.

2025-02-25: CO2 Year-to-date

I have long intended to have some sort of running live-ish measure of utility CO2 emissions, and they are now available (currently updating monthly): 16WW Energy Series Dataset: 16WW Utility Energy CO2 Year-to-date.

Raw and projected kgCO2 for year so far for utility grid electricity based on computed live grid intensity and 16WW imports and exports.

: year to date (31 days) 78kgCO2, full-year linearly projected (more-of-the-same) 918kgCO2.

I may also show the running kgCO2 on the home page or similar in due course...

Recent hotness

I have extended the stats on the heat-battery target dashboard to include simple text bar charts by day of system and DHW tanks full/hot:

  • Recent (8/8) days when tank or battery did not fill [nnnnnnnn]: 100%
  • Recent DHW cylinder pasteurisations w/ tank Hot [H---------].

2025-02-24: pagefind

I am considering using pagefind to replace the Google search box on the EOU site search page.

This would be like a (better!) version of my old Java applet. No dependency on any remote server at run-time.

There are some significant privacy and related issues to consider, but generating an index for the 'lite' site seems as simple as: 

% npx pagefind --site m

Because the main/desktop paged optimised away the body tag that pagefind normally indexes, and to avoid descending into private directories and capturing non-static pages, a slightly more complex formulation seems to be required: 

% pagefind --root-selector main --site . --glob '[A-Za-z0-9]*.html'

Running Pagefind v1.3.0 (Extended)
Running from: "..."
Source:       ""
Output:       "pagefind"

[Walking source directory]
Found 451 files matching [A-Za-z0-9]*.html

[Parsing files]
Did not find a data-pagefind-body element on the site.
. Indexing all <body> elements on the site.

[Reading languages]
Discovered 1 language: en-gb

[Building search indexes]
Total:
  Indexed 1 language
  Indexed 449 pages
  Indexed 56993 words
  Indexed 0 filters
  Indexed 0 sorts

Finished in 3.058 seconds

The default search UI (with the sample HTML snippet) seems reasonable. It requires JavaScript to run so would need a noscript warning. The JavaScript dependency is limited to the single search/sitemap page on each site variant.

This may need newer infrastructure than I wish to support on my server until it is upgraded, and there are some other difficulties evident.

2025-02-23: Bad Data Purge

When the Eddi lost some of its marbles on due to a latent firmware bug, all the data it reported from then until after I fully uploaded the firmware fix is wrong or at least tainted.

While preserving the raw data as captured at data/eddi/log/202501-raw.daily.csv, to remove noise from much of the stats I manually excised the bad days from the daily record at data/eddi/log/202501.daily.csv, and created a little helper to recompute the final summary line: 
% awk < data/eddi/log/202501.daily.csv -F, '/^[^#]/{for(i=2;i<=9;++i){sum[i]+=$i}} END{for(i=2;i<=9;++i){printf("%.3f,", sum[i])}}'

Storage CoP Threshold Raised

As work on reconnecting the Thermino has started, and I now do not think that much direct grid-top up should happen, ie it should be truly exceptional, I have raised the minimum effective storage CoP from 2.00 to 3.00 with SOFTPMINSTORAGECOPPC=300 in data/consolidated/softparams.txt.

2025-02-18: BIND9 DNS Daemon Died

DNS on the server died about a day ago (), which I discovered ~ because my intensity page was unable to fetch data (DNS lookup error).

I started by checking that the data source was not down (it happens), then blaming resolvconf, but a restart of BIND9 seems to have fixed things for now.

An innocent OOM (Out of Memory) kill is possible, but malicious attacks are possible. I was not manually killing any processes at that time.

2025-02-16: Adjusted Low Grid Times

I was marking 00h to 06h inclusive as 'L' potential low-intensity hours, but the first of those often sees a small uptick in demand on the GB grid because of Economy 7 etc kicking in after midnight.

Also the 'L' hours are 'local', so some of the year 00h is before midnight UTC. It would be nice to avoid that minor headache when looking at stats.

So I have removed 00h as an 'L' hour. The remaining 6 hours are long enough, as a typical pasteurisation time is 2h.

2025-02-14: Avoiding Bitly Ads

It seems that now, not unreasonably, we face Bitly Showing Ads on Page Redirects.

The tweets/toots for GB grid carbon intensity used to (for many years) redirect via a short bit.ly link to fit in an old tweet.

To trim the small extra implied carbon and privacy cost I have now set up a redirection from /GBGI to the GB Grid Intensity page.

Though it should work for http and https, and allowing for some other aliasing, I am now using https://earth.org.uk/GBGI in the toots on mastodon.energy.

Because toots can be longer than the old tweet limit, the 6-character longer (was http://bit.ly/3usoe) new URL is OK.

2025-02-21

This takes a couple of redirects to land, but that is OK. This is me clicking through from a toot: 

www.earth.org.uk:443 79.135.X.X - - [21/Feb/2025:18:11:22 +0000] "GET /GBGI HTTP/2.0" 301 382 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:135.0) Gecko/20100101 Firefox/135.0"
www.earth.org.uk:443 79.135.X.X - - [21/Feb/2025:18:11:22 +0000] "GET /GBGI HTTP/2.0" 301 331 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:135.0) Gecko/20100101 Firefox/135.0"
www.earth.org.uk:443 79.135.X.X - - [21/Feb/2025:18:11:22 +0000] "GET /_gridCarbonIntensityGB.html HTTP/2.0" 200 5862 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:135.0) Gecko/20100101 Firefox/135.0"

2025-02-10: Updating Interconnector Carbon Intensities

Today Electricity Maps gave me access to the whole-of- mean grid carbon intensities for the six countries at the other end of GB's interconnectors from its data portal: Granular historical electricity data.

Thank you!

I shall use these intensities from the start of for my live GB grid carbon intensity calculations. The roll-over will be automatic.

I am definitely not leaving things to the last minute!

The new () 'direct' intensity values: 

# Interconnectors to BE (NEM), DK (VKL), FR (IFA, IFA2, ELEC), IE (IRL, EW, GRNL), NL (NED), NO (NSL).
# BE 2024 mean 0.105: Electricity Maps / ENTSO-E.
# DK 2024 mean 0.075: Electricity Maps / ENTSO-E.
# FR 2024 mean 0.018: Electricity Maps / ENTSO-E.
# GB 2024 mean 0.107: Electricity Maps / nationalgrideso.com, elexon.co.uk
# IE 2024 mean 0.322: Electricity Maps / ENTSO-E.
# NL 2024 mean 0.199: Electricity Maps / ENTSO-E.
# NO 2024 mean 0.009: Electricity Maps / ENTSO-E.

2025-02-09: Reducing Redundant Archive Bandwidth

An archive of a copy 'lite' site version is captured each month and made available under out/monthly, in two formats (ZIP and LZMA).

Various somewhat brute-force bots come and fetch those blindly, even though their content is entirely redundant cf the live site.

I have displaced the archives one level down to out/monthly/archive, and created a robots.txt to forbid spidering that tree.

I have updated the Zenodo metadata [hart-davis2023EOUoffline] to reflect the new path on EOU.

2025-02-02: Unexpected Talk in FOSDEM Energy Area

I attended FOSDEM this year as usual (my first visit seems to have been 2011) and was sitting at the front of the Energy dev room when about the fifth scheduled speaker did not show up.

I volunteered my LOCO2024 lightning talk on the RSS inefficiency issue to fill some of the gap, was taken up on my offer, and was edited into the programme retrospectively [hart-davis2025feeds]!

That makes me a two-time FOSDEM speaker now, given the OpenTRV effort [hart-davis2014opentrv]...

The same has been done with the much larger data archive [hart-davis2023EOUdata].

References

(Count: 4)