Earth Notes: On Website Technicals (2025-09)
Updated 2025-09-17.2025-09-15: Amazon RSS Reader Improving?
The Amazon podcast client seems to be moderating, currently ~30/day vs ~62/day at . Good if so!
However, these polls are still apparently unconditional (so getting mainly429 responses) and not respecting Retry-After / Cache-Control.
# Greedy podcast feed pullers: keys are MD5 hashed User-Agent.
# A non-empty txt map lookup of the %{md5:%{HTTP:User-Agent}} means bad!
# Built: 2025-09-15T10:19+00:00Z
# 202 Spotify/1.0
4582d9bdbcef42af27d89da91c6eb804 202
# 138 Google-Podcast
8dea568b39db0451edd6b30f29238eaf 138
# 97 Gofeed/1.0
4a9d728c458902d6ff716779ff72841d 97
# 62 -
d41d8cd98f00b204e9800998ecf8427e 62
# 60 iTMS
97f76eb7e02c5ff923e1198ff1c288cd 60
# 35 axios/1.11.0
5c8ae194a6f98a725c992886f3da6e04 35
# 34 MuckRackFeedParse
62b46fff1cf5f8af7b4b37a2f783b57a 34
# 30 Amazon Music Podc
d69be2563c9f1929edf2906d41809aea 30
# 23 itms
2e7f714a929b3f52f3c094710819a99a 23
# 22 PocketCasts/1.0 (
5caee5a0a53fcbcae25244d8770516ed 22
# 15 Mozilla/5.0 (Maci
9bb586dfc3329d2b522978a294f5c138 15
# 14 Podimo/1.0 (+http
70469c7f54699380beb045260ce4d4ac 14
# 14 FeedBurner/1.0 (h
7678d16f662d98189ceb769c5dd002bc 14
# 13 Mozilla/5.0 (Wind
6b9a00393fb1607b0ada13520f814ab5 13
# 13 axios/1.6.8
b534882134248c9a5957e0c011a37037 13
# 11 TPA/1.0.0
86e71fcf5ba78d28f18270f7f83256bb 11
# 11 Overcast/1.0 Podc
c8bf931c39e0b216181afc441001e58b 11
# 10 deezer/curl-3.0
b99188f8b12adffe0f92ae9c03f03c7c 10
# 9 AntennaPod/3.9.0
3975d4c899c67ddd6e9fa312444f71a8 9
# 7 PodchaserParser/2
599ce17a1a1a11800a9905e39fa49f10 7
# 7 PlayerFM/1.0 Podc
93545f110333c421c67e562d8e5766d3 7
# 7 Mozilla/5.0 (Wind
87d166c91c0adb73ab1bd7fa82ca0662 7
KEHS redirect
I adjusted the KEHS short-cut to redirect to 2025 Bitesize, which is happening in 5 days...
Sequoia 15.7
A new macOS version is pending. Now is a relatively quiet moment, so I am pulling it down. I am currently on 15.6.1.
The first attempt got stuck having downloaded less than 20MB of several GB. Now the Checking for updates...
is continuing to whirl, so maybe my machine is low priority and the servers are maxed out...
Ah, now I am getting Tahoe 26. Sequoia 15.7 was but a dream.
2025-09-14: Rude Bots Ignoring Rules
There is a specific directive in EOU's robots.txt to tell bots not to waste lots of my bandwidth downloading big archive files over and over:
# DHD20250209: avoid huge consolidated archives being redundantly spidered. User-agent: * Disallow: /out/monthly/archive/
Here are some bots that consider themselves unimpeded by good behaviour:
34.174.X.X - [10/Sep/2025:01:34:14 +0000] "GET /out/monthly/archive/public-data-files.tar.xz HTTP/1.1" 200 112640 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36" 34.174.X.X - [11/Sep/2025:19:33:28 +0000] "GET /out/monthly/archive/public-data-files.tar.xz HTTP/1.1" 200 8139106 "https://dropbox.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36" 34.174.X.X - [11/Sep/2025:21:42:11 +0000] "GET /out/monthly/archive/public-data-files.tar.xz HTTP/1.1" 302 603 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:136.0) Gecko/20100101 Firefox/136.0" 34.174.X.X - [11/Sep/2025:21:42:12 +0000] "GET /out/monthly/archive/public-data-files.tar.xz HTTP/1.1" 200 21796672 "http://www.earth.org.uk/out/monthly/archive/public-data-files.tar.xz" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:136.0) Gecko/20100101 Firefox/136.0" 34.174.X.X - [11/Sep/2025:21:44:54 +0000] "GET /out/monthly/archive/public-data-files.tar.xz HTTP/1.1" 200 21315712 "https://msn.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36" 201.159.X.X - [12/Sep/2025:08:06:49 +0000] "GET /out/monthly/archive/public-data-files.tar.xz HTTP/1.1" 200 853391 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Edg/121.0.0.0" 113.163.X.X - [13/Sep/2025:00:39:23 +0000] "GET http://www.earth.org.uk/out/monthly/archive/public-data-files.tar.xz HTTP/1.1" 302 603 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" 14.186.X.X - [13/Sep/2025:00:39:24 +0000] "GET /out/monthly/archive/public-data-files.tar.xz HTTP/1.1" 200 51038 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" 34.174.X.X - [13/Sep/2025:18:58:57 +0000] "GET /out/monthly/archive/public-data-files.tar.xz HTTP/1.1" 200 22600123 "https://moonton.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" 34.174.X.X - [13/Sep/2025:19:17:09 +0000] "GET /out/monthly/archive/public-data-files.tar.xz HTTP/1.1" 200 10390469 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" 34.174.X.X - [13/Sep/2025:23:30:12 +0000] "GET /out/monthly/archive/public-data-files.tar.xz HTTP/1.1" 200 4902656 "https://google.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"
2025-09-10: SVG Load-profile Graphs
I am switching my future load-profile graph generation for 16WW Electricity Profiles to SVG from PNG for space efficiency, especially with maximal static pre-compression with brotli. The RPi server is not currently able to run svgo to optimise these, for about a halving of size, but that step is optional and anyway I usually build these graphs on my laptop.
For one graph that I have regenerated, optimised .svg output is about the same size as .webp, but the pre-compressed versions are many times smaller, much less than 10% of the size of the already-zopflipng-optimised .png.
This SVG generation with pre-compressed version(s) is faster, and scaled versions need not be created for smaller screens.
1622 img/PV/load-profile/202508/bucketed.enphase.202508.all.csv 20545 img/PV/load-profile/202508/bucketed.enphase.202508.all.png 10550 img/PV/load-profile/202508/bucketed.enphase.202508.all.png.webp 10973 img/PV/load-profile/202508/bucketed.enphase.202508.all.svg 1353 img/PV/load-profile/202508/bucketed.enphase.202508.all.svgbr 1480 img/PV/load-profile/202508/bucketed.enphase.202508.all.svggz
.svggz is .svg with zopfli/GZip, .svgbr with brotli. 
I will only automatically generate the pre-compressed .svnbr, since most browsers will understand/accept that Brotli encoding. For those few that do not, but understand GZip, that can be done on the fly. This reduces site clutter a little, for essentially no loss of performance.
2025-09-09: AntennaPod RSS
I seem to have about 4 real AntennaPod/3.9.0 RSS clients (plus myself) over a couple of feeds (plus one just polled by my AntennaPod client).
AntennaPod seems to be doing an average of ~7 polls per day across those, but unconditional, so getting 429 responses to most of them (9 out of the most recent 17).
2025-09-08: Purrmetrix API
Today's mission is to learn to use the Purrmetrix API and fetch all the data for 16WW for the last heating season, ie before we get into this winter...
Later...
I ended up pulling down getting on for 1GB of JSON for to , and as summer 'control'. The data compresses a bit under 10:1 with zopfli and a bit over 10:1 with xz.
2025-09-04: Codeberg
I just cloned (migrated?) a simple repository of mine on GitHub to Codeberg. Just in case GitHub experiences any wobbles, eg from being US based.
2025-09-01: Entry 100
This is the 100th site technicals note!
Upgrading the main Linux server (on the RPi3B) is getting more urgent as it is quite out of date and holding back progress on other things.
To make that upgrade work as long as possible, eg with apt-get working fully, I will postpone the upgrade until the next Raspberry Pi OS in the expectation that it will be derived from the new Debian version (Trixie).
I currently assume that I will go with a 64-bit OS version, and will have a long-enough change-over and shadow-running period to detect significant software or hardware peripheral problems.
Some ansible re-engineering is in my near future.
Accept Opus
I am working up to allowing Accept-like (or actual!) content negotiation for podcast (and other) audio, to transparently reduce bandwidth, in the same way that I substitute an equivalent smaller WEBP for a JPEG or PNG if the client supports it.
I could do this for both .mp3 to .opus and .mp3L to .opusL.
I plan to offer a ~96kbps VBR Opus file as default for the desktop AUDIO player, as an explicit RSS alternateEnclosure, and possibly with automatic content negotiation as above. Opus 96kbps should be nearly transparent even for complex music. MP3 will remain available and will remain as RSS podcast default.
Usually there will be a FLAC file for anyone really wanting to avoid unexpected audio artefacts.
I may wish to calculate the actual bit rate of a source MP3 and aim for half to ensure bandwidth savings.
Deezer 429 response
From being hit with 429s, deezer seems to have slowed to one query per day, from ~12 per day, but since that query seems to be unconditional, and my system has a multi-day memory, and the magic noon slot is being missed, deezer is still getting 429s so far...
185.159.X.X - [31/Aug/2025:15:12:57 +0000] "GET /rss/podcast.rss HTTP/2.0" 429 941 "-" "deezer/curl-3.0" 185.159.X.X - [01/Sep/2025:15:08:10 +0000] "GET /rss/podcast.rss HTTP/2.0" 429 941 "-" "deezer/curl-3.0"
Nope: it then reverted to hourly, unconditional:
185.159.X.X - [02/Sep/2025:00:10:46 +0000] "GET /rss/podcast.rss HTTP/2.0" 429 538 "-" "deezer/curl-3.0" 185.159.X.X - [02/Sep/2025:01:11:06 +0000] "GET /rss/podcast.rss HTTP/2.0" 429 538 "-" "deezer/curl-3.0" 185.159.X.X - [02/Sep/2025:02:11:23 +0000] "GET /rss/podcast.rss HTTP/2.0" 429 538 "-" "deezer/curl-3.0" ...