Earth Notes: On Website Technicals (2024-09)
Updated 2024-10-19.2024-09-28: MacBook Time Machine Drive Auto Defrag
When I want to do a backup on my MacBook Air with Time Machine I connect the external spinning-rust USB drive and manually start the backup. Except that it can take many minutes for the Mac to admit that the drive is actually present and allow me to kick off.
The APFS format not used for Time Machine volumes is apparently optimised for SSD rather than unfashionable spinning hard discs, and APFS performance is known to deteriorate over time on HDD.
So I have manually explicitly enabled automatic defragmentation of the volume, though my reading suggests that this will not help much because metadata is the problem and is not defragmented. Also, I think that some automatic defrag was happening anyway, accounting for mysterious continuing activity when the drive should otherwise have been idle.
% diskutil apfs defragment disk5 status APFS Container defragmentation is currently disabled for disk5 % diskutil apfs defragment disk5 enable APFS Container defragmentation has been successfully enabled for disk5 % sudo diskutil apfs defragment disk5s2 status APFS Volume defragmentation is currently disabled for disk5s2 % sudo diskutil apfs defragment disk5s2 enable APFS Volume defragmentation has been successfully enabled for disk5s2 % sudo diskutil apfs defragment disk5s1 status APFS Volume defragmentation is currently disabled for disk5s1 % sudo diskutil apfs defragment disk5s1 enable APFS Volume defragmentation has been successfully enabled for disk5s1
As of I think that backups may be starting more quickly after I connect the external USB (HDD) drive.
I now look out for iostat activity on disk4 falling to zero to indicate defrag is done, since the 'busy' indicator against the drive in Finder is now permanently on.
% iostat 5
disk0 disk4 cpu load average
KB/t tps MB/s KB/t tps MB/s us sy id 1m 5m 15m
13.29 86 1.11 16.63 0 0.01 6 4 90 2.05 2.02 2.01
16.00 1 0.02 8.77 193 1.65 16 10 75 2.05 2.02 2.01
177.13 9 1.59 8.37 207 1.69 14 8 78 2.28 2.07 2.03
21.00 54 1.11 6.51 197 1.25 11 7 81 2.18 2.05 2.02
6.36 15 0.09 6.73 223 1.47 17 10 73 2.25 2.06 2.03
...
disk0 disk4 cpu load average
KB/t tps MB/s KB/t tps MB/s us sy id 1m 5m 15m
13.24 428 5.53 0.00 0 0.00 5 3 91 1.22 1.23 1.45
10.37 10 0.10 0.00 0 0.00 2 2 96 1.36 1.26 1.46
14.62 76 1.08 13.96 18 0.25 4 3 93 1.33 1.26 1.46
28.13 6 0.17 0.00 0 0.00 4 2 94 1.31 1.25 1.46
5.54 35 0.19 0.00 0 0.00 3 2 95 1.28 1.25 1.45
0.00 0 0.00 0.00 0 0.00 2 2 96 1.34 1.26 1.46
The disk4 activity sometimes continues after the drive is ejected, which is somewhat alarming, though I assume safe else it would not happen.
For reference, some stats during an active backup, and entering the Cleaning Up...
phase:
11.79 60 0.69 5.88 115 0.66 5 4 91 1.93 1.93 1.98
7.42 98 0.71 5.16 117 0.59 10 8 82 1.93 1.93 1.98
4.16 76 0.31 8.94 78 0.68 5 5 90 1.78 1.90 1.96
5.35 80 0.42 8.51 99 0.82 7 7 87 1.80 1.90 1.96
7.12 60 0.42 7.18 76 0.53 8 6 86 1.89 1.92 1.97
9.42 908 8.36 9.62 75 0.71 11 8 81 1.90 1.92 1.97
345.37 179 60.24 282.00 150 41.26 5 6 90 1.83 1.90 1.96
216.09 343 72.42 483.07 175 82.69 7 6 87 2.24 1.99 1.99
354.43 302 104.65 940.50 100 92.02 3 4 92 2.06 1.95 1.98
94.76 97 8.97 202.23 95 18.70 7 5 88 2.14 1.97 1.99
13.40 608 7.96 10.63 96 1.00 7 6 87 2.05 1.95 1.98
7.79 37 0.28 4.34 156 0.66 5 5 90 2.20 1.99 1.99
55.91 66 3.59 4.77 96 0.45 7 4 89 2.10 1.97 1.99
Gallery traffic trimming
After quite a long time of being told that old URLs are dead, many spiders are still trying to download from them.
To try to trim nugatory traffic a little, I have added some new rules to robots.txt to deter access to large chunks of namespace currently empty and unsupported. (Some of this namespace may never be supported again.)
User-agent: * Disallow: /_cat/ Disallow: /_site/ Disallow: /_tn/ Disallow: /_virtual/
2024-09-26: RequestHeader unset If-None-Match
I have adjusted the configuration of a few more of the static sites (including m.earth.org.uk) to actively filter out any If-None-Match request headers to go along with no longer generating any ETag. This will help ensure that conditional GETs can work properly, even if clients retain and re-present very old ETag values.
# Until Apache 2.5, disable ETag entirely for this site. FileETag none Header unset ETag +RequestHeader unset If-None-Match
2024-09-22: BibTeX crossref
I have added very minimal support for BibTex crossref support, simply showing the citation token, not attempting to link, fill in fields, etc. I may not be using it as intended this way, but this can be improved.
2024-09-11: Adjusting Ps and Qs
I have adjusted P/p and Q/q to for a threshold CoP of 2, as that is the case now and shows that it can happen!
2024-09-05: Minding Ps and Qs
I have adjusted P/p to use CoP% (ie if current intensity is a little under predicted then the value is a little over 100), and over 150 now flags significant saving from topping-up storage.
This numeric value allows a smooth fading in of maximum top-up percentage from (say) a CoP% of 150 to 250, eg with a mean of 200%.
Right now I am seeing both P and Q flags, but no 'supergreen' G (though grid is ordinary green).
2024-09-05T14:26Z MT 41 GI 78 IT 169 F PQbvx ES 1
A matching change has been made for calculating Q/q and a CoP >150% significance threshold.
The grid went supergreen at 4pm ie peak time (H), so a top-up would have been locked out here:
2024-09-05T14:31Z MT 41 GI 78 IT 168 F PQbvx ES 1 2024-09-05T14:36Z MT 41 GI 78 IT 168 F PQbvx ES 1 2024-09-05T14:41Z MT 1 GI 78 IT 0 F PQbvx ES 1 2024-09-05T14:46Z MT 1 GI 78 IT 0 F PQbvx ES 1 2024-09-05T14:51Z MT 1 GI 80 IT 0 F PQbvx ES 1 2024-09-05T14:56Z MT 1 GI 82 IT 0 F PQbvx ES 1 2024-09-05T15:01Z MT 0 GI 86 IT 0 F HGPQbvx ES 1 2024-09-05T15:06Z MT 0 GI 85 IT 0 F HGPQbvx ES 1
(Intensity has sharply fallen over 24h and is due to rise again over 48h...)
Moments that are supergreen (G) and with at least one of P and Q can be picked out with something like:
% egrep ' F [^ ]*[PQ]' data/heatBattery/log/live/20240905.log | egrep ' F [^ ]*G'
2024-09-04: Q/q Flag
I added a Q (and converse q) flag (indicating that a big grid intensity fall has happened over the last 7d, so storing heat now would have an effective CoP >1) to script/heat-battery-target.sh output to estimate how often it might drive a top-up, with an initial threshold value of a CoP >2.0.
Yes, all the good and obvious flag letters have been used...
2024-09-03: P/p Flag and GSC 304s
I added a P (and converse p) flag (indicating that a big grid intensity rise is predicted over the next 48h, so storing heat now would have an effective CoP >1) to script/heat-battery-target.sh output to estimate how often it might drive a top-up eg 2024-09-03T14:31Z MT 0 GI 225 IT 139 F Rpbvx ES 3, with an initial threshold value of a CoP >2.0.
304s
Google Search Console now reports that 29% of recent crawls of EOU by Google get 304s now. A number that has been rising steadily, mainly because I stopped using ETags I suspect.
2024-09-02: Zero Gas Ahoy
Anticipting the heat pump getting installed, and thus gas consumption dropping to zero, I adjusted the recent utilities consumption working from the form At 16WW over 32 days from 2024-08-01 to 2024-09-02: gas consumption was 0.2kWh/d, electricity imports were 0.4kWh/d (net -9.5kWh/d).
to At 16WW over 32 days from 2024-08-01 to 2024-09-02: electricity imports 0.4kWh/d (net -9.5kWh/d), gas consumption 0.2kWh/d
, with the gas clause disappearing entirely when zero.