Earth Notes: On Website Technicals (2023-10)
Updated 2024-02-29.2023-10-30: De-Twitter-ing
It is high time to start stripping more Twitter-only metadata from pages, as I am not posting there any more (and free access seems likely to be terminated).
In particular from each header I am removing <meta name=twitter:site content=@EarthOrgUK> for those valuable bytes to be saved or used for other things.
I am taking the opportunity to trim the "previous article" teaser in the footer of desktop pages somewhat.
2023-10-26: Winterising
I have made some small tweaks to help the off-grid power system work better in winter. They should be enduring, ie they do not have to be backed out for summer. It is simply easier to find and adjust thresholds when on a cusp as now.
One tweak was to trim the 'reserve' off-grid battery SoC above the floor 50%, since it looks like I am cutting 'dumping' out too early and thus not only putting the router back on grid too much, but also not leaving enough space in storage to absorb energy the next day.
I have also set the make pages-incr incremental site rebuild to avoid running when it is dark outside as detected by the roof PV. I might adjust this further to dark or low, or to when the battery is (not) net discharging.
The pages-incr already avoids running when the battery is low.
, on a train(!): I have added CHARGING and DISCHARGING flags in powermgr to avoid using two subsystems' flags for this 1-cent job! I have switched cron job body to be:
if [ -e /run/CHARGING.flag -a -e /run/EXTERNAL_BATTERY_NOTLOW.flag ]; then if [ "`uptime | awk '{print $NF<2}'`" = 1 ]; then cd /rw/docs-public/www.hd.org/Damon/Env && nice make -k pages-incr > /tmp/Env.pages-incr.log 2>&1; fi; fi
2023-10-16: Bibliography Munging
I am trying to prepare a paper for an MDPI journal. Using my bibliography with their .bst bibliography style reveals some disconnects, and not all the information that I think should appear in the output PDF was, such as the URL for a Web page!
I adapted my processing to accept the www BibTeX type, and converted almost all misc type entries to it. Such entries are labelled as schema.org/Webpage. (One entry became a dataset.)
I also changed my patent to the patent type.
The MDPI styles do not seem to include dataset, so I see a little more pain ahead for those!
There has been some other clean-up and improvement of the conversion to HTML.
eprint
I have been misusing the eprint field to directly link to PDFs. It is useful, but wrong. To date eprint has been in 26 entry records.
The fastest fix is to convert eprint = {XXX}, to howpublished = {\url{XXX}},, and change the conversion script to work with that. Note that howpublished is not supported for all entry types. The scary command is:
% sed -I -e 's/^ *eprint *= *{\(.*\)},$/howpublished={\\url{\1}},/' db.bibliography/single/*.bib
@dataset
MDPI says to use @misc in place of @dataset, so I am using the power of sed and make to adjust on the fly from my general .bib!
2023-10-13: 18150 LaTeX Symbols
% texdoc symbols-a4
2023-10-12: Overleaf, MacTeX
Yesterday a uni colleague suggested that I try Overleaf to prepare my journal paper in LaTeX, especially as the journal seems to support that platform specifically. Most of his colleagues use Overleaf it seems.
It is about 30+ years since I last touched LaTeX, to set my MSc thesis for one thing.
I got as far this morning as making work a single citation from my collated set with the magic of \bibliography{general} in the References section and \cite{xu2023field} in the body of the text.
I received a warning that on the (totally free) account I have set up I am hitting compile time limits.
Thus, at another friend's suggestion, I am downloading and installing on my MBA MacTeX. I'll try to replicate the little bit of work done so far, though I may not get a 'live' editor as in Overleaf.
I will be happy if I can work offline (network access at uni was flaky yesterday with eduroam not working for me and at least one other), using Overleaf maybe for collaboration with my supervisors.
To get at the new toys I extend my path:
% set path= ( $path /usr/local/texlive/2023/bin/universal-darwin/ )
TEXMAKER and CLI
I am trying TEXMAKER as an interactive LaTeX editor. But I can always revert to CLI mode as in the olden days!
And on the command line, to get citations working too:
pdflatex sourcefile && bibtex sourcefile && pdflatex sourcefile && pdflatex sourcefile
2023-10-04: BibTeX Debugging
Very very soon I have to try to actually use my bibliography for a real journal paper.
My BibTeX entries are parsable by my awk script, but that does not mean that they are generally correct.
So I am trying biber downloaded with brew install biber to lint my files, eg:
% biber -V --tool -O - -logfile biber.log db.bibliography/general.bib
This is already turning up a number of minor horrors, with some help from tweaks to my converter awk:
% make db.bibliography/general.bib bibliography.html && biber -V --tool -O - -logfile biber.log db.bibliography/general.bib
Rebuilding .work/tmp/singlebib.mk
Rebuilding db.bibliography/general.bib
Rebuilding .work/inc/.bibliography.html.inc
INFO: Fake lockfile -r 0 -l 907 .work/inc/.bibliography.html.inc.lock
INFO: *** no URL for *** CEN2017EN12831
INFO: *** ambiguous author split for *** stulz20112000W
ERROR: missing trailing , for field in thomas2023policies on line 2461: annote={Quote: "California has required rooftop PV on all new buildings since 2020. The 2022 building code made the coupling of PV with battery storage mandatory for new commercial, public use and multifamily buildings. The PV must be sized to provide at least 60\% of the building's expected load, and the storage must be sufficient to reduce export to the grid to 10\% of total on-site solar generation. New residential homes must be storage-ready."}
make: *** [.work/inc/.bibliography.html.inc] Error 1
I did a mass fix with:
% sed -I -e 's/\(.}\)$/\1,/' db.bibliography/single/*.bib
I have also downloaded BibDesk, which rejected my original .bib files without explanation. Thus the search for a linting tool and biber!
After the linting and fixing, BibDesk will now read my files, hurrah!
2023-10-03: Stale Data Retry
V1.2.3 of reutils creates a xxx.stale.flag flag file while fresh FUELINST data is not available.
This can be used to re-poll sooner, as already done for green status.
After some drop-outs the next day while I was in front of the keyboard, I set this up to actually be used.
The updated rule in ansible is:
# Capture when green or data is stale, when not mid-winter and battery not VLOW.
- name: Capture extra grid intensity data when plenty of energy about.
cron:
name: "16WW: capture optional 5-min grid intensity points."
minute: "06,16,26,36,46,56"
user: "{{ workuser }}"
month: "2-11"
job: "( cd /rw/docs-public/www.hd.org/Damon/Env/.work && test -e /run/EXTERNAL_BATTERY_VLOW.flag -o \\( -e ../_gridCarbonIntensityGB.flag -a \\! -e ../_gridCarbonIntensityGB.stale.flag \\) || sh ./runIntensity.sh > /tmp/_gridCarbonIntensityGB.log 2>&1 )"
2023-10-02: Apache Icons Cacheing
I noticed that the Apache HTTP icons (under /icons/), used for directory listings, were getting the default EOU site expiry time of 922222 seconds (~11 days).
I added a rule to treat that subdirectory just like /img/, and cache for a year and with the immutable attribute.
The icons are tiny, but not requesting them at all should save a little effort all round!
2023-10-01: Ill Filesystem
After a power cut on the 29th caused by gas people (SGN) trying to fix a leak that needed two houses to be evacuated, the filesystem on pekoe was behaving very oddly, timing out a simple svn update or ls in a certain directory.
I set pekoe to fsck on reboot with sudo touch /forcefsck and it did not come back.
So I power cycled it again, was able to log in, and found stuff like this (amongst all the break-in attempts, etc) in syslog:
Sep 30 23:10:30 pekoe kernel: [89751.958919] mmc0: timeout waiting for hardware interrupt. Sep 30 23:10:30 pekoe kernel: [89751.958970] [253efcfc] TCM> c19a599c 0 Sep 30 23:10:30 pekoe kernel: [89751.958999] [253efcfc] FDA> c19a58d0 0 Sep 30 23:10:30 pekoe kernel: [89751.959019] [253efcfc] DMA> 0 0 Sep 30 23:10:30 pekoe kernel: [89751.959035] [253efd04] TSK< c19a58d0 0 Sep 30 23:10:30 pekoe kernel: [89751.959051] [253efd1c] TSK> c19a58d0 0 Sep 30 23:10:30 pekoe kernel: [89751.959067] [253f045b] PRD< c19a5b7c 0 Sep 30 23:10:30 pekoe kernel: [89751.959083] [253f045b] PRD1 c11a3420 0 Sep 30 23:10:30 pekoe kernel: [89751.959099] [253f0466] PRD2 1 0 Sep 30 23:10:30 pekoe kernel: [89751.959114] [253f0474] PRD3 c44318a4 0 Sep 30 23:10:30 pekoe kernel: [89751.959130] [253f0475] PDM> c19a5b7c 0 Sep 30 23:10:30 pekoe kernel: [89751.959148] [253f0477] REQ< c19a5ab0 10801 Sep 30 23:10:30 pekoe kernel: [89751.959165] [253f0478] CMD< 17 8 Sep 30 23:10:30 pekoe kernel: [89751.959181] [253f0482] FCM< c19a5ab0 c19a5aec Sep 30 23:10:30 pekoe kernel: [89751.959197] [253f0484] RSP 900 0 Sep 30 23:10:30 pekoe kernel: [89751.959213] [253f0485] CMD< 12 17ea3a0 Sep 30 23:10:30 pekoe kernel: [89751.959229] [253f0487] CMDD 8 200 Sep 30 23:10:30 pekoe kernel: [89751.959244] [253f0488] SDMA c19a5b7c c11a3420 Sep 30 23:10:30 pekoe kernel: [89751.959262] [253f048b] FCM< c19a5ab0 c19a5b1c Sep 30 23:10:30 pekoe kernel: [89751.959279] [253f048c] RSP 900 0 ... Sep 30 23:11:05 pekoe kernel: [89786.523232] [27ed60fa] TIM< 0 0 Sep 30 23:11:05 pekoe kernel: [89786.523249] mmc0: sbc op 23 arg 0x8 flags 0x15 - resp 00000900 00000000 00000000 00000000, err 0 Sep 30 23:11:05 pekoe kernel: [89786.523279] mmc0: cmd op 18 arg 0x17d2df0 flags 0xb5 - resp 00000900 00000000 00000000 00000000, err 0 Sep 30 23:11:05 pekoe kernel: [89786.523306] mmc0: data blocks 8 blksz 200 - err 0 Sep 30 23:11:05 pekoe kernel: [89786.523322] mmc0: stop op 12 arg 0x0 flags 0x95 - resp 00000000 00000000 00000000 00000000, err 0 Sep 30 23:11:05 pekoe kernel: [89786.523349] mmc0: =========== REGISTER DUMP =========== Sep 30 23:11:05 pekoe kernel: [89786.523361] mmc0: SDCMD 0x00004052 Sep 30 23:11:05 pekoe kernel: [89786.523373] mmc0: SDARG 0x017d2df0 Sep 30 23:11:05 pekoe kernel: [89786.523386] mmc0: SDTOUT 0x017d7840 Sep 30 23:11:05 pekoe kernel: [89786.523399] mmc0: SDCDIV 0x00000003 Sep 30 23:11:05 pekoe kernel: [89786.523411] mmc0: SDRSP0 0x00000900 Sep 30 23:11:05 pekoe kernel: [89786.523424] mmc0: SDRSP1 0x00001269 Sep 30 23:11:05 pekoe kernel: [89786.523436] mmc0: SDRSP2 0x7fffffff Sep 30 23:11:05 pekoe kernel: [89786.523449] mmc0: SDRSP3 0x00024007 Sep 30 23:11:05 pekoe kernel: [89786.523462] mmc0: SDHSTS 0x00000080 Sep 30 23:11:05 pekoe kernel: [89786.523476] mmc0: SDVDD 0x00000001 Sep 30 23:11:05 pekoe kernel: [89786.523488] mmc0: SDEDM 0x00010801 Sep 30 23:11:05 pekoe kernel: [89786.523501] mmc0: SDHCFG 0x0000040e Sep 30 23:11:05 pekoe kernel: [89786.523513] mmc0: SDHBCT 0x00000200 Sep 30 23:11:05 pekoe kernel: [89786.523526] mmc0: SDHBLC 0x00000008 Sep 30 23:11:05 pekoe kernel: [89786.523538] mmc0: =========================================== Sep 30 23:11:16 pekoe kernel: [89797.399144] mmc0: timeout waiting for hardware interrupt. ...
So I carefully moved the offending directory tree away. I am very gingerly leaving it alone but will get a (bigger) new SD card soon. I will probably just build a whole new image and rsync the Thermino logs over to the new one at the last moment.
AdSense GDPR cookies: left hand vs right hand
AdSense is managing to do all three of these at once:
- Complain that I need to set up GDPR cookie messaging.
- Show me the GDPR cookie messaging that has been set up for weeks.
- Complain that I am doing GDPR cookie messaging even though I have turned off all ad personalisation.
The last point, if properly implemented by Google, is why I do not think that I need such a pop-up at all. But now I have some extra controls to explicitly exclude third-party ad brokers and disallow even 'legitimate interest' data processing. So maybe my visitors will actually have less of their data abstracted.
Yes, I have provided "Feedback" to AdSense. No, I do not think that doing so will help.
I am now essentially only running any JavaScript on the relatively small set of pages where Google AdSense code is injected.