Earth Notes: On Website Technicals (2024-11)

As the number of energy series data streams increases, some trimming in various places to reduce bloat and to boost speed seems worthwhile.

Make series depend only on current year sources files

All energy series outputs such as consolidated CSV and MIDI files depend on all the input data sets, even though in almost all cases only the most recent file for each source data set changes.

% ls -al .work/tmp/consolidationdeps.mk
24833 Nov  4 11:27 .work/tmp/consolidationdeps.mk
% wc -l !$
wc -l .work/tmp/consolidationdeps.mk
312 .work/tmp/consolidationdeps.mk

A sample from the middle of that dependencies file looks is:

...
data/.flags/consolidated.flag: data/.private/YearlyMeterReadings.csv
data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2016.csv
data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2017.csv
data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2018.csv
data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2019.csv
data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2020.csv
data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2021.csv
data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2022.csv
data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2023.csv
data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2024.csv
data/.flags/consolidated.flag: data/WW-PV-roof/E2008.csv
data/.flags/consolidated.flag: data/WW-PV-roof/E2009.csv
data/.flags/consolidated.flag: data/WW-PV-roof/E2010.csv
data/.flags/consolidated.flag: data/WW-PV-roof/E2011.csv
data/.flags/consolidated.flag: data/WW-PV-roof/E2012.csv
data/.flags/consolidated.flag: data/WW-PV-roof/E2013.csv
...

Limiting to only data files with the current year in their name will save make a little time scanning file details and will still catch most stuff until the next unconditional rebuild of all series when any change is detected, so often daily anyway! No named script dependencies are excluded.

A one-line change in the makefile gets us to:

% wc -l .work/tmp/consolidationdeps.mk
     165 .work/tmp/consolidationdeps.mk

The previous sample reduces to:

...
data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2024.csv
data/.flags/consolidated.flag: data/WW-PV-roof/E2024.csv
...

Energy series page inserts top few only

Traces of all energy series have been inserted into relevant desktop page footers until now, albeit only with graphs at year cadence. A typical page size with things set up this way:

% ls -alS saving-electricity-2024.html*
81330 saving-electricity-2024.html
12035 saving-electricity-2024.htmlgz
 9877 saving-electricity-2024.htmlbr

Reducing such inserts to only be for the most important series (gas, imp, gen, geno) in this case roughly halves the overall uncompressed size:

% ls -alS saving-electricity-2024.html*
41947 saving-electricity-2024.html
 9414 saving-electricity-2024.htmlgz
 7749 saving-electricity-2024.htmlbr

Although the effects on compressed size are less dramatic, this change saves time building each page and decoding it in a browser!

New series

I was provoked into the optimisation when realising that I would probably find two more synthetic series helpful:

divDHW

all diverted solar PV DHW to immersion (resistive/CoP1) heating via the Eddi

immDHW

all DHW to immersion (resistive/CoP1) heating via the Eddi

There was initially some problem with the coverage being reported for at least divDHW, but this was tracked down to buggy conversion scripts from daily Eddi data.

Looking at some of the intermediate data files, it is event that values not yet being presented for h2x were not being handled well, as zeros:

% tail data/consolidated/energy/std/h2d/D/h2d-D.csv data/consolidated/energy/std/h1d/D/h1d-D.csv
==> data/consolidated/energy/std/h2d/D/h2d-D.csv <==
2024-10-23,Eddi,1,
2024-10-24,Eddi,1,
2024-10-25,Eddi,1,
2024-10-26,Eddi,1,
2024-10-27,Eddi,1,
2024-10-28,Eddi,1,
2024-10-29,Eddi,1,
2024-10-30,Eddi,1,
2024-10-31,Eddi,1,
#script,"script/consolidate/energy/std/h2d/D/Eddi/h2d-D-Eddi.sh"

==> data/consolidated/energy/std/h1d/D/h1d-D.csv <==
2024-10-23,Eddi,1,0.033
2024-10-24,Eddi,1,0.005
2024-10-25,Eddi,1,0
2024-10-26,Eddi,1,0
2024-10-27,Eddi,1,0.001
2024-10-28,Eddi,1,0
2024-10-29,Eddi,1,0
2024-10-30,Eddi,1,0
2024-10-31,Eddi,1,0.001

WAVE improvements

Using the WAVE Firefox plugin I have made some improvements/fixes to the energy series tables (replacing empty th items with td), which may improve accessibility eg for those using screen readers. I do not think there are any bad effects from the changes.

There is new data to be captured for the reconfigured heat-pump space heating and DHW system. The next set of these, which I will be starting to collect today, will be the Eddi second heater output and the heat pump H4 boundary flow.

h2

Eddi heater 2 output total

h2d

Eddi heater 2 output diversion only

h2b

Eddi heater 2 output boost only

hph4

Eddi second current clamp for heat pump H4 boundary consumption

Firstly, I have modified the data sources CSV:

--- data/consolidated/config_data_sources.csv	(revision 59843)
+++ data/consolidated/config_data_sources.csv	(working copy)
@@ -1,7 +1,7 @@
 # List of data sources, one per line.  Not all sources report all variables.
 # Expected variables listed after each source.
 # 'synth' must be last if present.
-Eddi,"Myenergi Eddi solar PV diverter",imp,exp,h1d,h1b,h1
+Eddi,"Myenergi Eddi solar PV diverter",imp,exp,h1d,h1b,h1,h2d,h2b,h2,hph4
 Enphase,"Enphase AC-coupled battery system",imp,exp,gen,con,battc,battd
 Loop,"Loop gas meter system",gas
 SSMPPT,"Morningstar SunSaver SS-MPPT-15L off-grid solar controller",geno

Next I will create updated versions of the existing Eddi data capture script script/myenergi/eddiDaySummary.sh to capture the new values h2d, h2b, ectp3 (for hph4), appending the new values to each daily record so as not to disturb any existing uses of those records.

On the Eddi I tried to configure CT2 to be Monitor (it is currently None) but I am unable to, I suspect because no clamp is electrically connected at the moment. I may wish to do a trial connection and configuration before the heat-pump installation.

Then I created the first of the scripts to create the new variables, editing the new hph4-M-Eddi.sh monthly script to extract the new (last) field for hph4:

% svn --parents mkdir script/consolidate/energy/std/hph4/M/Eddi
% svn cp script/consolidate/energy/std/imp/M/Eddi/imp-M-Eddi.sh script/consolidate/energy/std/hph4/M/Eddi/hph4-M-Eddi.sh

The diff looks like:

% svn diff script/consolidate/energy/std/hph4/M/Eddi/hph4-M-Eddi.sh
--- script/consolidate/energy/std/hph4/M/Eddi/hph4-M-Eddi.sh	(revision 59843)
+++ script/consolidate/energy/std/hph4/M/Eddi/hph4-M-Eddi.sh	(working copy)
@@ -1,10 +1,10 @@
 #!/bin/sh
-# Extract to stdout normalised (archived, not live) eddi grid import records.
+# Extract to stdout normalised (archived, not live) eddi hp H4 records.
 # Returns with status 0 in case of success.

 # The output record/line format is:
 #
-#YYYY-MM,device,coverage,imp,comment
+#YYYY-MM,device,coverage,hph4,comment

 # List of candidiate archived data files, in order.
 CANDS="`ls -1 data/eddi/log/20????.daily.csv`"
@@ -15,12 +15,16 @@
 #2022-03-02,24,0,0,6.047,0.032
 #2022-03-03,24,0,0,3.706,0.321

+##Eddi daily stats summary in kWh; partial if h lt 24
+##UTCISOdate,h,h1d,h1b,imp,exp,h2d,h2b,hph4
+#2024-11-10,19,0,0.51,4.949,0.01,0,0,0
+
 if [ "" = "$CANDS" ]; then
     echo "ERROR: no 20????.daily.csv files found." 1>&2; exit 1;
 fi

 # Create header which ensures non-zero length also.
-echo "#YYYY-MM,device,coverage,imp,comment"
+echo "#YYYY-MM,device,coverage,hph4,comment"
 # List input files.
 for f in $CANDS;
     do
@@ -36,13 +40,13 @@
         MM="`echo $f | sed -n -e 's/^.*\/20[0-9][0-9]\([01][0-9]\).daily.csv$/\1/p'`"
         DATE="$YYYY-$MM"
         # Coverage is as min 28 days, 24h/d.
-        cie="`awk < $f -F, '/^20/ {++d;h+=$2;simp+=$5}
+        cie="`awk < $f -F, '/^20/ {++d;h+=$2;shph4+=$10}
             END {
                 # Coverage is (imperfectly) day fraction for each day captured,
                 # further reduced pro-rata if fewer than 28 days seen.
                 df = h / (24*d); # Day fraction.
                 if(d < 28) { df *= d/28; }
-                print df","simp
+                print df","shph4
                 }'`"
         echo "$DATE,$DEVICE,$cie"
     done

All four new variables are added to config_highbad.csv, and the new hph4 is added to the generated CSVs, etc:

--- data/consolidated/config_variables.csv	(revision 59843)
+++ data/consolidated/config_variables.csv	(working copy)
@@ -16,3 +16,4 @@
 h1b,"DHW heater 1 boost from grid"
 h1,"DHW heater 1 all"
 heat,"DHW heater 1 all plus gas"
+hph4,"heat pump H4 consumption"

It turns out that an entirely-zero dataset (ie where the maximum is zero) had a divide-by-zero bug while normalising HTML bar charts, until just now...

Last is to create the extraction/generation scripts for h2 h2b h2d from their h1 counterparts in much the same way, and then update the relevant synthetic derived variables:

heat,Eddi.h1+Eddi.h2+Eddi.hph4+meter.gas
conexDHW,meter.con-Eddi.h1-Eddi.h2

I also encountered and fixed a minor bug in the daily h1d code.

I noticed that some status graphs were being cached too long in my browser. My attempt to boldly extend cacheing in winter (months 11, 12, 1, 2) seemed to be applying too widely. So I commented out as below:

# Set expiry.
# May depend on variables, so should come late in the config!
<IfModule mod_expires.c>
    ExpiresActive On
    # Crank up the cacheing for selected resources...
    # Default ~1 day to optimise Cache-Control max-age to 5 digits HPACK.
    ExpiresDefault "access plus 92222 seconds"
    # DHD20241109: this conditional seems to be overriding lots of stuff below.
    #<If "%{TIME_MON} -lt 3 || %{TIME_MON} -gt 10">
    #    # Winter ~11 days to reduce load a little.
    #    ExpiresDefault "access plus 922222 seconds"
    #</If>
...specific cases by <Location>...