Earth Notes: On 16WW Data Collections and Graphs

Updated 2024-05-02.
Open for research #dataset
A selection of data collected and used, especially at home (16WW). I intend to organise and collect more of it here, and graph it (and show the tools to graph it).

Some of the data is marked up with HTML5 microdata schema.org/Dataset. The intent is to "start with the basics" as suggested by Google, including 'DataDownload' and 'license' information where possible. (This mark-up is checked with Google's Structured Data Testing Tool.)

See available datasets for others.

This page also references circuit diagrams/schematics.

Dataset

Much data has been collected for our house at 16 Willingham Way (16WW) referred to in the various documents on this site. A lot of that data is structured and machine-readable.

The bulk of the data can be found here and is browsable, in simple text formats such as CSV. Common compression techniques have been applied for larger/non-live data sets.

name
16WW bulk energy and related data
description
16WW domestic energy and related data, including time series, Kingston-upon-Thames, London, UK
version
1
keywords
domestic, energy
date created
2007
date published
date modified
2024-05
temporal coverage
2007/..
spatial coverage
UK centre 51.406696N,-0.288789E elevation 16m
distribution
directory tree
distribution
xz-compressed tar VERY BIG archive monthly snapshot of the public data files
canonical URL
this descriptive text with markup
DOI
10.5281/zenodo.10206489 [hart-davis2023EOUdata]
licence
this dataset is licensed under CC0, ie it is effectively public domain; if you make use of this data, attribution is welcome but not obligatory
is accessible for free
true

Other datasets

Other available datasets (eg shards of this overall 16WW dataset).

Unstructured and Other Data

SAP and air-leakage

SAP cert extract

To set a baseline and understand better what needed fixing, I had SAP, air-leakage and other tests done in early 2009.

Tests/results/certificates by Tophouse 2009-03-30 for us were:

2024-02-10: new EPC

20240210 EPC A chart

In order to allow a BUS heat-pump grant, an in-date EPC is required, and it must not be suggesting any easy insulation works. The first one expired in 2019, so I booked in a new one, with the assessor here earlier in the week. Not everything done to improve this house can be captured by rdSAP.

The renewed 16WW EPC certificate came in from Ecoalex and it is an A 100 (it was previously B 84), which is ~0.3% of existing England and Wales homes.

Grid-tie PV generation

See live generation output when possible.

2019-03-12: now in OpenStreetMap!

Daily PV Output Since 2008

This data is collected a year at at time, though there is a chart for the whole data set, and collated daily kWh generation records with smoothing. (See the gnuplot commands.)

2008-02-25: the first phase/round of PV, 1.29kWp (west-facing, 6 x Sanyo HIT 215Wp hybrid crystalline/thinfilm panels with SMA Sunny Boy 1100 inverter), was installed.

See the 2008 daily PV generation logs for 2008 as .csv and .png chart.

2009-02-10: the second phase was installed, taking the system to 3.87kWp; same panels and inverters for each of the two new east-facing strings as for the first phase.

See the 2009 daily PV generation logs for 2009 as .csv and .png chart.

2010-04-26: the third phase (west-facing, same panels, SMA Sunny Boy 1200 inverter) of 1.29kWp was installed, taking the whole system to 5.16kWp. New tariff and generation meter for this last part.

See the 2010 daily PV generation logs for 2010 as .csv and .png chart.

See the 2011 daily PV generation logs for 2011 as .csv and .png chart.

See the 2012 daily PV generation logs for 2012 as .csv and .png chart.

See the 2013 daily PV generation logs for 2013 as .csv and .png chart.

See the 2014 daily PV generation logs for 2014 as .csv and .png chart.

See the 2015 daily PV generation logs for 2015 as .csv and .png chart.

See the 2016 daily PV generation logs for 2016 as .csv and .png chart.

See the 2017 daily PV generation logs for 2017 as .csv and .png chart.

See the 2018 daily PV generation logs for 2018 as .csv and .png chart.

See the 2019 daily PV generation logs for 2019 as .csv and .png chart.

See the 2020 daily PV generation logs for 2020 as .csv and .png chart.

See the 2021 daily PV generation logs for 2021 as .csv and .png chart.

See the 2022 daily PV generation logs for 2022 as .csv and .png chart.

See the 2023 daily PV generation logs for 2023 as .csv and .png chart.

See the 2024 daily PV generation logs for 2024 as .csv and .png chart.

(See the gnuplot commands for annual plots, and the current awk 'V2' smoothing code ('V1').)

See other system performance records, including per-string outputs.

Intraday data

Rob Ferber of ElectronVault wrote to me looking for intra-day generation data (with 10 minute sampling intervals). I had at that time exposed very little because of data clean-up issues. (Such as bizarre time offsets in the Sunny Beam apparently attempting to correct to German civil time even though I keep it on UTC, serial numbers, corrupt data, slight drift of Sunny Beam from true UTC, etc.)

EV graph

I capture these intraday samples manually after dusk about twice per month. Sometimes also I capture data on particularly 'interesting' days, of late using the SunnyBeamTool by Michaël Peeters on my SheevaPlug (and now, as of 2014, Raspberry Pi B+) ARM Linux system. I did not do this continuously because I have not been able to stop the Sunny Beam apparently drawing considerable power via the USB port, and because the SheevaPlug did not have a 'spare' USB port: I manually disconnected the USB I/O board temporarily to connect the Sunny Beam. (As of 2014-07 experimentally I have the Sunny Beam plugged in permanently, and take 5-minute samples during daylight hours.)

Rob produced lovely tweaked graphs from my raw data up to September 2012. Thanks, Rob: great job!

Prior to Rob's analysis I had made public but not announced these intraday samples from SMA's Windows application, with no explicit time correction: 2008-03-24 2008-04-01 2008-04-09 2008-04-17 2008-05-08 2008-09-13 2008-09-14 2008-09-19 2008-10-11 2008-12-13 2009-03-15 2009-04-08 2009-05-11 2009-05-24 2009-06-29

The entire raw data set is now available, including new samples as I take them, though with many caveats (beware timestamp wobbles in particular).

See the interesting 2015-03-20 (cloudy) morning dip in PV output (raw Sunny Beam data with time offset, per-minute data) during the partial solar eclipse (~85% maximum at ~09:30).

PVGIS Prediction

Here is an extract of the numbers from the PVGIS estimate not knowing anything about our local trees, etc!

Solar radiation database used: PVGIS-classic
Nominal power of the PV system: 5.2 kW (crystalline silicon)
Estimated losses due to temperature: 7.3% (using local ambient temperature)
Estimated loss due to angular reflectance effects: 4.1%
Other losses (cables, inverter etc.): 14.0%
Combined PV system losses: 23.5%

Fixed system: inclination=23°, orientation=-90°
MonthEdEmHdHm
Jan 2.7886.00.7322.6
Feb 5.361501.3437.4
Mar 8.832742.1666.8
Apr 14.404333.57107
May 17.805524.51140
Jun 18.205454.65140
Jul 18.205644.69145
Aug 15.604834.02125
Sep 10.703222.7281.7
Oct 6.572041.6751.7
Nov 3.431030.8926.8
Dec 1.9460.10.5216.2
Yearly average10.33152.6380.0
Total for year3780960

Ed: Average daily electricity production from the given system (kWh)
Em: Average monthly electricity production from the given system (kWh)
Hd: Average daily sum of global irradiation per square meter received by the modules of the given system (kWh/m2)
Hm: Average sum of global irradiation per square meter received by the modules of the given system (kWh/m2)

Note also that adding just 10% (~500Wp) vertically mounted south-facing PV would add about 1/3rd to mid-winter generation from the main array (ie ~1.6kWh/d actual from the roof and ~0.5kWh/d from the new array predicted.)

Nominal power of the PV system: 0.5 kW (crystalline silicon)
Estimated losses due to temperature and low irradiance: 6.1% (using local ambient temperature)
Estimated loss due to angular reflectance effects: 4.3%
Other losses (cables, inverter etc.): 14.0%
Combined PV system losses: 22.7%
Fixed system: inclination=90°, orientation=0°
MonthEd Em Hd Hm
Jan 0.5818.01.4043.5
Feb 0.7922.21.9454.2
Mar 1.1636.02.8989.7
Apr 1.2537.63.2597.4
May 1.0632.92.8789.1
Jun 1.0130.42.8184.2
Jul 1.0432.22.8889.2
Aug 1.0632.92.8788.9
Sep 1.1735.03.0591.5
Oct 0.9629.82.4475.5
Nov 0.7322.01.8053.9
Dec 0.5316.41.2739.3
Yearly average0.94628.82.4674.7
Total for year345896

Off-grid battery voltage/state

Battery Voltage

I log some aspects of my off-grid system such as live values for voltage, calendar month chart, and historic stats.

(See the gnuplot commands for live voltage plotting.)

Meter readings

Monthly meter readings and comments.

Gas consumption

Monthly Gas Consumption Since 2007

Gas consumption for hot water, central/space heating, and cooking, (manually) converted to monthly kWh/d figures, suitable for gnuplot to digest. (See the gnuplot commands.)

Yearly Gas Consumption Since 2005

Annual gas consumption in kWh, suitable for gnuplot to digest. (See the gnuplot commands.)

Gross electricity consumption

Monthly Electricity Consumption Since 2007 (gross)

Gross electricity consumption, ie compensating for microgeneration from PV, as if the PV were not there, in monthly kWh/d, suitable for gnuplot to digest. (See the gnuplot commands.)

Note the abnormally-high electricity consumption during the pandemic lockdown, roughly March 2020 to February 2021, that final month being the highest ever.

Yearly Electricity Consumption Since 2005 (gross)

Gross annual electricity consumption in kWh, suitable for gnuplot to digest. (See the gnuplot commands.)

Footprint from electricity and gas

Yearly E+G Carbon Footprint Electricity and gas (eg heat+light) annual carbon footprint in kgCO2e, suitable for gnuplot to digest. (See the gnuplot commands.)

Heating Degree Day data

Partial monthly summary of local HDD12 (base temperature of 12°C) data suitable for gnuplot to digest.

See annual data too.

Water mains inlet temperature

See 16WW water mains inlet temperatures for an idea of how much extra "lift" is required by the DHW system in winter, for example.

Relative humidity

See some manual relative humidity (RH) measurements at 16WW.

Live RH% for 16WW.

Also in the context of thermal discomfort.

Local temperature (etc) monitoring

See a graph of local internal temperature at 16WW collected from an OpenTRV unit tethered to the Web server.

201901 5s sensors left Radbot2 and SmartThings PIR right with window facing east to left

For the 16WW OpenTRV devices see:

and also see a 2016-09-22 snapshot.

recent 16WW temperatures

See initial data sets for server-local and all sensors, including external, for the end of April 2014.

See the OpenTRV/16WW public data set. Note that data formats have evolved over time, but are all plain text, in various common compressed archive formats for speed of download, etc.

iButton temperature monitoring

See: A Note On iButton Temperature Logging of Insulation Performance and data set.

all 4 traces for first fortnight or so

MyJoulo

I look up the offer at MyJoulo to place a small widget on top of my thermostat for one week to analyse our heating. Our unit arrived 2012-12-17 and ran until Christmas Eve.

See the generated analysis page (local copy) and the raw CSV data. (Also note that this sensor was about 2m away horizontally from button 3 in the iButton mission 5.)

snippet from temperature graph

The take-home message from the analysis is that it believes that we have our living room at the average UK 19°C and that our living room temperature varies between about 16°C and 21°C generally in a sawtooth pattern. (Note that 18°C in the morning before school seems to suffice.)

Note that from early evening 2012-12-19 I lowered the radiator flow temperature from about 60°C to about 45°C, independently of this, to see if the i30 PRC/eTRV could work with a lower heating rate. The reduced flow temperature seems to help reduce overshoot both in the target bedroom and in the living room, though the motivation was to try to test at typical heat-pump temperatures.

Note that we don't leave our house thermostat in the living room alone; we keep it lowish unless we feel cold and/or we want to force the heating on for the children's bedrooms before they go up. The latter action would be unnecessary if we had zoning with each bedroom able call for heat independently and those wasteful and soporific peaks >19°C could probably be trimmed. In the interim we should probably make sure that we get the thermostat down to ~18°C as the children go to bed to minimise overshoot.

A friend has generously provided his raw CSV data and chart too (from somewhat east of London), for comparison. In his case there is evidence of much tighter temperature control, at a similar mean level to ours, even with lower external temperatures.

Temperature and relative humidity data from thermal discomfort study

At the end of 2012 I participated in a UCL Energy Institute/LoLo study on responses to thermal discomfort, in return for access to the raw data collected by them on temperature and relative humidity at 16WW, which overlaps with some of my iButton mission 4 data collection.

Three draft plots (based on the sensors at 4 heights in each room):

were made available first, right at the end of the year. (The visible codes are: C04 Bedroom, C05 Living Room, C06 Kitchen.)

See the underlying raw data (note the room codes above on the filenames) in Mac .csv file format (with lines CR terminated, not CRLF or LF), from Stephanie Gauthier (then) of UCL whose project this was for, ZIPped.

Air Quality Egg (AQE)

I did not have much joy attempting to measure "air quality" with the AQE Air Quality Egg. These things are hard to do well and cheap at the same time.

Schematics

Logging (k8055)

All historic logging data from the k8055, mainly off-grid numbers, is available under data/k8055/YYYY-MM.gz, and a log of system changes is at the end of the historic stats.

Lead-acid voltage/SoC chart

Here is Solar John's SOC chart reproduced with permission, along with his note: "it provides only a rough estimate of the actual SOC":

% ChargeChargingAt RestDischarging
10014.7512.7012.50
9013.7512.5812.40
8013.4512.4612.30
7013.3012.3612.25
6013.2012.2812.15
5013.1012.2012.00
4012.9512.1211.90
3012.7512.0211.70
2012.5511.8811.50
1012.2511.7211.25

See also, for example Battery State-Of-Charge Chart For Voltage & Specific Gravity and Batteries Maintenance 101.

Lead-Acid Battery State of Charge vs Voltage (Richard Perez, Home Power #36, Aug/Sep 1993) notes that "At 32°F (0°C), the effect of temperature becomes pronounced enough to distinctly change not only the battery voltage vs. SOC profile, but also its useful Ampere-hour capacity. The discharge voltage curves may be depressed by as much as 0.5VDC from those shown on the graph. Charge voltages will be elevated by as much as 0.5VDC for a cold 12 Volt lead-acid battery." (This chimes with, for example, Victron's -24mV/°C threshold adjustment from a 20°C baseline.)

Factoids

A few random facts that are useful for rules of thumb, etc:

Code

GB grid intensity

Code used to collect and compute GB grid intensity numbers, live and historical, published under an BSD-style free/open licence on GitHub: reutils.

20110124: archival initial snapshot of Eclipse project:

Other

Just for giggles, note that there's nothing secret about the makefile used to build this site, though note that for tidiness not all of the supporting scripts are in public view. If anyone really cares I can make some or all of them visible too.

References

(Count: 1)