Earth Notes: On Data Collections and Graphs

Data collected in and used for our home (16WW) and elsewhere, open for research use.
temperature sensor graph

Here is a selection of the data collected and used; I intend to organise and collect more of it here, and graph it (and the tools to graph it). Please say if there is anything that you'd like me to do sooner...

Some of the data is marked up with JSON for Linking Data. The intent is to "start with the basics" as suggested by Google, including 'DataDownload' and 'license' information where possible. (This mark-up is checked with Google's Structured Data Testing Tool.)

This page also references circuit diagrams/schematics.

16WW Data

I've collected much data for our house at 16 Willingham Way (16WW) referred to in the various documents on this site.

Much of the data can be found here and is browsable, in simple text formats such as CSV, with common compression techniques applied for larger/non-live data sets.

Summary of continuing timeseries data sets available (and that may need manual archiving/attention):

SAP and Air-Leakage

SAP cert extract

To set a baseline and understand better what needed fixing, I had SAP, air-leakage and other tests done in early 2009.

Tests/results/certificates by Tophouse 2009/03/30 for us were:

Grid-Tie PV Generation

See live generation output when possible.

Daily PV Output Since 2008

This data is collected a year at at time, though there is a chart for the whole data set, and collated daily kWh generation records with smoothing. (See the gnuplot commands.)

2008/02/25: the first phase/round of PV, 1.29kWp (west-facing, 6 x Sanyo HIT 215Wp hybrid crystalline/thinfilm panels with SMA SunnyBoy 1100 inverter), was installed.

See the 2008 daily PV generation logs for 2008 as .csv and .png chart.

2009/02/10: the second phase was installed, taking the system to 3.87kWp; same panels and inverters for each of the two new east-facing strings as for the first phase.

See the 2009 daily PV generation logs for 2009 as .csv and .png chart.

2010/04/26: the third phase (west-facing, same panels, SMA SunnyBoy 1200 inverter) of 1.29kWp was installed, taking the whole system to 5.16kWp. New tariff and generation meter for this last part.

See the 2010 daily PV generation logs for 2010 as .csv and .png chart.

See the 2011 daily PV generation logs for 2011 as .csv and .png chart.

See the 2012 daily PV generation logs for 2012 as .csv and .png chart.

See the 2013 daily PV generation logs for 2013 as .csv and .png chart.

See the 2014 daily PV generation logs for 2014 as .csv and .png chart.

See the 2015 daily PV generation logs for 2015 as .csv and .png chart.

See the 2016 daily PV generation logs for 2016 as .csv and .png chart.

(See the gnuplot commands for annual plots, and the current awk 'V2' smoothing code ('V1').)

See other system performance records, including per-string outputs.

Intraday Data

Rob Ferber of ElectronVault wrote to me looking for intra-day generation data (with 10 minute sampling intervals), of which I had so far exposed very little because of data-cleanup issues (bizarre time offsets in the Sunny Beam apparently attempting to correct to German civil time even though I keep it on UTC, serial numbers, corrupt data, slight drift of Sunny Beam from true UTC, etc).

EV graph

I capture these intraday samples manually after dusk about twice per month, and possibly on particularly 'interesting' days, of late using the SunnyBeamTool c/o Michaël Peeters on my SheevaPlug (and now, as of 2014, Raspberry Pi B+) ARM Linux system. I didn't do this continuously because I have not been able to stop the Sunny Beam apparently drawing considerable power via the USB port, and because the SheevaPlug didn't have a 'spare' USB port: I manually disconnected the USB I/O board temporarily to connect the Sunny Beam. (As of 2014/07 experimentally I have the Sunny Beam plugged in permanently, and take 5-minute samples during daylight hours.)

Rob produced lovely tweaked graphs from my raw data up to September 2012. Thanks, Rob: great job!

Prior to Rob's analysis I had made public but not announced these intraday samples from SMA's Windows application, with no explicit time correction: 2008/03/24 2008/04/01 2008/04/09 2008/04/17 2008/05/08 2008/09/13 2008/09/14 2008/09/19 2008/10/11 2008/12/13 2009/03/15 2009/04/08 2009/05/11 2009/05/24 2009/06/29

The entire raw data set is now available, including new samples as I take them, though with many caveats (beware timestamp wobbles in particular).

See the interesting 2015/03/20 (cloudy) morning dip in PV output (raw SunnyBeam data with time offset, per-minute data) during the partial solar eclipse (~85% maximum at ~09:30).

PVGIS Prediction

Here is an extract of the numbers from the PVGIS estimate not knowing anything about our local trees, etc!

Solar radiation database used: PVGIS-classic
Nominal power of the PV system: 5.2 kW (crystalline silicon)
Estimated losses due to temperature: 7.3% (using local ambient temperature)
Estimated loss due to angular reflectance effects: 4.1%
Other losses (cables, inverter etc.): 14.0%
Combined PV system losses: 23.5%

Fixed system: inclination=23°, orientation=-90°
Yearly average10.33152.6380.0
Total for year3780960

Ed: Average daily electricity production from the given system (kWh)
Em: Average monthly electricity production from the given system (kWh)
Hd: Average daily sum of global irradiation per square meter received by the modules of the given system (kWh/m2)
Hm: Average sum of global irradiation per square meter received by the modules of the given system (kWh/m2)

Note also that adding just 10% (~500Wp) vertically mounted south-facing PV would add about 1/3rd to mid-winter generation from the main array (ie ~1.6kWh/d actual from the roof and ~0.5kWh/d from the new array predicted.)

Nominal power of the PV system: 0.5 kW (crystalline silicon)
Estimated losses due to temperature and low irradiance: 6.1% (using local ambient temperature)
Estimated loss due to angular reflectance effects: 4.3%
Other losses (cables, inverter etc.): 14.0%
Combined PV system losses: 22.7%
Fixed system: inclination=90°, orientation=0°
Yearly average0.94628.82.4674.7
Total for year345896

Off-grid Battery Voltage/State

Battery Voltage

I log some aspects of my off-grid system such as live values for voltage, calendar month chart, and historic stats.

(See the gnuplot commands for live voltage plotting.)

Meter Readings

Monthly meter readings and comments.

Gas Consumption

Monthly Gas Consumption Since 2007

Gas consumption for hot water, central/space heating, and cooking, (manually) converted to monthly kWh/d figures, suitable for gnuplot to digest. (See the gnuplot commands.)

Yearly Gas Consumption Since 2005

Annual gas consumption in kWh, suitable for gnuplot to digest. (See the gnuplot commands.)

Gross Electricity Consumption

Monthly Electricity Consumption Since 2007 (gross)

Gross electricity consumption, ie compensating for microgeneration from PV, as if the PV were not there, in monthly kWh/d, suitable for gnuplot to digest. (See the gnuplot commands.)

Yearly Electricity Consumption Since 2005 (gross)

Gross annual electricity consumption in kWh, suitable for gnuplot to digest. (See the gnuplot commands.)

Footprint From Electricity and Gas

Yearly E+G Carbon FootprintElectricity and gas (eg heat+light) annual carbon footprint in kgCO2e, suitable for gnuplot to digest. (See the gnuplot commands.)

Heating Degree Day Data

Partial monthly summary of local HDD12 (base temperature of 12°C) data suitable for gnuplot to digest.

See annual data too.

Water Mains Inlet Temperature

See 16WW water mains inlet temperatures for an idea of how much extra "lift" is required by the DHW system in winter, for example.

Local Temperature (etc) Monitoring

See a graph of local internal temperature at 16WW collected from an OpenTRV unit tethered to the Web server.

For the 16WW OpenTRV devices see:

and also see a 2016/09/22 snapshot.

recent 16WW temperatures

See initial data sets for server-local and all sensors, including external, for the end of April 2014.

iButton Temperature Monitoring

See: A Note On iButton Temperature Logging of Insulation Performance.

all 4 traces for first fortnight or so


I look up the offer at MyJoulo to place a small widget on top of my thermostat for one week to analyse our heating. Our unit arrived 2012/12/17 and ran until Christmas Eve.

See the generated analysis page (local copy) and the raw CSV data. (Also note that this sensor was about 2m away horizontally from button 3 in the iButton mission 5.)

snippet from temperature graph

The take-home message from the analysis is that it believes that we have our living room at the average UK 19°C and that our living room temperature varies between about 16°C and 21°C generally in a sawtooth pattern. (Note that 18°C in the morning before school seems to suffice.)

Note that from early evening 2012/12/19 I lowered the radiator flow temperature from about 60°C to about 45°C, independently of this, to see if the i30 PRC/eTRV could work with a lower heating rate. The reduced flow temperature seems to help reduce overshoot both in the target bedroom and in the living room, though the motivation was to try to test at typical heat-pump temperatures.

Note that we don't leave our house thermostat in the living room alone; we keep it lowish unless we feel cold and/or we want to force the heating on for the children's bedrooms before they go up. The latter action would be unnecessary if we had zoning with each bedroom able call for heat independently and those wasteful and soporific peaks >19°C could probably be trimmed. In the interim we should probably make sure that we get the thermostat down to ~18°C as the children go to bed to minimise overshoot.

A friend has generously provided his raw CSV data and chart too (from somewhat east of London), for comparison. In his case there is evidence of much tighter temperature control, at a similar mean level to ours, even with lower external temperatures.

Temperature and Relative Humidity Data from Thermal Discomfort Study

At the end of 2012 I participated in a UCL Energy Institute/LoLo study on responses to thermal discomfort, in return for access to the raw data collected by them on temperature and relative humidity at 16WW, which overlaps with some of my iButton mission 4 data collection.

Three draught plots (based on the sensors at 4 heights in each room):

were made available first, right at the end of the year. (The visible codes are: C04 Bedroom, C05 Living Room, C06 Kitchen.)

See the underlying raw data (note the room codes above on the filenames) in Mac .csv file format (with lines CR terminated, not CRLF or LF), c/o Stephanie Gauthier (then) of UCL whose project this was for, here.

Relative Humidity

See some manual relative humidity (RH) measurements at 16WW.

Live RH% for 16WW.


Logging (k8055)

All historic logging data from the k8055, mainly off-grid numbers, is available under data/k8055/YYYY-MM.gz, and a log of system changes is at the end of the historic stats.

Lead-Acid Voltage/SoC Chart

Here is Solar John's SOC chart reproduced with permission, along with his note: "it provides only a rough estimate of the actual SOC":

% ChargeChargingAt RestDischarging

See also, for example Battery State-Of-Charge Chart For Voltage & Specific Gravity and Batteries Maintenance 101.

Lead-Acid Battery State of Charge vs Voltage (Richard Perez, Home Power #36, Aug/Sep 1993) notes that "At 32°F (0°C), the effect of temperature becomes pronounced enough to distinctly change not only the battery voltage vs. SOC profile, but also its useful Ampere-hour capacity. The discharge voltage curves may be depressed by as much as 0.5VDC from those shown on the graph. Charge voltages will be elevated by as much as 0.5VDC for a cold 12 Volt lead-acid battery." (This chimes with, for example, Victron's -24mV/°C threshold adjustment from a 20°C baseline.)


A few random facts that are useful for rules of thumb, etc:


GB Grid Intensity

Code used to collect and compute GB grid intensity numbers, live and historical, published under an BSD-style free/open licence on GitHub: reutils.

20110124: archival initial snapshot of Eclipse project:


Just for giggles, note that there's nothing secret about the makefile used to build this site, though note that for tidiness not all of the supporting scripts are in public view. If anyone really cares I can make some or all of them visible too.