Earth Notes: On Setting Up a Raspberry Pi 3 Off-grid Server (2018)

Updated 2023-12-30.
RPi3B upgrade for more speed and space... #frugal #offGrid
As of 2020-06 I had been making progress with the RPi3B+ as an upgrade to the RPi2B+ again, after briefly flirting with the idea of upgrading straight to an RPi4B. Then I downgraded to an RPi3B (not plus) to save 0.5W and run cooler, but likely not slower. Also, my HAT setup is streamlined! I no longer expect to use the RPi3 as a primary WiFi AP since the Vigor has sophisticated support.

2018-07-13: Old Router Died?

As of 2018-07-13 I thought that my stand-alone Technicolor TG582n router had died, though it seemed OK again after several hours powered off. But it consumes ~8W and crashes about once per month. Also, since it is talking PPPoE (PPP over Ethernet) to a BT Openreach ECI Telecom ON316150 FTTC modem it does not need to do anything analogue, so the routing and filtering done by the Technicolor ought to be do-able by a Pi. Also the Pi 3 has on-board WiFi that can be used as an access point.

Another advantage of an upgrade to the Pi 3 B+ is a bit more speed, and a bigger range of power draw to make better use of available sunshine, etc. Though I have to see if the base consumption of a stripped back Pi 3 B+ is still OK.

I can also take the opportunity to buy a larger SD card to boost storage, and effectively keep the old SD card as backup. (Currently the RPi2 has a 128GB micro SD card.)

Yet another advantage of an upgrade is moving to a newer OS distribution, allowing access to a newer Apache, HTTP/2, and various other goodies.

So an RPi 3 Model B+ and mains and 12V power supplies and other goodies have been ordered from RS to get started.

I'll record below as I prototype various aspects of PPPoE networking, new HTTP/2 support, etc.

I anticipate a fair period of prototyping and testing before (re)building the production server.

Start

Starting with just the following RS parts, no keyboard or mouse or screen plugged in, and waiting for it to boot, settles to 2.5W power consumption measured at the mains.

Starter Kits
1373331Raspberry Pi 3 Model B+
8157334USB DC/DC Adapter 12Vin 5Vout 2.1A
9098135Official Pi 3 Power Supply Black
1213897SD Card preloaded with NOOBS - 16GB

Directly plugging in the USB keyboard and mouse from a previous Maplin Raspberry Pi kit (R45Pi) raises consumption to 3.4W. Plugging in HDMI does not add any further load/consumption.

Have logged in and changed the default password for user 'pi'.

The WiFi does not seem to be in use to grab an IP (with DHCP) and get an Internet connection. This may be normal for NOOBS.

Via the Preferences menu entry and the Raspberry Pi Configuration tool I have:

  • Set the system to boot to CLI rather than desktop.
  • Set GPU memory to minimum (16MB).
  • (NOT enabled SSH yet.)
  • Set WiFi country to GB.

I am being asked to reboot for some of those to take effect.

I am still being auto-logged-in, which on the console is probably OK.

ifconfig now shows wlan0, though not with an IP address.

uname -a shows ... 4.9.80-v7+ #1098 Fri Mar 9 19:11:42 GMT 2018 armv71 ...

Following Setting WiFi up via the command line I sudo raspi-config and in "Network Options" enter SSID and passphrase. I exit raspi-config and ... I have an IP address on wlan0 and can ping the outside world!

Security dictates that the very next thing to do is sudo apt-get update && sudo apt-get-dist-upgrade.

Space used in the root partition is ~1GB.

With all that finished, power consumption (measured at the mains) is 3.1W. (Unplugging the USB mouse seems to push that up to 3.2W!)

I have established that the RPi3B+ and WiFi are working. Also that power consumption is not outrageous even before any attempts to tune it.

PPPoE, AP, Downstream

The next step may be to load up a minimal Raspbian, and (briefly) get the RPi3B+ talking PPPoE to my ISP and providing a WiFi Access Point. This would probably have to be a quick temporary swap-out with the current server, as none of the other services would be there, so there will be HTTP and mail server fails while the RPi is in place.

I also have to provide down-stream wired routed (and NATted) Ethernet connections for devices that need it, such as my Loop and SmartThings hubs, and my new storage system. They may need some protection against external attack, and possibly against being used to attack other things inside the house. So I'll need to adjust my iptables config.

As a first pass attempt to support the two downstream wired devices I have bought an AX88179-based USB-to-dual-Ethernet dongle (VTOP USB 3.0 Gigabit RJ45 10/100/1000Mbps Network Lan Adapter 2 Port) which both my Mac and the RPi3B+ appear to see immediately. I can't find a published power-consumption figure, so I will take some measurements in due course. None of the connections need run very fast, since the traffic from the devices is essentially only buffered telemetry. For the RPi3B+ the new ports show up as eth1 and eth2, so for robust and secure configuration of that and the on-board port I may need a UUID or other more robust identifier.

Storage

On 2018-10-12 I ordered a 256GB Class 10 U3 micro SDXC memory card (Samsung MB-MC256GA Micro SDXC EVO PLUS Class 10 UHS-I U3, Amazon). The price was a little over £50+VAT.

My choice was partly informed by Raspberry Pi Dramble's microSD Card Benchmarks.

Looking at the root partition size given as 235GB, I suspect that there is some trailing space at the end of the device not being used. Worth investigating later to see if the partition can be grown a little. (Subsequent inspection with fdisk revealed only a couple of small unused slivers.)

OS Download

Given that the "... "lite" download is about half that size, for the entire OS. The full fat version is over 4GB" (Ken Hagan), I'm inclined to start with "lite" to save some of the new GB!

So, from the Raspbian download page I am pulling the "Raspbian Stretch Lite" "Minimal image based on Debian Stretch" with release date 2018-10-09 (kernel 4.14, 368MB, SHA-256 98444134e98cbb27e112f68422f9b1a42020b64a6fd29e2f6e941a3358d171b4) as a ZIP file.

For lolz, I'm trying Etcher as suggested on the Raspbian page. Nice and simple, but warned me that 256GB was "unusually large" for it to Flash. I told it to go ahead anyway. The process was fast, and the card was left unmounted afterwards.

Restart

2018-10-20: I inserted the "Lite" microSD card into the RPi3B+, connected up keyboard, plugged HDMI into the family TV, and powered up.

Some items noted during boot:

  • A message appeared (somewhere, now off-screen!) about the (root) partition being resized. (Looking at dmesg and df -h output shows that there is a small /boot partition of 44MB and a huge ext4 root partition of 235GB. OK, though previously I have had a separate /local with most of the actual data in it. It looks like there are plenty of inodes with 15M compared to ~360k used in RPi2 main filesystem.)
  • "Started LSB: Autogenerate and use a swap file." (I'll likely want to run swapless again, with some zram. There appears to be a 100MB swap file in /var/swap.)
  • SSH host keys have been regenerated.
  • Bluetooth service has been enabled (I'll most likely want to turn that off for security and power reasons).
  • A message says that "Wi-Fi is disabled because the country is not set. Use raspi-config to set the country before use."

I logged in as pi and changed the password. I should now be safe to connect up to the Intartubes and get updates, for example.

I have changed the fstab parameters of the root filesystem to be defaults,noatime,commit=120 as on the RPi2 to reduce write traffic and improve performance, and rebooted. I may further raise the commit interval to 300s as on the RPi2's /local partition.

I have set the hostname to sencha with raspi-config.

I have set the Wi-Fi country to GB with raspi-config, connected to the Net with the correct SSID and password by editing /etc/wpa_supplicant/wpa_supplicant.conf with a 'network' entry with 'ssid' and 'psk' items.

Then apt-get update and apt-get dist-upgrade to be up to date on security. At this moment, no updates were needed.

I have set up NTP with apt-get but have not yet fully configured it (ie accepting defaults for now).

At this moment, power consumption from the mains, with HDMI on, is shown as ~3.2W. The target is to get that as near as possible to 1W when idling without HDMI.

Avoiding need to use the Family TV!

2018-10-21: I have fixed the IP address handed out by the router to the RPi3B+ with DHCP. This will mean that the RPi3B+ has a stable LAN IP address. That in turn should make accessing the RPi3B+ via ssh easier, thus avoiding conflict over use of the family TV as console!

I also need to allow SSH to run on the RPi3B+ (with raspi-config), still at this point protected behind the firewall/NAT.

(Experimentally turning off HDMI at this point made no visible difference to power draw, still fluctuating around ~3.2W; similar to the original RPi experience. Turning off the red LED temporarily with echo none > /sys/class/leds/led1/trigger does not make a visible difference to mains consumption either. Doing this however may disable undervoltage detection by the system, which would be bad.)

I have created my user ID on the RPi with the same uid. I can rsync stuff across, repeating as I get close to bringing the RPi3B+ live.

Per making HTTP/2 work better I have added the following lines to /etc/sysctl.conf and rebooted:

net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_notsent_lowat = 16384

Incidentally, reboots seem much faster than on the RPi2, <30s, possibly because of systemd which seems to be in this Raspbian release.

Added main a/c to sudoers, and will disable pi a/c in due course.

Installed the Apache HTTP/Web server (apt-get install apache2). It looks as if it may be bringing TLS (HTTPS) support with it. (It serves pages claiming to be Apache 2.4.25 (Raspbian).)

The following additional packages will be installed:
  apache2-bin apache2-data apache2-utils libapr1 libaprutil1
  libaprutil1-dbd-sqlite3 libaprutil1-ldap liblua5.2-0 ssl-cert
Suggested packages:
  www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom
  openssl-blacklist

The event service model seems to be enabled by default, as a module, (mpm_event) and its for this low-traffic low-memory server all its parameters can be roughly halved, to more like the RPi2 worker values.

Other resource use may need pruning too.

For example, each Apache process+threads on the RPi3B+ shows a virtual memory size of 222MB compared to ~20MB on the RPi2 out of the box. Inserting the following in the Apache envvars file reduces the RPi3B+ vm size to more or less that of the RPi2:

# NPTL (modern Linux threads) defaults the thread stack size to the setting
# of your stack resource limit. The system-wide default for this is 8MB,
# which is waaay exaggerated when running httpd.
# 512kB should be more than enough (AIX manages on 96kB, Netware on 64kB).
ulimit -s 512

Performance

As a simple benchmark I run the command to (re)validate the generated HTML using the W3C Java tool (which examines all the desktop pages plus the mobile home page):

rm .work/tmp/pages.valid && time make .work/tmp/pages.valid

Wall clock time taken by my ~2014 MacBook Air is a little under 7s, the new RPi3B+ ~20s, and the RPi2 ~44s. (Some of this may be due to a newer faster Java (JDK8) on the RPi3B+.) But in any case this RPi3 is much nearer to laptop performance than the RPi2.

To-Do List

Roughly prioritised...

  1. Set timezone to UTC and localisation to en-GB.
  2. Networking:
    1. (Done) Temporarily give RPi3B+ stable LAN address for setup work.
    2. Temporarily have Pi as Wi-Fi AP and client during set-up to avoid needing the TV and as a recovery mode!
    3. Tighten sshd security, eg limit users allowed and forbid password-based logins.
    4. Set up PPPoE.
    5. Turn on anti-spoofing reverse path filters (net.ipv4.conf.default.rp_filter etc in /etc/sysctl.conf.
    6. Set up routing, NAT, DHCP and extended ipfilter/firewall local private LAN(s) wired and WiFi.
    7. Set up new local LAN with DHCP 'behind' RPi3B+; it may be possible to move Loop and SmartThings devices on to this new LAN well before rest of work is completed, and in any case they will buffer data through brief interruptions of connectivity.
    8. Possibly set up tc.
    9. Get USB/Ethernet multi-RJ45 dongle for Loop and SmartThings hub such as CU200.
    10. Install/configure BIND and DNS master.
  3. Swap and filesystems:
    1. (Done) Get swappiness right (1?). The RPi2 /etc/sysctl.conf has vm.swappiness=1 and vm.min_free_kbytes = 8192.
    2. (Done) Enable zram as on RPi2 in /etc/rc.local with modprobe zram; echo 128128128 > /sys/block/zram0/disksize; mkswap /dev/zram0; swapon /dev/zram0
    3. (Done) Disable the ~100MB automatically-configured /var/swap swap file with swapoff /var/swap; systemctl disable dphys-swapfile; rm -f /var/swap.
    4. (Done) Reduce SD card traffic per RPi2 with /etc/sysctl.conf:
      vm.dirty_background_ratio=20
      vm.dirty_ratio=40
      vm.dirty_writeback_centisecs=1500
      vm.dirty_expire_centisecs=12101
      
    5. (Done) Increase ext4 filesystem commit time to 300s.
    6. (Done) See if there is unused space at the end of the root partition that it could be grown into... (235GB looks a bit low even allowing for overheads and /boot. Inspection with fdisk revealed only a couple of small unused slivers.)
    7. [Done] Adding following to fstab to reduce SD traffic: tmpfs /tmp tmpfs defaults,noatime,nosuid 0 0. (Also retrofitted to RPi2 config as visibly (green led0) useful in reducing RPi3B+'s SD traffic during EOU site rebuild test.)
    8. [TODO] Reduce syslog logging to reduce write traffic.
    9. [Done] Add weekly fstrim to help maintain filesystem performance, with a new /etc/cron.daily/fstrim script.
    10. Add a daily root cron task for ionice -c 3 fstrim -v / and other non-memory partitions as per ext3/4 and SD-card wear leveling.
  4. Time:
    1. (Done) Install NTP FCS/vanilla config.
    2. (Done) Configure NTP per RPi2 such as bringing low-tier servers over and adding some defence against misuse. The restarted with systemctl restart ntp.
    3. Try again to join UK (or Debian) pool.ntp.org service; traffic was overwhelming before but faster connection and CPU may help now.
    4. (Done) Install/configure hardware clock (such as AB Electronics' Expander Pi).
  5. Power:
    1. (Done) Install/configure/enable ondemand governor.
    2. (Done) Turn off HDMI to save energy (~100mW) if no one logged in, per RPi2 rc.local.
    3. Set lower idling CPU clock to save energy (idle default apparently 600MHz, non-idle 1400MHz if not thermally throttled).
    4. (Done) Turn off Bluetooth to save energy (dtoverlay=pi3-disable-bt in /boot/config.txt or blacklist btbcm and blacklist hci_uart in /etc/modprobe.d/raspi-blacklist.conf and turn off Bluetooth services, eg sudo systemctl disable bluetooth and sudo systemctl disable hciuart?).
    5. (Done) Turn off unnecessary LEDs eg on network connector eg have green activity LED only show SD card activity when power OK or better, else off/'none'. (Turning off red power LED may disable low-voltage detection.)
    6. Find out if Ethernet port can save power in sub-Gbit mode, eg when storage is in a low- or very-low- power state. (ethtool -s eth0 speed [100|1000] duplex full: can probably safely keep at 100Mbps for PPPoE FTTC link, saving maybe ~500mW.)
    7. Selectively turn off some of the USB ports, eg for Sunny Beam overnight to force it to run on its own batteries (and power down its display). Two ports (for OpenTRV and battery stats) probably need to be left on, though maybe the latter could be powered up just to take readings once every 10 minutes.
    8. Construct 2.5A (12.5W) capable supply from 12V. Testing with a Muker 10VA USB Multimeter Charger Detector (UT-KWS-10VA) shows red power light goes out when current draw hits as little as ~730mA (3.8W) from a variety of nominal ≥2A USB supplies, and also MacBook Air USB port. Minimum/idle consumption immediately after boot ~2W. Consider Anker and USB-C to micro-USB 2 cable.
    9. Turn off other unnecessary h/w.
  6. Mail:
    1. (Done) Remove or disable exim4. (systemctl disable exim4.service.)
    2. Install and configure sendmail.
    3. Install and configure POP3 (dovecot?).
  7. Apache:
    1. (Done) Install Apache.
    2. (Done) Tune Apache for small-system memory and performance (roughly halved event mpm params and much reduced per-thread stack size).
    3. (Done) Use/configure Apache event model: event seems enabled by default and simply needs configuring. Initial tweak of mpm_event done.
    4. Copy Web sites (uid, data, cron) across from RPi2.
    5. Install/configure Apache w/ HTTPS and HTTP/2 support.
    6. Configure Apache log rolling.
    7. Configure (Apache) Brotli support and static pre-geneneration.
    8. Capture (eg in SVN) config changes (eg for mpm_event).
  8. Copy existing user accounts (uid, data, cron) across from RPi2.
  9. Copy Gallery (uid, data, metadata, app, cron) across from RPi2.
  10. Set up extra log directories for Sunny Beam, powermng, Enphase under /var/log.
  11. Copy power management code etc and move HATs.
  12. Security (misc):
    1. Install support for h/w RNG (rgnd / hwrng) per RPi2.
    2. Disable pi a/c in due course.
  13. SVN:
    1. Take snapshots of SVN repos, and archive some off-site.
    2. Disable RPi2 repo check-ins.
    3. Copy snapshots to RPi3B+ and unpack.
    4. Switch RPi3B+ remote use of RPi2 to local file:// access.
  14. Tune boot time eg using systemd-analyze blame. Top time hogs (>1s) as at 2018-10-25 as a WiFi client are:
    1. 8.851s dhcpcd.service
    2. 6.776s hciuart.service
    3. 1.862s dev-mmcblk0p2.device
    4. 1.324s apt-daily-upgrade.service
    5. 1.098s exim4.service
  15. Config to do and to capture explicitly in SVN:
    1. Copy appropriate rc.local features from RPi2 to RPi3B+ and capture in SVN.
    2. Capture (eg in SVN) /etc/sysctl.conf changes for TCP BBR/lowat/fq etc.
    3. Capture FCS and updated ntp.conf.
    4. Capture FCS and updated fstab.

Application Inventory

For significant applications added to the RPi 3, eg with apt-get or npm, a note will be made of what and when and why here.

  1. YYYY-MM-DD, apt-get/npm command, motivation and comments.
  2. 2018-10-20, apt-get install tcsh, because I like tcsh!
  3. 2018-10-20, apt-get install ntp, for good timekeeping.
  4. (2018-10-22, apt-get install htop, to help sysadmin; already installed!)
  5. 2018-10-22, apt-get install cpufrequtils, for better power management.
  6. 2018-10-22, apt-get install apache2, for Web serving.
  7. 2018-10-23, apt-get install npm, for EOU and other purposes; followed by sudo npm install npm -g (twice) to self-upgrade.
  8. 2018-10-23, apt-get install zopfli, for EOU and other purposes.
  9. 2018-10-23, apt-get install optipng, for EOU.
  10. 2018-10-23, apt-get install imagemagick, for EOU.
  11. 2018-10-24, apt-get install procmail, for lockfile for EOU and others, 29MB of diskspace for this one utility executable!
  12. 2018-10-24, apt-get install subversion, for local (EOU/ExNet/etc) repos.
  13. 2018-10-24, npm install html-minifier@3.5.19 -g, for EOU mobile and other pages. (Latest @3.5.20 does not work when fed from stdin; seems to be fixed in 3.5.21...)
  14. 2018-10-24, npm install uncss -g, for EOU mobile and desktop pages.
  15. 2018-10-24, npm install purify-css -g, for EOU mobile and desktop pages.
  16. 2018-10-24, apt-get install libjpeg-progs, for jpegtran for EOU.
  17. 2018-10-24, apt-get install libfile-slurp-perl, to support jpegrescan.
  18. 2018-10-24, npm install jpegrescan -g --unsafe-perm=true, for EOU.
  19. 2018-10-24, installed https://github.com/MegaByte/jpegultrascan/blob/master/jpegultrascan.pl as /usr/local/bin/jpegultrascan.pl, for EOU.
  20. 2018-10-24, npm install clean-css-cli -g, not cleancss!, for EOU.
  21. 2018-10-24, npm install reado-cli -g, for EOU.
  22. 2018-10-24, apt-get install oracle-java8-jdk, for EOU and other uses such as the Gallery.
  23. 2018-10-25, apt-get install gnuplot, for EOU and other uses. Uses ~315MB of file space!
  24. 2018-10-30, apt-get install brotli, for future EOU. (A matching brew upgrade brotli (0.6.0 to 1.0.7) on the Mac changes the executable name from bro to brotli to match the RPi.
  25. 2018-11-17, npm install amphtml-validator -g, for EOU AMP support. (Doesn't appear to install correctly by itself...)
  26. 2018-12-31 PENDING, npm i -g cssgip, for future EOU image support.
  27. 2019-04-25: apt-get install dnsutils, for dig.
  28. 2019-04-25: apt-get install tcpdump.
  29. 2019-07-14: apt-get install mediainfo, 0.7.91, for EOU.
  30. 2019-07-14: apt-get install libav-tools, (ffmpeg 3.2.14), for avconv for EOU.
  31. 2019-07-14: apt-get install ethtool, for checking link speed.
  32. 2020-06-12: downloaded and built and manually installed zopfli and zopflipng 1.0.3 from the official repo [zopfli] to /usr/local/bin. No 'apt-get' package seemed to be available for my RPi.

2020-06-17: snapshot converted to ansible role to automate future server configurations.

2020-06-18: Ansible role also fixes up npm permissions after global installs, and installs OpenJDK, as well as building and installing zopflipng on the fly.

Other:

  • sudo usermod -a -G adm dhd for read access to /var/log/apache2 logs.
  • Following appended to tail of /boot/config.txt:
    # For RTC on Expander Pi.
    dtoverlay=i2c-rtc,ds1307
    # To have system run cooler.
    temp_soft_limit=58
    temp_limit=70
    
  • Following appended to tail of /etc/rc.local:
    # Power down the TV output after a while to save 100mW after warning users.
    # Happens after a delay to allow system recovery using TV/keyboard.
    # Not done if anyone appears to be logged in.
    # COMMENT OUT THE FOLLOWING LINE TO KEEP THE TV OUTPUT RUNNING.
    (sleep 180 && if [ "X" = "X`w -h`" ]; then logger -p user.emerg "TV service off in 2 mins; see /etc/rc.local" && sleep 120 && /opt/vc/bin/tvservice -o; else logger -p user.emerg "TV service not being turned off; do it manually to save power"; fi) &
    # Set up small ZRAM compressing 'swap'.
    modprobe zram
    echo 128128128 > /sys/block/zram0/disksize
    mkswap /dev/zram0
    swapon /dev/zram0
    
  • Following appended to tail of /etc/sysctl.conf (TODO: these are from RPi2B+):
    # rpi tweaks
    vm.swappiness=1
    vm.min_free_kbytes = 8192
    # DHD20140628: further tweaks to reduce SD card traffic.
    vm.dirty_background_ratio=20
    vm.dirty_ratio=40
    vm.dirty_writeback_centisecs=1500
    vm.dirty_expire_centisecs=12101
    # DHD20181124: back porting to help reduce bufferbloat.
    net.ipv4.tcp_notsent_lowat = 16384
    

TODO: sendmail (and remove/disable exim4?)...

While on my RPi2 and Mac npm utilities have been installed in /usr/local/bin, for the RPi3B+ they've ended up in /usr/bin which mixes them up with system binaries and breaks a lot of my scripts relying on them being in the former.

Pi Day

2019-03-08: now waiting to see what happens on Pi Day...

Pi Day came and went with no RPi 4 announced, bah!

2019-06-24: The Raspberry Pi Foundation unveils the Raspberry Pi 4; faster and up to 4GB, but ~3W idle consumption is going in the wrong direction for off-grid IMHO...

2018-04-07: Emergency Network Fix Needed

In early April 2019 the router went largely deaf, so that only devices very close to it could talk to the outside world, and started dropping external connectivity at random and for extended periods.

The ISP ignored the problem hard enough to let a couple of weekends slip by, including shipping a "drop-in replacement" router which was essentially unconfigured and not capable of replacing the existing device at all.

Though its MD helpful was as ever, and offered to replace equipment at cost, since I have the RPi3B+, rather than spend over £200 buying a new router and spending time configuring it, I might as well strip down the RPi3B+ and try to configure it to do the job. If that does not work, I can splash the cash.

So the RPi3B+ was designated to be diverted to be the router and AP. Moving functionality from the RPi2 can happen by degrees if that all works.

So another apt-get update etc, then turn off any unwanted services for the router. I'm not sure if I should leave an Apache running (supporting something static such as the exnet.com site).

Some steps to get there, including information gathering (not all in right order, and largely following Raspberry Pi 3 as a Simple WiFi Access Point):

  1. (Done) apt-get update and apt-get dist-upgrade RPi3B+ for security patches.
  2. (Done) Capture current line configuration including username and password.
  3. *** Capture current IP use on RPi2, eg for interfaces and Web sites and mail, NTP, etc.
  4. (Done) Strip down RPi3B+ to minimum services.
    1. (Done) sudo update-rc.d apache2 disable
    2. (Done) Harden sshd, and reduce CPU impact (etc) of typical attacks.
  5. (Done) Load enabling packages:
    1. (Done - backed out 2019-06-24) sudo apt-get install dnsmasq hostapd
    2. (Done) sudo apt-get install iptables-persistent
    3. (Done) sudo apt-get install pppoeconf
  6. (Done) Plug in RPi3B+ eth0 port to spare current TG router port.
  7. (Done) Set temporary static IP address (in NATted range) for eth0 so that most of config including fowarding and NATting can be completed behind current router.
  8. (Done) Check log-in currently possible to RPi3B+ via WiFi and wired connections.
  9. (Done - backed out 2019-06-24) Disable DHCP client on RPi3B+: sudo systemctl disable dhcpcd.service.
  10. (Done - backed out 2019-06-24) Prevent DHCP client on RPi3B+ from attempting to assign an address to wlan0 by adding denyinterfaces wlan0 in /etc/dhcpcd.conf (above any other interfaces added there).
  11. (Done - backed out 2019-07-02) Configure IP forwarding: add net.ipv4.ip_forward=1 to /etc/sysctl.conf.
  12. (Done - backed out 2019-06-24) Provide static IP configuration for wlan0 in /etc/network/interfaces.d/wlan0 with temporary range 192.168.220.X, and reload interface with sudo ifdown wlan0; sudo ifup wlan0.
  13. (Done) Configure /etc/hostapd/hostapd.conf for wlan0 with temporary SSID and password and update /etc/default/hostapd to use it.
  14. Unmask and tentatively bring up the WiFi AP: sudo systemctl unmask hostapd; sudo systemctl enable hostapd; sudo systemctl start hostapd; the AP should be visible to WiFi users but not yet functional.
  15. (Done) Drop in IP filter to minimise attack surface and help with NAT/bridge based on WiFi Access Point rule set with some config from current RPi2 folded in.
  16. Edit /etc/resolvconf.conf?
  17. (Done) Edit /etc/dnsmasq.conf naming local servers and (say) Google, on the lines of:
    interface=wlan0       # Use interface wlan0
    listen-address=192.168.220.1   # Specify the address to listen on
    bind-interfaces      # Bind to the interface
    server=8.8.8.8       # Use Google DNS
    domain-needed        # Don't forward short names
    bogus-priv           # Drop the non-routed address spaces.
    dhcp-range=192.168.220.75,192.168.220.150,12h # IP range and lease time
    
  18. (Done) sudo service dnsmasq start.
  19. (Done) Reboot - at this point AP is up and running behind existing router.
  20. (Done) Plug VTOP USB-to-dual-Ethernet adapter into RPi3B+ ready to accept Loop monitoring device network gateway, and one other as needed: appears as eth1 and eth2.
  21. (Done) Widen existing config to extend DHCP to eth1 and eth2:
    1. (Done) In /etc/dhcpcd.conf extend denyinterfaces wlan0 eth1 eth2.
    2. (Done) In /etc/iptables/rules.v4 allow forwarding between eth0 and eth1/eth2 internally (copy wlan0 rules for eth1 and eth2).
    3. (Done) Provide static IP configuration for eth1 and eth2 in /etc/network/interfaces.d/eth12 with range 192.168.221.X and 192.168.222.X, and reload interface with sudo ifdown ethX; sudo ifup ethX.
    4. (Done) In /etc/dnsmasq.conf add extra interface=s, listen-address= and dhcp-range=s for eth1 and eth2
  22. (Done) Move Loop device to eth1 and check able to phone home.
  23. Prepare set up bridging between eth1 and ppp0 so that eth2 can see the whole public address space and effectively become the DMZ, and the RPi2 can be plugged into it:
    1. (Nominally ppp0 and eth0 should be added to /etc/dhcpcd.conf' denyinterfaces, but keeping the DHCP client off entirely is what is really wanted...)
    2. (Done) Take all mention of eth1 and the 192.168.221.X range out of /etc/dnsmasq.conf.
    3. Replace static config for eth1 in /etc/network/interfaces.d/eth12 to match publich address range with:
      allow-hotplug eth1
      iface eth1 inet static
          address 79.135.97.94
          netmask 255.255.255.224
          network 79.135.97.64
          broadcast 79.135.97.95
      
    4. Modify the rules.v4 to only NAT traffic coming from the wlan0 and eth2 interfaces (192.168.X.X):
      # Allow Access Point NAT only from wlan0 and eth2 (192.168.X.X).
      -A POSTROUTING -s 192.168.0.0/16 -o ppp0 -j SNAT --to-source 79.135.97.65
      
  24. (Done) Have eth0 marked as 'manual' in interfaces/core.
  25. (Done) Change all eth0 in iptables to ppp0.
  26. Work out how to route static addresses though to (say) eth2.
  27. Configure PPPoE connection.
  28. Remove old router.
  29. Tackle any remaining points above not done!

2019-04-26: Pause for Thought

So, annoyingly, I got a long way but not far enough.

I can bring up my RPi3B+ behind the existing (fading) Technicolor and the RPi3B+ provides what seems to be a fairly decent WiFi access point and outward connectivity to the Internet. My existing RPi2 server sits in the Technicolor DMZ and presents services on its ~6 static addresses to the outside world.

I can bring up my RPi3B+ directly on the PPPoE connection via the ECI VDSL2 modem you originally supplied with the FTTC service. It can continue to provide WiFi AP. I cannot however get it to route between the ppp0 (on eth0) connection and the eth1 connection that the RPi2 is on, primarily because they both end up with 79.135.97.65 in their range, it seems (and ppp0/eth0 cannot be bridged with eth1):

ppp0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1492
        inet 79.135.97.65  netmask 255.255.255.255  destination 195.26.38.252
        ppp  txqueuelen 3  (Point-to-Point Protocol)
        RX packets 42937  bytes 5363610 (5.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 34314  bytes 4267734 (4.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 79.135.97.65  netmask 255.255.255.224  broadcast 79.135.97.95
        inet6 fe80::20f:c9ff:fe10:f6d9  prefixlen 64  scopeid 0x20<link>
        ether 00:0f:c9:10:f6:d9  txqueuelen 1000  (Ethernet)
        RX packets 12377  bytes 1693664 (1.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 22276  bytes 2357947 (2.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions

Very frustrating and if I could sort that then I'd have the whole thing done, I think. I'm a bit out of practice on knotty routing issues these days.

There just don't seem to be any good example of static public IP address blocks being routed to in this way with a recent Linux.

I really don't need fancy DMZ or NATting, and at a pinch the top half of the static address range may just have enough space for short-lease DHCP for our laptops, phones etc.

Update 2019-05-02: starting the process of getting an ISP-issued Vigor 2862ac set up as modem, router and WiFi AP, at least for now!

2019-06-23: No RPi3B+ WiFi

Since the Vigor is seemingly doing a good job, and has good WiFi support, I no longer aim to get the RPi3B+ to be a WiFi AP (nor Bluetooth).

I will need to unpick some of the PPPoE and similar features, and plonk the RPi3B+ back with a static address on the public range. I'll have some other updating to do to get back on course to replace the RPi2.

(My alternate off-grid power-supply for the RPi3B+ has not turned up from RS, and turns out still to be on back order, and with no good alternative in sight. Ho hum...)

Reset

The first thing to do is rip out the WiFi, dnsmasq, and PPPoE, and select and set a unique static IP address, while sat in front of the TV while the kids are at school. Then I can plug it into the router (and be mains-powered ... hush) while I tinker. Plenty of spare solar PV on-grid to paper over that sin temporarily.

I've created a DNS entry for it too, which should help.

  • (Done) sudo apt-get remove dnsmasq
  • (Done) sudo apt-get remove hostapd
  • (Done) Simpify networking and set static IP.
  • (Done) Re-enable DHCP client on RPi3B+: sudo systemctl enable dhcpcd.service, temporarily.
  • (Done) Stop IP forwarding: remove net.ipv4.ip_forward=1 to /etc/sysctl.conf.
  • (Done) Trimmed PPP-driven /etc/iprules/rules.v4 to rebuild.
  • (Done) Unconfigure IP forwarding: remove net.ipv4.ip_forward=1 from /etc/sysctl.conf.
  • (Done) Fix IP filter for RPi3B+ use as leaf (copied config from green).
  • Re-enable WiFi as client (with DCHP for it), temporarily.
  • Power-down WiFi, at least temporarily.

2019-07-14: Kitchen Cabinet

I managed to bring together the RPi3B+, a wired connecton, and the TV, by virtue of moving everything into the kitchen while the family was out!

Connectivity to the outside world immediately worked, and so an immediate sudo apt-get update && sudo apt-get dist-upgrade was done for security.

(I installed mediainfo and avconv (libavi-tools) as I'll want them soon.)

Upon reboot (to make dbus happy) I saw that networking didn't restart happily, and was told to run systemctl status networking.service for more information. It looks like /sbin/ifup -a --read-environment is unhappy, possibly with my mauling of /etc/network/interfaces. And attempting to ssh in from my Mac is rejected, by sshd.

Commenting out anything to do with wlan0 from /etc/network/interfaces made networking happy on another reboot.

Boot is still showing an error: Failed to start dhcpcd on all interfaces, and I am invited to run systemctl status dhcpcd.service for details. As it happens, the 'details' are not helpful.

After some messing around, and discovering a key file that I had mis-named, I was able to set-up key-only ssh access again. So the kitchen setup can be dismantled, and I can work remotely (until I muck up networking again!).

I have applied a copy of the iptables config used on green, which should make things a little safer to leave connected.

HTTPS Warmup

Ready to support https on Apache with LetsEncrypt, I'm installing certbot starting with sudo apt-get install certbot python-certbot-apache.

I enable and start Apache to get its bland "It works!" page.

Running the suggested sudo certbot --apache mainly seems to generate errors, and does not seem to make meaningful updates to the Apache config files.

(I have had to remember to allow port 443 in the iptables config too!)

The 'manual' method certbot run -a webroot -i apache -w /var/www/html -d example.com seemed however to do the trick, and I have an https server running! It is accessible locally, and remotely to (at least) Google.

In future, to handle multiple sub-domains, the certbot --expand ... and certbot certonly --cert-name example.com ... commands look useful.

2020-06-11: Dead, Or Pining for the Fjords?

Circa 25th May, attempting to reboot the RPi3B+ for a periodic update, it refused to come up (didn't bring the network interface up).

I could attempt to recover the unit, eg by hooking it up to the TV, etc.

But given the mess that I probably made of the OS, it seems likely a better use of time to snap up a RPi4B with 64-bit OS on the horizon, and far more memory if required (4GB looks like the current sweet-spot for me). I'd carry across the current 256GB micro SD card.

(Not that I like generating e-waste...)

And a USB-C power cable/connector seems a big improvement over the micro-USB. I just need to ensure that I have at least a REV1.2 RPi4B to avoid the USB-C power snafu.

And the Anker PowerDrive Speed+ 2 car power adapter experience has been good, with fairly low idle power consumption itself. (There's a newer, cheaper, dual USB-C option too.)

A downside is that even with recent firmware updates, the idle power draw is ~2.3W, more than double the RPi2's. So idle consumption would rise from ~24Wh/d to ~56Wh/d, which may be a tricky issue in the depths of the winter. Still less than the SheevaPlug ~4W, however.

Peak consumption is ~6.4W (1280mA) cf 3B+ at 5.1W (980mA) and 2B at 2.1W (400mA) plus any peripherals plugged in over USB, etc. The official supply spec is 15.3W (3A). Datasheet suggests 13W capacity minimum: A good quality 2.5A power supply can be used if downstream USB peripherals consume less than 500mA in total. For this use with no significant USB peripherals I expect power draw ≤10W. Whatever supply and cable is used must therefore manage 10W to 15W comfortably.

RPi3 in clear case on desk

Before splashing the cash, I wanted to see what was wrong with the RPi3B+, eg if it was the SD card, then I'd need to buy a replacement. But ... connecting up to TV, mouse and keyboard the RPi3B+ seems to boot fine, with no obvious complaints in dmesg for example. Odd. I think that getting the Expansion Pi ordered and set up, whether for the RPi3B+ or a RPi4B, is sensible! This makes it worth trying to revive the RPi3B+ build. So some RPi4B elements, such as use of USB-C, and "Expander Pi", may be worth trying on the RPi3B+.

Swapping out the previous Ethernet cable for another made everything work. Swapping the old one back in did too (but I will keep an eye on it). So The RPi3B+ is back in the kitchen cupboard for now, for the next bits of brain surgery and power trimming.

Shopping List for RPi4B

Possible component selection, ie shopping list as of 2020-06-11 ish... (Budget £150 max, preferably under £100.)

RPi4B Shopping List
Part (linked if possible)
* = preferred
! = purchased
£ ex-VAT Supplier Notes
* Raspberry Pi 4 4GB Model B, REV1.2 or newer £44.35 RS
Raspberry Pi 4 4GB Model B, REV1.2 or newer £46.67 Amazon UK
* Case to protect PCB from surface underneath £3.90 RS Would have top operate with lid off. May still not passively cool enough. Could wedge into existing 'Really Useful' container currently used by RPi2B+.
Case to protect PCB from surface underneath and allowing at least one HAT board preferably Could wedge into existing 'Really Useful' container currently used by RPi2B+.
Mains PSU for dev: 5.1VDC 3A £6.35 RS May do without this to avoid more eventual e-waste.
! 12V PSU for off-grid: Anker USB C Car Charger, 48W 2-Port PIQ 3.0 Fast Charger Adapter £16.66 Amazon UK 30W and 18W ports. Good experience with previous Anker.
USB-C power cable with PD ≥~10W: AmazonBasics USB Type-C to USB Type-C 3.1 Gen1 Adapter Charger Cable 5V 3A £8.32 Amazon UK Marginal on power but should be OK. Already have one of these.
* USB-C power cable with PD ≥~10W: USB C to USB C 3.1 Gen2 Cable (3.3ft/1m), NIMASO PD 100W USB Type C to Type C Fast Charge Data Cable £8.32 Amazon UK
RTC Pi RTC £8.25 AB Electronics As for current RPi2B+ setup. Needs CR2032 cell.
ADC-DAC Pi Zero £11.49 AB Electronics Was ADC-DAC Pi for current RPi2B+ setup. From AB email 2020-06-11: The ADC-DAC Pi Zero is the same circuit as our old ADC-DAC Pi board.
! Expander Pi £22.99 AB Electronics Possible all-in one (ie smaller) substitute for RTC Pi RTC + ADC-DAC. Could add Mounting Kit Four Pack (£3.49) for stability, and do without my existing top connector for GPIO using one from this board. That would reduce my wobbly stack of 3 boards with a single one. Needs CR2032 cell. From AB email 2020-06-11: Expander Pi uses the same RTC as our RTC Pi ... The ADC on the Expander Pi is the 8 channel version of the ADC used on the ADC-DAC Pi Zero ... it uses an external voltage reference which by default uses a 4.096V reference included on the board ... (Ordered 2020-06-11 to test / set up with RPi3B+.)
! Satechi USB-C Power Meter Tester £19.16 Amazon UK For monitoring power consumption of RPi4B, and other USB-C devices. (Ordered 2020-06-11.)
(Ordered with AmazonBasics Double Braided Nylon USB Type-C to Micro-B 2.0 Male Cable to allow power of up to 5V 3A to RPi3B+ from USB-C supply.)

2020-06-12: Resuscitation

Today I have the RPi3B+ working again. The problem may have been simply a bad network connection.

So first up has been updating the system with apt-get, and bringing the EOU site copy up to date too.

There are weirdnesses such as AMP files simply not being built, that I'm looking at.

I'm also taking a base-line power measurement now, powering the device off-grid via the Anker (car charger w/ 19.5W max on USB 'A'), the Muker 10VA USB meter, and a short new Molex Male USB A to Male Micro USB B USB Cable for which the data sheet implies 1.8A max on power contacts. At least the Muker and the micro-USB lead fall short of the nominal 2.5A max that is recommended.

The RPi3B+ is able to boot properly, and is connected by wire to the router. Once I have logged in and things are idle, and with wlan0 apparently at least half running, the Muker shows 5.25V and 350mA, so 1.8W.

Running sudo rfkill block all seems to make no difference to the stray wlan0 not power consumption. In fact, after a reboot current draw has risen to 380mA apparently! Note that the HDMI is not turned off in this case, which may well account for the extra 30mA.

Getting a chunk of EOU rebuilt in parallel shows 5.26V and maybe 600mA to 900mA, ~4W, but with the red power light going out for significant stretches, implying that there's more CPU demand than power available. The device in its (DesignSpark, clear) case is also getting quite warm.

% make -k -j4 wpages

(When I designed in a Z80A to a toy robot in the early '80s, 1A at 5V bought about a million 8-bit instructions per second. In the RPi3B+ that same 5W likely does over three orders of magnitude more work.

Subsequently, just building the super-compressed copies of the pages, single-threaded, seems to result in a fairly steady 520mA at 5.26V ie 2.7W, without the red power light dropping out.

% make -k Wpages

Thus, I need a better power arrangement to use more than one CPU. Also, my idle consumption is a bit high at 1.8W, though bearable, with various things such as network LEDs and HDMI still to be forced off.

2020-06-13: Expander Pi And Other Tweaks

Expander Pi HAT board red for Raspberry Pi 3 Model B plus final assembly soldering then in situ 20200613 131050 DHD

The Expander Pi arrived today. (I wasn't expecting it even to be dispatched until Tuesday. AB's customer service remains very good!)

The new board is definitely not going to fit in the DesignSpark case that I have been using. Not without major surgery, anyhow!

The RPi3B+ can still sit in the lowest-third of the case, keeping it from touching the surface underneath while allowing lots of ventilation. This will do while developing and testing. But something to keep dust and fingers out will be needed for permanent return to the kitchen cupboard.

Before any of that, I have the GPIO 40-way connector and battery holder to solder on.

We suggest soldering the 40 pin GPIO connector first and then the battery connector. Soldering the battery connector first will make it difficult to access some of the pins on the GPIO connector.

I've also fitted one of the spacer pillars that I bought from AB, on the other side from the connector. This already makes the board much more mechanically stable, I could probably fit one or two more without snagging the DesignSpark base. However, this is better than the wobbly stack of three on the current RPi2, so I may well not bother.

In future I will need to very carefully solder connections for ADC and GPIO. Whether I shall leave the board in place or remove it while I do that is a poser for risk-balancing future me!

I am assuming that my 12V to USB converter (eg the Anker) has a through path for -ve/0V, else many things such as the local measurement of voltage may get more complicated. Assuming that that's OK, I may yet connect up a spare ADC input the the input side of the Anker, along with nominal 12V, to allow compensation for some float and ground currents.

Bah: looks like my Anker has just failed. Its blue light isn't coming on, and it's not supplying power. Input voltage is about 14V, well within the range that might be experienced in a car. Amazon has issued a refund, and I bought a replacement, alog with the newer 2 x USB-C model.

Equipment intended to be powered by the receptacle needs to account for intermittent contact, and voltages outside the nominal 12 V DC, such as maximum voltage 9–16 V continuously, or maximum voltage of 20 V lasting 1 hour, 24 V lasting 1 minute, and 40 V lasting 400 ms.

Italtronic Modulbox DIN Rail Series For Use With Raspberry Pi B+ has been suggested as an alternative case that could accommodate the Expander Pi. I'm not sure if it will provide enough ventilation, and I'm also not clear if this actually will accommodate the 3B+ given that this says B+ in one place and 3 in another, and I don't know what subtle differences in layout may exist and be important.

I'm trying DesignSpark For Use With Raspberry Pi 3B+, Raspberry Pi and a Pi HAT, Black Raspberry Pi Case (£3.38+VAT), on the basis of its claiming to have space for a HAT expansion board, being partly clear and with light-pipes to make use of whatever LEDs I leave on, and mentioning ventilation!

Raspberry Pi 3 Model B RPi3B in DesignSpark HAT case 20200616 134350 DHD Raspberry Pi 3 Model B RPi3B in DesignSpark HAT case 20200616 150542 DHD

2020-06-16: I received the case and assembling it turns out to be an IQ test that I nearly failed! The case comes in three parts, and snaps together. It has some hook and clip arrangements to hold the Pi3B+ down and the pieces together. I ended up moving the HAT support pillar to the other hole opposite the connector, since I think in the original position it may have been snagging something in the base. Anyhow, it all now fits together with the Expander Pi on top, neatly! I'll have to see if there is enough cooling. Unlike for the clear case, this base is is not perforated with vent holes, though enough may be drawn in over the microSD card, and around the bottom edge of the clear upper section, for example.

Later I'll have to work out how to tidily let a couple of signal cables in and out... There are knockouts in the clear top piece that would probably do. Or even threading through the vent slots in the top... Also, the black end-place over the Ethernet / USB ports can be unclipped and left off, and any cables threaded through there instead, which still leaves the RPi3B+ fairly well protected. Clever case design, and not an expensive item!

RTC

I already have NTP up and running. I have the Expander Pi board with CR2032 fitted.

Now it's time to try the magic line in /boot/config.txt to enable the RTC driver:

dtoverlay=i2c-rtc,ds1307

After a reboot, these appear in dmesg output:

[    5.287934] rtc-ds1307 1-0068: SET TIME!
[    5.291787] rtc-ds1307 1-0068: registered as rtc0

As with the original Pi and now the RPi2 (though I'm not sure if now necessary)...

To ensure that the NTP-maintained system clock is periodically saved to the hardware RTC to stop it drifting, I have created in /etc/cron.daily a file called hwRTC:

% cat /etc/cron.daily/hwRTC
#!/bin/sh
/sbin/hwclock --systohc -D --noadjfile --utc > /tmp/hwClock.log 2>&1

which should ensure once per day that the RTC time is forced to system time.

Having run the script manually and rebooted dmesg now shows:

[    5.158420] rtc-ds1307 1-0068: registered as rtc0

Bluetooth and WiFi off

Executing this (once):

sudo systemctl disable bluetooth
sudo service bluetooth stop
sudo systemctl disable hciuart
sudo service  hciuart stop

And adding this to /boot/config.txt:

dtoverlay=pi3-disable-wifi,pi3-disable-bt

and rebooting, should save a little power, an attack surface, and boot time. I can at least see that wlan0 has gone away.

I am enjoying the fast reboot (c/o systemd), by the way.

Ether power

Adding this to /boot/config.txt:

# 3B+ only:
dtparam=eth_max_speed=100
dtparam=eth_led0=14
dtparam=eth_led1=14

limits Ethernet speed (to 100Mbps) and turns off port LEDs to save power.

Watchdog

The kernel watchdog maybe helpful to reduce failures that happen when I'm not physically present to power-cycle the system, though network failures are far more common than any other mode.

Although it seems that the watchdog should be on be default since the Pi 3, I added to /boot/config.txt:

dtparam=watchdog=on
And in /etc/systemd/system.conf I set
RuntimeWatchdogSec to 30s.
Nominally I then run sudo systemctl daemon-reload,
but instead I reboot to get the config parameter read.

Probably not great, but now I see:

% sudo wdctl
wdctl: /dev/watchdog: watchdog already in use, terminating.
wdctl: cannot open /dev/watchdog: Device or resource busy

I'll wait and see if anything breaks...

2020-06-16: Good, Bad, and Cooling

I successfully put together the (simple) 3D jigsaw puzzle that is the new Pi case.

My USB-C to micro-USB cable turned up in the evening, and I intended to try powering the RPi3B+ via it.

Though they had both been working when they arrived, both Anker car chargers failed almost immediately that I plugged them in to 12V to use them. They are rated at 12/24V in, and battery voltages are a little over 13V, so benign compared to an automotive environment with an alternator running, for example. I cannot see what I'm doing wrong here. I am very disappointed.

To try to salvage some progress from this evening, I've plugged the RPi3B+ into its mains supply, and that into my mains power meter, which is showing ~3.4W as I write. I believe that the RPi3B+ itself is probably using ~1.8W, so ~1.6W is apparently being wasted in the mains adapter. That seems a little high.

Doing a rebuild of EOU (make -k -j4 pages) pushes power demand up, fluctuating between about 5W and 6W,

After a few minutes flat out the RPi3B+ feels quite warm on top and underneath, and power demand is ~5.2W +/- 0.1W, so the system may be thermally throttling.

Exposed metal parts, eg the Ethernet and USB connector shells, are feeling hot.

I'm trying standing the case on its long edge (red LED and micro-USB top) to see if that helps, but there is no obvious change in power draw for example. This arrangement is known to help with cooling.

Taking the top case off (leaving the case on its side) seems to allow power demand to peak at 5.9W, mainly hovering at a little below that (~5.8W, with dips to 5.0W).

I'm uncomfortable about keeping either the CR2032 cell or the microSD card hot. In both cases I would expect reliability and lifetime to suffer. Maybe I could turn the thermal throttling threshold temperature down to avoid this potential nasty side effects? (Likely to be temp_limit in /boot/config.txt.)

% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=1200000000
temp=73.6'C
throttled=0xa0008

Pausing (with ^Z) and after a few seconds the temperature drops, and the throttled value changes:

% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=600000000
temp=64.5'C
throttled=0xa0000

Resuming (fg) and temperature seems to rise again maybe 1°C/s util getting to a little over 70°C again.

Running make -k pages, ie nominally on a single CPU, is getting power draw in the range 3.5W to 4.9W, with lower temperature but still some throttling it seems:

% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=1200000000
temp=62.8'C
throttled=0xa0008

And when it's all finished and back to idle, consumption is 3.4W, and the temperature is hovering around the low 50°Cs (with the case still on its side and the lid off).

% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=600000000
temp=52.6'C
throttled=0xa0000

Putting the case down flat and the lid back on, and leaving the device idle (room temperature is ~25°C), the CPU temperature wobbles around 51°C.

A large rsync over ssh is pushing the temperature a little above 55°C.

According the RPi folks themselves:

For Raspberry Pi 3 Model B+, the PCB technology has been changed to provide better heat dissipation and increased thermal mass. In addition, a soft temperature limit has been introduced, with the goal of maximising the time for which a device can "sprint" before reaching the hard limit at 85°C. When the soft limit is reached, the clock speed is reduced from 1.4GHz to 1.2GHz, and the operating voltage is reduced slightly. This reduces the rate of temperature increase: we trade a short period at 1.4GHz for a longer period at 1.2GHz. By default, the soft limit is 60°C, and this can be changed via the temp_soft_limit setting in config.txt.

Maybe I could knock temp_soft_limit down to 55°C an even temp_limit down to (say) 60°C to keep the whole system cooler at the expense of sustained performance. I can potentially also have various tasks such as the Gallery and EOU builds sensitive to temperature as well as battery SoC and CPU load. (According to Overclocking options in config.txt for temp_limit Overheat protection. This sets the clocks and voltages to default when the SoC reaches this value in Celsius. Values over 85 are clamped to 85, and for emp_soft_limit 3A+/3B+ only. CPU speed throttle control. This sets the temperature at which the CPU clock speed throttling system activates. At this temperature, the clock speed is reduced from 1400Mhz to 1200Mhz. Defaults to 60, can be raised to a maximum of 70, but this may cause instability.

I am going to try:

temp_soft_limit=58
temp_limit=70

With make -k -j4 pages temperatures seemed to peak at just over 70°C, but more interesting down-clocking is happening, and power consumption is being held at 4.0/4.1W:

% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=1040108000
temp=66.6'C
throttled=0x20002
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=933000000
temp=68.8'C
throttled=0x20002
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=717000000
temp=69.8'C
throttled=0x20002
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=610000000
temp=70.4'C
throttled=0x20002
...
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=610000000
temp=71.4'C
throttled=0x60002
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=502000000
temp=70.9'C
throttled=0x60006

Taking the lid off seems to allow more time up at 4.8/4.9W, with brief drops to 4.0W. Removing the GPIO and little circular lid knock-outs and replacing the lid gets about 50% at 4.9W, the rest at 4.0W, so less good. Lid on, case on its long side (blocking the removed GPIO knockout) is between the two, interestingly. That position also avoids the HAT (and CR2032) sitting right above the CPU.

Note that -j3 seems to get about the same power consumption as -j4, but -j2 is noticeably less.

So, given that my RPi2B+ spends much of its time running at 900MHz, and under 50°C and under 1W, and gains no memory space (still at 1GB), and is still 32-bit ARM 7, will the RPi3B+ actually be faster/better?

(The Raspberry Pi 3 Model B+ idles 6.1&C warmer than the Pi Model 2.)

The main motivations would now seem to be ensuring that security is kept up to date, and having a bigger SD card for growth.

Maybe I need to automate the system setup process as far as possible, keyed on target hardware type, and driven by repo-hosted versions of parameterised config for such things as sendmail and Apache and the Gallery, so that I can near-trivially spin up a secondary clone of my current working system, with nominally only a few manual changes and some tidy-up to cut over. At least I coud get all the basics right, such as power-efficient config and all the tools I need, starting from a bare OS image, or updating a partial image.

Maybe Ansible is how I should be doing that? Note that Ansible only needs installing on the device that is doing the controlling, it does not need to be installed on the devices receiving the instructions..

Maybe I should go back to a RPi3B (no plus, £28+VAT at RS) to get idle power back to ~1.4W, lower nominal peak network performance (but that I am having to disable anyway), a little extra CPU oomph (900MHz to 1200MHz) at lower peak power demand (~3.7W)? Most of the work that I've done on the RPi3B+ should carry straight over to the RPi3B, with plus-only features such as temp_soft_limit simply being ignored.

(poweroff brings consumption down to 1.1W at the wall, and unplugging the RPi3B+ supply brings that down to 0.0W.)

2020-06-18: Downgrade to 3B?

Most of the extra 0.5W of idle consumption of the 3B+ over the 3B seems to be powering super-fast Ethernet and USB that I will not use.

Today arrived a plain old RPi3B, not '+', that I ordered from RS.

I commented out some only-3B+ entries in /boot/config.txt, powered down the RPi3B+, moved the SD card to the RPi3B, moved the HAT onto the RPi3B, and moved the RPi3B into the HAT case. I powered up and Hey Presto, it all works! Amazing.

Having rested for a little while after booting, the case still on its side, and with an ambient temperature somewhere around 23°C or above, I see:

sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=600062000
temp=48.3'C
throttled=0x0

Thus, an idling CPU at a little under 50°C, several degrees cooler than the RPi3B+.

I've left in the temp_limit=70, so what happens when I run up a stressy make -k -j4 pages on the RPi3B?

%: sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=1200000000
temp=53.2'C
throttled=0x0
...
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=1148000000
temp=65.5'C
throttled=0x20002
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=1094000000
temp=66.6'C
throttled=0x20002
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=986000000
temp=66.6'C
throttled=0x20002
...
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=933000000
temp=68.2'C
throttled=0x20002
...
sencha[dhd]: sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=933000000
temp=68.2'C
throttled=0x20002
sencha[dhd]: sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=825000000
temp=69.3'C
throttled=0x20002
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=825000000
temp=69.3'C
throttled=0x20002
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=825000000
temp=68.2'C
throttled=0x20002
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=825000000
temp=68.8'C
throttled=0x20002
...
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=933000000
temp=67.1'C
throttled=0x20002
...
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=610000000
temp=71.4'C
throttled=0x60002
...
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=556000000
temp=70.9'C
throttled=0x60006
...
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=502000000
temp=71.4'C
throttled=0x60006
...
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=771000000
temp=69.3'C
throttled=0x60002
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=825000000
temp=69.3'C
throttled=0x60002
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=771000000
temp=68.8'C
throttled=0x60002
...
... and later after a reboot and a wait ...
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=600000000
temp=51.0'C
throttled=0x20000
...
% sudo sh -c "vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled"
frequency(45)=600000000
temp=50.5'C
throttled=0x60000

(Note that the RPi2B+ sitting on the same shelf is idling at the same 50.5°C as I write. The RPi2B+ reports its CPU speed as 900MHz, so while the RPi3B runs faster than that then it will be ahead on performance.)

Temperature managed to sneak over 70°C, but not by much. I did manage to catch the CPUs fully thottling down to under 600MHz when busy, though that seemed to have taken longer to get to than with the RPi3B+ . And at no point have any of the case or connector shells (etc) felt hot, just warm. So likely this RPi3B will deliver as much throughput as the RPi3B+ when passively cooled and in the current case, and with a saved ~0.5W. Interesting...

2020-06-19: Planning

Basic plan up until 26th:

  • Have all applications installed, if not necessarily running (eg including the mail subsystem).
  • Have GPIO, ADC, DACs and ground all tested, and connections running eg via ribbon cable to breakout box:
    • Ground, but probably not 5V or 3V3 power.
    • GPIOs for dump-load control, study-desk light control, at least one spare.
    • ADCs in for B1 and B2 12V local, nominal 0V local (biased to provide differential floating ground reference)?, and maybe a spare.
    • Maybe a DAC output line or two (including room light sensor input to disable mood light when dark?).
  • Power from off-grid in parallel with existing Pi2B+.
  • Start moving some services across, eg http://www2.exnet.com and https:.

Incrementally move services across, start collecting data on new system, etc, with aim of switching RPi2B+ to become mostly-off staging server with refreshed OS by end July. Actually provide power switch to allow that!

Fantasy Timetable 1
Deliverable Date Notes
Prep for www2.exnet.com on sencha. 2020-06-22 Push down DNS TTL and retry times down to small values. Learn more Ansible.
Move RPi3B to off-grid supply.
www2.exnet.com on sencha.
2020-06-23 This should be downstream of the solar controller for consumption monitoring, and possibly some protection from nasty raw battery world. Do this now so that reliability problems are exposed early.
Bah! Powering RPi3B via JZK Car Power Converter DC 12V to 5V/ 3A Voltage Converter (idle power ~100mW apparently) and Molex Male USB A to Male Micro USB B USB is causing the red light to blink out periodically when under heavy CPU load, though ≤3W draw. Single-threaded this seems to get to ~4W consumption without much power light blinking.
I now have simple initial cron-tab jobs working, including some data collection (from the Enphase).
With both the RPi2B and the RPi3B off-grid, total consumption is reported to be 1.9W to 3.6W, maybe ~2.0W, of which just under 1W is from the RPi2B. This suggests that the RPi3B is also getting down to ~1W when quiet.
Crontab / system config file management. 2020-06-24 Finally! A micro-USB B (power) cable good enough to avoid the RPi3B power light blinking off (much) even when nominally maxed out with make -j4, though thermally throttling. Power drawn ~3W. All hail the 2.5A-rated "24 AWG wire" Startech Male USB A to Male USB Micro B USB Cable, USB 2.0, 0.5m manufacturer part number USBAUB50CMLA.
Another set of stats (RPi CPU temperature) is now being captured on the RPi3B. It will be added to the stats page shortly. Note that the RPi3B is idling just a fraction under 60°C with the RPi2B next to it idling about 5°C cooler on this hot ~30°C day. (53.7°C vs 50.8°C 2020-06-25T05:28Z, with outside temp ~20°C and room temp ~26°C.)
Able to manage (from multiple fragments) crontabs files for root and others. Able to manage interfaces file also.
www2.exnet.com on sencha. 2020-06-25 Made Apache logs roll weekly, last 3 weeks (1 current, 2 old).
(Even today, with room temp ~28°C and the cupboard hotter, the JZK Car Power Converter is running cool. Good!)
Moved http www2.exnet.com to sencha. Logs apparently under correct 'other vhosts' name and format. This may not hold for https since Let's Encrypt may be overriding a key option in its config included by Apache.
Get https://www2.exnet.com working
Get ADC input working.
2020-06-26 Got https://www2.exnet.com working with single-domain cert and HTTP/2. (Amongst other things, needed explicit Protocols h2 http/1.1 in the www2 config.) Then extended cert with www.exnet.com.
I have established that the JZK power converter seems to pass through OV/GND. So the simplest-possible battery voltage monitoring scheme via ADC is viable. More finesse can be added later, if needed.
Got ADC input working, with basic stripped-down powermng support for new ADC, and starting to log supply/battery voltage.
TO DO:
Possibly make differential input, with identical potential dividers, with one grounded at local analogue ground as now, and other grounded at remote supply ground, to be able to eliminate up to about ~1V of (common-mode) ground shift.
Move amp.EOU and others to RPi3. Make exnet.com 'default' in place of current sencha not-set-up page. Possibly make https://www2.exnet.com available (under own cert?).
Be able to manage etc /etc/sysctl.conf and /etc/rc.local.
ntp1.exnet.com or alternate IP.

Other thoughts:

  • As well as study-desk light, have provision for mood light(s) as large RGB LED (mainly mains powered, maybe also test with OpenTRV radio back-channel) in kitchen, at desk, and on trivial Web page somewhere, updating at appropriate speed, that glows brighter and angrier red as grid imports grow and grid intensity/cost is higher and at peak time. Eg, imports >500W off-peak or >200W peak-time are max red? Run local real-time process to software PWM, but try to let RPi sleep, along with more frequent polling of Enphase to control? Or off-load over WiFi to stand-alone lamp? (See BuildHub thread.)

2020-06-30: CPU Temperature Observations

A little after 10pm today, on a cool day with kitchen temperature ~20°C, recent (fairly idle) CPU temperatures in the RPi2B are:

2020/06/30T19:00Z 46540
2020/06/30T19:15Z 47078
2020/06/30T19:30Z 46540
2020/06/30T19:45Z 46002
2020/06/30T20:00Z 46540
2020/06/30T20:15Z 46540
2020/06/30T20:30Z 46540
2020/06/30T20:45Z 46540
2020/06/30T21:00Z 47615
2020/06/30T21:15Z 48692

and not so much higher (~1--2°C) for the RPi3B:

2020/06/30T19:00Z 49388
2020/06/30T19:15Z 49926
2020/06/30T19:30Z 49388
2020/06/30T19:45Z 49388
2020/06/30T20:00Z 48850
2020/06/30T20:15Z 49388
2020/06/30T20:30Z 49388
2020/06/30T20:45Z 49388
2020/06/30T21:00Z 49388
2020/06/30T21:15Z 49388

2020-08-02: GPIO

I am readying the RPi3B to take over the real battery power management I/O. Tere are some things still to sort out, such as the MODBUS libraries (the Raspbian/Debian distribution is too old, for example).

But one aspect that can be set up is the GPIO to control the dump load, and also my off-grid desk-side light.

Both of these should now go via the Expander PI GPIOs if I want to avoid transferring one of my old HAT boards over. (A hybrid appropach remains possible too, even with flying leads.)

As with the current (old) hand-crafted HAT, I have inserted in-line 100Ω resistors to provide a modicum of protection to the GPIO driver outputs, without affecting their ability to operate the target loads.

To try to need as little hand-crafted (C) code as possible, in /boot/config.txt I am going to try dtoverlay=mcp23017,noints to set up the Expander PI MCP23017.

How To Use A MCP23017 I2C Port Expander With The Raspberry Pi – Part 1 suggests that all of port A can be set to outputs with sudo i2cset -y 1 0x20 0x00 0x00, and GPA0 turned on with sudo i2cset -y 1 0x20 0x14 0x01 and off with sudo i2cset -y 1 0x20 0x14 0x00. GPA0 corresponds to '1' on the Expander Pi 20-way connector, next to the '5V' pad.

For the record, in the far end of the ribbon cable that already takes the ADC input sample from the 12V rail, I have soldered for GPIOs:

  • "GND": yellow
  • "1" (GPA0): green
  • "2" (GPA1): blue
  • "10" (GPB1): purple: desk light (via 100Ω to red signal wire)

After some faffing (so, by 19:30) I have the desk light GPIO and script set up, using GBP1. The current script crudely sets all of GBP0 to GBP8, but is simple. I have also moved across the relevant cron jobs.

(DONE) The next job is to set up powermng on the RPi3B to to able to drive GPA0 to control the dump load, then move dump wiring across. Given that the current supply monitoring tends to read low, that we be a conservative dump control until the (MODBUS from the solar controller). The code can be based on my existing i2c handlers and/or the AB Electronics demo code.

(DONE) In fairly short order should follow the SunnyBeam logging solar throughput. Preferably done after dusk so as not to loose any generation data points.

Some careful stitching together of log files from old and new will be needed, and there will be gaps.

2020-08-20: 5V for Dump Load and Full Ribbon I/O

I'm using the final (grey) free wire on the ribbon for +5V to power the dump load driver and relay.

The full set emerging from the RPi3B on the signal (input) side is:

  • "GND": black
  • "ADC1" 12V supply sense: white

The output set emerging from the RPi3B is:

  • "GND": yellow
  • "1" (GPA0): green: dump load control
  • "2" (GPA1): blue: currently unused
  • "10" (GPB1): purple: desk light (via 100Ω to red signal wire)
  • "5V": grey: dump load driver/relay power

2020-08-21: libmodbus Redux

Nearly there to move all the remaining powermng functions across to the RPi3...

I talk MODBUS over USB/serial from the RPi2 to my main solar controller. Though there is a libmodbus package available in the distribution I have on the RPi3 (stretch) it's a bit old at 3.0.6. I can compile against it, but at run time I can't create a 'context', possibly because of device filename limits. Possibly, however, it may just be because I don't have the appropriate USB device plugged in!

(Documentation on libmodbus.)

Recalling what I did four years ago, I could build from https://github.com/stephane/libmodbus/.

When I plug the USB connection for the MODBUS connection into the RPi3B then the /dev/serial/ tree appears, including the expected /dev/serial/by-id/usb-FTDI_UT232R_FTXIEVAK-if00-port0. But I cannot create the MODBUS context in mbaccess.cpp.

If I replace the robust long name with the short name of the USB device (where it happens to be shown this time) as below, all works:

//const char *DEFAULT_MODBUS_DEV = "/dev/serial/by-id/usb-FTDI_UT232R_FTXIEVAK-if00-port0";
const char *DEFAULT_MODBUS_DEV = "/dev/ttyUSB0";

So trying for the newer MODBUS code seems like a good idea!

I have downloaded v3.1.6 from github. I am following the build and installation instructions.

Amongst other things, this installs /usr/local/lib/libmodbus.so.5.1.0.

It turns out that the older (Raspbian/Debian) packages need to be removed to allow v3.1.6 to work properly, so they come off the system packages list for now.

It seems that once that was done powermng didn't even need to be rebuilt or reinstalled for everything to "just work". The magic of shared libraries and stable ABIs!

So as of 12:40:06Z, full power management is running on the RPi3B.

2020/08/21T12:40:06Z AL 921 B1 13575 B2 -1 P 7643 BV 13390 ST H D h A1P 12532 B1T 23 UC 100

The SunnyBeam stuff has been moved over after dusk. I have moved the OpenTRV log capture over (with my hand forced by pulling the wrong USB conenction!). I have moved EOU's 'www' pages over to the RPi3B host, and enabled https. I have still to carefully stitch/build/move some log files as appropriate. preferably before midnight.

2021-03-21: Ansible

I've been using Ansible for some months now to manage the RPi3 from my desk. It's not perfect, but it's better than hacking everything in with vi...

I'm also now managing my off-site secondary through a parallel set of Ansible scripts, and that is good too! Combining them (and maybe an additional on-site RPi secondary too) might be a clever thing to do, sharing the common definitons.

2022-03-03: The Great Raspberry Pi Appeal

From Matt at the Carbon Coop in Manchester arrived the following plea:

... Carbon Coop needs your help. As most of you know we're collaborating with energy co-operatives around Europe to develop a community-led virtual power plant providing demand flexibility to the grid. At the heart of this system is a home energy management system based around the Raspberry Pi 3B+, however due to global supply issues we're having significant problems sourcing this model.

It occurred to me that many of you have upgraded to the latest Pi or migrated to different boards and may therefore have a 3B+ or several sitting around in a drawer (I know I did). If that is the case we'd love to buy them off you - if you can help please contact [Matt].

So my idle 3B+ has been dispatched to Manchester and a new productive life, FOC.