Earth Notes: On Setting Up a Raspberry Pi 2 as Off-grid Server (2015)

The server uses less than 2W, even though I do some slightly inefficient things at the moment that prevent the CPU sleeping, and I have some loads such as the Sunny Beam on USB that I could trim. (Typical load including all these extras is reported overnight as under 1W, by the solar controller, as of 2016Q3 for example.)

Dirty Secret

My dirty secret is that I still have servers elsewhere on the Internet carrying some of the load from a busy site or two, plus DNS etc, including CDN services.

(I also have DNS secondary servers which are usefully not near my primary, but they don't need a lot of oomph or carbon footprint.)

Early 2015 I turned off a power-hungry server, a London-based 24-thread Sun Niagara "CoolThreads" box, and threw all the load over to its US-hosted sibling (x64 based though).

(I also maintained a small Australian virtual server for experimental reasons, for testing some of my distributed algorithms/code.)

Having upgraded my home/office Internet connection from ADSL (~1Mbps up) to FTTC (~10Mbps up or better), there is a prospect of bringing more load home, and killing off more carbon-emitting server power elsewhere in the world, but only if I can beef up my off-grid processing power at home since although the RPi can handle the primary services they are lightweight or can get away with being a little slow.

With the maybe 6x speed boost of the RPi2 over my existing RPi, my parallelism-friendly systems might just hack it.

Product Specs

I had ordered this (Model B, 1GB) RPi2, from Farnell:

• RASPBERRYPI-2-MODB-1GB (subsequently redirected to a more generic landing page...)

As of mid-October 2015 I still had to find a >128GB micro SD card for it if I am to test some of my theories, but hopefully I can do that soon!

Forced Upgrade!

2016-05-22: in the process of rearranging my off-grid system I seem to have bust the RPi B+ (it will no longer power up) so I am trying its SD card in the RPi2 B that I have to hand, and miraculously it just works!.

I have changed the overclocking settings to the Pi2 preset: Pi2 1000MHz ARM, 500MHz core, 500MHz SDRAM, 2 overvolt, though I note suggestions that this overclock may not be stable.

The system seems somewhat faster, though not hugely so. It may be helpful to allow a little more memory to key applications now that 1GB is available.

Stability

I'm seeing Java 8 crash in server mode, so I'm wondering if this is stability problem, though I am tweaking other parameters such as stack size to check.

This suggests that the 'Pi2' overclock RAM setting may be slightly too fast while "Certain divisor relationships between CPU clock and core (L2 cache) clock (such as 2:1) seem to enhance stability and performance," resulting in suggested settings of:

arm_freq=1000
over_voltage=0
core_freq=500
sdram_freq=483
over_voltage_sdram_p=0
over_voltage_sdram_i=0
over_voltage_sdram_c=0
gpu_mem=256

My preference would be to go back to the default 450MHz RAM speed, and the absolute minimum GPU memory.

I got nervous on seeing more odd behaviour and possible filesystem corruption and commented out all but the minimum CPU frequency parameter, leaving the relevant part looking like this:

# Work with CPU speed governor to save some juice...
initial_turbo=60
force_turbo=0 #turns on frequency scaling
# Scale CPU right back when idle.
arm_freq_min=100

# Pi2 settings; some suggest these may be borderline, especially the RAM.
#arm_freq=1000
#core_freq=500
#sdram_freq=500
#over_voltage=2

which gets me back to the equivalent of arm_freq=900 and sdram_freq=450 as checked with sudo vcgencmd get_config int. The 'core_freq' may be back to a very conservative value (250?).

Java Stability

I am more of the opinion that the Java 8 (server) version that was installed was unstable:

/usr/lib/jvm/jdk-8-oracle-arm-vfp-hflt/bin/java -server -version
java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) Server VM (build 25.0-b70, mixed mode)

so instead I have fetched the latest from the Oracle Web site:

/usr/lib/jvm/jdk1.8.0_91/bin/java -server -version
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) Server VM (build 25.91-b14, mixed mode)

Note that neither support -XX:+TieredCompilation for the RPi2.

2016-06-10: as of today the Java server is stable, as is the RPi2.

ZRAM

To try to keep the OOM killer at bay, and to try to be able to be less twitchy about default stack sizes, etc, I have enabled a small amount of ZRAM space (~128MB) in /etc/rc.local:

# Set up small ZRAM compressing 'swap'.
modprobe zram
echo 128128128 > /sys/block/zram0/disksize
mkswap /dev/zram0
swapon /dev/zram0

Power LED

2017-08-17: following the suggestions in "Controlling PWR and ACT LEDs on the Raspberry Pi" I changed the bright red power LED to come on only when a CPU is busy, so I hope consequently much less when the system is conserving power, with the following in my rc.local:

# Make the RPI2 red power LED reflect CPU activity (so off when idling).
echo cpu3 > /sys/class/leds/led1/trigger

I'm using cpu3 rather than cpu0 in the expectation that CPU 0 may get woken for everything including interrupts, and other cores may get woken only for real user-land activity, and so the LED can spend more time off.

I have now lost my visual reminder (the red LED going off when power is low) that I need to fix the power supply generally...

Further Tweaks

2017-10-09: I am always looking for unnecessary processes and subsystem that I can cull, and having noticed excess getty lines in top, I came across "Disable some useless stuff in Raspbian". I particularly like disabling stuff that also reduces attack surface.

% sudo a2dismod cgid
% sudo update-rc.d triggerhappy disable

In /etc/inittab I commented out the last three getty entries (they could possibly all go), followed by sudo telinit Q:

#4:23:respawn:/sbin/getty 38400 tty4
#5:23:respawn:/sbin/getty 38400 tty5
#6:23:respawn:/sbin/getty 38400 tty6

2017-10-13: I saw a lot of console-kit-daemon threads when running htop with the 'H' toggle to see all user threads within processes. So I followed the suggestion in Can consolekit sefely be removed, when running headless (no GUI)? and ran:

apt-get purge consolekit

If I do need to plug in a console/terminal to recover the system, I don't need X11, just a plain tty.

% sudo apt-get purge consolekit
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  consolekit* policykit-1*
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.
After this operation, 793 kB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 75713 files and directories currently installed.)
Removing policykit-1 ...
Purging configuration files for policykit-1 ...
dpkg: warning: while removing policykit-1, directory
'/etc/polkit-1/localauthority/50-local.d' not empty so not removed
Removing consolekit ...
Purging configuration files for consolekit ...
dpkg: warning: while removing consolekit, directory '/var/log/ConsoleKit' not
empty so not removed
Processing triggers for man-db ...

I also moved /etc/X11/default-display-manager to a .FCS version.

2018-11-24: I backported the following /etc/sysctl.conf setting from my in-progress RPi3 set-up, in case it helps avoid client bufferbloat even without the BBR congestion control also being applied on the RPi3:

net.ipv4.tcp_notsent_lowat = 16384

Note that I first applied it on the fly with:

sudo sysctl -w net.ipv4.tcp_notsent_lowat=16384

Still To Do

• Controlling power to USB ports in software (eg to cut power to the Sunny Boy at night).
• Upgrade of power supply to prevent power light flickering under load.
• Power backup from stack of ~150 'abused' NIMH 'hybrid' AA cells.
• Add a daily root cron task for ionice -c 3 fstrim -v / and other non-memory partitions as per ext3/4 and SD-card wear leveling.
• Periodically run iotop -oPa to check sources of SD card traffic.