Earth Notes: On Setting Up the SheevaPlug Linux 'Embedded' Computer (2009)
Updated 2022-09-21.2014-07-19: superseded by Raspberry Pi Model B+ server drawing under 2W.
2009: I ordered a SheevaPlug. (512MB Flash, 512MB RAM, 1.2GHz ARM CPU, USB and Ethernet, running Ubuntu Linux, in a small 'wallwart' mains-plug package.) My aim was getting my entire main Internet-facing server down to under 10W, using external plug-in low-power external HDD/SSD, I/O board and ADSL/WiFi. Maybe with the whole lot powered from my off-grid PV system even in winter.
The plug arrived 2009-08-17 after a little prompting from me. It uses ~7W from the mains out of the box. (A subsequent order was rather quicker to my door at two weeks.)
The next stage will be to move from the current separate ADSL/WiFi using ~8W, to USB-powered ADSL and WiFi connectivity. That will limit power consumption to well under 5W by USB power specs, possibly nearer 3W total with luck. Linux on the box will be providing any firewalling and routing services needed, and the on-board Ethernet port for wired (non-WiFi) access.
Power
The SheevaPlug contains a 'universal' mains power supply (AC 110--240V, 50--60Hz), and indeed can plug straight into a wall socket as if a power adapter. I would expect it to consume ~5W from the mains when deployed this way, which would be significant saving from the ~20W consumed by the current laptop (when not on solar power, which is most of the time). One potential issue is that there is apparently no 'power fail' signal within the SheevaPlug visible at the Linux level which might allow a graceful shutdown, eg sync the storage and quit.
A longer-term aim is to reduce the drain on the mains to zero by powering this entirely off my small solar off-grid system, (~120Wp with a 12V 40Ah SLA battery, thus ~250Wh storage, as of 2009-09-04.) Even in winter, though possibly with some emergency mains fallback. To this end I purchased 2009-09-04 from Maplin a which provides regulated outputs up to 3A from a DC 12V or DC 24V battery. It also has protection against over- and under- voltage (~11.5V cutoff apparently), and handily for me a voltage and current display. (Though the continuous consumption turns out to be higher than I want, ~1W--2W, since I can't turn off the LCD backlight which is a bit of a waste.) I believe that internally the SheevaPlug uses 5V (or slightly higher). Thus the 5V setting on this unit should be suitable. 2009-10-06 I ordered a Texas Instruments PT78HT205V from RS, which claims to have efficiency of 80%--90%, max 2A (10W) output. But the power into it seems to be still 3.5W even when the system is quiet. What it doesn't have is lead-acid battery-sensitive low-voltage cut-off, so I'm routing it via my solar controller to provide that. With my off-grid system boosted to a little under 200Wp, ~150Wp south-facing, the system seems to be getting enough energy in as of 2009-10-26 to have run uninterrupted off-grid since I switched to the TI regulator, in spite of some very gloomy days. So I have high hopes of making it through mid-winter!
(With no load, ie the SheevaPlug mains supply disconnected from the CPU board, the mains supply itself draws ~0.8W according to my most sensitive meter, thus ~80% efficiency with a 4W load from the plug's innards.)
I've verified [archive, 2010] which Linux build is most power-efficient as of 2009-09-04 with the 'tickless' kernel as I use in my x86 low-power laptop.
Note that as of 2009-09-13 simply plugging in a USB hub causes 10 extra wakeups per second (as seen by powertop) which raises power consumption by ~1W compared to the base ~4 wakeups/second.
When measuring DC power draw at (just over) 5V with my E-flite inline meter with 100mA current resolution, I'm seeing about 2.6W with an idle-ish plug with 8GB SD mounted card and Ethernet, 3.6W with a two-port hub plugged in and 4.3W with a 4-port plug (that gets slightly warm to the touch after a while).
I can also bump up consumption slightly by pulling a page repeatedly from the Apache server that I have installed.
With the external USB digital I/O board and a USB flash drive plugged in consumption is shown as 4.1W, which is very good indeed!
Without the USB stuff, but instead running a busy loop in the shell:
while true; do true; done
consumption goes up to 4.6W, which is still very good! (So going from a fully-idle system to one that wakes up frequently costs ~1W, and then running the CPU flat out eats ~1W more.) Doing a large Java compilation took the consumption to 5.1W in bursts, presumably reflecting the SD card activity (the SD card was warm), and took almost 50x longer than on my Intel Core Duo Linux laptop/server, about 25x of which is through lacking JIT (ie running fully interpreted), and the rest from having a single slower CPU (I parallelise when possible).
Without the USB stuff, and unplugging the Ethernet, seems to reduce consumption by ~0.5W which is roughly what I would expect; a WiFi connection might use a similar amount.
I observe that something like 0.5W is being wasted from continual break-in attempts across the Net on the untuned system. That includes the logging which will also be wearing out my Flash storage! So I've tuned settings a little to save energy and storage life...
2009-09-15 had probably only 2/3rds the insolation of the typical mid-December day here. I have also had some small extra loads on the battery. But there was still plenty of juice left which suggests that we may be fine mid-winter for power. The main Java app idles at about 3W+ (sans USB), with about 30 wakeups/sec. Upgrading to Tomcat 6 with a built-in compiler for JSP pages I can get by with a JRE rather than a full JDK. Tomcat 6.0.20 on top of JRE 1.6 handles Java-1.6-dependent pages fine. (With the Sun [now Oracle] 'embedded' SE JRE, total wakeups are at ~9/sec, uptime 0.02.)
As of 2009-10-08 it takes about 25 minutes and 3Wh for the entire system to settle down to a low-power state, including the heavy start-up processing in the Java/Tomcat app.
USB Power Revisited
Especially with a USB hub in place, it is important to have USB devices suspended when not in use, not only to minimise their own power consumption, but also to minimise (polling/wakeup) activity on any intermediate hubs which not only consume power themselves but also cause lots of CPU wakeups which waste power.
With CONFIG_USB_SUSPEND configured in the kernel then if everything downstream of a hub is suspended then the hub will be too. This is a very good start, but isn't quite enough.
A particular problem is the USB flash drive; if it is mounted then it cannot be suspended else errors may occur.
Using the autofs5 package and a couple of ugly kludges I am able to automatically unmount flash-drive partitions when not in use, and when none of them are mounted, suspend the USB device, saving 1W+ of power.
I have a script that will identify a USB device under /sys/bus/usb/devices/ by manufacturer and product.
For the automounter I use a 'program' (executable) map for the mount point for the USB partition(s) which is run every time that a mount is required. Whenever that program/script is invoked it looks for the USB flash device and sets its power/level to 'auto' (to let the kernel manage it if possible, though in fact 'on' would be the same) then if all is well returns the map entry, thus making the filesystem(s) mountable if required.
Periodically I look for the indirect mount point being empty and then attempt to set the USB device to suspended.
(I've also disabled hald and dbus to avoid any unwanted USB device prodding.)
There is definitely a risk of a dangerous race which may end up suspending a mounted filesystem thus causing damage, so I only do this suspend if we are short of power and after issuing a general 'sync' command. I also have a reasonably long timeout on the automount (>3x the fsck time) so the filesystem should only be unmounted if it now really is quiet. I also do some mutex locking between the 'powerup' and 'powerdown' operations, with the lock from 'powerup' lingering 15s after it returns in order to try to allow time for the mount to complete and be visible. (There's a further optimisation that, after the mount lock is released, an inotifywait is started to wait for a 'delete' on the mount point, and immediately tries a 'powerdown' if one is observed with ~30 minutes; at most one such 'watch' may run at once.)
I could probably improve the safety of this mechanism by, for example, building my own automounter which does the usb suspend/auto setting in a race-free way within the context of its own mutexes.
(Note that because my periodic reading of data from my k8055 interface board seems to wake up the USB flash (and presumably and other USB devices, I actually do the poll for powerdown just after the data read, so that might make co-ordination with a modified automounter more tricky, maybe requiring a filesystem-based lock to co-ordinate.)
With this mechanism in place, idle consumption drops back to ~2.5W, without it nearer ~4W with the 4-port USB hub.
More Detail: Automount, USB Suspend, and Other Horrors
Because the Linux usb-storage cannot suspend plugged-in storage (because for many of them, eg real spinning platters, it would be bad news) just plugging it in via a hub causes everything to be kept awake and chews through an extra ~1.5W (cf 2.5W for the entire rest of the system when fairly quiet) which is more than my off-grid system can spare on a dark day.
I use the automounter (autofs5 package) to act as a gateway to the USB filesystem to mount it when busy and unmount it when not. As it happens I've picked the indirect mount point /auto (all this could do with a heavy dose of sensible renaming and tweaking) to mount the USB filesystem containing the Gallery files, with a symlink from its nominal /rw/galleryDB location. Here's the relevant line from /etc/auto.master:
/auto /etc/auto.auto.sh --timeout=90
This causes the filesystem to get unmounted ~90s after it becomes idle.
auto.auto.sh
Here is the /etc/auto.auto.sh executable map source:
#!/bin/sh if [ "$1" = "galleryDB" ]; then # Attempt to ensure that DT200 is power up, else quit. logger -p daemon.info Powering up galleryDB storage... /etc/suspend_inactive_DT200.sh powerup || exit 1 logger -p daemon.info Mounting galleryDB... echo -fstype=ext3,rw,noatime,commit=600,data=ordered,barrier=1 :/dev/disk/by-label/galleryDB exit 0 fi # Error: unknown key. logger -p daemon.err UNKNOWN KEY $1 for /auto exit 1
This checks that the correct key (mount/directory) is being requested and if so powers up the USB key if necessary (exiting if this fails) and returns the correct ext3 mount options (including noatime,commit=600,data=ordered,barrier=1 for wear-reduction and safety), using the by-label mount for robustness.
After a near-death experience apparently provoked by a bad USB cable/connection (couldn't mount, streams of horrible fsck errors, though apparently little actual corruption) I have done tune2fs -e remount-ro /dev/sda2 to get the filesystem remount read-only if an error is detected to try to avoid causing any further damage but still giving read access which is all that is needed 99% of the time. (I also had to reinstate the journal with the -j option.)
The /etc/suspend_inactive_DT200.sh script locates the USB thumbdrive by manufacturer and product using a home-grown /etc/findUSBdir.sh script thus helping keep everything relatively insensitive to wherever the USB devices happen to be plugged and what hubs are involved, etc. This script can be called with powerup as here, or powerdown to force suspension of the USB device. An attempted powerdown is aborted if /proc/mounts shows anything mounted under /auto. The power-up retains the lock in the background long enough for any fsck and mount to complete. The script uses a filesystem mutex lock to prevent races between instances.
The powerdown is attempted periodically (being vetoed if inappropriate) and in particular after any other USB operation that may wake devices, such as reading from the K8055 USB digital I/O board.
Discussions with Alan Stern of Linux USB fame suggest that this may be the best that can be done with a 2.6.31 kernel.
I have suggested to Ian Kent of autofs5 that automount (the daemon) provide callouts before and after each mount and unmount, in the scope of its internal mutex locks so as to avoid races, where for example I might hang my powerup/powerdown operations. Typically the callout executable might always be passed simple arguments like:
{mount|unmount} {mountpoint} {mountedfile} {before|after|afterfailed}
eg for me I'd look for (to powerup):
mount /auto galleryDB before
and, to powerdown:
unmount /auto galleryDB after
and possible:
mount /auto galleryDB afterfailed
Maybe the callout program could be indicated with a --callout=PATH in /etc/auto.master? This being open source, of course, I can hack the automount daemon myself!
Wake-ups Trimmed
In December 2010 I reviewed the Java apps running on the plug and managed to reduce CPU wakeups caused by them from about 12 or more per second to about 2 or 3 per second total. Multiple software tweaks were needed to do this. Interrupts from incoming network traffic are now clearly the dominant source of wakeups, and a significant portion of that SPAM/malicious.
The system is now hovering at around 25 wakeups per second, though with significant excursions either side of that figure.
LiFePO4 Secondary Storage
In October 2010 I bought at installed a LiFePO4 'backup' battery for my SLA, attempting to roughly double overall storage to about 500Wh or about 5 days.
The system still takes its main view on storage state-of-charge (SoC) and thus available energy reserves primarily from the SLA, but as of January 2010 now also is forced into 'very low power' mode if the LiFePO4 battery seems exhausted regardless of the SLA state. This basically indicates that we've had no decent sunshine for a couple of days as the LiFePO4 battery is discharged preferentially for the server.
See the k8055 poll script as of 20110103.
Weather Forecast and Power System Management
Most of the system power usage control in the system is reactive, ie based on battery voltage(s), though there is some based on time of day and even calendar month (when PV generation is known to be low).
As of 2014-05-31 (see the power-management script) and other than in Nov/Dec/Jan when insolation is generally poor from short day length and cloudy weather, I now use the following day's weather forecast to adjust management thresholds. Unless tomorrow is forecast to be "sunny" in some way then the voltage levels used to establish battery system state-of-charge (SoC) — on a scale of very low, low, OK, good, very good — are bumped up a little to conserve more vigorously and keep more stored on the grounds that the following day is unlikely to replenish any used.
Power To-Do
There are various things that may help minimise power (and system wear):
- Reduce spurious logging other than syslog, eg from Apache.
- Reduce Ethernet connection speed when conserving power, to 10Mbps max, eg:
ethtool -s eth0 autoneg off speed 10to minimise power since nothing much needs go over 10Mbps here, and re-enable full-speed / auto-negotiate with:ethtool -s eth0 autoneg on speed 1000. - [text lost] ie when we think that we might be about to lose power and die.
- Find ways to minimise wake-ups (eg watching powertop stats) and spend more time in C1.
Already done for 2.6.31 kernel:
- Enable CONFIG_USB_SUSPEND which may save 1W or so. May be possible at run time, eg by adding to rc.local to allow all USB to autosuspend (quickly, but not instantly for better stability) something like:
echo 1 > /sys/module/usbcore/parameters/autosuspend for f in /sys/bus/usb/devices/*/power/autosuspend; do if [ -f "$f" ]; then echo 1 > $f; fi; done for f in /sys/bus/usb/devices/*/power/level; do if [ -f "$f" ]; then echo auto > $f; fi; doneNote however that rc.local may not be run at quite the right time to fully reduce consumption on some devices, so we may want to run some of it from cron... - Set noatime and a long commit (a la laptop-mode) on filesystems to minimise storage traffic and thus power consumption (and data=ordered,barrier=1 for better crash-proofing).
- Increase dirty_writeback_centisecs (to >1500) to reduce traffic, and use laptop-mode and other params to further increase I/O burstiness, (preferably matching up with the file-system commit interval) by adding to /etc/sysctl.conf:
vm.laptop_mode = 5 vm.dirty_ratio = 25 vm.dirty_background_ratio = 5 vm.dirty_expire_centisecs = 60000 vm.dirty_writeback_centisecs = 60000 vm.swappiness = 0
Note that we cannot have a dirty_ratio even approaching 50% as we expect the JVM to take up about (the other) 50% of physical memory. We don't want to be swapping at all if possible (and the initial configuration has no swap configured at all) but logically it may be needed in the future. - Reduce spurious logging from syslog, and use the "-" option to avoid syncing to storage on every entry.
- Mounting /tmp on tmpfs to avoid any writes to storage.
- Disabled hal by moving hal to hal.FCS in /etc/init.d and creating in its place a file just containing
exit 0; similarly for dbus, which also may be probing/waking USB devices.
Software
None of the software I use on my main server is CPU-specific, eg I have a mail server, static Web server, and a Java-based Web server, all of which should run my existing setup on top of ARM-based binaries rather than x86-based binaries, most of them standard with the OS.
I have spent quite some time tuning the memory-heavy Java-based Gallery to run with a much smaller (halved) memory footprint and with some tweaks for a single CPU, but it remains to be seen as of 2009-09 if performance will be adequate.
The 'apache2' package is not part of the Sheeva base so I have installed it.
Minimising Memory Overheads
A major challenge of a 512MB machine, especially with zero or minimal swap, is limiting the memory consumed be various tasks and daemons or eliminating them entirely.
(I don't want to be swapping at all, especially not to wear-prone flash, although some processes have much much larger virtual than physical memory use which may force me to set up some 'logical' swap to hopefully never be used.)
Done already:
- Disabled hald and dbus (for other reasons too).
- Tuned apache2 (and Tomcat) to support relatively few connections and in particular few 'spare' servers/connections, which all eat memory, virtual and/or physical.
- Set following system properties for Tomcat to avoid 'leaks':
-Dorg.apache.jasper.runtime.JspFactoryImpl.USE_POOL=false-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=trueand at the end of conf/catalina.properties disable String cacheing with:#tomcat.util.buf.StringCache.byte.enabled=true - Reduced the stack-space (and thus virtual memory) allocated per-apache2-thread from 8MB to 512kB with this inserted in /etc/init.d/apache2 in each of the two places (for start and restart) just before the daemons are started:
# NPTL (modern Linux threads) defaults the thread stack size to the setting # of your stack resource limit. The system-wide default for this is 8MB, # which is waaay exaggerated when running httpd. # 512kB should be more than enough (AIX manages on 96kB, Netware on 64kB). ulimit -s 512
thus reducing VM size from >100MB to maybe 20MB. Similarly for named/bind9. - More to save memory overheads: disable extra getty processed by commenting out the 'start' lines in /etc/event.d/tty3 onwards.
To do:
- Possibly I should set max-cache-size for my BIND/named.
- Possibly I should set the default stack size (soft limit?) system-wide in /etc/security/limits.conf to 512kB rather than the out-of-the-box 8MB.
Firewall, Apache, Mail
I am replicating my firewall, Apache (apache2) and mail (sendmail and qpopper) configuration from my previous server, though with more attention paid to reduced resources such as connections, memory and storage life/writes.
Also some attention is being paid to reduce bandwidth requirements which may reduce power draw and should improve perceived performance, eg by making sure that the Apache 'deflate' module is enabled with:
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css
so that common compressible static data files will be usually be sent in typically half the on-disc number of bytes on the wire, which should result in fewer packets sent and fewer CPU interrupts.
Note in particular that the mail spool directories (/var/spool/{mail,mqueue,mqueue-client}) are displaced via symlinks from the root partition (on the plug's NAND flash) to the external SD card to reduce wear.
Java
Java (1.6) is going to a critical component of the run-time environment, indeed a full JDK to support Tomcat 4 and/or local builds, and some variant of OpenJDK / Zero and Shark is my initial preferred route.
Something that I can apt-get would be best from my point of view, of course... apt-get install default-jdk seems to get a lot of stuff (84MB) including openjdk-6-jdk openjdk-6-jre openjdk-6-jre-headless openjdk-6-jre-lib. Attempting to get open-6-jdk looks much the same, but when I try it first there's an error fetching something, and then when I try again, another 250MB will be required apparently, which is more than the space I have left on NAND! This would be the point at which it would be useful to install Java stuff on external storage, eg SD card.
More modestly I'm now trying apt-get install openjdk-6-jre-headless which claims to need to download 30MB and take up 84MB afterwards, but it also fails with the:
Failed to fetch http://ports.ubuntu.com/pool/main/c/cups/libcups2_1.3.9-17ubuntu3_armel.deb 404 Not Found
I tried again in download-only mode (-d) and then without the -d:
# apt-get install -d openjdk-6-jre-headless # apt-get install openjdk-6-jre-headless
and it all seems to be installing... Bizarre...
java -version java version "1.6.0_0" OpenJDK Runtime Environment (IcedTea6 1.4.1) (6b14-1.4.1-0ubuntu10) OpenJDK Core VM (build 14.0-b08, interpreted mode)
So, an interpreter-only implementation for now, and down to 79M free space in / partition, but progress!
Creating a ~1GB usually-read-only partition on an SD card, and moving /usr/lib/jvm on to that (symlinked from its original location) now allows me to install the JDK (openjdk-6-jdk). And the same treatment is warranted for /usr/share/fonts to free up space in the root.
2009-09-15 Xerxes Rånby kindly sent me a private build of a newer IcedTea (that should be along in Ubuntu "Karmic") that "contains a new assembler optimized ARM interpreter that sits on top of zero that will improve your performance 3x-5x in most cases." I'm seeing probably at least 2x--3x speed-up: fab!
Doing a large Java build including extensive use of ProGuard to optimise, I see the following times:
- Plug (1.2GHz ARMv5 (Kirkwood MV88F6281 / Feroceon 88FR131), Java fast interp only armv4-asm-4.1.2-icedtea6-1.6): total time 210 minutes 54 seconds.
- Linux laptop/server, low-power mode (2x798MHz x86 cores (1 x Intel Core Duo)): total time 7 minutes 55 seconds.
- MacBook (2x2GHz x86 cores (1 x Intel Core 2 Duo), Java JIT): total time 3 minutes 28 seconds.
Thus the Plug is ~30x slower than x86 of similar clock speed...
2009-09-18 I am testing Sun's 'Embedded' SE JRE for ARMv5, which includes a C1 (client) HotSpot implementation, and it is fast enough to bring up a mirror of the Gallery, though is maybe nearly 10x slower than my 2x2GHz x86 MacBook where we might hope for 4x slower on the basis of clockspeed alone ie a gap of maybe 2x difference from the the energy tradeoff, but accomplishes on at ~5W rather than ~50W, so there is much more bang per Joule with the ARM!
Looking at dmesg from one of my x86 mirrors which seems ~10x faster,
CPU0: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 09 Total of 1 processors activated (5992.91 BogoMIPS).
ie 2 'integer' instructions per Hz, whereas for the plug:
Calibrating delay loop... 1192.75 BogoMIPS (lpj=5963776)
at 1 instruction per Hz, so maybe the C1/C2 difference is not that significant, ie Sun's C1 JRE is getting me fairly close to maximum available performance. This inclines me to do a little more hand-optimisation of code in conjunction with post-javac static optimisation (eg by ProGuard) and looking at the Java byte codes produced in order to reduce the amount of work that C1 has to do to optimise well. And I note that the SheevaPlug is only twice as slow on some intensive tasks as my Niagara T1000 1.2GHz 24-core host, so the plug isn't doing badly!
2010-11-12 upgraded JRE to 6u21 version, and over the next few days managed to reduce the virtual memory footprint by ~30MB (mainly by capping the PermGen at 32MB) and reduced the CPU wakeups per second for the entire Tomcat instance from about 12 to about 3 by tweaking both the Web apps running on it. Total wakeups per second for the plug now hovering around 30, mainly from eth0, quite probably ~10 packets per second of SPAM and similar malicious traffic; see this powertop fragment:
| Cn | Avg residency |
| C0 (cpu running) | ( 1.5%) |
| C0 | 6.8ms ( 7.3%) |
| C1 | 43.8ms (91.2%) |
Wakeups-from-idle per second : 31.6 interval: 15.0s Top causes for wakeups: 30.5% ( 23.9) <interrupt> : orion_tick 30.2% ( 23.7) <interrupt> : eth0 11.4% ( 8.9) <kernel core> : hrtimer_start (tick_sched_timer) 10.4% ( 8.1) <kernel core> : hrtimer_start_range_ns (tick_sched_timer) 6.3% ( 4.9) <kernel core> : sk_reset_timer (tcp_keepalive_timer) 2.2% ( 1.7) java : hrtimer_start_range_ns (hrtimer_wakeup) 1.3% ( 1.0) <interrupt> : mvsdio 1.3% ( 1.0) ntpd : hrtimer_start_range_ns (it_real_fn) 1.3% ( 1.0) kmmcd : mmc_rescan (delayed_work_timer_fn) 1.2% ( 0.9) apache2 : hrtimer_start_range_ns (hrtimer_wakeup) 1.1% ( 0.9) automount : hrtimer_start_range_ns (hrtimer_wakeup)
JRE 7u4
2012-05-07 upgraded to JRE 7u4 (thanks Andy Gilbert at Oracle once again):
java version "1.7.0_04-ea" Java(TM) SE Runtime Environment for Embedded (build 1.7.0_04-ea-b20, headless) Java HotSpot(TM) Embedded Client VM (build 23.0-b21, mixed mode)
This is running code built against JDK6, without any hitches at all. (The only problem that I hit was some of my JPEG-related test cases failing when run under OpenJDK7 (u6) due apparently to a long-standing mismatch in colour-space enumeration values between Java and the native JPEG library.) This version remains -client only.
I'm aiming to be able to run an instance of the Gallery on a Raspberry Pi, with 256MB main memory, and I have the app working comfortably as a mirror with 100MB heap (on my development machine's 64-bit ISA). On the ARM master node ~32MB of PermGen and ~12MB of CodeCache is needed too.
Garbage Collection and Inner/Anonymous Classes and Caches
One issue that became apparent with a 256MB heap for my main Java Web app, and that seems not to have been significant with a 512MB heap or bigger, is accidentally retaining garbage through unwanted strong references, especially via anonymous or non-static inner classes. Basically, unless an inner class is 'static' it may bring along all sorts of hidden references eg to large data structures that you want to let go of when short of memory! With the help of jmap (against a running server) and jhat (on my laptop) I've been removing such unwanted strong references and where some sort of reference is necessary, using a weaker reference instead. Also, in general, I've also been eliminating static data structures that might entrap such unwanted strong references.
Another sensitive issue with a small heap is over-greedy caches, so I have been doing a lot of work to monitor and tune those. That has included caches outside of my application, such as in Tomcat 6, that may have been retaining stuff that should otherwise have been GCed.
As ever, the tuning is done so as not to penalise my larger-memory hosts; indeed they should find themselves able to cache other data more productively, or even give some heap space back to the operating system!
Long-term Data Retention
One issue to be thought about longer term is the issue of long-term storage on SSD/Flash (ie non-magnetic, non-optical) media: valuable data such as code repositories and the Gallery content might last indefinitely on magnetic media, but probably at least require explicit refreshing every few years on SSD/Flash.
It will probably be useful to automate off-site backups, eg of non-sensitive data/repositories to another co-lo host.
Initial Access and Set-up
2009-09-09: My primary development machine is currently (2009-09-09) a MacBook, and as I'll need to have access to the plug's serial console for various administrative actions, I'm starting with the [archive, 2009] PlugWiki serial terminal program instructions and I have installed the Prolific PL2303 USB serial adapter driver for Mac OS X. Unfortunately if doesn't seem to work (I can't see the /dev/*PL* devices)! Now I'm trying the alternate FTDI drivers mentioned. Having installed and rebooted I can indeed get to the SheevaPlug console with the following command (note use of the 'B' channel and high baud):
sudo screen /dev/tty.usbserial-000010XDB 115200
Note that the mini-USB connection is not very snug and can fall out. I've also been experiencing 'USB overcurrent' warnings from my MacBook, which has then disconnected the SheevaPlug: a little worrying. So far the lead I use with my digital cameras seems more reliable! I've thrown away the USB lead that came with the SheevaPlug as broken.
I changed the root password for safety when this gets hooked up to my LAN which would make it statically reachable without much firewalling from the Net.
2009-09-10: Support for the SheevaPlug Kirkwood processor cpuidle is now in 2.6.31 of the Linux kernel.
I want to try the new [archive, 2009] V1.0 SheevaPlug installer "intended to replace the current recovery tool for reinstalling or reviving a bricked plug." This should help me get a new tickless/cpuidle kernel in, and allow me to boot quicker from NAND and/or from a (replaceable) SD card. I want to avoid wearing out the internal NAND flash that I cannot replace, and I'm fairly sure that I'll want to install more software (eg Java) than will fit on the NAND, and I'd like to keep the NAND image as a fall-back. The installer instructions firmly state Win32 or Linux as prerequisites, but OS X may be close enough to generic *nx to work... Nope:
**** exec(modprobe ftdi_sio vendor=0x9e88 product=0x9e8f)sh: line 1: modprobe: command not found modprobe ftdi_sio vendor=0x9e88 product=0x9e8f FAILED
Upgrade Kernel In-situ Directly
OK, next approach... Try and batten down the hatches so the plug is safe, then give it a static IP address (no DHCP) and attempt to use the self-running script to upgrade to 2.6.30.6, though with the risk that if I do brick anything then I'll have to use my Linux box to unbrick anyway...
There seem to be no exposed services other than ssh and bootpc, which is good:
root@debian:~# netstat -a Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 *:ssh *:* LISTEN udp 0 0 *:bootpc *:* ...
So I'm changing /etc/network/interfaces from the default:
auto lo iface lo inet loopback address 127.0.0.1 netmask 255.0.0.0 auto eth0 iface eth0 inet dhcp # /usr/share/doc/ifupdown/examples for more information.
to:
auto lo
iface lo inet loopback
address 127.0.0.1
netmask 255.0.0.0
auto eth0
iface eth0 inet static
address 79.135.97.79
netmask 255.255.255.224
gateway 79.135.97.65
to suit my network, and rebooting... (The primary interface address will change in due course, but will do for bootstrapping this system.)
It seems that my reconfig has worked:
eth0 Link encap:Ethernet HWaddr 00:50:43:01:d2:01
inet addr:79.135.97.79 Bcast:79.135.97.95 Mask:255.255.255.224
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:532
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:11
so I'm going to try plugging in my network interface and pinging something!
root@debian:~# ping 79.135.97.65 PING 79.135.97.65 (79.135.97.65) 56(84) bytes of data. 64 bytes from 79.135.97.65: icmp_seq=1 ttl=64 time=6.96 ms ...
Good!
Next I copy over (cut-n-paste) a sensible /etc/resolv.conf for DNS to work...
root@debian:/etc# ping kernel.org PING kernel.org (149.20.20.133) 56(84) bytes of data. 64 bytes from pub1.kernel.org (149.20.20.133): icmp_seq=1 ttl=56 time=154 ms ...
Kewl!
This should be enough to let me load a new kernel, but I'm curious to see if the RTC (real-time clock) will retain its settings when the plug is power-cycled, given that there does seem to be a button cell on the PCB at casual inspection. So I set the system clock with ntpdate and write to the RTC.
root@debian:/etc# ntpdate ntp.exnet.com 11 Sep 15:45:59 ntpdate[1288]: step time server 79.135.97.67 offset 283996801.623850 sec root@debian:/etc# ntpdate ntp.exnet.com 11 Sep 15:46:03 ntpdate[1289]: adjust time server 79.135.97.67 offset 0.000150 sec root@debian:/etc# hwclock -w
Powering down here also gives me a chance to insert my mains plug-in power meter which will give me a crude indication if I do manage to reduce power consumption especially on idle...
root@debian:/etc# poweroff
Broadcast message from root@debian
(/dev/ttyS0) at 15:52 ...
The system is going down for power off NOW!
* Saving the system clock
* Asking all remaining processes to terminate... [ OK ]
* All processes ended within 2 seconds.... [ OK ]
* Deconfiguring network interfaces... [ OK ]
* Unmounting temporary filesystems... [ OK ]
* Deactivating swap... [ OK ]
* Stopping remaining crypto disks... [ OK ]
* Stopping early crypto disks... [ OK ]
* Will now halt
halt: Unable to iterate IDE devices: No such file or directory
System halted
My plug-in power-meter is showing 7W--8W drawn from the mains...
It looks as if time was retained at least across a brief power-down as no huge step was needed this time:
root@debian:~# ntpdate ntp.exnet.com 11 Sep 15:58:50 ntpdate[1245]: adjust time server 79.135.97.67 offset 0.009734 sec
That should save a lot of trouble later (eg with strange file times) and is encouraging as I hope to run my primary NTP server on the plug.
Fetching the New Kernel (2.6.30.6)
See [archive, 2010] Install Prebuilt Kernels From sheeva.with-linux.com.
First I need to get 'wget', for which I'd normally just do:
# apt-get install wget
but that fails with the known problem:
E: Could not open lock file /var/cache/apt/archives/lock - open (2 No such file or directory)
which is ultimately fixed with a:
mkdir -p /var/cache/apt/archives/partial
So now in /tmp I need to do:
root@debian:/tmp# wget http://sheeva.with-linux.com/sheeva/README-2.6.30.6
and make it executable and run it.
Note that there is some human-targeted stuff at the top, actually to be read:
# Watch out for erase/flash errors # If errors are encountered you should redo the flash # # This is a mainline Linux Kernel and you must set # the mainlineLinux and arcNumber env variables in U-Boot # and change the bootargs for a successful boot. # # setenv mainlineLinux yes # setenv arcNumber 2097 ####### change bootargs, replace nand_mtd with orion_nand and add rootfstype=jffs2 # setenv bootargs rootfstype=jffs2 console=ttyS0,115200 mtdparts=orion_nand:0x400000@0x100000(uImage),0x1fb00000@0x500000(rootfs) rw root=/dev/mtdblock1 rw ip=192.168.1.9:192.168.1.4:192.168.1.4:255.255.255.0:DB88FXX81:eth0:none # saveenv ####### change vm security settings # Due to changes in vm security a change must be made in /etc/sysctl.d/10-process-security.conf. # vm.mmap_min_addr should be set to 32768 (This change is safe for any kernel version). # If this is not done it is likely that you will not be able to login remotely. # Although you should still be able to login as root on the main console.
It's been nice knowing you, plug...
root@debian:/tmp# chmod a+x README-2.6.30.6 root@debian:/tmp# ./README-2.6.30.6 Downloading files ...
I attempted a shutdown -r now to restart it but had to power-cycle the plug, and I hit return a few times to get to the Marvel>> prompt. Then:
Marvell>> setenv mainlineLinux yes Marvell>> setenv arcNumber 2097 Marvell>> printenv bootargs bootargs=console=ttyS0,115200 mtdparts=nand_mtd:0x400000@0x100000(uImage),0x1fb00000@0x500000(rootfs) rw root=/dev/mtdblock1 rw ip=10.4.50.4:10.4.50.5:10.4.50.5:255.255.255.0:DB88FXX81:eth0:none Marvell>> setenv bootargs rootfstype=jffs2 console=ttyS0,115200 mtdparts=onion_nand:0x400000@0x100000(uImage),0x1fb00000@0x500000(rootfs) rw root=/dev/mtdblock1 rw ip=10.4.50.4:10.4.50.5:10.4.50.5:255.255.255.0:DB88FXX81:eth0:none Marvell>> saveenv Marvell>> reset
The plug now reboots, loading the image Linux-2.6.30.6, and takes a while... (I have unplugged the network connection as a security precaution.)
Stuck!
Boo hoo, I seem to be stuck:
Reading data from 0x4ff800 -- 100% complete. 4194304 bytes read: OK ## Booting image at 00800000 ... Image Name: Linux-2.6.30.6 Created: 2009-09-10 1:43:29 UTC Image Type: ARM Linux Kernel Image (uncompressed) Data Size: 2626524 Bytes = 2.5 MB Load Address: 00008000 Entry Point: 00008000 Verifying Checksum ... OK OK Starting kernel ... Uncompressing Linux.......................................................................................................................................................................... done, booting the kernel
Discovered a typo in my bootargs above (removed from above transcript), fixed it, did a saveenv and reset, and I get a little further (the kernel starts but can't mount its /root):
... jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00280024: 0xe384 instead Further such events for this erase block will not be printed Cowardly refusing to erase blocks on filesystem with no valid JFFS2 nodes empty_blocks 0, bad_blocks 0, c->nr_blocks 32 VFS: Cannot open root device "mtdblock1" or unknown-block(31,1) Please append a correct "root=" boot option; here are the available partitions: 1f00 1024 mtdblock0 (driver?) 1f01 4096 mtdblock1 (driver?) 1f02 519168 mtdblock2 (driver?) Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(31,1) ...
Unbricking: Trying again from my Linux Server/Laptop
So I think this needs rescuing from another Linux environment. Happily my Ubuntu Feisty/Gusty system has the right driver for the plug console, and aside from the slight untidiness of an ssh into the Linux box (which ultimately this plug should be replacing) and then a 'screen' to get to the plug:
screen /dev/ttyUSB2 115200
this seems fine. (I have to be careful not to disturb the X-10 home-automation USB0 connection, so I'd better ensure that the plug console is disconnected if/when I reboot the Linux laptop, which should be fine for a temporary arrangement such as this.)
The installer's instructions are currently a little bit broken, so I'm [archive, 2009] slightly in the dark, in particular:
Other notable errors, are the part about editing the MAC address for the plug, the file no longer resides there, and you have to go hunting for 2 files with totally different filenames...
So just for starters I'm changing the 'ethaddr' value in the sheevaplug-installer-v1.0/uboot/uboot-env/uboot-*-custom.txt files to match my plug (only the last two bytes needed changing).
(I had to update my /etc/apt/sources.list entries from archive.ubuntu.com to old-releases.ubuntu.com to be able to apt-get install php5-cli; having the runme script a simple sh (Bourne Shell) script might have made life a little easier here!)
Since I've now messed up the NAND flash I might as well make it bootable, so I'll target 'nand' rather than 'mmc' with this image.
# php5 runme.php nandbut that gets me:
... openocd/openocd: error while loading shared libraries: libftdi.so.1: cannot open shared object file: No such file or directory ...So, looks like I'll need to [archive, 2009] build and install libftdi, giving me:
-rw-r--r-- 1 root root 50172 2009-09-12 13:10 /usr/lib/libftdi.a -rwxr-xr-x 1 root root 819 2009-09-12 13:10 /usr/lib/libftdi.la* lrwxrwxrwx 1 root root 17 2009-09-12 13:10 /usr/lib/libftdi.so -> libftdi.so.1.16.0* lrwxrwxrwx 1 root root 17 2009-09-12 13:10 /usr/lib/libftdi.so.1 -> libftdi.so.1.16.0* -rwxr-xr-x 1 root root 50306 2009-09-12 13:10 /usr/lib/libftdi.so.1.16.0*So now I have a new error:
$URL: http://svn.berlios.de/svnroot/repos/openocd/tags/openocd-0.2.0/src/openocd.c $ For bug reports, read http://svn.berlios.de/svnroot/repos/openocd/trunk/BUGS 2000 kHz jtag_nsrst_delay: 200 jtag_ntrst_delay: 200 dcc downloads are enabled Error: unable to open ftdi device: device not found Runtime error, file "command.c", line 469:So I may have to edit uboot/openocd/config/interface/sheevaplug.cfg for my [archive, 2009] 'newer plug' changing its content to:
interface ft2232 ft2232_layout sheevaplug ft2232_vid_pid 0x0403 0x6010 #ft2232_vid_pid 0x9e88 0x9e8f #ft2232_device_desc "SheevaPlug JTAGKey FT2232D B" jtag_khz 2000And ... oh ... things are happening:
Open On-Chip Debugger 0.2.0 (2009-07-26-14:56) Release $URL: http://svn.berlios.de/svnroot/repos/openocd/tags/openocd-0.2.0/src/openocd.c $ For bug reports, read http://svn.berlios.de/svnroot/repos/openocd/trunk/BUGS 2000 kHz jtag_nsrst_delay: 200 jtag_ntrst_delay: 200 dcc downloads are enabled Info : JTAG tap: feroceon.cpu tap/device found: 0x20a023d3 (mfg: 0x1e9, part: 0x0a02, ver: 0x2) Info : JTAG Tap/device matched target state: halted target halted in ARM state due to debug-request, current mode: Supervisor cpsr: 0x000000d3 pc: 0xffff0000 MMU: disabled, D-Cache: disabled, I-Cache: disabled 0 0 1 0: 00052078 NAND flash device 'NAND 512MiB 3,3V 8-bit' found successfully erased blocks 5 to 6 on NAND flash device 'NAND 512MiB 3,3V 8-bit' wrote file uboot-env.bin to NAND flash 0 up to offset 0x000c0000 in 19.210085s target state: halted target halted in ARM state due to debug-request, current mode: Supervisor cpsr: 0x000000d3 pc: 0xffff0000 MMU: disabled, D-Cache: disabled, I-Cache: disabled 0 0 1 0: 00052078 NAND flash device 'NAND 512MiB 3,3V 8-bit' found successfully erased blocks 0 to 4 on NAND flash device 'NAND 512MiB 3,3V 8-bit' wrote file uboot.bin to NAND flash 0 up to offset 0x00074000 in 69.370750s **** U-boot should be up and running now. Open your console ...
On reconnecting to the console I can see a kernel booting, mounting the USB stick, attempting to erase and write the NAND flash, etc, but then I get:
**** Mounting /dev/sda1 UBIFS: default file-system created UBIFS: mounted UBI device 0, volume 0, name "rootfs" UBIFS: file system size: 515321856 bytes (503244 KiB, 491 MiB, 3994 LEBs) UBIFS: journal size: 25804800 bytes (25200 KiB, 24 MiB, 200 LEBs) UBIFS: media format: w4/r0 (latest is w4/r0) UBIFS: default compressor: lzo UBIFS: reserved for root: 4952683 bytes (4836 KiB) ****** ERROR - no rootfs.tar.gz file found on USB stick Please press Enter to activate this console.
Another whoopsie, but closer I think... I can see, under /mnt/usb, the following:
modules.tar readme.txt rootfs.tar ubuntu-s.sh uimage
so I'm wondering if the .gz suffix was nuked by the FAT filesystem? And a restart doesn't far...
... UBIFS: file system size: 515321856 bytes (503244 KiB, 491 MiB, 3994 LEBs) UBIFS: journal size: 25804800 bytes (25200 KiB, 24 MiB, 200 LEBs) UBIFS: media format: w4/r0 (latest is w4/r0) UBIFS: default compressor: lzo UBIFS: reserved for root: 4952683 bytes (4836 KiB) VFS: Mounted root (ubifs filesystem) on device 253:1. Freeing init memory: 140K Warning: unable to open an initial console. Kernel panic - not syncing: No init found. Try passing init= option to kernel. [<c00309c0>] (unwind_backtrace+0x0/0xe0) from [<c003db50>] (panic+0x50/0x120) [<c003db50>] (panic+0x50/0x120) from [<c002b53c>] (init_post+0xd4/0xfc) [<c002b53c>] (init_post+0xd4/0xfc) from [<c0008738>] (kernel_init+0xc4/0xec)> [<c0008738>] (kernel_init+0xc4/0xec) from [<c0040b3c>] (do_exit+0x0/0x6ac)
I had copied the files on Linux having mounted my USB stick as 'msdos'; now I'm recopying the files from my Mac (!) to preserve the full name. (This will test my USB stick's wear-levelling algorithm...) And now I'm attempting the runme stuff again, which seems to be working, and having connected to the console, we seem to be getting further, eg:
... **** Copying root filesystem. This will take few minutes
And I'm in and unbricked! Hurrah! So I know I can do this again, at least until I wear something out. The plug is now loaded an older kernel (without the cpuidle stuff that I want), so I'll need to go back a few steps to get the latest...
root@ubuntu:~# uname -a Linux ubuntu 2.6.30.2 #11 PREEMPT Wed Jul 22 19:53:31 MDT 2009 armv5tel GNU/Linux
FWIW, power draw is showing as 6W from mains at the moment.
Getting 2.6.30.6 Again
Again I check with 'netstat -a' that no dangerous services are exposed, update /etc/network/interfaces to work nicely on my LAN, put some sensible entries in /etc/resolv.conf, change the root password, reboot, and attempt to connect to the outside world (eg ping, ntpdate): success!
So now I need to 'apt-get install wget', without any messing around needed, then 'wget http://sheeva.with-linux.com/sheeva/README-2.6.30.6' (in /tmp), set it executable and run it. (Note that there is currently no /etc/sysctl.d/10-process-security.conf...)
The README-2.6.30.6 file was updated 2009-09-11 and gained a new --rootkernel option and failed when $1 wasn't defined. I've manually moved sheeva-2.6.30.6-uImage into /boot as uImage, moving the old uImage aside.
Rebooting seems to work, but I still have a 2.6.30.2 kernel...
Still, I have cpuidle (etc) working at least to some degree, and with only the network (Ethernet) connection and a fairly idle machine I get a mains power consumption shown of somewhere between 5W and 6W, and powertop is showing maybe not enough time sleeping in C1:
Cn Avg residency P-states (frequencies) C0 (cpu running) ( 0.2%) C0 213.6ms (50.6%) C1 41.3ms (49.2%)
mvsdio is shown as the top cause for wakeups (~45% of ~15/sec total), with orion_tick next at about 17% and the kernel core close behind.
Running a tight loop in the shell bumps consumption up ~1W+, ie reading 7W/8W.
Additionally plugging in a USB memory stick via a hub, the serial console, and an SD card, brings the displayed total mains consumption to 9W, which is still only about half my Linux laptop!
Getting 2.6.31 Really
This time I've tried again using the new --nandflash option and the 2.6.31 kernel boots.
Plugging in an empty 2- or 4- port USB hub no longer soaks up 1W: That will give me so much more flexibility about what I plug in. (However, plugging anything into the hub eats up that power again...)
The 2.6.31 kernel seems to run a little cooler; measuring 4.6W rather than 5.1W on my E-flite meter when flat out with a large Java app.
Network Config
I have to start preparing for in-bound services, which implies allocating a name, putting it in DNS, etc. The host name will be "chai" (fully-qualified "chai.exnet.com").
I also have to set up some (iptables-based) firewalling for safety, as for my current laptop/server. I expose few services, but it is good to block anything unexpected.
When I moved my static (Apache2) Web sites across from the previous server I had to add virtual interfaces in /etc/network/interfaces for their addresses (I use a mixture of virtual and static addressing for Web sites).
Storage
I intend to do as few writes to the internal NAND as possible to prolong its life and thus that of the plug as a whole, and instead use an external SD memory card both to provide more room for read-only apps such as a full JDK and read-write space for mail spools, repositories and my home directory.
As at 2009-09-11 I have an 8GB card to hand (OCZ, class 4) and will thus spilt it into two partitions:
- ~1GB raw (125cyls) to be mounted read-only, for apps,
- ~7GB raw (871cyls) to be read/write for everything else.
and for safety I think I'll use ext3 (with data=ordered,barrier=1) rather than ext2 even at the cost of probably wearing out faster, mitigated slightly with options "noatime,commit=600" to reduce traffic.
32GB cards, the maximum that the SD standard allows, are hard to come by right now, but that's what I'll probably end up with.
Formatting the SD card involves fdisk /dev/mmcblk0. I'll then need to create filesystems with mkfs -t ext3 /dev/mmcblk0p1 (and ...p2) and mount them, permanently in /etc/fstab, with options to minimise traffic and maximise crash-robustness:
/dev/mmcblk0p1 /usr2 ext3 ro,noatime,commit=30,data=ordered,barrier=1 0 3 /dev/mmcblk0p2 /local ext3 noatime,commit=600,data=ordered,barrier=1 0 3
The ro /usr2 mount is a safety and traffic feature.
Note that several large items from /usr are moved/symlinked to /usr2, all non-essential so the OS can boot without them if /usr2 is unavailable, such as /usr/share/{fonts,java,man,man-db} and /usr/lib/jvm.
Non-critical for booting but busy/large caches and logs (such as mail) may similarly find themselves redirected from /var to under /local, to avoid / getting too full and/or prematurely worn.
Note that with the SD card in and mounted, the wakeup/interrupt rate from the mvsdio driver seems to have dropped to ~1Hz, with all wakeups <4Hz typically, and C1 (sleep state) residency very nearly 100% and ~300ms, which is very good.
Upgrading the SD card to 32GB
By upgrading to the largest possible SD card (32GB) I should be able to accommodate everything except bulk storage for the Gallery (which I intend to host on a 128GB USB flash thumb drive), including my home area, data gathering utilities, code repositories, etc.
I have inserted a SanDisk 32GB SD (class 2, SDSDB-032G-E11) into a USB adapter at it appears as /dev/sda:
# ls -al /dev/sd* brw-rw---- 1 root disk 8, 0 Sep 20 15:29 /dev/sda # fdisk /dev/sda The number of cylinders for this disk is set to 3880. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK)
Since I won't be booting off this, I can safely ignore the warnings I believe.
With this larger space I'll bump up the read-only (read-mostly!) first partition to ~2GB raw (243 cylinders) and have the rest read-write. Both partitions will be 'primary' partitions containing ext3 filesystems with 'noatime' and a large 'commit' interval: mkfs -t ext3 /dev/sda1 (and ...2).
I'm temporarily mounting these new filesystems on temporary mount points, stopping activity on the old mounts, copying data from the existing partitions into the new ones with cp -rp (as root, and avoiding lost+found), then unmounting everything and plugging the new SD card in the plug, and remounting /usr2 and /local. (The whole process is taking minutes rather than hours for magnetic disc.)
And I'm done an hour or so later...
df -h Filesystem Size Used Avail Use% Mounted on tmpfs 251M 0 251M 0% /lib/init/rw varrun 251M 68K 251M 1% /var/run varlock 251M 0 251M 0% /var/lock udev 251M 116K 251M 1% /dev tmpfs 251M 0 251M 0% /dev/shm rootfs 462M 233M 225M 51% / /dev/mmcblk0p1 1.9G 621M 1.2G 35% /usr2 /dev/mmcblk0p2 28G 2.2G 24G 9% /local
So now I have over 25GB free with most of the software I need already installed!
Bulk Storage
One of my applications needs a lot of storage (>50GB and growing, slowly), and I want to be able to power the storage from my off-grid solar, ideally via the plug's USB.
One particular problem with magnetic disc is the oomph required to 'spin up' even if using laptop-mode or similar I can keep it spun down much of the time. As far as I understand, bus-powered USB magnetic disc is not reliable.
To that end I have ordered a Kingston DataTraveler 200 USB 128 GB flash drive. I have put an ext3 (noatime,commit=600) filesystem on it, but in particular in order to minimise power consumption I explicitly automatically mount it on demand, unmounting it and explicitly suspending it when not in use to allow it and the USB hub to power down (avoids wasting ~1W). To make this less fragile I mount by LABEL or UUID, so its exact USB location will be less of an issue. (See blkid and fstab UUID= and LABEL= line formats.)
Using the numbers suggested by Theodore Ts'o to align filesystems to an SSD's erase block size to reduce wear and increase write performance, I used:
fdisk -H 224 -S 56 /dev/sda
to create cylinders of 49*128kB, then made a small (256MB) leading VFAT utility partition:
mkfs -t vfat -n PG2Kutil /dev/sda1
to allow the main/second partition to be cylinder- and erase-block- aligned, then to make that labelled main ext3 partition with better-than-default sizing (no space for expansion, directory indexes, sparse superblocks, an inode per 128kB which is a decent safety margin over the current gallery 500kB/file usage) I ran:
mkfs -t ext3 -i 131072 -L galleryDB -O ^resize_inod,dir_index,sparse_super /dev/sda2
which, with the current Gallery data copied in and taking about 50%:
Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 975872 105540 870332 11% /auto/galleryDB
Note that disc sync operations, eg during an svn commit, are slower than on magnetic media, probably as is to be expected, but better read performance does speed up some critical tasks for me.
Digital/Analogue I/O
I currently use a k8055 USB card to gather digital inputs on the state of the battery (voltage level), wind turbine output, etc. The k8055 code is simple to build but needs (for example) libusb-dev installed first.
I have rigged up a very crude measure of raw battery volts using one of the analogue inputs (in units of 0.1V by running a 200k resistor to the 12V-nominal supply and relying on the internal 100k pot to give a 3:1 potential divider and then trimming the pot to get within a few units in the last place (ulp)) and using that to drive the internal 'low battery' flag. To eliminate some noise there is a 1u5 (25V) tantalum capacitor across the analogue input (ie the 100k pot). When the low-battery flag is set most applications that can switch to a power-conserving mode and only do the essentials and/or trade speed for power/energy.
Fast Reboot
With almost everything important loaded on the system (as at 2009-09-25) system reboot is fast, about 60s from 'shutdown -r now' and about 30s from power-up (and the Marvell prompt/loader).
Future Work
There were many helpful comments on The Register write-up, eg Gordan Bobic's comments on boot-up write activity and compcache/ramzswap.