Earth Notes: On Website Technicals (2023-04)

Updated 2024-03-17.

Tech updates: sonifying data to house, Apache and CC, awk function, TiMidity++, MIDI deflate, Ableton, Bitwig, LMMS, mild.

The Easter weekend seems to have been an ideal time to start ~~messing around with~~ ahem "researching" some music/sonification... I am reminding myself of the adage do it, do it right, do it better, and I will be on stage 1 for quite a while!

2023-04-30: Sluggish Filesystem

The RPi that I use to talk to the Thermino heat battery has had its filesystem get quite full, partly is seems logging of the relentless break-in attempts via ssh now that the system is reachable from outside:

% df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        14G   14G  278M  98% /

Thanks to the nice clear explanation in How to Clear Systemd Journal Logs I ran a selection of command as below (most output not shown):

% journalctl --disk-usage
Archived and active journals take up 1.4G in the file system.
% sudo journalctl --rotate
% sudo journalctl --vacuum-time=2d
% sudo journalctl --vacuum-size=100M
sudo vi /etc/systemd/journald.conf
sudo systemctl restart systemd-journald

I edited into /etc/systemd/journald.conf SystemMaxUse=xxxM to keep future log size in reasonable bounds.

To then banish the slugs I ran sudo fstrim -av which released more than the equivalent of the entire journal space, but took a while!

I will probably build the journald.conf update into the ansible configuration.

2023-04-23: MIDI Format 1 Disaster

Garage Band did not seem to be paying attention to anything other than the first channel in my first track for my automated sonification of my 'consolidated' kWh data. (I have been optionally using an extra track for data-independent percussion.)

It seems that for MIDI format 1 (as I have been using) MIDI commands are assigned to more tracks, each track should have not more than one channel. First track is reserved to conductor.

Oh dear! I have been holding it wrong! GarageBand seems to be right... This also maybe explains why LMMS split my first track into several.

After examining a couple of big MIDI files (each from a pop song, one format 0 the other format 1), and playing with Signal - Online MIDI Editor, my new strategy is to generate MIDI format 1, one track (and channel) per voice, plus a leading 'control'/tempo track and an optional percussion track. This will require giving up any pretence of 'streaming' the data, instead making one pass over the data for each data stream to make its own track.

(I also tried making a format 0 track with each voice in its own channel, which worked, but folding in complex percussion beside that would have been horrible!)

2023-04-22: Mildly Augmented

Alongside the simple/bare MIDI files generated from consolidated kWh data, generated with V3 of the conversion script, I am now exposing versions 'mildly augmented' with very basic percussion and some time alignment (January to start each bar in monthly data) using V4.

V4 is still being hacked at and thought about, so that new set of sounds may be more volatile and flaky (and interesting).

2023-04-17: Bitwig et Al

A friend said You strike me as more of a bitwig person ... so I downloaded the 30-day free trial of 4.4.10.

And managed to get a complex MIDI playing within a minute or so. Though that was having being primed by Ableton to need to 'activate audio' and similar such unexpected steps. But getting audio out at all is hard.

I will look at FOSS tools Ardour and LMMS (brew gets V1.2.2).

(In LMMS, once I had downloaded a SoundFont2 (.sf2) file as it told me to, I could play MIDI files! Panning per voice worked too. I could even export MIDI.)

I could also try the 90-day free trial of the GarageBand upgrade to Logic Pro, though a licence would then be ~£200. Exporting MIDI is one feature.

Alignment

For monthly-cadence consolidated data I now force alignment so that the first data/melody note in the bar is -01 ie January, with the bar being 12 notes (thus a year).

2023-04-16: Pan

I'm trying to make the programmatically-generated MIDI from data sound more 'interesting'.

To that end, today's lesson has been how to add panning (by channel). Here I set up three (flute solo) channels, spread out left/right:

1, 0, Program_c, 0, 79
1, 0, Control_c, 0, 10, 21
1, 0, Program_c, 1, 79
1, 0, Control_c, 1, 10, 63
1, 0, Program_c, 2, 79
1, 0, Control_c, 2, 10, 105

I'm also annotating the generated MIDI with key timestamps from the input, eg on the first month of each year:

1, 480, Marker_t, "2010-01"

Ableton Live

I'm trying Ableton Live 11 free for 90 days. It is much more complex than GarageBand, and it took me about half enough to get a MIDI playing at all! (It turns out that until you attach a set of instruments to a MIDI track, you have the sound of no hands clapping...)

2023-04-12: TiMidity++

I installed TiMidity++ (on my Mac with brew, on the RPi with apt-get) to allow command-line automated conversion of MIDI to audio, eg FLAC:

% timidity input.mid -OF -o out.flac

MP3 format is not supported, but, for example, FFmpeg can convert from FLAC.

The sox utility is another possibility.

MIDI Compression

Since at least the MIDI files that I am generating seem to be significantly compressible, I have added to Apache's auto-deflate list both the mid filename suffix and the audio/midi MIME-type.

% ls -al data/consolidated/energy/std/flow/M/flow-M.*
3588 data/consolidated/energy/std/flow/M/flow-M.csv
1502 data/consolidated/energy/std/flow/M/flow-M.mid
% gzip -v9 < data/consolidated/energy/std/flow/M/flow-M.csv | wc -c
 74.7%
924
% gzip -v9 < data/consolidated/energy/std/flow/M/flow-M.mid | wc -c
 78.9%
335

Structure

At some point I might want to add a little more song structure to the sonifications, eg 'verse' and 'chorus' sections at different time resolutions, such as monthly vs yearly.

2023-04-11: awk function

Today I wrote my first awk function. It contains percussion generation so that I can share it between generative music projects.

Yes, in all the 30+ years that I must have been writing awk code I do not think that I have written a function in it!

2023-04-10: Apache and CC Licences

I just inserted a minimal Apache 2.0 licence statement in all the .sh scripts under scripts/mkaudio as below, just below any 'usage' section:

##########
# May be used / adapted / etc without any promise of fitness for purpose
# under the terms of the Apache License Version 2.0, January 2004
#     http://www.apache.org/licenses/LICENSE-2.0
##########

I hope that I did not break anything!

I also put most of my SoundCloud tracks under a Attribution 3.0 Unported (CC BY 3.0) licence, for which the only restriction is basically attribution (which, as for my Apache 2.0 code, I am happy with 'where reasonable'). I have removed even the attribution element for some trivial sounds.

Energy Series

Work has started on the script to generate a MIDI from every 'consolidated' format energy series CSV in 16WW Energy Series Dataset.

Each such file may get its own player and visualiser.

I intend to generate plain and 'house' versions of each series CSV.

Initially I have just the very simplest bare/plain MIDIs being generated immediately after the CSVs have been regenerated: 84 of them!

Longer Sample

Taking daily grid-tied solar generation from 2008 to now and the MIDI generated from it, then in GarageBand switching the instrument to Japanese String Movement, generates a fairly interesting and soothing piece, at over 15 minutes (~1 minute per year).

2023-04-09: House

It's a weak joke, but I'm running with it. Sonifying house data as house music. Though this time with MIDICSV rather than awk.

The new script textToMIDIv2 uses MIDICSV to convert monthly mains tap-water temperature samples for 16WW into a simple MIDI binary file (format 1, with 1 track). GarageBand is happy to open and play that file.

This replaces the previous awk hack, and is much more likely to be generating a completely legal MIDI file!

The output .mid file will be regenerated whenever a new data point is added.

The conversion plays 12 months (ie one year) into each 4/4 bar. This just about gets into bar 16 currently, with a grand total of 1691 bytes.

You may also have noticed recent entries in the Sources/Links (of this and previous months) prefixed with M3c. Those, in the same spirit as A11y (accessibility) and I18n internationalisation, etc, are for music. I am not short of weak jokes.

The script textToMIDIv2-house is a fork of textToMIDIv2 to generate a simple house-style percussion track alongside.

I had already put together a basic house percussion bar with a script after watching half a YouTube video, maybe not the good half. So sue me, I have no talent!

The first MIDI out of this process can be heard in all its MP3-rendered glory: