Earth Notes: On Website Technicals (2023-01)

Every time I encounter the ProGuard Java shrinker and optimiser I am newly impressed.

I was unhappy with the >600kB JAR file supporting the grid intensity page.

After a couple of hours wondering why ProGuard was blaring warnings at me, I was down to ~50kB fully obfuscated (but a little hard to debug)!

Done with ~20 lines added to my ant build.xml file.

I have now settled on a ~80kB unobfuscated, slightly statically optimised, and much cleaner and more robust JAR file. Hurrah!

684487 Jan 10 18:56 reutils-1.1.21.jar
 84974 Jan 15 19:30 edhMain.reutils-1.1.22.jar

Lower, Lower!

By being a bit bolder and allowing almost all obfuscation to reduce size, but retaining some information (source file names and line numbers) to help diagnose run-time exceptions, size is now ~10% of the original. (Removing unrelated functionality no longer used would cut that further!)

 65114 16 Jan 15:04 edhMain.reutils-1.1.23.jar

Example exception output now:

% sh extraTweet.sh "testing 1 2 3"
FAILED command: extraTweet
java.lang.IllegalArgumentException
        at org.hd.d.edh.E.a(TwitterUtils.java:370)
        at org.hd.d.edh.Main.main(Main.java:146)

Naturally there were a couple of astonishments due to the obfuscation. But those were fixed and some overdue related code improvements done.

A couple of pages on EOU, eg my PhD research page, are rated as "very difficult to read" by a Flesch–Kincaid scoring mechanism.

% reado --unfluff < PhD-research.html
...
score: 29.94
school level: college graduate
notes: Very difficult to read. Best understood by university graduates.

Pages tagged as 'technical' are allowed a lower readability than most. But the previous value (42) was way too high for these rogue pages, so that 'technical' threshold has been lowered to 25 for now!

It is fascinating that search engines (in this case apparently at Microsoft) are still polling for Gallery URLs that have been dead more than a decade, possibly two!

d.hd.org:80 40.86.XX.XX - - [11/Jan/2023:15:18:16 +0000] "GET /_I/cat/13/20yuxbmq0pc.HTM HTTP/1.1" 404 397 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.1 Safari/605.1.15"
www.hd.org:80 40.69.XX.XX - - [11/Jan/2023:15:18:18 +0000] "GET /Damon/_I/cat/19/2g4emqm6t78.HTM HTTP/1.1" 301 583 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.1 Safari/605.1.15"
d.hd.org:80 40.69.XX.XX - - [11/Jan/2023:15:18:18 +0000] "GET /_I/cat/19/2g4emqm6t78.HTM HTTP/1.1" 404 397 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.1 Safari/605.1.15"
d.hd.org:80 52.173.XX.XX - - [11/Jan/2023:15:18:19 +0000] "GET /_I/cat/19/rs1t27q1u0.HTM HTTP/1.1" 404 397 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.1 Safari/605.1.15"

I did not see a nice copy-and-paste example elsewhere though it was not that hard in the end, so here is the core of my solution:

// Fetch the auth tokens, or silently abort if not available...
final String authtoken = getMastodonAuthToken();

// Send message...

// Here is how to do it with curl...
// (MAT is a file containing the access token.)
// % curl https://mastodon.energy/api/v1/statuses -H "Authorization: Bearer `cat $MAT`" -F "status=$1"
// See https://dev.to/bitsrfr/getting-started-with-the-mastodon-api-41jj

// Use URL encoding to force into ASCII (7-bit) encoding.
final String formEncodedBody = "status=" +
    URLEncoder.encode(statusMessage, StandardCharsets.US_ASCII);

final int timeout_ms = 10000;

final URL u = new URL("https", md.hostname, "/api/v1/statuses");

final HttpsURLConnection uc = (HttpsURLConnection) u.openConnection();
uc.setUseCaches(false);
uc.setAllowUserInteraction(false);
uc.setDoOutput(true);
uc.setDoInput(true);
uc.setConnectTimeout(timeout_ms);
uc.setReadTimeout(timeout_ms);
uc.setRequestMethod("POST");
uc.setRequestProperty("Authorization", "Bearer " + authtoken);
uc.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
uc.setRequestProperty("Content-Length", String.valueOf(formEncodedBody.length()));
final OutputStream output = uc.getOutputStream();
output.write(formEncodedBody.getBytes(StandardCharsets.US_ASCII));
output.close();
final int responseCode = uc.getResponseCode();
final String responseMessage = uc.getResponseMessage();

uc.disconnect();

(Note the embedded cURL example and link!)

I have now put in place code to parse author lists in the bibliography (split by semicolon or "and"), truncate overly-long lists and tail them with et al. (the trailing "." makes it an unsexed abbreviation I understand), and standardise the separator in the HTML to be ";" for compactness. Each author gets their own itemprop=author section, which improves the metadata.

There are some potential potholes with ;s in the HTML entity codes for accented letters, but so far that has been swerved around by insisting on a space after a ";" as separator.

It turns out that some people have accents in their names! I am shocked, I tell you!

I want the bibliography .bib files to stay ASCII7, ie 7-bit ASCII, for robustness.

Accents don't fit in that reduced character set. HTML can get round that with entities such as "é" for "é". LaTeX, which is in effect the source language of BibTeX, uses backslash escapes such as \'{e} (or {\'e}).

I have set up my HTML conversion to recognise the single one of these that has so far appeared! [bouckaert2021net]

I have spent most of today gathering and archiving and commenting on energy stats for 16WW!

One improvement was to switch to the hour-by-hour carbon figure for electricity for the front-page 16WW carbon-footprint graph (a snapshot of which is shown here) where available, ie for 2020 onwards.

(A little time was spent updating copyright notices for 2023, too!)