Earth Notes: On Website Technicals (2023-02)

Today I started with the Friends of OpenJDK (aka "Java") track. After lunch I switched to the lightning talks.

GraalVM Native

Prompted by the Java talks, I am interested to get an idea of what AOT (Ahead Of Time) compilation looks like. It was in OpenJDK for a while, but has gone again, and probably will not be fully back until ~2027!

GraalVM has a native-image tool to create a fully-precompiled binary. This assumes that no features such as dynamic class loading by name are used. This VM is available for my Mac, but not for RPis older than 4 (and it seems as if 8GB RAM may be needed).

I installed V19 so:

% java -version
openjdk version "19.0.2" 2023-01-17
OpenJDK Runtime Environment GraalVM CE 22.3.1 (build 19.0.2+7-jvmci-22.3-b12)
OpenJDK 64-Bit Server VM GraalVM CE 22.3.1 (build 19.0.2+7-jvmci-22.3-b12, mixed mode, sharing)

I then installed the native-image tool with the GraalVM updater:

% /Library/Java/JavaVirtualMachines/graalvm-ce-java19-22.3.1/Contents/Home/lib/installer/bin/gu install native-image
% /Library/Java/JavaVirtualMachines/graalvm-ce-java19-22.3.1/Contents/Home/lib/svm/bin/native-image ...

A first attempt to run this against the current optimised reutils:

% /Library/Java/JavaVirtualMachines/graalvm-ce-java19-22.3.1/Contents/Home/lib/svm/bin/native-image -jar .work/edhMain.reutils-1.1.29.jar
================================================================================
GraalVM Native Image: Generating 'edhMain.reutils-1.1.29' (executable)...
================================================================================
[1/7] Initializing...                                            (5.1s @ 0.19GB)
 Version info: 'GraalVM 22.3.1 Java 19 CE'
 Java version info: '19.0.2+7-jvmci-22.3-b12'
 C compiler: cc (apple, arm64, 12.0.5)
 Garbage collector: Serial GC
[2/7] Performing analysis...  [******]                          (17.5s @ 1.66GB)
   5,010 (78.53%) of  6,380 classes reachable
   8,106 (58.03%) of 13,969 fields reachable
  23,583 (48.82%) of 48,310 methods reachable
     163 classes,    80 fields, and   739 methods registered for reflection
      58 classes,    58 fields, and    52 methods registered for JNI access
       4 native libraries: -framework Foundation, dl, pthread, z
[3/7] Building universe...                                       (2.1s @ 2.60GB)
[4/7] Parsing methods...      [**]                               (2.2s @ 1.61GB)
[5/7] Inlining methods...     [***]                              (1.3s @ 2.40GB)
[6/7] Compiling methods...    [****]                            (15.2s @ 3.91GB)
[7/7] Creating image...                                          (3.1s @ 0.83GB)
  10.34MB (45.99%) for code area:    14,471 compilation units
  11.61MB (51.63%) for image heap:  150,478 objects and 5 resources
 546.67KB ( 2.37%) for other data
  22.48MB in total
--------------------------------------------------------------------------------
Top 10 packages in code area:           Top 10 object types in image heap:
 822.49KB java.util                        2.17MB byte[] for code metadata
 641.95KB sun.font                         1.48MB java.lang.String
 497.89KB java.lang.invoke                 1.17MB byte[] for general heap data
 413.55KB com.sun.crypto.provider          1.10MB java.lang.Class
 389.12KB java.lang                      927.94KB byte[] for java.lang.String
 336.06KB java.text                      391.41KB c.o.s.c.h.DynamicHubCompanion
 281.64KB sun.java2d.marlin              280.09KB java.lang.String[]
 274.51KB com.sun.imageio.plugins.jpeg   271.27KB j.u.c.ConcurrentHashMap$Node
 270.15KB java.awt.image                 261.73KB java.lang.Object[]
 264.28KB java.util.concurrent           250.08KB java.util.HashMap$Node
   6.17MB for 178 more packages            2.42MB for 1063 more object types
--------------------------------------------------------------------------------
    1.1s (2.2% of total time) in 21 GCs | Peak RSS: 5.06GB | CPU load: 5.60
--------------------------------------------------------------------------------
Produced artifacts:
 /Users/dhd/SVN-controlled/www.hd.org/Damon/Env/edhMain.reutils-1.1.29 (executable)
 /Users/dhd/SVN-controlled/www.hd.org/Damon/Env/edhMain.reutils-1.1.29.build_artifacts.txt (txt)
================================================================================
Finished generating 'edhMain.reutils-1.1.29' in 49.0s.
% ls -al edhMain.reutils-1.1.29
-rwxr-xr-x  23576531  edhMain.reutils-1.1.29

My first attempt to run failed for a couple of reasons:

../edhMain.reutils-1.1.29 trafficLights ~/tmp/test.html
INFO: generating traffic-light summary [/Users/dhd/tmp/test.html]...
ERROR: could not fetch data from null error: Accessing an URL protocol that was not enabled. The URL protocol https is supported but not enabled by default. It must be enabled by adding the --enable-url-protocols=https option to the native-image command.
...
FAILED command: trafficLights
java.lang.NoSuchMethodError: org.hd.d.edh.D.a()
        at org.hd.d.edh.v.a(FUELINSTUtils.java:933)
        at org.hd.d.edh.f.a(FUELINST.java:377)
        at org.hd.d.edh.Main.main(Main.java:131)

So I added the --enable-url-protocols=https argument as above, and the new output is:

% ls -al edhMain.reutils-1.1.29
-rwxr-xr-x  1 dhd  staff  34150851  5 Feb 13:50 edhMain.reutils-1.1.29

That now fixes the first problem, ie the HTTPS connection works to fetch data:

(cd .work && ../edhMain.reutils-1.1.29 trafficLights ~/tmp/test.html)
INFO: generating traffic-light summary [/Users/dhd/tmp/test.html]...
INFO: record/row count of CSV FUELINST data: 1926ms, 288 records from source: https://www.bmreports.com/bmrs/?q=ajax/csv_download/FUELINST/csv/&filename=GenerationbyFuelType24HrInstantaneous fetch and parse
INFO: CHECKPOINT: data fetched etc: timestamp: 1952ms.
WARNING: could not load long store /Users/dhd/tmp/test.longstore.csv.gz error: java.io.FileNotFoundException: /Users/dhd/tmp/test.longstore.csv.gz (No such file or directory)
INFO: CHECKPOINT: 24h summmary computed: timestamp: 1960ms.
ERROR: missing directory for icons: ../out/hourly/button/
INFO: 24h summary: FUELINST.CurrentSummary:status=YELLOW:recentChange=RED:timestamp=20230205135000:currentMW=29459:currentIntensity=141:currentStorageDrawdownMW=292:histMinIntensity=76:histAveIntensity=120:histMaxIntensity=166:minIntensityRecordTimestamp=20230205044500:maxIntensityRecordTimestamp=20230204142000:histWindowSize=86100000:histSamples=288:lowerThreshold=91:upperThreshold=155:totalGridLosses=0.07:histAveIntensityByHourOfDay=80,85,89,79,78,89,102,101,102,118,133,139,140,162,156,160,162,160,157,156,141,117,100,87:computeTime_ms=8
...

Note that running the GraalVM JIT yields computeTime_ms=37 (on a single test run) and 227ms with OpenJDK 18 JIT vs the 8ms shown in the last line of the above output, so already promising.

Activating the kill switch that I have just added ahead of Twitter shutting down its free API access next week allows the process to complete without error. And in about half the time of the java JIT, though not generating the icons/buttons. Note also that the converted non-obfuscated JAR runs without this kill switch, so this issue presumably could be fixed with a ProGuard configuration change.

Creating a directory for the buttons reveals a new error:

ERROR: could not create/save buttons, error: java.lang.UnsatisfiedLinkError: no awt in java.library.path

So there would be more work to do, and I cannot deploy this on the RPi server where it would be useful. But this test drive has been very interesting, and it was easy to get a long way!

CDS Archive

Maybe I should look at using a CDS archive to speed up the existing JAR on the RPi, eg with JDK 19 onwards:

java -XX:+AutoCreateSharedArchive -XX:SharedArchiveFile=my.jsa -cp my.jar Hello