Earth Notes: On Website Technicals (2017/11)

Tech updates: Googlebot warp space, image re-optimisation, liter, defer, inlining...

2017/11/16: Image Inlining

For the mobile/lite site, to try to overcome a potentially-lengthy RTT (Round-Trip Time, ie latency) to fetch the ATF (Above The Fold) hero image (~300ms on 2G/3G), if the image (or its lo-fi variant as used for Save-Data is small enough, then it will be inlined directly into the HTML as a base-64 data URL. (Some other very small hero images could also be inlined where that would probably save overall bandwidth.)

The 'lite' site is also intended for bandwidth-limited connections such as dial-up. On modern mobile latency is the key constraint, on dial-up it is bandwidth, but the overal effects can be very similar!

This has the effect of preventing caching of the inlined images, and of getting less text in the initial inbound TCP packet(s), but the tradeoff is hopefully a better overall UX.

With this hero inlining in place the entire mobile home-page ATF content is rendered from the HTML on the first round trip in ~1.7s according to WebPagetest's 'simple' test: Dulles, VA - Moto G4 - Chrome - 3GSlow. The (gzipped) home page HTML is still well within the initcwd at 8.4kB.

2017/11/14: Defer

Given Prefer DEFER Over ASYNC: "DEFER scripts don’t execute until the HTML document is done being parsed", I'm testing switching the already-at-the-end Share42 script from async to defer to possibly marginally improve behaviour on mobiles in particular (CPU starved, share icons not shown ATF).

A preliminary WebPagetest suggests at least that the loading of the script stays late (low priority) and that there is no obvious harm from doing this.

Note that defer seems to be slightly less supported than async, eg Opera was reported as not.

2017/11/12: Even Liter

As an experiment I'm dropping most of the ads from the mobile/lite site as the nominal revenue (loss) is tiny but the effect on page weight is huge! Partly inspired by "Banner Ads Considered Harmful (Here)".

2017/11/11: Image Re-optimisation

Always poking the hornets' nest out of sheer devilment, I went and took another run at PageSpeed Insights.

By folding in some of Google's specific ImageMagick convert suggestions from "Optimize Images", in particular -sampling-factor 4:2:0 for JPEGs, I was able to hit 100/100 for the mobile home page in a mobile browser, with all other site/browser combinations in the high 90s.

2017/11/05: Firework Routing

Googlebot owns an interesting warp in the fabric of space-time!

See GSC Crawl Stats up to a couple of days ago. (All the record highs are in that snapshot, plus the record low download time.) Up before the last day the usual rule of thumb of ~200ms + ~1ms/kB was holding true, but for the very last point the download time drops ~50ms below the usual baseline, which was due to nothing at my end that I know of, and usually only happens with a very small number of fetches in a day.

I looked through my logs for being crawled locally (eg from within the EU) which could knock down the RTT signifcantly (vs ~150ms London to California).

Other than a few obvious fakes all the (non-Images) Googlebot IPs appear to be registered in Mountain View, but a couple of traceroutes are interesting:

% traceroute
traceroute to (, 30 hops max, 60 byte packets
 1 (  0.766 ms  0.997 ms  1.160 ms
18 (  20.784 ms  20.212 ms *
% traceroute
traceroute to (, 30 hops max, 60 byte packets
 1 (  0.825 ms  1.116 ms  1.407 ms
19 (  103.652 ms  103.580 ms  95.009 ms

Note that it can take ~10--20ms RTT across my local FTTC link to my ISP's infrastructure. StatusCake's London server sees a minimum of ~27ms and a mean of ~70ms to pull down the mobile home page.

One Googlebot is 80ms RTT closer than the other, and both seem closer than Mountain View (for normal traffic)!

(Just after writing the above the data for the 4th became visible, with a more normal download time but still fairly vigorous spidering.)