Earth Notes: On Website Technicals (2021-12)

What else is there to do while waiting for New Year other than set up IndexNow support?

I have this set up incrementally pushing changed, reasonably-new, main pages. This rotates randomly between IndexNow participants, since they should share all URLs that they receive.

This was provoked by Bing suddenly (unannounced, in the last few days) stopping accepting sitemap pings, rejecting them with a 410 Gone.

This is pretty much the entire support in the makefile:

# Recreate/expose the IndexNow key as necessary.
# It is not built in to the makefile since it is meant to be 'secret'.
# https://www.indexnow.org/documentation
IndexNowKeySrc=.work/IndexNow.key.txt
IndexNowKey := $(shell cat $(IndexNowKeySrc))
IndexNowKeyFile := $(IndexNowKey).txt
all:: $(IndexNowKeyFile)
$(IndexNowKeyFile): $(IndexNowKeySrc)
        @echo "Rebuilding $@"
        ln $(IndexNowKeySrc) $(IndexNowKeyFile)
        chmod a+r $(IndexNowKeyFile)
# Ping main-page updates to IndexNow and remembers which have been done.
# https://www.indexnow.org/
# Errs on the side of under-reporting.
# Submits updates incrementally.
# Only considers pages up to a few days old.
IndexNowMaxDaysOld=7
# Eliminates explicit 'NOINDEX' pages.
# Does not attempt to ping any one page more than once between updates.
# All pings could be sent to the primary (URL1) or can be shared at random.
IndexNowSEURL1=https://yandex.com/indexnow
IndexNowSEURL2=https://www.bing.com/indexnow
IndexNowFlags=.work/IndexNow.flags
.PHONY: IndexNow.ping
IndexNow.ping: $(WORKTMP)/IndexNow.ping
all:: $(WORKTMP)/IndexNow.ping
$(WORKTMP)/IndexNow.ping: makefile $(IndexNowKeyFile) $(SCWPAGES)
        @echo "Rebuilding $@"
        @$(LOCKFILENRSLOW) $@.lock
        @for f in `find $(PAGES) -mtime -${IndexNowMaxDaysOld} | sort -R`; do \
            if egrep -q '<!-- *NOINDEX *-->' .$$f; then continue; fi; \
            count=0; \
            n=$$f; \
            flag=${IndexNowFlags}/$$f.log; \
            if [ ! -f $$flag -o $$f -nt $$flag ]; then \
                echo IndexNow: $$n; \
                URL=`( echo ${IndexNowSEURL1} ; echo ${IndexNowSEURL2} ) | sort -R | head -1`; \
                wget -O $$flag "$$URL"'?url=$(URLLISTPREFIX)'"$$n"'&key=${IndexNowKey}'; \
                count=1; break; \
            fi; \
            done; \
            if [ 0 = "$$count" ]; then echo "All done..."; touch $@; fi
        @/bin/rm -f $@.lock