039: A Finished Art Build Whose Index File Is Missing Isn't "Ready"

README.md

@@ -429,7 +429,7 @@ Keep several, and delete old ones yourself.
 
 A restore needs a **new, empty volume** and runs **before the instance's first start**, as a one-off command in a container of the image you will run. It refuses a volume that already holds a Collector database (or the `-wal`, `-shm` or `-journal` file of one, or the queue, cache or cable database) and changes nothing: to restore over a running instance, restore onto a new volume and switch to it. It restores a backup taken by the same release or an older one, which the first start then migrates as any upgrade does; a backup from a newer release is refused. Use the image tag you run (`COLLECTOR_IMAGE`), and a release that has `backup:restore`.
 
-A restore ends every session (everyone signs in again) and drops pending bulk-removal Undos and selections; no collection is lost. It leaves the art index file behind: with art matching on, press **Build art index** on the admin catalog page afterwards, which needs no downloads.
+A restore ends every session (everyone signs in again) and drops pending bulk-removal Undos and selections; no collection is lost. It leaves the art index file behind: with art matching on, press **Build art index** on the admin catalog page afterwards, which needs no downloads. Until you do, that page reads "Index file missing" and `catalog:status` "Art matching: index file missing".
 
 **Docker Compose.** Create the volume without starting Collector, and find its real name (Compose prefixes the project's name; a different project name is a different, empty volume):
 
@@ -478,7 +478,7 @@ bin/kamal setup --skip-push --version 0.1.0
 
 Give the exact release as the tag: releases deploy by version, so no `latest` tag is on the server. This path is documented and its container command is tested in CI, but it hasn't been run through Kamal itself.
 
-**After a restore**, before you hand the instance back: compare the counts the command printed (accounts, users, lots) with what you expect; start it; sign in; with art matching on press **Build art index**; check your settings (`COLLECTOR_*`) and your secret.
+**After a restore**, before you hand the instance back: compare the counts the command printed (accounts, users, lots) with what you expect; start it; sign in; with art matching on press **Build art index** (until then the admin catalog page reads "Index file missing" and `catalog:status` "Art matching: index file missing"); check your settings (`COLLECTOR_*`) and your secret.
 
 **A damaged, cut-short or wrong file is refused** with a plain message and the volume is left as it was: a file still being copied, a file from another tool, a collector's export (use **Import collection** for that), a backup of a newer release, a snapshot that fails SQLite's integrity or foreign-key check.
 

app/models/mtg/art.rb

@@ -34,7 +34,8 @@ def self.after_refresh(run)
   # The art line `catalog:status[mtg]` prints (AC-3.11), from the latest build run that wasn't skipped. A build still
   # marked running reads as the admin catalog page words it (MTG::Art::Operation): building with no heartbeat while a
   # worker holds the job of a build that stopped beating (MTG::ArtBuild#stalled?), interrupted once nothing is working
-  # on it (MTG::ArtBuild#left_behind?), both judged by the queue as it was read before the build.
+  # on it (MTG::ArtBuild#left_behind?), both judged by the queue as it was read before the build. A finished build whose
+  # index file isn't on the volume reads as missing, and says to press the admin page's button (spec 039 decision 3).
   def self.status_line
     return "Art matching: off (set #{ENV_NAME}=true to turn it on)" unless enabled?
 
@@ -52,6 +53,10 @@ def self.status_line
     elsif run.running?
       "Art matching: building since #{run.started_at.utc.iso8601}; #{progress}, #{run.failed_count} failed " \
         "(last heartbeat #{run.heartbeat_at.utc.iso8601})"
+    elsif run.finished? && !run.index_file_present?
+      "Art matching: index file missing (#{run.index_file.presence || "no file name recorded"}); press Build art index on the " \
+        "admin catalog page to write it again. #{run.indexed_count} artworks indexed, #{run.without_image_count} without an image, " \
+        "#{run.failed_count} failed images"
     elsif run.finished?
       "Art matching: ready; #{run.indexed_count} artworks indexed, #{run.without_image_count} without an image, " \
         "#{run.failed_count} failed images; index #{run.index_file}"

app/models/mtg/art/index.rb

@@ -25,6 +25,11 @@ def self.current = files.first
 
   def self.built_for?(catalog_version) = files.any? { it.basename.to_s.start_with?("art-index-#{catalog_version}-#{MTG::Art::Settings.digest}-") }
 
+  # True when a file of exactly this name is in the index directory (spec 039 decision 4). The name must be one this
+  # class writes, so nothing outside the directory is looked at; the file isn't opened. A blank name, a name of another
+  # shape and a directory that can't be examined are all "not there".
+  def self.file?(name) = NAME.match?(name.to_s) && dir.join(name.to_s).file?
+
   # Only the current or previous index, by its exact name; anything else is nil (AC-4.2).
   def self.path_for(name) = NAME.match?(name.to_s) ? files.first(KEEP).find { it.basename.to_s == name } : nil
 

app/models/mtg/art/operation.rb

@@ -3,9 +3,10 @@
 # both by MTG::ArtBuild#stalled? and #left_behind?). The two differ only while the queue holds a build job that no
 # running build is being worked on by: one that waits, or one a worker has taken that hasn't opened its build yet. The
 # page then says "Queued." or "Queued to retry.", and the status line still describes the latest build (interrupted,
-# ready or failed), or says that no build has run yet.
+# ready, index file missing or failed), or says that no build has run yet.
 class MTG::Art::Operation < Catalog::Operation
   OFF = "Art matching is off. Set #{MTG::Art::ENV_NAME}=true to turn it on.".freeze
+  INDEX_MISSING = "Index file missing. Build the art index again.".freeze
   NEEDS_CATALOG = "Refresh the catalog first: the art index is built from its cards.".freeze
   # A restart that stops the worker in good order releases the job it holds: the queue runs it again, and it closes its
   # own build and carries on. After a crash or a kill the job stays held until the queue notices its worker is gone
@@ -36,17 +37,19 @@ def build
     @build = MTG::ArtBuild.latest
   end
 
-  # :off, :queued, :building, :stalled, :interrupted, :never, :ready or :failed. A build still marked running that
-  # nothing is working on any more is interrupted: its heartbeat stopped 10 minutes ago, or 2 minutes ago and the queue
-  # has failed its job or no longer holds it (issue #73). One whose heartbeat stopped 10 minutes ago while a worker
-  # still holds its job is stalled instead (issue #79).
+  # :off, :queued, :building, :stalled, :interrupted, :never, :ready, :index_missing or :failed. A build still marked
+  # running that nothing is working on any more is interrupted: its heartbeat stopped 10 minutes ago, or 2 minutes ago
+  # and the queue has failed its job or no longer holds it (issue #73). One whose heartbeat stopped 10 minutes ago while
+  # a worker still holds its job is stalled instead (issue #79). A finished build whose index file isn't on the volume
+  # is :index_missing (spec 039); it is judged only once nothing is in flight or running, so the states above it win.
   def state
     @state ||=
       if !MTG::Art.enabled? then :off
       elsif build&.running? then running_state
       elsif queue.any? then :queued
       elsif build.nil? then :never
-      else build.finished? ? :ready : :failed
+      elsif build.finished? then build.index_file_present? ? :ready : :index_missing
+      else :failed
       end
   end
 
@@ -64,6 +67,7 @@ def summary
     when :interrupted then "Interrupted. Build it again to carry on from where it stopped."
     when :never then catalog_loaded? ? "No build has run yet. One starts after the next catalog refresh, or you can start it here." : NEEDS_CATALOG
     when :ready then "Ready."
+    when :index_missing then INDEX_MISSING
     when :failed then failed_summary
     end
   end
@@ -84,9 +88,7 @@ def facts
     when :building, :stalled, :interrupted
       [ fact("Started", build.started_at), fact("Images fetched", build.fetched_count), fact("Failed images", build.failed_count),
         fact("Last heartbeat", build.heartbeat_at, relative: true) ]
-    when :ready
-      [ fact("Finished", build.finished_at), fact("Artworks indexed", build.indexed_count),
-        fact("Without an image", build.without_image_count), fact("Failed images", build.failed_count) ]
+    when :ready, :index_missing then finished_facts
     when :failed then failed_facts
     else []
     end
@@ -114,6 +116,14 @@ def retrying?
     # Why the build failed in plain words, with the error it stored as the detail (issue #84).
     def failure = Catalog::FailureReason.for(build.message, collectible_type)
 
+    # What a finished build records, plus the file's name when that file is the point (spec 039 decision 2).
+    def finished_facts
+      [ fact("Finished", build.finished_at),
+        (fact("Index file", build.index_file.presence || "none recorded") if state == :index_missing),
+        fact("Artworks indexed", build.indexed_count), fact("Without an image", build.without_image_count),
+        fact("Failed images", build.failed_count) ].compact
+    end
+
     def failed_summary = "Failed. #{failure.sentence}"
 
     def failed_facts

app/models/mtg/art_build.rb

@@ -71,6 +71,10 @@ def stalled_minutes = ((Time.current - heartbeat_at) / 60).floor
   # the first two ask #stalled? before it.
   def left_behind?(queue) = stale? || (quiet? && job_ended?(queue))
 
+  # Whether the index file this build wrote is still on the volume (spec 039). The row and the file can disagree after
+  # a restore (a backup holds the database only) or when the directory was lost.
+  def index_file_present? = MTG::Art::Index.file?(index_file)
+
   def counts = COUNTS.index_with { |name| public_send(:"#{name}_count") }
 
   private