Skip to content

feat(resume): Resume a match from its replay - #673

Open
bill-rich wants to merge 15 commits into
GeneralsOnlineDevelopmentTeam:mainfrom
bill-rich:up/resume-from-replay
Open

bill-rich wants to merge 15 commits into
GeneralsOnlineDevelopmentTeam:mainfrom
bill-rich:up/resume-from-replay

Conversation

@bill-rich

Copy link
Copy Markdown

Adds a way to pick a match back up after someone crashes or drops. Everyone gets back in a lobby, the host types /resume (or hits the Resume button), and the game starts by replaying the last recording at full speed until it reaches the point where the match died. Then it hands control back to the players and the match keeps going.

Right now when a game dies the only option is to start over. In longer team games that can mean throwing away 30+ minutes, and getting the same group back together to replay from scratch often doesn't happen. With this the group just needs to get back in a lobby.

How it works:

  • The host arms the resume with /resume in lobby chat or the Resume button. /resume off (or CANCEL RESUME) disarms it
  • The lobby is realigned from the recording: map, starting cash, superweapon limit, and every slot's occupant, faction, color, start position and team. Online, the host sends the whole slot arrangement in one HOST_ACTION_ALIGN_SLOTS request and the service rebuilds the lobby to match or refuses with a reason
  • The host's last replay is copied to Replays/Resume.rep and pushed to every guest (LAN over the map transfer path, online through an upload to the service). Guests validate their copy against the lobby when it arrives and again at game start. The host can't start until every human has it
  • On start every client injects the recorded commands frame by frame while the network game runs in lockstep at an uncapped rate. The last 10 seconds before the handoff run at normal speed, then there's a 10 second countdown, then players are live
  • The resumed match records a new replay from frame 0, so the result is one complete replay of the whole match
  • Anything that doesn't line up (wrong player in a slot, missing AI, mismatched map, a replay that ends early) is posted to the whole lobby with what's wrong and who needs to fix it, since that's usually not the host

Also included because resume hit them:

  • Base64Encode padded a one-byte tail as XXX= instead of XX==, so inputs of length 1 mod 3 decoded one byte long
  • Requests with a body now send a JSON content type. Without it libcurl labels the body as form data and large payloads fail with 400 "form key length limit 2048 exceeded"
  • The host bulk slot update (HOST_ACTION_BULK_SLOT_UPDATE). This is shared with feat(gui): Add Random button to LAN and Online game lobbies #421, so whichever lands first carries it

Testing: I've run it with two and three LAN clients. Two clients caught up 4500 frames in about 6 seconds and resumed on the same frame. With three, the resume restored the recorded slot order, factions and colors after the players rejoined in a different order with different picks. An earlier version of this has also been running in Zulu GO since October and has been used to recover real online games. I haven't tested this exact branch online against the upstream service yet.

The Resume buttons are looked up by name and everything works without them, so the layouts can ship separately (LanGameOptionsMenu.wnd:ButtonResumeFromReplay and GameSpyGameOptionsMenu.wnd:ButtonResume). All text uses FETCH_OR_SUBSTITUTE, so there's no csf change.

Depends on: the companion Services change (lobby resume fields, ALIGN_SLOTS and the replay upload). GeneralsOnlineDevelopmentTeam/Services#83

Base64Encode emitted a one-byte tail as XXX= instead of XX==, because its padding test compared the consumed index against size + 1, which is never true. Any input whose length is 1 mod 3 therefore decoded to one byte too many. Found streaming replay chunks for live spectating, where the extra byte shifted every later offset and zeroed a byte of the replay header.
Without a Content-Type libcurl labels the body application/x-www-form-urlencoded, so the service form-parses it and rejects anything past its form key limit: a large JSON payload reads as one enormous key and the request fails with 400 "form key length limit 2048 exceeded". Small bodies happened to survive because the parse succeeded and nothing read the result. Requests that set their own type, such as the S3 uploads, are left alone.
Lets the host push side, color, start position and team for every occupied slot in one request (HOST_ACTION_BULK_SLOT_UPDATE, field 19, supported by Services since PR TheSuperHackers#13). Shared with the Random button change; whichever lands first carries it.
Adds RECORDERMODETYPE_RESUME_CATCHUP. When the lobby arms a replay, every client opens its own copy on game start and injects its recorded commands frame by frame while the live network game runs in lockstep at an uncapped rate, renders on a wall-clock cadence and skips CRC validation. The last 10 seconds before the handoff frame run at normal speed, then the simulation is held for a 10 second countdown, then control is handed back and the replay file is taken over for recording so the resumed match is one complete replay. Camera scrolling works throughout; selection, commands, control bar and hotkeys are inert until the countdown ends. LAN lobbies arm with the /resume chat command (or a ButtonResumeFromReplay if the layout has one): the last recorded replay is validated against the lobby roster, the lobby is aligned to the replay's slots and seed, and RF=/RHF= tokens carry the file and handoff frame to every client.
The host types /resume in the online lobby chat to arm the last recorded replay (or /resume off to disarm). The replay is validated against the current game version and the lobby: the map must match and every human must already sit in the slot they held in the replay, since the service owns slot order; the host is told the first difference. The recorded faction, color, start position and team are restored through the bulk slot update and the replay's seed, file name and handoff frame are sent with the new HOST_ACTION_ARM_RESUME lobby field, which the companion Services change stores and serializes as ResumeReplayFile and ResumeHandoffFrame. Depends on the Random button branch for the bulk slot update.
Running out of recorded frames during catch-up now ends the catch-up instead of the live game (readNextFrame no longer calls stopPlayback in that mode). Arming falls back to walking the replay's records for its last frame when the header count is 0, which is the case for every recording cut short by a crash, using the recorder's analysis mode. The catch-up start, lead-in, handoff, countdown ticks and handoff completion are written to the GeneralsOnline release log. Verified in a two-client LAN run under wine: both clients caught up 4500 frames in 6 seconds, played the 10 second lead-in, showed the same 10 second countdown and resumed live play on the same frame with the replay files continuing to record.
…ches

Roster problems are now announced to the whole lobby instead of only the host, since the player who has to fix a name or a slot is usually not the host. When a resume is armed, the host's start re-reads the recording and refuses to start if any recorded player is missing from their slot or a stranger sits in one, telling the lobby which slot must hold whom and how to disarm. LAN gets /resume off to disarm. Verified with three clients under wine: a renamed player blocks arming with a lobby-wide message, and a player who leaves and rejoins under another name after arming blocks the start with the offending slot named on every client. The three-player resume itself restored the recorded slot order, factions and colors after the joiners came back in a different order with different preferences.
The online lobby routes every slash command through handleGameSetupSlashCommands, which answered /resume with Unknown command before the chat entry ever saw it. The command now lives there, with a /help line.
Every client now replays the same file. /resume copies the host's last recording to Replays/Resume.rep (RecorderClass::getResumeReplayFileName), aligns the lobby with it and broadcasts only the handoff frame (RH= in the LAN options, ResumeHandoffFrame online; the file name is fixed, so RF= and replay_file are gone). The file is pushed to the guests before the game starts: on LAN over the map transfer path at OnGameStart (DoResumeReplayTransfer, with .rep admitted to the transfer whitelist behind a GENREP check and a Replays\ portable path), online by an upload to the service that must finish before the arm is sent, after which guests download it, report "Resume: replay downloaded" and flag themselves as holding it; the host cannot press start until every human does. The handoff is the host's scanReplayLastFrame(Resume.rep) minus a named slack, so there is no shorter-replay case any more.

The resumed match records a brand new 00000000.rep from frame 0 like any other game: the recorder starts recording as usual, opens Resume.rep on a second handle, injects its commands frame by frame and writes them straight into the new recording, then keeps recording live after the handoff. The old truncate-the-source hand-over is gone.

Guests validate their copy against the lobby (version, map, map CRC, seed, starting cash, superweapon limit, every slot's occupant, and that the recording reaches the handoff) when it arrives and again at game start, through one helper (GameNetwork/ResumeFromReplay). A mismatch backs out of the start the way a missing map does, and a source that ends before the handoff aborts the game instead of handing off locally; nothing falls back to recording a fresh match. Changing the map, starting cash or superweapon limit disarms on the client and on the service.

Also: local MSG_LOGIC_CRC messages are dropped during catch-up and validation stays off for one run-ahead after the handoff, which removes the false "Not enough CRCs" on the handoff frame; the throttled render path still emits MSG_FRAME_TICK every logic frame so the network's per-frame bookkeeping keeps running; a torn replay record ends the usable data instead of being parsed as garbage; the dead FPS and logic-rate plumbing is removed (catch-up runs on the isResumeCatchupMode bypasses only); the recorder no longer reaches into LANAPI or NGMPGame; the arm is announced online only from the service's success reply.
…w upload

The slot check now covers every slot, not just the humans' names: occupant (human by name, AI by difficulty, or empty, with open and closed both counting as empty), faction, color, start position and team, since all of it seeds the simulation the recorded commands drive. Arming aligns the faction, color, start spot and team of every occupied slot whose kind already matches and then runs the full check, so a missing or wrong AI is reported for the host to fix instead of slipping through to the start.

Online guests download the resume replay whenever the lobby is armed with an upload generation they have not fetched (the service bumps it on every arm), so a re-arm with the same handoff frame is fetched again; a chunk upload answered 409 resumes from the length the service reports, up to three times. The lead-in keeps the network's realtime gate so its last ten seconds run at the logic rate regardless of the render rate. Once the recorder holds the arm at game start the handoff frame is cleared on the game info, keeping RH= out of the new recording's options string. scanReplayLastFrame refusing to run while a file is open is now an explicit, logged error that readSource reports rather than a silent zero.
…e Resume button

Online resume used to refuse whenever the lobby's slot order differed
from the recording, because the service owns slot order. The host now
sends the whole recorded arrangement in one HOST_ACTION_ALIGN_SLOTS (22)
request and the service rebuilds the lobby to match or refuses it with
a reason. The arm is a chain, each step from the previous completion:
map, starting cash and superweapon limit (only what differs), the slot
table, the replay upload, then the arm itself, which must come last
because the service disarms on every one of those changes.

Every refusal is posted to the whole lobby as what went wrong and who
fixes it. Guests print the armed notice from the lobby state itself
rather than relying on the host's chat line, and failed or mismatched
downloads say what to do. Start on a drifted lobby points at Resume or
Cancel Resume.

The Resume button is the same action as /resume and /resume off: the
host arms with it and, while armed, it reads CANCEL RESUME and disarms;
guests see RESUME ARMED, disabled. Both lobbies look the button up by
name and work without it, so the layouts can ship separately
(LanGameOptionsMenu.wnd:ButtonResumeFromReplay and
GameSpyGameOptionsMenu.wnd:ButtonResume). All texts use
FETCH_OR_SUBSTITUTE, so no csf change is needed.
Reasons that already end in a period were getting a second one, and
the wait for a guest's download now says what to do if it never comes.
A transferred .rep could land under any name in the guest's replay folder. The
only legitimate file is Resume.rep, so refuse every other leaf.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant