Repository navigation
moss-infer-vulkan: the engine's request checks, --help/--version, --vram-budget - #10
Merged
Merged
Conversation
…ram-budget Found by the moss-infer-hip review, which mirrors this program: - --help prints the usage to stdout and exits 0; --version prints the version. - pipeline::validate()'s bounds and messages: a whitespace-only prompt, at most 4.2 megapixels and 4096 pixels on a side, at most 10000 steps (was 1000), the seed, cfg, shift and max-seq messages; --device below -1 is refused. - The PNG metadata names the bundle by util::path_key(model).filename(), as the engine does (a trailing slash or "--model ." gave an empty name or "."), and records noise_layout official. - --vram-budget GiB, with vram_option_bytes' bounds: caps the device memory of the VAE and the DiT (weights, the DiT's activations, the decode); the DiT's blocks past it stream. A budget that cannot hold the decode or the activations is refused before the text encoder runs. RTX 4090 Laptop, 512x512 20 steps: --vram-budget 4 keeps 10 of 42 blocks resident and gives the same latents and PNG bytes as no budget; 0.5 GiB is refused before any work; 2 GiB at 1024x1024 streams every block and completes.
…block streamed Review of the previous commit: the early refusal counted the DiT's activations only, so a budget just above them passed, then the DiT's global tensors, text refiner and two block slots (about 1 GiB) went past it. Dit::minimum_weight_bytes() gives that footprint from the shape lists the constructor checks (now shared); the refusal before the text encoder and a second one before the DiT loads (should the encoder's lengths differ from the token counts) both include it, instead of falling back to every block streamed past the budget. RTX 4090 Laptop: 1.3 GiB at 512x512 and 2 GiB at 1024x1024 (both ran before, over the budget) are refused before the text encoder; 3 GiB at 1024x1024 runs with 3 blocks resident; 4 GiB at 512x512 gives the same latents as no budget.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings
moss-infer-vulkan's command line in line withmoss-infer generate. The gaps were found by the review ofmoss-infer-hip, which mirrors this program.--helpprints the usage to stdout and exits 0 (it went to stderr with exit 2).--versionis new.pipeline::validate()'s bounds and messages:--devicebelow -1 is refused.util::path_key(model).filename(), as the engine names it. Before, a trailing slash gave an empty name and--model .gave ".".noise_layout: officialis recorded.--vram-budget GiB, withvram_option_bytes' bounds. It caps the device memory of the VAE and the DiT: their weights, the DiT's activations and the decode. The DiT's blocks past the budget stream from host memory.Dit::minimum_weight_bytes(): the global tensors, the text refiner and the two block slots), is refused before the text encoder runs.RTX 4090 Laptop GPU:
--vram-budget 4: 10 of 42 blocks resident, and the latents and PNG bytes are identical to the run without a budget.