Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions apps/api/src/opendirector_api/localization.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,10 @@
"minimax_image_reference": "MiniMax H3 references must be PNG, JPEG, or WebP, with edges of 128–8192 pixels and a long-to-short edge ratio of at most 4:1.",
"minimax_image_count": "MiniMax H3 image editing accepts up to 16 reference images.",
"minimax_image_bytes": "MiniMax H3 image reference requests cannot exceed 64 MiB.",
"minimax_speech_text": "Enter the text to read aloud, up to 9,999 characters.",
"minimax_speech_model": "Unsupported MiniMax speech model. Choose a model from the preset.",
"minimax_speech_voice": "Choose a voice available to your MiniMax account from the voice library.",
"minimax_speech_reference": "MiniMax speech synthesis does not accept references. Enter the text to read aloud and choose a voice.",
"moss_image_size": "MOSS image dimensions must be 256–2048, multiples of 16, and no more than 2,097,152 pixels in total.",
"moss_image_reference": "MOSS image generation does not accept image references or editing inputs.",
"moss_layout_reference": "MOSS layout JSON does not accept additional text references. Put the requirements in the JSON or remove the text references.",
Expand Down
1 change: 1 addition & 0 deletions docs/development/lessons.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ When work exposes a consequential risk or a recurring problem, add a lesson or u
- **Trigger:** Accepting a cross-origin result URL from a configured model service.
- **Lesson:** [PR #210](https://github.com/ThinkFlowLab/OpenDirector/pull/210) added compatible image CDN downloads without provider credentials, but initially left arbitrary local and private hosts reachable. Credential isolation and bounded responses do not constrain the network destination.
- **Check:** Preserve explicitly configured local service origins; validate all DNS answers for unconfigured hosts, connect to a checked public IP while retaining the original Host and TLS identity, and revalidate every redirect. Cover local metadata addresses, IPv6, mixed DNS answers, redirect loops, and recovery without another model submission.
- **Recovery follow-up:** [PR #283](https://github.com/ThinkFlowLab/OpenDirector/pull/283) reproduced download recovery losing its adapter's public-host checks after a model was removed or changed. Recover with the accepted model snapshot and no credentials instead of an untyped download. Test private URLs, changed DNS answers and redirects after configuration removal or replacement.


## An acknowledged editor save invalidates earlier automatic callbacks
Expand Down
5 changes: 5 additions & 0 deletions docs/development/validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,7 @@ The voice browser workflow uses a separate isolated fixture. It runs the real sh

```sh
.venv/bin/python -m pytest tests/test_voice_library.py -q
.venv/bin/python -m pytest tests/test_minimax_speech.py -q
node --test tests/voiceLibrary.test.ts tests/creationParameters.test.ts
# Terminal 1
.venv/bin/python tests/browser/voice_library_server.py --port 8030
Expand All @@ -280,6 +281,10 @@ CREATION_API_URL=http://127.0.0.1:8030 CREATION_WEB_URL=http://127.0.0.1:5190 no

For a slow-response regression, set `VOICE_REOPEN_DELAY_MS=17000` for the browser command. It delays the reopened asset request beyond the ordinary 15-second interaction timeout. Cold navigation, saved-state and reopened-media waits have explicit 45-second limits; HTTP errors still fail immediately with their status. No generation is retried.

The audio-reopen check waits for the saved indicator and acknowledges the page's session-release request before reloading. Its result must not depend on an unload-time request reaching the server; keep the server's editing-lease checks intact.

慢响应回归:在浏览器命令的环境变量中设置 `VOICE_REOPEN_DELAY_MS=17000`,让重开后的资源请求超过普通交互的 15 秒等待。冷启动导航、保存状态和重开媒体分别有明确的 45 秒上限;HTTP 错误仍立即报告状态并失败,不重试生成。

The bilingual browser workflow covers opening/outside click/Escape, keyboard tabs, four-column and narrow layouts, real previews and unavailable previews, metadata filters, scoped favorites, loading/error/retry, invalid and valid local audio, creation, selection, Drama3 transcript and Nano reference requirements, supported deletion and failure feedback, model/service switching, project reopening and persisted job input. `moss_speech.mjs` is retained as an alias for this integrated workflow. The pagination regression includes a 15,001-voice catalog. With a configured MOSS account, open the library, go to the next page, and return to the first. Verify the page counts and that returning to the first page restores its entries using read-only requests. Python tests verify provider fields, pagination, credentials, account boundaries, actual audio/container/duration validation, and equivalent Agent operations. Actual provider synthesis still requires a configured service, account permission and a recording authorized for use.

The same browser workflow also covers MiniMax's mainland speech preset in both languages, localized rejection of unsupported model IDs, saved credentials remaining private, read-only system voices, IDs containing spaces, selection retention between HD and Turbo, the 9,999-character precheck, and generation/reopen/playback. Its generated audio is a local WAV fixture, not MiniMax output. `test_minimax_speech.py` uses mocked HTTP with real MP3 bytes to check `/t2a_v2` and `/get_voice`, HTTP-200 error envelopes, invalid or incomplete responses, public result downloads without credentials, UI/Agent acceptance and receipts, actual media probing, and recovery after model removal or replacement without another synthesis call. Recovery must still reject private destinations, DNS changes to private addresses, and redirects into private networks. Live MiniMax synthesis and perceptual voice quality remain separate real-account checks.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 7 additions & 1 deletion docs/guides/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,10 +87,16 @@ For video generation, choose a mode supported by the model:
- **First and last frames**: Connect up to two image sources and assign them to the first- and last-frame slots. Last-frame support depends on the model's capabilities and the guidance in the interface.
- **Continuation**: Select a model that supports continuation and a source video. If the service restarts or the source is no longer available, you may need to regenerate the source video.

For audio generation, enter the text to be read aloud in the prompt. Supported vLLM-Omni speech models and MOSS Open Platform offer **Voice library** beside the model controls. The library opens above the composer with **All voices**, **My voices**, and **Favorites**. Click a compatible voice’s name to select it. When a preview is available, click the circular play button to listen. When a compatible voice has no preview, its icon and card background also select it. Metadata filters appear only when supplied by the service.
For audio generation, enter the text to be read aloud in the prompt. Supported vLLM-Omni speech models, MOSS Open Platform, and MiniMax Official Speech offer **Voice library** beside the model controls. The library opens above the composer with **All voices**, **My voices**, and **Favorites**. Click a compatible voice’s name to select it. When a preview is available, click the circular play button to listen. When a compatible voice has no preview, its icon and card background also select it. Metadata filters appear only when supplied by the service.

On MOSS Open Platform, select the stable `moss-tts-1.5-flash` or `moss-tts-1.0-pro` model, or choose its dated snapshot to pin a version. All four use the same voice library on a given account connection. The library shows up to 150 MOSS voices per page. Use **Next page** and **Previous page** to browse. Filters and the Favorites tab show matches on the current page.

For MiniMax, choose **Settings → Model connections → Audio → Add service → MiniMax Official Speech**, enter your mainland China API key, and save `speech-2.8-hd` or `speech-2.8-turbo`. The preset uses `https://api.minimax.cn/v1`. Unsupported model IDs are rejected when saving the service; choose a supported model from the preset. In an audio node, select that model, choose a system voice from **Voice library**, and enter up to 9,999 characters. Reference assets, voice cloning, voice design, and deletion are not supported by this integration. The system voice API does not provide previews; selecting a voice does not start a paid synthesis request. Compatible MiniMax speech models on the same account connection retain the selection.

MiniMax speech uses the [synchronous T2A API](https://platform.minimax.cn/docs/api-reference/speech-t2a-http) and saves an MP3 in the project. A saved result URL permits download recovery without another synthesis request, even after removing the model configuration. The provider's URLs expire after 24 hours; an expired link cannot be recovered locally. An unknown submission is not retried automatically. This integration covers the mainland China endpoint; international endpoints and real-account availability need separate verification.

![MiniMax system voice selection with a mocked catalog](images/minimax-speech-voices-en-US.png)

Choose **Create voice** to upload a local recording, name it, and review its playback, format, size and duration before submitting. vLLM-Omni requires the consent recording ID; adapters such as Drama3 also require the recording transcript. MOSS-TTS-Nano needs a registered voice with reference audio: a listed `default` voice is insufficient. This upload registers the voice with the selected service without adding a canvas reference node.

Selected voices are saved in the project; favorites are saved as browser preferences. Both are scoped to the configured service and account. Compatible models sharing a service can retain the selection; another service or account requires its own registered voice. Use a voice’s secondary menu to copy its provider ID or delete an uploaded voice where supported. Deletion requires confirmation; MOSS deletion is unavailable until its API supports it. Service errors are shown without credentials; an unsupported model keeps its existing voice controls.
Expand Down
8 changes: 7 additions & 1 deletion docs/guides/user-guide.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,10 +87,16 @@ HTTP 400。
- **首尾帧**:最多连接两个图片来源,按槽位选择首帧和尾帧;能否使用尾帧以模型能力和界面提示为准。
- **接续**:选择支持该功能的模型和来源视频。服务重启或来源不再可用时,可能需要重新生成来源视频。

音频生成时,在提示词中填写需要朗读的正文。受支持的 vLLM-Omni 语音模型和 MOSS 开放平台会在模型控件旁显示**音色库**。点击后在创作区上方打开**全部音色 / 我的音色 / 收藏**。点击兼容音色的名称即可选用。有试听音频时,点击圆形播放按钮即可试听。兼容音色没有试听音频时,点击图标或卡片空白处也可以选用。筛选项仅根据服务实际提供的元数据显示。
音频生成时,在提示词中填写需要朗读的正文。受支持的 vLLM-Omni 语音模型、MOSS 开放平台和 MiniMax 官方语音会在模型控件旁显示**音色库**。点击后在创作区上方打开**全部音色 / 我的音色 / 收藏**。点击兼容音色的名称即可选用。有试听音频时,点击圆形播放按钮即可试听。兼容音色没有试听音频时,点击图标或卡片空白处也可以选用。筛选项仅根据服务实际提供的元数据显示。

在 MOSS 开放平台上,可选择稳定版 `moss-tts-1.5-flash` 或 `moss-tts-1.0-pro`,也可以选择带日期的快照来固定版本。同一账户连接下,这四个型号共用音色库。每页最多显示 150 个 MOSS 音色,点击**下一页**或**上一页**可以翻页。筛选和收藏标签显示当前页中符合条件的音色。

使用 MiniMax 时,选择**设置 → 模型接入 → 音频 → 添加服务 → MiniMax 官方语音**,填写中国站 API Key,保存 `speech-2.8-hd` 或 `speech-2.8-turbo`。预设地址为 `https://api.minimax.cn/v1`。保存服务时会拒绝不支持的型号,请选择预设中的受支持型号。在音频节点中选择该模型,从**音色库**选择系统音色,再填写最多 9,999 个字符的朗读文本。本接入不支持参考素材、音色克隆、音色设计或删除。系统音色接口不提供试听文件;选择音色不会发起付费合成请求。同一账户连接下的兼容 MiniMax 语音模型可以保留选择。

MiniMax 配音使用[同步语音合成接口](https://platform.minimax.cn/docs/api-reference/speech-t2a-http),将 MP3 保存到工程。已有结果链接可用于恢复下载,不会重新提交合成;移除模型配置后仍可恢复。供应商链接有效期为 24 小时,过期后无法在本地恢复。提交结果不明时不会自动重试。本接入覆盖中国站地址,国际站和真实账户可用性需要单独验证。

![MiniMax 系统音色选择,使用模拟音色目录](images/minimax-speech-voices-zh-CN.png)

选择**创建音色**,上传本地录音、填写名称,并在提交前检查试听、格式、大小和时长。vLLM-Omni 需要填写授权录音 ID;Drama3 等适配器还必须填写录音逐字稿。MOSS-TTS-Nano 需要带有参考音频的已注册音色,列表中的 `default` 不能代替录音。上传会直接在当前服务注册音色,不创建画布参考音频节点。

所选音色随工程保存,收藏作为浏览器偏好保存,均按服务连接和账户隔离。同一连接下的兼容模型可以保留选择;其他服务或账户必须单独注册音色。音色的更多操作菜单可复制服务端 ID,或在服务支持时删除上传音色。删除前需要确认;MOSS 尚未开放删除操作。服务错误不会显示凭据;不支持音色库的模型保留原有音色控件。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,8 @@ def _parameters(self, spec, config, node, records):
params.update(prompt_enhance=False, watermark=not self.settings.get()['removeWatermark'])
if spec.kind == 'audio' and input_rules(config)['profile'] == 'moss-speech':
params = dict(voice=intent['voice'], response_format='mp3', delivery_method='url', stream=False)
if spec.kind == 'audio' and input_rules(config)['profile'] == 'minimax-speech':
params = dict(voice=intent['voice'], format='mp3', output_format='url', stream=False)
return params

@staticmethod
Expand All @@ -231,7 +233,7 @@ def _execution(kind, parameters, config):
return dict(kind=kind, creationType='music', format='mp3', maxDurationSeconds=parameters['duration'],
scoreMode=parameters.get('score_mode'))
if kind == 'audio':
if input_rules(config)['profile'] == 'moss-speech':
if input_rules(config)['profile'] in {'moss-speech', 'minimax-speech'}:
return dict(kind=kind, voice=parameters['voice'], format='mp3')
return dict(kind=kind, voice=parameters.get('voice') or ('default' if config['connectionType'] == 'self_hosted' else 'alloy') if config['protocol'] == 'openai' else None,
speaker=parameters.get('speaker') if config['protocol'] != 'openai' else None,
Expand Down Expand Up @@ -600,10 +602,8 @@ async def recover(self, project_id, job_id, session=None, *, explicit=False):
valid = False
if not valid:
if mode == 'download':
# The private locator belongs to the accepted result, not
# the current model selection. Never attach another model's
# credentials or turn this read into a new submission.
config = None
# Keep the accepted adapter and download policy without current credentials.
config = {**job.inputSnapshot.model.model_dump(mode='json'), 'apiKey': '', 'authMode': 'none'}
else:
task.update(status='interrupted', recovery='review', resumeAllowed=False)
await asyncio.to_thread(self.checkpoint, project_id, task, pinned)
Expand Down
21 changes: 19 additions & 2 deletions packages/core/src/opendirector/application/voice_library.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,12 @@ def _config(self, model_config_id):
if capabilities is None: raise voice_error('voice_library_unsupported', 409)
return config, capabilities

def _request(self, config, method, suffix='', **kwargs):
def _request(self, config, method, suffix='', path='/audio/voices', **kwargs):
try:
# Sync service runs in worker threads, including job preparation.
with httpx.Client(transport=self.transport, trust_env=trust_environment(config['url'], config=config),
timeout=httpx.Timeout(120, connect=15), follow_redirects=False) as client:
with client.stream(method, endpoint(config['url'], '/audio/voices') + suffix,
with client.stream(method, endpoint(config['url'], path) + suffix,
headers=auth_headers(config), **kwargs) as response:
data = bytearray()
for chunk in response.iter_bytes():
Expand Down Expand Up @@ -134,6 +134,21 @@ def _vllm_page(self, config, capabilities):
records[identity] = (record, locator)
return list(records.values()), None

def _minimax_page(self, config, capabilities):
# https://platform.minimaxi.com/docs/api-reference/voice-management-get
# Errors arrive as HTTP 200 with a nonzero base_resp.status_code.
value = self._request(config, 'POST', path='/get_voice', json={'voice_type': 'system'})
status = value.get('base_resp')
if not isinstance(status, dict) or type(status.get('status_code')) is not int:
raise voice_error('voice_provider_protocol', 502)
if status['status_code'] != 0:
raise voice_error('voice_provider_error', 502)
items = value.get('system_voice')
if not isinstance(items, list) or len(items) > 10000 or not all(isinstance(item, dict) for item in items):
raise voice_error('voice_provider_protocol', 502)
return [self._record({'id': item.get('voice_id'), 'name': item.get('voice_name')}, capabilities,
source='builtin', has_audio=False, config=config) for item in items], None

@staticmethod
def _valid_cursor(cursor):
return isinstance(cursor, str) and 0 < len(cursor) <= 2048 and cursor == cursor.strip() and cursor.isprintable()
Expand Down Expand Up @@ -169,6 +184,8 @@ def _pages(self, config, capabilities, limit, after=None):
self._assert_current(config, capabilities)
if capabilities['provider'] == 'moss':
records, cursor = self._moss_page(config, capabilities, limit, after)
elif capabilities['provider'] == 'minimax':
records, cursor = self._minimax_page(config, capabilities)
else:
records, cursor = self._vllm_page(config, capabilities)
for record, _ in records:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,7 @@ def command_fingerprint(command: Command) -> str:
'moss_speech_text', 'moss_speech_voice', 'moss_speech_reference', 'moss_prompt_limit',
'moss_image_size', 'moss_image_reference', 'moss_layout_reference',
'minimax_image_reference', 'minimax_image_count', 'minimax_image_bytes',
'minimax_speech_text', 'minimax_speech_voice', 'minimax_speech_reference', 'minimax_speech_model',
'voice_library_unsupported', 'voice_connection_changed', 'voice_reference_required',
'voice_transcript_required', 'voice_transcript_too_long', 'voice_description_too_long',
'voice_not_found', 'voice_delete_unsupported',
Expand Down
Loading
Loading