OpenRec Algorithms

Release v0.2.0 · Changelog

The offline side of open-rec: computes the recall tables and trains the rank model that the online service serves. It supports both one-machine development (pandas / gensim / torch) and scheduled cluster execution (Hive / Spark), with the same recall formulas and serving schemas.

It plays two roles:

• a batch tool — generate recall tables as CSV, which openrec/init loads into Redis and Elasticsearch
• a library — rank-engine imports LRModel, UserFeature and ItemFeature from it at serving time, pinned as rec-algorithm==0.2.0

install

pip install -r requirements.txt
pip install -e .

Install only what the selected execution mode needs:

pip install -e .                 # local library development
pip install -e ".[spark]"       # Hive/Spark jobs
pip install -e ".[publish]"     # Redis/Elasticsearch publishing
pip install -e ".[cluster]"     # complete distributed runtime

For local tests, copy the fixture into the gitignored data directory:

mkdir -p data/test
cp ../openrec/data/test/*.csv data/test/     # if you have the example repo checked out

data/ holds CSV datasets. Local model helpers save to OPENREC_MODEL_HOME/rank/{scene} when configured, otherwise to the sibling model/rank/{scene} if that store exists, or rec-algorithm/model/rank/{scene} as a fallback. The default scene is default; directories are created on demand. Runtime cluster releases use a separate artifact volume.

Build the wheel that rank-engine depends on:

bash package.sh                              # -> dist/rec_algorithm-0.2.0-*.whl

layout

Path	Contents
algorithm/recall/	Item and user recall strategies: CF, BM25 sparse text, content similarity, hot/new, sequence embedding, and ALS-based U2U
algorithm/rank/	LR/FM/LightGBM rank models and a candidate-aware Transformer building block
algorithm/feature/	feature encoders for users and items
algorithm/meta/	table and column definitions — the source of truth for CSV headers
algorithm/structure/	ScoreItem, JSON helpers
tool/	dataset generation and recall dumping scripts
test/	pytest suites, doubling as usage examples
jobs/spark/	categorized recall/ranking/analytics jobs and shared Spark infrastructure; extension guide
publisher/	Redis and Elasticsearch serving-layer publication
conf/	cluster configuration examples

execution modes

Local mode remains the fast feedback path: existing classes under algorithm/ accept pandas frames and produce ScoreItem objects or model artifacts. Spark mode is an execution layer, not a second algorithm library: it reads the Hive entity contract, applies the same deduplication and scoring rules, writes versionable Parquet/Hive results, and can publish them to the online stores.

data-processor -> Hive ODS/DWD -> jobs.spark -> Hive/Parquet artifacts
                                            -> versioned Elasticsearch recall tables
                                            -> Elasticsearch embeddings

Run with the cluster's Spark installation:

spark-submit --master spark://spark-master:7077 \
  --py-files dist/rec_algorithm-0.2.0-py3-none-any.whl \
  jobs/spark/recall/job.py hot \
  --event-table openrec.event_entity \
  --date 2026-08-18 --output-table openrec.recall_hot --size 2000 --publish

spark-submit --master spark://spark-master:7077 \
  --py-files dist/rec_algorithm-0.2.0-py3-none-any.whl \
  jobs/spark/recall/job.py item_cf_i2i \
  --event-table openrec.event_entity \
  --date 2026-08-18 --output-table openrec.recall_item_cf_i2i --size 20 --publish

spark-submit --master spark://spark-master:7077 \
  --py-files dist/rec_algorithm-0.2.0-py3-none-any.whl \
  jobs/spark/recall/job.py item_seq_emb \
  --event-table openrec.event_entity \
  --date 2026-08-18 --output-table openrec.recall_item_seq_emb --vector-size 64 --publish \
  --es-password "$ELASTIC_PASSWORD" --es-ca-certs /path/to/ca.crt

Use new with --item-table openrec.item_entity. Output schemas are stable:

• hot/new: scene, item, score
• new additionally carries publish_time for the online time-window filter
• item_cf_i2i: scene, left_item, right_item, score
• content_i2i: scene, left_item, right_item, score
• user_cf_u2i: scene, user, item, score
• item_seq_emb: scene, item, vector
• text_search: scene, item, text or scene, user, text (target_type=user)
• user_cf_u2u/content_u2u/user_als_u2u: scene, left_user, right_user, score; the ALS job publishes to user-als-u2u for the serving user_als_u2u channel

All source and result tables are partitioned by UTC dt. --date defaults to yesterday, so a daily scheduler can invoke the same command without calculating a date; passing it explicitly is recommended for backfills and reproducibility. Source reads use the cumulative warehouse snapshot through --date, not just that day's partition: events are de-duplicated and accumulated, while items are collapsed to their latest mutation and latest DELETE tombstones are excluded. Every recall algorithm then semi-joins its events with this active item snapshot before scoring, so deleted items do not consume hot/item_cf_i2i/user_cf_u2i/item_seq_emb resources or get published online. In cluster mode the runner reads the daily ODS directories directly with --event-path and --item-path. This preserves Hive-style partition discovery and reads source data directly from HDFS without requiring metastore table reads. The runner uses Spark 4.0.4. Result data is written beneath --output-path, remains partitioned by the requested day, and uses dynamic partition overwrite so rerunning one day does not rewrite other result partitions. Table-based reads and writes remain available for compatible metastores.

With --publish, non-vector algorithms are staged under their serving table names: hot, new, item-cf-i2i, content-i2i, user-cf-u2i, user-cf-u2u, content-u2u, and user-als-u2u. Physical indexes use openrec-recall-{tableName}-{YYYYMMDD}-{revision}. rec-console creates the staging index before Spark writes it, then verifies the document count, atomically moves openrec-recall-{tableName}-active, and removes versions beyond the configured retention after the switch succeeds. The active version plus one previous physical index are retained by default. Use --revision r002 for a rerun that must coexist with r001, and --max-index-versions to change the maximum number of loaded physical indexes. Scene remains a document field and query condition; it is never part of the physical index name.

Embedding publishes one versioned physical index per scene as {scene}-item-seq-emb-index-{YYYYMMDD}-{revision} and atomically moves the legacy serving name {scene}-item-seq-emb-index as an alias after document-count validation. The previous version is retained for rollback instead of deleting the active vector index before a replacement is ready.

Publishing the same date and revision again is idempotent when its document count matches. An active physical index is never deleted or overwritten; use the next revision when recomputation changes its contents. Index creation, activation, retention and rollback belong to rec-console; this project only computes recall data and writes documents into the staging index authorized by the console. In cluster mode Airflow calls the internal rec-algorithm-runner service, which owns the Spark submission command while Airflow remains Docker-socket-free. Enable the openrec_daily_recall DAG for the 02:00 UTC schedule, or trigger it with {"revision":"r002"} for a corrected daily release. Each submitted runner job is capped at four Spark cores by default; set RECALL_SPARK_CORES for a different deployment policy. Worker capacity itself is not artificially capped.

Emergency rollback does not rerun Spark or reload documents. POST {"algorithm":"item-cf-i2i","target_index":"openrec-recall-item-cf-i2i-20260819-r001"} to the internal rec-console endpoint /api/recall/releases/rollback; it atomically moves only the active alias. When the target is omitted, the newest retained non-active index is selected. The same operation is exposed in the Airflow UI as the manual openrec_recall_rollback DAG. Trigger it with the same algorithm and optional target_index configuration.

The cluster runner also exposes internal POST /jobs/rank/train. Its Spark job reads cumulative event, item, and user partitions through date. Labels are mutation-resolved at the immutable end of the requested UTC business window, so a later update, delete, or late arrival cannot rewrite a rerun. Item rank uses click/expose labels from that day. User rank uses a seven-day label window by default (user_label_window_days, range 1-30): shared positive interactions form U2U positives, while an equal number of deterministic negatives are sampled only from users active in that window. Generated negatives inherit their source user's label time.

Spark then performs a separate point-in-time join for every label. Entity profiles use the latest mutation visible at that label time, a latest DELETE removes the sample, and behavioural windows include only interactions strictly before the label. It materializes aligned events.jsonl, sample_users.jsonl, and sample_items.jsonl directories. The Spark driver invokes algorithm.rank.training in its dedicated Python/PyTorch environment to fit the FeatureSpace, train, evaluate and atomically write an immutable release. max_history_rows and max_materialization_seconds bound the job. The release request records the observation and feature time bounds, source/constructed/materialized sample counts, history rows, and elapsed materialization time. Rank submissions default to four total executor cores (RANK_SPARK_CORES=4) and emit a version manifest for the Airflow openrec_rank_model registration task. Activation remains a separate rec-console operation.

The runner also serves GET /features and POST /features/validate for the console's global catalog and training selection. Neither these operations nor training call rank-engine. The online service may be stopped throughout training.

The image contains separate Spark and training Python environments. Spark performs distributed sample preparation; LR/FM use CPU PyTorch and LightGBM uses its CPU learner in a subprocess on the offline driver, not distributed parameter training. The subprocess failure fails the Spark job, so Airflow cannot register a failed release. No checkpoint is published until the evaluation gate passes. Invalid/duplicate sample identities, empty samples, single-class labels and undefined AUC fail training. Fitted encoders use only the training time slice. Existing checkpoint and sidecar formats remain compatible with online inference.

Build arguments TRAINING_BASE_IMAGE and TRAINING_PIP_INDEX_URL select the independent PyTorch base and package mirror. Compose exposes these as RANK_TRAINING_BASE_IMAGE and RANK_TRAINING_PIP_INDEX_URL; --local in example uses a separate PyTorch 2.8 / CUDA 12.9 training image and the local pip mirror. The training base must provide /opt/conda for copying into Spark; the PyTorch 2.10 serving image uses system Python and cannot be substituted for this stage. RANK_TRAINING_PYTHON selects the trainer interpreter (default /opt/conda/bin/python), RANK_TRAINING_THREADS=2 limits CPU threads, and RANK_RUNNER_CPUS=4 / RANK_RUNNER_MEMORY=8g limit the offline runner container. Training dependencies are in requirements-training.txt; torch comes from the base image. Large training sets still need sufficient driver memory for pandas and PyTorch after Spark preparation.

POST /jobs/analytics runs the business dashboard aggregation with four Spark cores by default. It scans only the selected daily event partitions, de-duplicates mutation-envelope events by trace identity, and returns overall PV/UV CTR, PV/UV CVR, active items and GMV plus daily trend rows.

jobs.spark.ranking.samples.labelled_interactions performs the large join against the same active item snapshot, excluding deleted-item samples before deterministic train/validation splitting. PyTorch/FeatureSpace training remains the model contract so its checkpoint is still loadable by rank-engine; distributed sample preparation does not introduce an incompatible Spark ML model format.

generate data

Generate deterministic raw inputs, then build the deployable model bundle:

python tool/gen_test_data.py --output ../openrec/data/test --seed 42
python -m tool.build_default_artifacts \
  --data ../openrec/data/test --model-root ../model

gen_test_data.py creates an impression funnel with expose, stay, click, collect and buy events. The conversion probability depends on user interests, item metadata, popularity and position, so rank and recall receive learnable rather than independent random signals. The artifact builder emits feature snapshots, fitted LR/FM/LightGBM feature spaces, model artifacts, all recall tables and a hash manifest. tool/gen_recall_data.py remains available as a standalone CLI when only recall is needed.

recall

Every algorithm implements recall(user_triggers, item_triggers); item_cf_i2i and item_seq_emb also implement a dump_* method used to write the offline tables.

Algorithm	Class	Method
item_cf_i2i	ItemBasedI2I	item co-occurrence within a user's sequence, damped by 1/log(len+1), normalized by sqrt(count_i * count_j)
user_cf_u2i	UserBasedCF	inverse-popularity weighted user similarity, followed by unseen-item aggregation from the top similar users
content	ContentBasedI2I	TF-IDF cosine similarity over field-prefixed category, tags and title tokens
item_seq_emb	EventEmbedding	word2vec (gensim) over per-user item sequences; dump_vectors exports 10-dim vectors
user_cf_u2u	Spark user_cf_u2u	inverse-popularity weighted shared-item similarity between users
content_u2u	Spark content_u2u	profile-token Jaccard similarity between users
user_als_u2u	Spark user_als_u2u	Nearest users from ALS latent vectors, published under the user-als-u2u serving name
hot	Hot	click counts normalized by the maximum
new	New	freshness min-max normalized over the observed pub_time range, raised to power (31)

ItemEntityEmbedding remains reserved for learned dense content vectors. ContentBasedI2I is the implemented sparse content recall path and requires no model service or external download.

All of them are computed per scene — group events by scene before constructing them, as gen_recall_data.py does. item_cf_i2i, content_i2i, and item_seq_emb require item triggers; user_cf_u2i requires a user trigger; User U2U algorithms produce candidate-user rows or vectors; hot and new do not require triggers. Spark recall jobs write Hive/Parquet results and publish their serving tables through the same versioned release protocol as hot and item-CF.

Two behaviours worth knowing:

• item_cf_i2i merges across triggers. An item reachable from several triggers is emitted once, keeping its best score; triggers themselves are never recalled back; the result is truncated to recall_size. The similarity matrix is quadratic in sequence length, so it is computed once per instance and reused by recall() and dump_i2i().
• item_seq_emb tolerates unknown triggers. min_count=5 keeps rare items out of the word2vec vocabulary, so a trigger may be absent; those are skipped, and a request where none are known returns an empty list rather than raising.

rank

LRRecModel wraps a torch logistic regression over concatenated user and item features. FMRecModel reuses that exact feature vector and training/scoring contract while learning second-order interactions in a compact latent space.

user_feature = UserFeature(users=users, events=events)
item_feature = ItemFeature(items=items, events=events)

lr_model = LRRecModel(user_feature=user_feature, item_feature=item_feature, events=events)
lr_model.train(epoch_num=10, batch_size=256, learning_rate=0.003)
lr_model.save()                    # -> model/rank/default/lr.pth

lr_model.load()
lr_model.score("user_0", ["item_0", "item_1"])

To train FM instead:

from algorithm.rank.fm import FMRecModel

fm_model = FMRecModel(user_feature=user_feature, item_feature=item_feature,
                      events=events, factor_dim=8)
fm_model.train(epoch_num=10, batch_size=256, learning_rate=0.003)
fm_model.save()                    # -> model/rank/default/fm.pth

CandidateAwareTransformerModel is the reusable single-objective sequence ranker. It projects dense candidate/history item vectors, encodes a bounded history with self-attention, applies candidate-aware attention, and fuses the result with the existing flattened OpenRec global feature vector. The experiments repository owns its first EB-NeRD training protocol. The cluster release job and rank-engine lifecycle support the LR/FM/LightGBM tabular models only, so Transformer serving activation requires a separately versioned deployment contract.

The cluster rank job accepts model_type=lr|fm|lightgbm and factor_dim (FM only). Its release manifest records the effective LightGBM tree count (max(50, epochs * 100)) because the shared epochs control is translated to boosting rounds. Cluster LightGBM uses binary click/expose classification over independent point-in-time samples; the separate LambdaRank wrapper is intended for datasets that provide contiguous impression/query groups. The release manifest keeps the model type, latent width, feature-set identity, fitted input dimension and sidecar checksum so rec-console can validate and atomically deploy or roll back each supported type.

Labels come from the event type: click is 1 and an unclicked expose is 0. When an impression has both events, its expose remains available to behavioural aggregation but is not a negative label. For User rank, validation reserves the newest examples independently within each label so tied synthetic-negative timestamps cannot leave the holdout single-class and make AUC undefined.

Features used are deliberately a subset — country, city, gender, age and tags for users; category, scene and weight for items — plus the event snapshot statistics described below. Raw ids, names and titles are excluded because one-hot encoding them explodes the tensor size. The categorical input dimension still depends on the training vocabulary, so FeatureSpace is saved beside every checkpoint and must be loaded by rank-engine.

feature catalog and model feature sets

algorithm/feature/definitions/feature.catalog.json is a generated, package-local copy of the global catalog owned by model/feature/catalog/feature.catalog.json; do not edit it directly. It also registers ecommerce user features derived from the Item context frozen into each event, with point-in-time Item-history lookup for legacy events. These include version visible at event.time: weighted category/subcategory preferences, per-action price mean/deviation, 1/7/30-day click and buy price means, and recent/long-term price ratios. Pandas and Spark point-in-time materializers and realtime snapshots implement the same formulas. contains every entity and behavioural feature OpenRec currently produces. lr.feature-set.json and fm.feature-set.json declare each model family's implemented feature capabilities and default selection. A training request can select an ordered subset of those supported features. These files do not replace the canonical catalog; new features require matching offline and online implementations before being offered for training.

Training fits the selected set against that version's data and writes a self-contained lr.features.json, fm.features.json, or lightgbm.features.json beside the model. This fitted sidecar includes the ordered columns, category vocabularies, numeric normalization statistics, catalog/set provenance, and computed widths. New sidecars persist fingerprints of selected feature definitions; deployment checks these against the packaged catalog, so unrelated additions do not invalidate an existing release. Older sidecars retain whole-catalog version/SHA-256 validation. Deployment uses the fitted checkpoint and encoders without refitting or consulting a live catalog service. LR/FM/LightGBM can select different subsets even when their default capability sets contain the same features.

materialize online features

Create flat user/item snapshots that can be imported into the online user:* and item:* tables:

python tool/gen_feature_data.py \
  --user ../openrec/data/test/user.csv \
  --item ../openrec/data/test/item.csv \
  --event ../openrec/data/test/event.csv \
  --output data/feature/test

The output keeps all entity columns and adds event count/value statistics, active days, distinct scene and counterpart counts, first/last time, recency, 1/7/30-day counts, fixed event-type counts (click, expose, buy, collect, stay) and click rate. The default snapshot time is the newest event; production jobs should pass --as-of-time explicitly. Use a snapshot before the model label period to avoid target leakage.

Serve a trained checkpoint with rank-engine; generated bootstrap artifacts for the deterministic example fixture live in model. Always deploy the fitted sidecar with its checkpoint; feature dimensions depend on that artifact, not a fixed dataset-wide constant.

test

Run tests from the repository root. Shared helpers resolve the deterministic fixture under data/test; copy it from ../openrec/data/test as described in the install section when it is not already present:

pytest -q test
pytest -q test/algorithm/recall/test_item_cf_i2i.py
pytest -q test/algorithm/rank/test_lr.py::test_train
pytest -q test/jobs/spark/test_runner.py

For the smallest recall-contract check, the merge test uses fixed sequences and needs neither Spark nor external services:

pytest -q test/algorithm/recall/test_item_cf_i2i_merge.py

data structure

item

name	type	required	description
id	string	yes	item uniq id
title	string	yes
category	string	yes	single value
subcategory	string	no	one or more content subcategories
tags	string	no	multi value
scene	string	yes	relation recommend or guess you like
pub_time	int	yes
modify_time	int	no	update time
expire_time	int	no
status	bool	yes	could be recommend
weight	int	no
ext_fields	json	no

user

name	type	required	description
id	string	yes	user uniq id
device_id	string	yes	user device id
name	string	no	fake name
gender	string	no
age	int	no
country	string	no
city	string	no	update time
phone	long	no
tags	string	no	multi value
register_time	int	no
login_time	int	no
ext_fields	json	no

event

name	type	required	description
id	string	yes	event uniq id
user_id	string	yes
item_id	string	yes	single value
trace_id	string	yes	eg: openrec
scene	string	yes	relation recommend or guess you like
type	string	yes	eg: click, expose, buy, collect, stay
value	string	yes	the value of the type
time	int	yes
is_login	bool	no	could be recommend
ext_fields	json	no

Selected global features

select_features(model_type, target_type, selection) validates an ordered subset of the LR/FM/LightGBM capability declarations. selection has required user and candidate lists and optional session, context, and interaction lists; candidate features belong to the target entity. Omitting it keeps the model's default feature set. FeatureSpace.for_model(..., selection) persists the selection and fingerprints of selected definitions alongside fitted encoders. Definition fingerprints include referenced policies. Adding an unrelated catalog feature does not change a fitted subset; changing a dependency rejects it.

The Spark rank job accepts --feature-selection JSON and --scene global for all-scene training. Neither registration nor selection creates a feature producer: new features require aligned offline and online implementations before being added to a model's supported set. Rebuild the wheel and downstream images together.

Item ranking also supports cold-start content features from the canonical item schema: title, subcategory, tags, and publication-time-derived content_age_hours. Unbounded text uses persisted fixed-width signed BLAKE2b feature hashing, so training and rank-engine encode unseen text identically without shipping a vocabulary. Content age is calculated at each label time in offline point-in-time materialization and at feature refresh time online.

When upgrading a shared model volume, Compose runs rank-artifact-init first to transfer release/training directory ownership from the former online writer to Spark. The online activation directory is unchanged. If starting the runner with --no-deps, run this initialization service explicitly first.

Dynamic feature rollout boundaries

The feature catalog describes encoder capabilities. The runner's /features response additionally exposes training_models: features actually connected to its Spark release pipeline. /features/validate, /jobs/rank/train, and the Spark command line apply the same select_training_features gate. Omitted selections use user/candidate features; UTC hour/weekday context is opt-in and is derived from each label timestamp.

Spark currently exports sample_users.jsonl and sample_items.jsonl, not session or interaction samples. Session aggregates, user-item interactions, candidate position/count, device type and subscription/authentication context therefore cannot be selected through this cluster pipeline yet. Model/experiment support for these roles remains available with explicitly materialized, sample-aligned inputs. The trainer rejects absent selected values instead of synthesizing zero columns. sessionId, requestId, and position survive ODS event reads; legacy typed event tables may omit these optional columns.

Do not infer request candidate counts from the number of labelled events or use post-ranking display positions as pre-ranking candidate positions. Connecting these features requires logging the matching request/candidate context, exporting point-in-time dynamic sample files, and validating the corresponding online producer before expanding training_models.

Spark runtime compatibility

The Spark extra and cluster runner use Spark 4.0.4 / Scala 2.13 with Java 21. The runner image must match the bigdata-platform master/workers; rebuild it when upgrading the platform. The separate PyTorch training interpreter and model contracts are unchanged.

Query embedding names

The default item language vectors are item_title_emb.csv; the old duplicate query_embedding.csv is removed. User factors remain user_als_emb.csv. query_qwen_emb identifies Qwen query-to-item recall from the Music CRS experiment: Qwen3-Embedding-8B, last-token pooling and normalized 4096D vectors. Its encoder runs upstream; QueryQwenEmbeddingRecall consumes matching precomputed vectors.

algorithm.recall.query_embedding.QueryEmbeddingRecall remains the generic cosine API for existing sequence/history callers. A generic vector operation does not identify its encoder; strategy and artifact names do. No other recall algorithm changes in this rename.

文本向量模型配置

algorithm.recall.text_embedding.TextEmbeddingEncoder 通过模型名称加载 Qwen3 文本模型， 离线 item 与在线调用方的 query 必须使用同一模型及 revision。 当前默认名称为 Qwen/Qwen3-Embedding-0.6B，可以通过环境变量或生成器 --model 覆盖。 此编码器使用 Qwen3 的 query instruction、last-token pooling 和 L2 归一化； 更换其他模型家族时需要对应的编码实现，不能仅替换名称沿用 Qwen3 的预处理。

环境变量	用途	默认值
OPENREC_TEXT_EMBEDDING_MODEL	模型名称	Qwen/Qwen3-Embedding-0.6B
OPENREC_TEXT_EMBEDDING_REVISION	远端提交号或本地模型文件指纹	main
OPENREC_MODEL_CACHE	模型缓存目录	模型库默认缓存位置
OPENREC_TEXT_EMBEDDING_DEVICE	推理设备	cpu
OPENREC_TEXT_EMBEDDING_MAX_LENGTH	最大 token 数	512
OPENREC_TEXT_EMBEDDING_INSTRUCTION	query 检索任务说明	Given a search query, retrieve relevant items that match the query

安装 .[text] 后执行（也可在本地 rec-algorithm 镜像的训练 Python 环境中运行）：

python -m tool.gen_text_embedding_data --item ../openrec/data/test/item.csv --output ../model/recall/item

默认仅生成 item_title_emb.csv（title）。显式增加 --include-content 可生成 可选的 item_content_emb.csv（title/category/tags），该文件不随默认产物发布。 生成器返回模型名称、实际 revision、维度、预处理和数据校验值，CLI 将其输出到日志， 不再单独生成 Qwen 专属 manifest。 不记录机器上的绝对路径。模型缓存可挂载至容器任意目录，并通过 OPENREC_MODEL_CACHE 指定。 已有 ModelScope 模型按 <cache>/<namespace>/<model> 解析并离线加载， 例如模型名称 Qwen/Qwen3-Embedding-0.6B；本地模型通过权重和 tokenizer 等文件的 SHA-256 生成 revision 指纹，查询侧可使用生成日志中的 revision 验证是否加载了相同文件。

查询侧复用 TextEmbeddingEncoder().encode_query(text)，得到的向量用于检索这两张语言向量表。 发布时应保留生成日志中的 model_name/revision/max_length 和 query instruction 配置；更换模型或预处理需重新生成两张表。 该 Python 入口尚未接入 rec-server 的 HTTP 请求流程。

离线全文检索数据统一输出为 recall/item/text_search.csv 和 recall/user/text_search.csv。离线算法名为 text_search，线上通道名为 item_search / user_search；item/user 分别发布到 item-search / user-search， user 任务设置 target_type=user，输出路径独立为 user_text_search。

Scheduled serving-index coverage

Graph	Serving table	Scheduled algorithm	Serving alias
item	hot / new	hot / new	openrec-recall-{table}-active
item	item-cf-i2i / content-i2i / user-cf-u2i	item_cf_i2i / content_i2i / user_cf_u2i	openrec-recall-{table}-active
item	item-seq-emb	item_seq_emb	{scene}-item-seq-emb-index
item	item-search	text_search (item)	openrec-recall-item-search-active
item	query-title-emb	query_title_emb	query-title-emb-index (global)
user	user-cf-u2u / content-u2u / user-als-u2u	user_cf_u2u / content_u2u / user_als_u2u	openrec-recall-{table}-active
user	user-search	text_search (user)	openrec-recall-user-search-active

query_title_emb reads the latest non-deleted item snapshot as of the business date, ignores scene, skips blank titles, and rejects duplicate global item IDs. It sends raw titles (no query instruction) to the configured vLLM /v1/embeddings endpoint, validates model, dimensions, indexes and finite nonzero vectors, then L2-normalizes the results. The output schema is item,vector,dt, saved under the query_title_emb HDFS output path. Spark caches generated vectors on executor disk during output and publication, avoiding repeated inference for each action. Long titles exceeding the server token limit fail the job; no partial candidate corpus is activated.

The runner accepts algorithm=query_title_emb. Configure EMBEDDING_ENDPOINT, EMBEDDING_MODEL, EMBEDDING_DIMENSIONS, and optional EMBEDDING_API_KEY consistently with rec-server and rec-console. Offline batching uses EMBEDDING_BATCH_SIZE=32, EMBEDDING_PARTITIONS=1, and EMBEDDING_BATCH_TIMEOUT_SECONDS=60 by default, independently of the online request timeout. Model paths and GPU loading remain owned by vLLM.

rec-console prepares openrec-recall-query-title-emb-YYYYMMDD-rNNN, validates the staged count, blocks writes, and atomically activates the global alias. It retains at least two releases for rollback. The existing per-scene sequence-vector publisher remains separate.

CPU bootstrap and optional title vectors (bundle v19)

tool.build_default_artifacts and tool.gen_recall_data now default to CPU recall and rank artifacts without loading a language model. They do not require transformers or download Qwen weights. The bundle checker accepts both CPU-only bundles and bundles with recall/item/item_title_emb.csv; every included output is checksummed. A rebuild preserves existing title vectors when the item CSV and vector checksum match the previous manifest, even if events or rank feature definitions changed.

To generate a complete language-enabled bundle, install rec-algorithm[text], configure the model cache as described above, then run:

python -m tool.build_default_artifacts --data ../openrec/data/test \
  --model-root ../model --include-text-embeddings

The same flag is available on tool.gen_recall_data. Ordinary standalone/cluster bootstrap does not enable it; enabling online query embedding requires a matching precomputed title table and inference service. No substitute vectors are generated.