Skip to content

docs: document Kafka skipped_line limitation - #567

Open
n24q02m wants to merge 1 commit into
apache:masterfrom
n24q02m:docs/hg-3-1.8-audit
Open

n24q02m wants to merge 1 commit into
apache:masterfrom
n24q02m:docs/hg-3-1.8-audit

Conversation

@n24q02m

@n24q02m n24q02m commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Purpose of the PR

Main Changes

  • Add the same note to the English and Chinese Loader guides, directly below the Kafka input options.
  • Recommend preprocessing the topic or using a FILE/HDFS input source when line filtering is required.
  • Keep the existing skipped_line description for file-based input sources unchanged.

Context

The loader's KafkaSource declares @JsonProperty("skipped_line"), while KafkaReader has no reference to it. The current guides describe skipped_line as if it filtered Kafka messages.

Verification

  • scripts/hugo.sh build with Hugo 0.165.0 extended and --panicOnWarning: passes, 570 pages (286 EN / 284 CN).
  • The new note renders under the Kafka input source in both languages.

- Clarify that Kafka input accepts skipped_line but does not apply it.
- Recommend preprocessing the topic or using FILE/HDFS for line filtering.
- Update both English and Chinese guides.
@n24q02m
n24q02m force-pushed the docs/hg-3-1.8-audit branch from 94c6d6f to 0536c41 Compare November 16, 2026 13:33

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: The note matches KafkaReader, which hands each message to the parser and never reads skippedLine, but both pages already state this in the Kafka skipped_line bullet, so the note duplicates it above the page intro.

search_boost: 1.6
description: "Bulk import graph data into HugeGraph with Loader from files, HDFS, relational databases, Kafka, and other HugeGraph graphs."
---
> **Kafka and `skipped_line`:** HugeGraph Loader accepts `skipped_line` for

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: the skipped_line bullet under 3.3.2.4 Kafka input source already documents this limitation on both pages, and this note renders above the page intro instead of below the Kafka options. Remove the note from the English and Chinese pages, or move the FILE/HDFS suggestion into the existing skipped_line bullet.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Score: 8.0/10 · Confidence: medium

Blocking: no

Summary

The existing inline concern remains: the note duplicates the Kafka option description and sits above the introduction. Consolidating the FILE/HDFS suggestion in that option would keep the guidance together.

Evidence

KafkaSource declares skipped_line and KafkaReader passes each message to the parser without applying it; FileLineFetcher applies the file filter. Both language pages already document the Kafka limitation in their Kafka options.

Validation notes

Source review only; no live Kafka/HDFS test or Hugo build was run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants