Skip to content

[Context]: add MapParallel with chunking and bounded concurrency #20

Open

Description

@rian-be

Summary

Add MapParallel — the recommended batch-processing primitive combining mapping (Map) with bounded concurrency (MaxConcurrency) and chunking (partition batches) to cut per-item Task/AsyncLocal/GC overhead.

Goal

Provide the recommended method for batch processing (e.g. batch import), with partition-based context management and concurrency caps.

Problem

Mapping each of 1000 items into its own Task + BeginContext is GC-heavy for small in-memory jobs. There is no single call that bounds concurrency and batches work into a few context scopes.

Scope

  • MapParallel(ctx, source, parallelOptions, workItem) where parallelOptions has MaxConcurrency and ChunkSize.
  • Chunking: source of N items split into M partitions (ChunkSize); one BeginContext per partition → only M tasks.
  • Reuses the caller-supplied SemaphoreSlim limits across chunks (static MaxConcurrency semaphores).
  • Sample: import 10,000 products in chunks of 100 with MaxConcurrency = 4.
  • Unit + benchmark tests (per performance section 2.5).

Design Expectations

  • Composes Map + MaxConcurrency into a single, recommended primitive.
  • Chunking reduces GC/allocations for small jobs.
  • Result order defaulting to ordered; opt-out reserved for the Ordered issue.
  • Async-only.

Acceptance Criteria

  • Bounded concurrency respected across chunks.
  • Partitioning runs fewer tasks than items and still maps all items.
  • Benchmark shows chunking reduces overhead vs per-item mapping.

Non-Goals

  • No retry / partial-failure aggregation (Polly, TryEach).
  • No graph orchestration (DAG).
  • No streaming (IAsyncEnumerable).

Activity

  1. added
    enhancementNew feature or request
    extensionExtension behaviors / helpers
    concurrencyParallel / batch / async execution
    on Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    concurrencyParallel / batch / async executionenhancementNew feature or requestextensionExtension behaviors / helpers

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions