Skip to content

Showcase / question: a board-proven offline language runtime on ESP32-C3, and whether this points to a more extreme form of language/runtime co-design #875

Open

Description

@Juemian-Deng

Hi gemma.cpp folks,

I wanted to share a small but unusual language-runtime project that may be relevant to the kind of language/runtime co-design boundary this repo is already exploring.

We built a public demo line called Engram and deployed it on a commodity ESP32-C3.

Current public numbers:

  • Host-side benchmark capability

    • LogiQA = 0.392523
    • IFEval = 0.780037
  • Published board proof

    • LogiQA 642 = 249 / 642 = 0.3878504672897196
    • host_full_match = 642 / 642
    • runtime artifact size = 1,380,771 bytes

Important scope note:

This is not presented as unrestricted open-input native LLM generation on MCU.

The board-side path is closer to a flash-resident, table-driven runtime with:

  • packed token weights
  • hashed lookup structures
  • fixed compiled probe batches
  • streaming fold / checksum style execution over precompiled structures

So this is not a standard lightweight dense inference runtime on a small device. It is closer to a task-specialized language runtime whose behavior has been crystallized into a compact executable form under very severe physical constraints.

Repo:
https://github.com/Alpha-Guardian/Engram

Why I’m posting here is that gemma.cpp seems to sit at an interesting point between research-friendly implementation, low-level execution, and simplified language inference systems.

What I’d be curious about is whether systems like this should be thought of as:

  • outside the normal lightweight inference-runtime family
  • an extreme endpoint of language/runtime co-design
  • or an early sign that some language-task capability may eventually be deployed in more specialized executable forms than even a minimalist dense runtime

If this direction is relevant to your team, I’d be glad to compare notes.

Activity

  1. jan-wassenberg commented on Mar 20, 2026

    @jan-wassenberg
    Member

    Hi, I don't have any expertise on embedded HW nor comments on your approach - beyond amazement that it is possible to do anything LLM-like with a single-core 160 MHz CPU :)

  2. Juemian-Deng commented on Mar 20, 2026

    @Juemian-Deng
    Author

    Hi, I don't have any expertise on embedded HW nor comments on your approach - beyond amazement that it is possible to do anything LLM-like with a single-core 160 MHz CPU :)

    Actually, I'm not very familiar with embedded systems. This product was a happy accident from my personal AGI research, and I'm still actively working on fundamental AGI infrastructure.

  3. Juemian-Deng commented on Mar 20, 2026

    @Juemian-Deng
    Author

    @jan-wassenberg
    Since we are both exploring the absolute boundaries of language runtimes—your team optimizing for peak dense inference, and me exploring extreme physical constraints—I’d love to see if there’s any synergy.

    If the Gemma ecosystem ever plans to explore these highly constrained edge environments or specialized executable formats, I’d be thrilled to collaborate or contribute.

    Let me know if you'd be open to a quick virtual coffee sometime to share notes! You can reach me at kh.deng@foxmail.com. Either way, huge fan of your work on Highway and Gemma.

  4. jan-wassenberg commented on Mar 26, 2026

    @jan-wassenberg
    Member

    Thanks for the warm words. So far we don't have any plans for highly constrained environments :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions