Metadata-Driven Application Runtime

This repository implements a Metadata-Driven Application Runtime. Instead of hardcoding routes, database schemas, or validation rules for specific database entities (e.g. Contacts, Accounts, Leads), this system generates them dynamically at runtime based on JSON Metadata (either supplied statically, or generated using the Gemini AI API).

The system features user-session auto-generation, multi-tenant record isolation, and per-user UI folder/form generation.

---

📂 Project Directory Structure

src/
├───ai/                      # Integration with Gemini AI
│   ├───ai.prompt.ts         # System instructions for AI schema generation
│   ├───ai.service.ts        # Client service invoking GoogleGenAI sdk
│   └───ai.types.ts          # Typing definitions for AI outputs
├───auth/                    # Middleware & Session handling
│   └───auth.middleware.ts   # Cookie-based session generation & parsing
├───modules/
│   ├───metadata/
│   │   └───metadata.types.ts # TypeScript interfaces for user-defined metadata
│   └───records/             # Storage, CRUD orchestration, and Validation
│       ├───crud.engine.ts    # Main logic orchestrator for DB CRUD actions
│       ├───memory.store.ts   # Key-Value store simulating database records
│       ├───payload.validator.ts # Validates incoming request body against schema
│       └───validation.types.ts # Validation return types
├───normalizer/              # Sanitization & Fallback rules
│   ├───app.normalizer.ts    # Normalizes application level schema & metadata
│   └───field.normalizer.ts  # Normalizes individual field constraints (types, defaults)
├───runtime/                 # The Engine that compiles metadata into Express routes & HTML
│   ├───api.generator.ts     # Generates entity-specific secured endpoints (e.g., /contact)
│   ├───dynamic.router.ts    # Catch-all router mapping route variables to CrudEngine
│   ├───entity.registry.ts   # Catalog mapping entity names to their configuration objects
│   ├───metadata.compiler.ts # Compiles raw metadata into a structured runtime schema
│   ├───runtime.manager.ts   # Stores compiled runtime settings
│   ├───runtime.schema.ts    # Types for compiled runtime schemas
│   ├───runtime.types.ts     # Types for database records (id, ownerId, payload data)
│   ├───ui.file.generator.ts # Generates user-specific form directories and files
│   └───ui.generator.ts      # Renders HTML structures from field definition metadata
├───shared/                  # Constants, custom warning definitions, and errors
│   ├───constants.ts
│   ├───errors.ts
│   └───warning_type.ts
├───validator/               # Zod validation schema
│   └───app.validator.ts     # Safe-parses raw incoming metadata structure
├───workflow/                # Lifecycle hooks
│   └───workflow.engine.ts   # Simulates running workflows (e.g. on CREATE/UPDATE/DELETE)
├───index.ts                 # Main startup file orchestrating system compilation
└───server.ts                # Configures the Express web application

---

🔄 The Data Journey (Step-by-Step Flow)

To understand how data flows through the application, let's look at the initialization and runtime cycles:

graph TD
    A[index.ts Entry] --> B(ai.service.ts)
    B -->|Generates metadata| C(app.validator.ts)
    C -->|Validates structure| D(app.normalizer.ts)
    D -->|Sanitizes fields & registers warnings| E(entity.registry.ts)
    E -->|Catalog entity schemas| F(metadata.compiler.ts)
    F -->|Compile runtime configurations| G(server.ts)
    G -->|Start HTTP Server & Attach middleware| H(api.generator.ts & dynamic.router.ts)

Loading

1. Startup & Schema Compilation

1. Metadata Retrieval (index.ts → ai.service.ts): The program asks the Gemini AI to design a schema (e.g., a CRM app schema). If the API is unavailable, the service gracefully falls back to a mock CRM configuration.
2. Structural Validation (app.validator.ts): The schema JSON goes through Zod to verify it is structured correctly (e.g. entities have names and arrays of fields).
3. Normalization (app.normalizer.ts & field.normalizer.ts): Sanitizes and updates any missing fields (like setting default rules or logging warnings if a field's configuration is incomplete).
4. Registration (entity.registry.ts): The normalized entity definitions are cataloged case-insensitively in EntityRegistry to avoid duplicate setups.
5. Compilation (metadata.compiler.ts): Turns the user-facing definitions into a compiler-optimized schema representation, stored in RuntimeManager.
6. Express Server Boot (server.ts & api.generator.ts): Express starts listening on port 3000. api.generator.ts reads the registered schemas and spawns standard dynamic endpoints (e.g., /contact, /opportunity) for each entity.

2. Runtime HTTP Request-Response Flow

sequenceDiagram
    participant Browser as Browser Client
    participant Auth as auth.middleware.ts
    participant Router as api.generator.ts / dynamic.router.ts
    participant Engine as crud.engine.ts
    participant Validator as payload.validator.ts
    participant Store as memory.store.ts

    Browser->>Auth: HTTP Request (GET/POST/PUT/DELETE)
    Note over Auth: Check for user-id cookie.<br/>Generate user-UUID if missing.
    Auth->>Router: Sets req.userId & passes control
    Router->>Engine: Call CRUD action with req.userId
    Note over Engine: Verify record ownership (if modifying/deleting)
    Engine->>Validator: Validate payload fields against metadata
    Validator-->>Engine: Validated & Sanitized Data
    Engine->>Store: Perform DB lookup or persistence
    Store-->>Engine: DB Result
    Engine-->>Router: Response (Success / Record / Error)
    Router-->>Browser: JSON Output + Set-Cookie Header

Loading

---

🔍 Detailed File & Module Explanations

1. auth.middleware.ts

• Why it exists: Secures the application and assigns unique identities to clients.
• How data is transferred: Intercepts every HTTP request. If the incoming request has no x-user-id header or user-id cookie, it creates a new ID (e.g. user-ab8700ec...) using crypto.randomUUID(). It sets this in the response header as an HttpOnly cookie, and binds the value to req.userId to pass it down the Express middleware stack.

2. crud.engine.ts

• Why it exists: Serves as the central backend logic controller.
• How data is transferred: Takes queries from routing files, validates input payloads against the registry configuration using PayloadValidator, checks ownership using the request's userId, and writes or reads records from the MemoryStore.
• Tenant Isolation: When calling update() or delete(), it retrieves the existing record first and blocks the request if the client's userId does not match the record's ownerId, preventing unauthorized edits.

3. memory.store.ts

• Why it exists: Acts as our database (currently in-memory, but abstraction matches SQL/Postgres engines).
• How data is transferred: Receives records (structured with id, ownerId, and data fields) from CrudEngine and updates its internal storage Map.

4. ui.file.generator.ts & ui.generator.ts

• Why they exist: Generate user interface assets dynamically.
• How data is transferred: When a user accesses GET /:entity/form, api.generator.ts uses generateUserUIFiles(userId, entities) to parse the entity schemas. It writes a distinct user-isolated HTML form file to generated-ui/{userId}/{entity}.html. The generated forms submit POST requests back to the dynamic entity routes.

---

💡 Key Lessons

1. Keep Layers Separated:
   • MemoryStore doesn't care about API logic or permissions; it just saves and reads files.
   • CrudEngine manages business rules, validation, and security check constraints.
   • api.generator.ts and dynamic.router.ts only handle parsing incoming HTTP headers, bodies, cookies, and formatting outgoing JSON/HTML.
2. Case Insensitivity: Always normalize lookups (e.g. converting name.toLowerCase()) when matching URL parameters (which are often lowercase like /contact) with structured data models (which are often PascalCase like Contact).
3. Data Isolation: Never trust client inputs (like headers) blindly without checking database record owners. Always cross-reference request credentials (req.userId) against database resource metadata (record.ownerId).