sdk	docker
app_port	7860

CodeGuardian Inference Script

This script runs an LLM agent against the CodeGuardian multi-step code review environment.

Setup

Environment Variables

Set the following environment variables before running:

# Required: API endpoint (for OpenAI-compatible APIs)
export API_BASE_URL="https://api.openai.com/v1"

# Required: Model name to use
export MODEL_NAME="gpt-4"

# Required: Authentication token (HF_TOKEN takes precedence over API_KEY)
export HF_TOKEN="your-huggingface-token"
# OR
export API_KEY="your-openai-api-key"

Dependencies

Install required packages:

pip install openai pydantic

Usage

Run the inference script:

python inference.py

What It Does

The script:

1. Initializes the CodeGuardian environment and OpenAI client
2. Loops through all tasks in the environment (3 tasks × 3 steps each = 9 total steps)
3. For each step:
   • Step 0: Detect issues in the code
   • Step 1: Assess risk level (low/medium/high)
   • Step 2: Suggest corrected code
4. Sends structured prompts to the LLM with the current observation
5. Parses JSON responses safely with fallback handling
6. Executes actions in the environment and accumulates rewards
7. Logs progress including rewards, feedback, and running totals

Output Format

2026-04-06 02:26:51,542 - INFO - Starting inference with model: gpt-4
2026-04-06 02:26:51,561 - INFO - Initial task: Return sum of two numbers
2026-04-06 02:26:51,561 - INFO - --- Step 1: DETECT ---
2026-04-06 02:26:51,562 - INFO - LLM response: {"detected_issues": ["wrong operator"], ...}
2026-04-06 02:26:51,564 - INFO - Action type: detect
2026-04-06 02:26:51,564 - INFO - Reward: +0.30
2026-04-06 02:26:51,564 - INFO - Feedback: []
2026-04-06 02:26:51,565 - INFO - Running total: +0.30
...
2026-04-06 02:26:51,586 - INFO - === Inference Complete ===
2026-04-06 02:26:51,586 - INFO - Final score: 0.00
2026-04-06 02:26:51,586 - INFO - Total steps: 9

Reward Structure

• Detect step: +0.3 (correct) / -0.2 (incorrect)
• Risk step: +0.2 (correct) / -0.2 (incorrect)
• Fix step: +0.5 (correct) / -0.3 (incorrect)
• Invalid action order: -0.5

Error Handling

The script is robust against:

• Invalid JSON responses (falls back to empty responses)
• API errors (falls back to empty responses)
• Missing fields (provides sensible defaults)
• Infinite loops (50-step safety limit)

Expected Performance

• Perfect score: 3.0 (all 3 tasks completed perfectly)
• Random baseline: ~0.0 (rewards balance out)
• Typical LLM: 0.5-2.0 (depending on model capability)

The script provides a stable, reproducible baseline for evaluating LLM performance on multi-step code review tasks.