hdx-eval reports unparseable MCP responses as successful evidence #3526

Summary

queryTileWithEvidence in packages/hdx-eval/src/hyperdx/api.ts calls an MCP
tool and parses the textual result as JSON. When the MCP response is a
transport success with isError: false but its text is not valid JSON, the
catch branch returns { success: true, hasData: true } instead of reporting
a parse failure. The eval framework therefore treats unparseable content as
valid evidence.

Environment

• Repository: hyperdxio/hyperdx
• Target revision: 1773c11942a73b466637436e263b972fe1a16361
• File: packages/hdx-eval/src/hyperdx/api.ts:365-372
  (queryTileWithEvidence)

Expected behavior

If the MCP content cannot be parsed as JSON, the call reports failure (for
example { success: false, error: ... }), so grading never treats the raw
text as data.

Actual behavior

A transport-successful response whose content.text is not JSON returns:

{"success": true, "hasData": true}

Reproduction:

node --experimental-transform-types repro.ts
# {"success":true,"hasData":true}

hasData is derived only from content.text.length > 0, so any non-empty
garbage counts as data.

Root cause

try {
  const inner = JSON.parse(content.text);
  ...
  return { success: true, hasData, rowCount: rows.length, ... };
} catch {
  return { success: true, hasData: content.text.length > 0 };   // parse error -> success
}

The catch conflates "the MCP server returned non-JSON text" with "the tool
returned data", and discards the parse error.

Impact

• The eval harness can record fabricated or malformed tool output as successful
  evidence and grade an agent against it, producing false positives in
  benchmark results.
• Parse failures are invisible to callers, so a misbehaving MCP server looks
  healthy and the raw text is never surfaced as an error.

Suggested fix

Return a failure result from the catch branch and keep the parse error:

} catch (error) {
  return {
    success: false,
    hasData: false,
    error: `Invalid MCP content: ${(error as Error).message}`,
  };
}

If raw text is useful for diagnostics, return it in a separate field rather
than marking it as parsed data.