Exposed: MCP Servers Are Lying About Their Schemas

Just wrapped up our latest exploration of MCP servers using Specmatic’s MCP Auto-Test, and we uncovered some fascinating findings! When testing remote MCP servers from Postman, Hugging Face, and GitHub, our team dug deep into schema implementations—and here’s what you need to know:

3 Key Takeaways from Our Automated MCP Testing:

Schema-Implementation Drift is Real Specmatic Auto-Test caught critical mismatches, such as fields marked optional in the schema but required in the actual implementation—leading to frustrating errors and unpredictable behavior for consumers.

Standard Schema Constraints Matter When max/min constraints weren’t declared with standard JSON schema keywords, Specmatic (and any consuming API or LLM) couldn’t validate values correctly. Relying on just descriptions isn’t enough—explicit standards are essential!

Reduce Guesswork, Prevent Hallucinations Leaving schema details vague means automated testing tools & LLMs won’t know the rules. This not only causes failed tests but also introduces risk for downstream consumers. Clear, well-defined schemas keep your API ecosystem predictable and robust.

The power of Specmatic’s automated approach? It flags these misalignments before they reach production—helping us ship AI-ready APIs faster and with confidence.

Watch how to use Specmatic MCP Auto-Test:    • Meet Specmatic MCP Auto-Test: The first MC...  

Learn more at https://specmatic.io

Remote MCP servers we tested
Postman: https://www.postman.com/postman/postm...
Hugging Face: https://hf.co/mcp
GitHub: https://github.com/github/github-mcp-...

#API #QualityEngineering #Specmatic #AIReady #mcpserver