To MCP or not to MCP

Making Sense of the MCP Landscape

Introduction

A few months back, Google CEO Sundar Pichai posed the question "To MCP or Not to MCP" and sought others' views. He was referring to the Model Context Protocol (MCP) released by Anthropic late last year. His quote coincided with Google's release of the Agent2Agent (A2A) protocol. Both MCP and A2A are complementary protocols that can coexist.

Previously, users would ask questions and Large Language Models (LLMs) like ChatGPT, Gemini, Nova and numerous others would provide responses. This worked well but had one limitation. The models would only respond based on their pre-training data. Users would often see messages like "My training data includes information up until April 2024, and I don't have information after that date." To address this limitation, LLMs began using Retrieval Augmented Generation (RAG) to incorporate new information. While RAG systems primarily retrieve relevant information from knowledge bases to augment LLM responses, MCP offers broader capabilities. MCPs enable LLMs to invoke external tools and APIs. RAG typically requires custom integrations with specific data sources, which becomes challenging to manage as external systems multiply. MCP provides a standard for connecting AI systems to external sources, making integrations more streamlined and modular. This allows LLMs to discover and use various tools and data sources without requiring individual integrations.

MCP Architecture

MCP follows a client server architecture where the host application connects to multiple servers. An MCP Host is an application with an embedded MCP client that interfaces with servers. The key distinction between a standard client application and an MCP Host is that the MCP Host interacts with an LLM. When processing user requests, it leverages MCP servers to provide more relevant and accurate responses.

<Source:https://modelcontextprotocol.io/introduction>

Key Components:

• MCP Hosts: Programs such as Claude Desktop, IDEs, or AI tools that access data through MCP

• MCP Clients: Protocol clients maintaining one to one connections with servers

• MCP Servers: Lightweight programs exposing specific capabilities through the standardized Model Context Protocol

• Local Data Sources: Computer files, databases, and services that MCP servers access securely

• Remote Services: External systems available through the internet that MCP servers can connect to

  
  Evolving MCP Architecture

  The rapid adoption of MCP in the last six months has revealed the need for additional components. Authentication and Authorization remain under development, with OAuth emerging as the preferred authorization approach. The proliferation of servers and their supported tools creates new challenges. A fundamental question arises: How does an MCP Host discover all available servers? This need led to the creation of the MCP Registry, which functions as a directory service where servers can register themselves.
  
  While the registry solves server discovery, managing connections to these servers presents another challenge. The MCP Gateway addresses this by serving as the primary interface for server interactions. Although the gateway handles remote MCP server communications, local server interactions require special consideration. Teams can choose to let the MCP Host interact with local servers directly or implement an MCP Proxy to manage both local and remote server communications uniformly.

  

  Key Components:

  • MCP Registry: A central service discovery component that maintains a registry of all available MCP servers, services, and tools, allowing components to locate and communicate with each other dynamically

  • MCP Gateway: Acts as a routing and load balancing layer that sits between MCP clients and servers, managing and directing traffic while providing a single entry point for client requests

  • MCP Proxy: A component that facilitates communication between MCP Host and rest of the MCP ecosystem

    
    Challenges

    As we continue building components of the MCP ecosystem, we need to assess which parts are essential and which are optional. While MCP helps LLMs provide more relevant and accurate information, several challenges must be addressed before implementation becomes unwieldy.

    1. Tool Proliferation and Selection Overhead

    The widespread adoption of MCP has led many teams to automatically generate MCP servers from their existing OpenAPI or Smithy API definitions. This introduces two key challenges:

    • Context Window Consumption: Each tool description consumes valuable context window space.

    • Tool Selection Complexity: LLMs struggle to select appropriate tools when presented with too many options, leading to incorrect selections and hallucinations.

    Solution: Dynamic Tool Selection

    The tool proliferation challenge can be addressed through Dynamic Tool Selection. This approach enables the MCP Host to identify the most suitable tools based on user requests. By limiting exposure to the top N relevant tools, we reduce context window consumption and minimize LLM confusion. Teams can also proactively limit their MCP servers to expose only essential tools, eliminating the need for dynamic selection mechanisms.

    2. Documentation Quality Misalignment

    Traditional API documentation practices do not align with LLM requirements. Engineers often rely on self-documenting code or technical writing teams to create developer-focused documentation, but LLMs require a different style of documentation to function effectively.

    Solution: LLM-Optimized Documentation

    Tool descriptions must be crafted specifically for LLM consumption. Unlike developers, LLMs rely on multiple structured examples to generalize tool usage across contexts. Documentation should provide rich contextual information, clear descriptions of the tool’s purpose and limitations, and multiple usage scenarios. One effective approach is to use LLMs themselves to generate initial documentation, then have human experts refine these observations into comprehensive descriptions.

    3. The Pitfall of Direct API-to-Tool Mapping

    Autogenerated MCP servers expose all API endpoints as tools, resulting in granular operations that lack proper abstraction for effective LLM task execution.

    Solution: Task-Oriented Tool Design

    MCP tools should be designed with specific tasks and LLM interactions in mind. Creating purpose-built, high-level tools that encapsulate related operations improves LLM task execution capability and reduces complexity. For example, rather than exposing individual API endpoints for user management, a single comprehensive “createUserProfile” tool can handle all aspects of profile creation internally. This strategy enhances both the reliability and interpretability of LLM-driven task execution.

    Conclusion

    Anthropic redefined what MCP means and in doing so gave a powerful boost to LLMs. While MCP servers are undeniably useful, the real focus should be on building the right tools and only as many as are truly needed to increase productivity. Instead of flooding the context with a long list of tools, we should carefully include just the essential ones. The winner will not be the MCP server with the largest number of tools but the one with the right tools for the job. We have learned from past mistakes with monolith services and the overuse of microservices. Those lessons should guide us in building MCP servers that are the right size and fit for purpose. It surprises me that the MCP specification does not yet mention roles, operation modes, or profiles that apply to specific user scenarios. For example, a software developer might want to start by reviewing requirements and working on high-level design, iterating until it is ready. The next step could be building the solution and submitting pull requests for peer review. There is also the testing mode, where the developer focuses on writing test cases, canaries, and so on. Each of these roles requires different MCP servers and tools. Rather than having a single static MCP configuration file, it makes sense for these configurations to be dynamically generated and loaded based on the user’s current role. I would not be surprised if future versions of the MCP spec address this to solve the problem of too many tools.
    
    If you are an MCP server vendor, do not simply rely on auto-generated servers and consider your job done. MCP is a complement to APIs, not a replacement. It acts as a semantic layer that helps models interact intelligently with existing APIs while preserving APIs as the foundation of system-to-system communication. MCP will not replace APIs. Instead, invest time to understand how your customers use your existing APIs and build an abstraction that presents high-level, task-oriented tools. These will be far more effective than exposing low-level APIs that force LLMs to piece together complex workflows. Also, establish an MCP evaluation process similar to how SWE Bench evaluates LLMs. If your MCP server supports developer workflows, you can leverage existing benchmarks to measure how well your server assists with coding tasks.
    
    Lastly, the MCP ecosystem often includes clients, servers, tools, gateways, proxies, and registries. Not all of these components are necessary for every use case. In many cases, a handful of specialized servers and tools will suffice. Do not wait for every piece of the ecosystem to be in place before you start experimenting. You can work around missing features like authentication by running local MCP servers, which removes dependency on other components. MCP is just another tool in your AI toolbox. Treat it that way rather than inflating its importance beyond what it deserves.