Self-Hosting Your Gemini Stack on OpenMetal

1. What runs where

Gemini is closed. The flagship runs on Google’s infrastructure, or on Vertex AI, and never on a machine you control. (Google’s open-weight cousin, Gemma, you can run yourself, but it is not the flagship.) That fixes one row of the stack on the provider’s side; the rest can be yours.

The only data that must leave is the prompt and its completion. Your egress proxy decides what each prompt actually carries.

2. Deployment options for Gemini

Google offers three practical routes for keeping more of the system inside your boundary. None puts the model on your hardware.

• API direct (Google AI). Your application on OpenMetal calls the Gemini API with your key. On enterprise and Vertex deployments, Google does not use stored data to train its models. You should verify the data terms attached to your specific access tier.
• Vertex AI (cloud-partner route). The route with the strongest controls. Vertex supports HIPAA, customer-managed encryption keys, and VPC Service Controls; combined with Private Service Connect, VPC-SC can fully isolate Gemini API traffic at the network level, which became generally available in 2025. Data residency is set by region selection at project creation under the Google Cloud Data Processing Addendum, and zero-data-retention-equivalent terms are available for eligible enterprise customers via a DPA amendment. You should current VPC-SC scope, regions, and ZDR-equivalent eligibility.
• Self-hosted UI. An open-source chat front-end on your OpenMetal servers, connected to the API. Your users, prompt templates, and conversation history live in your database.

3. What you host on OpenMetal

Every layer you own here is ordinary compute. It does not need a GPU, and it benefits from single-tenant hardware, a private network, and predictable cost.

On a Hosted Private Cloud or Bare Metal deployment, the chat UI, your agent tooling, the retrieval corpus and its vector database, your logs, and the egress proxy all run on hardware only you use. Traffic between them is east-west private-network traffic on per-customer private networks, included rather than metered, so your retrieval and logging layers can be as busy as they need without a transfer bill. Public egress ships with a generous included allotment, which is all the Gemini calls need.

The guardrail and egress-proxy tier is the part to design deliberately. Sitting at the single north-bound egress point on hardware you control, it inspects every outbound prompt: redacting or tokenizing fields, applying policy on what may be sent, and recording exactly what crossed. On the Vertex route you can pair this with VPC Service Controls so the API traffic itself is confined to a service perimeter you define. Either way, the rule for what leaves is yours to write and yours to audit.

4. The honest ceiling

State the limits plainly, because they are what make the rest trustworthy.

• Gemini never runs locally. The flagship is closed. There is nothing to download and nothing to load on an OpenMetal GPU. If you need weights you can host, that is Gemma, the open-weight cousin, not the flagship.
• Prompts and completions still transit to Google or Vertex AI. The no-training-on-stored-data commitment, optional ZDR-equivalent terms, and VPC-SC isolation govern how and where that data is handled, but they do not keep it inside your network. Sovereignty here is boundary control, not zero egress.
• A Gemini BAA runs through Google Cloud, and is separate from OpenMetal’s HIPAA. Gemini can be used in HIPAA-regulated workflows through Vertex AI on a HIPAA-aligned Google Cloud account with a BAA signed with Google. That agreement covers Google’s processing, not your infrastructure, and OpenMetal’s organization-level HIPAA posture covers your infrastructure, not the API call. The two are additive and both are required for regulated data. You should verify current BAA scope and eligible Vertex services against Google’s terms.
• Air-gapped or classified workloads exclude Gemini entirely. If nothing may leave the network, no Gemini route qualifies. Route those workloads to an open-weight model on your own OpenMetal GPUs instead.

5. When local GPUs enter (hybrid)

The data-plane tier and the full-sovereignty tier are not mutually exclusive. A hybrid design routes the bulk of traffic, and anything too sensitive to send out, to an open-weight model running on your own OpenMetal GPU servers, behind the same egress proxy, and escalates only the hardest prompts to Gemini’s API. That shrinks what leaves your boundary and trims token spend at the same time. The open-weight side of that design is our Model Deployment Fit series, where models like Llama and Gemma run on OpenMetal H200 and NVIDIA RTX PRO 6000 servers, each of which uses discrete GPUs rather than a pooled accelerator.

6. Running your Gemini stack on OpenMetal

The reason to run this stack on OpenMetal rather than a hyperscaler is that the data-plane tier is only worth it if the boundary is real. Single-tenancy means your vector store and your logs sit on hardware no other tenant shares. Per-customer private networking means the traffic between your app, your retrieval layer, and your proxy never touches the public internet. Included egress and fixed monthly pricing mean the architecture does not penalize you for keeping data close.

There is a strategic payoff as well. Because Gemini sits behind your own proxy, your data plane is provider-neutral: you can point it at Gemini today, add another provider tomorrow, or fail over between them, without moving your corpus, your logs, or your guardrails. You own the substrate, and the model is a swappable backend. That is the opposite of building your whole stack inside one vendor’s managed platform.