The Crossroads of AI: Why Wall Street is Saying "No" to ChatGPT and Claude?

Author: Jeff @IOSG

Why Private AI is Needed

On July 1, Palantir CEO Alex Karp contributed a 20-minute interview on CNBC, which some media referred to as a "mental breakdown." According to Karp, companies are paying a token premium to cutting-edge labs while watching their own IP flow to model vendors. He referred to this leakage as the transfer of alpha, which is occurring at the architectural level: every request sent to a closed-source model arrives at the service provider's server in plaintext. Just days before the show aired, Palantir had announced a partnership with NVIDIA to run open Nemotron models in customer-controlled environments, accompanied by a nine-point AI sovereignty declaration. After the CNBC program aired, PLTR jumped 8%.

For the past twenty years, companies have adopted cloud software based on trust at the protocol level, and it has worked. Each SaaS vendor only sees a slice of enterprise data, and most have little incentive to feed customer data back into their core products. Salesforce sees sales channels, Workday sees HR, Jira sees development iterations, and AWS provides the foundation for storage and computing. However, today's AI workflows advocate for the one-time upload of all assets, along with the structured context that connects various departments, in pursuit of maximizing productivity. Setting goodwill aside, upstream service providers can now use this data to develop new features instead of letting it sit on servers gathering dust.

No one is slowing down; Anthropic's annual revenue reached $47 billion in May, a significant leap from $9 billion at the end of 2025, while OpenAI surpassed 900 million weekly active users in February. Both companies completed new rounds of financing this spring, with valuations approaching $1 trillion, and are expected to IPO at even higher valuations. Years of privacy and IP allegations have not caused either company to lose momentum.

Some companies have already taken action. In February 2023, less than three months after ChatGPT launched, major Wall Street banks restricted its use. In May 2023, after a Samsung engineer leaked chip source code into ChatGPT, the company banned generative AI across its network. In response, OpenAI launched ChatGPT Enterprise in August of that year, promising not to use commercial data for training, along with a zero-data-retention (ZDR) protocol, which subsequently became a standard requirement for enterprise procurement.

However, contracts only lock in company accounts. IBM found that by 2025, shadow AI (employees feeding company data into unauthorized AI tools through personal accounts) was involved in one-fifth of data breach incidents, and heavy shadow AI usage added an average of $670,000 to breach costs. In a 2025 survey by security training company Anagram, four employees indicated they would be willing to violate AI usage policies to complete tasks faster.

Companies can at least spend money to buy their way out, with ZDR contracts and non-training service tiers, and if you are a government or Palantir client, there are sovereign deployment options. However, for ordinary users like you and me, the importance of privacy AI remains debated until a court summons arrives at the door.

A court order in May 2025 forced OpenAI to retain even consumer chat logs that users had deleted, and in November, a judge ordered the transfer of 20 million of those logs to lawyers for The New York Times as evidence. Then came criminal cases: ChatGPT records from the defendant in the Palisades arson case entered evidence, and an affidavit in a double homicide case in Florida cited the suspect's questions about how to dispose of a body. Sam Altman also admitted in a July 2025 interview that ChatGPT conversations are not protected by legal privilege, and in litigation, OpenAI "may be required to hand over" user chat logs.

The point is that it's not just criminals who need private conversations. Conversations between people and AI being archived and subject to subpoena represent a surveillance aspect that most users are unaware of. A survey by Kolmogorov Law in October 2025 of 1,000 American AI users found that 50% were unaware that these conversations could be subpoenaed, while two-thirds believed that these chats should receive the same protection as consultations with lawyers or doctors.

Self-hosted or open-source models running in verifiable environments are quickly catching up, but the strongest batch still lags behind cutting-edge closed-source models by about four months in general capabilities. This places enterprises and individuals engaged in tokenmaxxing at a crossroads: either sacrifice a few months of model quality for this privacy or continue uploading sensitive materials to Anthropic's servers, as competitors are gaining productivity advantages this way.

Currently, there are no perfect solutions on the market. A report outlines attempts by various parties to narrow the gap and observes how far cutting-edge intelligence under provable privacy is from being delivered to enterprises and ordinary users.

How Privacy is Currently Achieved

Privacy AI is not a single project, but every mechanism on the market currently addresses the same event: a prompt leaves your device, traverses the network, lands on a machine running the model, and then returns a response. The difference between mechanisms lies in where plaintext exists along this path, who can read it there, and what is used to verify the privacy of the response.

Protocol-Level Privacy
At this level, besides you, others can read your plaintext prompt, and what happens next relies entirely on a promise.

• Contractual Zero Retention is the enterprise solution. The service provider knows who you are, processes your prompt, and promises not to retain it, enforced by contracts and reputation.

• Anonymous Proxies erase who you are but do not encrypt what you said; downstream service providers still handle plaintext according to their own policies. Terms vary by provider; for example, Duck.ai (DuckDuckGo's chatbot product) may negotiate deletion agreements with model vendors, while Venice assumes users will assume the service provider will retain everything, but neither side can verify.

Every segment of the route between machines runs on TLS, which only encrypts the pipeline; the receiving party can read all information. Relays typically use Oblivious HTTP (RFC 9458) to separate this right to know, functioning like a friend passing a note. The friend knows who passed it but cannot read the content; the recipient can read the content but does not know who wrote it. OHTTP became an IETF standard in January 2024, and many companies now run production traffic on OHTTP relays rented from Cloudflare and Fastly.

This is also the privacy limit achievable when accessing closed-source models, due to a mathematical problem. The current cost of flagship-level training is in the billions, and these labs' nearly trillion-dollar valuations are based on the exclusivity of model weights. The duration of the model capability gap dictates the duration of the premium; thus, labs guard weight files as state secrets.

Meta has passively conducted this experiment. The LLaMA model released in February 2023 was initially only open to researchers, but within a week, the weights leaked to 4chan in seed form. A week later, llama.cpp allowed the smallest 7B model to run locally on a MacBook, and three days later, Stanford fine-tuned a chat assistant, Alpaca, on the same model for under $600. This leak reduced Llama's operating costs to electricity, allowing anyone with the file to run it at home. In July 2023, Meta officially open-sourced Llama 2 with a commercial license that excluded a clause for 700 million monthly active users. The weights ran, and the premium followed.

In theory, cutting-edge labs could provide attestation (remote proof) for inference on closed-source models, but attestation can only prove which segment of code read the prompt; it cannot prove what that code did with it. To determine whether the server retained data, we need to audit the serving code and reconstruct it to the hash reported by the hardware. However, once the serving code is handed over, the lab also relinquishes the batch processing and caching techniques that support profit margins, and these techniques will migrate to every future generation of models. Apple and Meta can provide remote proof for the service stacks behind iPhone and WhatsApp because their profits come from devices and advertising, and publicly disclosing service code incurs almost no cost.

This is why the weights and service code of flagship models do not reach external operators. Without external operators, there is no third-party attestation; without attestation, verifiable privacy exists only on open-source models.

Structural-Level Privacy
Each mechanism in this category replaces trust promises with hardware, cryptographic, or physical proofs, but each incurs different costs for privacy upgrades, primarily because they can only run open-source models.

• TEE (Trusted Execution Environment) Confidential Computing runs inference inside a hardware enclave (a sealed chamber on the chip that even the machine operator cannot open); the chip will sign an attestation detailing which model and segment of code was run.

• The prompt is only sealed at the endpoint. There remains a role along the path mediated by the proxy that can read plaintext, and only the protocol prevents the proxy from recording or leaking the content of the relay.

• E2EE (End-to-End Encryption) seals off readable relays. The user's device encrypts the prompt with the enclave's key, and every hop in between carries a sealed envelope that only the enclave can open.

• Trust falls on the client side. The code responsible for encrypting the prompt and verifying the attestation also has the ability to revoke this guarantee. Therefore, verifiable E2EE requires both a proven enclave and open, reproducible client code.

• Compared to the simplicity of TEE, the cost of E2EE is the engineering burden, which also slows down functional integration. E2EE turns the proxy into a blind messenger, so all functions that rely on reading plaintext must be rebuilt around the client key or only reconstructed internally within the enclave.

• FHE (Fully Homomorphic Encryption, and its MPC variants) completely removes the trusted party. The server performs computations on ciphertext in a locked box that it can never open; the key is only in your hands, while MPC (Multi-Party Computation) splits the prompt into secret shares distributed among multiple parties, achieving the same effect unless all participants conspire.

• The cost is speed. FHE natively only performs addition and multiplication, so the nonlinear steps required for transformer operation must be reconstructed at a high cost. The inference cost on ciphertext is 10,000 to 100,000 times that of plaintext, with each token on small models taking seconds to minutes, while under unencrypted conditions, it only takes milliseconds.

• Chips customized for encrypted computation are expected to narrow the gap, but the first prototype will not complete its demo until early 2026, and commercial versions will take a few more years.

• Local Inference directly eliminates this path. The model runs on your own hardware, with no relay, no server, no service provider, and no verification requirements.

• The obvious costs are expense and model capability. gpt-oss-120b scores about half that of GLM-5.2 on the Artificial Analysis index, but its size is 65GB, exceeding the combined VRAM of two flagship gaming graphics cards on the market. The full-precision GLM-5.2 can only run on an 8-card data center node, costing over $300,000 just for the GPUs.

However, beyond these structural limitations, the cost of putting inference into enclaves is decreasing. In single-card inference, enclave cloud service provider Phala's benchmarks show that the throughput loss for enclave mode H100 averages less than 7%, and is nearly zero on large models because the main cost lies in moving data into the chip, not in computing inside it. In multi-card inference, NVIDIA's next-generation GPU Blackwell now supports direct encryption of inter-chip traffic, while the older H100 can only achieve the same effect by routing through the CPU host at one-seventh of the bandwidth. NVIDIA's own benchmarks on Blackwell show that the throughput loss for a 397B model in enclave mode is less than 8%. With these advancements, the performance loss of privacy inference is no longer a decisive constraint.

In fact, the enclave itself adds almost no extra operational cost for the operator. Every H100 after 2023 comes with enclave mode, and the additional cost is the throughput loss caused by encryption, not extra chips. Currently, the rental price for confidential H100 SKUs on Azure is still $8.90 per hour, while without enclave it is $6.98, representing a 27% markup over traditional cloud facilities. On the other hand, operators like Phala that specialize in enclaves are renting confidential mode H100s starting at $3.80 per hour, lower than the price range of $3.99 to $4.29 for Lambda's standard SXM cards. In hosted API solutions, NEAR AI offers endpoints with attestation at $0.15 per million tokens input and $0.55 output for gpt-oss-120b, comparable to plaintext routes on Amazon Bedrock, Together, and Groq. Even for models requiring multi-chip parallelism, NEAR AI's pricing on GLM 5.2 is identical to Fireworks, and on the larger Kimi K2.6, inputs are 15% cheaper and outputs 4% cheaper.

Although these new privacy inference providers may be burning profits to grab market share (this applies to any company in the market looking to grow), the structural trend is that the cost of privacy is decreasing for both consumers and operators.

How Open-Source Models Win

Despite the performance overhead being compressed, there remains a visible gap between cutting-edge models and SOTA open-source models. An entity pursuing maximum productivity still needs to trust cutting-edge labs not to steal its IP.

The gap still exists, but AIA Labs under Bridgewater and Thinking Machines provided a case on June 30: an open model fine-tuned with expert annotations simultaneously outperformed a cutting-edge model in both accuracy and cost.

In the study, the team fine-tuned Qwen3-235B on Tinker (Thinking Machines' hosted fine-tuning API service). They first procured annotations from the vendor, trained the first round with this batch of data, and then sent the divergent samples back to the company's investment personnel for re-annotation. The training used reinforcement learning (GRPO) along with three modifications: round-robin batching (each task takes turns producing a batch), CISPO loss (limiting how far a single answer can pull the model), and on-policy distillation (anchoring to the current optimal checkpoint to ensure the model does not learn from weaker copies).

All tasks were drawn from the daily workflows of investment personnel: whether a news article is important to C-suite investment professionals, whether a central bank document suggests future interest rate changes, and where to start with template phrases in a document or email. Scoring came from an independent test set, with cutting-edge models averaging about 50% on simple prompts and only reaching 78.2% with expert prompts, below the 80% threshold set by investment personnel. The fine-tuned Qwen achieved 84.7%, which, according to the original text, means it made 29.8% fewer mistakes than the cutting-edge optimal model, with inference costs 13.8 times lower.

https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/
This case demonstrates that open-source models can win in both accuracy and cost, but the training process is still not private. The expert annotations used in the process are proprietary data from Bridgewater, passing through Tinker’s third-party service, landing at the same trust level as the ZDR protocol. The fund also rented computing power, and the entire training ran on machines it never controlled. Buyers wanting this formula without the trust assumption have very few options today. Renting bare GPU clusters makes the training process readable to cloud operators. Buying a cluster solves the data hosting problem, but costs skyrocket.

The route with attestation has just arrived. In March, Workshop Labs and Tinfoil released Silo, a post-training stack running in Tinfoil enclaves on a single 8-card node, with keys controlled solely by the customer. The article states that the enclave cost adds 11 minutes to a two-hour training session, and this stack can accommodate a trillion-parameter model (Kimi K2 Thinking) by freezing base weights and only training small adapters on top of them. The challenge lies in the fact that reinforcement learning requires moving data back and forth between components, and moving data is precisely where the enclave costs arise.

Less than a month after Silo's release, Workshop Labs was acquired by Thinking Machines, and all the components needed to run the next Bridgewater-style RL loop in the enclave are now under the same company.

Privacy at the Harness Layer

There is another issue that lies outside all private inference mechanisms. These mechanisms each manage the path from prompt to model, while every external tool call initiated by an agent opens a route that the inference layer fundamentally cannot touch. The recent trend of harness engineering amplifies the problem; every tool, memory store, and data source surrounding the model is yet another destination that reads its slice of the workflow in plaintext. Calendar servers read schedules, database servers read queries. A completely local agent still needs to submit search terms in plaintext to a search engine if it wants anything outside the training set, as the server cannot read plaintext and thus cannot answer questions.

Mainstream solutions still default to the protocol layer. Companies like Runlayer and MintMCP use a central gateway to control all tool traffic, masking personal identifiable information (PII) before requests leave. The gateway also decides which servers can receive traffic, blocking unvetted ones, and logs the destination and content of each call for future reference. Even if these controls carry independent audits (SOC 2), tool servers still need to read plaintext queries to respond, and whether they retain copies depends on their own retention policies, multiplied by every tool in the harness. Moreover, the gateway itself is an additional reading party dependent on trust along the path, rather than verification.

Structural-level solutions target that middle layer. For example, Phala directly hosts the MCP server within a TEE, covering wallets, code execution, and data sources, allowing users to verify privacy statements with an attestation rather than trusting the operator. However, tools hosted in TEE still need to submit queries in plaintext to service providers; the enclave only seals the messenger, not the destination.

Only a few destinations have learned to respond without reading, but only for structured queries. Apple provides private information retrieval for the iPhone, allowing caller numbers to match against spam call databases without exposing the number, and Microsoft uses the same scheme for passwords in the Edge browser. MongoDB's Queryable Encryption allows clients to encrypt fields before they leave, enabling the server to perform equality and range matching based solely on ciphertext.

However, for open-ended searches, the best answers today still rely on trust; verifiable encrypted search has yet to emerge from the lab. Brave promises zero data retention on its own index of 40 billion pages (rather than Google's), but it still falls under the protocol layer. Exa built a neural index that embeds user keywords semantically, matching results based on semantics, but this embedding step still starts from plaintext on Exa's servers. MIT's 2023 Tiptoe paper achieved ranking on 360 million web pages without exposing queries, but each search consumes a significant amount of server computing power, and the ranking quality differs from unencrypted searches. Apple's 2024 Wally paper reduces communication costs by up to 31 times by hiding real queries among decoys, but this math only becomes cheap at millions of concurrent queries, a scale that no private search system currently possesses.

Encrypted search is feasible; it just hasn't reached a commercially viable level in terms of performance and price.

Outlook

The demand for private AI is growing. Venice AI recently surpassed 3.5 million registered users and a monthly throughput of 1.3 trillion tokens, subsequently completing a new round of Series A equity financing valued at $1 billion. Proton is its direct competitor, with its chat product Lumo exceeding 10 million users within a year of launch. In terms of infrastructure, Phala is currently running 2 to 3 billion tokens daily on OpenRouter. Duck.ai routes gpt-oss-120b and Gemma into Tinfoil's enclave, providing verifiable privacy beyond user proxies. This doesn't even account for self-hosting, which is likely the largest channel for private inference, as models run on one's own hardware, leaving no usage traces.

However, in the broader wave of mainstream AI, privacy AI occupies only a tiny portion, and this gap will only close when cutting-edge labs intentionally meet this demand. In May, Google processed 320 trillion tokens across all its products, meaning Venice's monthly throughput is roughly equivalent to Google's 18 minutes. Last November, Google launched Private AI Compute (PAC), running some Gemini-driven features in sealed TPU enclaves isolated from the company itself, with design independently audited by NCC Group. However, the problem is that PAC only covers a few Pixel features like personalized recommendations and audio summaries, not the Gemini applications used by hundreds of millions. Google is willing to hand over designs to auditors because these features monetize through devices and advertising, not by selling tokens.

Current hosted solutions are also imperfect. Users seeking the highest privacy through E2EE must wait for new features to be rebuilt in places that service providers cannot read. Private harnesses still rely on protocols at the service layer. Reasonably priced post-training still requires trusting third-party suppliers to achieve the best fine-tuning results. Self-hosting completely eliminates all service providers, but running the strongest open-source models locally may cost more than the building housing them.

Flaws aside, private AI has become a real and affordable option, and the remaining gaps are narrowing. For ordinary consumers, open model private chats on Lumo and Venice under no-logs promises cost nothing, while subscriptions from Venice or Tinfoil at $18 to $20 encapsulate the same chats in enclaves, not more expensive than a ChatGPT subscription. For enterprise workflows, endpoints with attestation are now even cheaper than plaintext routes. Endpoints like NEAR's E2EE API can now bring encrypted context into enclaves, with memory, file uploads, and custom instructions operating on top of E2EE today. As for post-training with attestation, NVIDIA's upcoming Vera Rubin NVL72 will extend confidential computing from Blackwell's 8-card nodes to 72-card racks, making cutting-edge RL loops more feasible without exposing IP.

However, the key value capture lies beyond these price compression levels. Privacy is nearly free where it already exists, but it hasn't covered mainstream agentic workflows. Operators focused on renting and selling enclaves hold a switch on standard chips, not a moat, while protocol layer gateways compete alongside traditional middleware. The defensible territory is the half of this report that remains unsolved: training loops locked in enclaves, end-to-end sealed tool calls, and invisible term search indexes. Whoever first accomplishes one of these will sell something that cannot be commoditized by any price war. Capital chasing privacy AI should be buying the gaps, not that switch.

So, trust or verify? For tasks that are execution-heavy and agent-heavy, choose trust, because every tool call inherently delivers plaintext to destinations that enclaves cannot seal, and cutting-edge models deserve their price in such loops. As for high-level thinking that distinguishes a company from its competitors, choose verification. Strategy, planning, and judgments distilled from years of professional experience are precisely that contested alpha. The way forward is to fine-tune open-source models with these proprietary insights within the boundaries of company control. In the domains where a company's alpha resides, expert-tuned open models have already simultaneously outperformed cutting-edge ones in accuracy and cost, and the infrastructure to build them in privacy environments is arriving node by node.