The Inference Stack

💰Capital Shift

From training compute to inference platforms

Foundation Model training investment (2023-2024) has passed its peak and approximately 2/3 of AI capex has shifted to inference execution. At the point where models are sufficiently powerful, the bottleneck moved from 'better models' to 'faster and cheaper execution.' 81% of Q1 2026's $297B is AI — inference platforms and inference chips taking the largest share.

open_in_new startupxo.com/ko/news/2026/05/ai-infra-vc-landscape

Stack Fragmentation

Every layer becomes a separate company

Inference compiler (SGLang), inference chip (Cerebras WSE-3), inference platform (Baseten), inference networking (Nexthop AI) — each stack layer receives independent VC rounds and fragments into separate companies. The web era's OS→DB→middleware→app fragmentation pattern is replaying in ML infrastructure at 10x speed. The thesis that owning one layer well is sufficient has been validated by capital first.

Compiler Layer

Open source → $400M in under a year

SGLang (RadixArk) and vLLM are the de facto standards for inference serving. RadixArk received $100M Seed at $400M valuation from Accel and Spark Capital. The VC round came 6 months after becoming the de facto standard as open source. ML infrastructure is traversing in under a year the path that took MongoDB 8 years and HashiCorp 5 years.

Chip Layer

NVIDIA alternatives go institutional

Cerebras WSE-3 IPO at $26.6B valuation signals NVIDIA alternatives crossing from 'interesting experiment' to 'institutional investor thesis.' Capital concentrates toward reducing sole NVIDIA dependency across the stack: Groq LPU, SambaNova, Nexthop AI ($500M, networking layer). Hardware competition will shift purchasing negotiating power within 2-3 years.

Demand Floor

Top 3 deals = 45.8% of all AI capital

OpenAI, Anthropic, and xAI absorb 45.8% of all AI capital. Every dollar these companies spend on compute becomes infrastructure founders' revenue. A demand base this concentrated is rare even in venture history — for infrastructure layer founders, the most reliable demand base; for attackers, the most attractive target.

🔒Software Trust Layer

pip install is not safer than chmod +x

As ML pipelines grow more complex, the supply chain attack surface expands. Structural vulnerabilities of the ML ecosystem vs. web services: insufficient version pinning, no GPU node auditing, package account hijacking vectors, payload reach extending to model parameters, training data, and GPU compute. Convenience for speed converts to trust costs.

ML Pipeline Attack Surface

Four structural gaps vs. web services

① Version range specification in requirements.txt (torch>=2.0) is routine ② On-demand GPU instances install latest version without image pinning ③ PyPI account hijacking is the attack path for millions-of-downloads packages ④ Malicious payloads can reach model weights, training data, credentials, and GPU resources. The structural cause is that dependency auditing habits are less established in data science environments.

The 42-Minute Window

Fast response ≠ zero exposure

2026-04-30 PyPI lightning 2.6.2/2.6.3 supply chain attack. 42 minutes from community detection to removal — a fast response. However, during those 42 minutes, ML pipelines, CI/CD systems, and local development environments without version pinning received the malicious version. Fast detection is the community's strength, but unless the exposure window reaches zero, defense lies in prevention not detection.

Defense Architecture

Pin → Hash → Audit → Mirror

① Exact version pinning (lightning==2.6.1, pip-compile/poetry.lock) ② Pin Docker images to SHA-256 digest (FROM pytorch/pytorch:2.3.1@sha256:...) ③ Verify download integrity with pip install --require-hashes ④ Integrate pip-audit and safety into CI. Internal Artifactory/Nexus mirrors are the most powerful layer for structurally reducing the attack surface.

🔑Cryptographic Foundation

The standard is set. Migration is now.

Since NIST FIPS 203/204/205 finalization (August 2024), post-quantum cryptography (PQC) has transitioned from 'future plans' to 'infrastructure currently being deployed.' Cloudflare's TLS traffic is 66% X25519MLKEM768 hybrid. Chrome 131 and OpenSSH 10.0 changed their defaults. The logic of 'we'll do it when the standard is finished' was already wrong in August 2024.

HNDL Threat Model

Harvest the ciphertext today. Decrypt after Q-Day.

Harvest Now Decrypt Later: attackers store today's encrypted traffic and bulk-decrypt it on Q-Day. Forrester/PostQuantum.com consensus: Q-Day 2030±2 years. Medical records (decades), defense intelligence (30+ years), financial contracts (10-30 years) — data with confidentiality requirements extending past Q-Day is already within the HNDL risk window.

Q-Day Estimates Narrowing

2030±2 is the current consensus

Gidney (2025, arXiv 2505.15917): RSA-2048 factoring in under a week with fewer than 1 million qubits. IBM Quantum Blue Jay roadmap: 2,000 logical qubits by 2033. The estimate range has narrowed from 'decades away' to '5-10 years.' Uncertainty has shifted from 'will it come?' to 'exactly when?'

Data Shelf Life Matters

Encrypted today, decryptable tomorrow

If data encrypted with RSA today must remain confidential past Q-Day, it needs to change now. 30-year medical records, long-term defense intelligence, 20-year bond contracts — if you have backups of this category of data encrypted with RSA, migration is priority one. 'Later' has already become more expensive.

Hybrid Deployment

X25519 + ML-KEM: one breaks, the other holds

Current standard: X25519 + ML-KEM-768. If either algorithm is safe, the whole is safe. Chrome 131+ (enabled by default), OpenSSH 10.0 (default), AWS KMS/ACM, Apple iMessage PQ3, Cloudflare edge — a list of implementations already deployed in production. Even if ML-KEM has an undiscovered flaw, X25519 provides the safety net.

Algorithm Tradeoffs

Right tool, right context

ML-KEM (FIPS 203): best performance/size balance, for TLS key exchange. ML-DSA (FIPS 204): general-purpose signing, but 52x the size of ECDSA P-256. SLH-DSA (FIPS 205): most conservative mathematical assumptions, slow and large so for firmware/code signing. FN-DSA (Falcon): compact but side-channel attacks demonstrated — prohibited for general signing. Common mistakes: using FN-DSA for general API signing, SLH-DSA for high-performance services.

⚖️Velocity vs. Security Debt

The faster the stack, the deeper the debt

The pace at which the AI infrastructure stack fragments by layer and capital concentrates creates the security debt accumulating on top. OSS adoption outpaces dependency management habits, and standard finalization precedes migration execution. Benefiting from rapid infrastructure transitions and managing the vulnerabilities those transitions create both come from the same engineering competency.

OSS Adoption Outpaces Dependency Discipline

Convenience and attack surface are the same feature

An open-source ecosystem becoming the de facto standard in 6 months also means becoming a dependency for millions of pipelines without a validation period. Version range specification in requirements.txt is convenient and simultaneously the cause of the PyTorch Lightning incident. The balance between speed and control is the core tension of ML engineering.

Standards Precede Migration

The gap between FIPS and your TLS config

NIST FIPS 203/204/205 took effect in August 2024, but most systems still use RSA/ECDH. The lag time from standard finalization → library support → gateway deployment → internal system migration varies by organization. This lag is the time of exposure to HNDL risk. 'We'll do it when the standard is finalized' — that argument is already 2 years old.