Kog is going deeper to squeeze more inference out of GPUs

The IPO markets celebrated Cerebras in May. Purpose-built chips, fresh capital, a clean narrative. But across the Atlantic, a solo French founder is betting the real money stays in the metal enterprises already own.

Kog hit Hacker News with a demo that made a simple promise: extremely fast single-request decoding on the AMD MI300X and NVIDIA H200 GPUs sitting in your datacenter right now. Not on future silicon. Not on specialized accelerators. On the hardware collecting dust while procurement cycles grind.

The comment section split. Some complained the demo didn't run on laptop GPUs. Others saw the implication immediately. Two hundred tangible business leads arrived in the inbox. Gaël Delalleau counted them.

Software engineering comes first. Veteran Claude Code users know the wait. Hours for a result. Anthropic knows the pain too — they charge a multiple for Fast Mode. Kog targets the professionals who cannot afford that latency, who build workflows around AI that stall when the model chokes.

The startup also courts design partners generating games and apps from prompts. Faster inference means more iterations. More iterations means more revenue. The math is straightforward.

But the market isn't ready. Prospective customers won't fine-tune small models. They want the big ones. So Kog pivoted hard after launch — full focus on accelerating larger models to meet actual demand.

Here is where the claim meets the cliff. The demo delivered 3,000 tokens per second on Laneformer 2B, a purpose-built small model Kog open-sourced. Two billion parameters. The promise is 30x faster inference on LLMs — models whose size breaks inference chips for breakfast.

Delalleau doesn't flinch. He calls the GPU-unsuitable-for-decoding narrative a misconception. Newer GPUs pack memory bandwidth that only begs to be unlocked. His conviction: the same software architecture that flies on 2B parameters scales to 70B and beyond.

Skepticism is warranted. Memory bandwidth helps. But kernel fusion, speculative decoding, custom scheduling — these break differently at scale. The attention pattern changes. The memory hierarchy pressure shifts. What works on a model that fits in HBM with room to spare may collapse when the KV cache alone exceeds capacity.

Kog isn't alone in the software layer. ZML, also French, bypasses CUDA to run fast inference across competing chips. Hardware-agnostic. Practical. Delalleau positions Kog closer to Stanford's Hazy Research — deeper in the GPU architecture, closer to the metal. The distinction matters. ZML abstracts. Kog excavates.

Delalleau himself is not a researcher. His first startup, Stribe, landed on TechCrunch50 in 2009. Consumer social. Zero overlap with kernel optimization. His former cofounder turned VC. The pedigree is entrepreneurial, not academic.

That matters. Research prototypes. Products ship. The gap between a 3,000 TPS demo on a curated 2B model and a production engine that handles 405B parameters at consistent latency across variable batch sizes, context lengths, and quantization schemes — that gap is where startups die.

But the incentive structure is brutal and clear. Every enterprise running LLMs today over-provisions GPUs because inference throughput is poor. They buy H100s by the rack to serve traffic that optimized software could handle on a fraction of the iron. Nvidia's margin is Kog's addressable market.

Delalleau's bet: the memory subsystem is the lever. Modern GPUs stream data at terabytes per second. The decoder touches a fraction. Unlock the rest and the arithmetic changes — not incrementally, but multiplicatively.

The proof point approaches. Kog must show 30x on Llama-3-70B or equivalent. Real prompts. Real KV cache pressure. Real concurrent users. Not a curated benchmark on a model designed for the kernel.

If they deliver, the procurement calculus flips. Enterprises stop buying iron to solve software problems. They license the engine that makes their existing iron sufficient.

If they don't, the demo joins the graveyard of impressive small-model numbers that vanished at scale.

The clock is running. Two hundred leads won't wait for a research project. They have products shipping, customers churning, Anthropic bills stacking. Kog chose the harder path — no custom silicon, no hardware moat, pure software depth on someone else's metal. Respect the audacity. Watch the execution.