Introduction

NVIDIA today released the first public demo of Chat with RTX. No, you can’t talk to your graphics card and ask “how’s it going?,” you’ll need TechPowerUp GPU-Z to do that. Chat with RTX is something else. Imagine a fully localized AI chat that’s running entirely on your PC, accelerated by the highly capable cores in your GeForce RTX graphics card; and which sends none of your queries to a cloud-based chat server. That’s Chat with RTX. This thing is being developed by NVIDIA to be a ChatGPT alternative that has all its knowledge stored locally on your PC, and a GeForce RTX GPU to use for a brain.

While 2024 promises to be the “year of the AI PC,” as industry leaders Microsoft and Intel would have you believe, NVIDIA has had an incredible six-year head-start with AI acceleration. The company introduced on-device accelerated AI for its RTX real-time ray tracing technology. As part of this innovation, its 2017 GeForce RTX GPUs were equipped with Tensor cores. These components significantly boosted AI deep-learning neural network (DNN) building and training compared to using CUDA cores alone. This advancement marked a substantial leap forward in performance, enhancing the capabilities of the GPUs for AI-driven tasks. Besides the denoiser, NVIDIA leverages AI acceleration to drive its DLSS performance enhancement feature. Can’t max out a game? Simply enable DLSS and pick one of its presets until the game is playable at the settings you choose.

In our recent interactions with NVIDIA, the company made it clear that they aren’t too impressed with the newest processors from Intel and AMD, which introduce NPUs (neural processing units); with performance figures around the 10-16 TOPS mark for the NPU itself, and no more than 40 TOPS for the whole chip (NPU + CPU + iGPU). NVIDIA GeForce RTX GPUs with their Tensor cores in contrast, tend to offer anywhere between 20x to 100x (!) this performance due to the sheer scale at which NVIDIA has deployed AI acceleration on its GPU silicon.

While CPU-based NPUs are intended to drive simple text-based and light image-based generative AI tasks; NVIDIA is incorporating AI at a different level, even today—think of generating every alternate frame in DLSS 3 Frame Generation, or denoising a 4K in-game scene at 60+ FPS, depending on the resolution. Put simply, GeForce RTX GPUs have enormous amounts of AI acceleration hardware resources that remain dormant when you’re not gaming; and so NVIDIA has taken it upon itself to show gamers they can run fully localized generative AI tools leveraging this hardware. The company is just getting started, and one of its first projects is Chat with RTX, for which we’re reviewing a preview build today. NVIDIA has a vast install base—millions of gamers with GeForce RTX GPUs, and so in the near future, we expect NVIDIA to take a more active role in the AI PC ecosystem, by providing additional AI-driven experiences and productivity tools for PCs with a GeForce RTX GPU.

Chat with RTX, as we said, is a text-based generative AI platform—a ChatGPT or Copilot of sorts—but one that doesn’t send a single bit of your data to a cloud server, or use web-based datasets. The dataset is whatever you provide. You even have the flexibility to choose an AI model, between Llama2, and Mistral. For the tech demo of Chat with RTX, NVIDIA provided both Llama2 and Mistral, along with their native datasets that are updated till mid-2022.

In this article, we take Chat with RTX for a spin to show you its potential to bring powerful, completely-offline AI chat to gamers.