Nvidia is throwing its hat in the generative AI ring with a new tech demo called Chat with RTX for late-model Nvidia GPU owners. The company calls it a "demo" as it's version 0.2 and still in beta, but it gives us a glimpse of our "AI on device" future. It lets you run a chatbot directly on your PC instead of requiring "the cloud" for horsepower and access to AI models like with Microsoft Copilot. The benefits of this approach are much faster response times and enhanced data privacy. However, this technology is still very much in its infancy, so the benefits are quite limited at this time.
Nvidia has opened up the Chat with RTX program to anyone who wants to try it, so all you need is a 40-series or 30-series GPU, 8GB of system memory, and Windows 11. Our sister site PCMag got to take it for a spin and noted the download was around 35GB. It requires 100GB of free space, on par with a AAA game. The installation takes between 30 minutes and an hour, so you might want to let this one run unattended. It's unclear why the demo won't work with RTX 20-series GPUs, which have the requisite tensor cores. Perhaps there's just not enough of them, as generative AI wasn't a thing when those GPUs launched six years ago.
Given the software's beta status, it will surprise nobody that it only allows access to one large language model, which Nvidia calls Mistral. You can feed the model queries, and it spits them out quickly—with the main benefit being it can pull data from both the web and the complex documents you submit, including PDFs and even YouTube videos. Nvidia says it can also do language translation, though that functionality doesn't seem to exist yet. Overall, PCMag says when compared head-to-head with ChatGPT, Chat with RTX performed similarly but was less sophisticated overall. For example, it found Chat with RTX would ignore some parts of its queries, whereas ChatGPT had no problem following directions.
The main benefit of Chat with RTX, at this time, seems to be summarizing text files (PDF, Doc, txt) and YouTube videos and answering questions about the content contained in the files or links. Let's be real; we all love a good 18-minute GPU deep dive, but it would also be nice to have a tidy summary in text form (olds, assemble). Chat with RTX is reportedly quite adept at video summaries, but the results were mixed with a PDF of GPU test results, indicating it was unclear on the instructions. For example, PCMag says it accurately summarized a YouTube video and answered questions about it. However, when given a PDF file, it gave inaccurate answers and didn't understand follow-up questions.
For now, Chat with RTX seems like a fun tool to tinker with, and Nvidia has certainly beaten AMD to the punch. On-device AI has been in the pipeline for 2024, and Nvidia gets kudos for being ahead of the competition here. However, with CPUs from AMD and Intel now having neural processing units (NPUs), we have to wonder if there will soon be a battle within our computers over which hardware runs which AI we're trying to use. Honestly, it's already a bit exhausting to even think about.
Tagged In
Artificial IntelligenceMore from Computing
