intel-is-using-machine-learning-to-make-gta-v-look-incredibly,-unsettlingly-realistic

Intel is using machine learning to make GTA V look incredibly, unsettlingly realistic

One of the more impressive aspects of Grand Theft Auto V is how closely the game’s San Andreas approximates real-life Los Angeles and Southern California, but a new machine learning project from Intel Labs called “Enhancing Photorealism Enhancement” might take that realism in a unsettlingly photorealistic direction (via Gizmodo).

Putting the game through the processes researchers Stephan R. Richter, Hassan Abu Alhaija, and Vladlen Kolten created produces a surprising result: a visual look that has unmistakable similarities to the kinds of photos you might casually take through the smudged front window of your car. You have to see it in motion to really appreciate it, but the combination of slightly washed-out lighting, smoother pavement, and believably reflective cars just sells the fact you’re looking out at the real street from a real dashboard, even if it’s all virtual.

The Intel researchers suggest some of that photorealism comes from the datasets they fed their neural network. The group offers a more in-depth and thorough explanation for how image enhancement actually works in their paper (PDF), but as I understand it, the Cityscapes Dataset that was used — built largely from photographs of German streets — filled in a lot of the detail. It’s dimmer and from a different angle, but it almost captures what I imagine a smoother, more interactive version of scrolling through Google Maps’ Street View could be like. It doesn’t entirely behave like it’s real, but it looks very much like it’s built from real things.

The researchers say their enhancements go beyond what other photorealistic conversion processes are capable of by also integrating geometric information from GTA V itself. Those “G-buffers,” as the researchers call them, can include data like the distance between objects in the game and the camera, and the quality of textures, like the glossiness of cars.

While you might not see an official “photorealism update” roll out to GTA V tomorrow, you may have already played a game or watched a video that’s benefited from another kind of machine learning — AI upscaling. The process of using machine learning smarts to blow up graphics to higher resolutions doesn’t show up everywhere, but has been featured in Nvidia’s Shield TV and in several different mod projects focused on upgrading the graphics of older games. In those cases a neural network is making predictions to fill in missing pixels of detail from a lower resolution game, movie, or TV show to reach those higher resolutions.

Photorealism probably shouldn’t be the only graphical goal for video games to have (artistry aside, it looks kind of creepy), but this Intel Labs project does show there’s probably as much room to grow on the software side of things as there is in the raw GPU power of new consoles and gaming PCs.

valve-head-gabe-newell-hints-at-upcoming-console-news

Valve Head Gabe Newell Hints at Upcoming Console News

(Image credit: Valve)

Gabe Newell, the head of Valve Software, hinted to students in New Zealand that the company might expand its Steam platform, or at least some games, to the console space later this year. The comment is vague at best, but at least it shows that the owner of one of the largest game distribution platforms has not given up its living room gaming plans. 

Earlier this week, Gabe Newell spoke to students at Sancta Maria College in Auckland, New Zealand, and was asked whether Steam would be “porting any games on consoles, or [would] it just stay on PC?” The response was imprecise, but we cannot really expect anyone to disclose business plans at an event like this. 

“You will get a better idea of that by the end of this year… and it won’t be the answer you expect,” Newell said. “You’ll say, ‘Ah-ha! Now I get what he was talking about.'” 

The whole conversation had been recorded by a student who later uploaded it to Reddit, Ars Technica first reported.  

Valve’s track record with game consoles in particular and living room gaming in general has been bumpy at best. On the one hand, the company successfully ported its games to consoles from Microsoft and Sony in the past, including the very successful The Orange Box on Xbox 360 and PS3. On the other hand, Valve’s Steam Machines initiatives has failed, just like its console oriented SteamOS. Valve’s Steam Link, which allowed to stream games from a local PC to a TV, has also failed to get popular enough for the company to keep selling the product. 

For gamers, getting their Steam libraries on consoles would be a thing of dreams. It’s unclear how Steam would work on those systems, though, as Nintendo, Xbox and Sony all run their own exclusive stores on their platforms. Furthermore, far from all Steam games have versions for consoles, and developing a Windows or Linux emulator for Xbox or PlayStation is one heck of a task. Streaming games to consoles might be a way into the living room for Valve, but controller options may be an obstacle there.

Newell has been living in New Zealand since March 2020, when he was stuck as the Covid-19 pandemic hit. Since he doesn’t often do big appearances in the games industry, it makes some sense that a bunch of New Zealand students were the first to hear his thoughts on upcoming announcements.