Isn’t that a “for now” problem rather than something intractable for the performance anyway? Presumably HW and SW algorithms will continue to improve. Art directable may be a problem but it feels like Gaussian splats + genAI models could be a match made in heaven with the genAI mode generating the starting image and splats generating the 3d scene from it

Sure, given an unlimited amount of time and resources, it’s possible that Gaussian splats could be performant. But that’s just too vague a discussion point to be meaningful.

It’s definitely not in the cards in the near term without a dramatic breakthrough. Splats have been a thing for decades so I’m not holding my breath.

I mean here it is running at 22fps. In another 5 years it’s reasonable to conservatively believe hardware and software to be 3x as powerful which gets you to a smooth 60fps.

What am I missing on the performance front?

Well my critique of your comment is just that it’s unbounded. Yes, eventually all compute will get better and we can use once slow technologies. But that’s not a very valuable discussion because nobody is saying it’ll never be useful, just that it isn’t for games today.

It also ignores that everything else will be faster too then as well, and ignores needing to target different baselines of hardware.

Either way 5 years for a 3x improvement seems unrealistic. 4 years saw a little over a doubling of performance at the highest end with a significant increase in power requirements as well, where we’re now hitting realistic power limits.

Taking the 2080 vs 4080 as their respective tiers

153% performance increase 50% more power consumption 50% price increase.

So yes performance at the high end will increase, but it’s scaling pretty poorly with cost and power. And the lower end isn’t scaling as linearly.

On the lower/mid end (1060 Ti vs 2060 Super) we saw only a 53% increase in that same time period.

I guess it's to me that's still just an pessimistic perception. Ray tracing was also extremely slow for a long time until Nvidia built dedicated HW to accelerate it. Is there reason to believe that splats are already well served by generic GPU compute that dedicated HW won't accelerate it in a meaningful way?

Here's splats from 2020 working at 50-60fps [1]. I think my overall point is I don't think it's performance that's holding it back in games but tooling & whether it saves meaningful costs elsewhere in the game development pipeline.

[1] https://x.com/8Infinite8/status/1699460316529090568

Again, I’m not saying it won’t be possible someday. Any number of things could happen, even though the trajectory doesn’t imply it will be in the next 5 years. All I’m saying is that the question is pointless without bounds.

Otherwise flying cars will also be possible.

Also your splat is running in isolation. Any single system can run by itself at a good clip. That’s not indicative of anything when running as part of a larger system. Again, the discussion of performance is pointless without bounds.

> they’re not art directable or dynamic

This is not true I believe. There are plenty of papers out there revolving around dynamic/animated splat-based models, some using generative models for that aspect too.

There are also some tools out there that let you touch up/rig splat models. Still not near what you can do with meshes but I think fundamentally it’s not impossible.

You can touch up a splat in the same way you can apply gross edits to an image (cropping, color corrections etc), but you can’t easily change it in a way like “make this bicycle handle bar more rounded”. Ergo it’s not art directable.

With regards to dynamicism, there’s some papers yes but with heavy limitations. Rigging is doable but relighting is still hit and miss, while most complex rigs require a mesh underneath to drive a splats surface. There’s also the issue of making sure the splats are tight to the surface boundary, which is difficult without significant other input.

Other dynamics like animation operate at a very gross level, but you can’t for example do a voronoi fracture for destruction along a surface easily. And again, even at a large scale motion, you still have the issue of splat isolation and fitting to contend with.

The neural motion papers you mention are interesting, but have a significant overhead currently outside of small use cases.

Meshes are much more straightforward, and with advancements in neutral materials and micropolygons (nanite etc) it’s really difficult to make a splat scene that isn’t first represented as a mesh have the quality and performance needed. And if you’re creating splats from a captured real world scene, they need significant cleanup first.

Are they good for that either? I haven't seen one where the data isn't huge

The data is definitely an issue, but they do make for fairly convenient alternatives to something like matterport where you need their cameras rented etc.

Though I think matterport will just start using them since the other half of their product is the user experience on the web.

Will they though? I saw a siggraph demo of a matterport like apartment preview using gaussian splatting. It downloaded 1.6gig! for a single apartment. Checking out a current matterport demo on their site for a similar sized space it was 60meg or 26x smaller

Tbh most splats data today is not optimally stored. There’s a lot that could be done for streaming, data reduction and segmentation. So I think it’s definitely both possible and easy to reduce that data size in half if not more.

They’ll likely never be smaller than a mesh and texture though, because the data frequency will be higher. A wall can be two triangles and a texture. The same representation as splats will have to be many hundreds of points, roughly at the count of the pixels of the lowest resolvable version of that texture.

So I agree they’re far from optimal for data size. But they greatly reduce the complexity of data capture and representation.