yeah, at some point memory footprint becomes an architectural constraint again, not just a number you throw more hardware at.
The memory capacity wall has been coming at us at high speed in plain sight for decades. We've been ignoring it.
Demand outstrips supply. Efficient software is great, but it doesn’t directly resolve insufficient supply of hardware.
If training and inference is hardware constrained, and you can train and server better and bigger models on the same hardware with memory optimizations, that's exactly what I would expect companies to do.
