Let's say we have some specific nodes for specific tasks. E.g. we deployed Elasticsearch (3 nodes) and assigned Elasticsearch to those nodes with node-pool labels. We grew up and we need more resources for Elasticsearch node pool specifically. Currently, we have to:
Either:
Deploy new node-pool with a different name
Wait for deployment process
Change nodeSelector for Elasticsearch (if it's statefulset, it's even more complicated)
Wait until all pods move to different nodes
Kill old node pool
Or:
Destroy old node pool (downtime guaranteed)
Wait for node pool destruction
Create new node pool with same name, but different resources scale for nodes
Wait for everything to start up
Both processes are either complicated and require a lot of manual work, or have downtime.
It would be better if we could resize node pools by nodes sizes rather than node count only. For example, scale from 4 cores per node to 8 cores per node, with rolling node replacement rather than all-at-once.