Use local instance storage as root volume

/kind feature

1. Describe IN DETAIL the feature/behavior/change you would like to see.

Hi!

• if an instance has local instance storage (physically attached NVMe) available
• and the root volume size request does not exceed the size of the local instance storage

kOps can safely mount the local instance storage as root volume, instead of EBS, for a substantial performance boost whilst saving money.

2. Feel free to provide a design supporting your feature request.

Pasting the motivation and afaik the current state:

I have some containers that need fast temporary storage (around 100GB) we were using gp2 type AWS EBS volumes however they would quickly run out of burst balance. Local instance storage seemed like the perfect replacement as it would reduce the spend on slow EBS volumes and provide fast temporary storage. However I quickly found that Kubernetes doesn't seem to have quite implemented a way to use the local instance storage yet.

I wanted to use emptyDir volumes on my containers that needed fast local temporary storage so I tried moving /var/lib/kubelet to local instance storage by specifying it in the instance group configuration:

  volumeMounts:
    - device: /dev/nvme1n1
      filesystem: ext4
      path: /var/lib/kubelet

However like previous posters have mentioned I started seeing issues with disk pressure and the pods being evicted even though the local instance storage had only used 35% capacity.

Instead we have now switched to using a hostPath volume with an initContainer to set the correct permissions in the host directory. Our kops instance group now looks like this:

  volumeMounts:
    - device: /dev/nvme1n1
      filesystem: ext4
      path: /mnt/localssd

Relevant container configuration:

      initContainers:
      - name: fix-tmp-perms
        image: busybox
        securityContext:
          runAsUser: 0
        command: ["sh", "-c", "chown -R 201:201 /tmp/worker-temp; chmod 1777 /tmp/worker-temp; rm -rf /tmp/worker-temp/*"]
        volumeMounts:
        - name: worker-temp
          mountPath: /tmp/worker-temp

      volumes:
        - name: worker-temp
          hostPath:
            path: /mnt/localssd/worker-temp
            type: DirectoryOrCreate

What would be nice is to be able to specify the root volume in kops to use the local instance storage rather than having to be backed by EBS. I think this makes sense as the EBS volume is only used for temporary storage and is deleted when the instance is deleted.

Originally posted by @elliotdobson in https://github.com/kubernetes/kops/issues/429#issuecomment-689803817