The giant feature is "start a VM in less than 1 second". Followed by things like suspending/resuming VMs.
We also need precise placement. We have people who want to run 8 VMs in London, 3 VMs in Los Angeles, and one VM in Sydney. They also want to do rolling deploys that revert on failure. Nomad's scheduling is not that precise.
Databases too. Both Nomad and k8s have ways to do persistent workloads, but they rely heavily on the underlying infrastructure to "move" volumes between throwaway containers. We don't want that, we actually want something closer to old school VM provisioning than "orchestration".
> Both Nomad and k8s have ways to do persistent workloads, but they rely heavily on the underlying infrastructure to "move" volumes between throwaway containers
I did this with Nomad by writing a small API to "place" host volumes which Nomad can use for job placement constraints automatically. It works shockingly well for my use-case which I think is somewhat similar to Fly's? A smallish number of containers that require host persistent storage, and don't expect anything to move -- just disappear forever (persistent til' it ain't).
This is what we do now, basically. We have a Nomad Device Plugin and create local LVMs. We also have plumbing to move these between hosts. It works pretty well, it's just very limiting.
What we need is to be able to target machines to specific volumes. And for things like database clusters, we need to control restart order during upgrades.
> The giant feature is "start a VM in less than 1 second". Followed by things like suspending/resuming VMs.
Is this for scaling existing apps up (eg flyctl regions add ...) or for new deployments (flyctl deploy)?
I suspect the former (scaling existing) as artifact distribution and deployment parameters like health checks seem like they prevent subsecond deployments (even ignoring the time for initial artifact uploading/building). Although my view is probably quite biased by Nomad having a much better chance at solving the former than the latter!
(Also hi! I appreciate our chats in the past and feel free to reach out if you're willing to brain dump Nomad gripes on me. :) )
It's very different than Nomad! The basic model is direct "machine" management with very little orchestration. Machines are kind of like allocations, but more permanent.
Some machines operations are slow, some are fast. Create is slow, update is slow. Start and stop are fast. Future "suspend" and "resume" are very fast.
For the regions case, people use these by having a couple of machines per region they care about. Then they start and stop as needed. A stopped machine can be migrated between nodes, but we ensure it's ready to start quickly if we do that.
Some interesting things happen with this model. Create is slow, but we are able to make it fast _sometimes_ because we can target creates at hardware that already has the rootfs cached.
Updates don't work like nomad, we update machines in place for a rolling deploy. This lets us do things like pull an image in parallel, then stop the machine, update, and restart very quickly.
We're also doing things differently with health checks. We may not wait for an active health checks before sending a machine an HTTP request. Many workloads don't even care about health checks, they just want a VM running and streaming logs back as fast as possible.
There are features I could ask for in Nomad to do what we'd need, but they'd make it not Nomad. :D
