Nomad: basics

Tue, Oct 25, 2022 4-minute read

Nomad: basics

Nomad is a scheduler/orchestrator for tasks. It can handle containers, VMs, .NET and Java application through its scheduler.

Highlights

  • Single binary; distributed; leader election and replication for HA
  • Plugin support, and GPU, FPGA and TPU out of the box support
  • Multi-region federation
  • Highly scalable (10k+ nodes in production in real-world examples)
  • Native integration into Hashicorp products like Vault, Consul and Terraform
  • Batch processing support (Kubernetes doesn’t do this well)

Versus Kubernetes

Kubernetes is an orchestration system for containers originally designed by Google, now governed by the Cloud Native Computing Foundation (CNCF) and developed by Google, Red Hat, and many others. Kubernetes and Nomad support similar core use cases for application deployment and management, but they differ in a few key ways. Kubernetes aims to provide all the features needed to run Linux container-based applications including cluster management, scheduling, service discovery, monitoring, secrets management and more. Nomad only aims to focus on cluster management and scheduling and is designed with the Unix philosophy of having a small scope while composing with tools like Consul for service discovery/service mesh and Vault for secret management.

source

Installing

Ubuntu:

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install nomad
# grab all Hashicorp tools if needed with
# sudo apt-get install packer terraform consul vault nomad

Definitions

Cluster:

  • agent - An agent is a Nomad process running in server or client mode. These are the building blocks of a Nomad cluster.
  • dev agent - Developer agent with defaults for running experiments or a dev cluster. It does not persist any state to disk.
  • server - Server mode agents are the “Control Plane” for the cluster. There is a cluster of servers per region and they manage all the jobs and clients, run evaluations, and create the task allocations.
  • leader - This is the server which performs the majority of the cluster management. It is in charge of applying plans, deriving vault tokens for workloads and administering the cluster state.
  • follower - Followers submit plans to the leader to provide more scheduling capacity to the cluster. The leader executes plans from the followers.
  • client - An agent running in client mode. These agents watch for any work and execute tasks. Client connections are multiplexed to servers.

Nomad Objects:

  • job - Defines one or more task groups which contain one or more tasks.
  • job spec - Schema definition for a Nomad job. Describes the type, tasks and resources required to run the job.
  • task group - A set of tasks which must be run together. Example, a web server may require a log shipping co-process is always running as well. A task group ensures that these two tasks are always run together.
  • task - The smallest unit of work in Nomad. These are executed by task drivers which allow Nomad to be flexible in the type of tasks it supports.
  • allocation - A mapping between a task group and a client node. A job can have hundreds or thousands of task groups which means an equivalent number of allocations must be mapped to client machines. Allocations are created from scheduling decisions known as an evaluation.
  • evaluation - The way in which Nomad makes scheduling decisions. When the state changes a new evaluation is triggered to determine the next action.

Scheduling:

  • bin packing - This is how Nomad maximises allocations on a device.
  • spread scheduling - The opposite of bin packing; this attempts to evenly spread work across all the machines in the fleet.

Starting Nomad

Nomad relies upon agent; an agent must be on every machine in the cluster. The server is responsible for managing the cluster and all other agents should be in client mode. Clients register the machine, conduct heart beats and run tasks when assigned from servers.

In development:

sudo nomad agent -dev -bind 0.0.0.0 -log-level INFO

Tags:

#hashicorp #nomad