Today, we are breaking convention to make it easier to run Kubernetes (K8s) on bare metal servers to support stateful applications that need stable, persistent storage. To do this, we built MetalK8s, an open source, opinionated K8s distribution, to simplify on-premises deployments. It is available now on GitHub, under the Apache Software License v2.
We made the early design choice to trust K8s as the infrastructure foundation for Scality’s Zenko Multi-Cloud Data Controller. Since Zenko is meant to run in any cloud environment and also on-premises for the most demanding enterprise customers, K8s is the perfect tool to run Zenko the same way whether it’s on Amazon, Azure, Google Cloud, in a private cloud or on bare metal.
Because the on-premises bare metal K8s experience is not prevalent in the community, we decided to take the time to find the best way to deploy Kubernetes on bare metal servers for Zenko.
Why are we choosing to go Bare Metal?
Kubernetes itself grew up in virtualized environments, which is natural given its purpose to orchestrate distributed container environments. We realized that very few people dare to run K8s on bare metal, and actually most have no choice but to run it on virtual infrastructure. In the course of our development, though, we discovered that there are several huge benefits to be gained from deploying on bare metal. But, this is only true when developers and operators find all of the tools they need for smooth, long-term operations.
While developing Zenko on Kubernetes, we required efficient access to stateful local storage for both metadata and data. Moreover, as Zenko is a distributed environment, we really wanted to optimize the proximity of compute and storage to the same machine that has the local storage. For applications that require this type of storage access efficiency, the K8s environment has never been optimal. By default, K8s can otherwise force the use of an expensive SAN or cloud block-storage volumes. With MetalK8s, we resolve this problem and are enabling fast local storage access for container-based applications.
Why an Opinionated Distribution?
We chose to go the ‘opinionated’ route because we have made some specific choices in the course of our development: MetalK8s’ goal is to provide great functionality while reducing complexity for other users and delivering the stateful storage efficiencies described earlier.
Our team specifically chose to leverage an existing project rather than reinvent the wheel, so we based MetalK8s on top of the excellent open-source Kubespray ‘playbook’ project. Kubespray enables us to install a base Kubernetes cluster reliably using the Ansible provisioning tool with its dependencies (e.g.; the etcd distributed database system). This allowed us to quickly iterate and implement the features we need to run Kubernetes at the scale needed by Scality customers. This is where our own Scality DevOps team excels, and so this stayed in line with our focus on ease of operations. Contrary to Kubespray’s general-purpose approach, we decided to make hard choices like use Calico as the only Container Network Interface (CNI) implementation. Further, an “ingress controller” is deployed by default, based on Nginx. And for simplicity, all of these are managed as Helm packages.
The installation is further augmented with a set of powerful operational tools for monitoring and metering, including Prometheus, Grafana, ElasticSearch and Kibana.
Unlike hosted Kubernetes solutions, where network-attached storage is available and managed by the provider, MetalK8s assumes no such system to be available in environments where MetalK8s itself is deployed. This means its focus is on managing node-local storage and exposing local volumes to containers managed in the cluster.
The team plans to work with upstream projects including Kubespray, Kubernetes, Helm Charts and others to release back all useful contributions and eventually implement new features.
You can learn more from Scality Architect Nicolas Trangez at OpenStack Summit in Vancouver on Tuesday, May 22 at 3:10pm PDT Convention Centre West – Level One – Marketplace Demo Theater. Scality’s CTO Giorgio Regni and the Zenko team are also available for interviews on site: please book time with them.
Nicolas Trangez co-authored this post.