We just successfully ran kubernetes 1.5 on LinuxONE, just curious anyone else using or interested with kubernetes on LinuxONE? We could learn from each other and see what’s missing from current solution.
3 Likes
Very cool! Can you detail more on the use case? Any gotchas or quirks to be aware of?
We’re trying to use kubernetes to manage the containers orchestration and management, kubernetes could help us to scale the containers, ensure the container is re-started when it failed, rolling update docker images without shutdown current running containers.
Here comes more details about the setup:
To setup a single node kubernetes cluster is quite straightforward, the interesting part is the HA cluster. We have integrated several open source project with kubernetes to have the HA cluster up and running.
- HA - etcd
etcd is installed to 3 nodes, each node has its own etcd data.
etcd is configured to act as a cluster within 3 nodes, each node knows he is one member of a 3-nodes cluster.
Kubernetes connects to etcd using 3 IP address together. In case one node failed, Kubernetes will switch to next node automatically.
- HA – Kubernetes
Master node
Kubernetes API server, controller manager and scheduler are installed to 3 nodes.
Pacemaker is installed to 3 nodes and configured as the HA manager using a virtual IP.
Pacemaker is responsible for the traffic to virtual IP, re-direct to a working API server address. If one node went offline, pacemaker on another node will take over the virtual IP and the end users are not impacted.
Kubernetes controller manager and scheduler are installed to 3 nodes and managed by pacemaker, pacemarker will try to re-start those services if any service stopped. If one node went offline, Kubernetes controller manager and scheduler on other nodes will take over.
Worker node
Kubernetes kubelet and proxy on each worker node will use virtual IP to connect to API server.
- HA – Storage
Ceph is providing distributed storage for each Stateful containers.
3 OSD nodes to store the data.
3 node for monitoring node as a cluster.
1 node for administration.
The reference link from kubernetes HA setup document.
We built kubernetes from source code following the instructions here (https://github.com/linux-on-ibm-z/docs/wiki/Building-kubernetes). From Kubernetes 1.6.0 alpha 1, s390x build is provided by the official kubernetes release download page (https://github.com/kubernetes/kubernetes/releases/tag/v1.6.0-alpha.1), so in the near future, no need to build kubernetes on LinuxONE from source code any more.
3 Likes
We have OpenShift 1.5 up and running on LinuxONE. It already includes the necessary code for building on s390x, for 1.3 and 1.4 I just copied the relevant sections from the golang src tree for s390x syscalls etc. The only deficiency I can see, and this is not a z specific issue, is that it only allows for FCP connected devices with LUN between 1 and 255. In addition it expects device by-path names to correspond to template xxxx-fcp-0xnnnnnnnnnnnn-lun-mmm. This works for LUNs 0 < n < 256 but when above this range the devices appear as xxxx-fcp-0xnnnnnnnnnnn-lun-0xmmmmmmmmmmm. The code in kubernetes doesn’t cater for this.
My experiences, so far, have been documented at Getting Started with OpenShift on LinuxONE
2 Likes