Practicing Nebula Operator on Cloud

Nebula Operator

Hi, everybody! As you know, Nebula Operator has been open-source, and we published an introduction to it. Today, I would like to introduce my practice on Nebula Operator on cloud.

About Nebula Operator

What is Nebula Operator? You can refer to Nebula Operator Overview: Automated Operation on Kubernetes. Today, this article will focus on the practice on Nebula Operator. I hope it could help you get started with it soon and enjoy NebulaGraph.

Practicing Nebula Operator on Cloud

In this section, I will introduce the process of deploying and using Nebula Operator on cloud. Alibaba Cloud services are used, but the process is applicable to the similar services of other cloud service providers.

Installation Tools

Install the following tools on your on-premises machine:

For the installation information, see their documentations.

Creating Kubernetes Cluster on Cloud

Nebula Operator is implemented for Kubernetes applications, so Kubernetes is necessary for practicing Nebula Operator.

This practice was performed on a managed Kubernetes cluster on Alibaba Cloud, so the first thing is to log on to the Alibaba Cloud Container Service for Kubernetes (ACK) console and then to create a managed Kubernetes cluster. The configurations must meet your needs.

NOTE: To facilitate access to the Kubernetes API Server from Internet, Expose API Server with EIP was enabled for this practice. Whether to enable it or not is determined by your needs. If it is not enabled, you need to make sure that your on-premises machine can connect to the Kubernetes cluster. Configure other parameters as required.

When the Kubernetes cluster starts, in the Connection Information tab, copy the Public Access content in the code block to the $HOME/.kube/config file of your on-premises machine.

And then on your on-premises machine, run the following command to check the connection to your Kubernetes cluster.

$ kubectl get nodes
NAME                         STATUS   ROLES    AGE   VERSION
cn-beijing.192.168.250.13    Ready    <none>   51m   v1.20.4-aliyun.1
cn-beijing.192.168.250.185   Ready    <none>   51m   v1.20.4-aliyun.1
cn-beijing.192.168.250.89    Ready    <none>   51m   v1.20.4-aliyun.1

Installing Dependencies for Nebula Operator

Install all the dependencies listed in this section for Nebula Operator.

Installing CertManager

Run the following command to install CertManager.

 Install CertManager
$ helm install cert-manager cert-manager --repo https://charts.jetstack.io \
    --namespace cert-manager --create-namespace --version v1.3.1 \
    --set installCRDs=true
# Wait a moment and then check whether CertManager is started successfully.
$ kubectl -n cert-manager get pod
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7998c69865-jfw9x              1/1     Running   0          93s
cert-manager-cainjector-7b744d56fb-846w9   1/1     Running   0          93s
cert-manager-webhook-7d6d4c78bc-ssk4w      1/1     Running   0          93s

Installing OpenKruise

Run the following command to install OpenKruise.

# Install OpenKruise
$ helm install kruise \
    https://github.com/openkruise/kruise/releases/download/v0.8.1/kruise-chart.tgz
# Wait for a moment and then check whether OpenKruise is started successfully.
$ kubectl -n kruise-system get pod
NAME                                         READY   STATUS    RESTARTS   AGE
kruise-controller-manager-6797f89d9b-ppv65   1/1     Running   0          49s
kruise-controller-manager-6797f89d9b-wlkbd   1/1     Running   0          49s
kruise-daemon-7rljq                          1/1     Running   0          49s
kruise-daemon-8kd8d                          1/1     Running   0          49s
kruise-daemon-n6tdw                          1/1     Running   0          49s

Adding Nebula Operator Charts

Run the following command to add Nebula Operator Charts.

# Add the Nebula Operator Charts repository.
$ helm repo add nebula-operator https://vesoft-inc.github.io/nebula-operator/charts
# Update the repository.
$ helm repo update

Installing Nebula Operator

If you cannot pull the gcr.io or k8s.gcr.io image, use mirrors. For example, in this practice, the following mirrors were used.

Original Images	Mirrors
gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0	kubesphere/kube-rbac-proxy:v0.8.0
k8s.gcr.io/kube-scheduler:v1.18.8	kubesphere/kube-scheduler:v1.18.8

You can run the following command to view all the configurable parameters.

$ helm show values nebula-operator/nebula-operator

In this practice, I followed this procedure to install Nebula Operator.

# Install Nebula Operator
$ helm install nebula-operator nebula-operator/nebula-operator \
    --namespace nebula-operator-system --create-namespace --version 0.1.0 \
    --set image.kubeRBACProxy.image=kubesphere/kube-rbac-proxy:v0.8.0 \
    --set image.kubeScheduler.image=kubesphere/kube-scheduler:v1.18.8
# Wait for a moment and check whether Nebula Operator is started successfully.
$ kubectl -n nebula-operator-system get pod
NAME                                                             READY   STATUS    RESTARTS   AGE
nebula-operator-controller-manager-deployment-6968547fff-k62b4   2/2     Running   0          19s
nebula-operator-controller-manager-deployment-6968547fff-lhpdx   2/2     Running   0          19s
nebula-operator-scheduler-deployment-7c5fc7945-hbkv8             2/2     Running   0          19s
nebula-operator-scheduler-deployment-7c5fc7945-sxc7w             2/2     Running   0          19s

If Cluster Domain of Kubernetes are customized, modify the preceding installation command by adding configuration of kubernetesClusterDomain.

# Install Nebula Operator and replace <<YourCustomCLusterDomain>> with your own cluster domain.
$ helm install nebula-operator nebula-operator/nebula-operator \
    --namespace nebula-operator-system --create-namespace --version 0.1.0 \
    --set image.kubeRBACProxy.image=kubesphere/kube-rbac-proxy:v0.8.0 \
    --set image.kubeScheduler.image=kubesphere/kube-scheduler:v1.18.8 \
    --set kubernetesClusterDomain=<<YourCustomCLusterDomain>>

Deploying NebulaGraph Cluster

So far, Nebula Operator has been ready. Now, let's deploy a NebulaGraph cluster for handling graph data.

The first one is to know StorageClass information, which will be used to configure the persistent volume storage for the NebulaGraph cluster.

$ kubectl get sc
NAME                       PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
alicloud-disk-available    diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   100m
alicloud-disk-efficiency   diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   100m
alicloud-disk-essd         diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   100m
alicloud-disk-ssd          diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   100m
alicloud-disk-topology     diskplugin.csi.alibabacloud.com   Delete          WaitForFirstConsumer   true                   100m

As you can see from the result of the preceding command, five types of StorageClass are supported on Alibaba Cloud. For this practice, alicloud-disk-ssd was used. If you want to use the services provided by another cloud service provider, choose an applicable StorageClass type.

NOTE: Each cloud service provider may set limits on the storage capacity for your application. For example, on Alibaba Cloud, the limit on SSD is from 20 GiB to 32,768 GiB. Therefore, you must specially consider this configuration when deploying a NebulaGraph cluster.

To view all the configurable parameters, you can run the following command.

$ helm show values nebula-operator/nebula-cluster

In this practice, all these commands were run to deploy a NebulaGraph cluster.

# Name the NebulaGraph cluster.
$ export NEBULA_CLUSTER_NAME=nebula

# Create a namespace for the NebulaGraph cluster.
$ export NEBULA_CLUSTER_NAMESPACE=nebula

# Specify StorageClass type for the NebulaGraph cluster. In this example, alicloud-disk-ssd is used.
$ export STORAGE_CLASS_NAME=alicloud-disk-ssd

# Specify the storage size for the three processes of NebulaGraph.
$ export STORAGE_SIZE_GRAPHD=20Gi
$ export STORAGE_SIZE_METAD=20Gi
$ export STORAGE_SIZE_STORAGED=20Gi

# Deploy the NebulaGraph cluster.
$ helm install ${NEBULA_CLUSTER_NAME} nebula-operator/nebula-cluster \
    --namespace ${NEBULA_CLUSTER_NAMESPACE} --create-namespace --version 0.1.0 \
    --set nameOverride=${NEBULA_CLUSTER_NAME} \
    --set nebula.storageClassName="${STORAGE_CLASS_NAME}" \
    --set nebula.graphd.storage="${STORAGE_SIZE_GRAPHD}" \
    --set nebula.metad.storage="${STORAGE_SIZE_METAD}" \
    --set nebula.storaged.storage="${STORAGE_SIZE_STORAGED}"

# Wait a moment and check whether the NebulaGraph cluster is started successfully.
$ kubectl -n ${NEBULA_CLUSTER_NAMESPACE} get nebulacluster
NAME     GRAPHD-DESIRED   GRAPHD-READY   METAD-DESIRED   METAD-READY   STORAGED-DESIRED   STORAGED-READY   AGE
nebula   2                2              3               3             3                  3                4m10s
$ kubectl -n ${NEBULA_CLUSTER_NAMESPACE} get pod
NAME                READY   STATUS    RESTARTS   AGE
nebula-graphd-0     1/1     Running   0          96s
nebula-graphd-1     1/1     Running   0          96s
nebula-metad-0      1/1     Running   0          97s
nebula-metad-1      1/1     Running   0          97s
nebula-metad-2      1/1     Running   0          97s
nebula-storaged-0   1/1     Running   0          97s
nebula-storaged-1   1/1     Running   0          97s
nebula-storaged-2   1/1     Running   0          97s

Of course, you can also run the following command to upgrade the cluster to have five nebula-storaged process instances.

# Upgrade the NebulaGraph cluster.
$ helm upgrade ${NEBULA_CLUSTER_NAME} nebula-operator/nebula-cluster \
    --namespace ${NEBULA_CLUSTER_NAMESPACE} --create-namespace --version 0.1.0 \
    --set nameOverride=${NEBULA_CLUSTER_NAME} \
    --set nebula.storageClassName="${STORAGE_CLASS_NAME}" \
    --set nebula.graphd.storage="${STORAGE_SIZE_GRAPHD}" \
    --set nebula.metad.storage="${STORAGE_SIZE_METAD}" \
    --set nebula.storaged.storage="${STORAGE_SIZE_STORAGED}" \
    --set nebula.storaged.replicas=5
#  Wait for a moment and check whether the cluster is started successfully.
$ kubectl -n ${NEBULA_CLUSTER_NAMESPACE} get nebulacluster
NAME     GRAPHD-DESIRED   GRAPHD-READY   METAD-DESIRED   METAD-READY   STORAGED-DESIRED   STORAGED-READY   AGE
nebula   2                2              3               3             5                  5                6m12s
$ kubectl -n ${NEBULA_CLUSTER_NAMESPACE} get pod
NAME                READY   STATUS    RESTARTS   AGE
nebula-graphd-0     1/1     Running   0          2m30s
nebula-graphd-1     1/1     Running   0          2m30s
nebula-metad-0      1/1     Running   0          2m30s
nebula-metad-1      1/1     Running   0          2m30s
nebula-metad-2      1/1     Running   0          2m30s
nebula-storaged-0   1/1     Running   0          2m30s
nebula-storaged-1   1/1     Running   0          2m30s
nebula-storaged-2   1/1     Running   0          2m30s
nebula-storaged-3   1/1     Running   0          52s
nebula-storaged-4   1/1     Running   0          52s

For more information, see Install Nebula Cluster with helm in NebulaGraph Database Manual.

Connecting to NebulaGraph Cluster

If the NebulaGraph cluster is started successfully, here comes the last step: Connect to the cluster.

Connecting from Inside of Kubernetes

Firstly, on Kubernetes, run the following command to start NebulaGraph Console.

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nebula-console
spec:
  containers:
    - name: nebula-console
      image: vesoft/nebula-console:v2-nightly
      command:
      - sleep
      - "1000000"
EOF

And then, run this command to connect to the cluster via the created NebulaGraph Console.

$ kubectl exec -it nebula-console -- \
    nebula-console -u u -p p --addr ${NEBULA_CLUSTER_NAME}-graphd-svc.${NEBULA_CLUSTER_NAMESPACE}.svc --port 9669
2021/06/23 06:21:22 [INFO] connection pool is initialized successfully
Welcome to NebulaGraph!
(u@nebula) [(none)]> show hosts
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| Host                                                                  | Port | Status   | Leader count | Leader distribution  | Partition distribution |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-3.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-4.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "Total"                                                               |      |          | 0            |                      |                        |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
Got 4 rows (time spent 7669/9367 us)
Wed, 23 Jun 2021 06:21:26 UTC

Connecting from Outside of Kubernetes

If you want to connect to the services inside Kubernetes from outside, use hostPort, hostNetwork, Ingress, LoadBalancer, and so on. In this practice, I took advantages of the functionalities of Alibaba Cloud and used LoadBalancer.

DO NOT: This operation will expose your NebulaGraph Cluster to the public, so do not use it in the production environment.

Firstly, change the type of nebula-graphd to LoadBalancer, and then view EXTERNAL-IP.

# Change the type of nebula-graphd to LoadBalancer.
$ kubectl patch -n ${NEBULA_CLUSTER_NAMESPACE} svc ${NEBULA_CLUSTER_NAME}-graphd-svc \
    -p '{"spec": {"type": "LoadBalancer"}}'
# View EXTERNAL-IP. If "pending" is shown, wait a moment and try again.
$ kubectl -n ${NEBULA_CLUSTER_NAMESPACE} get svc nebula-graphd-svc
NAME                TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                                          AGE
nebula-graphd-svc   LoadBalancer   172.16.85.222   x.x.x.x         9669:31460/TCP,19669:32579/TCP,19670:31481/TCP   27m

Now, you can use the EXTERNAL-IP address (for example, x.x.x.x in this example) to connect to the cluster.

$ export EXTERNAL_IP=x.x.x.x
$ docker run -it --rm vesoft/nebula-console:v2-nightly -u u -p p --addr ${EXTERNAL_IP} --port 9669
2021/06/23 06:42:17 [INFO] connection pool is initialized successfully
Welcome to NebulaGraph!
(u@nebula) [(none)]> show hosts
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| Host                                                                  | Port | Status   | Leader count | Leader distribution  | Partition distribution |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-3.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-4.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "Total"                                                               |      |          | 0            |                      |                        |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
Got 4 rows (time spent 3747/60433 us)
Wed, 23 Jun 2021 06:42:21 UTC

Enjoy

You made it!

Have fun with NebulaGraph!

Join our Slack channel if you want to discuss with the rest of the NebulaGraph community!