Kubernetes Configuration Synchronization Mechanism

# Synchronization Mechanism

The configuration synchronization process of Kubernetes can be divided into two stages: Initialization Stage and Synchronization Stage.

1. Initialization Stage: When OpenResty Edge Admin just starts, it initializes the Kubernetes configuration monitoring manager. In the scenario of starting multiple Edge Admin instances, only the Edge Admin that gets the lock will monitor the Kubernetes configuration.

2. Synchronization Stage: The listener manager will regularly start the listener, each Kubernetes cluster will start two listeners (for example, in the case of three Kubernetes clusters, there will be six listeners), which will listen to the related events of /api/v1/watch/services and /api/v1/watch/endpoints and process them.

# Event Processing

Currently, we support processing the following events:

# Endpoint Addition Event

• Event Type: ADDED

• Triggering Time:

  • When connecting to the Kubernetes watch interface for the first time.
  • When exposing the deployment (creating service), for example, using the command kubectl expose deployment/test-hello.

• Event Handling: Edge Admin will update the upstream node information corresponding to the pod on Kubernetes.

• Event Source: /api/v1/watch/endpoints

• Event Example:

  {
      "type": "ADDED",
      "object": {
          "apiVersion": "v1",
          "kind": "Endpoints",
          "metadata": {
              "name": "test-hello",
              "namespace": "default",
              ...
          },
          ...
      }
  }
  

# Endpoint Modification Event

• Event Type: MODIFIED

• Triggering Time:

  • When scaling up or down the pod, this event will be triggered.
  • When deleting the deployment, for example, using the command kubectl delete deployment test-hello.

• Event Handling: Edge Admin will update the upstream node information corresponding to the pod on Kubernetes.

• Event Source: /api/v1/watch/endpoints

• Event Example:

  {
      "type": "MODIFIED",
      "object": {
          "apiVersion": "v1",
          "kind": "Endpoints",
          "metadata": {
              "name": "test-hello",
              "namespace": "default",
              ...
          },
          ...
      }
  }
  

# Service Addition Event

• Event Type: ADDED

• Triggering Time: This event will be triggered after adding the service.

• Event Handling: Edge Admin will add the corresponding DNS record.

• Event Source: /api/v1/watch/services

• Event Example:

  {
      "type": "ADDED",
      "object": {
          "kind": "Service",
          "metadata": {
              "name": "test-hello",
              "resourceVersion": "3686",
              "namespace": "default",
              ...
          },
          "apiVersion": "v1",
          "spec": {
              "selector": {
                  "app": "test-hello"
              },
              "ports": [
                  {
                      "port": 80,
                      "protocol": "TCP",
                      "nodePort": 30476,
                      "targetPort": 80
                  }
              ],
              "clusterIPs": [
                  "10.68.181.72"
              ],
              ...
          }
      }
  }
  

# Service Modification Event

• Event Type: MODIFIED

• Triggering Time: This event will be triggered after modifying the service, for example using the command kubectl edit svc test-hello.

• Event Handling: Edge Admin will modify the corresponding DNS record.

• Event Source: /api/v1/watch/services

• Event Example:

  {
      "type": "MODIFIED",
      "object": {
          "kind": "Service",
          "metadata": {
              "name": "test-hello",
              "namespace": "default",
              ...
          },
          "apiVersion": "v1",
          "spec": {
              "selector": {
                  "app": "test-hello"
              },
              "clusterIP": "10.68.181.72",
              "ports": [
                  {
                      "port": 80,
                      "name": "http",
                      "protocol": "TCP",
                      "nodePort": 30476,
                      "targetPort": 80
                  },
                  {
                      "port": 88,
                      "name": "http2",
                      "protocol": "TCP",
                      "nodePort": 30478,
                      "targetPort": 88
                  }
              ],
              "clusterIPs": [
                  "10.68.181.72"
              ],
              ...
          }
      }
  }
  

# Service Deletion Event

• Event Type: DELETE

• Triggering Time: This event will be triggered after deleting the service.

• Event Handling: Edge Admin will delete the corresponding DNS record.

• Event Source: /api/v1/watch/services

• Event Example:

  {
      "type": "DELETED",
      "object": {
          "kind": "Service",
          "metadata": {
              "name": "test-hello",
              "namespace": "default",
              ...
          },
          "apiVersion": "v1",
          "status": {
              "loadBalancer": []
          },
          "spec": {
              "selector": {
                  "app": "test-hello"
              },
              "clusterIP": "10.68.217.25",
              "ports": [
                  {
                      "port": 81,
                      "protocol": "TCP",
                      "nodePort": 31183,
                      "targetPort": 81
                  }
              ],
              "clusterIPs": [
                  "10.68.217.25"
              ],
              ...
          }
      }
  }
  

# Error Code Explanation

During the configuration synchronization process, there may be listening errors or event handling failures. Here are explanations of all error codes:

# Frequently Asked Questions

1. What is the delay of Kubernetes configuration synchronization?

   Events in Kubernetes will be processed in a merged way, with a merging window of 1 second, so the configuration synchronization usually has a delay of about 1 second. For example, if Edge Admin receives two pod change event notifications from the same deployment within 1 second, Edge Admin will combine the two notifications and make one configuration change after 1 second.

2. Will Kubernetes configuration be fully synchronized?

   When starting the watcher, you will immediately receive the ADDED type events of service and endpoint in the Kubernetes cluster, and then fully synchronize the corresponding configuration.

3. Will changes to Kubernetes pod result in release records in Edge Admin?

   Due to the frequent changes in Kubernetes, Edge Admin does not record release logs, but there will be audit logs.

4. When will watchers be added and deleted?

   After adding the Kubernetes configuration to Edge Admin, the watcher will be restarted regularly to prevent coroutine resource leakage. After deleting the Kubernetes configuration from Edge Admin, the corresponding watcher will be recycled.

5. If the configuration does not change, will the received events still be processed?

   When Edge Admin receives an event from the Kubernetes cluster, it will compare the related configurations. If the configuration does not change, it will not update the upstream, etc.