Hello, I'm Yan Zhenfan, a researcher of Microsoft MVP laboratory in this issue. Today, I will share with you through code examples how to write components using Kubernetes API Server to obtain the resource object information of the cluster from K8S.
Yan Zhenfan, Microsoft's most valuable expert, is currently learning about microservices. You can communicate more~
preface
Some time ago, I used C# to write a project, using Kubernetes API Server to obtain information and monitor Kubernetes resources, and then combined with Neting to make API gateway.
Experience address http://neting.whuanle.cn:30080/
Account admin, password admin123
This article mainly introduces how to obtain the information of various resources in Kubernetes and the prerequisite knowledge of Conroller through C# developing the application based on Kubernetes.
Kubernetes API Server
Kube apiserver is one of the k8s main processes. Apiserver component discloses Kubernetes API (HTTP API). Apiserver is the front end of Kubernetes control surface. We can write code in Go, C# and other programming languages and call Kubernetes remotely to control the operation of the cluster. The exposed endiont port of apiserver is 6443.
In order to control the operation of the cluster, Kubernetes officially provides a binary command-line tool called kubectl. Apiserver provides the interface service. Kubectl parses the instructions entered by the user, sends an HTTP request to apiserver, and then feeds back the results to the user.
kubectl is a very powerful cluster control tool provided by Kubernetes. It manages the whole cluster through command line operation.
Kubernetes has many visual panels, such as Dashboard. Behind it is the API calling apiserver, which is equivalent to calling the front end to the back end.
In short, the backend of various cluster management tools we use is apiserver. Through apiserver, we can also customize various cluster management tools, such as grid management tool istio. Tencent cloud, Alibaba cloud and other cloud platforms provide online kubernetes services, as well as console visual operation, which also makes use of apiserver.
You can refer to the Kubernetes e-book written by the author to learn more: https://k8s.whuanle.cn/1.basic/5.k8s.html
In short, Kubernetes API Server is the entry point for third-party operation of Kubernetes.
Expose Kubernetes API Server
First, check the Kubernetes component running in Kube system. There is a Kube apiserver master running.
root@master:~# kubectl get pods -o wide -n kube-system NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ... ... kube-apiserver-master 1/1 Running 2 (76d ago) 81d 10.0.0.4 master <none> <none> ... ...
Although these components are important, there will only be one instance and run in the form of Pod instead of Deployment. These components can only be run on the master node.
Then check admin Conf file, which can be accessed through / etc / kubernetes / Admin Conf or $home / Kube / config path.
admin.conf file is the certificate to access Kubernetes API Server. Through this file, we can access Kubernetes API interface programmatically.
But admin Conf is a very important file. If it is a development environment development cluster, make it casually. If it is a production environment, do not use it. You can restrict API access authorization through role binding and other methods.
Then put admin Download the conf or config file locally.
You can use the kubectl edit pods Kube apiserver master - n Kube system command to view some configuration information of Kubernetes API Server.
Since the Kubernetes API Server is accessed within the cluster by default, if remote access is required, it needs to be exposed outside the cluster (it has nothing to do with whether it is all in the intranet and whether it is in the cluster).
Expose the API Server outside the cluster:
kubectl expose pod kube-apiserver-master --type=NodePort --port=6443 -n kube-system
View ports randomly assigned by nodes:
root@master:~# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-apiserver-master NodePort 10.101.230.138 <none> 6443:32263/TCP 25s
32263 port is automatically assigned by Kubernetes, and everyone's is different.
Then, you can test the access through {IP:32263}.
If your cluster has CoreDNS installed, you can also access this service through the IP of other nodes.
Then download the admin Conf or config file (please change the name to admin.conf) and modify the "server" attribute in it, because we access it remotely at this time.
Connect to API Server
Create a new MyKubernetes console project, and then add admin The conf file is copied into the project and output is generated with the project.
Then search the KubernetesClient package in Nuget. The author currently uses 7.0.1.
Then set the environment variables in the project:
The environment variable itself is ASP Net core comes with it, not in the console program.
Write a method to instantiate and obtain Kubernetes client:
private static Kubernetes GetClient()
{
KubernetesClientConfiguration config;
if (Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") == "Development")
{
// By profile
config = KubernetesClientConfiguration.BuildConfigFromConfigFile("./admin.conf");
}
else
{
// Accessed through the default Service Account, it can only be used when running in kubernetes
config = KubernetesClientConfiguration.BuildDefaultConfig();
}
return new Kubernetes(config);
}The logic is very simple. If it is a development environment, use admin Conf file access. If it is a non development environment, buildedefaultconfig() automatically obtains the access credentials. This method is only valid when running in Pod and uses Service Account authentication.
Let's test and get all the namespaces:
static async Task Main()
{
var client = GetClient();
var namespaces = await client.ListNamespaceAsync();
foreach (var item in namespaces.Items)
{
Console.WriteLine(item.Metadata.Name);
}
}
okay! You can already get Kubernetes resources. The first step to getting started! Xiuer!
Client side knowledge
Although the first step of getting started is opened, don't rush to use various API s. Here, let's learn about the definition of various Kubernetes resources in the client and how to parse the structure.
First, in the code of Kubernetes Client C#, all model classes of Kubernetes resources are in k8s Records in models.
If we want to view the definition of an object in Kubernetes, such as the "Kube Systtem" namespace:
kubectl get namespace kube-system -o yaml
apiVersion: v1
kind: Namespace
metadata:
creationTimestamp: "2021-11-03T13:57:10Z"
labels:
kubernetes.io/metadata.name: kube-system
name: kube-system
resourceVersion: "33"
uid: f0c1f00d-2ee4-40fb-b772-665ac2a282d7
spec:
finalizers:
- kubernetes
status:
phase: ActiveC# as like as two peas:
In the client, the name of the model is prefixed with the apiVersion version version, and the list of such objects is obtained through V1NamespaceList.
If you want to obtain a certain type of resources, its interfaces start with List, such as client ListNamespaceAsync(),client.ListAPIServiceAsync(),client.ListPodForAllNamespacesAsync(), etc.
It seems that learning is on the right track. Let's experiment and practice!
Practice 1: how to parse a Service
Here, the author has carefully prepared some exercises for readers. The first exercise is to analyze the information of a Service.
View the service created earlier:
kubectl get svc kube-apiserver-master -n kube-system -o yaml
The corresponding structure is as follows:
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2022-01-24T12:51:32Z"
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver-master
namespace: kube-system
resourceVersion: "24215604"
uid: ede0e3df-8ef6-45c6-9a8d-2a2048c6cb12
spec:
clusterIP: 10.101.230.138
clusterIPs:
- 10.101.230.138
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- nodePort: 32263
port: 6443
protocol: TCP
targetPort: 6443
selector:
component: kube-apiserver
tier: control-plane
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
We define a model class in C #:
public class ServiceInfo
{
/// <summary>
///SVC name
/// </summary>
public string Name { get; set; } = null!;
/// <summary>
///One of three types < see CREF = "ServiceType" / >
/// </summary>
public string? ServiceType { get; set; }
/// <summary>
///Namespace
/// </summary>
public string Namespace { get; set; } = null!;
/// <summary>
///Some services do not have this option
/// </summary>
public string ClusterIP { get; set; } = null!;
/// <summary>
///Internet access IP
/// </summary>
public string[]? ExternalAddress { get; set; }
public IDictionary<string, string>? Labels { get; set; }
public IDictionary<string, string>? Selector { get; set; }
/// <summary>
/// name,port
/// </summary>
public List<string>? Ports { get; set; }
public string[]? Endpoints { get; set; }
public DateTime? CreationTime { get; set; }
// Associated pod and ip of pod
}Next, specify which namespace to get the Service and its associated Endpoint information.
static async Task Main()
{
var result = await GetServiceAsync("kube-apiserver-master","kube-system");
Console.WriteLine(System.Text.Json.JsonSerializer.Serialize(result));
}
public static async Task<ServiceInfo> GetServiceAsync(string svcName, string namespaceName)
{
var client = GetClient();
var service = await client.ReadNamespacedServiceAsync(svcName, namespaceName);
// Get the information of the service itself
ServiceInfo info = new ServiceInfo
{
Name = service.Metadata.Name,
Namespace = service.Metadata.NamespaceProperty,
ServiceType = service.Spec.Type,
Labels = service.Metadata.Labels,
ClusterIP = service.Spec.ClusterIP,
CreationTime = service.Metadata.CreationTimestamp,
Selector = service.Spec.Selector.ToDictionary(x => x.Key, x => x.Value),
ExternalAddress = service.Spec.ExternalIPs?.ToArray(),
};
// Service - > endpoint information
var endpoint = await client.ReadNamespacedEndpointsAsync(svcName, namespaceName);
List<string> address = new List<string>();
foreach (var sub in endpoint.Subsets)
{
foreach (var addr in sub.Addresses)
{
foreach (var port in sub.Ports)
{
address.Add($"{addr.Ip}:{port.Port}/{port.Protocol}");
}
}
}
info.Endpoints = address.ToArray();
return info;
}The output results are as follows:
Pro, if you don't know much about Kubernetes' network knowledge, please open it first https://k8s.whuanle.cn/4.network/1.network.html Find out.
Practice 2: parsing Service attributes in detail
We know that a service can associate multiple pods and provide load balancing and other functions for multiple pods. At the same time, a service has attributes such as externalIP and clusterIP. It is difficult to really parse a service. For example, only service ports can be used; You can also use only DNS domain name to access; You can also access service B indirectly from service a DNS - > Service B IP without binding any Pod;
Services contain many situations. Readers can refer to the following figure. Next, we obtain the IP and port information of a Service through code, and then generate the corresponding IP + port structure.
It is useless to simply obtain the IP and port, because they are separate. The IP you obtain may be from Cluter, Node and LoadBalancer. It is possible that DNS has no IP. How can you access this port? At this time, we must analyze the information and filter the invalid data according to certain rules in order to get a useful access address.
First, define some enumerations and models:
public enum ServiceType
{
ClusterIP,
NodePort,
LoadBalancer,
ExternalName
}
/// <summary>
///Kubernetes Service and IP
/// </summary>
public class SvcPort
{
// LoadBalancer -> NodePort -> Port -> Target-Port
/// <summary>
/// 127.0.0.1:8080/tcp,127.0.0.1:8080/http
/// </summary>
public string Address { get; set; } = null!;
/// <summary>
/// LoadBalancer,NodePort,Cluster
/// </summary>
public string Type { get; set; } = null!;
public string IP { get; set; } = null!;
public int Port { get; set; }
}
public class SvcIpPort
{
public List<SvcPort>? LoadBalancers { get; set; }
public List<SvcPort>? NodePorts { get; set; }
public List<SvcPort>? Clusters { get; set; }
public string? ExternalName { get; set; }
}Write parsing code:
static async Task Main()
{
var result = await GetSvcIpsAsync("kube-apiserver-master","kube-system");
Console.WriteLine(System.Text.Json.JsonSerializer.Serialize(result));
}
public static async Task<SvcIpPort> GetSvcIpsAsync(string svcName, string namespaceName)
{
var client = GetClient();
var service = await client.ReadNamespacedServiceAsync(svcName, namespaceName);
SvcIpPort svc = new SvcIpPort();
// LoadBalancer
if (service.Spec.Type == nameof(ServiceType.LoadBalancer))
{
svc.LoadBalancers = new List<SvcPort>();
var ips = svc.LoadBalancers;
// Load balancer IP
var lbIP = service.Spec.LoadBalancerIP;
var ports = service.Spec.Ports.Where(x => x.NodePort != null).ToArray();
foreach (var port in ports)
{
ips.Add(new SvcPort
{
Address = $"{lbIP}:{port.NodePort}/{port.Protocol}",
IP = lbIP,
Port = (int)port.NodePort!,
Type = nameof(ServiceType.LoadBalancer)
});
}
}
if (service.Spec.Type == nameof(ServiceType.LoadBalancer) || service.Spec.Type == nameof(ServiceType.NodePort))
{
svc.NodePorts = new List<SvcPort>();
var ips = svc.NodePorts;
// Load balancer IP. In some cases, ClusterIP can be set to None; It can also be manually set to None, as long as there is a public IP
var clusterIP = service.Spec.ClusterIP;
var ports = service.Spec.Ports.Where(x => x.NodePort != null).ToArray();
foreach (var port in ports)
{
ips.Add(new SvcPort
{
Address = $"{clusterIP}:{port.NodePort}/{port.Protocol}",
IP = clusterIP,
Port = (int)port.NodePort!,
Type = nameof(ServiceType.NodePort)
});
}
}
// The following part of the code is normal. Use {} to isolate part of the code and avoid duplicate variable names
// if (service.Spec.Type == nameof(ServiceType.ClusterIP))
// If the Service does not have Cluster IP, it may use headless mode, or it may not want to have Cluster IP
//if(service.Spec.ClusterIP == "None")
{
svc.Clusters = new List<SvcPort>();
var ips = svc.Clusters;
var clusterIP = service.Spec.ClusterIP;
var ports = service.Spec.Ports.ToArray();
foreach (var port in ports)
{
ips.Add(new SvcPort
{
Address = $"{clusterIP}:{port.Port}/{port.Protocol}",
IP = clusterIP,
Port = port.Port,
Type = nameof(ServiceType.ClusterIP)
});
}
}
if (!string.IsNullOrEmpty(service.Spec.ExternalName))
{
/* NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
myapp-svcname ExternalName <none> myapp.baidu.com <none> 1m
myapp-svcname -> myapp-svc
Visit myapp SVC default. svc. cluster. Local, become myapp baidu. com
*/
svc.ExternalName = service.Spec.ExternalName;
}
return svc;
}The rule analysis is complex, so it will not be explained in detail here. If you have any questions, you can contact the author for discussion.
Main rules: loadbalancer - > nodeport - > port - > target port.
The final results are as follows:
Through this part of the code, you can analyze the address list that the Service can really access under the conditions of External Name, LoadBalancer, NodePort, ClusterIP, etc.
Practice 3: parsing Endpoint list
If you don't know much about Endpoint, please open it https://k8s.whuanle.cn/4.network/2.endpoint.html Take a look at the relevant knowledge.
In Kubernetes, Service is not directly associated with Pod, but indirectly proxies Pod through Endpoint. Of course, in addition to Service - > Pod, you can also access third-party services outside the cluster through Endpoint. For example, if the database cluster is not in the Kubernetes cluster, but you want unified access through the Kubernetes Service, you can use Endpoint to decouple. Not much to say here, readers can refer to https://k8s.whuanle.cn/4.network/2.endpoint.html .
In this section, the author will also explain how to get resources by paging in Kubernetes.
First define the following model:
public class SvcInfoList
{
/// <summary>
///Paging attribute, with temporary validity period, which is determined by Kubernetes
/// </summary>
public string? ContinueProperty { get; set; }
/// <summary>
///Estimated remaining quantity
/// </summary>
public int RemainingItemCount { get; set; }
/// <summary>
///SVC list
/// </summary>
public List<SvcInfo> Items { get; set; } = new List<SvcInfo>();
}
public class SvcInfo
{
/// <summary>
///SVC name
/// </summary>
public string Name { get; set; } = null!;
/// <summary>
///One of three types < see CREF = "ServiceType" / >
/// </summary>
public string? ServiceType { get; set; }
/// <summary>
///Some services have no IP and the value is None
/// </summary>
public string ClusterIP { get; set; } = null!;
public DateTime? CreationTime { get; set; }
public IDictionary<string, string>? Labels { get; set; }
public IDictionary<string, string>? Selector { get; set; }
/// <summary>
/// name,port
/// </summary>
public List<string> Ports { get; set; }
public string[]? Endpoints { get; set; }
}Paging in Kubernetes does not include PageNo, PageSize, Skip, Take and Limit, and paging may only be expected and may not be completely accurate.
You cannot use the ContinueProperty property property the first time you access the get object list.
After accessing Kubernets for the first time and obtaining 10 pieces of data, Kubernets will return a ContinueProperty token and the remaining quantity RemainingItemCount.
Then we can calculate the approximate paging number through RemainingItemCount. Because Kubernetes cannot page directly, it records the current access position through something similar to a cursor, and then continues to get the object down. ContinueProperty holds the token of the current query cursor, but the token is valid for a few minutes.
Resolution method:
public static async Task<SvcInfoList> GetServicesAsync(string namespaceName,
int pageSize = 1,
string? continueProperty = null)
{
var client = GetClient();
V1ServiceList services;
if (string.IsNullOrEmpty(continueProperty))
{
services = await client.ListNamespacedServiceAsync(namespaceName, limit: pageSize);
}
else
{
try
{
services = await client.ListNamespacedServiceAsync(namespaceName,
continueParameter: continueProperty,
limit: pageSize);
}
catch (Microsoft.Rest.HttpOperationException ex)
{
throw ex;
}
catch
{
throw;
}
}
SvcInfoList svcList = new SvcInfoList
{
ContinueProperty = services.Metadata.ContinueProperty,
RemainingItemCount = (int)services.Metadata.RemainingItemCount.GetValueOrDefault(),
Items = new List<SvcInfo>()
};
List<SvcInfo> svcInfos = svcList.Items;
foreach (var item in services.Items)
{
SvcInfo service = new SvcInfo
{
Name = item.Metadata.Name,
ServiceType = item.Spec.Type,
ClusterIP = item.Spec.ClusterIP,
Labels = item.Metadata.Labels,
Selector = item.Spec.Selector,
CreationTime = item.Metadata.CreationTimestamp
};
// Processing port
if (item.Spec.Type == nameof(ServiceType.LoadBalancer) || item.Spec.Type == nameof(ServiceType.NodePort))
{
service.Ports = new List<string>();
foreach (var port in item.Spec.Ports)
{
service.Ports.Add($"{port.Port}:{port.NodePort}/{port.Protocol}");
}
}
else if (item.Spec.Type == nameof(ServiceType.ClusterIP))
{
service.Ports = new List<string>();
foreach (var port in item.Spec.Ports)
{
service.Ports.Add($"{port.Port}/{port.Protocol}");
}
}
var endpoint = await client.ReadNamespacedEndpointsAsync(item.Metadata.Name, namespaceName);
if (endpoint != null && endpoint.Subsets.Count != 0)
{
List<string> address = new List<string>();
foreach (var sub in endpoint.Subsets)
{
if (sub.Addresses == null) continue;
foreach (var addr in sub.Addresses)
{
foreach (var port in sub.Ports)
{
address.Add($"{addr.Ip}:{port.Port}/{port.Protocol}");
}
}
}
service.Endpoints = address.ToArray();
}
svcInfos.Add(service);
}
return svcList;
}The rule analysis is complex, so it will not be explained in detail here. If you have any questions, you can contact the author for discussion.
Call method:
static async Task Main()
{
var result = await GetServicesAsync("default", 2);
Console.WriteLine(System.Text.Json.JsonSerializer.Serialize(result.Items));
if (result.RemainingItemCount != 0)
{
while (result.RemainingItemCount != 0)
{
Console.WriteLine($"surplus {result.RemainingItemCount} Data,{result.RemainingItemCount / 3 + (result.RemainingItemCount % 3 == 0 ? 0 : 1)} Page, press enter to continue to obtain!");
Console.ReadKey();
result = await GetServicesAsync("default", 2, result.ContinueProperty);
Console.WriteLine(System.Text.Json.JsonSerializer.Serialize(result.Items));
}
}
}
In the above practice, there are many codes. It is recommended that readers debug after startup, debug step by step, slowly check the data, compare various objects in Kubernetes, and gradually deepen their understanding.
The next article will explain how to implement Conroller and Kubernetes Operator. Coming soon!
Microsoft MVP
Microsoft's most valuable expert is a global award awarded by Microsoft to third-party technology professionals. Over the past 29 years, technology community leaders around the world have won this award for sharing expertise and experience in their online and offline technology communities.
MVP is a strictly selected team of experts. They represent the most skilled and intelligent people. They are experts who are enthusiastic and helpful to the community. We are committed to creating Microsoft's open-source website, writing a speech, organizing a conference, and helping others by using Microsoft's MVP technology.
For more details, please visit the official website:
https://mvp.microsoft.com/zh-cn