Author: Yuanyi, senior development engineer of Alibaba cloud intelligent business group

background

Message queuing for Apache Kafka is a distributed, high throughput and scalable Message Queuing service provided by alicloud. Message queuing for Apache Kafka is widely used in log collection, monitoring data aggregation, streaming data processing, online and offline analysis and other big data fields, and has become an indispensable part of big data ecology.

Combined with the support of KafkaSource event source provided in Knight, Kafka message service can be easily docked.

You can also install community Kafka 2.0.0 and above.

Create Kafka instance on alicloud

Create Kafka instance

Sign in Message queuing Kafka console , select purchase instance. Since the current Kafka event source in known supports version 2.0.0 and above, creating Kafka instances on Alibaba cloud requires purchasing the yearly, monthly and professional versions, and upgrading to 2.0.0 after purchase.

Deploy instance and bind VPC

After purchase, deploy. When deploying, set the VPC where the knowledgeable cluster is located.

Create Topic and Consumer Group

Next, we create topics and consumption groups. < br / > Enter topic management and click Create topic. Here we create a topic named demo:

Enter Consumer Group management, and click create Consumer Group. Here we create a Consumer Group named demo consumer:

Deploy Kafka data source

Deploy Kafka addon components

Log in to the container service console, enter "knowledge component management", and deploy Kafka addon component.

Create KafkaSource instance

First, create the service event display for receiving events:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: event-display
spec:
  template:
    spec:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/knative-sample/eventing-sources-cmd-event_display:bf45b3eb1e7fc4cb63d6a5a6416cf696295484a7662e0cf9ccdf5c080542c21d

Next, create KafkaSource:

apiVersion: sources.eventing.knative.dev/v1alpha1
kind: KafkaSource
metadata:
  name: alikafka-source
spec:
  consumerGroup: demo-consumer
  # Broker URL. Replace this with the URLs for your kafka cluster,
  # which is in the format of my-cluster-kafka-bootstrap.my-kafka-namespace:9092.
  bootstrapServers: 192.168.0.6x:9092,192.168.0.7x:9092,192.168.0.8x:9092
  topics: demo
  sink:
    apiVersion: serving.knative.dev/v1alpha1
    kind: Service
    name: event-display

Explain:

• Bootstrap servers: Kafka VPC access address
• consumerGroup: set consumption group
• Topic s: setting topics

After creation, we can check that the corresponding instance has been run:

[root@iZ2zeae8wzyq0ypgjowzq2Z ~]# kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
alikafka-source-k22vz-db44cc7f8-879pj   1/1     Running   0          8h

<a name="cd8992b6"></a>

Verification

In the Kafka console, select topic to send the message. Note that the message format here must be json format:

We can see that we have received the Kafka message:

[root@iZ2zeae8wzyq0ypgjowzq2Z ~]# kubectl logs event-display-zl6m5-deployment-6bf9596b4f-8psx4 user-container

☁️  CloudEvent: valid ✅
Context Attributes,
  SpecVersion: 0.2
  Type: dev.knative.kafka.event
  Source: /apis/v1/namespaces/default/kafkasources/alikafka-source#demo
  ID: partition:7/offset:1
  Time: 2019-10-18T08:50:32.492Z
  ContentType: application/json
  Extensions: 
    key: demo
Transport Context,
  URI: /
  Host: event-display.default.svc.cluster.local
  Method: POST
Data,
  {
    "key": "test"
  }

<a name="5db9fd7c"></a>

Summary

Combined with Alibaba cloud Kafka products, is it simple and efficient to trigger service (function) execution through event driven? In this way, we can fully release the cloud's native ability by using knight, and bring us more imagination space. Welcome to communicate with people who are interested in knight.

Link to demonstration of operation video: https://v.qq.com/x/page/d3013wzs3tp.html

About the author: Yuan Yi, senior development engineer of Alibaba cloud container platform, is responsible for the work related to the knowledge of Alibaba cloud container platform.

Welcome to the interactive group