Deploying Apache pulsar on kubesphere

The authors introduce ： Xu Wentao ,StreamNative Content Strategist, Love cloud native and open source technologies , Active in localization / file / Technology blog contribution , hold K8s CKA/CKAD/CKS authentication .

Apache Pulsar Introduce

Apache Pulsar As Apache Software foundation top projects , It is the next generation cloud native distributed message flow platform , Set message 、 Storage 、 Lightweight functional calculation as a whole , Using the separation of computing and storage architecture design , Multi tenant support 、 Persistent storage 、 Cross regional replication 、 Hierarchical storage , With strong consistency 、 High throughput 、 Low latency and high scalability streaming data storage features , It is the cloud native era to solve the real-time message stream data transmission 、 The best solution for storage and Computing .

Pulsar Of Broker There is no state , Don't store data .BookKeeper Responsible for storing data , Among them Bookie Support horizontal expansion , Metadata information is stored in ZooKeeper On . This architecture also facilitates the expansion and contraction of the container environment , Support high availability and other scenarios .

KubeSphere Introduce

KubeSphere It's based on Kubernetes A distributed operating system for cloud native applications , Fully open source , Support multi cloud and multi cluster management 、 The app store 、 Observability （ monitor 、 Alarm, audit, etc ） And multi tenancy , Provide a full stack of IT Automatic operation and maintenance capability , Simplify the enterprise DevOps workflow .

stay KubeSphere Installation on Apache Pulsar

As an application centric K8s Container platform ,KubeSphere Provide users with a variety of ways to install applications . For rapid deployment of applications , Here we recommend the following two methods ：

• Directly from KubeSphere Install... In the app store of Pulsar.KubeSphere from 3.2.0 Added at the beginning of the version “ Dynamically load the app store ” The function of . Small partners who want to contribute applications can directly report to KubeSphere Of Helm Warehouse Contribute to the application Helm Chart, stay PR After approval, the latest app list will be loaded on the app store .
• add to Apache Pulsar Of Helm Warehouse to KubeSphere Application warehouse of , Then install... From the application repository Pulsar Application of the template . This installation is similar to the command line helm Related commands add the warehouse and install .

notes ：KubeSphere The app store and the full lifecycle management function of apps are based on open source projects OpenPitrix. In the installation Pulsar front , You need first. stay KubeSphere Enable OpenPitrix.

Let's use the second method in KubeSphere Console Add Pulsar Of Helm Warehouse and install .

1. Prepare one KubeSphere colony . You can directly Use KubeKey install Or through ks-installer Already exist K8s Build on the cluster KubeSphere colony . meanwhile , Please note that Pulsar In the document for K8s Clusters and Helm Of Version for .

2. Create enterprise spaces and projects . The enterprise space is KubeSphere A logical unit in , For managing projects 、DevOps Resources such as projects and application templates , The administrator can manage the resource access rights ; Project means K8s Medium Namespace. For the convenience of demonstration , This article adopts admin The user action . be based on KubeSphere Multi tenant system , In a real-world environment, you can base on the responsibilities of each tenant in the organization , Create different accounts and assign corresponding roles .

3. In your corporate space , Click on Application Management > Application warehouse , Then click Add . After defining the warehouse name , add to https://pulsar.apache.org/charts Warehouse address . After the warehouse address is verified successfully , Click ok .

4. After the warehouse is successfully added , We also need to run a script , Create some and Pulsar dependent Secret And so on , For our follow-up visit Grafana and Pulsar Manager Tools such as . Go to your K8s node , Execution of the following command will pulsar-helm-chart Of GitHub Clone the warehouse locally . It should be noted that , You must be able to execute on this node kubectl and helm command .

   git clone https://github.com/apache/pulsar-helm-chartcd pulsar-helm-chart

5. Execute the following command to run prepare_helm_release.sh Script . among -n Of or relating to Secret Generated Namespace,-k Refers to the use helm Install the application release name , These two places need to be with us KubeSphere The project specified on the corresponds to the application name . More information about the parameters available for this script , Can run ./scripts/pulsar/prepare_helm_release.sh --help Check the details .

   ./scripts/pulsar/prepare_helm_release.sh -n pulsar -k ks-pulsar

6. Go to KubeSphere Project created in , The project in this example is named pulsar.

7. Click apply load > application , Under the template based application tab , Click Create .

8. Select apply template from , Select... From the drop-down list of the application template Pulsar The warehouse of . Click on Pulsar after , We can see it here Chart Some information about the file .

9. Click on the install . Define application name （ That is, when executing the script just now release name ）, Select the version and installation location （ If you enable the multi cluster feature and add multiple clusters , There is also a choice Pulsar The cluster where the installation is located ）, Click next .

10. Go to the application settings page , This is similar to using helm Command time customization values.yaml Document parameters . We set up namespace Is the project name ,initialize by true（ First installation only Pulsar You need to set ）, And enable the Pulsar Manager（ Set up components.pulsar_manager by true）, Then click Install . More specific information about these parameters , May refer to Pulsar file .

11. Wait a while , see Pulsar Whether the components of have been in normal operation .

Use Pulsar Manager establish Tenant、Namespace as well as Topic

Pulsar After the cluster is built , We can use Web Tools Pulsar Manager To manage Pulsar Tenants in （Tenant）、 Namespace （Namespace） And the theme （Topic）. These resources can be used to ensure data isolation between different departments in the same organization , As shown in the figure below .

1. Go to the application load > service , Entry exposure Pulsar Manager Of Service The details page of , And according to which LoadBalancer Type of IP Address plus slogan access Pulsar Manager. Note here , Depending on the deployment environment, you may also need to configure the corresponding firewall or port forwarding rules .
2. Through the default account password pulsar/pulsar Sign in , Then add the environment .Service URL Fill in here Broker Of Service Address and port number , In this example http://ks-pulsar-broker:8080.
3. Get into Pulsar Manager Of Dashboard after , Add... Under different menus Tenant、Namespace as well as Topic. Definition Topic We need to select the Topic The type of （ Persistent or non persistent ）, And designate Topic The partition （Partition） Number . stay Pulsar in ,Topic The format is ：{persistent | non-persistent}://tenant/namespace/topic. We need to record this information first , Back to the Topic When sending a message .

towards Pulsar The cluster sends information and consumes

Now we have created the relevant resources , You can try to Pulsar The cluster sends information and consumes .

1. Download it locally first Pulsar Binary file , Then decompress .

2. Get into Pulsar After the directory is modified conf/client.conf In the document webServiceUrl and brokerServiceUrl Address , Is used to Pulsar Client and Pulsar Cluster binding . In this example, the proxy access cluster is enabled , The address here needs to fill in the... Exposed by the agent IP Address . We can do it in KubeSphere Service page to find the proxy Service, And see... On its details page IP Address .

   webServiceUrl=http://34.71.201.104:8080brokerServiceUrl=pulsar://34.71.201.104:6650

3. After modification , stay Pulsar Execute the following command under the directory to create a subscription and consume information , In the command kubesphere/kubesphere/kubesphere Corresponding to just now Pulsar Manager Created in Tenant、Namespace as well as Topic.

   ./bin/pulsar-client consume -s sub kubesphere/kubesphere/kubesphere -n 0

Be careful ： Please make sure that... Is installed before operation Applicable Java JRE/JDK edition .

4. Open another command line window , Execute the following command to produce information .

   ./bin/pulsar-client produce kubesphere/kubesphere/kubesphere -m “Hello KubeSphere and Pulsar” -n 10

5. You can see that the message has been successfully consumed .

adopt Grafana monitor Pulsar indicators

Pulsar Of Helm Chart Will be installed during deployment Grafana Provide with Pulsar Relevant observability indicators , It is convenient for us to conduct subsequent operation and maintenance analysis .

1. And visit Pulsar Manager similar , Go to the application load > service , Entry exposure Grafana Of Service The details page of , And according to which LoadBalancer Type of IP Address plus slogan access Grafana.
2. Through the default account password pulsar/pulsar Sign in . Select one in the upper left corner Dashboard To see Pulsar Related indicators .
3. You can see that the relevant data is successfully displayed in Grafana On .

summary

This article demonstrates the application of KubeSphere Upper Department Apache Pulsar Operation steps of . With the help of Pulsar Nature adapts to the characteristics of cloud native environment , We can use KubeSphere The operation and maintenance friendly operation interface is easy to deploy and manage Pulsar. Interested friends can read KubeSphere and Pulsar For more information .

6 month 25 On the daily line Meetup Notice

6 month 25 Japan ,KubeSphere Community unity Apache Pulsar Community , Jointly organize an online Meetup, A total of four topics , Registration is now open , Welcome to scan the QR code in the poster to sign up ！

This article by the blog one article many sends the platform OpenWrite Release ！