倍受关注的 Cilium Service Mesh 到底怎么玩？- 上手实践

Cilium是一个基于eBPF技术，用于为容器工作负载间提供安全且具备可观测性的网络连接的开源软件。

如果你对Cilium还不太了解，可以参考我之前的两篇文章：

K8S生态周报|Google选择Cilium作为GKE下一代数据面

Cilium上手实践

最近正式发布了，增加OpenTelemetry的支持以及其他一些增强特性。同时，也宣布了CiliumServiceMesh的计划。当前CiliumServiceMesh正处于测试阶段，预期在2022年会合并到版本中。

CiliumServiceMesh也带来了一个全新的模式。

Cilium直接通过eBPF技术实现的ServiceMesh相比我们常规的Istio/Linkerd等方案，最显著的特点就是将Sidecarproxy模型替换成了Kernel模型，如下图：

不再需要每个应用程序旁边都放置一个Sidecar了，直接在每台Node上提供支持。

本篇我带你实际体验下CiliumServiceMesh。

这里我使用KIND作为测试环境，我的内核版本是5.15.8。

关于KIND命令行工具的安装这里就不再赘述了，感兴趣的小伙伴可以参考我之前的文章《使用KIND搭建自己的本地Kubernetes测试环境》。

以下是我创建集群时使用的配置文件：

apiVersion:/v1alpha4kind:Clusternodes:-role:control-plane-role:worker-role:worker-role:workernetworking:disableDefaultCNI:true

创建集群：

➜"kind"✓Ensuringnodeimage(kindest/node:)✓Preparingnodes✓Writingconfiguration✓Startingcontrol-plane️✓InstallingStorageClass✓JoiningworkernodesSetkubectlcontextto"kind-kind"Youcannowuseyourclusterwith:kubectlcluster-info--contextkind-kindNotsurewhattodonext?Checkout

这里我们使用CiliumCLI工具进行Cilium的部署。

➜cilium-meshcurl-L--remote-name-all\{,.sha256sum\}[1/2]:;_curl_--[2/2]:;_curl_--➜➜

在部署Cilium的过程中需要一些镜像，我们可以提前下载后加载到KIND的Node节点中。如果你的网络比较顺畅，那这一步可以跳过。

➜cilium-meshciliumMeshImage=("/cilium/cilium-service-mesh:""/cilium/operator-generic-service-mesh:""/cilium/hubble-relay-service-mesh:")➜cilium-meshforiin${ciliumMeshImage[@]}dodockerpull$ikindloaddocker-image$idone

接下来我们直接使用CiliumCLI完成部署。注意这里的参数。

➜cilium-meshciliuminstall--version-service-mesh:=true--kube-proxy-replacement=probe--agent-image='/cilium/cilium-service-mesh:'--operator-image='/cilium/operator-generic-service-mesh:'--datapath-mode=vxlanAuto-detectedKuberneteskind:kind:sparkles:Running"kind"validationchecks:white_check_mark:Detectedkindversion"0.12.0":information_source:usingCiliumversion"-service-mesh:"Auto-detectedclustername:kind-kindAuto-detectedIPAMmode:kubernetesCustomdatapathmode:vxlanFoundCAinsecretcilium-caGeneratingcertificatesforHubbleCreatingServic:information_source:ManualoverwriteinConfigMap:enable-envoy-config=trueCreatingAgentDaemonSetCreatingOperatorDeployment:hourglass:WaitingforCiliumtobeinstalledandready:white_check_mark:Ciliumwassuccessfullyinstalled!Run'ciliumstatus'toviewinstallationhealth

在安装成功后，可以通过ciliumstatus命令来查看当前Cilium的部署情况。

➜cilium-meshciliumstatus/¯¯\/¯¯\__/¯¯\Cilium:OK\__/¯¯\__/Operator:OK/¯¯\__/¯¯\Hubble:disabled\__/¯¯\__/ClusterMesh:disabled\__/Deploymentcilium-operatorDesired:1,Ready:1/1,Available:1/1DaemonSetciliumDesired:4,Ready:4/4,Available:4/4Containers:ciliumRunning:4cilium-operatorRunning:1ClusterPods:3/3/cilium/cilium-service-mesh::4/cilium/operator-generic-service-mesh::1

Hubble主要是用来提供观测能力的。在启用它之前，需要先加载一个镜像，如果网络畅通可以跳过。

/envoyproxy/envoy:@sha256:e8b37c1d75787dd1e712ff389b0d37337dc8a174a63bed9c34ba73359dc67da7

然后使用CiliumCLI开启Hubble：

➜cilium-meshciliumhubbleenable--relay-image='/cilium/hubble-relay-service-mesh:'--uiFoundCAinsecretcilium-ca:sparkles:PatchingConfigMapcilium-configtoenableHubble:recycle:RestartedCiliumpods:hourglass:WaitingforCiliumtobecomereadybeforedeployingotherHubblecomponent(s)GeneratingcertificatesforRelay:sparkles:/cilium/hubble-relay-service-mesh::sparkles:/cilium/hubble-ui:/cilium/hubble-ui-back::hourglass:WaitingforHubbletobeinstalled/¯¯\/¯¯\__/¯¯\Cilium:OK\__/¯¯\__/Operator:OK/¯¯\__/¯¯\Hubble:OK\__/¯¯\__/ClusterMesh:disabled\__/DaemonSetciliumDesired:4,Ready:4/4,Available:4/4Deploymentcilium-operatorDesired:1,Ready:1/1,Available:1/1Deploymenthubble-relayDesired:1,Ready:1/1,Available:1/1Deploymenthubble-uiDesired:1,Unavailable:1/1Containers:ciliumRunning:4cilium-operatorRunning:1hubble-relayRunning:1hubble-uiRunning:1ClusterPods:5/5/cilium/cilium-service-mesh::4/cilium/operator-generic-service-mesh::1/cilium/hubble-relay-service-mesh::1/cilium/hubble-ui::1/cilium/hubble-ui-back::1/envoyproxy/envoy:@sha256:e8b37c1d75787dd1e712ff389b0d37337dc8a174a63bed9c34ba73359dc67da7:1

这里我们可以给KIND集群中安装MetaLB，以便于我们可以使用LoadBalancer类型的svc资源（Cilium会默认创建一个LoadBalancer类型的svc）。如果不安装MetaLB，那也可以使用NodePort的方式来进行替代。

具体过程就不一一介绍了，直接按下述操作步骤执行即可。

➜cilium-meshkubectlapply-f➜cilium-meshkubectlcreatesecretgeneric-nmetallb-systemmemberlist--from-literal=secretkey="$(opensslrand-base64128)"secret/memberlistcreated➜cilium-meshkubectlapply-f➜cilium-meshdockernetworkinspect-f'{{.}}'kind[{172.18.0.0/16172.18.0.1map[]}{fc00:f853:ccd:e793::/64fc00:f853:ccd:e793::1map[]}]➜➜:v1kind:ConfigMapmetadata:namespace:metallb-systemname:configdata:config:|address-pools:-name:defaultprotocol:layer2addresses:-172.18.255.200-172.18.255.250➜/configcreated

这里我们使用hashicorp/http-echo:0.2.3作为示例程序，它们可以按照启动参数的不同响应不同的内容。

➜cilium-meshdockerpullhashicorp/http-echo:0.2.30.2.3:Pullingfromhashicorp/http-echo86399148984b:PullcompleteDigest:sha256:ba27d460cd1f22a1a4331bdf74f4fccbc025552357e8a3249c40ae216275de96Status:Downloadednewerimageforhashicorp/http-echo:0.2.3/hashicorp/http-echo:0.2.3➜cilium-meshkindloaddocker-imagehashicorp/http-echo:0.2.3Image:"hashicorp/http-echo:0.2.3"withID"sha256:a6838e9a6ff6ab3624720a7bd36152dda540ce3987714398003e14780e61478a"notyetpresentonnode"kind-worker",loadingImage:"hashicorp/http-echo:0.2.3"withID"sha256:a6838e9a6ff6ab3624720a7bd36152dda540ce3987714398003e14780e61478a"notyetpresentonnode"kind-worker2",loadingImage:"hashicorp/http-echo:0.2.3"withID"sha256:a6838e9a6ff6ab3624720a7bd36152dda540ce3987714398003e14780e61478a"notyetpresentonnode"kind-control-plane",loadingImage:"hashicorp/http-echo:0.2.3"withID"sha256:a6838e9a6ff6ab3624720a7bd36152dda540ce3987714398003e14780e61478a"notyetpresentonnode"kind-worker3",loading

本文件中的所有配置文件均可在。

我们使用如下配置进行测试服务的部署：

apiVersion:v1kind:Podmetadata:labels:run:foo-appname:foo-appspec:containers:-image:hashicorp/http-echo:0.2.3args:-"-text=foo"name:foo-appports:-containerPort:5678resources:{}dnsPolicy:ClusterFirstrestartPolicy:Alwaysstatus:{}---apiVersion:v1kind:Servicemetadata:labels:run:foo-appname:foo-appspec:ports:-port:5678protocol:TCPtargetPort:5678selector:run:foo-app---apiVersion:v1kind:Podmetadata:labels:run:bar-appname:bar-appspec:containers:-image:hashicorp/http-echo:0.2.3args:-"-text=bar"name:bar-appports:-containerPort:5678resources:{}dnsPolicy:ClusterFirstrestartPolicy:Always---apiVersion:v1kind:Servicemetadata:labels:run:bar-appname:bar-appspec:ports:-port:5678protocol:TCPtargetPort:5678selector:run:bar-app

新建如下的Ingress资源文件：

apiVersion:/v1kind:Ingressmetadata:name:cilium-ingressnamespace:defaultspec:ingressClassName:ciliumrules:-http:paths:-back:service:name:foo-appport:number:5678path:/foopathType:Prefix-back:service:name:bar-appport:number:5678path:/barpathType:Prefix

创建Ingress资源，然后可以看到产生了一个新的LoadBalancer类型的svc。

➜/cilium-ingresscreated➜cilium-meshkubectlgetsvcNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)/:31643///TCP81m➜cilium-meshkubectlgetingNAMECLASSHOSTSADDRESSPORTSAGEcilium-ingresscilium*172.18.255.200801m

使用curl命令进行测试访问，发现可以按照Ingress资源中的配置得到正确的响应。查看响应头，我们会发现这里的代理实际上还是使用的Envoy来完成的。

➜➜/foofoo➜/barbar➜/barHTTP/1.1200OKContent-Length:4Connection:keep-aliveContent-Type:text/plain;charset=utf-8Date:Sat,18Dec202106:02:56GMTKeep-Alive:timeout=4Proxy-Connection:keep-aliveServer:envoyX-App-Name:http-echoX-App-Version:0.2.3X-Envoy-Upstream-Service-Time:0➜/fooHTTP/1.1200OKContent-Length:4Connection:keep-aliveContent-Type:text/plain;charset=utf-8Date:Sat,18Dec202106:03:01GMTKeep-Alive:timeout=4Proxy-Connection:keep-aliveServer:envoyX-App-Name:http-echoX-App-Version:0.2.3X-Envoy-Upstream-Service-Time:0

在使用上述方式部署CIlium后，它其实还安装了一些CRD资源。其中有一个是CiliumEnvoyConfig用于配置服务之间代理的。

➜cilium-meshkubectlapi-resources|/v2falseCiliumClusterwideNetworkPolicyciliumpointscep,////v2falseCiliumIdentityciliumnetworkpoliciescnp,/v2trueCiliumNetworkPolicyciliumnodescn,/v2falseCiliumNode

可以先进行Hubble的port-forward

➜cilium-meshciliumhubbleport-forward

默认会监听到4245端口上，如果不提前执行此操作就会出现下述内容

EnablingHubbletelescope:warning:UnabletocontactHubbleRelay,disablingHubbletelescopeandflowvalidation:rpcerror:code=Unavailabledesc=connectionerror:desc="transport:Errorwhiledialingdialtcp[::1]:4245:connect:connectionrefused"

如果已经开启Hubble的port-forward，正常情况下会得到如下输出：

➜cilium-meshciliumconnectivitytest--testegress-l7:information_source:Monitoraggregationdetected,willskipsomeflowvalidationsteps:hourglass:[kind-kind]Waitingfordeployments[clientclient2echo-same-node]tobecomeready:hourglass:[kind-kind]Waitingfordeployments[echo-other-node]tobecomeready:hourglass:[kind-kind]WaitingforCiliumpointforpodcilium-test/client-6488dcf5d4-pk6w9toappear:hourglass:[kind-kind]WaitingforCiliumpointforpodcilium-test/client2-5998d566b4-hrhrbtoappear:hourglass:[kind-kind]WaitingforCiliumpointforpodcilium-test/echo-other-node-f4d46f75b-bqpcbtoappear:hourglass:[kind-kind]WaitingforCiliumpointforpodcilium-test/echo-same-node-745bd5c77-zpzdntoappear:hourglass:[kind-kind]WaitingforServicecilium-test/echo-other-nodetobecomeready:hourglass:[kind-kind]WaitingforServicecilium-test/echo-same-nodetobecomeready:hourglass:[kind-kind]:32751(cilium-test/echo-other-node)tobecomeready:hourglass:[kind-kind]:32133(cilium-test/echo-same-node)tobecomeready:hourglass:[kind-kind]:32133(cilium-test/echo-same-node)tobecomeready:hourglass:[kind-kind]:32751(cilium-test/echo-other-node)tobecomeready:hourglass:[kind-kind]:32751(cilium-test/echo-other-node)tobecomeready:hourglass:[kind-kind]:32133(cilium-test/echo-same-node)tobecomeready:hourglass:[kind-kind]:32751(cilium-test/echo-other-node)tobecomeready:hourglass:[kind-kind]:32133(cilium-test/echo-same-node)tobecomeready:information_source:SkippingIPCachecheck:hourglass:[kind-kind]Waitingforpodcilium-test/client-6488dcf5d4-pk6w9toreachdefault/kubernetesservice:hourglass:[kind-kind]Waitingforpodcilium-test/client2-5998d566b4-hrhrbtoreachdefault/kubernetesserviceEnablingHubbletelescope:information_source:HubbleisOK,flows:16380/16380Runningtests[=]SkippingTest[no-policies][=]SkippingTest[allow-all][=]SkippingTest[client-ingress][=]SkippingTest[echo-ingress][=]SkippingTest[client-egress][=]SkippingTest[to-entities-world][=]SkippingTest[to-cidr-1111][=]SkippingTest[echo-ingress-l7][=]Test[client-egress-l7].[=]SkippingTest[dns-only][=]SkippingTest[to-fqdns]:white_check_mark:All1tests(10actions)successful,10testsskipped,0scenariosskipped.

我们也可以同时打开UI看看：

➜cilium-meshciliumhubbleui:information_source:Opening"http://localhost:12000"inyourbrowser

效果图如下：

这个操作实际上会进行如下部署：

➜cilium-meshkubectl-ncilium-testgetallNAMEREADYSTATUSRESTARTSAGEpod/client-6488dcf5d4-pk6w91/1Running066mpod/client2-5998d566b4-hrhrb1/1Running066mpod/echo-other-node-f4d46f75b-bqpcb1/1Running066mpod/echo-same-node-745bd5c77-zpzdn1/1Running066mNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGEservice/:32751/TCP66mservice/:32133//client1/11166/client21/11166/echo-other-node1/11166/echo-same-node1/11166////echo-same-node-745bd5c7711166m

我们也可以看看它的label：

➜cilium-meshkubectlgetpods-ncilium-test--show-labels-owideNAMEREADYSTATUSRESTARTSAGEIPNODENOMINATEDNODEREADINESSGATESLABELSclient-6488dcf5d4-pk6w91/1=client,name=client,pod-template-hash=6488dcf5d4client2-5998d566b4-hrhrb1/1=client,name=client2,other=client,pod-template-hash=5998d566b4echo-other-node-f4d46f75b-bqpcb1/1=echo,name=echo-other-node,pod-template-hash=f4d46f75becho-same-node-745bd5c77-zpzdn1/1=echo,name=echo-same-node,other=echo,pod-template-hash=745bd5c77

这里我们在主机上进行操作下，先拿到client2的Pod名称，然后通过Hubble命令观察所有访问此Pod的流量。

➜cilium-meshexportCLIENT2=client2-5998d566b4-hrhrb➜cilium-meshhubbleobserve--from-podcilium-test/$CLIENT2-fDec1814:07:37.200:cilium-test/client2-5998d566b4-hrhrb:44805kube-system/coredns-78fcd69978-7lbwh:53to-overlayFORWARDED(UDP)Dec1814:07:37.200:cilium-test/client2-5998d566b4-hrhrb:44805-kube-system/coredns-78fcd69978-7lbwh:53to-pointFORWARDED(UDP)Dec1814:07:37.200:cilium-test/client2-5998d566b4-hrhrb:44805kube-system/coredns-78fcd69978-7lbwh:53to-overlayFORWARDED(UDP)Dec1814:07:37.200:cilium-test/client2-5998d566b4-hrhrb:44805-kube-system/coredns-78fcd69978-7lbwh:53to-pointFORWARDED(UDP)Dec1814:07:37.200:cilium-test/client2-5998d566b4-hrhrb:42260-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-pointFORWARDED(TCPFlags:SYN)Dec1814:07:37.201:cilium-test/client2-5998d566b4-hrhrb:42260-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-pointFORWARDED(TCPFlags:ACK)Dec1814:07:37.201:cilium-test/client2-5998d566b4-hrhrb:42260-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-pointFORWARDED(TCPFlags:ACK,PSH)Dec1814:07:37.202:cilium-test/client2-5998d566b4-hrhrb:42260-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-pointFORWARDED(TCPFlags:ACK,FIN)Dec1814:07:37.203:cilium-test/client2-5998d566b4-hrhrb:42260-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-pointFORWARDED(TCPFlags:ACK)Dec1814:07:50.769:cilium-test/client2-5998d566b4-hrhrb:36768kube-system/coredns-78fcd69978-7lbwh:53to-overlayFORWARDED(UDP)Dec1814:07:50.769:cilium-test/client2-5998d566b4-hrhrb:36768kube-system/coredns-78fcd69978-7lbwh:53to-overlayFORWARDED(UDP)Dec1814:07:50.769:cilium-test/client2-5998d566b4-hrhrb:36768-kube-system/coredns-78fcd69978-7lbwh:53to-pointFORWARDED(UDP)Dec1814:07:50.769:cilium-test/client2-5998d566b4-hrhrb:36768-kube-system/coredns-78fcd69978-7lbwh:53to-pointFORWARDED(UDP)Dec1814:07:50.770:cilium-test/client2-5998d566b4-hrhrb:42068cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-overlayFORWARDED(TCPFlags:SYN)Dec1814:07:50.770:cilium-test/client2-5998d566b4-hrhrb:42068-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-pointFORWARDED(TCPFlags:SYN)Dec1814:07:50.770:cilium-test/client2-5998d566b4-hrhrb:42068cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-overlayFORWARDED(TCPFlags:ACK)Dec1814:07:50.770:cilium-test/client2-5998d566b4-hrhrb:42068-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-pointFORWARDED(TCPFlags:ACK)Dec1814:07:50.770:cilium-test/client2-5998d566b4-hrhrb:42068cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-overlayFORWARDED(TCPFlags:ACK,PSH)Dec1814:07:50.770:cilium-test/client2-5998d566b4-hrhrb:42068-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-pointFORWARDED(TCPFlags:ACK,PSH)Dec1814:07:50.771:cilium-test/client2-5998d566b4-hrhrb:42068cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-overlayFORWARDED(TCPFlags:ACK,FIN)Dec1814:07:50.771:cilium-test/client2-5998d566b4-hrhrb:42068-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-pointFORWARDED(TCPFlags:ACK,FIN)Dec1814:07:50.772:cilium-test/client2-5998d566b4-hrhrb:42068cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-overlayFORWARDED(TCPFlags:ACK)Dec1814:07:50.772:cilium-test/client2-5998d566b4-hrhrb:42068-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-pointFORWARDED(TCPFlags:ACK)

以上输出是由于我们执行了下面的操作：

kubectlexec-it-ncilium-test$CLIENT2--curl-vecho-same-node:8080/kubectlexec-it-ncilium-test$CLIENT2--curl-vecho-other-node:8080/

日志中基本上都是to-point或者to-overlay的。

需要先安装networkpolicy，我们可以直接从CiliumCLI的仓库中拿到。

kubectlapply-f

然后重复上面的请求：

Dec1814:33:40.570:cilium-test/client2-5998d566b4-hrhrb:44344-kube-system/coredns-78fcd69978-2ww28:53L3-L4REDIRECTED(UDP)Dec1814:33:40.570:cilium-test/client2-5998d566b4-hrhrb:44344-kube-system/coredns-78fcd69978-2ww28:53to-proxyFORWARDED(UDP)Dec1814:33:40.570:cilium-test/client2-5998d566b4-hrhrb:44344-kube-system/coredns-78fcd69978-2ww28:53to-proxyFORWARDED(UDP)Dec1814:33:40.570:cilium-test/client2-5998d566b4-hrhrb:44344-kube-system/coredns-78fcd69978-2ww28:53dns-requestFORWARDED()Dec1814:33:40.570:cilium-test/client2-5998d566b4-hrhrb:44344-kube-system/coredns-78fcd69978-2ww28:53dns-requestFORWARDED()Dec1814:33:40.571:cilium-test/client2-5998d566b4-hrhrb:42074-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080L3-L4REDIRECTED(TCPFlags:SYN)Dec1814:33:40.571:cilium-test/client2-5998d566b4-hrhrb:42074-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-proxyFORWARDED(TCPFlags:SYN)Dec1814:33:40.571:cilium-test/client2-5998d566b4-hrhrb:42074-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-proxyFORWARDED(TCPFlags:ACK)Dec1814:33:40.571:cilium-test/client2-5998d566b4-hrhrb:42074-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-proxyFORWARDED(TCPFlags:ACK,PSH)Dec1814:33:40.572:cilium-test/client2-5998d566b4-hrhrb:42074-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080http-requestFORWARDED(HTTP/1.1GEThttp://echo-other-node:8080/)Dec1814:33:40.573:cilium-test/client2-5998d566b4-hrhrb:42074-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-proxyFORWARDED(TCPFlags:ACK,FIN)Dec1814:33:40.573:cilium-test/client2-5998d566b4-hrhrb:42074-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-proxyFORWARDED(TCPFlags:ACK)

执行另一个请求：

➜cilium-meshkubectlexec-it-ncilium-test$CLIENT2--curl-vecho-same-node:8080/

也可以看到如下输出，其中有to-proxy的字样。

Dec1814:45:18.857:cilium-test/client2-5998d566b4-hrhrb:58895-kube-system/coredns-78fcd69978-2ww28:53L3-L4REDIRECTED(UDP)Dec1814:45:18.857:cilium-test/client2-5998d566b4-hrhrb:58895-kube-system/coredns-78fcd69978-2ww28:53to-proxyFORWARDED(UDP)Dec1814:45:18.857:cilium-test/client2-5998d566b4-hrhrb:58895-kube-system/coredns-78fcd69978-2ww28:53to-proxyFORWARDED(UDP)Dec1814:45:18.857:cilium-test/client2-5998d566b4-hrhrb:58895-kube-system/coredns-78fcd69978-2ww28:53dns-requestFORWARDED()Dec1814:45:18.857:cilium-test/client2-5998d566b4-hrhrb:58895-kube-system/coredns-78fcd69978-2ww28:53dns-requestFORWARDED()Dec1814:45:18.858:cilium-test/client2-5998d566b4-hrhrb:42266-cilium-test/echo-same-node-745bd5c77-zpzdn:8080L3-L4REDIRECTED(TCPFlags:SYN)Dec1814:45:18.858:cilium-test/client2-5998d566b4-hrhrb:42266-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-proxyFORWARDED(TCPFlags:SYN)Dec1814:45:18.858:cilium-test/client2-5998d566b4-hrhrb:42266-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-proxyFORWARDED(TCPFlags:ACK)Dec1814:45:18.858:cilium-test/client2-5998d566b4-hrhrb:42266-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-proxyFORWARDED(TCPFlags:ACK,PSH)Dec1814:45:18.858:cilium-test/client2-5998d566b4-hrhrb:42266-cilium-test/echo-same-node-745bd5c77-zpzdn:8080http-requestFORWARDED(HTTP/1.1GEThttp://echo-same-node:8080/)Dec1814:45:18.859:cilium-test/client2-5998d566b4-hrhrb:42266-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-proxyFORWARDED(TCPFlags:ACK,FIN)Dec1814:45:18.859:cilium-test/client2-5998d566b4-hrhrb:42266-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-proxyFORWARDED(TCPFlags:ACK)

其实看请求头更加方便：

➜cilium-meshkubectlexec-it-ncilium-test$CLIENT2--curl-Iecho-same-node:8080/HTTP/1.1403Forbiddencontent-length:15content-type:text/plaindate:Sat,18Dec202114:47:39GMTserver:envoy

之前都是如下：

0)GET/HTTP/1.1Host:echo-same-node:8080User-Agent:curl/7.78.0Accept:*/**MarkbundleasnotsupportingmultiuseHTTP/1.1200OKX-Powered-By:ExpressVary:Origin,Accept-EncodingAccess-Control-Allow-Credentials:trueAccept-Ranges:bytesCache-Control:public,max-age=0Last-Modified:Sat,26Oct198508:15:00GMTETag:W/"809-7438674ba0"Content-Type:text/html;charset=UTF-8Content-Length:2057Date:Sat,18Dec202114:07:37GMTConnection:keep-aliveKeep-Alive:timeout=5

以前请求响应是404，现在是403,并得到如下内容

➜cilium-meshkubectlexec-it-ncilium-test$CLIENT2--curl-vecho-same-node:8080/foo*:8080*Connectedtoecho-same-node(10.96.136.252)port8080(0tohostecho-same-nodeleftintact

日志中也都是to-proxy的字样。

Dec1814:50:39.185:cilium-test/client2-5998d566b4-hrhrb:37683-kube-system/coredns-78fcd69978-7lbwh:53L3-L4REDIRECTED(UDP)Dec1814:50:39.185:cilium-test/client2-5998d566b4-hrhrb:37683-kube-system/coredns-78fcd69978-7lbwh:53to-proxyFORWARDED(UDP)Dec1814:50:39.185:cilium-test/client2-5998d566b4-hrhrb:37683-kube-system/coredns-78fcd69978-7lbwh:53to-proxyFORWARDED(UDP)Dec1814:50:39.185:cilium-test/client2-5998d566b4-hrhrb:37683-kube-system/coredns-78fcd69978-7lbwh:53dns-requestFORWARDED()Dec1814:50:39.185:cilium-test/client2-5998d566b4-hrhrb:37683-kube-system/coredns-78fcd69978-7lbwh:53dns-requestFORWARDED()Dec1814:50:39.186:cilium-test/client2-5998d566b4-hrhrb:42274-cilium-test/echo-same-node-745bd5c77-zpzdn:8080L3-L4REDIRECTED(TCPFlags:SYN)Dec1814:50:39.186:cilium-test/client2-5998d566b4-hrhrb:42274-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-proxyFORWARDED(TCPFlags:SYN)Dec1814:50:39.186:cilium-test/client2-5998d566b4-hrhrb:42274-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-proxyFORWARDED(TCPFlags:ACK)Dec1814:50:39.186:cilium-test/client2-5998d566b4-hrhrb:42274-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-proxyFORWARDED(TCPFlags:ACK,PSH)Dec1814:50:39.186:cilium-test/client2-5998d566b4-hrhrb:42274-cilium-test/echo-same-node-745bd5c77-zpzdn:8080http-requestDROPPED(HTTP/1.1GEThttp://echo-same-node:8080/foo)Dec1814:50:39.186:cilium-test/client2-5998d566b4-hrhrb:42274-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-proxyFORWARDED(TCPFlags:ACK,FIN)Dec1814:50:39.187:cilium-test/client2-5998d566b4-hrhrb:42274-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-proxyFORWARDED(TCPFlags:ACK)

我们使用如下内容作为Envoy的配置文件，其中包含rewrite策略。

apiVersion:/v2alpha1kind:CiliumEnvoyConfigmetadata:name:envoy-lb-listenerspec:services:-name:echo-other-nodenamespace:cilium-test-name:echo-same-nodenamespace:cilium-testresources:-"@type":/:envoy-lb-listenerfilter_chains:-filters:-name:_connection_managertyped_config:"@type":/_connection__prefix:envoy-lb-listenerrds:route_config_name:lb_routehttp_filters:-name:"@type":/:lb_routevirtual_hosts:-name:"lb_route"domains:["*"]routes:-match:prefix:"/"route:weighted_clusters:clusters:-name:"cilium-test/echo-same-node"weight:50-name:"cilium-test/echo-other-node"weight:50retry_policy:retry_on:5xxnum_retries:3per_try_timeout:1sregex_rewrite:pattern:google_re2:{}regex:"^/foo.*#34;substitution:"/"-"@type":/:"cilium-test/echo-same-node"connect_timeout:5slb_policy:ROUND_ROBINtype:EDSoutlier_detection:split_external_local_origin_errors:trueconsecutive_local_origin_failure:2-"@type":/:"cilium-test/echo-other-node"connect_timeout:3slb_policy:ROUND_ROBINtype:EDSoutlier_detection:split_external_local_origin_errors:trueconsecutive_local_origin_failure:2

测试请求时，发现可以正确的得到响应了。

➜cilium-meshkubectlexec-it-ncilium-test$CLIENT2--curl-XGET-Iecho-same-node:8080/HTTP/1.1200OKx-powered-by:Expressvary:Origin,Accept-Encodingaccess-control-allow-credentials:trueaccept-ranges:bytescache-control:public,max-age=0last-modified:Sat,26Oct198508:15:00GMTetag:W/"809-7438674ba0"content-type:text/html;charset=UTF-8content-length:2057date:Sat,18Dec202115:00:01GMTx-envoy-upstream-service-time:1server:envoy

并且请求/foo地址时，也可以正确的得到响应了。

➜cilium-meshkubectlexec-it-ncilium-test$CLIENT2--curl-XGET-Iecho-same-node:8080/fooHTTP/1.1200OKx-powered-by:Expressvary:Origin,Accept-Encodingaccess-control-allow-credentials:trueaccept-ranges:bytescache-control:public,max-age=0last-modified:Sat,26Oct198508:15:00GMTetag:W/"809-7438674ba0"content-type:text/html;charset=UTF-8content-length:2057date:Sat,18Dec202115:01:40GMTx-envoy-upstream-service-time:2server:envoy

同时：请求/foo的时候，流量如下:直接转换成功了对/的访问

Dec1815:02:22.541:cilium-test/client2-5998d566b4-hrhrb:38860-kube-system/coredns-78fcd69978-2ww28:53L3-L4REDIRECTED(UDP)Dec1815:02:22.541:cilium-test/client2-5998d566b4-hrhrb:38860-kube-system/coredns-78fcd69978-2ww28:53to-proxyFORWARDED(UDP)Dec1815:02:22.541:cilium-test/client2-5998d566b4-hrhrb:38860-kube-system/coredns-78fcd69978-2ww28:53to-proxyFORWARDED(UDP)Dec1815:02:22.541:cilium-test/client2-5998d566b4-hrhrb:38860-kube-system/coredns-78fcd69978-2ww28:53dns-requestFORWARDED()Dec1815:02:22.541:cilium-test/client2-5998d566b4-hrhrb:38860-kube-system/coredns-78fcd69978-2ww28:53dns-requestFORWARDED()Dec1815:02:22.542:cilium-test/client2-5998d566b4-hrhrb:53062-cilium-test/echo-same-node:8080noneREDIRECTED(TCPFlags:SYN)Dec1815:02:22.542:cilium-test/client2-5998d566b4-hrhrb:53062-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:SYN)Dec1815:02:22.542:cilium-test/client2-5998d566b4-hrhrb:53062-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:ACK)Dec1815:02:22.542:cilium-test/client2-5998d566b4-hrhrb:53062-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:ACK,PSH)Dec1815:02:22.542:cilium-test/client2-5998d566b4-hrhrb:53048-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-proxyFORWARDED(TCPFlags:ACK,PSH)Dec1815:02:22.542:cilium-test/client2-5998d566b4-hrhrb:53048-cilium-test/echo-same-node-745bd5c77-zpzdn:8080http-requestFORWARDED(HTTP/1.1GEThttp://echo-same-node:8080/)Dec1815:02:22.543:cilium-test/client2-5998d566b4-hrhrb:53062-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:ACK,FIN)Dec1815:02:22.544:cilium-test/client2-5998d566b4-hrhrb:53062-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:ACK)

多次请求看日志：

Dec1815:07:20.883:cilium-test/client2-5998d566b4-hrhrb:49656-kube-system/coredns-78fcd69978-2ww28:53L3-L4REDIRECTED(UDP)Dec1815:07:20.883:cilium-test/client2-5998d566b4-hrhrb:49656-kube-system/coredns-78fcd69978-2ww28:53to-proxyFORWARDED(UDP)Dec1815:07:20.883:cilium-test/client2-5998d566b4-hrhrb:49656-kube-system/coredns-78fcd69978-2ww28:53to-proxyFORWARDED(UDP)Dec1815:07:20.883:cilium-test/client2-5998d566b4-hrhrb:49656-kube-system/coredns-78fcd69978-2ww28:53dns-requestFORWARDED()Dec1815:07:20.884:cilium-test/client2-5998d566b4-hrhrb:49656-kube-system/coredns-78fcd69978-2ww28:53dns-requestFORWARDED()Dec1815:07:20.885:cilium-test/client2-5998d566b4-hrhrb:53070-cilium-test/echo-same-node:8080noneREDIRECTED(TCPFlags:SYN)Dec1815:07:20.885:cilium-test/client2-5998d566b4-hrhrb:53070-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:SYN)Dec1815:07:20.885:cilium-test/client2-5998d566b4-hrhrb:53070-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:ACK)Dec1815:07:20.885:cilium-test/client2-5998d566b4-hrhrb:53070-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:ACK,PSH)Dec1815:07:20.885:cilium-test/client2-5998d566b4-hrhrb:53064-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-proxyFORWARDED(TCPFlags:ACK,PSH)Dec1815:07:20.885:cilium-test/client2-5998d566b4-hrhrb:53064-cilium-test/echo-same-node-745bd5c77-zpzdn:8080http-requestFORWARDED(HTTP/1.1GEThttp://echo-same-node:8080/)Dec1815:07:20.886:cilium-test/client2-5998d566b4-hrhrb:53070-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:ACK,FIN)Dec1815:07:20.886:cilium-test/client2-5998d566b4-hrhrb:53070-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:ACK)Dec1815:07:26.086:cilium-test/client2-5998d566b4-hrhrb:53048-cilium-test/echo-same-node-745bd5c77-zpzdn:8080to-proxyFORWARDED(TCPFlags:ACK)Dec1815:07:44.739:cilium-test/client2-5998d566b4-hrhrb:39057-kube-system/coredns-78fcd69978-7lbwh:53L3-L4REDIRECTED(UDP)Dec1815:07:44.739:cilium-test/client2-5998d566b4-hrhrb:39057-kube-system/coredns-78fcd69978-7lbwh:53to-proxyFORWARDED(UDP)Dec1815:07:44.740:cilium-test/client2-5998d566b4-hrhrb:39057-kube-system/coredns-78fcd69978-7lbwh:53to-proxyFORWARDED(UDP)Dec1815:07:44.740:cilium-test/client2-5998d566b4-hrhrb:39057-kube-system/coredns-78fcd69978-7lbwh:53dns-requestFORWARDED()Dec1815:07:44.740:cilium-test/client2-5998d566b4-hrhrb:39057-kube-system/coredns-78fcd69978-7lbwh:53dns-requestFORWARDED()Dec1815:07:44.741:cilium-test/client2-5998d566b4-hrhrb:53072-cilium-test/echo-same-node:8080noneREDIRECTED(TCPFlags:SYN)Dec1815:07:44.741:cilium-test/client2-5998d566b4-hrhrb:53072-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:SYN)Dec1815:07:44.741:cilium-test/client2-5998d566b4-hrhrb:53072-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:ACK)Dec1815:07:44.741:cilium-test/client2-5998d566b4-hrhrb:53072-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:ACK,PSH)Dec1815:07:44.742:cilium-test/client2-5998d566b4-hrhrb:53068-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080to-proxyFORWARDED(TCPFlags:ACK,PSH)Dec1815:07:44.742:cilium-test/client2-5998d566b4-hrhrb:53068-cilium-test/echo-other-node-f4d46f75b-bqpcb:8080http-requestFORWARDED(HTTP/1.1GEThttp://echo-same-node:8080/)Dec1815:07:44.744:cilium-test/client2-5998d566b4-hrhrb:53072-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:ACK,FIN)Dec1815:07:44.744:cilium-test/client2-5998d566b4-hrhrb:53072-cilium-test/echo-same-node:8080to-proxyFORWARDED(TCPFlags:ACK)

可以看到它真的成功的进行了负载均衡。

本文我带你部署了CiliumServiceMesh，并通过两个示例，带你体验了CiliumServiceMesh的工作情况。

整体而言，这种方式能带来一定的便利性，但它的服务间流量配置主要依靠于CiliumEnvoyConfig，不算太方便。