来，在Kubernetes中部署高可用Harbor镜像仓库

系统环境：

kubernetes版本：1.18.10

HarborChart版本：1.5.2

Harbor版本：2.1.2

Helm版本：3.3.4

持久化存储驱动：CephRBD

简介

Harbor是一个开放源代码容器镜像注册表，可通过基于角色权限的访问控制来管理镜像，还能扫描镜像中的漏洞并将映像签名为受信任。Harbor是CNCF孵化项目，可提供合规性，性能和互操作性，以帮助跨Kubernetes和Docker等云原生计算平台持续，安全地管理镜像。

特性

管理：多租户、可扩展

安全：安全和漏洞分析、内容签名与验证

安装Harbor我们会默认使用HTTPS协议，需要TLS证书，如果我们没用自己设定自定义证书文件，那么Harbor将自动创建证书文件，不过这个有效期只有一年时间，所以这里我们生成自签名证书，为了避免频繁修改证书，将证书有效期为100年，操作如下：

安装cfssl

fssl是CloudFlare开源的一款PKI/TLS工具,cfssl包含一个命令行工具和一个用于签名，验证并且捆绑TLS证书的HTTPAPI服务,使用Go语言编写.

github:/cloudflare/…

下载地址:/

macOS安装步骤：

→brewinstallcfssl

通用安装方式：

→wget→wget→wget→chmod+x/usr/local/bin/cfssl*

获取默认配置

→→

生成CA证书

将内容修改为：

{"signing":{"default":{"expiry":"876000h"},"profiles":{"harbor":{"expiry":"876000h","usages":["signing","keyencipherment","serverauth"]}}}}

修改文件内容为：

{"CN":"CA","key":{"algo":"rsa","size":2048},"names":[{"C":"CN","ST":"hangzhou","L":"hangzhou","O":"harbor","OU":"System"}]}

修改好配置文件后,接下来就可以生成CA证书了：

→|cfssljson-bareca2020/12/3000:45:55[INFO]generatinganewCAkeyandcertificatefromCSR2020/12/3000:45:55[INFO]generatereceivedrequest2020/12/3000:45:55[INFO]receivedCSR2020/12/3000:45:55[INFO]generatingkey:rsa-20482020/12/3000:45:56[INFO]encodedCSR2020/12/3000:45:56[INFO]signedcertificatewithserialnumber529798847867094212963042958391637272775966762165

此时目录下会出现三个文件：

→tree├──这也是刚才申请证书的json├──├──

这样我们就生成了:

根证书文件:

根证书私钥:

根证书申请文件:(csr是不是clientsslrequest?)

签发证书

创建,内容为：

{"CN":"harbor","hosts":["","*."],"key":{"algo":"rsa","size":2048},"names":[{"C":"US","ST":"CA","L":"SanFrancisco","O":"harbor","OU":"System"}]}

使用之前的CA证书签发harbor证书：

→cfsslgencert-ca====|cfssljson-bareharbor2020/12/3000:50:31[INFO]generatereceivedrequest2020/12/3000:50:31[INFO]receivedCSR2020/12/3000:50:31[INFO]generatingkey:rsa-20482020/12/3000:50:31[INFO]encodedCSR2020/12/3000:50:31[INFO]signedcertificatewithserialnumber37264109865546268794440162

此时目录下会多几个文件：

→tree-L1├──├──├──├──

至此，harbor的证书生成完成。

生成Secret资源

创建Kubernetes的Secret资源，且将证书文件导入：

-n：指定创建资源的Namespace

--from-file：指定要导入的文件地址

→kubectlcreatensharbor→kubectl-nharborcreatesecretgenericharbor-tls--from-file======

查看是否创建成功：

→kubectl-nharborgetsecretharbor-tlsNAMETYPEDATAAGEharbor-tlsOpaque31m

创建radosgw

如果你是通过ceph-deploy部署的，可以通过以下步骤创建radosgw：

先安装radosgw：

→

然后创建radosgw：

→

如果你是通过cephadm部署的，可以通过以下步骤创建radosgw：

cephadm将radosgw部署为管理特定领域和区域的守护程序的集合。例如，要在172.16.7.1上部署1个服务于mytest领域和myzone区域的rgw守护程序：

接下来创建一个新的区域组：→radosgw-adminzonegroupcreate--rgw-zonegroup=myzg--master--default为特定领域和区域部署一组radosgw守护程序：→cephorchapplyrgwmytestmyzone--placement="1172.16.7.1"

查看服务状态：

→cephorchls|/15mago7wcount:1/ceph/ceph:v154405f6339e35

测试服务是否正常：

→curl-s

正常返回如下数据：

?xmlversion="1.0"encoding="UTF-8"?ListAllMyBucketsResultxmlns=""OwnerIDanonymous/IDDisplayName/DisplayName/OwnerBuckets/Buckets/ListAllMyBucketsResult

查看zonegroup：

→radosgw-adminzonegroupget{"id":"ed34ba6e-7089-4b7f-91c4-82fc856fc16c","name":"myzg","api_name":"myzg","is_master":"true","points":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"650e7cca-aacb-4610-a589-acd605d53d23","zones":[{"id":"650e7cca-aacb-4610-a589-acd605d53d23","name":"myzone","points":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e63c234c-e069-4a0d-866d-1ebdc69ec5fe","sync_policy":{"groups":[]}}

→'allowrwx'mon'allowrwx'-o/etc/ceph/

分发/etc/ceph/到其它radosgw节点。

创建对象存储用户和访问凭证

Createaradosgwuserfors3access→radosgw-adminusercreate--uid="harbor"--display-name="HarborRegistry"

Createaswiftuser→adosgw-adminsubusercreate--uid=harbor--subuser=harbor:swift--access=full

CreateSecretKey→radosgw-adminkeycreate--subuser=harbor:swift--key-type=swift--gen-secret记住keys字段中的access_keysecret_key

创建存储桶（bucket）

首先需要安装awscli：

→pip3installawscli-i

查看秘钥：

→radosgw-adminuserinfo--uid="harbor"|[{"user":"harbor","access_key":"VGZQY32LMFQOQPVNTDSJ","secret_key":"YZMMYqoy1ypHaqGOUfwLvdAj9A731iDYDjYqwkU5"}]

配置awscli：

→awsconfigure--profile=cephAWSAccessKeyID[None]:VGZQY32LMFQOQPVNTDSJAWSSecretAccessKey[None]:YZMMYqoy1ypHaqGOUfwLvdAj9A731iDYDjYqwkU5Defaultregionname[None]:Defaultoutputformat[None]:json

配置完成后，凭证将会存储到~/.aws/credentials：

→cat~/.aws/credentials[ceph]aws_access_key_id=VGZQY32LMFQOQPVNTDSJaws_secret_access_key=YZMMYqoy1ypHaqGOUfwLvdAj9A731iDYDjYqwkU5

配置将会存储到~/.aws/config：

→cat~/.aws/config[profileceph]region=cn-hangzhou-1output=json

创建存储桶（bucket）：

→aws--profile=ceph--point=

查看存储桶（bucket）列表：

→radosgw-adminbucketlist["harbor"]

查看存储桶状态：

→radosgw-adminbucketstats[{"bucket":"harbor","num_shards":11,"tenant":"","zonegroup":"ed34ba6e-7089-4b7f-91c4-82fc856fc16c","placement_rule":"default-placement","explicit_placement":{"data_pool":"","data_extra_pool":"","index_pool":""},"id":"650","marker":"650","index_type":"Normal","owner":"harbor","ver":"01,21,41,61,81,100,10,30,50,70,90","mtime":"2020-12-29T17:19:02.481567Z","creation_time":"2020-12-29T17:18:58.940915Z","max_marker":"0,2,4,6,8,10入口配置，我只在内网使用，所以直接使用cluserIPexpose:type:clusterIPtls:Thecommonnameusedtogeneratethecertificate,it'snecessaryThenameofsecretwhichcontainskeysnamed:""-theprivatekeysecretName:"harbor-tls"""-thecertificateOnlyneededwhenthe""is"ingress".notarySecretName:""如果Harbor部署在代理后，将其设置为代理的URLexternalURL:，并将storageClass设置为集群默认的storageClasspersistence:enabled:nts,)resourcePolicy:"keep"persistentVolumeClaim:registry:andspecifythe"subPath"ifthePVCissharedwithothercomponentsexistingClaim:""StorageClasswillbeused(thedefault).Ifexternaldatabaseisused,thefollowingsettingsfordatabasewillIfexternalRedisisused,thefollowingsettingsforRediswill默认用户名admin的密码配置，注意：密码中一定要包含大小写字母与数字harborAdminPassword:"Mydlq123456"各个组件CPUMemory资源相关配置nginx:resources:requests:memory:256Micpu:500mportal:resources:requests:memory:256Micpu:500mcore:resources:requests:memory:256Micpu:1000mjobservice:resources:requests:memory:256Micpu:500mregistry:registry:resources:requests:memory:256Micpu:500mcontroller:resources:requests:memory:256Micpu:500mclair:clair:resources:requests:memory:256Micpu:500madapter:resources:requests:memory:256Micpu:500mnotary:server:resources:requests:memory:256Micpu:500msigner:resources:requests:memory:256Micpu:500mdatabase:internal:resources:requests:memory:256Micpu:500mredis:internal:resources:requests:memory:256Micpu:500mtrivy:enabled:trueresources:requests:cpu:200mmemory:512Milimits:cpu:1000mmemory:1024MiSpecifywhethertodisable`redirect`forimagesandchartstorage,,simplyset`disableredirect`to`true`instead."caBundleSecretName"'sandchartmuseum':"filesystem","azure","gcs","s3","swift",mustbe"filesystem"ifyouwanttousepersistentvolumesforregistryrootdirectory:/s3/object/name/prefixmultipartcopychunksize:"33554432"multipartcopythresholdsize:"33554432"

添加Helm仓库

→helmrepoaddharbor

部署Harbor

→helminstallharborharbor/

查看应用是否部署完成

→kubectl-nharborgetpodNAMEREADYSTATUSRESTARTSAGEharbor-harbor-chartmuseum-55fb975fbd-74vnh1/1Running03mharbor-harbor-clair-695c7f9c69-7gpkh2/2Running03mharbor-harbor-core-687cfb49b6-zmwxr1/1Running03mharbor-harbor-database-01/1Running03mharbor-harbor-jobservice-88994b9b7-684vb1/1Running03mharbor-harbor-nginx-6758559548-x9pq61/1Running03mharbor-harbor-notary-server-6d55b785f-6jsq91/1Running03mharbor-harbor-notary-signer-9696cbdd8-8tfw91/1Running03mharbor-harbor-portal-6f474574c4-8jzh21/1Running03mharbor-harbor-redis-01/1Running03mharbor-harbor-registry-5b6cbfb4cf-42fm92/2Running03mharbor-harbor-trivy-01/1Running03m

Host配置域名

接下来配置Hosts，客户端想通过域名访问服务，必须要进行DNS解析，由于这里没有DNS服务器进行域名解析，所以修改hosts文件将Harbor指定clusterIP和自定义host绑定。首先查看nginx的clusterIP：

→kubectl-nharborgetsvcharbor-harbor-nginxNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)/TCP,443/TCP22h

打开主机的Hosts配置文件，往其加入下面配置：

10.109.50.142

如果想在集群外访问，建议将Servicenginx的type改为nodePort或者通过ingress来代理。当然，如果你在集群外能够直接访问clusterIP，那更好。

输入地址。

用户：admin

密码：Mydlq123456(在安装配置中自定义的密码)

进入后可以看到Harbor的管理后台：

对于Containerd来说，不能像docker一样dockerlogin登录到镜像仓库，需要修改其配置文件来进行认证。/etc/containerd/需要添加如下内容：

[plugins."".registry][plugins."".][plugins."".][plugins.""."".tls]insecure_skip_verify=true[plugins.""."".auth]username="admin"password="Mydlq123456"

由于Harbor是基于Https的，理论上需要提前配置tls证书，但可以通过insecure_skip_verify选项跳过证书认证。

当然，如果你想通过Kubernetes的secret来进行用户验证，配置还可以精简下：

[plugins."".registry][plugins."".][plugins."".][plugins.""."".tls]insecure_skip_verify=true

Kubernetes集群使用docker-registry类型的Secret来通过镜像仓库的身份验证，进而拉取私有映像。所以需要创建Secret，命名为regcred：

→kubectlcreatesecretdocker-registryregcred\--docker-server=你的镜像仓库服务器\--docker-username=你的用户名\--docker-password=你的密码\--docker-email=你的邮箱地址

然后就可以在Pod中使用该secret来访问私有镜像仓库了，下面是一个示例Pod配置文件：

apiVersion:v1kind:Podmetadata:name:private-regspec:containers:-name:private-reg-containerimage:your-private-imageimagePullSecrets:-name:regcred

如果你不嫌麻烦，想更安全一点，那就老老实实将CA、证书和秘钥拷贝到所有节点的/etc/ssl/certs/目录下。/etc/containerd/需要添加的内容更多一点：

[plugins."".registry][plugins."".][plugins."".][plugins.""."".tls]ca_file="/etc/ssl/certs/"cert_file="/etc/ssl/certs/"key_file="/etc/ssl/certs/"

至于Docker的配置方式，大家可以自己去搜一下，这里就跳过了，谁让它现在不受待见呢。

这里为了测试推送镜像，先下载一个用于测试的helloworld小镜像，然后推送到仓库：

将下载的镜像使用tag命令改变镜像名→/library/hello-world:/library/hello-world:/library/hello-world:latest删除之前镜像→/library/hello-world:latest→/library/hello-world:latest#测试从下载新镜像→/library/hello-world:/library/hello-world:latest:resolved|++++++++++++++++++++++++++++++++++++++|manifest-sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042:done|++++++++++++++++++++++++++++++++++++++|layer-sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689:done|++++++++++++++++++++++++++++++++++++++|config-sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b:done|++++++++++++++++++++++++++++++++++++++|elapsed:0.6stotal:525.0(874.0B/s)unpackinglinux/amd64sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042done