centos7的系统服务管理Harbor企业级镜像管理系统实战

导读0.安装harbor0.1下载安装包github下载巨慢链接：HTTPS:///s/1_l_FaqkKleqoiR3FAi2p5A提取码：7b1r--来自百....

0.安装harbor0.1 下载安装包

github下载巨慢

链接：HTTPS:///s/1_l_FaqkKleqoiR3FAi2p5A 提取码：7b1r --来自百度网盘超级会员V4的分享

0.2安装

解压下载的压缩包修改配置并执行install

tar -zxvf harbor-offline-installer-v1.10.10.tgz harbor]# ll-rw-r--r-- 1 root root 612306524 Jan 12 12:09 harbor.v1.10.10.tar.gz-rw-r--r-- 1 root root 5895 Apr 22 10:02 harbor.yml-rwxr-xr-x 1 root root 2284 Jan 12 12:08 install.sh-rw-r--r-- 1 root root 11347 Jan 12 12:08 LICENSE-rwxr-xr-x 1 root root 1750 Jan 12 12:08 prepare#修改配置修改hostname 和port 以及数据存储目录hostname: 10.50.10.185http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 8199# The default data volumedata_volume: /opt/harbor/data [root@p1edaspk02 harbor]# sh ./install.sh[Step 0]: checking if Docker is installed ...Note: docker version: 18.06.3[Step 1]: checking docker-compose is installed ...Note: docker-compose version: 1.29.1[Step 2]: loading Harbor images ...Loaded image: goharbor/harbor-portal:v1.10.10...[Step 3]: preparing environment ...[Step 4]: preparing harbor configs ...prepare base dir is set to /opt/harbor/usr/src/app/utils/configs.py:100: YAMLLoadWARNING: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. configs = yaml.load(f)....[Step 5]: starting Harbor ...Creating nginx ... done✔ ----Harbor has been installed and started successfully.----

0.3启动和关闭

# 启动harbordocker-compose up -d# 关闭harbordocker-compose down

要配置HTTPS，必须创建SSL证书。您可以使用由受信任的第三方CA签名的证书，也可以使用自签名证书

默认情况下，Harbor不附带证书。可以在没有安全性的情况下部署Harbor，以便您可以通过HTTP连接到它。但是，只有在没有外部网络连接的空白测试或开发环境中，才可以使用HTTP。在没有空隙的环境中使用HTTP会使您遭受中间人攻击。在生产环境中，请始终使用HTTPS。如果启用Content Trust with Notary来正确签名所有图像，则必须使用HTTPS。

1. 生成证书颁发机构证书

在生产环境中，您应该从CA获得证书。在测试或开发环境中，您可以生成自己的CA。要生成CA证书，请运行以下命令。

1.1 生成CA证书私钥。

openssl genrsa -out ca.key 4096

1.2 生成CA证书

调整-subj选项中的值以反映您的组织。如果使用FQDN连接Harbor主机，则必须将其指定为通用名称（CN）属性。

openssl req -x509 -new -nodes -sha512 -days 3650 \ -subj "/C=CN/ST=XianYang/L=XianYang/O=example/OU=Personal/CN=10.50.10.185" \ -key ca.key \ -out ca.crt

如果是ip访问， 将 10.50.10.185 改成 ip地址

2. 生成服务器证书

证书通常包含一个.crt文件和一个.key文件

2.1 生成私钥

Copyopenssl genrsa -out 10.50.10.185.key 4096

2.2 生成证书签名请求（CSR）

openssl req -sha512 -new \ -subj "/C=CN/ST=XianYang/L=XianYang/O=example/OU=Personal/CN=10.50.10.185" \ -key 10.50.10.185.key \ -out 10.50.10.185.csr

如果是ip访问， 将 10.50.10.185 改成 ip地址

2.3 生成一个x509 v3扩展文件

无论您使用FQDN还是IP地址连接到Harbor主机，都必须创建此文件，以便可以为您的Harbor主机生成符合主题备用名称（SAN）和x509 v3的证书扩展要求。替换DNS条目以反映您的域

Copycat > v3.ext <<-EOFauthorityKeyIdentifier=keyid,issuerbasicConstraints=CA:FALSEkeyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEnciphermentextendedKeyUsage = serverAuthsubjectAltName = @alt_names[alt_names]DNS.1=10.50.10.185DNS.2=10.50.10.185DNS.3=10.50.10.185EOF

• 如果是ip访问

cat > v3.ext <<-EOFauthorityKeyIdentifier=keyid,issuerbasicConstraints=CA:FALSEkeyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEnciphermentextendedKeyUsage = serverAuthsubjectAltName = IP:10.50.10.185EOF

2.4 使用该v3.ext文件为您的Harbor主机生成证书

openssl x509 -req -sha512 -days 3650 \ -extfile v3.ext \ -CA ca.crt -CAkey ca.key -CAcreateserial \ -in 10.50.10.185.csr \ -out 10.50.10.185.crt

如果是ip访问， 将 10.50.10.185 改成 ip地址

3. 提供证书给Harbor和Docker

生成后ca.crt，10.50.10.185.crt和10.50.10.185.key文件，必须将它们提供给Harbor和docker，重新配置它们

3.1 将服务器证书和密钥复制到Harbor主机上的/data/cert/文件夹中

mkdir -p /data/cert/cp 10.50.10.185.crt /data/cert/cp 10.50.10.185.key /data/cert/

3.2 转换10.50.10.185.crt为10.50.10.185.cert，供Docker使用

Docker守护程序将.crt文件解释为CA证书，并将.cert文件解释为客户端证书

openssl x509 -inform PEM -in 10.50.10.185.crt -out 10.50.10.185.cert

3.3 将服务器证书，密钥和CA文件复制到Harbor主机上的Docker证书文件夹中。您必须首先创建适当的文件夹

mkdir -p /etc/docker/certs.d/10.50.10.185/cp 10.50.10.185.cert /etc/docker/certs.d/10.50.10.185/cp 10.50.10.185.key /etc/docker/certs.d/10.50.10.185/cp ca.crt /etc/docker/certs.d/10.50.10.185/harbor]# tree /etc/docker/certs.d/10.50.10.185//etc/docker/certs.d/10.50.10.185/├── 10.50.10.185.cert├── 10.50.10.185.key└── ca.crt

如果将默认nginx端口443 映射到其他端口，请创建文件夹/etc/docker/certs.d/yourdomain.com:port或/etc/docker/certs.d/harbor_IP:port

例如https的port为8443

mkdir -p /etc/docker/certs.d/10.50.10.185:8843cp 10.50.10.185.cert 10.50.10.185.key ca.crt /etc/docker/certs.d/10.50.10.185:8843

3.4 重新启动Docker Engine

systemctl restart docker

3.5 证书的目录结构

ca]# tree /etc/docker/certs.d//etc/docker/certs.d/└── 10.50.10.185 ├── 10.50.10.185.cert ├── 10.50.10.185.key └── ca.crt

4. 部署或重新配置Harbor

harbor.ymlhostname: 10.50.10.185http: port: 8199https: port: 443 certificate: /opt/harbor/ca/10.50.10.185.crt private_key: /opt/harbor/ca/10.50.10.185.keyexternal_url: https://10.50.10.185harbor_admin_password: Harbor12345database: password: root123 max_idle_conns: 50 max_open_conns: 100data_volume: /data/harborclair: updaters_interval: 12jobservice: max_job_workers: 10notification: webhook_job_max_retry: 10chart: absolute_url: disabledlog: level: info local: rotate_count: 50 rotate_size: 200M location: /data/harbor/logs_version: 1.10.0proxy: http_proxy: https_proxy: no_proxy: components: - core - jobservice - clair

4.1 运行prepare脚本以启用HTTPS

Harbor将nginx实例用作所有服务的反向代理。您可以使用prepare脚本来配置nginx为使用HTTPS

./prepare

4.2 如果Harbor正在运行，请停止并删除现有实例

您的images数据保留在文件系统中，因此不会丢失任何数据

harbor]# docker-compose down -vStopping harbor-jobservice ... doneStopping nginx ... doneStopping harbor-core ... doneStopping registryctl ... doneStopping harbor-db ... doneStopping redis ... doneStopping registry ... doneStopping harbor-portal ... doneStopping harbor-log ... doneRemoving harbor-jobservice ... doneRemoving nginx ... doneRemoving harbor-core ... doneRemoving registryctl ... doneRemoving harbor-db ... doneRemoving redis ... doneRemoving registry ... doneRemoving harbor-portal ... doneRemoving harbor-log ... doneRemoving network harbor_harbor

4.3 重启Harbor

harbor]# docker-compose up -dCreating network "harbor_harbor" with the default driverCreating harbor-log ... doneCreating harbor-db ... doneCreating registryctl ... doneCreating redis ... doneCreating harbor-portal ... doneCreating registry ... doneCreating harbor-core ... doneCreating harbor-jobservice ... doneCreating nginx ... done

找到Harbor服务的服务目录

~]# find / -name harbor.yml -type f/opt/harbor/harbor.yml

查看Harbor服务状态

2 ~]# cd /opt/harbor/You have mail in /var/spool/mail/root[root@p1edaspk02 harbor]# docker-compose ps Name Command State Ports-----------------------------------------------------------------------------------------------------------------harbor-core /harbor/harbor_core Up (healthy)harbor-db /docker-entrypoint.sh Up (healthy) 5432/tcpharbor-jobservice /harbor/harbor_jobservice ... Up (healthy)harbor-log /bin/sh -c /usr/local/bin/ ... Up (healthy) 127.0.0.1:1514->10514/tcpharbor-portal nginx -g daemon off; Up (healthy) 8080/tcpnginx nginx -g daemon off; Up (healthy) 0.0.0.0:8199->8080/tcp, 0.0.0.0:443->8443/tcpredis redis-server /etc/redis.conf Up (healthy) 6379/tcpregistry /home/harbor/entrypoint.sh Up (healthy) 5000/tcpregistryctl /home/harbor/start.sh Up (healthy)

5. 验证HTTPS连接

打开浏览器，然后输入https://10.50.10.185。它应该显示Harbor界面

6. 推送以及拉去镜像6.1 登录harbor仓库

# docker login harbor域名地址:端口号harbor]# docker login https://10.50.10.185 -u adminPassword:WARNING! Your password will be stored unencrypted in /root/.docker/config.json.Configure a credential helper to remove this warning. Seehttps://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded

如上所示为登录成功

如果登录报错:

harbor]# docker login 10.50.10.185:8199 -u adminPassword:Error response from daemon: Get https://10.50.10.185:8199/v2/: http: server gave HTTP response to HTTPS client

6.2 在harbor dashboar创建项目

项目 -> 新建项目 ->

在项目中标记镜像：docker tag SOURCE_IMAGE[:TAG] 10.50.10.185/harbortest/IMAGE[:TAG]docker tag registry.aliyuncs.com/openspug/spug:latest 10.50.10.185/harbortest/registry.aliyuncs.com/openspug/spug:latest```在项目中标记镜像：docker tag SOURCE_IMAGE[:TAG] 10.50.10.185/harbortest/IMAGE[:TAG]推送镜像到当前项目：docker push 10.50.10.185/harbortest/IMAGE[:TAG]​```

例子：

harbor]# docker tag registry.aliyuncs.com/openspug/spug:latest 10.50.10.185/harbortest/registry.aliyuncs.com/openspug/spug:latest[root@p1edaspk02 harbor]#[root@p1edaspk02 harbor]#[root@p1edaspk02 harbor]# docker login https://10.50.10.185 -u adminPassword:WARNING! Your password will be stored unencrypted in /root/.docker/config.json.Configure a credential helper to remove this warning. Seehttps://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded[root@p1edaspk02 harbor]#[root@p1edaspk02 harbor]#[root@p1edaspk02 harbor]# docker push 10.50.10.185/harbortest/registry.aliyuncs.com/openspug/spug:latestThe push refers to repository [10.50.10.185/harbortest/registry.aliyuncs.com/openspug/spug]7f7d97906ce8: Pushed260a2403f5c7: Pushed26b6e6155c9a: Pushed8e1aef93890d: Pushedb220652480d3: Pushedd30f3e7469cb: Pushed8d395243207e: Pushed4b4158158262: Pushed5fcede3d79f6: Pushed72bd99349a58: Pushed27e935fbee66: Pushed737c272b1ba6: Pushedfb82b029bea0: Pushedlatest: digest: sha256:8137ad64f0e6ae455171fd4c45a4c0ca42d069262d66f15f66a487f357312350 size: 3032

注意

然后登陆推送镜像测试， 如果服务器要推送代码到harbor， 必须在docker的配置文件的目录 /etc/docker/certs.d/10.50.10.185/ 配置 服务器证书（10.50.10.185.cert），密钥（10.50.10.185.key）和CA文件（ca.crt）

7. harbor 配合kubernetes使用7.1 k8s拉取镜像的方式

Always：当容器失效时，由Kubelet自动重启该容器。RestartPolicy的默认值。OnFailure：当容器终止运行且退出码不为0时由Kubelet重启。Never：无论何种情况下，Kubelet都不会重启该容器。注意，这里的重启是指在 Pod 所在 Node 上面本地重启，并不会调度到其他 Node 上去。

7.2使用私有镜像仓库拉去镜像7.2.1 k8s-node节点添加验证

将harbor服务器上如下三个文件分发到kubernetes集群的node节点/etc/docker/certs.d/10.50.10.185/的这目录

1. 服务器证书（10.50.10.185.cert）
2. 密钥（10.50.10.185.key）
3. CA文件（ca.crt）

7.2.2 拉去镜像

在node节点执行

docker pull 10.50.10.185/harbortest/nginx:latest

如果可以成功拉取代表node节点的证书已生效

7.2.3 创建一个 docker registry secret

使用私有仓库

kubectl create secret docker-registry regsecret --docker-server=https://10.50.10.185 --docker-username=admin --docker-password=Harb2323 --docker-email=ninesun@126.com

使用 Azure Container Registry（ACR）： https://kubernetes.feisky.xyz/concepts/objects/pod

ACR_NAME=dregistrySERVICE_PRINCIPAL_NAME=acr-service-principal# Populate the ACR login server and resource id.ACR_LOGIN_SERVER=$(az acr show --name $ACR_NAME --query loginServer --output tsv)ACR_REGISTRY_ID=$(az acr show --name $ACR_NAME --query id --output tsv)# Create a contributor role assignment with a scope of the ACR resource.SP_PASSWD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME --role Reader --scopes $ACR_REGISTRY_ID --query password --output tsv)# Get the service principle client id.CLIENT_ID=$(az ad sp show --id http://$SERVICE_PRINCIPAL_NAME --query appId --output tsv)# Create secretkubectl create secret docker-registry acr-auth --docker-server $ACR_LOGIN_SERVER --docker-username $CLIENT_ID --docker-password $SP_PASSWD --docker-email local@local.domain

7.2.4 引用docker registry secret的两种方式直接在pod中引用

apiVersion: v1kind: Podmetadata: name: harbor-testspec: containers: - name: harbor-test image: 10.50.10.185/harbortest/nginx:latest imagePullSecrets: - name: regsecret

secret 添加到 service account 中 s并通过ervice account 引用

显然如果将secret添加到sa中，就又多了一层抽象，不用在每个pod或者deployment中的每个container都写一遍imagePullSecrets。而且向用户屏蔽了细节。用户不需要关心

kubectl patch serviceaccount default -p {"imagePullSecrets": [{"name": "myregistrykey"}]}

kubectl patch serviceaccount default -p {"imagePullSecrets": [{"name": "regsecret"}]} /opt/k8s]#kubectl get sa -oyamlapiVersion: v1items:- apiVersion: v1 imagePullSecrets: - name: regsecret kind: ServiceAccount metadata: creationTimestamp: "2022-03-18T12:31:44Z" name: default namespace: default resourceVersion: "4202955" uid: a9b88295-630e-4121-94e1-ab53a17f4f49 secrets: - name: default-token-qvnrckind: Listmetadata: resourceVersion: "" selfLink: ""

实战

使用deployment部署nginx

apiVersion: apps/v1kind: Deploymentmetadata: name: harbor-test labels: app: nginxspec: replicas: 10 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx #image: nginx:latestimage: 10.50.10.185/harbortest/nginx:latest ports: - containerPort: 80

<!--== 注意 ==-->

image字段一定要写harbor全路径，否则将使用默认的源拉去镜像

如何查找harbor中的镜像

# vim Harbor-image-list-100.sh #!/bin/bash#镜像清单文件，将获取到的镜像信息存到该文件中File=harbor-images-`date %Y-%m-%d`.txt## 定义Harbor连接地址,这里需要改为你们自己的Harbor地址Address=https://10.50.10.185## 定义连接Harbor的用户名和密码（因为是获取全部的镜像，只有admin用户才有该权限）Hamin=admin:Harbor12345## 获取Harbor中有哪些项目（Project）Project_List=$(curl -u "$Hamin" -X GET $Address/api/projects -H "Content-Type: application/json" | grep name | awk /"name": / | awk -F " {print $4})for Project in $Project_List;do # 循环获取每个项目下所有的镜像 Image_Names=$(curl -u "$Hamin" -X GET $Address/api/search?q=$Project -H "Content-Type: application/json" | grep "repository_name" | awk -F "\"" {print $4}) for Image in $Image_Names;do # 循环获取每个镜像所有的标签（版本） Image_Tags=$(curl -u "$Hamin" -X GET $Address/api/repositories/$Image/tags -H "Content-Type: application/json" | awk /"name": / | awk -F " {print $4}) for Tag in $Image_Tags;do # 将获取到的镜像完整路径存档到镜像清单文件 echo "$Address/$Image:$Tag" | grep -v Base | grep -v Image | grep -v CentOS >> $File done donedone