应用程序触发的docker构建无法从私有伪影注册表中提取pip要求
应用程序触发的docker构建无法从私有伪影注册表中提取pip要求
提问于 2022-11-17 13:50:35
我有一个谷歌云构建的Python烧瓶应用程序，触发了拉请求我的git回购。我正在尝试添加一个私有python包依赖项，它存储在google注册中心上。当我将服务帐户json复制到docker容器并将GOOGLE_APPLICATION_CREDENTIALS指向它时，这在本地工作得很好，但我不希望将服务密钥提交给github，并且希望避免服务密钥位于容器中。
这类似于这个问题，但没有得到回答，只是建议使用一个短暂的访问令牌，但是没有关于如何将其集成到自动云构建触发器中的详细信息/文档。
我的Dockerfile如下所示：
WORKDIR $APP_HOME
COPY . ./
# Let Hive know what env it is running in
ARG HIVE_BUILD_ENV=UNSET
ENV HIVE_ENV=$HIVE_BUILD_ENV
# setup timezone for Ireland and magics library for file type detection
ENV DEBIAN_FRONTEND=noninteractive
ENV TZ="Europe/Dublin"
RUN apt-get update && apt-get install -y tzdata libmagic1
FROM base as staging
ENV DB_NAME=hive-staging
RUN pip install --upgrade pip
RUN pip install keyrings.google-artifactregistry-auth
RUN pip install --no-cache-dir -r requirements.txt **# <<< build fails here**
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 "hive:create_app()"我的cloudbuild-preview.yaml start看起来是这样的：(我可以提供整个文件，但是构建在这个步骤上失败了)
steps:
- id: "build image"
name: "gcr.io/cloud-builders/docker"
args:
[
"build",
"-t",
"gcr.io/${PROJECT_ID}/${_SERVICE_NAME}:${_PR_NUMBER}-${SHORT_SHA}",
".",
"--target",
"staging",
"--no-cache"
]我的requirements.txt开始如下所示：
--index-url https://europe-west1-python.pkg.dev/hive-347910/hive-commons-art-repo/simple
--extra-index-url https://pypi.org/simple
hive-commons==0.0.6
Flask==2.2.2当我尝试运行构建时，会得到以下错误输出：
Step #0 - "build image": Step 15/16 : RUN pip install --no-cache-dir -r requirements.txt
Step #0 - "build image": ---> Running in 1d17c8df7022
Step #0 - "build image": Looking in indexes: https://europe-west1-python.pkg.dev/hive-347910/hive-commons-art-repo/simple, https://pypi.org/simple
Step #0 - "build image": WARNING: Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
Step #0 - "build image": WARNING: Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: timed out
Step #0 - "build image": WARNING: Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out
Step #0 - "build image": WARNING: Authentication failed using Compute Engine authentication due to unavailable metadata server.
Step #0 - "build image": WARNING: Failed to retrieve Application Default Credentials: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
Step #0 - "build image": WARNING: Trying to retrieve credentials from gcloud...
Step #0 - "build image": WARNING: Failed to retrieve credentials from gcloud: gcloud command exited with status: [Errno 2] No such file or directory: 'gcloud'
Step #0 - "build image": WARNING: Artifact Registry PyPI Keyring: No credentials could be found.
Step #0 - "build image": WARNING: Keyring is skipped due to an exception: Failed to find credentials, Please run: `gcloud auth application-default login or export GOOGLE_APPLICATION_CREDENTIALS=<path/to/service/account/key>`
Step #0 - "build image": User for europe-west1-python.pkg.dev: ERROR: Exception:
Step #0 - "build image": Traceback (most recent call last):
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
Step #0 - "build image": status = run_func(*args)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
Step #0 - "build image": return func(self, options, args)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 400, in run
Step #0 - "build image": requirement_set = resolver.resolve(
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
Step #0 - "build image": result = self._result = resolver.resolve(
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
Step #0 - "build image": state = resolution.resolve(requirements, max_rounds=max_rounds)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
Step #0 - "build image": self._add_to_criteria(self.state.criteria, r, parent=None)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
Step #0 - "build image": if not criterion.candidates:
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
Step #0 - "build image": return bool(self._sequence)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
Step #0 - "build image": return any(self)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
Step #0 - "build image": return (c for c in iterator if id(c) not in self._incompatible_ids)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 44, in _iter_built
Step #0 - "build image": for version, func in infos:
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 279, in iter_index_candidate_infos
Step #0 - "build image": result = self._finder.find_best_candidate(
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 889, in find_best_candidate
Step #0 - "build image": candidates = self.find_all_candidates(project_name)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 830, in find_all_candidates
Step #0 - "build image": page_candidates = list(page_candidates_it)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/sources.py", line 134, in page_candidates
Step #0 - "build image": yield from self._candidates_from_page(self._link)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 790, in process_project_url
Step #0 - "build image": index_response = self._link_collector.fetch_response(project_url)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 461, in fetch_response
Step #0 - "build image": return _get_index_content(location, session=self.session)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 364, in _get_index_content
Step #0 - "build image": resp = _get_simple_response(url, session=session)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 135, in _get_simple_response
Step #0 - "build image": resp = session.get(
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 600, in get
Step #0 - "build image": return self.request("GET", url, **kwargs)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/session.py", line 518, in request
Step #0 - "build image": return super().request(method, url, *args, **kwargs)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 587, in request
Step #0 - "build image": resp = self.send(prep, **send_kwargs)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 708, in send
Step #0 - "build image": r = dispatch_hook("response", hooks, r, **kwargs)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_vendor/requests/hooks.py", line 30, in dispatch_hook
Step #0 - "build image": _hook_data = hook(hook_data, **kwargs)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/auth.py", line 270, in handle_401
Step #0 - "build image": username, password, save = self._prompt_for_password(parsed.netloc)
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/auth.py", line 233, in _prompt_for_password
Step #0 - "build image": username = ask_input(f"User for {netloc}: ")
Step #0 - "build image": File "/usr/local/lib/python3.8/site-packages/pip/_internal/utils/misc.py", line 204, in ask_input
Step #0 - "build image": return input(message)
Step #0 - "build image": EOFError: EOF when reading a line
Step #0 - "build image": The command '/bin/sh -c pip install --no-cache-dir -r requirements.txt' returned a non-zero code: 2
Finished Step #0 - "build image"
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 2
Step #0 - "build image": 这两行是我问题的根源：
警告:由于不可用元数据服务器，使用Compute Engine身份验证失败。警告:无法检索应用程序默认凭据:无法自动确定凭据。请设置GOOGLE_APPLICATION_CREDENTIALS或显式创建凭据并重新运行应用程序.有关更多信息，请参见https://cloud.google.com/docs/authentication/getting-started
但是，查看错误消息中链接的文档，我看不到一种在没有服务帐户json的情况下验证在码头容器构建中运行的pip的方法。我还看到了与这个问题相关的其他问题，人们建议进行多阶段构建，如果我在开发机器上运行构建，那么这个问题就会奏效，但我面临的问题是，多阶段构建中的第一阶段从什么地方提取服务帐户密钥，而它只是作为自动云构建的一部分从源代码中提取的。
如能就如何处理这一问题提出建议，将不胜感激。
回答 1
Stack Overflow用户
回答已采纳
发布于 2022-11-18 11:14:37
编辑：在与google联系后，向我展示了一种无需使用秘密管理器或服务帐户json执行此步骤的方法。
文档中的这一页显示，如果将"--network=cloudbuild"添加到yaml文件中的停靠构建步骤，则它将步骤的容器附加到名为cloudbuild的本地docker网络，该网络具有pip访问伪影注册表所需的ADC creds。此解决方案非常简单，并删除了对服务帐户creds文件的任何要求。更安全、更不复杂的Dockerfile。所以我的yaml文件现在看起来如下：
steps:
- id: "build image"
name: "gcr.io/cloud-builders/docker"
args:
[
"build",
"-t",
"gcr.io/${PROJECT_ID}/${_SERVICE_NAME}:${_PR_NUMBER}-${SHORT_SHA}",
".",
"--target",
"staging",
"--network=cloudbuild"
]原始解决方案：找到了一个不需要将服务帐户json提交到源代码管理的解决方案。
云构建可以访问秘密管理器中存储的秘密。
因此，我将服务帐户json作为- build传递到docker命令中，然后将其保存到一个文件中，并将GOOGLE_APPLICATION_CREDENTIALS指向该位置。之后，我删除文件并取消设置env变量。
以下是我的yaml文件的相关部分：
- id: "build image"
name: "gcr.io/cloud-builders/docker"
entrypoint: 'bash'
args:
[
"-c",
'docker build --tag gcr.io/${PROJECT_ID}/${_SERVICE_NAME}:${_PR_NUMBER}-${SHORT_SHA} --target staging --build-arg SERVICE_ACCOUNT_JSON="$$CREDS_JSON" --no-cache .'
]
secretEnv: ['CREDS_JSON']
availableSecrets:
secretManager:
- versionName: projects/$PROJECT_ID/secrets/fake_secret_name/versions/latest
env: 'CREDS_JSON'以及Dockerfile中处理此问题的部分：
ARG CREDS_JSON
WORKDIR $APP_HOME
RUN touch creds.json
RUN bash -c 'echo -E "$CREDS_JSON" >> ./creds.json'
ARG GOOGLE_APPLICATION_CREDENTIALS="$APP_HOME/creds.json"
RUN pip install --upgrade pip
RUN pip install keyrings.google-artifactregistry-auth
RUN pip install --no-cache-dir -r requirements.txt
RUN rm $APP_HOME/creds.json特别重要的是要注意-build周围的双引号，并使用RUN bash -c 'echo -E "$CREDS_JSON" >> ./creds.json'而不是内置的Docker命令RUN echo来保留json文件中的空白和回车，否则keyring包无法将cred文件作为有效的json文件处理。
这解决了我在不需要在源代码管理中存储凭据的情况下从伪存储库中访问pip需求的问题，但是服务帐户creds json仍然在docker映像构建阶段的历史中公开，所以我不是100%使用这个解决方案，所以我正在考虑使用多阶段构建来进一步限制这种暴露。我也会在短时间内轮换这些凭证。
理想情况下，我希望docker构建使用运行构建的云构建服务帐户的凭据，并且我已经与google云支持部门联系，看看这是否可行。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/74476618
复制相关文章
相似问题
腾讯云开发者
