使用GitHub镜像加速TensorFlow编译

windmaple

发布于 2022-03-30 10:13:33

1.6K0

发布于 2022-03-30 10:13:33

由于GitHub的访问不是很稳定，速度经常很慢，这就给某些需要编译TensorFlow的同学带来了问题，因为TensorFlow主代码库放在GitHub上，更麻烦的是TensorFlow有很多第三方依赖库也在GitHub上，即使通过访问gitee等国内代码库把TF的代码clone下来，编译的时候也可能出问题，因为bazel需要在编译的时候动态下载GitHub上的第三方库。

比如我在clone TF代码库的时候就出了问题：

(python3.7) user-macbookpro2:project-src user$ git clone https://github.com/tensorflow/tensorflow.git
Cloning into 'tensorflow'...
fatal: unable to access 'https://github.com/tensorflow/tensorflow.git/': LibreSSL SSL_connect: Operation timed out in connection to github.com:443

这时候我们可以使用gitee镜像

(python3.7) user-macbookpro2:project-src user$ git clone https://gitee.com/mirrors/tensorflow.git
Cloning into 'tensorflow'...
remote: Enumerating objects: 1296821, done.
remote: Counting objects: 100% (30826/30826), done.
remote: Compressing objects: 100% (14970/14970), done.
remote: Total 1296821 (delta 23630), reused 20179 (delta 15701), pack-reused 1265995
Receiving objects: 100% (1296821/1296821), 707.78 MiB | 3.54 MiB/s, done.
Resolving deltas: 100% (1076054/1076054), done.
Updating files: 100% (25434/25434), done.

这样就解决了第一个问题。第二个问题更麻烦一些，需要将workspace里面的第三方依赖库的GitHub地址替换为国内镜像。比如我在编译TFLite benchmark的时候就出现问题，LLVM文件无法下载

(python3.7) user-macbookpro2:tensorflow user$ bazel build -c opt   --config=android_arm64   tensorflow/lite/tools/benchmark:benchmark_model
..........
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/7c02776567cc9561e6691dc05235799da45e72d6.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
WARNING: Download from https://github.com/llvm/llvm-project/archive/7c02776567cc9561e6691dc05235799da45e72d6.tar.gz failed: class java.io.IOException connect timed out
ERROR: An error occurred during the fetch of repository 'llvm-raw':
   Traceback (most recent call last):
  File "/Users/user/Desktop/project-src/tensorflow/third_party/repo.bzl", line 64, column 33, in _tf_http_archive_impl
    ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/7c02776567cc9561e6691dc05235799da45e72d6.tar.gz, https://github.com/llvm/llvm-project/archive/7c02776567cc9561e6691dc05235799da45e72d6.tar.gz] to /private/var/tmp/_bazel_user/49b53f01b9950541462cad273fc3407a/external/llvm-raw/temp3906885934226153945/7c02776567cc9561e6691dc05235799da45e72d6.tar.gz: connect timed out
ERROR: Error fetching repository: Traceback (most recent call last):
  File "/Users/user/Desktop/project-src/tensorflow/third_party/repo.bzl", line 64, column 33, in _tf_http_archive_impl
    ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/7c02776567cc9561e6691dc05235799da45e72d6.tar.gz, https://github.com/llvm/llvm-project/archive/7c02776567cc9561e6691dc05235799da45e72d6.tar.gz] to /private/var/tmp/_bazel_user/49b53f01b9950541462cad273fc3407a/external/llvm-raw/temp3906885934226153945/7c02776567cc9561e6691dc05235799da45e72d6.tar.gz: connect timed out
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/runtime/archive/c63eaf2a55892d5c4cdfedba70e83257688f7eac.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
ERROR: no such package '@llvm-raw//utils/bazel': java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/7c02776567cc9561e6691dc05235799da45e72d6.tar.gz, https://github.com/llvm/llvm-project/archive/7c02776567cc9561e6691dc05235799da45e72d6.tar.gz] to /private/var/tmp/_bazel_user/49b53f01b9950541462cad273fc3407a/external/llvm-raw/temp3906885934226153945/7c02776567cc9561e6691dc05235799da45e72d6.tar.gz: connect timed out
INFO: Elapsed time: 76.762s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

其实这时候很简单，我们只要把workspace里面的github.com替换成国内镜像（比如https://github.com.cnpmjs.org）就可以了

sed -i -e 's#https:\/\/github\.com#https://github.com.cnpmjs.org#g' ./tensorflow/*.bzl

再次尝试编译bazel就会去国内镜像下载文件进入编译过程，这样就绕过了对GitHub的依赖

(python3.7) user-macbookpro2:tensorflow user$ bazel build -c opt   --config=android_arm64   tensorflow/lite/tools/benchmark:benchmark_model
..........
DEBUG: Rule 'io_bazel_rules_docker' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1596824487 -0400"
DEBUG: Repository io_bazel_rules_docker instantiated at:
  /Users/user/Desktop/project-src/tensorflow/WORKSPACE:23:14: in <toplevel>
  /Users/user/Desktop/project-src/tensorflow/tensorflow/workspace0.bzl:108:34: in workspace
  /private/var/tmp/_bazel_user/49b53f01b9950541462cad273fc3407a/external/bazel_toolchains/repositories/repositories.bzl:35:23: in repositories
Repository rule git_repository defined at:
  /private/var/tmp/_bazel_user/49b53f01b9950541462cad273fc3407a/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
INFO: Analyzed target //tensorflow/lite/tools/benchmark:benchmark_model (120 packages loaded, 9986 targets configured).
INFO: Found 1 target...
[371 / 1,353] 8 actions running
    Compiling src/idl_parser.cpp [for host]; 19s local
    Compiling src/idl_parser.cpp; 11s local
    Compiling src/f32-vunary/gen/vsqr-neon-x8.c; 0s local
    Compiling src/qu8-vadd/gen/minmax-neon-ld64-x32.c; 0s local
    Compiling src/f32-vmulcaddc/gen/c4-minmax-neon-2x.c; 0s local
    Compiling src/qc8-igemm/gen/4x16-minmax-fp32-aarch64-neon-mlal-lane-prfm-cortex-a53.S; 0s local
    Compiling src/x32-packx/x4-neon-st4.c; 0s local
    Compiling src/qc8-igemm/gen/4x16c4-minmax-fp32-aarch64-neondot-cortex-a55.S; 0s local

当然大家可以选择自己熟悉的GitHub镜像来加速对TensorFlow的编译，CNPM只是其中的一个选择

本文参与腾讯云自媒体分享计划，分享自微信公众号。

原始发表：2022-01-28，如有侵权请联系 cloudcommunity@tencent.com 删除

tensorflow