Loading [MathJax]/jax/output/CommonHTML/config.js
文档建议反馈控制台
首页
学习
活动
专区
圈层
工具
文章/答案/技术大牛
发布
首页
学习
活动
专区
圈层
工具
返回腾讯云官网
社区首页 >问答首页 >不能再在requirements.txt文件中安装Google数据流作业的“Google- Cloud -datastore`”依赖项

问不能再在requirements.txt文件中安装Google数据流作业的“Google- Cloud -datastore`”依赖项
EN

Stack Overflow用户
提问于 2018-07-31 16:31:56
回答 1查看 894关注 0票数 3

类似于上一篇文章的事情再次发生：Google Python cloud-dataflow instances broke without new deployment (failed pubsub import)

本质上，一夜之间，我们所有的云数据流工作似乎都无缘无故地中断了。没有新的部署，没有SDK更新，也没有什么，我们的团队只是唤醒StackDriver通知，说我们的映射作业一夜之间就失败了。

这是作业失败的堆栈跟踪。

代码语言：javascript
代码运行次数：0
运行
AI代码解释
复制
Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 609, in do_work work_executor.execute()
    File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 167, in execute op.start()
    File "apache_beam/runners/worker/operations.py", line 340, in apache_beam.runners.worker.operations.DoOperation.start def start(self):
    File "apache_beam/runners/worker/operations.py", line 341, in apache_beam.runners.worker.operations.DoOperation.start with self.scoped_start_state:
    File "apache_beam/runners/worker/operations.py", line 346, in apache_beam.runners.worker.operations.DoOperation.start pickler.loads(self.spec.serialized_fn))
    File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 225, in loads return dill.loads(s)
    File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 277, in loads return load(file)
    File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 266, in load obj = pik.load()
    File "/usr/lib/python2.7/pickle.py", line 864, in load dispatch[key](self)
    File "/usr/lib/python2.7/pickle.py", line 1096, in load_global klass = self.find_class(module, name)
    File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 423, in find_class return StockUnpickler.find_class(self, module, name)
    File "/usr/lib/python2.7/pickle.py", line 1130, in find_class __import__(module)
    File "/usr/local/lib/python2.7/dist-packages/dataflow_pipeline/invoice_overages.py", line 26,
in <module> from google.cloud.datastore.helpers import entity_from_protobuf ImportError: No module named datastore.helpers

我尝试在本地复制错误，这似乎是因为无法安装google-cloud-datastore。

这是我们当前的requirements.txt文件内容。

代码语言：javascript
代码运行次数：0
运行
AI代码解释
复制
Flask==0.12.2
apache-beam[gcp]
google-cloud-dataflow
gunicorn==19.7.1
google-cloud-datastore==1.3.0
pytz
google-cloud-pubsub
google-gax
grpc-google-iam-v1
googleapis-common-protos
google-cloud==0.32
six==1.10.0
protobuf

我现在基本上无法在当地复制这个。我安装了这些要求

代码语言：javascript
代码运行次数：0
运行
AI代码解释
复制
httplib2==0.9.1
oauth2client==3.0.0
google-cloud-dataflow==2.5.0

我得到了上面显示的错误，

代码语言：javascript
代码运行次数：0
运行
AI代码解释
复制
$ python main.py
Traceback (most recent call last):
  File "main.py", line 25, in <module>
    import dataflow_pipeline.summarize_intervals as summarization_pipeline
  File "/Users/john/camio-mappers/box-counters-pipeline/dataflow_pipeline/summarize_intervals.py", line 31, in <module>
    from google.cloud.datastore.helpers import entity_from_protobuf
ImportError: No module named datastore.helpers
(venv)

但如果我那么做pip install --ignore-installed google-cloud-datastore

我得到了这个疯狂的错误

代码语言：javascript
代码运行次数：0
运行
AI代码解释
复制
$ python main.py
Traceback (most recent call last):
  File "main.py", line 25, in <module>
    import dataflow_pipeline.summarize_intervals as summarization_pipeline
  File "/Users/john/camio-mappers/box-counters-pipeline/dataflow_pipeline/summarize_intervals.py", line 31, in <module>
    from google.cloud.datastore.helpers import entity_from_protobuf
  File "/Users/john/camio-mappers/box-counters-pipeline/venv/lib/python2.7/site-packages/google/cloud/datastore/__init__.py", line 61, in <module>
    from google.cloud.datastore.batch import Batch
  File "/Users/john/camio-mappers/box-counters-pipeline/venv/lib/python2.7/site-packages/google/cloud/datastore/batch.py", line 24, in <module>
    from google.cloud.datastore import helpers
  File "/Users/john/camio-mappers/box-counters-pipeline/venv/lib/python2.7/site-packages/google/cloud/datastore/helpers.py", line 29, in <module>
    from google.cloud.datastore_v1.proto import datastore_pb2
  File "/Users/john/camio-mappers/box-counters-pipeline/venv/lib/python2.7/site-packages/google/cloud/datastore_v1/__init__.py", line 17, in <module>
    from google.cloud.datastore_v1 import types
  File "/Users/john/camio-mappers/box-counters-pipeline/venv/lib/python2.7/site-packages/google/cloud/datastore_v1/types.py", line 26, in <module>
    from google.cloud.datastore_v1.proto import datastore_pb2
  File "/Users/john/camio-mappers/box-counters-pipeline/venv/lib/python2.7/site-packages/google/cloud/datastore_v1/proto/datastore_pb2.py", line 17, in <module>
    from google.cloud.datastore_v1.proto import entity_pb2 as google_dot_cloud_dot_datastore__v1_dot_proto_dot_entity__pb2
  File "/Users/john/camio-mappers/box-counters-pipeline/venv/lib/python2.7/site-packages/google/cloud/datastore_v1/proto/entity_pb2.py", line 28, in <module>
    dependencies=[google_dot_api_dot_annotations__pb2.DESCRIPTOR,google_dot_protobuf_dot_struct__pb2.DESCRIPTOR,google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,google_dot_type_dot_latlng__pb2.DESCRIPTOR,])
  File "/Users/john/camio-mappers/box-counters-pipeline/venv/lib/python2.7/site-packages/google/protobuf/descriptor.py", line 878, in __new__
    return _message.default_pool.AddSerializedFile(serialized_pb)
TypeError: Couldn't build proto file into descriptor pool!
Invalid proto descriptor for file "google/cloud/datastore_v1/proto/entity.proto":
  google.datastore.v1.PartitionId.project_id: "google.datastore.v1.PartitionId.project_id" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.PartitionId.namespace_id: "google.datastore.v1.PartitionId.namespace_id" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.PartitionId: "google.datastore.v1.PartitionId" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Key.partition_id: "google.datastore.v1.Key.partition_id" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Key.path: "google.datastore.v1.Key.path" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Key.PathElement.id_type: "google.datastore.v1.Key.PathElement.id_type" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Key.PathElement.kind: "google.datastore.v1.Key.PathElement.kind" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Key.PathElement.id: "google.datastore.v1.Key.PathElement.id" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Key.PathElement.name: "google.datastore.v1.Key.PathElement.name" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Key.PathElement: "google.datastore.v1.Key.PathElement" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Key: "google.datastore.v1.Key" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.ArrayValue.values: "google.datastore.v1.ArrayValue.values" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.ArrayValue: "google.datastore.v1.ArrayValue" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.value_type: "google.datastore.v1.Value.value_type" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.null_value: "google.datastore.v1.Value.null_value" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.boolean_value: "google.datastore.v1.Value.boolean_value" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.integer_value: "google.datastore.v1.Value.integer_value" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.double_value: "google.datastore.v1.Value.double_value" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.timestamp_value: "google.datastore.v1.Value.timestamp_value" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.key_value: "google.datastore.v1.Value.key_value" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.string_value: "google.datastore.v1.Value.string_value" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.blob_value: "google.datastore.v1.Value.blob_value" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.geo_point_value: "google.datastore.v1.Value.geo_point_value" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.entity_value: "google.datastore.v1.Value.entity_value" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.array_value: "google.datastore.v1.Value.array_value" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.meaning: "google.datastore.v1.Value.meaning" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value.exclude_from_indexes: "google.datastore.v1.Value.exclude_from_indexes" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Value: "google.datastore.v1.Value" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Entity.key: "google.datastore.v1.Entity.key" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Entity.properties: "google.datastore.v1.Entity.properties" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Entity.PropertiesEntry.key: "google.datastore.v1.Entity.PropertiesEntry.key" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Entity.PropertiesEntry.value: "google.datastore.v1.Entity.PropertiesEntry.value" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Entity.PropertiesEntry: "google.datastore.v1.Entity.PropertiesEntry" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Entity: "google.datastore.v1.Entity" is already defined in file "google/cloud/proto/datastore/v1/entity.proto".
  google.datastore.v1.Key.partition_id: "google.datastore.v1.PartitionId" seems to be defined in "google/cloud/proto/datastore/v1/entity.proto", which is not imported by "google/cloud/datastore_v1/proto/entity.proto".  To use it here, please add the necessary import.
  google.datastore.v1.Key.path: "google.datastore.v1.Key.PathElement" seems to be defined in "google/cloud/proto/datastore/v1/entity.proto", which is not imported by "google/cloud/datastore_v1/proto/entity.proto".  To use it here, please add the necessary import.
  google.datastore.v1.ArrayValue.values: "google.datastore.v1.Value" seems to be defined in "google/cloud/proto/datastore/v1/entity.proto", which is not imported by "google/cloud/datastore_v1/proto/entity.proto".  To use it here, please add the necessary import.
  google.datastore.v1.Value.key_value: "google.datastore.v1.Key" seems to be defined in "google/cloud/proto/datastore/v1/entity.proto", which is not imported by "google/cloud/datastore_v1/proto/entity.proto".  To use it here, please add the necessary import.
  google.datastore.v1.Value.entity_value: "google.datastore.v1.Entity" seems to be defined in "google/cloud/proto/datastore/v1/entity.proto", which is not imported by "google/cloud/datastore_v1/proto/entity.proto".  To use it here, please add the necessary import.
  google.datastore.v1.Value.array_value: "google.datastore.v1.ArrayValue" seems to be defined in "google/cloud/proto/datastore/v1/entity.proto", which is not imported by "google/cloud/datastore_v1/proto/entity.proto".  To use it here, please add the necessary import.
  google.datastore.v1.Entity.PropertiesEntry.value: "google.datastore.v1.Value" seems to be defined in "google/cloud/proto/datastore/v1/entity.proto", which is not imported by "google/cloud/datastore_v1/proto/entity.proto".  To use it here, please add the necessary import.
  google.datastore.v1.Entity.key: "google.datastore.v1.Key" seems to be defined in "google/cloud/proto/datastore/v1/entity.proto", which is not imported by "google/cloud/datastore_v1/proto/entity.proto".  To use it here, please add the necessary import.
  google.datastore.v1.Entity.properties: "google.datastore.v1.Entity.PropertiesEntry" seems to be defined in "google/cloud/proto/datastore/v1/entity.proto", which is not imported by "google/cloud/datastore_v1/proto/entity.proto".  To use it here, please add the necessary import.

我不知道这意味着什么。我确实丢失了列出的.proto文件，但是为什么会丢失它们呢？

我的主要问题是，这是最后一次发生这种情况:为什么当我们没有进行任何新的部署时，作业就可以中断呢？假设，如果我们不更改任何代码，代码就不会中断。如果我们不改变任何依赖项，那么我们就不应该最终导致管道破裂，然后我们不得不去修复。已经脱离了beta版，应该是稳定的，如果它处于beta版，并且受到API更改的影响，这是意料之中的，但此时它应该是稳定的。到目前为止，我们的管道已经关闭，我们不知道要多长时间。

我对google-cloud-datastore==1.4.0做了建议的修改，它部分地修复了我的管道，但现在它又失败了。现在看来，google-cloud-pubsub失败了，出现了类似的丢失.proto文件的错误。

代码语言：javascript
代码运行次数：0
运行
AI代码解释
复制
ERROR: (gcloud.app.deploy) Error Response: [9]
Application startup error:
    import dataflow_pipeline.tally_overages as overaging_pipeline
  File "/home/vmagent/app/dataflow_pipeline/tally_overages.py", line 29, in <module>
    from google.cloud import pubsub
  File "/env/local/lib/python2.7/site-packages/google/cloud/pubsub.py", line 17, in <module>
    from google.cloud.pubsub_v1 import PublisherClient
  File "/env/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/__init__.py", line 17, in <module>
    from google.cloud.pubsub_v1 import types
  File "/env/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/types.py", line 30, in <module>
    from google.cloud.pubsub_v1.proto import pubsub_pb2
  File "/env/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/proto/pubsub_pb2.py", line 29, in <module>
    dependencies=[google_dot_api_dot_annotations__pb2.DESCRIPTOR,google_dot_protobuf_dot_duration__pb2.DESCRIPTOR,google_dot_protobuf_dot_empty__pb2.DESCRIPTOR,google_dot_protobuf_dot_field__mask__pb2.DESCRIPTOR,google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,])
  File "/env/local/lib/python2.7/site-packages/google/protobuf/descriptor.py", line 878, in __new__
    return _message.default_pool.AddSerializedFile(serialized_pb)
TypeError: Couldn't build proto file into descriptor pool!
Invalid proto descriptor for file "google/cloud/pubsub_v1/proto/pubsub.proto":
  google.pubsub.v1.Topic.name: "google.pubsub.v1.Topic.name" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Topic: "google.pubsub.v1.Topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.data: "google.pubsub.v1.PubsubMessage.data" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.attributes: "google.pubsub.v1.PubsubMessage.attributes" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.message_id: "google.pubsub.v1.PubsubMessage.message_id" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.publish_time: "google.pubsub.v1.PubsubMessage.publish_time" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.AttributesEntry.key: "google.pubsub.v1.PubsubMessage.AttributesEntry.key" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.AttributesEntry.value: "google.pubsub.v1.PubsubMessage.AttributesEntry.value" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.AttributesEntry: "google.pubsub.v1.PubsubMessage.AttributesEntry" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage: "google.pubsub.v1.PubsubMessage" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.GetTopicRequest.topic: "google.pubsub.v1.GetTopicRequest.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.GetTopicRequest: "google.pubsub.v1.GetTopicRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishRequest.topic: "google.pubsub.v1.PublishRequest.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishRequest.messages: "google.pubsub.v1.PublishRequest.messages" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishRequest: "google.pubsub.v1.PublishRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishResponse.message_ids: "google.pubsub.v1.PublishResponse.message_ids" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishResponse: "google.pubsub.v1.PublishResponse" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsRequest.project: "google.pubsub.v1.ListTopicsRequest.project" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsRequest.page_size: "google.pubsub.v1.ListTopicsRequest.page_size" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsRequest.page_token: "google.pubsub.v1.ListTopicsRequest.page_token" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsRequest: "google.pubsub.v1.ListTopicsRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsResponse.topics: "google.pubsub.v1.ListTopicsResponse.topics" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsResponse.next_page_token: "google.pubsub.v1.ListTopicsResponse.next_page_token" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsResponse: "google.pubsub.v1.ListTopicsResponse" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsRequest.topic: "google.pubsub.v1.ListTopicSubscriptionsRequest.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsRequest.page_size: "google.pubsub.v1.ListTopicSubscriptionsRequest.page_size" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsRequest.page_token: "google.pubsub.v1.ListTopicSubscriptionsRequest.page_token" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsRequest: "google.pubsub.v1.ListTopicSubscriptionsRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsResponse.subscriptions: "google.pubsub.v1.ListTopicSubscriptionsResponse.subscriptions" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsResponse.next_page_token: "google.pubsub.v1.ListTopicSubscriptionsResponse.next_page_token" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsResponse: "google.pubsub.v1.ListTopicSubscriptionsResponse" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.DeleteTopicRequest.topic: "google.pubsub.v1.DeleteTopicRequest.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.DeleteTopicRequest: "google.pubsub.v1.DeleteTopicRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.name: "google.pubsub.v1.Subscription.name" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.topic: "google.pubsub.v1.Subscription.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.push_config: "google.pubsub.v1.Subscription.push_config" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.ack_deadline_seconds: "google.pubsub.v1.Subscription.ack_deadline_seconds" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.retain_acked_messages: "google.pubsub.v1.Subscription.retain_acked_messages" is already defined in 
  google.pubsub.v1.Subscriber.Seek: "google.pubsub.v1.SeekRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Subscriber.Seek: "google.pubsub.v1.SeekResponse" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.CreateTopic: "google.pubsub.v1.Topic" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.CreateTopic: "google.pubsub.v1.Topic" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.UpdateTopic: "google.pubsub.v1.Topic" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.Publish: "google.pubsub.v1.PublishRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.Publish: "google.pubsub.v1.PublishResponse" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.GetTopic: "google.pubsub.v1.GetTopicRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.GetTopic: "google.pubsub.v1.Topic" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.ListTopics: "google.pubsub.v1.ListTopicsRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.ListTopics: "google.pubsub.v1.ListTopicsResponse" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.ListTopicSubscriptions: "google.pubsub.v1.ListTopicSubscriptionsRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.ListTopicSubscriptions: "google.pubsub.v1.ListTopicSubscriptionsResponse" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.DeleteTopic: "google.pubsub.v1.DeleteTopicRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.

我不太明白这些错误意味着什么。这是我的新requirements.txt文件

代码语言：javascript
代码运行次数：0
运行
AI代码解释
复制
httplib2==0.9.1
oauth2client==3.0.0
google-cloud-dataflow==2.5.0
dill==0.2.6
Flask
gunicorn
pytz
googledatastore
google-cloud-datastore==1.4.0
google-cloud
google-cloud-pubsub

编辑-以下是代码的导入行

档案1

代码语言：javascript
代码运行次数：0
运行
AI代码解释
复制
from __future__ import absolute_import

# standard imports
import datetime
import json
import logging
import base64
import collections
import traceback
import hashlib
from functools import reduce

# apache beam / dataflow imports
import apache_beam as beam
from apache_beam.io.gcp.datastore.v1.datastoreio import ReadFromDatastore
from apache_beam.options.pipeline_options import PipelineOptions

# google cloud datastore imports
from google.cloud.proto.datastore.v1 import query_pb2
from googledatastore import helper as datastore_helper, PropertyFilter, CompositeFilter
from google.cloud.datastore.helpers import entity_from_protobuf

# custom utility imports
from .util import *

档案2

代码语言：javascript
代码运行次数：0
运行
AI代码解释
复制
from __future__ import absolute_import

import datetime
import math
import json
import logging
import traceback
import collections
import hashlib
from functools import reduce

import apache_beam as beam
from apache_beam.io.gcp.datastore.v1.datastoreio import ReadFromDatastore
from apache_beam.options.pipeline_options import PipelineOptions

from google.cloud.proto.datastore.v1 import query_pb2
from googledatastore import helper as datastore_helper, PropertyFilter, CompositeFilter
from google.cloud.datastore.helpers import entity_from_protobuf
from google.cloud import pubsub

from .util import *

档案3

代码语言：javascript
代码运行次数：0
运行
AI代码解释
复制
from __future__ import absolute_import

import math
import json
import logging
import traceback
import uuid
import collections
import hashlib
import datetime
from functools import reduce

import apache_beam as beam
from apache_beam.io.gcp.datastore.v1.datastoreio import ReadFromDatastore
from apache_beam.options.pipeline_options import PipelineOptions

from google.cloud.proto.datastore.v1 import query_pb2
from googledatastore import helper as datastore_helper, PropertyFilter, CompositeFilter
from google.cloud.datastore.helpers import entity_from_protobuf
from google.cloud import pubsub

from .util import *

以及从util.*文件

代码语言：javascript
代码运行次数：0
运行
AI代码解释
复制
import re
import os
import math
import datetime
import json
import logging
import base64
import traceback
import itertools
import pytz
import requests
from functools import reduce

# google cloud datastore imports
from google.cloud.proto.datastore.v1 import query_pb2
from googledatastore import helper as datastore_helper, PropertyFilter, CompositeFilter
from google.cloud.datastore.helpers import entity_from_protobuf
python
google-cloud-platform
google-cloud-datastore
python-import
google-cloud-dataflow
EN

回答 1

Stack Overflow用户

回答已采纳

发布于 2018-08-10 18:04:29

最后，我通过创建一个最小的管道集合并一个一个地添加依赖项来使它工作，直到我找到了一些有用的组合。依赖项如下

代码语言：javascript
代码运行次数：0
运行
AI代码解释
复制
Flask
gunicorn
apache-beam[gcp]==2.6.0
oauth2client==3.0.0
google-cloud-datastore==1.3.0
google-cloud-pubsub==0.28.0
google-cloud-core==0.27.0
google-cloud==0.34.0

我们的生产管道已经离线一周了，但是现在已经上线了，希望这能帮助到外面的人。

编辑-

说得太早了。此组合在本地工作，但仍然在云中中断>:-(

EDIT2 -

所以..。谷歌似乎停止了从requirements.txt文件中安装。我把这个放在我的setup.py文件里

REQUIRED_PACKAGES = ['google-cloud']

看起来这是安装的唯一依赖项！当我添加这段代码

代码语言：javascript
代码运行次数：0
运行
AI代码解释
复制
if os.path.exists('requirements.txt'):
    with open('requirements.txt') as fh:
        REQUIRED_PACKAGES=[line.strip() for line in fh.readlines()]

一切都成功了！现在我的管道开动了。

所以看起来谷歌改变了一些他们停止从requirements.txt安装的东西。

什么！？！

票数 4
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：

https://stackoverflow.com/questions/51623806

复制
相关文章
Hive 视图和索引
hive大数据存储
Hive 中的视图和 RDBMS 中视图的概念一致，都是一组数据的逻辑表示，本质上就是一条 SELECT 语句的结果集。视图是纯粹的逻辑对象，没有关联的存储 (Hive 3.0.0 引入的物化视图除外)，当查询引用视图时，Hive 可以将视图的定义与查询结合起来，例如将查询中的过滤器推送到视图中。
每天进步一点点
2022/12/15
1.4K0
Hive 视图和索引
SQL Server 索引和视图
server数据索引数据库sql
索引（Index）是数据库中一种用于加速对表中数据的检索速度的数据结构。索引类似于书籍的目录，它提供了一种快速查找数据行的方法，避免了全表扫描的开销。通过使用索引，数据库系统可以直接定位到符合特定搜索条件的数据，而不必逐行遍历整个表。
神秘泣男子
2024/06/03
1140
SQL Server 索引和视图
事务、视图、索引、备份和恢复
云推荐引擎数据库sql云数据库 SQL Server
创建视图：CREATE VIEW view_name AS <SELECT 语句>;
小陈运维
2021/10/13
3570
数据库：视图和索引
javahttpssql网络安全数据结构
视图是一张虚拟表，并不在数据库中以存储数据值集的形式存在。在引用过程中依据基表动态生成。
全栈程序员站长
2022/09/16
6310
视图索引
云推荐引擎sql
大家好，又见面了，我是你们的朋友全栈君。创建索引视图 视图也称为虚拟表，这是因为由视图返回的结果集其一般格式与由列和行组成的表相似，并且，在 SQL 语句中引用视图的方式也与引用表的方式相同。标准视图的结果集不是永久地存储在数据库中。查询每次引用视图时，Microsoft&reg; SQL Server&#8482; 2000 会动态地将生成视图结果集所需的逻辑合并到从基表数据生成完整查询结果集所需的逻辑中。生成视图结果的过程称为视图具体化。有关更多信息，请参见视图解析。 对于标准视图而言，为每个引用视图的查询动态生成结果集的开销很大，特别是对于那些涉及对大量行进行复杂处理（如聚合大量数据或联接许多行）的视图更为可观。若经常在查询中引用这类视图，可通过在视图上创建唯一聚集索引来提高性能。在视图上创建唯一聚集索引时将执行该视图，并且结果集在数据库中的存储方式与带聚集索引的表的存储方式相同。有关用于存储聚集索引的结构的更多信息，请参见聚集索引。 说明 只有安装了 Microsoft SQL Server 2000 企业版或 Microsoft SQL Server 2000 开发版，才可以创建索引视图。 在视图上创建索引的另一个好处是：查询优化器开始在查询中使用视图索引，而不是直接在 FROM 子句中命名视图。这样一来，可从索引视图检索数据而无需重新编码，由此带来的高效率也使现有查询获益。有关更多信息，请参见在视图上使用索引。 在视图上创建聚集索引可存储创建索引时存在的数据。索引视图还自动反映自创建索引后对基表数据所做的更改，这一点与在基表上创建的索引相同。当对基表中的数据进行更改时，索引视图中存储的数据也反映数据更改。视图的聚集索引必须唯一，从而提高了 SQL Server 在索引中查找受任何数据更改影响的行的效率。 与基表上的索引相比，对索引视图的维护可能更复杂。只有当视图的结果检索速度的效益超过了修改所需的开销时，才应在视图上创建索引。这样的视图通常包括映射到相对静态的数据上、处理多行以及由许多查询引用的视图。 视图的要求 在视图上创建聚集索引之前，该视图必须满足下列要求： 当执行 CREATE VIEW 语句时，ANSI_NULLS 和 QUOTED_IDENTIFIER 选项必须设置为 ON。OBJECTPROPERTY 函数通过 ExecIsAnsiNullsOn 或 ExecIsQuotedIdentOn 属性为视图报告此信息。 为执行所有 CREATE TABLE 语句以创建视图引用的表，ANSI_NULLS 选项必须设置为 ON。 视图不能引用任何其它视图，只能引用基表。 视图引用的所有基表必须与视图位于同一个数据库中，并且所有者也与视图相同。 必须使用 SCHEMABINDING 选项创建视图。SCHEMABINDING 将视图绑定到基础基表的架构。 必须已使用 SCHEMABINDING 选项创建了视图中引用的用户定义的函数。 表和用户定义的函数必须由 2 部分的名称引用。不允许使用 1 部分、3 部分和 4 部分的名称。 视图中的表达式所引用的所有函数必须是确定性的。OBJECTPROPERTY 函数的 IsDeterministic 属性报告用户定义的函数是否是确定性的。有关更多信息，请参见确定性函数和非确定性函数。 视图中的 SELECT 语句不能包含下列 Transact-SQL 语法元素： 选择列表不能使用 * 或 table_name.* 语法指定列。必须显式给出列名。 不能在多个视图列中指定用作简单表达式的表的列名。如果对列的所有（或只有一个例外）引用是复杂表达式的一部分或是函数的一个参数，则可多次引用该列。例如，下列选择列表是非法的： SELECT ColumnA, ColumnB, ColumnA 下列选择列表是合法的： SELECT ColumnA, AVG(ColumnA), ColumnA + Column B AS AddColAColB SELECT SUM(ColumnA), ColumnA % ColumnB AS ModuloColAColB 派生表。 行集函数。 UNION 运算符
全栈程序员站长
2022/09/16
1.2K0
数据库视图和索引
其他
虚拟视图是由其他其他关系上的查询所定义的一种关系。虚拟视图并不在数据库中存储，但可对其进行查询，就好像它被存储在数据库中一样。查询处理器会在执行查询时用视图的定义来替换视图。 试图也可以被物化，即它们从数据库中定期的进行构造并存储。物化可以加快查询的执行，一种典型的“物化视图”就是索引。 虚拟视图 视图定义 CREATE VIEW <视图名> AS <视图定义>; 例如：有个关系如下： Movies(title, year, length, name, producer) 要在其上创建一个视图，包括2018
SuperHeroes
2018/05/31
1.3K0
视图和索引（数据库学习）
数据结构数据库sql serversql
引入—数据库的基本表是按照数据库设计人员的观点设计的，并不一定符合所有用户的需求。
营琪
2019/11/04
1.1K0
mysql创建索引视图_mysql中创建视图、索引[通俗易懂]
全文检索云推荐引擎数据库sql存储
视图中的数据依赖于原来表中的数据，一旦表中数据发生改变，显示在视图中的数据也会发生改变。
全栈程序员站长
2022/09/16
7.7K0
索引与视图
数据库sqlhttpsoraclejava
一个多列索引可以认为是包含通过合并(concatenate)索引列值创建的值的一个排序数组。 当查询语句的条件中包含last_name 和 first_name时
全栈程序员站长
2022/09/16
3260
视图索引问题
httpsjava网络安全
　　最近和一直在研究如何加快查询数据库的速度，看了网络上说用索引，会加快查询的速度．我就认真看了索引了一些文章，也建立了表索引．但是在建立视图索引的时候遇到问题了， 无法在视图 ‘SBK_VIEW’ 上创建 索引，因为该视图未绑定到架购．查一下相关的资料，发现：
全栈程序员站长
2022/09/16
4220
Oracle 视图索引
replaceselectsetunionview
注：为了在当前用户模式中创建视图，要求数据库用户必须有create any view（创建任何视图）的权限。
全栈程序员站长
2022/09/16
1.3K0
Oracle 视图索引
solr合并索引
lucene/solr其他
solr下可以通过mergeindexes命令完成一个或多个core到其他core的索引合并，合并索引需要保证要被合并到的core的schema要与其他来源core的schema兼容，否则就会出现无法启动core的问题，我们可以通过代码CoreAdminRequest.MergeIndexes类实现，实例代码如下：
johnhuster的分享
2022/03/28
5270
【iOS 开发】父视图外部子视图点击响应 - hitTest
ios
面试被问过两次了的一个问题，记录一下：假如一个 UIView（我们称作 FatherView），有一个 subview（我们称作 ChildView），这个 ChildView 在 FatherView 的 frame 外部，那么默认情况下，这个 ChildView 被点击的时候，并不会触发 FatherView 的点击响应链（这是合情合理的，FatherView 所在的位置都没被点击，它不遍历 subviews，不作回应是正常的），这时候怎样才能让这个 ChildView 相应点击事件？
KyXu
2019/04/11
1.1K0
【iOS 开发】父视图外部子视图点击响应 - hitTest
【MySQL】表的内外连接和视图
连接数据语法mysqlselect
内连接实际上就是利用 where 子句对两种表形成的笛卡尔积进行筛选，我们前面学习的查询都是内连接，也是在开发过程中使用的最多的连接查询。
YoungMLet
2024/03/01
1840
【MySQL】表的内外连接和视图
MySQL中的索引、视图和DBA操作
数据库sql云数据库 SQL Server
索引就相当于一本书的目录，通过目录可以快速地找到对应的资源。在数据库方面，查询一张表的时候有两种检索方式：
共饮一杯无
2022/11/28
1.1K0
hive之路9-hive索引和视图
存储hive索引
本文中主要是介绍了hive中索引和视图的相关操作。 修改表 修改表主要是对表的结构和属性进行操作，包含： 重命名 alter table oldname rename to new_table; 修改表属性 alter table table_name set tblproperties (property_name=property_value); 修改表注释 alter table table_name set tblproperties('comment'=new_comment); 修改存储属
皮大大
2021/03/02
7340
SQLServer中使用索引视图(物化视图)
其他
物化视图：以前用的普通的视图，普通视图就是一段逻辑语句，对性能没有任何的提升，也不能创建索引，而物化视图会把视图里查询出来的数据在数据库上建立快照，它和物理表一样，可以创建 索引，主键约束等等，性能会有质的提升，但是其有缺点，会占用，可以设置它定时自动更新一次，也可以手动更新，当然也是可以设置及时更新的，但是会拉慢基表的增删改查操作，在这里我只讲思路，具体的话大家可以自己去研究。
跟着阿笨一起玩NET
2018/09/19
3.3K0
mysql之视图、索引
全文检索数据库sql云数据库 SQL Server
视图（View）是一种虚拟存在的表，同真实表一样，视图也由列和行构成，但视图并不实际存在于数据库中。行和列的数据来自于定义视图的查询中所使用的表，并且还是在使用视图时动态生成的。
全栈程序员站长
2022/09/16
9860
sqlserver 视图创建索引_Oracle创建索引
javahttpssql网络安全
1、添加索引 create index 索引对象名 on 索引对应表名(表内索引对象字段名); 例：需创建包含userid属性的userinfo表。 create index userid on system.userinfo(userid);
全栈程序员站长
2022/10/04
1.3K0
sqlserver 视图创建索引_数据库视图可以建立索引吗
数据库sql存储java
视图(View)是从一个或多个表或其它视图导出的，用来导出视图的表称为基表，导出的视图又称为虚表。在数据库中，只存储视图的定义，不存放视图对应的数据，这些数据仍然存放在原来的基表中。 使用视图前，必须先创建视图，创建视图要遵守以下原则： （1）只有在当前数据库中才能创建视图，视图命名必须遵循标识符规则。 （2）不能将规则、默认值或触发器与视图相关联。 （3）不能在视图上建立任何索引。
全栈程序员站长
2022/09/23
2.8K0
sqlserver 视图创建索引_数据库视图可以建立索引吗

相似问题

使用完全外部联接对视图进行索引优化

10

完全外部连接减慢查询

20

无法运行完全外部连接

10

oracle数据库过滤器与完全外部连接和合并相交

10

外部连接抑制索引使用？

10
添加站长 进交流群

领取专属 10元无门槛券

AI混元助手 在线答疑

扫码加入开发者社群
关注 腾讯云开发者公众号

洞察 腾讯核心技术

剖析业界实践案例

扫码关注腾讯云开发者公众号
领券

腾讯云开发者

扫码关注腾讯云开发者

扫码关注腾讯云开发者

领取腾讯云代金券

热门产品

热门推荐

更多推荐

Copyright © 2013 - 2025 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有 

深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 深公网安备号 44030502008569

腾讯云计算（北京）有限责任公司 京ICP证150476号 |  京ICP备11018762号 | 京公网安备号11010802020287

问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档
查看详情【社区公告】 技术创作特训营有奖征文