httpx.ReadTimeout在Supabase-py上的距离滤波器
httpx.ReadTimeout在Supabase-py上的距离滤波器
提问于 2022-10-24 20:16:22
我桌子上有375000件东西。我正在做一个循环以获得所有条目的所有id，每个API调用的api限制设置为20000项。在200000之后，我总是开始获得httpx.ReadTimeout:读取操作有时超时--可能达到240000，但永远不要继续。
在每个循环之后，我尝试有不同的等待时间。我试图将api限制更改为10000，并将其提高到30000或50000，但在所有情况下，它只能停留在150000或200000左右。
existing_search_result = supabase.table('vehicles').select('ref_id', count='exact').order('id', desc=False).execute()
existing_items = []
range_step = len(existing_search_result.data)
total_existing_items = existing_search_result.count
print(total_existing_items)
while len(existing_items) < total_existing_items:
try:
existing_items += (
supabase.table(
'vehicles'
).select('ref_id')
.order('id', desc=False)
.range(range_start, range_start + range_step)
.execute()
).data
range_start += range_step
except Exception as e:
logging.exception(e)
print(range_start, len(existing_items))
time.sleep(0.30)错误日志：
2022-10-23 21:04:14,168:ERROR - The read operation timed out
Traceback (most recent call last):
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
yield
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpcore/backends/sync.py", line 26, in read
return self._sock.recv(max_bytes)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/ssl.py", line 1226, in recv
return self.read(buflen)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/ssl.py", line 1101, in read
return self._sslobj.read(len)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
yield
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 204, in handle_request
resp = self._pool.handle_request(req)
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 253, in handle_request
raise exc
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 237, in handle_request
response = connection.handle_request(request)
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 90, in handle_request
return self._connection.handle_request(request)
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 102, in handle_request
raise exc
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 81, in handle_request
) = self._receive_response_headers(**kwargs)
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 143, in _receive_response_headers
event = self._receive_event(timeout=timeout)
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 172, in _receive_event
data = self._network_stream.read(
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpcore/backends/sync.py", line 26, in read
return self._sock.recv(max_bytes)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/contextlib.py", line 135, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
raise to_exc(exc)
httpcore.ReadTimeout: The read operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/ak4zh/updater/main.py", line 278, in job
supabase.table(
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/postgrest/_sync/request_builder.py", line 53, in execute
r = self.session.request(
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpx/_client.py", line 802, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpx/_client.py", line 889, in send
response = self._send_handling_auth(
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpx/_client.py", line 917, in _send_handling_auth
response = self._send_handling_redirects(
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpx/_client.py", line 954, in _send_handling_redirects
response = self._send_single_request(request)
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpx/_client.py", line 990, in _send_single_request
response = transport.handle_request(request)
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 204, in handle_request
resp = self._pool.handle_request(req)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/contextlib.py", line 135, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/ak4zh/updater/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ReadTimeout: The read operation timed out回答 1
Stack Overflow用户
回答已采纳
发布于 2022-10-31 10:02:39
我也有同样的问题，所以我创建了一个函数来获取1000项并合并它！这是我的密码：
def count_data(tb_name: str, field_name: str):
return supabase.table(tb_name).select(field_name, count='exact').execute().count
def get_field_data(tb_name: str, src_field: str, len_record: int, id_field: str = 'id'):
if len_record <= 1000:
field_data = supabase.table(tb_name).select(id_field, src_field).order(
column=id_field).execute().data
else:
rnk = int(len_record / 1000)
field_data = []
for i in range(rnk):
min_rg = (i * 1000) + 1
max_rg = (i + 1) * 1000
field_data = field_data + supabase.table(tb_name).select(id_field, src_field).order(
column=id_field).range(min_rg - 1, max_rg).execute().data
field_data = field_data + supabase.table(tb_name).select(id_field, src_field).order(
column=id_field).range(max_rg, len_record).execute().data
return field_data页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/74186356
复制相关文章
相似问题
腾讯云开发者
