blocks|key|2485305|text|这不是一个解决方案，但我有一个解释。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2485306|Python3在标准库中支持sqlite+(稳定到pip知道并且不允许安装pysqlite)。但是，这个库需要在运行时在机器上安装sqlite开发人员工具(+libs)。默认情况下，Amazon的linux+AMI没有安装这些程序，而AWS+Lambda是在其上运行的(裸机ami实例)。不过，我不确定这是否意味着没有安装sqlite支持，或者只是在添加库之前无法工作，因为我测试的顺序不对。|2485307|Python2在标准库中不支持sqlite，你必须使用像pysqlite这样的第三方库来获得支持。这意味着可以更容易地构建二进制文件，而不依赖于机器状态或path变量。|2485308|我的建议是，如果可以的话，只在python2.7中运行该函数(并使您的单元测试变得更加困难:/)。|2485309|由于这些限制(它是在3中烘焙到python的基本库中的东西)，创建一个对lambda友好的部署包变得更加困难。我唯一能建议的就是申请AWS向lambda添加这种支持，或者(如果你可以不用实际使用nltk中的sqlite块)复制anaconda，将具有正确方法和属性的空库放入其中，但实际上什么也不做。|2485310|如果您对后者感兴趣，可以查看anaconda安装中的任何fake/_sqlite3文件。这样做只是为了避免导入错误。|offset|length|style|CODE|2485311|entityMap^0|0|0|0|0|0|S|D|0^^$0|@$1|2|3|4|5|6|7|R|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|S|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|T|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|U|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|V|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|W|8|@$L|X|M|Y|N|O]]|9|@]|A|$]]|$1|P|3|-4|5|6|7|Z|8|@]|9|@]|A|$]]]|Q|$]]

This isn't a solution, but I have an explanation why.

Python 3 has support for sqlite in the standard library (stable to the point of pip knowing and not allowing installation of pysqlite). However, this library requires the sqlite developer tools (C libs) to be on the machine at runtime. Amazon's linux AMI does not have these installed by default, which is what AWS Lambda runs on (naked ami instances). I'm not sure if this means that sqlite support isn't installed or just won't work until the libraries are added, though, because I tested things in the wrong order.

Python 2 does not support sqlite in the standard library, you have to use a third party lib like pysqlite to get that support. This means that the binaries can be built more easily without depending on the machine state or path variables. 

My suggestion, which you've already done I see, is to just run that function in python 2.7 if you can (and make your unit testing just that much harder :/).

Because of the limitations (it being something baked into python's base libs in 3) it is more difficult to create a lambda-friendly deployment package. The only thing I can suggest is to either petition AWS to add that support to lambda or (if you can get away without actually using the sqlite pieces in nltk) copying anaconda by putting blank libraries that have the proper methods and attributes but don't actually do anything.

If you're curious about the latter, check out any of the <code>fake/_sqlite3</code> files in an anaconda install. The idea is only to avoid import errors.

blocks|key|2485316|text|正如apathyman所描述的那样，直到亚马逊将sqlite3所需的C库捆绑到用于在lambda上运行Python的AMI中，才有一个直接的解决方案。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2485317|不过，一种解决方法是使用纯SQLite实现，如PyDbLite。这回避了问题，因为像这样的库不需要安装任何特定的C库，只需要安装Python即可。|2485318|不幸的是，如果您使用的是一个库，而这个库又使用了sqlite3模块，那么这对您没有什么帮助。|2485319|entityMap|0|LINK|mutability|MUTABLE|url|https://pydblite.readthedocs.io/en/latest/^0|O|7|0|N|8|0|0|O|7|0^^$0|@$1|2|3|4|5|6|7|R|8|@$9|S|A|T|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|U|8|@]|D|@$9|V|A|W|1|X]]|E|$]]|$1|H|3|I|5|6|7|Y|8|@$9|Z|A|10|B|C]]|D|@]|E|$]]|$1|J|3|-4|5|6|7|11|8|@]|D|@]|E|$]]]|K|$L|$5|M|N|O|E|$P|Q]]]]

As apathyman describes, there isn't a direct solution to this until Amazon bundle the C libraries required for <code>sqlite3</code> into the AMI's used to run Python on lambda.

One workaround though, is using a pure Python implementation of SQLite, such as <a href="https://pydblite.readthedocs.io/en/latest/" rel="nofollow noreferrer">PyDbLite</a>. This side-steps the problem, as a library like this doesn't require any particular C libraries to be installed, just Python.

Unfortunately, this doesn't help you if you are using a library which in turn uses the <code>sqlite3</code> module.

blocks|key|28300|text|根据您对NLTK所做的工作，我可能已经找到了一个解决方案。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|28301|基本的nltk模块导入了许多依赖项，其中许多并未被其功能集的大部分使用。在我的用例中，我只使用nltk.sent_tokenize，它对sqlite3没有函数依赖，即使sqlite3是作为依赖导入的。|offset|length|style|CODE|28302|我能够让我的代码在AWS+Lambda上运行|28303|import+nltk|code-block|syntax|javascript|28304|至|28305|import+imp
import+sys
sys.modules["sqlite"]+=+imp.new_module("sqlite")
sys.modules["sqlite3.dbapi2"]+=+imp.new_module("sqlite.dbapi2")
import+nltk|28306|这将为sqlite和sqlite.dbapi2动态创建空模块。当nltk.corpus.reader.panlex_lite尝试导入sqlite时，它将获得空模块，而不是标准库版本。这意味着导入将成功，但也意味着当nltk尝试使用sqlite模块时，它将失败。|28307|如果您使用的功能实际上依赖于sqlite，恐怕我无能为力。但是，如果您正在尝试使用其他nltk功能，并且只需要解决缺少sqlite的问题，则此技术可能会起作用。|28308|entityMap^0|0|1B|I|0|0|0|0|0|3|6|A|D|W|U|1U|6|0|0^^$0|@$1|2|3|4|5|6|7|Y|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Z|8|@$D|10|E|11|F|G]]|9|@]|A|$]]|$1|H|3|I|5|6|7|12|8|@]|9|@]|A|$]]|$1|J|3|K|5|L|7|13|8|@]|9|@]|A|$M|N]]|$1|O|3|P|5|6|7|14|8|@]|9|@]|A|$]]|$1|Q|3|R|5|L|7|15|8|@]|9|@]|A|$M|N]]|$1|S|3|T|5|6|7|16|8|@$D|17|E|18|F|G]|$D|19|E|1A|F|G]|$D|1B|E|1C|F|G]|$D|1D|E|1E|F|G]]|9|@]|A|$]]|$1|U|3|V|5|6|7|1F|8|@]|9|@]|A|$]]|$1|W|3|-4|5|6|7|1G|8|@]|9|@]|A|$]]]|X|$]]

Depending on what you're doing with NLTK, I may have found a solution. 

The base nltk module imports a lot of dependencies, many of which are not used by substantial portions of its feature set. In my use case, I'm only using the <code>nltk.sent_tokenize</code>, which carries no functional dependency on sqlite3 even though sqlite3 gets imported as a dependency.

I was able to get my code working on AWS Lambda by changing

<pre><code>import nltk
</code></pre>

to 

<pre><code>import imp
import sys
sys.modules["sqlite"] = imp.new_module("sqlite")
sys.modules["sqlite3.dbapi2"] = imp.new_module("sqlite.dbapi2")
import nltk
</code></pre>

This dynamically creates empty modules for <code>sqlite</code> and <code>sqlite.dbapi2</code>. When <code>nltk.corpus.reader.panlex_lite</code> tries to import <code>sqlite</code>, it will get our empty module instead of the standard library version. That means the import will succeed, but it also means that when nltk tries to use the sqlite module it will fail.

If you're using any functionality that actually depends on sqlite, I'm afraid I can't help. But if you're trying to use other nltk functionality and just need to get around the lack of sqlite, this technique might work.

blocks|key|2485336|text|这是一个小技巧，但我通过将CentOS+7上的Python3.6中的_sqlite3.so文件直接放到使用Zappa+to+AWS部署的项目的根目录中，从而使其正常工作。这应该意味着，如果您可以将_sqlite3.so直接包含到压缩包的根目录中，那么它应该可以工作，因此可以通过cpython中的以下行将其导入|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2485337|https://github.com/python/cpython/blob/3.6/Lib/sqlite3/dbapi2.py#L27|2485338|不是很漂亮，但很管用。您可以在此处找到_sqlite.so的副本：|2485339|https://github.com/Miserlou/lambda-packages/files/1425358/_sqlite3.so.zip|2485340|祝好运!|2485341|entityMap|0|LINK|mutability|MUTABLE|url|1^0|Y|B|2Q|B|3V|7|0|0|1W|0|0|J|A|0|0|21|1|0|0^^$0|@$1|2|3|4|5|6|7|V|8|@$9|W|A|X|B|C]|$9|Y|A|Z|B|C]|$9|10|A|11|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|12|8|@]|D|@$9|13|A|14|1|15]]|E|$]]|$1|H|3|I|5|6|7|16|8|@$9|17|A|18|B|C]]|D|@]|E|$]]|$1|J|3|K|5|6|7|19|8|@]|D|@$9|1A|A|1B|1|1C]]|E|$]]|$1|L|3|M|5|6|7|1D|8|@]|D|@]|E|$]]|$1|N|3|-4|5|6|7|1E|8|@]|D|@]|E|$]]]|O|$P|$5|Q|R|S|E|$T|G]]|U|$5|Q|R|S|E|$T|K]]]]

This is a bit of a hack, but I've gotten this working by dropping the <code>_sqlite3.so</code> file from Python 3.6 on CentOS 7 directly into the root of the project being deployed with Zappa to AWS. This should mean that if you can include <code>_sqlite3.so</code> directly into the root of your ZIP, it should work, so it can be imported by this line in <code>cpython</code>:

<a href="https://github.com/python/cpython/blob/3.6/Lib/sqlite3/dbapi2.py#L27" rel="noreferrer">https://github.com/python/cpython/blob/3.6/Lib/sqlite3/dbapi2.py#L27</a>

Not pretty, but it works. You can find a copy of <code>_sqlite.so</code> here:

<a href="https://github.com/Miserlou/lambda-packages/files/1425358/_sqlite3.so.zip" rel="noreferrer">https://github.com/Miserlou/lambda-packages/files/1425358/_sqlite3.so.zip</a>

Good luck!

blocks|key|2485343|text|我的解决方案可能适用于你，也可能不适用(因为它依赖于Python3.5)，但希望它能为类似的问题提供一些启发。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2485344|sqlite3附带了标准库，但不是使用亚马逊网络服务使用的python3.6构建的，原因由apathyman和其他答案解释。|offset|length|style|CODE|2485345|快速技巧是将共享对象.so包含到您的lambda包中：|2485346|find+~+-name+_sqlite3.so|2485347|在我的例子中：|2485348|/home/user/anaconda3/pkgs/python-3.5.2-0/lib/python3.5/lib-dynload/_sqlite3.so|2485349|然而，这还不够。您将获得：|2485350|ImportError:+libpython3.5m.so.1.0:+cannot+open+shared+object+file:+No+such+file+or+directory|2485351|因为_sqlite3.so是用python3.5构建的，所以它还需要python3.5share对象。在您的包部署中，您也将需要它：|2485352|find+~+-name+libpython3.5m.so*|2485353|2485354|/home/user/anaconda3/pkgs/python-3.5.2-0/lib/libpython3.5m.so.1.0|2485355|如果您使用的是用python3.6构建的_sqlite3.so，那么这个解决方案可能不起作用，因为由亚马逊网络服务构建的libpython3.6可能不支持这一点。然而，这只是我的教育猜测。如果任何人已经成功完成，请让我知道。|2485356|entityMap^0|0|0|7|19|9|0|A|3|0|0|O|0|0|0|26|0|0|0|2K|0|2|B|0|0|U|0|0|0|1T|0|K|B|0^^$0|@$1|2|3|4|5|6|7|14|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|15|8|@$D|16|E|17|F|G]|$D|18|E|19|F|G]]|9|@]|A|$]]|$1|H|3|I|5|6|7|1A|8|@$D|1B|E|1C|F|G]]|9|@]|A|$]]|$1|J|3|K|5|6|7|1D|8|@$D|1E|E|1F|F|G]]|9|@]|A|$]]|$1|L|3|M|5|6|7|1G|8|@]|9|@]|A|$]]|$1|N|3|O|5|6|7|1H|8|@$D|1I|E|1J|F|G]]|9|@]|A|$]]|$1|P|3|Q|5|6|7|1K|8|@]|9|@]|A|$]]|$1|R|3|S|5|6|7|1L|8|@$D|1M|E|1N|F|G]]|9|@]|A|$]]|$1|T|3|U|5|6|7|1O|8|@$D|1P|E|1Q|F|G]]|9|@]|A|$]]|$1|V|3|W|5|6|7|1R|8|@$D|1S|E|1T|F|G]]|9|@]|A|$]]|$1|X|3|M|5|6|7|1U|8|@]|9|@]|A|$]]|$1|Y|3|Z|5|6|7|1V|8|@$D|1W|E|1X|F|G]]|9|@]|A|$]]|$1|10|3|11|5|6|7|1Y|8|@$D|1Z|E|20|F|G]]|9|@]|A|$]]|$1|12|3|-4|5|6|7|21|8|@]|9|@]|A|$]]]|13|$]]

My solution may or may not apply to you (as it depends on Python 3.5), but hopefully it may shed some light for similar issue.

<code>sqlite3</code> comes with standard library, but is not built with the python3.6 that AWS use, with the reason explained by <code>apathyman</code> and other answers.

The quick hack is to include the share object <code>.so</code> into your lambda package:

<code>find ~ -name _sqlite3.so</code>

In my case:

<code>/home/user/anaconda3/pkgs/python-3.5.2-0/lib/python3.5/lib-dynload/_sqlite3.so</code>

However, that is not totally sufficient. You will get: 

<code>ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory</code>

Because the <code>_sqlite3.so</code> is built with python3.5, it also requires python3.5 share object. You will also need that in your package deployment:

<code>find ~ -name libpython3.5m.so*</code>

In my case:

<code>/home/user/anaconda3/pkgs/python-3.5.2-0/lib/libpython3.5m.so.1.0</code>

This solution is likely not work if you are using <code>_sqlite3.so</code> that is built with python3.6, because the libpython3.6 built by AWS will likely not support this. However, this is just my educational guess. If anyone has successfully done, please let me know.

blocks|key|2695904|text|根据AusIV的回答，这个版本适合我在AWS+Lambda和NLTK中工作，我创建了一个dummysqllite文件来模拟所需的引用。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2695905|spec+=+importlib.util.spec_from_file_location("_sqlite3","/dummysqllite.py")
sys.modules["_sqlite3"]+=+importlib.util.module_from_spec(spec)
sys.modules["sqlite3"]+=+importlib.util.module_from_spec(spec)
sys.modules["sqlite3.dbapi2"]+=+importlib.util.module_from_spec(spec)|code-block|syntax|javascript|2695906|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

From AusIV's answer, This version works for me in AWS Lambda and NLTK, I created a dummysqllite file to mock the required references. 

<pre><code>spec = importlib.util.spec_from_file_location("_sqlite3","/dummysqllite.py")
sys.modules["_sqlite3"] = importlib.util.module_from_spec(spec)
sys.modules["sqlite3"] = importlib.util.module_from_spec(spec)
sys.modules["sqlite3.dbapi2"] = importlib.util.module_from_spec(spec)
</code></pre>

blocks|key|27105|text|您需要sqlite3.so文件(正如其他人所指出的)，但是获取它的最健壮的方法是从(半官方的？)AWS+Lambda+docker镜像在lambci/lambda中提供。例如，对于Python+3.7，有一个简单的方法可以做到这一点：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|27106|首先，让我们从docker镜像中获取sqlite3.so+(库文件)：|27107|mkdir+lib
docker+run+-v+$PWD:$PWD+lambci/lambda:build-python3.7+bash+-c+"cp+sqlite3.cpython*.so+$PWD/lib/"|code-block|syntax|javascript|27108|接下来，我们将使用我们的需求和代码创建一个压缩的可执行文件：|27109|pip+install+-t+output+requirements.txt
pip+install+.+-t+output
zip+-r+output.zip+output|27110|最后，我们将库文件添加到映像中：|27111|cd+lib+&&+zip+-r+../output.zip+sqlite3.cpython*.so|27112|如果您想使用AWS+SAM构建/打包，请将其复制到lambda环境包的顶层(即，位于其他python文件旁边)。|27113|entityMap^0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|U|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|V|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|W|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|X|8|@]|9|@]|A|$]]|$1|K|3|L|5|F|7|Y|8|@]|9|@]|A|$G|H]]|$1|M|3|N|5|6|7|Z|8|@]|9|@]|A|$]]|$1|O|3|P|5|F|7|10|8|@]|9|@]|A|$G|H]]|$1|Q|3|R|5|6|7|11|8|@]|9|@]|A|$]]|$1|S|3|-4|5|6|7|12|8|@]|9|@]|A|$]]]|T|$]]

You need the sqlite3.so file (as others have pointed out), but the most robust way to get it is to pull from the (semi-official?) AWS Lambda docker images available in lambci/lambda. For example, for Python 3.7, here's an easy way to do this:

First, let's grab the sqlite3.so (library file) from the docker image:

<pre><code>mkdir lib
docker run -v $PWD:$PWD lambci/lambda:build-python3.7 bash -c "cp sqlite3.cpython*.so $PWD/lib/"
</code></pre>

Next, we'll make a zipped executable with our requirements and code:

<pre><code>pip install -t output requirements.txt
pip install . -t output
zip -r output.zip output
</code></pre>

Finally, we add the library file to our image:

<pre><code>cd lib &amp;&amp; zip -r ../output.zip sqlite3.cpython*.so
</code></pre>

If you want to use AWS SAM build/packaging, instead copy it into the top-level of the lambda environment package (i.e., next to your other python files).

blocks|key|2485391|text|TL;DR|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2485392|虽然不是一个很好的方法，但它工作得很好。使用pickle+module将所需的任何功能转储到使用本地系统的单独文件中。将该文件导出到AWS项目目录，从文件加载功能并使用它。|offset|length|style|CODE|2485393|在Long中|2485394|所以，这是我尝试过的，当我试图从nltk.corpus导入stopwords时，我遇到了同样的问题，但亚马逊网络服务不允许我导入它，甚至在Amazon+AMI上安装它似乎也不可能，因为同样的错误_sqlite3+module+not+found。为此，我尝试使用我的本地系统生成一个文件并将停用字转储到其中。下面是代码|2485395|#+Only+for+Local+System

from+nltk.corpus+import+stopwords
import+pickle

stop+=+list(stopwords.words('English'))
with+open('stopwords.pkl',+'wb')+as+f:
++++pickle.dump(stop,+f)|code-block|syntax|javascript|2485396|现在，我使用winscp将此文件导出到AWS+Directory，然后通过从我的项目上的文件加载它来使用该功能。|2485397|import+pickle

#+loading+trained+model
with+open('stopwords.pkl',+'rb')+as+f:
++++stop+=+pickle.load(f)|2485398|而且它工作得很好|2485399|entityMap^0|0|M|D|0|0|G|B|T|9|2P|P|0|0|6|6|0|0|0^^$0|@$1|2|3|4|5|6|7|Y|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Z|8|@$D|10|E|11|F|G]]|9|@]|A|$]]|$1|H|3|I|5|6|7|12|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|13|8|@$D|14|E|15|F|G]|$D|16|E|17|F|G]|$D|18|E|19|F|G]]|9|@]|A|$]]|$1|L|3|M|5|N|7|1A|8|@]|9|@]|A|$O|P]]|$1|Q|3|R|5|6|7|1B|8|@$D|1C|E|1D|F|G]]|9|@]|A|$]]|$1|S|3|T|5|N|7|1E|8|@]|9|@]|A|$O|P]]|$1|U|3|V|5|6|7|1F|8|@]|9|@]|A|$]]|$1|W|3|-4|5|6|7|1G|8|@]|9|@]|A|$]]]|X|$]]

<h1>TL;DR</h1>
Although not a very good approach but it works just fine. Use <code>pickle module</code> to dump whatever functionality you want in a separate file using you Local System. Export that file to AWS Project Directory, load functionality from file and use it.
<h1>In Long</h1>
So, here's what I tried, I had this same problem when I was trying to import <code>stopwords</code> from <code>nltk.corpus</code> but AWS doesn't let me import it, even installing it wasn't seems to be possible for me on Amazon AMI because of the same error <code>_sqlite3 module not found</code>. So to do that, what I tried was, using my local system I generated a file and dump stopwords into it. Here's the code
<pre><code># Only for Local System

from nltk.corpus import stopwords
import pickle

stop = list(stopwords.words('English'))
with open('stopwords.pkl', 'wb') as f:
 pickle.dump(stop, f)
</code></pre>
Now, I exported this file to AWS Directory using <code>winscp</code> and then used that functionality by loading it from the file on my project.
<pre class="lang-py prettyprint-override"><code>import pickle

# loading trained model
with open('stopwords.pkl', 'rb') as f:
 stop = pickle.load(f)
</code></pre>
and it works just fine

I am building a <code>python 3.6</code> AWS Lambda deploy package and was facing an issue with <code>SQLite</code>. 

In my code I am using <code>nltk</code> which has a <code>import sqlite3</code> in one of the files.

Steps taken till now:

<ol>
<li>Deployment package has only python modules that I am using in the root. I get the error:
<code>Unable to import module 'my_program': No module named '_sqlite3'</code></li>
<li>Added the _sqlite3.so from <code>/home/my_username/anaconda2/envs/py3k/lib/python3.6/lib-dynload/_sqlite3.so</code> into package root. Then my error changed to:

<code>Unable to import module 'my_program': dynamic module does not define module export function (PyInit__sqlite3)</code></li>
<li>Added the SQLite precompiled binaries from <code>sqlite.org</code> to the root of my package but I still get the error as point #2.</li>
</ol>

My setup: <code>Ubuntu 16.04</code>, <code>python3 virtual env</code>

AWS lambda env: <code>python3</code>

How can I fix this problem?

sqlite3 error on AWS lambda with Python 3

单元测试

Docker

SQLite

Python

CentOS

Ubuntu

我正在构建一个python 3.6 AWS Lambda部署包，并且遇到了SQLite的问题。在我的代码中，我使用的是nltk，其中一个文件中有一个import sqlite3。到目前为止采取的步骤：python 部署包中只有我在根目录中使用的python模块。我得到了错误:将_sqlite3.so从/home/my_...

问Python3在AWS lambda上出现sqlite3错误
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Python3在AWS lambda上出现sqlite3错误EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Python3在AWS lambda上出现sqlite3错误
EN