blocks|key|3034975|text|看起来pyparsing的人已经预料到你的问题了。来自https://github.com/pyparsing/pyparsing/blob/master/HowToUsePyparsing.rst：|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|3034976|对于复杂的语法和/或大型输入字符串，pyparsing的|style|CODE|3034977|性能可能会很慢。psyco包可用于在不更改语法或程序逻辑的情况下提高pyparsing模块的速度-观察到的改进在20-50%25的范围内。|blockquote|3034978|3034979|3034980|然而，正如万格尔在下面的评论中指出的那样，截至2012年3月，psyco是一个过时的项目。它的后继者是PyPy项目，该项目从相同的基本性能方法开始:使用JIT本机代码编译器而不是字节码解释器。如果切换PyPy实现对您有效，那么您应该能够获得类似或更大的收益。|3034981|如果你真的很喜欢速度，但又想保留一些易读性和声明性语法，我建议你看看ANTLR。可能不是Python生成后端；我怀疑它是否足够成熟或高性能来满足您的需求。我说的是好东西:开启这一切的C后端。|3034982|在解析器的入口点周围包装一个Python+C扩展模块，并使其松散。|3034983|话虽如此，在这个过渡过程中，您将放弃很多东西:基本上，您想在解析器中执行的任何Python都必须通过C+API完成(并不完全美观)。此外，你还必须习惯于完全不同的做事方式。ANTLR有它的魅力，但是它不是基于组合符的，所以你的语法和你的语言之间不存在像pyparsing那样简单和流畅的关系。另外，它有自己的DSL，很像lex/yacc，它可以呈现一个学习曲线-但是，因为它是基于LL的，你可能会发现它更容易适应你的需求。|3034984|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/pyparsing/pyparsing/blob/master/HowToUsePyparsing.rst|1|http://pypy.org/^0|R|20|0|0|I|9|0|8|5|Y|9|0|0|0|V|5|1F|4|1|0|0|0|0^^$0|@$1|2|3|4|5|6|7|14|8|@]|9|@$A|15|B|16|1|17]]|C|$]]|$1|D|3|E|5|6|7|18|8|@$A|19|B|1A|F|G]]|9|@]|C|$]]|$1|H|3|I|5|J|7|1B|8|@$A|1C|B|1D|F|G]|$A|1E|B|1F|F|G]]|9|@]|C|$]]|$1|K|3|-4|5|6|7|1G|8|@]|9|@]|C|$]]|$1|L|3|-4|5|6|7|1H|8|@]|9|@]|C|$]]|$1|M|3|N|5|6|7|1I|8|@$A|1J|B|1K|F|G]]|9|@$A|1L|B|1M|1|1N]]|C|$]]|$1|O|3|P|5|6|7|1O|8|@]|9|@]|C|$]]|$1|Q|3|R|5|6|7|1P|8|@]|9|@]|C|$]]|$1|S|3|T|5|6|7|1Q|8|@]|9|@]|C|$]]|$1|U|3|-4|5|6|7|1R|8|@]|9|@]|C|$]]]|V|$W|$5|X|Y|Z|C|$10|11]]|12|$5|X|Y|Z|C|$10|13]]]]

It looks like the pyparsing folks have anticipated your problem. From <a href="https://github.com/pyparsing/pyparsing/blob/master/HowToUsePyparsing.rst" rel="nofollow noreferrer">https://github.com/pyparsing/pyparsing/blob/master/HowToUsePyparsing.rst</a> :

<blockquote>
 Performance of <code>pyparsing</code> may be slow for complex grammars and/or large input strings. The <code>psyco</code> package can be used to improve the speed of the <code>pyparsing</code> module with no changes to grammar or program logic - observed improvments have been in the 20-50% range.
</blockquote>

However, as Vangel noted in the comments below, <code>psyco</code> is an obsolete project as of March 2012. Its successor is the <a href="http://pypy.org" rel="nofollow noreferrer" title="PyPy">PyPy</a> project, which starts from the same basic approach to performance: use a JIT native-code compiler instead of a bytecode interpreter. You should be able to achieve similar or greater gains with PyPy if switching Python implementations will work for you.

If you're really a speed demon, but want to keep some of the legibility and declarative syntax, I'd suggest having a look at ANTLR. Probably not the Python-generating backend; I'm skeptical whether that's mature or high-performance enough for your needs. I'm talking about the goods: the C backend that started it all.

Wrap a Python C extension module around the entry point to the parser, and turn it loose.

Having said that, you'll be giving up a lot in this transition: basically any Python you want to do in your parser will have to be done through the C API (not altogether pretty). Also, you'll have to get used to very different ways of doing things. ANTLR has its charms, but it's not based on combinators, so there's not the easy and fluid relationship between your grammar and your language that there is in pyparsing. Plus, it's its own DSL, much like lex/yacc, which can present a learning curve – but, because it's LL based, you'll probably find it easier to adapt to your needs.

blocks|key|306150|text|如果不对其进行测试，就无法知道您将获得什么样的好处，但是如果您的流程是长期运行和重复的，那么仅使用Unladen+Swallow就可以获得10倍的好处。(而且，如果你有很多东西要解析，并且你通常会为每个东西启动一个新的解释器，那么Unladen+Swallow在一定程度上会变得更快--你运行你的进程的时间越长，所以尽管解析一个输入可能不会显示出太多的收益，但在同一个进程中，你可能会在第二个和第三个输入上获得显著的收益)。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|306151|(注意:从SVN中提取最新版本--您将获得比最新tarball更好的性能)|306152|entityMap|0|LINK|mutability|MUTABLE|url|http://code.google.com/p/unladen-swallow/^0|1D|F|0|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@]|9|@$A|O|B|P|1|Q]]|C|$]]|$1|D|3|E|5|6|7|R|8|@]|9|@]|C|$]]|$1|F|3|-4|5|6|7|S|8|@]|9|@]|C|$]]]|G|$H|$5|I|J|K|C|$L|M]]]]

There's no way to know what kind of benefit you'll get without just testing it, but it's within the range of possibility that you could get 10x benefit just from using <a href="http://code.google.com/p/unladen-swallow/" rel="nofollow noreferrer">Unladen Swallow</a> if your process is long-running and repetitive. (Also, if you have many things to parse and you typically start a new interpreter for each one, Unladen Swallow gets faster - to a point - the longer you run your process, so while parsing one input might not show much gain, you might get significant gains on the 2nd and 3rd inputs in the same process).

(Note: pull the latest out of SVN - you'll get far better performance than the latest tarball)

blocks|key|3035021|text|切换到生成的C/C%2B%2B解析器(使用ANTLR、flex/bison等)。如果你可以延迟所有的动作规则，直到你完成解析之后，你也许能够用简单的代码构建一个AST，然后通过像SWIG这样的东西将它传递回你的python代码，并使用你当前的动作规则对其进行处理。OTOH，为了给你带来速度提升，解析必须是繁重的工作。如果你的动作规则是很大的代价，那么这不会给你带来任何好处，除非你也用C编写你的动作规则(但你可能无论如何都要这样做，以避免为你得到的python和C代码之间的阻抗不匹配买单)。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3035022|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

Switch to a generated C/C++ parser (using ANTLR, flex/bison, etc.). If you can delay all the action rules until after you are done parsing, you might be able to build an AST with trivial code and then pass that back to your python code via something like SWIG and process on it with your current actions rules. OTOH, for that to give you a speed boost, the parsing has to be the heavy lifting. If your action rules are the big cost, then this will buy you nothing unless you write your action rules in C as well (but you might have to do it anyway to avoid paying for whatever impedance mismatch you get between the python and C code).

blocks|key|3035057|text|如果您真的想要大型语法的性能，只需看看SimpleParse+(它本身依赖于mxTextTools，一个C扩展)。然而，现在要知道，它的代价是变得更加隐秘，并要求您精通EBNF。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|3035058|这绝对不是Pythonic式的路线，您必须从头开始使用EBNF语法才能使用SimpleParse。|3035059|entityMap|0|LINK|mutability|MUTABLE|url|http://simpleparse.sourceforge.net/|1|http://simpleparse.sourceforge.net/simpleparse_grammars.html^0|J|B|0|2C|4|1|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@$A|Q|B|R|1|S]|$A|T|B|U|1|V]]|C|$]]|$1|D|3|E|5|6|7|W|8|@]|9|@]|C|$]]|$1|F|3|-4|5|6|7|X|8|@]|9|@]|C|$]]]|G|$H|$5|I|J|K|C|$L|M]]|N|$5|I|J|K|C|$L|O]]]]

If you really want performance for large grammars, look no farther than <a href="http://simpleparse.sourceforge.net/" rel="nofollow noreferrer">SimpleParse</a> (which itself relies on mxTextTools, a C extension). However, know now that it comes at the cost of being more cryptic and requiring that you be well-versed in <a href="http://simpleparse.sourceforge.net/simpleparse_grammars.html" rel="nofollow noreferrer">EBNF</a>. 

It's definitely not the more Pythonic route, and you're going to have to start all over with an EBNF grammar to use SimpleParse.

blocks|key|3035108|text|参加聚会有点晚了，但是PLY+(Python+Lex-Yacc)，对我很有帮助。PLY为您提供了一个纯Python框架来构造基于lex的记号赋予器和基于yacc的LR+parsers。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|3035109|当我遇到pyparsing的性能问题时，我就走了这条路。|3035110|这是一篇关于Python解析的老文章，但仍然很有趣，其中包含了benchmarks+for+ANTLR,+PLY+and+pyparsing。在这个测试中，PLY大约比pyparsing快4倍。|3035111|entityMap|0|LINK|mutability|MUTABLE|url|http://www.dabeaz.com/ply/|1|https://en.wikipedia.org/wiki/LR_parser|2|http://www.dalkescientific.com/writings/diary/archive/2007/11/03/antlr_java.html^0|B|L|0|29|A|1|0|0|V|13|2|0^^$0|@$1|2|3|4|5|6|7|T|8|@]|9|@$A|U|B|V|1|W]|$A|X|B|Y|1|Z]]|C|$]]|$1|D|3|E|5|6|7|10|8|@]|9|@]|C|$]]|$1|F|3|G|5|6|7|11|8|@]|9|@$A|12|B|13|1|14]]|C|$]]|$1|H|3|-4|5|6|7|15|8|@]|9|@]|C|$]]]|I|$J|$5|K|L|M|C|$N|O]]|P|$5|K|L|M|C|$N|Q]]|R|$5|K|L|M|C|$N|S]]]]

A bit late to the party, but <a href="http://www.dabeaz.com/ply/" rel="nofollow noreferrer">PLY (Python Lex-Yacc)</a>, has served me very well. PLY gives you a pure Python framework for constructing lex-based tokenizers, and yacc-based <a href="https://en.wikipedia.org/wiki/LR_parser" rel="nofollow noreferrer">LR parsers</a>.

I went this route when I hit performance issues with pyparsing.

Here is a somewhat old but still interesting article on Python parsing which includes <a href="http://www.dalkescientific.com/writings/diary/archive/2007/11/03/antlr_java.html" rel="nofollow noreferrer">benchmarks for ANTLR, PLY and pyparsing</a>. PLY is roughly 4 times faster than pyparsing in this test.

I have a huge grammar developed for pyparsing as part of a large, pure Python application.
I have reached the limit of performance tweaking and I'm at the point where the diminishing returns make me start to look elsewhere. Yes, I think I know most of the tips and tricks and I've profiled my grammar and my application to dust.

What next? 

I hope to find a parser that gives me the same readability, usability (I'm using many advanced features of pyparsing such as parse-actions to start the post processing of the input which is being parsed) and python integration but at 10× the performance.

I love the fact the the grammar is pure Python.

All my basic blocks are regular expressions, so reusing them would be nice.

I know I can't have everything so I am willing to give up on some of the features I have today to get to the requested 10× performance.

Where do I go from here?

what next after pyparsing?

编译器

解释器

Python

作为一个大型纯Python应用程序的一部分，我为pyparsing开发了一个庞大的语法。我已经达到了性能调整的极限，我已经到了回报递减让我开始寻找其他地方的地步。是的，我想我知道大部分的技巧和技巧，我已经概述了我的语法和我的应用程序。下一步呢？我希望找到一个解析器，给我同样的可读性和可用性(我正在使用pyparsing...

问pyparsing之后的下一步是什么？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问pyparsing之后的下一步是什么？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问pyparsing之后的下一步是什么？
EN