blocks|key|4015660|text|对于所提供的记号赋予器类，这不是一个典型的场景。更像是一个成熟的解析器必须处理的东西。即使您需要手动构建这样的记号生成器，您也会发现研究由解析器生成器(如javacc或antlr+)生成的代码是很有教育意义的。专注于他们如何处理“前瞻”，这就是你在这里问的。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4015661|除非这是一个作业问题，不允许使用解析器生成器，否则使用解析器生成器会得到更好的结果。|4015662|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|F|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|G|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|H|8|@]|9|@]|A|$]]]|E|$]]

This isn't a typical scenario for the provided tokenizer classes. More like something that a fully-blown parser has to handle. Even if you need to build such a tokenizer by hand, you may find it educational to study code produced by parser generators such as javacc or antlr. Focus on how they handle "lookahead", which is what you are asking about here.

Unless this is a homework problem where you aren't allowed to use a parser generator, you will get better results by using one.

blocks|key|2329644|text|看起来StreamTokenizer有点太基础了。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2329645|我建议你在StreamTokenizer之上构建一个词法分析器。这个词法分析器所做的就是在通常意义上给你一个实际的令牌流。也就是说，<=将作为单个令牌给出，而不是两个单独的令牌。|2329646|更好的方法是，将StreamTokenizer绑定并编写一个只直接查看字符的lexer。StreamTokenizer所做的工作太少，对解析高级语法没有用处。|2329647|entityMap^0|3|F|0|1U|2|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@$9|M|A|N|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|O|8|@$9|P|A|Q|B|C]]|D|@]|E|$]]|$1|H|3|I|5|6|7|R|8|@]|D|@]|E|$]]|$1|J|3|-4|5|6|7|S|8|@]|D|@]|E|$]]]|K|$]]

It looks like <code>StreamTokenizer</code> is a bit on the basic side.

I'd recommend you build a lexer on top of StreamTokenizer. What this lexer would do is give you a stream of actual tokens in the usual sense. That is, <code>&lt;=</code> would be given as a single token, not two separate tokens.

Better still, bin StreamTokenizer and write a lexer that just looks at the characters directly. StreamTokenizer does too little to be useful for parsing an advanced grammar.

blocks|key|324092|text|StreamTokenizer为您提供的是基本词法分析器的功能。你必须使用这些来制作你的高端版本。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|324093|您必须非常明智地使用nextToken()和pushBack()。例如，在下面的示例中，我负责处理<、<<和<=。如果您看到操作符<，则向前查看流中的线索，如果没有找到后面的<或=，则将look+ahead标记推回到流中。|324094|>>示例代码|BOLD|324095|import+java.io.IOException;
import+java.io.StreamTokenizer;
import+java.io.StringReader;

public+class+LexerTest+
{
++++private+StringReader+r;

++++public+LexerTest(StringReader+stringReader)+{
++++++++r+=+stringReader;
++++}

++++public+static+void+main(String[]+args)+throws+IOException+
++++{
++++++++String+s+=+"test+=+test1+%2B+(test2+*+test3+*+(test4+-+2);";
++++++++new+LexerTest(new+StringReader(s)).printTokens();

++++++++System.out.println("\n###+Test+2+###\n");
++++++++s+=+"test+=+if(test1+<+test2){+test3+=+(test4+-+2);}";
++++++++new+LexerTest(new+StringReader(s)).printTokens();

++++++++System.out.println("\n###+Test+3+###\n");
++++++++s+=+"test+=+if(test1+<=+test2){+test3+=+(test4+-+2);}";
++++++++new+LexerTest(new+StringReader(s)).printTokens();

++++++++System.out.println("\n###+Test+4+###\n");
++++++++s+=+"test+=+if(test1+<+test2){+test3+=+(test4+<<+2);}";
++++++++new+LexerTest(new+StringReader(s)).printTokens();
++++}

++++private+void+printTokens()+throws+IOException+
++++{
++++++++StreamTokenizer+st+=+new+StreamTokenizer(r);
++++++++st.eolIsSignificant(true);

++++++++int+token+=+st.nextToken();
++++++++while+(token+!=+StreamTokenizer.TT_EOF)+
++++++++{
++++++++++++token+=+st.nextToken();
++++++++++++switch+(token)+
++++++++++++{
++++++++++++case+StreamTokenizer.TT_NUMBER:
++++++++++++++++double+num+=+st.nval;
++++++++++++++++System.out.println("Number+found:+"+%2B+num);
++++++++++++++++break;
++++++++++++case+StreamTokenizer.TT_WORD:
++++++++++++++++String+word+=+st.sval;
++++++++++++++++System.out.println("Word+found:+"+%2B+word);
++++++++++++++++break;
++++++++++++case+'%2B':
++++++++++++++++break;
++++++++++++case+'-':
++++++++++++++++break;
++++++++++++case+'/':
++++++++++++++++break;
++++++++++++case+'*':
++++++++++++++++break;
++++++++++++case+'<':
++++++++++++{
++++++++++++++++int+t+=+st.nextToken();
++++++++++++++++switch(t)
++++++++++++++++{
++++++++++++++++case+'=':
++++++++++++++++++++System.out.println("<=");
++++++++++++++++++++break;
++++++++++++++++case+'<':
++++++++++++++++++++System.out.println("<<");
++++++++++++++++++++break;
++++++++++++++++++++default:
++++++++++++++++++++++++st.pushBack();
++++++++++++++++++++++++System.out.println("<");
++++++++++++++++++++++++break;
++++++++++++++++}
++++++++++++}
++++++++++++}
++++++++}

++++}
}|code-block|syntax|javascript|324096|希望这能有所帮助。|324097|entityMap^0|0|F|0|A|B|M|A|1D|1|1F|2|1I|2|1T|1|2F|1|2H|1|0|0|6|0|0|0^^$0|@$1|2|3|4|5|6|7|T|8|@$9|U|A|V|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|W|8|@$9|X|A|Y|B|C]|$9|Z|A|10|B|C]|$9|11|A|12|B|C]|$9|13|A|14|B|C]|$9|15|A|16|B|C]|$9|17|A|18|B|C]|$9|19|A|1A|B|C]|$9|1B|A|1C|B|C]]|D|@]|E|$]]|$1|H|3|I|5|6|7|1D|8|@$9|1E|A|1F|B|J]]|D|@]|E|$]]|$1|K|3|L|5|M|7|1G|8|@]|D|@]|E|$N|O]]|$1|P|3|Q|5|6|7|1H|8|@]|D|@]|E|$]]|$1|R|3|-4|5|6|7|1I|8|@]|D|@]|E|$]]]|S|$]]

What <code>StreamTokenizer</code> provides you is the functionality of basic Lexer. You have to use these to make your high end version.

You have to make use of <code>nextToken()</code> and <code>pushBack()</code> very judiciously. For example in the below I am taking care of <code>&lt;</code>, <code>&lt;&lt;</code> and <code>&lt;=</code>. If you see an operator <code>&lt;</code> then look ahead in the stream for a clue and if you don't find a following <code>&lt;</code> or <code>=</code> then push back the look ahead token back into the stream.

>> Sample Code

<pre><code>import java.io.IOException;
import java.io.StreamTokenizer;
import java.io.StringReader;

public class LexerTest 
{
 private StringReader r;

 public LexerTest(StringReader stringReader) {
 r = stringReader;
 }

 public static void main(String[] args) throws IOException 
 {
 String s = "test = test1 + (test2 * test3 * (test4 - 2);";
 new LexerTest(new StringReader(s)).printTokens();

 System.out.println("\n### Test 2 ###\n");
 s = "test = if(test1 &lt; test2){ test3 = (test4 - 2);}";
 new LexerTest(new StringReader(s)).printTokens();

 System.out.println("\n### Test 3 ###\n");
 s = "test = if(test1 &lt;= test2){ test3 = (test4 - 2);}";
 new LexerTest(new StringReader(s)).printTokens();

 System.out.println("\n### Test 4 ###\n");
 s = "test = if(test1 &lt; test2){ test3 = (test4 &lt;&lt; 2);}";
 new LexerTest(new StringReader(s)).printTokens();
 }

 private void printTokens() throws IOException 
 {
 StreamTokenizer st = new StreamTokenizer(r);
 st.eolIsSignificant(true);

 int token = st.nextToken();
 while (token != StreamTokenizer.TT_EOF) 
 {
 token = st.nextToken();
 switch (token) 
 {
 case StreamTokenizer.TT_NUMBER:
 double num = st.nval;
 System.out.println("Number found: " + num);
 break;
 case StreamTokenizer.TT_WORD:
 String word = st.sval;
 System.out.println("Word found: " + word);
 break;
 case '+':
 break;
 case '-':
 break;
 case '/':
 break;
 case '*':
 break;
 case '&lt;':
 {
 int t = st.nextToken();
 switch(t)
 {
 case '=':
 System.out.println("&lt;=");
 break;
 case '&lt;':
 System.out.println("&lt;&lt;");
 break;
 default:
 st.pushBack();
 System.out.println("&lt;");
 break;
 }
 }
 }
 }

 }
}
</code></pre>

Hope this will help.

blocks|key|4015782|text|nextToken()将跳过空格，因此%2B%2B和%2B+%2B将被识别为相同的！|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|4015783|entityMap^0|0|B|J|2|M|3|0^^$0|@$1|2|3|4|5|6|7|H|8|@$9|I|A|J|B|C]|$9|K|A|L|B|C]|$9|M|A|N|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|O|8|@]|D|@]|E|$]]]|G|$]]

<code>nextToken()</code> will skip over whitespace so <code>++</code> and <code>+ +</code> will be recognized as the same!

blocks|key|1943797|text|StreamTokenizer是处理这种情况的非常基本的工具。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1943798|您可以创建自己的lookAhead函数来解决您的问题。|1943799|你读一个'<‘，然后调用你的前视，如果有'=’或没有-相应地采取行动|1943800|你可以使用堆栈来保存你之前的状态。|1943801|PS:对于更大的表达式，这会变得更加复杂，如果你想要更多的功能，你当然应该深入研究词法分析器和解析器|1943802|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|M|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|N|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|O|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|P|8|@]|9|@]|A|$]]|$1|J|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|K|$]]

StreamTokenizer is very basic tool for handling this.

You can create your own lookAhead function to solve your purpose.

You read a '&lt;' then call your lookahead and if there is a '=' or not 
- act accordingly

you could use a stack to save your previous state.

PS: This would get much more complicated with bigger expressions and If you want more functionality surely you should delve into lexers &amp; parsers

I'm writing a tokenizer in Java which has to deal with operators, and whitespace chars between tokens is not necessary.

I need to recognize something like "&lt;=" as a token, while also recognizing "&lt;" and "=".

Right now I have:

<pre><code>if (token == '&lt;')
 if (nextToken == '=')
 this.tokenList.add(27); // &lt;=
 else
 // add 2 tokens separately
</code></pre>

Is there anyway for StreamTokenizer to do this on its own? I've read through the API, but I don't see anything.

Can I specify combination of token that can be counted as one? Ideally, getNextToken would remove both token at once.

Thanks!

How to deal with operators while tokenizing in Java (StreamTokenizer)

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我正在用Java编写一个记号赋予器，它必须处理操作符，并且记号之间的空格字符不是必需的。我需要将"<=“之类的东西识别为标记，同时还需要识别"<”和"=“。现在我有：if (token == '<') if (nextToken == '=') this.tokenList.add(...

问在Java中如何在标记化时处理运算符(StreamTokenizer)
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在Java中如何在标记化时处理运算符(StreamTokenizer)EN