blocks|key|346821|text|您可能希望创建忽略型号的字母/数字组合的逻辑(因为它们几乎总是非常相似)。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|346822|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

You might want to create logic that ignores the letter/number combination of model numbers (since they're nigh always extremely similar).

blocks|key|348330|text|Spell+checking算法浮现在脑海中。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|348331|虽然我找不到一个好的示例实现，但我相信你可以修改一个基本的拼写检查算法来得到令人满意的结果。也就是说，将单词作为一个单元而不是一个字符来工作。|348332|留在我记忆中的点点滴滴：|348333|删除所有常用单词(a、an、、|348334|)。什么是“常见”取决于上下文。|ordered-list-item|348335|获取每个单词的第一个字母及其长度，并将其作为单词关键字。|348336|当出现可疑单词时，会查找具有相同或相似关键字的单词。|348337|348338|它可能不能直接解决你的问题...但你说你是在找点子，对吧？|348339|:-)|348340|entityMap|0|LINK|mutability|MUTABLE|url|http://en.wikipedia.org/wiki/Spell_checker^0|0|E|0|0|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|13|8|@]|9|@$A|14|B|15|1|16]]|C|$]]|$1|D|3|E|5|6|7|17|8|@]|9|@]|C|$]]|$1|F|3|G|5|6|7|18|8|@]|9|@]|C|$]]|$1|H|3|I|5|6|7|19|8|@]|9|@]|C|$]]|$1|J|3|K|5|L|7|1A|8|@]|9|@]|C|$]]|$1|M|3|N|5|L|7|1B|8|@]|9|@]|C|$]]|$1|O|3|P|5|L|7|1C|8|@]|9|@]|C|$]]|$1|Q|3|-4|5|6|7|1D|8|@]|9|@]|C|$]]|$1|R|3|S|5|6|7|1E|8|@]|9|@]|C|$]]|$1|T|3|U|5|6|7|1F|8|@]|9|@]|C|$]]|$1|V|3|-4|5|6|7|1G|8|@]|9|@]|C|$]]]|W|$X|$5|Y|Z|10|C|$11|12]]]]

<a href="http://en.wikipedia.org/wiki/Spell_checker" rel="nofollow noreferrer">Spell checking</a> algorithms come to mind.

Although I could not find a good sample implementation, I believe you can modify a basic spell checking algorithm to comes up with satisfactory results. i.e. working with words as a unit instead of a character.

The bits and pieces left in my memory:

<ol>
<li>Strip out all common words (a, an, the, new). What is "common" depends on context.</li>
<li>Take the first letter of each word and its length and make that an word key.</li>
<li>When a suspect word comes up, looks for words with the same or similar word key.</li>
</ol>

It might not solve your problems directly... but you say you were looking for ideas, right?

:-)

blocks|key|3126489|text|我认为，这将归结为区分、联想、等关键词和、等关键字。|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|3126490|我会对姓名数据库进行一些分析，以确定关键字。您可以使用类似于生成单词云的代码。|3126491|然后我会手动编辑列表，删除任何明显的废话，比如也许New实际上是通用的，但不是key。|3126492|然后，您将有一个关键字列表，可以用来帮助识别相似性。您可以将“原始”名称与其关键字相关联，并在比较两个或更多原始名称的相似性时使用这些关键字(字面意思是共享关键字的百分比)。|3126493|无论如何，这都不是一个完美的解决方案，但我不认为你在期待一个解决方案？|3126494|entityMap^0|B|4|K|1|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@$9|Q|A|R|B|C]|$9|S|A|T|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|U|8|@]|D|@]|E|$]]|$1|H|3|I|5|6|7|V|8|@]|D|@]|E|$]]|$1|J|3|K|5|6|7|W|8|@]|D|@]|E|$]]|$1|L|3|M|5|6|7|X|8|@]|D|@]|E|$]]|$1|N|3|-4|5|6|7|Y|8|@]|D|@]|E|$]]]|O|$]]

I think this will boil down to distinguishing key words such as Lenovo from chaff such as New.

I would run some analysis over the database of names to identify key words. You could use code similar to that used to generate a word cloud. 

Then I would hand-edit the list to remove anything obviously chaff, like maybe New is actually common but not key.

Then you will have a list of key words that can be used to help identify similarities. You would associate the "raw" name with its keywords, and use those keywords when comparing two or more raw names for similarities (literally, percentage of shared keywords).

Not a perfect solution by any stretch, but I don't think you are expecting one?

blocks|key|348354|text|我对这类问题没有任何经验，但我认为一个非常幼稚的实现是对搜索项进行标记化，并搜索恰好包含任何标记的匹配项。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|348355|例如，"Canon+PowerShot+A20+IS“标记为：|348356|348357|Canon|unordered-list-item|348358|Powershot|348359|A20|348360|IS|348361|348362|这将与您希望在结果中显示的其他每个项目相匹配。当然，这种策略也可能会产生大量的错误匹配。|348363|另一种策略是将“关键字”与每个项目一起存储，如“相机”、“佳能”、“数码相机”，并基于具有匹配关键字的项目进行搜索。此外，如果您存储了其他属性，如Maker、Brand等，则可以在每个属性上进行搜索。|348364|entityMap^0|0|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|U|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|V|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|W|8|@]|9|@]|A|$]]|$1|E|3|F|5|G|7|X|8|@]|9|@]|A|$]]|$1|H|3|I|5|G|7|Y|8|@]|9|@]|A|$]]|$1|J|3|K|5|G|7|Z|8|@]|9|@]|A|$]]|$1|L|3|M|5|G|7|10|8|@]|9|@]|A|$]]|$1|N|3|-4|5|6|7|11|8|@]|9|@]|A|$]]|$1|O|3|P|5|6|7|12|8|@]|9|@]|A|$]]|$1|Q|3|R|5|6|7|13|8|@]|9|@]|A|$]]|$1|S|3|-4|5|6|7|14|8|@]|9|@]|A|$]]]|T|$]]

Not having any experience with this type of problem, but I think a very naive implementation would be to tokenize the search term, and search for matches that happen to contain any of the tokens.

"Canon PowerShot A20 IS", for example, tokenizes into:

<ul>
<li>Canon</li>
<li>Powershot</li>
<li>A20</li>
<li>IS</li>
</ul>

which would match each of the other items you want to show up in the results. Of course, this strategy will likely produce a whole lot of false matches as well.

Another strategy would be to store "keywords" with each item, such as "camera", "canon", "digital camera", and searching based on items that have matching keywords. In addition, if you stored other attributes such as Maker, Brand, etc., you could search on each of these.

blocks|key|3339380|text|我认为，edg的答案是正确的--你需要区分关键词和模糊的东西。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3339381|上下文很重要。举个例子，Core2Duo在查看T400的两个实例时是模糊的，但在查看CPU包时就不是这样了。|3339382|如果您可以在数据库中标记产品名称的规范形式的哪些部分更重要，并且必须以一种或另一种形式出现以标识产品，那么您应该这样做。也许是通过使用某种语义标记？你能负担得起让人类标记数据库的费用吗？|3339383|您可以尝试为"T-400“、"T400”、“T400”等定义等价类。也许可以定义一组规则，说明“数字比附加在这些数字上的字母更紧密地绑定在一起”。|3339384|根据制造商、型号等拆分案例可能是一种不错的方法。我建议您查看术语查找的技术来尝试实现这一点：http://www.worldcat.org/isbn/9780262100854|offset|length|3339385|在一个灵活的框架中设计一切，这个框架主要是规则驱动的，其中规则可以根据你的需要和新出现的坏模式(即:破坏你的算法的东西)进行修改，这也是一个好主意。这样，您就能够基于真实世界的数据来提高系统的性能。|3339386|entityMap|0|LINK|mutability|MUTABLE|url|http://www.worldcat.org/isbn/9780262100854^0|0|0|0|0|1A|16|0|0|0^^$0|@$1|2|3|4|5|6|7|V|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|W|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|X|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|Y|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|Z|8|@]|9|@$J|10|K|11|1|12]]|A|$]]|$1|L|3|M|5|6|7|13|8|@]|9|@]|A|$]]|$1|N|3|-4|5|6|7|14|8|@]|9|@]|A|$]]]|O|$P|$5|Q|R|S|A|$T|U]]]]

edg's answer is in the right direction, I think - you need to distinguish key words from fluff.

Context matters. To take your example, Core 2 Duo is fluff when looking at two instances of a T400, but not when looking at a a CPU OEM package.

If you can mark in your database which parts of the canonical form of a product name are more important and must appear in one form or another to identify a product, you should do that. Maybe through the use of some sort of semantic markup? Can you afford to have a human mark up the database?

You can try to define equivalency classes for things like "T-400", "T400", "T 400" etc. Maybe a set of rules that say "numbers bind more strongly than letters attached to those numbers." 

Breaking down into cases based on manufacturer, model number, etc. might be a good approach. I would recommend that you look at techniques for term spotting to try and accomplish that: <a href="http://www.worldcat.org/isbn/9780262100854" rel="nofollow noreferrer">http://www.worldcat.org/isbn/9780262100854</a>

Designing everything in a flexible framework that's mostly rule driven, where the rules can be modified based on your needs and emerging bad patterns (read: things that break your algorithm) would be a good idea, as well. This way you'd be able to improve the system's performance based on real world data.

blocks|key|3339429|text|这正是我在业余时间解决的问题。我的想法是:基于关键字缩小搜索范围：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3339430|在这种情况下，你可以有一些层次结构：|3339431|类型-->公司-->型号|3339432|这样就可以将"Digital+Camera“匹配到一个类型|3339433|“佳能”的公司，在那里你会留下更窄的范围来搜索。|3339434|你可以通过引入产品线等方式进一步降低这一点，但主要的一点是，这可能必须迭代完成。|3339435|entityMap^0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|O|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|P|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|Q|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|R|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|S|8|@]|9|@]|A|$]]|$1|L|3|-4|5|6|7|T|8|@]|9|@]|A|$]]]|M|$]]

That is exactly the problem I'm working on in my spare time. What I came up with is:
based on keywords narrow down the scope of search:

in this case you could have some hierarchy:

type --> company --> model

so that you'd match
"Digital Camera" for a type

"Canon" for company and there you'd be left with much narrower scope to search.

You could work this down even further by introducing product lines etc.
But the main point is, this probably has to be done iteratively.

blocks|key|349253|text|为此，您可能能够使用trigram+search。我必须承认，我从来没有见过实现索引的算法，但我见过它在制药应用程序中工作，在那里它确实很好地处理了严重拼写错误的药品名称。您也许能够将相同类型的逻辑应用于此问题。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|349254|entityMap|0|LINK|mutability|MUTABLE|url|http://en.wikipedia.org/wiki/Trigram_search^0|A|E|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

You might be able to make use of a <a href="http://en.wikipedia.org/wiki/Trigram_search" rel="nofollow noreferrer">trigram search</a> for this. I must admit I've never seen the algorithm to implement an index, but have seen it working in pharmaceutical applications, where it copes very well indeed with badly misspelt drug names. You might be able to apply the same kind of logic to this problem.

blocks|key|3126667|text|这里的关键理解是，您确实有一个适当的距离度量。事实上，这根本不是你的问题。你的问题出在分类上。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3126668|让我给你举个例子。假设您有20个条目用于Foo+X1，20个条目用于Foo+Y1。您可以放心地假设它们是两个组。另一方面，如果条形图X1有39个条目，条形图Y1有1个条目，则应将它们视为一个单独的组。|3126669|现在，距离X1+<->+Y1在两个示例中是相同的，那么为什么分类会有差异？这是因为Bar+Y1是一个异常值，而Foo+Y1不是。|3126670|有趣的是，您实际上不需要做大量的工作来预先确定这些组。你只需要做一个递归分类。您从每个组的节点开始，然后为最近的两个节点添加一个超级节点。在超节点中，存储最佳假设、其子树的大小和其中的变化。由于您的许多字符串将是相同的，您很快就会得到具有相同条目的大型子树。递归以包含在树的根部的超级节点结束。|3126671|现在将规范名称映射到此树上。您很快就会看到，每一个都将匹配整个子树。现在，使用这些树之间的距离来拾取该条目的距离截止点。如果数据库中同时有Foo+X1和Foo+Y1产品，则需要降低截止距离以反映这一点。|3126672|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|M|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|N|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|O|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|P|8|@]|9|@]|A|$]]|$1|J|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|K|$]]

The key understanding here is that you do have a proper distance metric. That is in fact not your problem at all. Your problem is in classification. 

Let me give you an example. Say you have 20 entries for the Foo X1 and 20 for the Foo Y1. You can safely assume they are two groups. On the other hand, if you have 39 entries for the Bar X1 and 1 for the Bar Y1, you should treat them as a single group. 

Now, the distance X1 &lt;-> Y1 is the same in both examples, so why is there a difference in the classification? That is because Bar Y1 is an outlier, whereas Foo Y1 isn't. 

The funny part is that you do not actually need to do a whole lot of work to determine these groups up front. You simply do an recursive classification. You start out with node per group, and then add the a supernode for the two closest nodes. In the supernode, store the best assumption, the size of its subtree and the variation in it. As many of your strings will be identical, you'll soon get large subtrees with identical entries. Recursion ends with the supernode containing at the root of the tree.

Now map the canonical names against this tree. You'll quickly see that each will match an entire subtree. Now, use the distances between these trees to pick the distance cutoff for that entry. If you have both Foo X1 and Foo Y1 products in the database, the cut-off distance will need to be lower to reflect that.

blocks|key|349285|text|我们可以使用Datadecision+service来匹配产品。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|349286|它将允许您使用统计算法自动匹配您的产品数据。此操作是在定义置信度的阈值得分之后完成的。|349287|所有无法自动匹配的数据都必须通过专用用户界面进行手动审查。|349288|在线服务使用查找表来存储同义词以及手动匹配历史记录。这允许您在下次导入新数据时提高数据匹配的自动化程度。|349289|entityMap|0|LINK|mutability|MUTABLE|url|http://www.datadecision.com/^0|6|K|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|R|8|@]|9|@$A|S|B|T|1|U]]|C|$]]|$1|D|3|E|5|6|7|V|8|@]|9|@]|C|$]]|$1|F|3|G|5|6|7|W|8|@]|9|@]|C|$]]|$1|H|3|I|5|6|7|X|8|@]|9|@]|C|$]]|$1|J|3|-4|5|6|7|Y|8|@]|9|@]|C|$]]]|K|$L|$5|M|N|O|C|$P|Q]]]]

We can use the <a href="http://www.datadecision.com" rel="nofollow" title="data matching software by Datadecision">Datadecision service</a> for matching products. 

It will allow you to automatically match your product data using statistical algorithms. This operation is done after defining a threshold score of confidence.

All data that cannot be automatically matched will have to be manually reviewed through a dedicated user interface.

The online service uses lookup tables to store synonyms as well as your manual matching history. This allows you to improve the data matching automation next time you import new data.

blocks|key|3126713|text|这是record+linkage的问题。dedupe+python+library提供了一个完整的实现，但是即使您不使用python，文档中也有一个good+overview+of+how+to+approach+this+problem。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|3126714|简而言之，在标准范例中，此任务分为三个阶段|3126715|3126716|比较字段，在本例中仅比较名称。为此，您可以使用一个或多个比较器，例如，编辑距离(如Levenshtein距离)或余弦距离(用于比较常用单词的数量)。|ordered-list-item|3126717|将距离分数数组转换为一对记录确实与同一事物有关的概率|3126718|将这些成对的概率分数聚类为多组记录，这些记录可能都指向同一事物。|3126719|3126720|entityMap|0|LINK|mutability|MUTABLE|url|https://en.wikipedia.org/wiki/Record_linkage|1|https://github.com/datamade/dedupe|2|https://dedupe.io/developers/library/en/latest/How-it-works.html^0|2|E|0|K|L|1|22|19|2|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|10|8|@]|9|@$A|11|B|12|1|13]|$A|14|B|15|1|16]|$A|17|B|18|1|19]]|C|$]]|$1|D|3|E|5|6|7|1A|8|@]|9|@]|C|$]]|$1|F|3|-4|5|6|7|1B|8|@]|9|@]|C|$]]|$1|G|3|H|5|I|7|1C|8|@]|9|@]|C|$]]|$1|J|3|K|5|I|7|1D|8|@]|9|@]|C|$]]|$1|L|3|M|5|I|7|1E|8|@]|9|@]|C|$]]|$1|N|3|-4|5|6|7|1F|8|@]|9|@]|C|$]]|$1|O|3|-4|5|6|7|1G|8|@]|9|@]|C|$]]]|P|$Q|$5|R|S|T|C|$U|V]]|W|$5|R|S|T|C|$U|X]]|Y|$5|R|S|T|C|$U|Z]]]]

This is a problem of <a href="https://en.wikipedia.org/wiki/Record_linkage" rel="nofollow noreferrer">record linkage</a>. The <a href="https://github.com/datamade/dedupe" rel="nofollow noreferrer">dedupe python library</a> provides a complete implementation, but even if you don't use python, the documentation has a <a href="https://dedupe.io/developers/library/en/latest/How-it-works.html" rel="nofollow noreferrer">good overview of how to approach this problem</a>.

Briefly, within the standard paradigm, this task is broken into three stages

<ol>
<li>Compare the fields, in this case just the name. You can use one or more comparator for this, for example an edit distance like the Levenshtein distance or something like the cosine distance that compares the number of common words.</li>
<li>Turn an array fo distance scores into a probability that a pair of records are truly about the same thing</li>
<li>Cluster those pairwise probability scores into groups of records that likely all refer to the same thing.</li>
</ol>

blocks|key|3126743|text|我以前也做过同样的事情。我所做的是使用NLP方法；TF-IDF向量器为每个单词分配权重。例如，在您的案例中：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3126744|佳能PowerShot+a20IS|3126745|weight标准-->+weight+=+0.05+(不是很有区别的word)|3126746|PowerShot+-->+weight+=+0.37+(可以是distinguishing)|unordered-list-item|3126747|
|3126748|a20IS+-->+weight=+0.96+(very+distinguishing)|3126749|3126750|这将告诉你的模型应该关注哪些单词，而不关注哪些单词。多亏了TF-IDF，我有了很好的比赛。但请注意:+a20IS不能像a20那样被识别，您可以考虑使用某种正则表达式来过滤这种情况。|3126751|在此之后，您可以使用像余弦相似度这样的数值计算。|3126752|entityMap^0|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|T|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|U|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|V|8|@]|9|@]|A|$]]|$1|F|3|G|5|H|7|W|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|X|8|@]|9|@]|A|$]]|$1|K|3|L|5|H|7|Y|8|@]|9|@]|A|$]]|$1|M|3|-4|5|6|7|Z|8|@]|9|@]|A|$]]|$1|N|3|O|5|6|7|10|8|@]|9|@]|A|$]]|$1|P|3|Q|5|6|7|11|8|@]|9|@]|A|$]]|$1|R|3|-4|5|6|7|12|8|@]|9|@]|A|$]]]|S|$]]

I worked on the exact same thing in the past. What I have done is using an NLP method; TF-IDF Vectorizer to assign weights to each word. For example in your case:
Canon PowerShot a20IS
<ul>
<li>Canon --&gt; weight = 0.05 (not a very distinguishing word)</li>
<li>PowerShot --&gt; weight = 0.37 (can be distinguishing)</li>
<li>a20IS --&gt; weight = 0.96 (very distinguishing)</li>
</ul>
This will tell your model which words to care and which words to not. I had quite good matches thanks to TF-IDF.
But note this: a20IS cannot be recognized as a20 IS, you may consider to use some kind of regex to filter such cases.
After that, you can use a numeric calculation like cosine similarity.

I need to automatically match product names (cameras, laptops, tv-s etc) that come from different sources to a canonical name in the database.

For example "Canon PowerShot a20IS", "NEW powershot A20 IS from Canon" and "Digital Camera Canon PS A20IS" 
should all match "Canon PowerShot A20 IS". I've worked with levenshtein distance with some added heuristics (removing obvious common words, assigning higher cost to number changes etc), which works to some extent, but not well enough unfortunately.

The main problem is that even single-letter changes in relevant keywords can make a huge difference, but it's not easy to detect which are the relevant keywords. Consider for example three product names: 
Lenovo T400 
Lenovo R400 
New Lenovo T-400, Core 2 Duo 
The first two are ridiculously similar strings by any standard (ok, soundex might help to disinguish the T and R in this case, but the names might as well be 400T and 400R), the first and the third are quite far from each other as strings, but are the same product.

Obviously, the matching algorithm cannot be a 100% precise, my goal is to automatically match around 80% of the names with a high confidence.

Any ideas or references is much appreciated

Fuzzy matching of product names

数据库

我需要自动匹配来自不同来源的产品名称(相机，笔记本电脑，电视等)到数据库中的规范名称。例如，佳能PowerShot a20IS，新的powershot A20是从佳能和数码相机佳能PS A20IS应该都匹配佳能PowerShot A20是。我已经使用levenshtein distance添加了一些启发式方法(删除明显...

问产品名称的模糊匹配
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问产品名称的模糊匹配EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问产品名称的模糊匹配
EN