干货 | 一步步拆解 Elasticsearch BM25 模型评分细节

     特定分词单元 Term 出现次数 (Number of times term t appears in a document)
TF =  ---------------------------------------------- 
         所在文档 Terms 总数 (Total number of terms in the document)

                文档数(Total number of documents) 
IDF = log  ---------------------------------------
          包含特定分词单元 Term 的文档数 (Number of documents with term t in it)

DELETE got
PUT got
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "properties": {
      "quote": {
        "type": "text",
        "analyzer": "english"
      }
    }
  }
}

POST _bulk
{ "index" : { "_index" : "got", "_id" : "1" } }
{ "quote" : "A mind needs books as a sword needs a whetstone, if it is to keep its edge." }
{ "index" : { "_index" : "got", "_id" : "2" } }
{ "quote" : "Never forget what you are, for surely the world will not. Make it your strength. Then it can never be your weakness. Armor yourself in it, and it will never be used to hurt you." }
{ "index" : { "_index" : "got", "_id" : "3" } }
{ "quote" : "Let them see that their words can cut you, and you’ll never be free of the mockery. If they want to give you a name, take it, make it your own. Then they can’t hurt you with it anymore." }
{ "index" : { "_index" : "got", "_id" : "4" } }
{ "quote" : "When you play the game of thrones, you win or you die. There is no middle ground." }
{ "index" : { "_index" : "got", "_id" : "5" } }
{ "quote" : "The common people pray for rain, healthy children, and a summer that never ends. It is no matter to them if the high lords play their game of thrones, so long as they are left in peace." }
{ "index" : { "_index" : "got", "_id" : "6" } }
{ "quote" : "If you would take a man’s life, you owe it to him to look into his eyes and hear his final words. And if you cannot bear to do that, then perhaps the man does not deserve to die." }
{ "index" : { "_index" : "got", "_id" : "7" } }
{ "quote" : "Sorcery is the sauce fools spoon over failure to hide the flavor of their own incompetence." }
{ "index" : { "_index" : "got", "_id" : "8" } }
{ "quote" : "Power resides where men believe it resides. No more and no less." }
{ "index" : { "_index" : "got", "_id" : "9" } }
{ "quote" : "There’s no shame in fear, my father told me, what matters is how we face it." }
{ "index" : { "_index" : "got", "_id" : "10" } }
{ "quote" : "Love is poison. A sweet poison, yes, but it will kill you all the same." }
{ "index" : { "_index" : "got", "_id" : "11" } }
{ "quote" : "What good is this, I ask you? He who hurries through life hurries to his grave." }
{ "index" : { "_index" : "got", "_id" : "12" } }
{ "quote" : "Old stories are like old friends, she used to say. You have to visit them from time to time." }
{ "index" : { "_index" : "got", "_id" : "13" } }
{ "quote" : "The greatest fools are ofttimes more clever than the men who laugh at them." }
{ "index" : { "_index" : "got", "_id" : "14" } }
{ "quote" : "Everyone wants something, Alayne. And when you know what a man wants you know who he is, and how to move him." }
{ "index" : { "_index" : "got", "_id" : "15" } }
{ "quote" : "Always keep your foes confused. If they are never certain who you are or what you want, they cannot know what you are like to do next. Sometimes the best way to baffle them is to make moves that have no purpose, or even seem to work against you." }
{ "index" : { "_index" : "got", "_id" : "16" } }
{ "quote" : "One voice may speak you false, but in many there is always truth to be found." }
{ "index" : { "_index" : "got", "_id" : "17" } }
{ "quote" : "History is a wheel, for the nature of man is fundamentally unchanging." }
{ "index" : { "_index" : "got", "_id" : "18" } }
{ "quote" : "Knowledge is a weapon, Jon. Arm yourself well before you ride forth to battle." }
{ "index" : { "_index" : "got", "_id" : "19" } }
{ "quote" : "I prefer my history dead. Dead history is writ in ink, the living sort in blood." }
{ "index" : { "_index" : "got", "_id" : "20" } }
{ "quote" : "In the game of thrones, even the humblest pieces can have wills of their own. Sometimes they refuse to make the moves you’ve planned for them. Mark that well, Alayne. It’s a lesson that Cersei Lannister still has yet to learn." }
{ "index" : { "_index" : "got", "_id" : "21" } }
{ "quote" : "Every man should lose a battle in his youth, so he does not lose a war when he is old." }
{ "index" : { "_index" : "got", "_id" : "22" } }
{ "quote" : "A reader lives a thousand lives before he dies. The man who never reads lives only one." }
{ "index" : { "_index" : "got", "_id" : "23" } }
{ "quote" : "The fisherman drowned, but his daughter got Stark to the Sisters before the boat went down. They say he left her with a bag of silver and a bastard in her belly. Jon Snow, she named him, after Arryn." }
{ "index" : { "_index" : "got", "_id" : "24" } }
{ "quote" : "You could make a poultice out of mud to cool a fever. You could plant seeds in mud and grow a crop to feed your children. Mud would nourish you, where fire would only consume you, but fools and children and young girls would choose fire every time." }
{ "index" : { "_index" : "got", "_id" : "25" } }
{ "quote" : "Men live their lives trapped in an eternal present, between the mists of memory and the sea of shadow that is all we know of the days to come." }
{ "index" : { "_index" : "got", "_id" : "26" } }
{ "quote" : "No. Hear me, Daenerys Targaryen. The glass candles are burning. Soon comes the pale mare, and after her the others. Kraken and dark flame, lion and griffin, the sun’s son and the mummer’s dragon. Trust none of them. Remember the Undying. Beware the perfumed seneschal." }

GET got/_search
{
  "query": {
    "match": {
      "quote": "live"
    }
  }
}

GET got/_search
{
  "query": {
    "match": {
      "quote": "you"
    }
  },
  "explain": true
}

总评分 = 2.2 * 2.043074 * 0.74080354 = 3.3297362。

{
        "_shard" : "[got][0]",
        "_node" : "m9VCQfPDRqqMuupU_Xz5Eg",
        "_index" : "got",
        "_type" : "_doc",
        "_id" : "22",
        "_score" : 3.3297362,
        "_source" : {
          "quote" : "A reader lives a thousand lives before he dies. The man who never reads lives only one."
        },
        "_explanation" : {
          "value" : 3.3297362,
          "description" : "weight(quote:live in 21) [PerFieldSimilarity], result of:",
          "details" : [
            {
              "value" : 3.3297362,
              "description" : "score(freq=3.0), computed as boost * idf * tf from:",
              "details" : [
                {
                  "value" : 2.2,
                  "description" : "boost",
                  "details" : [ ]
                },
                {
                  "value" : 2.043074,
                  "description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                },
                {
                  "value" : 0.7408035,
                  "description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                }
              ]
            }
          ]
        }
      }

{
    "value":0.7408035,
    "description":"tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
    "details":[
        {
            "value":"3.0",
            "description":"freq, occurrences of term within document",
            "details":[

            ]
        },
        {
            "value":1.2,
            "description":"k1, term saturation parameter",
            "details":[

            ]
        },
        {
            "value":0.75,
            "description":"b, length normalization parameter",
            "details":[

            ]
        },
        {
            "value":"14.0",
            "description":"dl, length of field",
            "details":[

            ]
        },
        {
            "value":16.807692,
            "description":"avgdl, average length of field",
            "details":[

            ]
        }
    ]
}

POST got/_analyze
{
  "text": "A reader lives a thousand lives before he dies. The man who never reads lives only one", 
  "analyzer": "english"
}

TF = freq / (freq + k1 * (1 - b + b * dl / avgdl))  
= 3 / (3 + 1.2 *( 1 - 0.75 + 0.75 * 14 / 16.807692)) 
= 3 / (3 + 1.2 *0.87471397)
= 3/（3+1.049656764）
= 3/4.049656764
= 0.740803524

{
    "value":2.043074,
    "description":"idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
    "details":[
        {
            "value":3,
            "description":"n, number of documents containing term",
            "details":[

            ]
        },
        {
            "value":26,
            "description":"N, total number of documents with field",
            "details":[

            ]
        }
    ]
}

IDF = log(1 + (N - n + 0.5) / (n + 0.5)) 
= log( 1 + ( 26 - 3 + 0.5) / ( 3 + 0.5)) 
= log( 1 + 23.5/3.5) 
= log( 1 + 6.714285) 
= log(7.714285) 
= 2.043074

总评分 
= boost * TF * IDF 
= 2.2 * 0.74080354  * 2.043074 = 3.3297362