我想把这个JSON
{"api":"POST /n2mt/translateLog","n2mt":true,"params":{"deviceId":"D4E3-4239-9188-92E67D246D13","caller":"NID.IOS","source":"ko","target":"ja","text":"넌 친구랑도 금방 친해질 것 같아","dict":true,"dictDisplay":3,"dictHl":"all","honorific":false,"instant":false,"agree":false,"domainType":"","splitSents":false,"sessionId":"9345-41B9-9632-94852823146E","n2mtResut":null},"result":{"json":{"message":{"result":{"srcLangType":"ko","tarLangType":"ja","translatedText":"あなたは友達ともすぐ親しくなると思う。","engineType":"N2MT","pivot":null,"delay":400,"delaySmt":400},"@type":"response","@service":"proxy","@version":"1.0.0"}},"elapsed":0,"textLen":18},"langDetect":"ko","modelVer":"1.2.12"}
基于Scala的星火流对弹性搜索的post请求
我得到了一个
{"error":{"root_cause":[{"type":"not_x_content_exception","reason":"not_x_content_exception: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"not_x_content_exception","reason":"not_x_content_exception: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}},"status":400}
如果我复制粘贴字符串并使用Postman发送请求,它将成功。我认为错误信息是说字符串的结构有问题,但我并不完全是问题所在。
代码如下所示
def makeHttpCall(row: String): Unit = {
val client = HttpClientBuilder.create().setDefaultCredentialsProvider(credentialsProvider).build()
val post = new HttpPost(s"$elasticsearchHost:$elasticsearchPort/$indexAndDocType")
post.addHeader(HttpHeaders.CONTENT_TYPE, "application/json;charset=UTF-8")
val nameValuePairs = new ArrayList[NameValuePair]()
nameValuePairs.add(new BasicNameValuePair("JSON", row))
post.setEntity(new UrlEncodedFormEntity(nameValuePairs))
val response = client.execute(post)
val entity = response.getEntity()
println(Seq(response.getStatusLine.getStatusCode(), response.getStatusLine.getReasonPhrase()))
val responseAsString = EntityUtils.toString(response.getEntity())
println(responseAsString)
}
resultDF.writeStream
.option("truncate", false)
.option("encoding", "UTF-8")
.outputMode(outputMode)
.foreachBatch { (batchDF: DataFrame, batchId: Long) =>
batchDF.persist()
val result = batchDF
.select("json")
.collect()
.map(_.getString(0))
.mkString("\n")
makeHttpCall(result)
println(result)
batchDF.unpersist()
()
}
我不喜欢makeHttpCall之前的collect(),任何改进代码的建议都会很有帮助
我不能使用ES-Hadoop,因为集群的版本是7.10
发布于 2022-06-20 03:49:05
我想出来了
将字符串作为StringEntity而不是UrlEncodedFormEntity发送,并设置新的StringEntity( string,"UTF-8")
https://stackoverflow.com/questions/72657181
复制相似问题