【Rust日报】 2020-01-11 postgres-query, 新的分词器库tokenizers

MikeLoveRust

发布于 2020-02-12 15:05:41

5720

发布于 2020-02-12 15:05:41

文章被收录于专栏：Rust语言学习交流Rust语言学习交流

postgres-query

这个crate提供了方便的宏和trait，可帮助编写SQL查询并将其结果收集到静态类型的结构中。

示例代码：

// Connect to the database
let client: Client = connect(/* ... */);

// Construct the query
let query = query!(
    "SELECT name, age FROM people WHERE age >= $min_age",
    min_age = 18
);

// Define the structure of the data returned from the query
#[derive(FromSqlRow)]
struct Person {
    age: i32,
    name: String,
}

// Execute the query
let people: Vec<Person> = query.fetch(&client).await?;

// Use the results
for person in people {
    println!("{} is {} years young", person.name, person.age);
}

Github

新的分词器库tokenizers

分词器的核心是用Rust编写的。提供当今最常用的分词器的实现，重点是性能和多功能性。

示例代码

use tokenizers::tokenizer::{Result, Tokenizer, EncodeInput};
use tokenizers::models::bpe::BPE;

fn main() -> Result<()> {
let bpe_builder = BPE::from_files("./path/to/vocab.json", "./path/to/merges.txt")?;
let bpe = bpe_builder
		.dropout(0.1)
		.unk_token("[UNK]".into())
		.build()?;

let mut tokenizer = Tokenizer::new(Box::new(bpe));

let encoding = tokenizer.encode(EncodeInput::Single("Hey there!".into()))?;
println!("{:?}", encoding.get_tokens());

Ok(())
}

Github