一个系统最初的线上业务量并不会很大,比如说单库的数据量在百万级别以下(事实上千万级别以下都还能支撑),那么MySQL的单库即可完成任何增/删/改/查的业务操作。随着业务的发展,单个DB中保存的数据量(用户、订单、计费明细和权限规则等数据)呈现指数级增长,那么各种业务处理操作都会面临单DB的IO读写瓶颈带来的性能问题。
Sharding-JDBC分库分表就是其中一个解决方法,目前用的还挺广泛的,虽然还是有蛮多问题,但是对于公司的普通应用已经足够了。
其实,如果仅仅是分表,Mybatis等中间件就可以帮我们实现简单分表功能,不需要使用Sharding-JDBC,但是Sharding-JDBC可以支持分库,而且支持分库的本地事务(弱事务):
Sharding-JDBC本地事务官方说明:
在《Spring整合Sharding-JDBC分库分表详情》一篇中,已经使用Spring mvc 整合Spring-data-jpa、Sharding-JDBC 进行分库分表操作;这一篇将使用SpringBoot与Mybatis整合Sharding-JDBC 进行分库分表操作。
本文假设你已经引入spring-boot-starter-web。已经是个SpringBoot项目了,如果不会搭建,可以打开这篇文章看一看《SpringBoot入门建站全系列(一)项目建立》。
本篇使用Mybatis做数据库访问,并整合Sharding-JDBC。
需要引入mybatis-spring-boot-starter,这里要访问数据库进行操作,所以要依赖数据库相关jar包。还要引入sharding-jdbc-spring-boot-starter。
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.mybatis.spring.boot</groupId> <artifactId>mybatis-spring-boot-starter</artifactId> </dependency> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-dbcp2</artifactId> </dependency> <dependency> <groupId>org.apache.shardingsphere</groupId> <artifactId>sharding-jdbc-spring-boot-starter</artifactId> <version>${sharding-sphere.version}</version> </dependency>
在application.properties 中需要添加下面的配置:
worker.id=1 spring.shardingsphere.datasource.names=master,slave spring.shardingsphere.datasource.master.type=org.apache.commons.dbcp2.BasicDataSource spring.shardingsphere.datasource.master.driver-class-name=com.mysql.jdbc.Driver spring.shardingsphere.datasource.master.url=jdbc:mysql://127.0.0.1:3306/boot?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC spring.shardingsphere.datasource.master.username=cff spring.shardingsphere.datasource.master.password=123456 spring.shardingsphere.datasource.master.max-wait-millis=60000 spring.shardingsphere.datasource.master.min-idle=20 spring.shardingsphere.datasource.master.initial-size=2 spring.shardingsphere.datasource.master.validation-query=SELECT 1 spring.shardingsphere.datasource.master.connection-properties=characterEncoding=utf8 spring.shardingsphere.datasource.master.validation-query=SELECT 1 spring.shardingsphere.datasource.master.test-while-idle=true spring.shardingsphere.datasource.master.test-on-borrow=true spring.shardingsphere.datasource.master.test-on-return=false spring.shardingsphere.datasource.slave.type=org.apache.commons.dbcp2.BasicDataSource spring.shardingsphere.datasource.slave.driver-class-name=com.mysql.jdbc.Driver spring.shardingsphere.datasource.slave.url=jdbc:mysql://127.0.0.1:3306/cff?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC spring.shardingsphere.datasource.slave.username=cff spring.shardingsphere.datasource.slave.password=123456 spring.shardingsphere.datasource.slave.max-wait-millis=60000 spring.shardingsphere.datasource.slave.min-idle=20 spring.shardingsphere.datasource.slave.initial-size=2 spring.shardingsphere.datasource.slave.validation-query=SELECT 1 spring.shardingsphere.datasource.slave.connection-properties=characterEncoding=utf8 spring.shardingsphere.datasource.slave.validation-query=SELECT 1 spring.shardingsphere.datasource.slave.test-while-idle=true spring.shardingsphere.datasource.slave.test-on-borrow=true spring.shardingsphere.datasource.slave.test-on-return=false spring.shardingsphere.sharding.tables.t_chat_info.actual-data-nodes=ds0.t_chat_info_$->{0..9} spring.shardingsphere.sharding.tables.t_chat_info.table-strategy.inline.sharding-column=live_id spring.shardingsphere.sharding.tables.t_chat_info.table-strategy.inline.algorithm-expression=t_chat_info_$->{live_id % 10} spring.shardingsphere.sharding.tables.t_chat_info.key-generator.column=chat_no spring.shardingsphere.sharding.tables.t_chat_info.key-generator.type=SNOWFLAKE spring.shardingsphere.sharding.binding-tables=t_chat_info spring.shardingsphere.sharding.master-slave-rules.ds0.master-data-source-name=master spring.shardingsphere.sharding.master-slave-rules.ds0.slave-data-source-names=slave mybatis.configuration.log-impl=org.apache.ibatis.logging.stdout.StdOutImpl
这里的配置主要就是数据库及数据源、sharding-jdbc的配置。
上面都配置完成之后,代码的实现上和普通的数据库操作没区别了。
需要注意的是,增删改都要带上分表字段,查询可以不带分表字段;分表字段就是上面定义的sharding-column="live_id"
中的live_id
.
下面写好了增删改查,因为要带分表字段,参数都用实体来接收。
ChatInfoMapper :
package com.cff.springbootwork.sharding.jdbc.dao; import java.util.List; import org.apache.ibatis.annotations.Delete; import org.apache.ibatis.annotations.Insert; import org.apache.ibatis.annotations.Mapper; import org.apache.ibatis.annotations.Param; import org.apache.ibatis.annotations.Select; import org.apache.ibatis.annotations.Update; import com.cff.springbootwork.sharding.jdbc.domain.ChatInfo; @Mapper public interface ChatInfoMapper { @Select({ "<script>", "SELECT ", "chat_no as chatNo,user_id as userId,live_id as liveId,nick_name as nickName,create_time as createTime,delete_flag as deleteFlag,read_flag as readFlag", "FROM t_chat_info", "</script>"}) List<ChatInfo> findAll(); @Select({ "<script>", "SELECT ", "chat_no as chatNo,user_id as userId,live_id as liveId,nick_name as nickName,create_time as createTime,delete_flag as deleteFlag,read_flag as readFlag", "FROM t_chat_info", "<trim prefix=\" where \" prefixOverrides=\"AND\">", "<if test='paramIn.chatNo != null '> and chat_no = #{paramIn.chatNo} </if>", "<if test='paramIn.userId != null '> and user_id = #{paramIn.userId} </if>", "<if test='paramIn.liveId != null '> and live_id = #{paramIn.liveId} </if>", "<if test='paramIn.nickName != null '> and nick_name = #{paramIn.nickName} </if>", "<if test='paramIn.createTime != null '> and create_time = #{paramIn.createTime} </if>", "<if test='paramIn.deleteFlag != null '> and delete_flag = #{paramIn.deleteFlag} </if>", "<if test='paramIn.readFlag != null '> and read_flag = #{paramIn.readFlag} </if>", "</trim>", "</script>"}) public ChatInfo findByCondition(@Param("paramIn") ChatInfo paramIn); @Insert({ "<script> ", "INSERT INTO t_chat_info", "( chat_no,user_id,live_id,nick_name,create_time,delete_flag,read_flag ) ", " values ", "( #{chatNo},#{userId},#{liveId},#{nickName},#{createTime},#{deleteFlag},#{readFlag} ) ", "</script>"}) int save(ChatInfo item); @Update({ "<script>", "update t_chat_info", "<trim prefix=\"set\" suffixoverride=\",\"> ", "<if test='paramIn.userId != null '> user_id = #{paramIn.userId}, </if>", "<if test='paramIn.liveId != null '> live_id = #{paramIn.liveId}, </if>", "<if test='paramIn.nickName != null '> nick_name = #{paramIn.nickName}, </if>", "<if test='paramIn.createTime != null '> create_time = #{paramIn.createTime}, </if>", "<if test='paramIn.deleteFlag != null '> delete_flag = #{paramIn.deleteFlag}, </if>", "<if test='paramIn.readFlag != null '> read_flag = #{paramIn.readFlag}, </if>", "</trim>", "where chat_no = #{chatNo}", "</script>" }) int update(@Param("updateIn") ChatInfo updateIn, @Param("chatNo") Long chatNo); @Delete({ "<script>", " delete from t_chat_info", "<trim prefix=\" where \" prefixOverrides=\"AND\">", "<if test='paramIn.chatNo != null '> and chat_no = #{paramIn.chatNo} </if>", "<if test='paramIn.userId != null '> and user_id = #{paramIn.userId} </if>", "<if test='paramIn.liveId != null '> and live_id = #{paramIn.liveId} </if>", "<if test='paramIn.nickName != null '> and nick_name = #{paramIn.nickName} </if>", "<if test='paramIn.createTime != null '> and create_time = #{paramIn.createTime} </if>", "<if test='paramIn.deleteFlag != null '> and delete_flag = #{paramIn.deleteFlag} </if>", "<if test='paramIn.readFlag != null '> and read_flag = #{paramIn.readFlag} </if>", "</trim>", "</script>" }) int delete(@Param("paramIn") ChatInfo paramIn); }
没啥逻辑,就是个调用。注意主键要自己生成填入。
ChatInfoService:
package com.cff.springbootwork.sharding.jdbc.service; import java.util.List; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; import com.cff.springbootwork.sharding.jdbc.dao.ChatInfoMapper; import com.cff.springbootwork.sharding.jdbc.domain.ChatInfo; @Service public class ChatInfoService { @Autowired ChatInfoMapper chatInfoMapper; @Autowired SnowflakeIdGenerator snowflakeIdGenerator; public void save(ChatInfo tChatInfo) { tChatInfo.setChatNo(snowflakeIdGenerator.nextId()); chatInfoMapper.save(tChatInfo); } public void delete(ChatInfo tChatInfo) { chatInfoMapper.delete(tChatInfo); } public void update(ChatInfo tChatInfo) { chatInfoMapper.update(tChatInfo, tChatInfo.getChatNo()); } public List<ChatInfo> findAll() { return chatInfoMapper.findAll(); } public ChatInfo findByChatNoAndLiveId(Integer liveId, Long id) { ChatInfo condition = new ChatInfo(); condition.setChatNo(id); condition.setLiveId(liveId); return chatInfoMapper.findByCondition(condition); } public ChatInfo findById(Long id) { ChatInfo condition = new ChatInfo(); condition.setChatNo(id); return chatInfoMapper.findByCondition(condition); } }
下面的web请求层,涵盖了sharding-jdbc大多数情况下的使用。
ShardingRest :
package com.cff.springbootwork.sharding.jdbc.web; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestMethod; import org.springframework.web.bind.annotation.RequestParam; import org.springframework.web.bind.annotation.RestController; import com.cff.springbootwork.sharding.jdbc.domain.ChatInfo; import com.cff.springbootwork.sharding.jdbc.dto.ResultModel; import com.cff.springbootwork.sharding.jdbc.service.ChatInfoService; @RestController @RequestMapping("/sharing") public class ShardingRest { @Autowired ChatInfoService chatInfoService; @RequestMapping(value = "/add", method = { RequestMethod.POST }) public ResultModel add(@RequestBody ChatInfo chatInfo) { chatInfoService.save(chatInfo); return ResultModel.ok(); } /** * 查询也要携带分表字段,方便查找数据 * @param chatInfo * @return */ @RequestMapping(value = "/info", method = { RequestMethod.GET }) public ResultModel info(@RequestParam("liveId") Integer liveId, @RequestParam("chatNo") Long chatNo) { return ResultModel.ok(chatInfoService.findByChatNoAndLiveId(liveId,chatNo)); } /** * 不带分表字段查询 * @param chatInfo * @return */ @RequestMapping(value = "/detail", method = { RequestMethod.GET }) public ResultModel detail(@RequestParam("chatNo") Long chatNo) { return ResultModel.ok(chatInfoService.findById(chatNo)); } @RequestMapping(value = "/delete", method = { RequestMethod.POST }) public ResultModel delete(@RequestBody ChatInfo chatInfo) { chatInfoService.delete(chatInfo); return ResultModel.ok(); } @RequestMapping(value = "/update", method = { RequestMethod.POST }) public ResultModel update(@RequestBody ChatInfo chatInfo) { chatInfoService.update(chatInfo); return ResultModel.ok(); } }
SnowflakeIdGenerator :
package com.cff.springbootwork.sharding.jdbc.service; import javax.annotation.PostConstruct; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component; /** * SnowFlake 算法修改 * 0 - 0000000000 00 - 0000000000 0000000000 0000000000 000000000 - 0000 - 00000000 * 符号位 -12位年月位(表示yyMM,最大4096,即可用至2040年)-39位时间戳 (可用17年,即可用至2035年)-4位机器ID(最大16,即可部署16个节点)-8位序列号(z最大256) * @author yujinlong * */ @Component public class SnowflakeIdGenerator { // ==============================Fields=========================================== /** 开始时间截 (2018-01-01) */ private final long twepoch = 1514736000000L; /** 时间戳占的位数 */ public static final long timestampBits = 39L; /** 机器id所占的位数 */ public static final long workerIdBits = 4L; /** 支持的最大机器id,结果是15 (这个移位算法可以很快的计算出几位二进制数所能表示的最大十进制数) */ private final long maxWorkerId = -1L ^ (-1L << workerIdBits); /** 序列在id中占的位数 */ public static final long sequenceBits = 8L; /** 机器ID向左移6位 */ private final long workerIdShift = sequenceBits; /** 时间截向左移12位(4+8) */ private final long timestampLeftShift = sequenceBits + workerIdBits; /** 年月标识左移51位(39 + 4 + 8)*/ private final long yearMonthLeftShift = sequenceBits + workerIdBits + timestampBits; /** 生成序列的掩码,这里为255 */ private final long sequenceMask = -1L ^ (-1L << sequenceBits); /** 工作机器ID(0~16) */ @Value("${worker.id}") private long workerId; /** 毫秒内序列(0~256) */ private long sequence = 0L; /** 上次生成ID的时间截 */ private long lastTimestamp = -1L; // ==============================Methods========================================== @PostConstruct public void init(){ System.out.println(workerId); if(this.workerId < 0 || this.workerId > maxWorkerId){ throw new RuntimeException("workerId(" + this.workerId + ") is out of range [0, 15]"); } } /** * 获得下一个ID (该方法是线程安全的) * @return SnowflakeId */ public synchronized long nextId(long yyMM) { long timestamp = timeGen(); //如果当前时间小于上一次ID生成的时间戳,说明系统时钟回退过这个时候应当抛出异常 if (timestamp < lastTimestamp) { throw new RuntimeException( String.format("Clock moved backwards. Refusing to generate id for %d milliseconds", lastTimestamp - timestamp)); } //如果是同一时间生成的,则进行毫秒内序列 if (lastTimestamp == timestamp) { sequence = (sequence + 1) & sequenceMask; //毫秒内序列溢出 if (sequence == 0) { //阻塞到下一个毫秒,获得新的时间戳 timestamp = tilNextMillis(lastTimestamp); } } else { //时间戳改变,毫秒内序列重置 sequence = 0L; } //上次生成ID的时间截 lastTimestamp = timestamp; //移位并通过或运算拼到一起组成64位的ID long preId = (yyMM << yearMonthLeftShift) | ((timestamp - twepoch) << timestampLeftShift) | (workerId << workerIdShift) | sequence; return preId; } /** * 获得不带年月位的id * @return */ public synchronized long nextId() { long timestamp = timeGen(); //如果当前时间小于上一次ID生成的时间戳,说明系统时钟回退过这个时候应当抛出异常 if (timestamp < lastTimestamp) { throw new RuntimeException( String.format("Clock moved backwards. Refusing to generate id for %d milliseconds", lastTimestamp - timestamp)); } //如果是同一时间生成的,则进行毫秒内序列 if (lastTimestamp == timestamp) { sequence = (sequence + 1) & sequenceMask; //毫秒内序列溢出 if (sequence == 0) { //阻塞到下一个毫秒,获得新的时间戳 timestamp = tilNextMillis(lastTimestamp); } } else { //时间戳改变,毫秒内序列重置 sequence = 0L; } //上次生成ID的时间截 lastTimestamp = timestamp; //移位并通过或运算拼到一起组成64位的ID long preId = ((timestamp - twepoch) << timestampLeftShift) | (workerId << workerIdShift) | sequence; return preId; } /** * 阻塞到下一个毫秒,直到获得新的时间戳 * @param lastTimestamp 上次生成ID的时间截 * @return 当前时间戳 */ protected long tilNextMillis(long lastTimestamp) { long timestamp = timeGen(); while (timestamp <= lastTimestamp) { timestamp = timeGen(); } return timestamp; } /** * 返回以毫秒为单位的当前时间 * @return 当前时间(毫秒) */ protected long timeGen() { return System.currentTimeMillis(); } public void setWorkerId(long workerId) { this.workerId = workerId; } }
ChatInfo:
package com.cff.springbootwork.sharding.jdbc.domain; import java.util.Date; public class ChatInfo { private Long chatNo; private Integer userId; private Integer liveId; private String nickName; private Date createTime; private Integer deleteFlag; private Integer readFlag; public void setChatNo(Long chatNo) { this.chatNo = chatNo; } public Long getChatNo() { return chatNo; } public Integer getUserId() { return userId; } public void setUserId(Integer userId) { this.userId = userId; } public Integer getLiveId() { return liveId; } public void setLiveId(Integer liveId) { this.liveId = liveId; } public String getNickName() { return nickName; } public void setNickName(String nickName) { this.nickName = nickName; } public Date getCreateTime() { return createTime; } public void setCreateTime(Date createTime) { this.createTime = createTime; } public Integer getDeleteFlag() { return deleteFlag; } public void setDeleteFlag(Integer deleteFlag) { this.deleteFlag = deleteFlag; } public Integer getReadFlag() { return readFlag; } public void setReadFlag(Integer readFlag) { this.readFlag = readFlag; } }
ResultModel:
package com.cff.springbootwork.sharding.jdbc.dto; public class ResultModel { private String errorCode; private String message; private Object data; public ResultModel() { } public ResultModel(String errorCode) { this.errorCode = errorCode; } public ResultModel(String errorCode, String message) { this.errorCode = errorCode; this.message = message; } public ResultModel(String errorCode, String message, Object data) { this.errorCode = errorCode; this.message = message; this.data = data; } public String getErrorCode() { return errorCode; } public void setErrorCode(String errorCode) { this.errorCode = errorCode; } public String getMessage() { return message; } public void setMessage(String message) { this.message = message; } public Object getData() { return data; } public void setData(Object data) { this.data = data; } public static ResultModel ok() { ResultModel resultModel = new ResultModel("0000", "成功"); return resultModel; } public static ResultModel ok(Object data) { ResultModel resultModel = new ResultModel("0000", "成功"); resultModel.setData(data); return resultModel; } public static ResultModel error() { ResultModel resultModel = new ResultModel("1111", "失败"); return resultModel; } public static ResultModel resultModel(String message) { ResultModel resultModel = new ResultModel("1111", message); return resultModel; } }
得出的结论如下:
null
should extends Comparable for sharding value.原创声明,本文系作者授权云+社区发表,未经许可,不得转载。
如有侵权,请联系 yunjia_community@tencent.com 删除。
我来说两句