当图片较小时,上传时间很快,而且可以直接显示原像素。
如果我们的图片达到几兆时,我就不说几个G了,我是为了模拟分片上传,并顺便解决我的垃圾服务器的上传速度慢问题。
而且,图片较大时,如果直接显示在前端,会因为文件过大加载很长时间,这就需要对图片进行处理。
如果大家正在寻找一个java的学习环境,或者在开发中遇到困难,可以加入我们的java学习圈,点击即可加入,共同学习,节约学习时间,减少很多在学习中遇到的难题。
本次分块上传的主要思路是:
我用一张表来存储上传的图片记录,并配合实现分片上传,也可以用配置文件这种形式。这里只展示表列和字段的对应,不单独列出表:
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@Table(name = "f_resources")
public class ResourceInfo {
@Column(name = "file_path")
private String filePath;
@Column(name = "resource_url")
private String resourceUrl;
@Id
@Column(name = "resource_id")
private Long resourceId;
@Column(name = "create_time")
private Date createTime;
@Column(name = "is_slice")
private Integer isSlice;
@Column(name = "slice_num")
private Integer sliceNum;
@Column(name = "bit_map")
private Integer bitMap;
}
该接口的动作是前端发起分片上传请求到后端,后端处理生成唯一标识,返回前端。
/**
* 分片上传
*
* @param file
* @param principal
* @return
*/
@RequestMapping(value = "/slicePictureStart", method = { RequestMethod.POST })
public ResultModel uploadSlicePicture(SilceFileReq silceFileReq, Principal principal) {
try {
if (silceFileReq.getChunkTotalNum() < 1) {
return new ResultModel(ResultCode.CODE_00003);
}
SilceFileRes silceFileRes = resourceService.startSlicePicture(silceFileReq.getFileName(),
silceFileReq.getChunkTotalNum());
silceFileRes.setSliceProcess(0f);
return ResultModel.ok(silceFileRes);
} catch (Exception e) {
e.printStackTrace();
return new ResultModel(ResultCode.CODE_00004);
}
}
这里,必须要把分片总数(chunkTotalNum)、文件名(fileName)提交,调用resourceService.startSlicePicture
进行预处理,并返回对象silceFileRes
.
resourceService.startSlicePicture:
/**
* 开始分片上传
*
* @param file
* @return
* @throws Exception
*/
public SilceFileRes startSlicePicture(String fileName, Integer chunckNum) throws Exception {
SilceFileRes silceFileRes = new SilceFileRes();
String datePath = DateUtil.format(new Date(), DateUtil.SimpleDatePattern);
String uploadPath = resourcesDir + resourcesRelative + SLICE_PATH + datePath + "/";
Long resourceId = resourceIdGenerator.nextId();
silceFileRes.setFileId(resourceId);
String newfileName = resourceId + fileName.substring(fileName.lastIndexOf('.'));
String filePath = uploadPath + newfileName;
String webPath = resourcesSlice + datePath + "/" + newfileName;
silceFileRes.setWebUrl(webPath);
File dir = new File(uploadPath);
if (!dir.exists()) {
dir.mkdirs();
}
dir.setReadable(true, false);
dir.setExecutable(true, false);
dir.setWritable(true, false);
new File(filePath).createNewFile();
ResourceInfo resourceInfo = new ResourceInfo();
resourceInfo.setResourceId(resourceId);
resourceInfo.setFilePath(filePath);
resourceInfo.setResourceUrl(webPath);
resourceInfo.setCreateTime(new Date());
resourceInfo.setIsSlice(DbConstant.DB_SLICE_FILE);
resourceInfo.setSliceNum(chunckNum);
resourceInfo.setBitMap(0);
resourceInfoDao.save(resourceInfo);
return silceFileRes;
}
在startSlicePicture这一步,生成了唯一标识resourceId(也就是fileId),入库并生成文件。
/**
* 分片上传
*
* @param file
* @param principal
* @return
*/
@RequestMapping(value = "/uploadSlicePicture", method = { RequestMethod.POST })
public ResultModel uploadSlicePicture(SilceFileReq silceFileReq, Principal principal) {
try {
if (silceFileReq.getFileId() == null || silceFileReq.getFile() == null) {
return new ResultModel(ResultCode.CODE_00003);
}
ResourceInfo resourceInfo = resourceService.findById(silceFileReq.getFileId());
if (resourceInfo == null) {
return new ResultModel(ResultCode.CODE_00011);
}
if (resourceInfo.getSliceNum() != silceFileReq.getChunkTotalNum()
|| silceFileReq.getChunkTotalNum() < silceFileReq.getChunkNum()) {
return new ResultModel(ResultCode.CODE_00017);
}
// 将数据库的bitMap转换成2进制计算进度
Integer bitMap = resourceInfo.getBitMap();
String totalBinaryNum = Integer.toBinaryString(bitMap);
char[] bitArray = new char[resourceInfo.getSliceNum()];
for (int i = 0; i < totalBinaryNum.length() && i < resourceInfo.getSliceNum(); i++) {
bitArray[i] = totalBinaryNum.charAt(totalBinaryNum.length() - i - 1);
}
if (totalBinaryNum.length() < resourceInfo.getSliceNum()) {
for (int i = totalBinaryNum.length(); i < resourceInfo.getSliceNum(); i++) {
bitArray[i] = '0';
}
}
SilceFileRes silceFileRes = new SilceFileRes();
int count = 0;
for (int i = 0; i < resourceInfo.getSliceNum(); i++) {
char charZeroOrOne = bitArray[i];
if (charZeroOrOne == '1') {
count++;
}
}
// 如果bitmap标识全部上传完了,直接返回100%
if (count >= resourceInfo.getSliceNum()) {
silceFileRes.setFileId(silceFileReq.getFileId());
silceFileRes.setSliceProcess(100f);
return ResultModel.ok(silceFileRes);
}
Integer chunkNum = silceFileReq.getChunkNum();
// 如果该分片还没上传
if (bitArray[chunkNum - 1] == '0') {
// 使用RandomAccessFile定位到文件起始位置并写入
RandomAccessFile accessTmpFile = new RandomAccessFile(resourceInfo.getFilePath(), "rw");
accessTmpFile.seek((silceFileReq.getChunkNum() - 1) * (500 * 1024));
accessTmpFile.write(silceFileReq.getFile().getBytes());
accessTmpFile.close();
String bNum = "1";
for (int i = 1; i < chunkNum; i++) {
bNum += "0";
}
// 将当前块的bitmap拼接成10000这种形式,并转换成10进制,比如第4个分块,是1000,10进制就是8
Integer cnum = Integer.parseInt(bNum, 2);
// 写入完成,更新bitmap
resourceService.updateBitMap(resourceInfo.getResourceId(), cnum);
}
count++;
silceFileRes.setFileId(silceFileReq.getFileId());
silceFileRes.setSliceProcess((float) count * 100 / (float) resourceInfo.getSliceNum());
return ResultModel.ok(silceFileRes);
} catch (Exception e) {
e.printStackTrace();
return new ResultModel(ResultCode.CODE_00004);
}
}
这里,bitmap的计算是这样的,
19 | 8 = 10011 | 1000 = 27
,等第3片上传完成再加上 4 即可。updateBitMap:
@Update({
"<script>",
" update f_resources set",
"bit_map = bit_map | #{bitMap}",
" where resource_id=#{resourceId}",
"</script>"
})
int updateBitMap(@Param("resourceId") Long resourceId, @Param("bitMap") int bitMap);
因为多线程的存在,后端这个进度可能是不准确的,可以增加一次查询,使进度更准确点,也可以让前端去控制进度。
完成接口,就是给一个进度的反馈(如果分片的文件并不是最终名称,可以在这个方法中将文件重命名)。
/**
* 分片上传
*
* @param file
* @param principal
* @return
*/
@RequestMapping(value = "/slicePictureEnd", method = { RequestMethod.POST })
public ResultModel uploadSlicePicture(SilceFileReq silceFileReq, Principal principal) {
try {
ResourceInfo resourceInfo = resourceService.findById(silceFileReq.getFileId());
if (resourceInfo == null) {
return new ResultModel(ResultCode.CODE_00011);
}
int complateProgress = (int) (Math.pow(2, resourceInfo.getSliceNum()) - 1);
if (resourceInfo.getBitMap() == complateProgress) {
String md5 = DigestUtils.md5DigestAsHex(new FileInputStream(resourceInfo.getFilePath()));
SilceFileRes silceFileRes = new SilceFileRes();
silceFileRes.setWebUrl(resourceInfo.getResourceUrl());
silceFileRes.setFileId(silceFileReq.getFileId());
silceFileRes.setSliceProcess(100f);
silceFileRes.setMd5(md5);
return ResultModel.ok(silceFileRes);
}else {
String totalBinaryNum = Integer.toBinaryString(resourceInfo.getBitMap());
int count = 0;
for (int i = 0; i < totalBinaryNum.length() && i < resourceInfo.getSliceNum(); i++) {
if(totalBinaryNum.charAt(totalBinaryNum.length() - i - 1) == '1') {
count ++;
}
}
return new ResultModel(ResultCode.CODE_00011, (float) count * 100 / (float) resourceInfo.getSliceNum());
}
} catch (Exception e) {
e.printStackTrace();
return new ResultModel(ResultCode.CODE_00004);
}
}
这里,同时计算了个md5给前端。
我这里将三个接口汇总成一个接口,用type区分:
/**
* 分片上传
*
* @param file
* @param principal
* @return
*/
@RequestMapping(value = "/uploadSlicePicture", method = { RequestMethod.POST })
public ResultModel uploadSlicePicture(SilceFileReq silceFileReq, Principal principal) {
try {
if (silceFileReq.getType() == 0) {
if (silceFileReq.getChunkTotalNum() < 1) {
return new ResultModel(ResultCode.CODE_00003);
}
SilceFileRes silceFileRes = resourceService.startSlicePicture(silceFileReq.getFileName(),
silceFileReq.getChunkTotalNum());
silceFileRes.setSliceProcess(0f);
return ResultModel.ok(silceFileRes);
} else if (silceFileReq.getType() == 2) {
if (silceFileReq.getChunkTotalNum() < 1) {
return new ResultModel(ResultCode.CODE_00003);
}
ResourceInfo resourceInfo = resourceService.findById(silceFileReq.getFileId());
if (resourceInfo == null) {
return new ResultModel(ResultCode.CODE_00011);
}
int complateProgress = (int) (Math.pow(2, resourceInfo.getSliceNum()) - 1);
if (resourceInfo.getBitMap() == complateProgress) {
String md5 = DigestUtils.md5DigestAsHex(new FileInputStream(resourceInfo.getFilePath()));
SilceFileRes silceFileRes = new SilceFileRes();
silceFileRes.setWebUrl(resourceInfo.getResourceUrl());
silceFileRes.setFileId(silceFileReq.getFileId());
silceFileRes.setSliceProcess(100f);
silceFileRes.setMd5(md5);
return ResultModel.ok(silceFileRes);
}else {
String totalBinaryNum = Integer.toBinaryString(resourceInfo.getBitMap());
int count = 0;
for (int i = 0; i < totalBinaryNum.length() && i < resourceInfo.getSliceNum(); i++) {
if(totalBinaryNum.charAt(totalBinaryNum.length() - i - 1) == '1') {
count ++;
}
}
return new ResultModel(ResultCode.CODE_00011, (float) count * 100 / (float) resourceInfo.getSliceNum());
}
} else {
if (silceFileReq.getFileId() == null || silceFileReq.getFile() == null) {
return new ResultModel(ResultCode.CODE_00003);
}
ResourceInfo resourceInfo = resourceService.findById(silceFileReq.getFileId());
if (resourceInfo == null) {
return new ResultModel(ResultCode.CODE_00011);
}
if (resourceInfo.getSliceNum() != silceFileReq.getChunkTotalNum()
|| silceFileReq.getChunkTotalNum() < silceFileReq.getChunkNum()) {
return new ResultModel(ResultCode.CODE_00017);
}
Integer bitMap = resourceInfo.getBitMap();
String totalBinaryNum = Integer.toBinaryString(bitMap);
char[] bitArray = new char[resourceInfo.getSliceNum()];
for (int i = 0; i < totalBinaryNum.length() && i < resourceInfo.getSliceNum(); i++) {
bitArray[i] = totalBinaryNum.charAt(totalBinaryNum.length() - i - 1);
}
if (totalBinaryNum.length() < resourceInfo.getSliceNum()) {
for (int i = totalBinaryNum.length(); i < resourceInfo.getSliceNum(); i++) {
bitArray[i] = '0';
}
}
SilceFileRes silceFileRes = new SilceFileRes();
int count = 0;
for (int i = 0; i < resourceInfo.getSliceNum(); i++) {
char charZeroOrOne = bitArray[i];
if (charZeroOrOne == '1') {
count++;
}
}
if (count >= resourceInfo.getSliceNum()) {
silceFileRes.setFileId(silceFileReq.getFileId());
silceFileRes.setSliceProcess(100f);
return ResultModel.ok(silceFileRes);
}
Integer chunkNum = silceFileReq.getChunkNum();
if (bitArray[chunkNum - 1] == '0') {
RandomAccessFile accessTmpFile = new RandomAccessFile(resourceInfo.getFilePath(), "rw");
accessTmpFile.seek((silceFileReq.getChunkNum() - 1) * (500 * 1024));
accessTmpFile.write(silceFileReq.getFile().getBytes());
accessTmpFile.close();
String bNum = "1";
for (int i = 1; i < chunkNum; i++) {
bNum += "0";
}
Integer cnum = Integer.parseInt(bNum, 2);
resourceService.updateBitMap(resourceInfo.getResourceId(), cnum);
}
count++;
silceFileRes.setFileId(silceFileReq.getFileId());
silceFileRes.setSliceProcess((float) count * 100 / (float) resourceInfo.getSliceNum());
return ResultModel.ok(silceFileRes);
}
} catch (Exception e) {
e.printStackTrace();
return new ResultModel(ResultCode.CODE_00004);
}
}
前端使用后端提供的汇总接口user/uploadSlicePicture
。
function uploadLargeFile(file, fn){
var chunkSize = 500 * 1024;
var totalNum = Math.ceil(file.size / chunkSize);
var fileName = file.name;
var formData = new FormData();
formData.append('chunkNum', 0);
formData.append('chunkTotalNum', totalNum);
formData.append('type', 0);
formData.append('fileName', fileName);
$.ajax({
type : "post",
url : "user/uploadSlicePicture",
data: formData,
contentType: false,
processData: false,
dataType : "json",
beforeSend: function(){
if($("#modalUploadProcess")){
$("#modalUploadProcess").modal('open');
}
},
success : function(data) {
if (data.errorCode == "00000") {
var process = 0;
var percentProcess = (1 / totalNum) * 100;
$("#processBarProcess").width("1%");
var fileId = data.data.fileId;
for(var i =0;i<totalNum;i++){
var chunckFormData = new FormData();
chunckFormData.append('fileId', fileId);
chunckFormData.append('chunkNum', i + 1);
chunckFormData.append('chunkTotalNum', totalNum);
chunckFormData.append('type', 1);
chunckFormData.append('fileName', fileName);
chunckFormData.append('file', file.slice(i*chunkSize,(i+1) * chunkSize));
$.ajax({
type : "post",
url : "user/uploadSlicePicture",
data: chunckFormData,
contentType: false,
processData: false,
dataType : "json",
success : function(cdata) {
if (cdata.errorCode == "00000") {
process += percentProcess;
// 进度条更新进度
$("#processBarProcess").width(process+"%");
// 上传结束同步结果
if(cdata.data.sliceProcess > 99.9 || process > 99.9){
sliceUploadEnd(fileId, totalNum, fileName, fn);
}
}
},
error : function(XMLHttpRequest, textStatus, errorThrown) {
$("#progressMessage").html("图片上传失败" + data.message);
return;
}
});
}
}else{
if($("#modalUploadProcess")){
$("#modalUploadProcess").modal('close');
}
$("#modalAlertMessage").modal("open");
$("#alertMessage").html("图片上传失败" + data.message);
}
},
error : function(XMLHttpRequest, textStatus, errorThrown) {
if($("#modalUploadProcess")){
$("#modalUploadProcess").modal('close');
}
$("#modalAlertMessage").modal("open");
$("#alertMessage").html("图片上传失败" + data.message);
}
});
}
这里,默认分块是500K,先请求接口分片开始,然后循环调用分片上传,当前端进度或者后端进度返回大于99.9,调用sliceUploadEnd()方法请求后端分片结束接口。
function sliceUploadEnd(fileId, totalNum, fileName, fn){
var endFormData = new FormData();
endFormData.append('fileId', fileId);
endFormData.append('chunkNum', totalNum);
endFormData.append('chunkTotalNum', totalNum);
endFormData.append('type', 2);
endFormData.append('fileName', fileName);
$.ajax({
type : "post",
url : apiUrl + "user/uploadSlicePicture",
data: endFormData,
contentType: false,
processData: false,
dataType : "json",
success : function(edata) {
var spark = new SparkMD5();
var reader = new FileReader();
reader.onload = function(e) {
spark.appendBinary(e.target.result);
var fileMd5 = spark.end();
console.log(edata.data.md5);
console.log(fileMd5);
if (cdata.errorCode == "00000") {
if(edata.data.md5 == fileMd5){
if(typeof fn == "function"){
//调用它,既然我们已经确定了它是可调用的
$("#progressMessage").html("上传完成!");
fn(edata.data.webUrl);
return;
}
}else{
$("#progressMessage").html("上传进度" + edata.data.sliceProcess + ",上传未完成");
return;
}
}else{
$("#progressMessage").html("图片上传失败!");
return;
}
}
reader.readAsBinaryString(file);
},
error : function(XMLHttpRequest, textStatus, errorThrown) {
$("#progressMessage").html("图片上传失败" + data.message);
return;
}
});
}
这里,使用SparkMD5进行前端的文件md5的计算。并和后端文件进行对比。
<div id="modalUploadPicture" class="modal">
<div class="modal-content">
<h4>上传图片</h4>
<div class="file-field input-field">
<div class="btn" style="color: white;">
<span>图片</span>
<input type="file" id="uploadPictureFile">
</div>
<div class="file-path-wrapper">
<input class="file-path validate" type="text">
</div>
</div>
</div>
<div class="modal-footer">
<a href="javascript:uploadPictureNote()" class=" modal-action modal-close waves-effect waves-green btn-flat">确定</a>
<a href="javascript:void(0)" class=" modal-action modal-close waves-effect waves-green btn-flat">关闭</a>
</div>
</div>
<div id="modalUploadProcess" class="modal">
<div class="modal-content">
<h4>文件上传进度</h4>
<div class="progress">
<div class="determinate" id="processBarProcess" style="width: 0%"></div>
</div>
<div id="progressMessage" class="red-text">
</div>
</div>
<div class="modal-footer">
<a href="javascript:void(0)" class=" modal-action modal-close waves-effect waves-green btn-flat">关闭</a>
</div>
</div>
modalUploadPicture窗口弹出之后,是这样的:
在这里插入图片描述
点击上传,弹出modalUploadProcess窗口,是这样的:
在这里插入图片描述
分开上传完成有返回图片的地址,可以将图片显示在任意位置。
function uploadPictureNote(){
var file = $('#uploadPictureFile')[0].files[0];
uploadLargeFile(function(data){
var url = staticUrl + data;
var htmlContent = $("#contentDetail").html();
htmlContent += `
<img style="max-width:80%;max-height:90%;" src="${url}"/>
`;
$("#contentDetail").html(htmlContent);
});
}
因为图片比较大,直接显示流量不允许,可以将图片进行压缩。
nginx有一个模块叫ngx_http_image_filter_module
,网上有很多配置方法,但是在我的机器上,这个模块是nginx默认加载的。
因为加载了配置文件/usr/share/nginx/modules/mod-http-image-filter.conf
;文件里就一句话load_module "/usr/lib64/nginx/modules/ngx_http_image_filter_module.so";
在nginx的配置文件中,增加这几行即可。
location /largepic/ {
alias /upload/slice/;
index index.html;
image_filter resize 300 400;
image_filter_buffer 10M;
}
这是将图片按照300 * 400的像素显示出来,缓冲区大小为10M,如果设置太小,就会出现无法加载图片的情况,默认是1M.