压缩算法选型(gzip/snappy/lz4)及性能对比

原创

用户7255712

修改于 2021-11-21 17:16:12

12.5K0

修改于 2021-11-21 17:16:12

常见压缩算法对比:Gzip/Snappy/Lz4性能对比

0 Gzip方式
1 Snappy方式
2 Lz4方式

0 Gzip方式

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.Closeable;
import java.io.IOException;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;

public class GzipUtil {

    /**
     * @param bytes 字节数组
     * @return byte[]
     * @description 压缩字节数组
     */
    public static byte[] compress(byte[] bytes) {
        if (bytes == null || bytes.length == 0) {
            return null;
        }

        //将byte数据读入文件流
        ByteArrayOutputStream bos = null;
        GZIPOutputStream gzipos = null;
        try {
            bos = new ByteArrayOutputStream();
            gzipos = new GZIPOutputStream(bos);
            gzipos.write(bytes);
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            closeStream(gzipos);
            closeStream(bos);
        }
        return bos.toByteArray();
    }

    /**
     * @param bytes Gzip压缩的字节数组
     * @return byte[]
     * @description 解压缩字节数组
     */
    public static byte[] unCompress(byte[] bytes) {
        if (bytes == null || bytes.length == 0) {
            return null;
        }
        ByteArrayOutputStream byteAos = null;
        ByteArrayInputStream byteArrayIn = null;
        GZIPInputStream gzipIn = null;
        try {
            byteArrayIn = new ByteArrayInputStream(bytes);
            gzipIn = new GZIPInputStream(byteArrayIn);
            byteAos = new ByteArrayOutputStream();
            byte[] b = new byte[4096];
            int temp = -1;
            while ((temp = gzipIn.read(b)) > 0) {
                byteAos.write(b, 0, temp);
            }
        } catch (Exception e) {
            e.printStackTrace();
            return null;
        } finally {
            closeStream(byteAos);
            closeStream(gzipIn);
            closeStream(byteArrayIn);
        }
        return byteAos.toByteArray();
    }

    /**
     * 关闭流
     *
     * @param oStream
     */
    private static void closeStream(Closeable oStream) {
        if (null != oStream) {
            try {
                oStream.close();
            } catch (IOException e) {
                oStream = null;//赋值为null,等待垃圾回收
                e.printStackTrace();
            }
        }
    }


    public static void main(String[] args) throws Exception{
        String sss = "test";
        byte[] compress = compress(sss.getBytes("utf-8"));
        System.out.println(compress[0]);
        System.out.println(compress[1]);
        System.out.println(compress[0]==0x1F);
        System.out.println(compress[1]==0x8b);
        System.out.println(0x1F);
        System.out.println(0x8b);
        //有符号-117 无符号139


        String sss2 = "{}";
        byte[] compress2 = compress(sss2.getBytes());
        System.out.println(compress2[0]);
        System.out.println(compress2[1]);

        System.out.println(sss.getBytes()[0]);
        System.out.println(sss.getBytes()[1]);
    }
}

1 Snappy方式

1.0 官网参考

https://github.com/xerial/snappy-java

1.1 引入依赖

 <dependency>
            <groupId>org.xerial.snappy</groupId>
            <artifactId>snappy-java</artifactId>
            <version>1.1.2.6</version>
</dependency>

1.2 工具类

import lombok.extern.slf4j.Slf4j;
import org.xerial.snappy.Snappy;

@Slf4j
public class SnappyUtil {

    public static byte[] compress(byte bytes[]) {
        if (bytes == null || bytes.length == 0) {
            return null;
        }
        try {
            return Snappy.compress(bytes);
        } catch (Exception e) {
            log.error("SnappyUtil compress error", e);
            return null;
        }
    }

    public static byte[] unCompress(byte[] bytes) {
        if (bytes == null || bytes.length == 0) {
            return null;
        }
        try {
            return Snappy.uncompress(bytes);
        } catch (Exception e) {
            log.error("SnappyUtil uncompress error", e);
            return null;
        }
    }
}

1.3 高级用法

通过Snappy.compress()进行压缩，压缩后的数据没有magic header

通过SnappyInputStream进行压缩，压缩后的数据有固定的header(多20个字节)

@Slf4j
public class SnappyUtil {

    


    public static byte[] compressWithHeader(byte bytes[]) {
        if (bytes == null || bytes.length == 0) {
            return null;
        }
        try {
            ByteArrayInputStream is = new ByteArrayInputStream(bytes);
            ByteArrayOutputStream os = new ByteArrayOutputStream();
            SnappyOutputStream sos = new SnappyOutputStream(os);
            int count;
            byte temp[] = new byte[4096];
            try {
                while ((count = is.read(temp)) != -1) {
                    sos.write(temp, 0, count);
                }
                sos.flush();
                return os.toByteArray();
            } finally {
                sos.close();
                is.close();
            }
        } catch (Exception e) {
            log.error("SnappyUtil compressWithHeader error", e);
            return null;
        }
    }

    public static byte[] unCompressWithHeader(byte[] bytes) {
        if (bytes == null || bytes.length == 0) {
            return null;
        }
        ByteArrayOutputStream byteAos = null;
        ByteArrayInputStream byteArrayIn = null;
        SnappyInputStream gzipIn = null;
        try {
            byteArrayIn = new ByteArrayInputStream(bytes);
            gzipIn = new SnappyInputStream(byteArrayIn);
            byteAos = new ByteArrayOutputStream();
            byte[] b = new byte[4096];
            int temp = -1;
            while ((temp = gzipIn.read(b)) > 0) {
                byteAos.write(b, 0, temp);
            }
        } catch (Exception e) {
            log.error("SnappyUtil unCompressWithHeader error", e);
            return null;
        } finally {
            closeStream(byteAos);
            closeStream(gzipIn);
            closeStream(byteArrayIn);
        }
        return byteAos.toByteArray();
    }

    private static void closeStream(Closeable oStream) {
        if (null != oStream) {
            try {
                oStream.close();
            } catch (IOException e) {
                oStream = null;//赋值为null,等待垃圾回收
                e.printStackTrace();
            }
        }
    }

}

/**
     * 输出如下：
     * Snappy.compress  压缩结果：2b a8 54 68 65 20 71 75 69 63 6b 20 62 72 6f 77 6e 20 66 6f 78 20 6a 75 6d 70 73 20 6f 76 65 72 20 74 68 65 20 6c 61 7a 79 20 64 6f 67
     * SnappyInputStream压缩结果：82 53 4e 41 50 50 59 00 00 00 00 01 00 00 00 01 00 00 00 2d 2b a8 54 68 65 20 71 75 69 63 6b 20 62 72 6f 77 6e 20 66 6f 78 20 6a 75 6d 70 73 20 6f 76 65 72 20 74 68 65 20 6c 61 7a 79 20 64 6f 67
     *                         |---------------------magic header(16bytes)-----|size(4bytes)|----compressed data-----
     */

2 Lz4方式

2.1 引入依赖

<dependency>
            <groupId>org.lz4</groupId>
            <artifactId>lz4-java</artifactId>
            <version>1.8.0</version>
</dependency>

2.2 工具类

public class Lz4Util {

    private static final int ARRAY_SIZE = 4096;
    private static LZ4Factory factory = LZ4Factory.fastestInstance();
    private static LZ4Compressor compressor = factory.fastCompressor();
    private static LZ4FastDecompressor decompressor = factory.fastDecompressor();


    public static byte[] compress(byte bytes[]) {
        if (bytes == null || bytes.length == 0) {
            return null;
        }
        try {
            ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
            LZ4BlockOutputStream lz4BlockOutputStream = new LZ4BlockOutputStream(outputStream, ARRAY_SIZE, compressor);
            lz4BlockOutputStream.write(bytes);
            lz4BlockOutputStream.finish();
            return outputStream.toByteArray();
        } catch (Exception e) {
            log.error("Lz4Util compress error", e);
            return null;
        }
    }

    public static byte[] unCompress(byte[] bytes) {
        if (bytes == null || bytes.length == 0) {
            return null;
        }
        try {
            ByteArrayOutputStream outputStream = new ByteArrayOutputStream(ARRAY_SIZE);

            ByteArrayInputStream inputStream = new ByteArrayInputStream(bytes);
            LZ4BlockInputStream decompressedInputStream
                    = new LZ4BlockInputStream(inputStream, decompressor);
                int count;
                byte[] buffer = new byte[ARRAY_SIZE];
                while ((count = decompressedInputStream.read(buffer)) != -1) {
                    outputStream.write(buffer, 0, count);
                }

            return outputStream.toByteArray();
        } catch (Exception e) {
            e.printStackTrace();
            System.out.println(e.getMessage());
            log.error("Lz4Util uncompress error", e);
            return null;
        }
    }


    public static void main(String[] args) {
        String sss ="as";
        byte[] compress = compress(sss.getBytes());
        System.out.println("sss:"+sss.getBytes().length+"compress:"+compress.length);
        byte[] bytes = unCompress(compress);
        System.out.println(new String(bytes));
    }
}

2.3 lz4高级使用

官网:https://github.com/lz4/lz4-java

3 对比总结

3.1 一些压测报告

https://gitee.com/wayz/compress snappy好

https://github.com/lz4/lz4 官网lz4号

https://bbs.pediy.com/thread-197445.htm （snappy压缩速度要快于lz4,但是lz4解压缩速度快了snappy一大截）:各有优点

3.2 自己压测

压缩算法	原始数据大小	压缩后数据大小	解压缩次数	耗时
gzip	3260	590	10000	171/163/152/146/148
lz4	3260	1103	10000	61/60/62/73/63
snappy	3260	1056	10000	36/39/33/33/33
snappy-header	3260	1076	10000	57/57/56/57/56

3.3 最终总结

综合考虑,snappy具有更优秀的性能,对耗时要求更高的情况,优先考虑。

对压缩比更看重,而耗时可以接受的情况,优先考虑Gzip

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

java

spring boot

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

java

spring boot

#压缩算法

登录后参与评论

0 条评论

热度

压缩算法选型(gzip/snappy/lz4)及性能对比

压缩算法选型(gzip/snappy/lz4)及性能对比

常见压缩算法对比:Gzip/Snappy/Lz4性能对比

0 Gzip方式

1 Snappy方式

1.0 官网参考

1.1 引入依赖

1.2 工具类

1.3 高级用法

2 Lz4方式

2.1 引入依赖

2.2 工具类

2.3 lz4高级使用

3 对比总结

3.1 一些压测报告

3.2 自己压测

3.3 最终总结

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐