zookeeper 删除snapshot和transaction log的源码解读
zookeeper 删除snapshot和transaction log的源码解读
转载请注明源地址http://www.cnblogs.com/dongxiao-yang/p/4910059.html
zookeeper具有自动清除快照日志和事务日志的工能，可以在配置文件设置autopurge.purgeInterval来实现，问题是这个属性的时间单位是小时，
有些情况下，一小时的日志过大（比如把事务日志放到内存），需要手动删除，所以需要研究下zk删除日志文件的源码。
清理日志主类:org.apache.zookeeper.server.PurgeTxnLog,包含如下几个静态工具方法
static void printUsage(){
System.out.println("PurgeTxnLog dataLogDir [snapDir] -n count");
System.out.println("\tdataLogDir -- path to the txn log directory");
System.out.println("\tsnapDir -- path to the snapshot directory");
System.out.println("\tcount -- the number of old snaps/logs you want to keep");
System.exit(1);
}
常见的帮助方法，告诉使用者参数的传入顺序，其中snapdir参数为可选，假如两种日志配置在同一个路径下，只传一个路径参数就好。
main方法，没什么好说的，只是解析参数。
public static void purge(File dataDir, File snapDir, int num) throws IOException {
if (num < 3) {
throw new IllegalArgumentException("count should be greater than 3");
}
FileTxnSnapLog txnLog = new FileTxnSnapLog(dataDir, snapDir);
List<File> snaps = txnLog.findNRecentSnapshots(num);
retainNRecentSnapshots(txnLog, snaps);
}
删除文件的主方法，主要分两个部分
1：txnLog.findNRecentSnapshots(num);
找到需要保留的文件
主要逻辑代码为
public List<File> findNRecentSnapshots(int n) throws IOException {
List<File> files = Util.sortDataDir(snapDir.listFiles(), "snapshot", false);
int i = 0;
List<File> list = new ArrayList<File>();
for (File f: files) {
if (i==n)
break;
i++;
list.add(f);
}
return list;
}
private static class DataDirFileComparator
implements Comparator<File>, Serializable
{
private static final long serialVersionUID = -2648639884525140318L;
private String prefix;
private boolean ascending;
public DataDirFileComparator(String prefix, boolean ascending) {
this.prefix = prefix;
this.ascending = ascending;
}
public int compare(File o1, File o2) {
long z1 = Util.getZxidFromName(o1.getName(), prefix);
long z2 = Util.getZxidFromName(o2.getName(), prefix);
int result = z1 < z2 ? -1 : (z1 > z2 ? 1 : 0);
return ascending ? result : -result;
}
}
/**
* Sort the list of files. Recency as determined by the version component
* of the file name.
*
* @param files array of files
* @param prefix files not matching this prefix are assumed to have a
* version = -1)
* @param ascending true sorted in ascending order, false results in
* descending order
* @return sorted input files
*/
public static List<File> sortDataDir(File[] files, String prefix, boolean ascending)
{
if(files==null)
return new ArrayList<File>(0);
List<File> filelist = Arrays.asList(files);
Collections.sort(filelist, new DataDirFileComparator(prefix, ascending));
return filelist;
}
2 删除文件
// VisibleForTesting
static void retainNRecentSnapshots(FileTxnSnapLog txnLog, List<File> snaps) {
// found any valid recent snapshots?
if (snaps.size() == 0)
return;
File snapShot = snaps.get(snaps.size() -1);
int ii=snaps.size() -1;
System.out.println(ii);
final long leastZxidToBeRetain = Util.getZxidFromName(
snapShot.getName(), PREFIX_SNAPSHOT);
class MyFileFilter implements FileFilter{
private final String prefix;
MyFileFilter(String prefix){
this.prefix=prefix;
}
public boolean accept(File f){
if(!f.getName().startsWith(prefix + "."))
return false;
long fZxid = Util.getZxidFromName(f.getName(), prefix);
if (fZxid >= leastZxidToBeRetain) {
return false;
}
return true;
}
}
// add all non-excluded log files
List<File> files = new ArrayList<File>(Arrays.asList(txnLog
.getDataDir().listFiles(new MyFileFilter(PREFIX_LOG))));
// add all non-excluded snapshot files to the deletion list
files.addAll(Arrays.asList(txnLog.getSnapDir().listFiles(
new MyFileFilter(PREFIX_SNAPSHOT))));
// remove the old files
for(File f: files)
{
System.out.println("Removing file: "+
DateFormat.getDateTimeInstance().format(f.lastModified())+
"\t"+f.getPath());
if(!f.delete()){
System.err.println("Failed to remove "+f.getPath());
}
}
}
Util.getZxidFromName工具方法代码
public static long getZxidFromName(String name, String prefix) {
long zxid = -1;
String nameParts[] = name.split("\\.");
if (nameParts.length == 2 && nameParts[0].equals(prefix)) {
try {
zxid = Long.parseLong(nameParts[1], 16);
} catch (NumberFormatException e) {
}
}
return zxid;
}