首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >Python文本去重

Python文本去重

作者头像
py3study
发布2020-01-06 14:28:54
2.5K0
发布2020-01-06 14:28:54
举报
文章被收录于专栏:python3python3

用法:命令行python unique.py -f file.txt 输出:去除重复字符后的output.txt

# -*- coding:utf-8 -*-
#auther_cclarence_2016_4_6
#open file and read out the characters
from optparse import OptionParser
import sys
reload(sys)
sys.setdefaultencoding("utf-8")
def readfile(filename):
    try:
        f = open(filename)
    except Exception, e:
        print "No such file"
        exit(0)
    text = f.readlines()
    f.close()
    for i in range(0,len(text)-1):
        text[i] = text[i][:-1]
    return text
#deduplication
def unique(arr):
    arr1 = list(set(arr))
    arr1.sort(key = arr.index)
    return arr1
def main():
    parser = OptionParser()
    parser.add_option("-f", "--file", dest="filename",help="write report to FILE", metavar="FILE")
    (options, args) = parser.parse_args()
    filename = options.filename
    text = readfile(filename)
    text_dealed = unique(text)
    for i in range(0,len(text_dealed)-1):
        text_dealed[i] = text_dealed[i] +'\n'
    f = open("output.txt","w")
    f.writelines(text_dealed)
    f.close()
    deduplication_num = len(text) - len(text_dealed) 
    print "success"
    print "The num of data from the source file        :" + str(len(text))
    print "The num of data from the preprocessed file: :" + str(len(text_dealed))
    print "The num of data removed                     :" + str(deduplication_num)
if __name__ == '__main__':
    main()
本文参与 腾讯云自媒体分享计划,分享自作者个人站点/博客。
原始发表:2019-09-18 ,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档