在大数据集中查找重复项
在大数据集中查找重复项
提问于 2017-01-10 16:56:28
我有一个数据集，里面有关于控制系统故障的数据。这些数据的结构如下：
TYPE OF FAILURE (string), START DATE (dd/mm/yyyy), START TIME (hh/mm/ss), DURATION (ss), LOCALIZATION (string), WORKING TEAM (A,B,C), SHIFT (morning, afternoon, night)包含数据的表有555000行。首先，我想分析一下关于开始日期参数是否有重复的故障序列。基本上，我希望找到这样的东西：
3月10日出现故障1，3月15日出现故障2，间隔5天。然后在4月10日和4月15日出现了故障1，这也是它们之间的5天。故障1出现在5月10日和5月15日之间，间隔为5天。然而，故障1也可能出现在不同的日期，但对我来说，有趣的是，有更大的可能性，故障2将在故障1之后5天出现，并且这些事件(F1->F2)之间的时间间隔为一个月。
我不知道我的解释是否足够清楚。然而，我正在寻找合适的方法/算法，以便能够从上面描述的数据中提取这样的序列。你能告诉我一些方法吗？或者让我们一起集思广益:)。感谢您的帮助。
PS:我计划用C#或MATLAB (取决于合适的方法)实现这一点，谢谢。
回答 1
Stack Overflow用户
发布于 2017-01-10 19:07:38
你的文件看起来像一个很大的CSV，因为matlab有一个很好的Data Store实现
https://es.mathworks.com/help/matlab/import_export/what-is-a-datastore.html
并具有用于处理大文件的以下工具：
https://es.mathworks.com/help/matlab/large-files-and-big-data.html
，还要看一看working with tables in matlab。
在您的示例中，您可以执行以下操作：
示例文件airlinessmall.csv (123524行)
Year,Month,DayofMonth,DayOfWeek,DepTime,CRSDepTime,ArrTime,CRSArrTime,UniqueCarrier,FlightNum,TailNum,ActualElapsedTime,CRSElapsedTime,AirTime,ArrDelay,DepDelay,Origin,Dest,Distance,TaxiIn,TaxiOut,Cancelled,CancellationCode,Diverted,CarrierDelay,WeatherDelay,NASDelay,SecurityDelay,LateAircraftDelay
1987,10,21,3,642,630,735,727,PS,1503,NA,53,57,NA,8,12,LAX,SJC,308,NA,NA,0,NA,0,NA,NA,NA,NA,NA
1987,10,26,1,1021,1020,1124,1116,PS,1550,NA,63,56,NA,8,1,SJC,BUR,296,NA,NA,0,NA,0,NA,NA,NA,NA,NA
1987,10,23,5,2055,2035,2218,2157,PS,1589,NA,83,82,NA,21,20,SAN,SMF,480,NA,NA,0,NA,0,NA,NA,NA,NA,NA
1987,10,23,5,1332,1320,1431,1418,PS,1655,NA,59,58,NA,13,12,BUR,SJC,296,NA,NA,0,NA,0,NA,NA,NA,NA,NA
1987,10,22,4,629,630,746,742,PS,1702,NA,77,72,NA,4,-1,SMF,LAX,373,NA,NA,0,NA,0,NA,NA,NA,NA,NA
1987,10,28,3,1446,1343,1547,1448,PS,1729,NA,61,65,NA,59,63,LAX,SJC,308,NA,NA,0,NA,0,NA,NA,NA,NA,NA
1987,10,8,4,928,930,1052,1049,PS,1763,NA,84,79,NA,3,-2,SAN,SFO,447,NA,NA,0,NA,0,NA,NA,NA,NA,NA
1987,10,10,6,859,900,1134,1123,PS,1800,NA,155,143,NA,11,-1,SEA,LAX,954,NA,NA,0,NA,0,NA,NA,NA,NA,NA
...有了数据存储，你可以将数据作为表格来处理，并获得所需的变量，例如，要获得到达延迟的平均值：
>> ds = datastore('airlinesmall.csv','TreatAsMissing','NA');
>> ds.MissingValue = 0;
>> ds.SelectedVariableNames = 'ArrDelay';
>> data = preview(ds)
data =
ArrDelay
________
8
8
21
13
4
59
3
11
>> data % this is a table
data =
ArrDelay
________
8
8
21
13
4
59
3
11
>> sums = [];
counts = [];
while hasdata(ds)
T = read(ds); % this is a table, but this is not all loaded in memory
sums(end+1) = sum(T.ArrDelay);
counts(end+1) = length(T.ArrDelay);
end
>> avgArrivalDelay = sum(sums)/sum(counts)
avgArrivalDelay =
6.9670让我们来处理你的样本。检查此文件：
sample.csv
TYPE OF FAILURE, START DATE, START TIME, DURATION, LOCALIZATION, WORKING TEAM, SHIFT
failure 1, 06/01/2017, 12/13/20, 300, Area 1, A, morning
failure 2, 06/01/2017, 12/13/20, 300, Area 1, A, night
failure 3, 06/01/2017, 12/13/20, 400, Area 1, A, afternoon
failure 1, 08/01/2017, 12/13/20, 300, Area 1, A, morning
failure 2, 09/01/2017, 12/13/20, 300, Area 1, A, morning
failure 3, 09/01/2017, 12/13/20, 300, Area 1, A, night
failure 3, 09/01/2017, 14/13/20, 200, Area 1, A, morning
failure 1, 10/01/2017, 12/13/20, 300, Area 1, A, morning
failure 1, 12/01/2017, 12/13/20, 300, Area 1, A, afternoon
failure 2, 12/01/2017, 12/13/20, 500, Area 1, A, morning
failure 1, 14/01/2017, 12/13/20, 300, Area 1, A, night你可以看到失败1是每隔两天发生一次，让我们看看这个：
>> ds = tabularTextDatastore('sample.csv')
Warning: Variable names were modified to make them valid MATLAB identifiers.
ds =
TabularTextDatastore with properties:
Files: {
'/home/anquegi/learn/matlab/stackoverflow/sample.csv'
}
FileEncoding: 'UTF-8'
ReadVariableNames: true
VariableNames: {'TYPEOFFAILURE', 'STARTDATE', 'STARTTIME' ... and 4 more}
Text Format Properties:
NumHeaderLines: 0
Delimiter: ','
RowDelimiter: '\r\n'
TreatAsMissing: ''
MissingValue: NaN
Advanced Text Format Properties:
TextscanFormats: {'%q', '%q', '%q' ... and 4 more}
ExponentCharacters: 'eEdD'
CommentStyle: ''
Whitespace: ' \b\t'
MultipleDelimitersAsOne: false
Properties that control the table returned by preview, read, readall:
SelectedVariableNames: {'TYPEOFFAILURE', 'STARTDATE', 'STARTTIME' ... and 4 more}
SelectedFormats: {'%q', '%q', '%q' ... and 4 more}
ReadSize: 20000 rows
>> ds.SelectedVariableNames = {'TYPEOFFAILURE', 'STARTDATE', 'STARTTIME', 'DURATION', 'LOCALIZATION', 'WORKINGTEAM', 'SHIFT'}
ds =
TabularTextDatastore with properties:
Files: {
'/home/anquegi/learn/matlab/stackoverflow/sample.csv'
}
FileEncoding: 'UTF-8'
ReadVariableNames: true
VariableNames: {'TYPEOFFAILURE', 'STARTDATE', 'STARTTIME' ... and 4 more}
Text Format Properties:
NumHeaderLines: 0
Delimiter: ','
RowDelimiter: '\r\n'
TreatAsMissing: ''
MissingValue: NaN
Advanced Text Format Properties:
TextscanFormats: {'%q', '%q', '%q' ... and 4 more}
ExponentCharacters: 'eEdD'
CommentStyle: ''
Whitespace: ' \b\t'
MultipleDelimitersAsOne: false
Properties that control the table returned by preview, read, readall:
SelectedVariableNames: {'TYPEOFFAILURE', 'STARTDATE', 'STARTTIME' ... and 4 more}
SelectedFormats: {'%q', '%q', '%q' ... and 4 more}
ReadSize: 20000 rows
>> reset(ds)
accum = [];
while hasdata(ds)
T = read(ds);
accum = datetime(T(strcmp(T.TYPEOFFAILURE,'failure 1'),:).STARTDATE, 'InputFormat','dd/MM/yyyy');
mean(diff(accum))
end
ans =
48:00:00%恰好每48小时，然后你可以尝试任何你想要的东西
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/41564809
复制相关文章
相似问题
腾讯云开发者
