首页
学习
活动
专区
工具
TVP
发布
社区首页 >问答首页 >如何在Excel VBA中加速数据文件导入

如何在Excel VBA中加速数据文件导入
EN

Stack Overflow用户
提问于 2019-05-31 00:17:35
回答 3查看 946关注 0票数 0

截至2019年6月11日的更新:我仍然没有弄清楚为什么我所有的延迟都发生在这两条线上,但目前的状态是我忍受了延迟。到目前为止,我在主文档中有大约6000行数据,无论我导入多少行,导入过程都需要大约20秒。

-

我有一个“主文档”,我整天从很多很多的小文档中导入数据。我承认我在这方面不是一个超级天才,而且我的很多编程习惯都是“老一套”的,所以可能有一些我不知道的"Excel方法“(但我想学习!)。

我看到的问题是数据文件导入需要多长时间。

当我开始使用这个工具时,数据导入只需要几秒钟。

现在我有大约3500行数据,数据导入大约需要15-20秒。不管我是导入一行还是导入100行。我预计这个数字还会继续上升。当我读到7000行或10,000行时,我希望它会变得无法忍受。

通过使用消息框(记住:“老式”),我已经能够将速度瓶颈缩小到两行代码。“步骤1”和“步骤2”之间的延迟大约是我延迟的30%,而“步骤2”和“步骤3”之间的延迟大约是我延迟的70%。

为了确保我没有遗漏一些明显的东西,我在下面包含了整个sub,但我确保取消我的消息框,以便您可以转到r-i-g-h-t我怀疑的代码。此外,我包含了整个子集,因为通常第一个响应之一是“您能显示整个子集以便我有更好的上下文吗?”

感谢您的任何想法或建议。:)

Private Sub Btn_ImportDataFiles_Click()
  ' Search the current worksheet and assign the next TransactionID
    Dim TransactionCounter As Integer
    Dim TransactionID As Long ' This is the next available Transaction ID
    TransactionID = Application.WorksheetFunction.Max(Range("a:a")) + 1
  ' open the file and import the data
    Dim customerBook As Workbook
    Dim filter As String
    Dim caption As String
    Dim customerFilename As String
    Dim customerWorkbook As Workbook
    Dim targetWorkbook As Workbook

    ' make weak assumption that active workbook is the target
      Set targetWorkbook = Application.ActiveWorkbook

    ' get the customer workbook
      filter = "Text files (*.xlsx),*.xlsx"
      caption = "Please Select an input file "
      customerFilename = Application.GetOpenFilename(filter, , caption)

    If customerFilename <> "False" Then
    ' If they have uploaded the file before, let them know.
    ' If they want to keep uploading it, no harm done,
    ' but no need to stupidly add data that is already present.
    ' Select the archive sheet
      Sheets("Upload_Archive").Select
      Dim FileNameHunt As String
      Dim cell As Range
      Dim ContinueUpload As Boolean
      ContinueUpload = True
      FileNameHunt = Mid(customerFilename, InStrRev(customerFilename, "\") + 1)
      Columns("A:A").Select
      Set cell = Selection.Find(what:=FileNameHunt, after:=ActiveCell, LookIn:=xlFormulas, lookat:=xlWhole, searchorder:=xlByRows, searchdirection:=xlNext, MatchCase:=False, searchformat:=False)
      If cell Is Nothing Then ' Add the new filename to the archive
        Sheets("Upload_Archive").Select
        Rows(1).Insert shift:=xlDown
        Range("a1:a1").Value = FileNameHunt
        Sheets("MasterSheet").Select
        Application.Cells.Font.Name = "Calibri Light"
        Application.Cells.Font.Size = "8"
        Application.Cells.Font.Bold = False
      Else
        response = MsgBox("This data file has previously been uploaded. " & vbCrLf & "Do you want to cancel this upload?" & vbCrLf & vbCrLf & "Pressing [yes] will cancel the process." & vbCrLf & "Pressing [no] will continue with the file upload" & vbCrLf & "and add the data to the tracking sheet.", vbYesNo)
        If response = vbYes Then
          ContinueUpload = False
          Sheets("MasterSheet").Select
          Exit Sub
        End If
      End If ' If cell Is Nothing Then...

      If ContinueUpload = True Then
        ' Continue with data upload procedure
          Sheets("MasterSheet").Select
          Set customerWorkbook = Application.Workbooks.Open(customerFilename)
        ' Copy data from customer to target workbook
          Dim targetSheet As Worksheet
          Set targetSheet = targetWorkbook.Worksheets(1)
          Dim sourceSheet As Worksheet
          Set sourceSheet = customerWorkbook.Worksheets(1)
          Dim ImportRecordCount As Integer
          ImportRecordCount = sourceSheet.Range("B1")
          Dim ReconciliationID As String
          ReconciliationID = ""
          If sourceSheet.Range("E3") = "Removed from Depot" Then ReconciliationID = "1"
MsgBox ("Step 1")
          targetSheet.Range("A1").EntireRow.Offset(1).Resize(ImportRecordCount).Insert shift:=xlDown ' Add the blank rows
MsgBox ("Step 2")
          targetSheet.Range("B2:AB" & ImportRecordCount + 1).Value = sourceSheet.Range("A3:AA" & ImportRecordCount + 2).Value ' Bring in the big pile of data
MsgBox ("Step 3")
          targetSheet.Range("AJ2:AJ" & ImportRecordCount + 1).Value = ReconciliationID ' To help with reconciling shipments
          targetSheet.Range("AK2:AK" & ImportRecordCount + 1).Value = ReconciliationID ' To help with deployment timing
          'targetSheet.Range("AI2:AI" & ImportRecordCount + 1).Value = "=COUNTIFS($D:$D, D2, $F:$F, F2)" ' This is the helper formula for identifying duplicates (deprecated, but I'm saving the code)
          For TransactionCounter = 2 To ImportRecordCount + 1 ' Create and add the new Transaction ID values
            targetSheet.Range("a" & TransactionCounter) = TransactionID + ImportRecordCount - TransactionCounter + 1
          Next
        ' Close customer workbook
          customerWorkbook.Close
        ' Format the sheet properly
          Application.Cells.Font.Name = "Calibri Light"
          Application.Cells.Font.Size = "8"
          Application.Cells.Font.Bold = False
          Application.Range("1:1").Font.Size = "10"
          Application.Range("1:1").Font.Bold = True
        ' Query the User -- delete the file?
          If MsgBox("Delete the local client-generated data file?" & vbCrLf & vbCrLf & "(this will NOT affect your email)", vbYesNo, "Confirm") = vbYes Then
            Kill customerFilename
            ' MsgBox ("File: " & vbCrLf & customerFilename & vbCrLf & "has been deleted.")
          End If
      End If ' If ContinueUpload = True Then
    End If ' If customerFilename <> "False" Then

End Sub

编辑

我编辑了您的原始问题,以突出显示我发现的可疑内容。这些都是我觉得值得向你指出的事情。为了专注于这些特定的问题,我剔除了所有其他的东西。回顾它们,做一些研究,看看你是否能发现自己处于更好的境地。

    MsgBox ("Step 2")

        'Ive never moved large amounts of data using this method. Ive always just used arrays. I have moved smaller bits of data though.
        ' I suspect that this might take a moment if the data set is large. Again use arrays to grab the data and move it.
        ' Edward says “This step takes about 70% of my delay — even if bringing in only a single line of data.”

        targetSheet.Range("B2:AB" & ImportRecordCount + 1).Value = sourceSheet.Range("A3:AA" & ImportRecordCount + 2).Value ' Bring in the big pile of data

    MsgBox ("Step 3")

      ' this loop is probably your main culprit of your performance issue. 
      ' Edward says “Nope, this flies by. It is not the issue at all. I have verified this already.”
      ' Learn how to construct an array of data on the fly and then learn how to dump the entire array to 
      ' sheet using a simple method.

        For TransactionCounter = 2 To ImportRecordCount + 1 ' Create and add the new Transaction ID values
            targetSheet.Range("a" & TransactionCounter) = TransactionID + ImportRecordCount - TransactionCounter + 1
        Next
EN

回答 3

Stack Overflow用户

发布于 2019-05-31 00:56:55

看起来你在这里有很多好东西。我看到的一些事情可能会被改变,以提高您的性能。

首先,在“步骤1”和“步骤2”之间:根据我的经验,添加行比使用已经存在的行需要更长的时间。看起来您基本上是在向下“推”所有东西,以便为新数据腾出空间,这样新输入的数据在顶部,最旧的数据在底部。(如果我在这方面有任何错误,请纠正我。)如果您只是简单地将数据添加到工作表的末尾,您可能会看到一些性能改进,尽管我不知道它会有多大的改进。

其次,在“第2步”和“第3步”之间:我发现使用.Value2而不是.Value可以给您带来一些性能改进,并且数据越大,改进越大。这有一个缺点- Value2不保留任何可能存在的格式,这意味着数字类型(日期、会计等)不能正确拉出。如果这是您不需要的东西,那么您可以使用Value2。

最后,其他方法:当我运行大量的宏时,我总是尝试做我能做的一切来提高性能。你可以通过使用像关闭屏幕更新(Application.ScreenUpdating = False)这样的技巧来获得轻微的提升,只是要确保在宏的末尾重新打开它。

我希望这能帮助你解决这个问题!如果所有其他方法都失败了,您可以手动执行一到两次,以记住使用宏的速度有多快!哈哈。祝好运!

票数 0
EN

Stack Overflow用户

发布于 2020-03-25 22:42:23

你试过使用.value2吗?在某些情况下,它可能会给您带来更好的性能。在此处查看一些性能比较:https://fastexcel.wordpress.com/2011/11/30/text-vs-value-vs-value2-slow-text-and-how-to-avoid-it/

如果不能访问原始表,就很难看出问题出在哪里。也许问题出在数据本身,而不是您的VBA代码,有时您可能需要清除源数据中的大量内容,然后根据需要重新添加它。

你也可以考虑用Python来做一些工作,但是如果你不想在你的解决方案中添加额外的软件层,我想这是不可能的。

票数 0
EN

Stack Overflow用户

发布于 2019-05-31 02:00:03

尝试在脚本的开头和结尾添加此代码。只需确保将所有内容设置为TRUE!!

Application.ScreenUpdating = False
Application.DisplayAlerts = False

...CODE HERE...

Application.ScreenUpdating = True
Application.DisplayAlerts = True
票数 -1
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/56381960

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档