我想知道,当我从CSV文件向数据库插入数据时,您是否可以帮助我处理一个错误。
问题是,我正在读取一个CSV文件,并将其分批处理。由于hibernate缓存内存中的所有对象,所以在完成一定数量的记录后,我执行如下操作:
private void cleanUpGorm() {
def session = sessionFactory.currentSession
session.flush()
session.clear()
}这是为了避免当许多对象存储到内存中时出现已知的性能问题。
问题是,对于一些CSV文件(15个中的4个),当调用session.flush() (会话不为null)时,我会得到一个session.flush。这些CSV文件没有任何奇怪的东西,编码是好的,没有坏的字符,等等。
我有两种方法:
@NotTransactional
void loadDataFromCSV() {
// Group the CSV lines into groups of 1000.
loadBatch(myGroupOfLines)
}
// No annotation here.. I assume that by default is Transactional
void loadBatch(myGroupOfLines) {
// Iterate over myGroupOfLines Inserting or updating the entries in the db.
// Clean GORM cache
if(index > 0 && index % CLEAN_GORM_CACHE_FREQUENCY == 0) {
log.info "**** Cleaning GORM cache ****"
cleanUpGorm()
}
}如果你能指导我解决这个问题,我将不胜感激,因为我从现在开始就在试图找到一个解决方案。
我得到的详细NPE是:
2017-01-19 18:00:52,440 - ERROR: GrailsExceptionResolver - NullPointerException occurred when processing request: [POST] /api/admin/upload - parameters:
entity: OCEAN_COSTS
Stacktrace follows:
java.lang.reflect.InvocationTargetException: null
at org.grails.core.DefaultGrailsControllerClass$ReflectionInvoker.invoke(DefaultGrailsControllerClass.java:210)
at org.grails.core.DefaultGrailsControllerClass.invoke(DefaultGrailsControllerClass.java:187)
at org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter.handle(UrlMappingsInfoHandlerAdapter.groovy:90)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77)
at org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: null
at org.hibernate.engine.spi.BatchFetchQueue.removeBatchLoadableEntityKey(BatchFetchQueue.java:147)
at org.hibernate.engine.internal.StatefulPersistenceContext.addEntity(StatefulPersistenceContext.java:387)
at org.hibernate.engine.internal.StatefulPersistenceContext.addEntity(StatefulPersistenceContext.java:462)
at org.hibernate.action.internal.AbstractEntityInsertAction.makeEntityManaged(AbstractEntityInsertAction.java:126)
at org.hibernate.engine.spi.ActionQueue.addResolvedEntityInsertAction(ActionQueue.java:279)
at org.hibernate.engine.spi.ActionQueue.addInsertAction(ActionQueue.java:254)
at org.hibernate.engine.spi.ActionQueue.addAction(ActionQueue.java:299)
at org.hibernate.event.internal.AbstractSaveEventListener.addInsertAction(AbstractSaveEventListener.java:318)
at org.hibernate.event.internal.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:275)
at org.hibernate.event.internal.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:182)
at org.hibernate.event.internal.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:113)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:192)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:177)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.performSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:97)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:73)
at org.grails.orm.hibernate.support.ClosureEventTriggeringInterceptor.onSaveOrUpdate(ClosureEventTriggeringInterceptor.java:81)
at org.hibernate.internal.SessionImpl.fireSaveOrUpdate(SessionImpl.java:651)
at org.hibernate.internal.SessionImpl.saveOrUpdate(SessionImpl.java:643)
at org.hibernate.engine.spi.CascadingActions$5.cascade(CascadingActions.java:218)
at org.hibernate.engine.internal.Cascade.cascadeToOne(Cascade.java:391)
at org.hibernate.engine.internal.Cascade.cascadeAssociation(Cascade.java:316)
at org.hibernate.engine.internal.Cascade.cascadeProperty(Cascade.java:155)
at org.hibernate.engine.internal.Cascade.cascadeCollectionElements(Cascade.java:424)
at org.hibernate.engine.internal.Cascade.cascadeCollection(Cascade.java:356)
at org.hibernate.engine.internal.Cascade.cascadeAssociation(Cascade.java:319)
at org.hibernate.engine.internal.Cascade.cascadeProperty(Cascade.java:155)
at org.hibernate.engine.internal.Cascade.cascade(Cascade.java:104)
at org.hibernate.event.internal.AbstractFlushingEventListener.cascadeOnFlush(AbstractFlushingEventListener.java:150)
at org.hibernate.event.internal.AbstractFlushingEventListener.prepareEntityFlushes(AbstractFlushingEventListener.java:141)
at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:74)
at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:38)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295)
at maersk.calculator.api.MaritimeTransportationService$$EQ8n1X8K.cleanUpGorm(MaritimeTransportationService.groovy:401)
at maersk.calculator.api.MaritimeTransportationService$__tt__loadBatch_closure10$$EQ8n1X8K.doCall(MaritimeTransportationService.groovy:227)
at maersk.calculator.api.MaritimeTransportationService$$EQ8n1X8K.$tt__loadBatch(MaritimeTransportationService.groovy:200)
at grails.transaction.GrailsTransactionTemplate$2.doInTransaction(GrailsTransactionTemplate.groovy:96)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
at grails.transaction.GrailsTransactionTemplate.execute(GrailsTransactionTemplate.groovy:93)
at maersk.calculator.api.MaritimeTransportationService$$EQ8n1X8K.loadDataFromCSV(MaritimeTransportationService.groovy:139)
at maersk.calculator.api.UploadController.triggerJob(UploadController.groovy:88)
at maersk.calculator.api.UploadController.saveFile(UploadController.groovy:55)
... 14 common frames omitted编辑
实际上,我可以隔离这个问题,它在loadBatch方法中。它似乎与#9290有关
错误发生在以下代码中:
mtc = new MaritimeTransportationCharge(charge, amount, currency, container, mt)
mt.addToCharges(mtc)
if(!mt.save()) {
errors.add("Error in line $line: Invalid charge for container ${container}.\nErrors: ${Utils.getEntityErrors(mtc)}")
}似乎这些关联不再对齐(从Grails 2开始),并且正在发生一些事情,可能子对象无效。
这些关系是:
MaritimeTransportation hasMany: MaritimeTransportationCharges
MaritimeTransportationCharge belongsTo: MaritimeTransportation发布于 2017-01-21 21:55:23
我建议在application.yml中添加SQL日志记录。
environments:
local:
dataSource:
logSql: true
formatSql: true我在升级到grails 3时看到的是,GORM完成的一些自动表映射可能是错误的,所以请检查它在日志中使用的表是否正确。如果发现它们不正确,则可以在域类中显式地设置它们。
MaritimeTransportationCharges joinTable: [name:"join_table_name", key: "key", column: "column"]https://stackoverflow.com/questions/41754753
复制相似问题