我有一个简单的csv文件,包含大约400,000行(仅一列)--我需要很长的时间来读取和处理这些记录
用couchbase验证记录的处理器
作者写到遥远的话题花了我大约30分钟。太疯狂了。
我读到flatfileItemreader并不是线程安全的。所以我的块值是1。
我读了异步处理可以帮助。但我看不出有什么改进。
这是我的代码:
@Configuration
@EnableBatchProcessing
public class NotificationFileProcessUploadedFileJob {
    @Value("${expected.snid.header}")
    public String snidHeader;
    @Value("${num.of.processing.chunks.per.file}")
    public int numOfProcessingChunksPerFile;
    @Autowired
    private InfrastructureConfigurationConfig infrastructureConfigurationConfig;
    private static final String OVERRIDDEN_BY_EXPRESSION = null;
    @Inject
    private JobBuilderFactory jobs;
    @Inject
    private StepBuilderFactory stepBuilderFactory;
    @Inject
    ExecutionContextPromotionListener executionContextPromotionListener;
    @Bean
    public Job processUploadedFileJob() throws Exception {
        return this.jobs.get("processUploadedFileJob").start((processSnidUploadedFileStep())).build();
    }
    @Bean
    public Step processSnidUploadedFileStep() {
        return stepBuilderFactory.get("processSnidFileStep")
                .<PushItemDTO, PushItemDTO>chunk(numOfProcessingChunksPerFile)
                .reader(snidFileReader(OVERRIDDEN_BY_EXPRESSION))
                .processor(asyncItemProcessor())
                .writer(asyncItemWriter())
            //    .throttleLimit(20)
             //   .taskJobExecutor(infrastructureConfigurationConfig.taskJobExecutor())
                        //     .faultTolerant()
                        //   .skipLimit(10) //default is set to 0
                        //     .skip(MySQLIntegrityConstraintViolationException.class)
                .build();
    }
    @Inject
    ItemWriter writer;
    @Bean
    public AsyncItemWriter asyncItemWriter() {
        AsyncItemWriter asyncItemWriter=new AsyncItemWriter();
        asyncItemWriter.setDelegate(writer);
        return asyncItemWriter;
    }
    @Bean
    @Scope(value = "step", proxyMode = ScopedProxyMode.INTERFACES)
    public ItemStreamReader<PushItemDTO> snidFileReader(@Value("#{jobParameters[filePath]}") String filePath) {
        FlatFileItemReader<PushItemDTO> itemReader = new FlatFileItemReader<PushItemDTO>();
        itemReader.setLineMapper(snidLineMapper());
        itemReader.setLinesToSkip(1);
        itemReader.setResource(new FileSystemResource(filePath));
        return itemReader;
    }
    @Bean
    public AsyncItemProcessor asyncItemProcessor() {
        AsyncItemProcessor<PushItemDTO, PushItemDTO> asyncItemProcessor = new AsyncItemProcessor();
        asyncItemProcessor.setDelegate(processor(OVERRIDDEN_BY_EXPRESSION, OVERRIDDEN_BY_EXPRESSION, OVERRIDDEN_BY_EXPRESSION,
                OVERRIDDEN_BY_EXPRESSION, OVERRIDDEN_BY_EXPRESSION, OVERRIDDEN_BY_EXPRESSION, OVERRIDDEN_BY_EXPRESSION));
        asyncItemProcessor.setTaskExecutor(infrastructureConfigurationConfig.taskProcessingExecutor());
        return asyncItemProcessor;
    }
    @Scope(value = "step", proxyMode = ScopedProxyMode.INTERFACES)
    @Bean
    public ItemProcessor<PushItemDTO, PushItemDTO> processor(@Value("#{jobParameters[pushMessage]}") String pushMessage,
                                                             @Value("#{jobParameters[jobId]}") String jobId,
                                                             @Value("#{jobParameters[taskId]}") String taskId,
                                                             @Value("#{jobParameters[refId]}") String refId,
                                                             @Value("#{jobParameters[url]}") String url,
                                                             @Value("#{jobParameters[targetType]}") String targetType,
                                                             @Value("#{jobParameters[gameType]}") String gameType) {
        return new PushItemProcessor(pushMessage, jobId, taskId, refId, url, targetType, gameType);
    }
    @Bean
    public LineMapper<PushItemDTO> snidLineMapper() {
        DefaultLineMapper<PushItemDTO> lineMapper = new DefaultLineMapper<PushItemDTO>();
        DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
        lineTokenizer.setDelimiter(",");
        lineTokenizer.setStrict(true);
        lineTokenizer.setStrict(true);
        String[] splittedHeader = snidHeader.split(",");
        lineTokenizer.setNames(splittedHeader);
        BeanWrapperFieldSetMapper<PushItemDTO> fieldSetMapper = new BeanWrapperFieldSetMapper<PushItemDTO>();
        fieldSetMapper.setTargetType(PushItemDTO.class);
        lineMapper.setLineTokenizer(lineTokenizer);
        lineMapper.setFieldSetMapper(new PushItemFieldSetMapper());
        return lineMapper;
    }
}
 @Bean
    @Override
    public SimpleAsyncTaskExecutor taskProcessingExecutor() {
        SimpleAsyncTaskExecutor simpleAsyncTaskExecutor = new SimpleAsyncTaskExecutor();
        simpleAsyncTaskExecutor.setConcurrencyLimit(300);
        return simpleAsyncTaskExecutor;
    }你认为我怎样才能提高加工性能,使它们更快?谢谢
ItemWriter代码:
 @Bean
    public ItemWriter writer() {
        return new KafkaWriter();
    }
public class KafkaWriter implements ItemWriter<PushItemDTO> {
    private static final Logger logger = LoggerFactory.getLogger(KafkaWriter.class);
    @Autowired
    KafkaProducer kafkaProducer;
    @Override
    public void write(List<? extends PushItemDTO> items) throws Exception {
        for (PushItemDTO item : items) {
            try {
                logger.debug("Writing to kafka=" + item);
                sendMessageToKafka(item);
            } catch (Exception e) {
                logger.error("Error writing item=" + item.toString(), e);
            }
        }
    }发布于 2015-02-19 17:01:42
增加你的提交数量是我要开始的地方。记住提交计数是什么意思。由于将其设置为1,因此对每个项目执行以下操作
您的配置没有显示委托ItemWriter是什么,所以我无法判断,但至少您要执行多个语句(每项)来更新作业存储库。
您是正确的,因为FlatFileItemReader并不是线程安全的。但是,您没有使用多个线程来读取,而是使用进程,因此没有理由根据我所看到的将提交计数设置为1。
https://stackoverflow.com/questions/28606634
复制相似问题