Spring Batch 5.1.0 ThottleLimit Causing Deadlock on ThreadPoolExecutor

Posted on Edited on

Spring Batch 5.1.0 ThottleLimit Causing Deadlock on ThreadPoolExecutor

Facts

In Spring Batch 5.1.0,

  1. The throttleLimit in Spring Batch is intended to cap the number of concurrent tasks submitted to the executor, but it is not a hard guarantee.
  2. The actual number of submitted tasks can exceed the throttleLimit if the executor rejects tasks (e.g., due to a full queue or pool).
  3. When using a ThreadPoolExecutor with AbortPolicy, rejected tasks throw a TaskRejectedException.
    1. Spring Batch logs this but does not retry or make any adjustment.
    2. Except output an error log, Spring Batch won’t provide any feedback to user.
    3. Spring Batch will still wait the task result from an failed submitted task, this can lock the whole process except the running subthread.

Adviced practice

  1. When using ThreadPoolExecutor + AbortPolicy (by default) ThrottleLimit need to be far less than ThreadPoolExecutor CorePoolSize, to avoid rejected tasks.
  2. When letting ThottleLimit >= CorePoolSize, we have to use ThreadPoolExecutor’s CallerRunsPolicy so rejected tasks are executed in the calling thread, ensuring all expected results are eventually produced.

Details

We can use the Spring official example to reproduce following situations:

1. Testing: DeadLock when Spring Batch submit more task than CorePoolSize, with ThreadPoolExecutor + AbortPolicy

1.1 Source code branch

test/v5.1.0/deadlock-when-spring-batch-submit-more-task-than-core-pool-size.

1.2 Source code change

Setting up the ThreadPoolExecutor like this:

1
2
3
4
5
6
7
8
9
10
@Bean
public ThreadPoolTaskExecutor threadPoolTaskExecutor(){
    ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
    threadPoolTaskExecutor.setCorePoolSize(2);
    threadPoolTaskExecutor.setMaxPoolSize(3);
    threadPoolTaskExecutor.setQueueCapacity(1);
    threadPoolTaskExecutor.setDaemon(true);
    threadPoolTaskExecutor.initialize();
    return threadPoolTaskExecutor;
}

While we set thottleLimit = 8 > MaxPoolSize + QueueCapacity = 4

1
2
.taskExecutor(threadPoolTaskExecutor())
.throttleLimit(8)

1.3 Diagram

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

Deadlock Case (task rejected):
+-------------+         +-------------------+         +---------------------+
| Main Thread |         | TaskExecutor      |         | Worker Thread(s)    |
+-------------+         +-------------------+         +---------------------+
      |                        |                               |
      | for (i < throttle)     |                               |
      |----------------------->|                               |
      | execute(runnable)      |                               |
      |----------------------->|                               |
      |   (TaskRejectedException thrown here)                  |
      |<---------------------- |                               |
      |   (logs error, does not retry)                         |
      |                        |                               |
      | waitForResults()       |                               |
      |-----                   |                               |
      |    |                   |                               |
      |<----                   |                               |
      | queue.take()           |                               |
      |-----                   |                               |
      |    |                   |                               |
      |<----                   |                               |
      | (waits forever for     |                               |
      |  missing result)       |                               |
      |   (DEADLOCK)           |                               |
      |                        |                               |

1.4 Running phenomenon

It will output a the following log and stuck there

See log detail 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
2025-07-06 17:59:54 INFO  [main] BatchRegistrar:70 - Finished Spring Batch infrastructure beans configuration in 0 ms.
2025-07-06 17:59:54 WARN  [main] PostProcessorRegistrationDelegate$BeanPostProcessorChecker:437 - Bean 'jobRegistry' of type [org.springframework.batch.core.configuration.support.MapJobRegistry] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying). Is this bean getting eagerly injected into a currently created BeanPostProcessor [jobRegistryBeanPostProcessor]? Check the corresponding BeanPostProcessor declaration and its dependencies.
2025-07-06 17:59:54 INFO  [main] EmbeddedDatabaseFactory:189 - Starting embedded database: url='jdbc:hsqldb:mem:d99a2b92-7e2d-4b5d-87a8-30a2855a354b', username='sa'
2025-07-06 17:59:54 INFO  [main] JobRepositoryFactoryBean:274 - No database type set, using meta data indicating: HSQL
2025-07-06 17:59:54 INFO  [main] TaskExecutorRepeatTemplate:79 - Set throttleLimit to 8
2025-07-06 17:59:54 INFO  [main] BatchObservabilityBeanPostProcessor:62 - No Micrometer observation registry found, defaulting to ObservationRegistry.NOOP
2025-07-06 17:59:54 INFO  [main] SimpleJobLauncher:232 - No TaskExecutor has been set, defaulting to synchronous executor.
2025-07-06 17:59:55 INFO  [main] SimpleJobLauncher:154 - Job: [SimpleJob: [name=ioSampleJob]] launched with the following parameters: [{'inputFile':'{value=org/springframework/batch/samples/file/delimited/data/delimited.csv, type=class java.lang.String, identifying=true}','outputFile':'{value=file:./target/test-outputs/delimitedOutput.csv, type=class java.lang.String, identifying=true}'}]
2025-07-06 17:59:55 INFO  [main] SimpleStepHandler:150 - Executing step: [step1]
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:133 - Entering RepeatStatus.iterate
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:161 - RepeatStatus.executeInternal
Thread main is setting throtleLimit 8 to ResultHolderResultQueue
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:127 - Submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:129 - Done submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:127 - Submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] TaskExecutorRepeatTemplate:259 - run method start
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:129 - Done submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:127 - Submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-2] TaskExecutorRepeatTemplate:259 - run method start
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:129 - Done submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:127 - Submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:129 - Done submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:127 - Submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-3] TaskExecutorRepeatTemplate:259 - run method start
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:298 - Handling exception: org.springframework.core.task.TaskRejectedException, caused by: org.springframework.core.task.TaskRejectedException: ExecutorService in active state did not accept task: org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@c4d2c44
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:224 - result is CONTINUABLE
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:162 - Entering waitForResults
2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:176 - Running queue.take()
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] RepeatTemplate:133 - Entering RepeatStatus.iterate
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-2] RepeatTemplate:133 - Entering RepeatStatus.iterate
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:133 - Entering RepeatStatus.iterate
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:161 - RepeatStatus.executeInternal
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-2] RepeatTemplate:161 - RepeatStatus.executeInternal
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] RepeatTemplate:161 - RepeatStatus.executeInternal
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] FlatFileItemReader:188 - Reading customer3,30
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-2] FlatFileItemReader:188 - Reading customer2,20
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-3] FlatFileItemReader:188 - Reading customer1,10
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-2] FlatFileItemReader:188 - Reading customer6,60
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-3] FlatFileItemReader:188 - Reading customer5,50
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] FlatFileItemReader:188 - Reading customer4,40
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-2] RepeatTemplate:224 - result is CONTINUABLE
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] RepeatTemplate:224 - result is CONTINUABLE
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:224 - result is CONTINUABLE
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] RepeatTemplate:383 - Entering no-op waitForResults
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-2] RepeatTemplate:383 - Entering no-op waitForResults
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:383 - Entering no-op waitForResults
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] RepeatTemplate:229 - Get result CONTINUABLE
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:229 - Get result CONTINUABLE
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-2] RepeatTemplate:229 - Get result CONTINUABLE
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] RepeatTemplate:149 - Quit RepeatStatus.iterate
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:149 - Quit RepeatStatus.iterate
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-2] RepeatTemplate:149 - Quit RepeatStatus.iterate
2025-07-06 17:59:55 WARN  [threadPoolTaskExecutor-3] TaskExecutorRepeatTemplate:276 - run method got error Conversion = ' '
2025-07-06 17:59:55 WARN  [threadPoolTaskExecutor-2] TaskExecutorRepeatTemplate:276 - run method got error Conversion = ' '
2025-07-06 17:59:55 WARN  [threadPoolTaskExecutor-1] TaskExecutorRepeatTemplate:276 - run method got error Conversion = ' '
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-2] ResultHolderResultQueue:99 - Put this in queue org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@2c6cfb64
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-3] ResultHolderResultQueue:99 - Put this in queue org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@6802dc5f
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] ResultHolderResultQueue:99 - Put this in queue org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@7e8cc5f8
2025-07-06 17:59:55 INFO  [main] ResultHolderResultQueue:138 - Queue size before take: 1
2025-07-06 17:59:55 INFO  [main] ResultHolderResultQueue:139 - Take value: isContinuable = false
2025-07-06 17:59:55 INFO  [main] ResultHolderResultQueue:140 - Queue size after take: 2
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] ResultHolderResultQueue:105 - Notify all
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] TaskExecutorRepeatTemplate:285 - run method end finally
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-3] ResultHolderResultQueue:105 - Notify all
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-3] TaskExecutorRepeatTemplate:285 - run method end finally
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] TaskExecutorRepeatTemplate:259 - run method start
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-2] ResultHolderResultQueue:105 - Notify all
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-2] TaskExecutorRepeatTemplate:285 - run method end finally
2025-07-06 17:59:55 INFO  [main] ResultHolderResultQueue:150 - Enter in waiting, count = 5 , queue.size() = 3
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] TaskExecutorRepeatTemplate:272 - run method end try
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] ResultHolderResultQueue:99 - Put this in queue org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@27a5151b
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] ResultHolderResultQueue:105 - Notify all
2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-1] TaskExecutorRepeatTemplate:285 - run method end finally
2025-07-06 17:59:55 INFO  [main] ResultHolderResultQueue:154 - Wake up, count = 5 , queue.size() = 4
2025-07-06 17:59:55 INFO  [main] ResultHolderResultQueue:150 - Enter in waiting, count = 5 , queue.size() = 4
Another log detail 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
 2025-07-06 19:03:56 INFO  [main] BatchRegistrar:70 - Finished Spring Batch infrastructure beans configuration in 0 ms.
2025-07-06 19:03:56 WARN  [main] PostProcessorRegistrationDelegate$BeanPostProcessorChecker:437 - Bean 'jobRegistry' of type [org.springframework.batch.core.configuration.support.MapJobRegistry] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying). Is this bean getting eagerly injected into a currently created BeanPostProcessor [jobRegistryBeanPostProcessor]? Check the corresponding BeanPostProcessor declaration and its dependencies.
2025-07-06 19:03:56 INFO  [main] EmbeddedDatabaseFactory:189 - Starting embedded database: url='jdbc:hsqldb:mem:51832490-316a-4a89-b4cc-57683b46885a', username='sa'
2025-07-06 19:03:56 INFO  [main] JobRepositoryFactoryBean:274 - No database type set, using meta data indicating: HSQL
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:79 - Set throttleLimit to 8
2025-07-06 19:03:56 INFO  [main] BatchObservabilityBeanPostProcessor:62 - No Micrometer observation registry found, defaulting to ObservationRegistry.NOOP
2025-07-06 19:03:56 INFO  [main] SimpleJobLauncher:232 - No TaskExecutor has been set, defaulting to synchronous executor.
2025-07-06 19:03:56 INFO  [main] SimpleJobLauncher:154 - Job: [SimpleJob: [name=ioSampleJob]] launched with the following parameters: [{'inputFile':'{value=org/springframework/batch/samples/file/delimited/data/delimited.csv, type=class java.lang.String, identifying=true}','outputFile':'{value=file:./target/test-outputs/delimitedOutput.csv, type=class java.lang.String, identifying=true}'}]
2025-07-06 19:03:56 INFO  [main] SimpleStepHandler:150 - Executing step: [step1]
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:133 - Entering RepeatStatus.iterate
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:161 - RepeatStatus.executeInternal
Thread main is setting throtleLimit 8 to ResultHolderResultQueue
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:127 - Submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:129 - Done submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:127 - Submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] TaskExecutorRepeatTemplate:259 - run method start
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:129 - Done submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:127 - Submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] TaskExecutorRepeatTemplate:259 - run method start
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:129 - Done submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:127 - Submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:129 - Done submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:127 - Submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] TaskExecutorRepeatTemplate:259 - run method start
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:298 - Handling exception: org.springframework.core.task.TaskRejectedException, caused by: org.springframework.core.task.TaskRejectedException: ExecutorService in active state did not accept task: org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@c4d2c44
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:224 - result is CONTINUABLE
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:162 - Entering waitForResults
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:176 - Running queue.take()
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:137 - Queue size before take: 0
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] RepeatTemplate:133 - Entering RepeatStatus.iterate
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:133 - Entering RepeatStatus.iterate
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] RepeatTemplate:133 - Entering RepeatStatus.iterate
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] RepeatTemplate:161 - RepeatStatus.executeInternal
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:161 - RepeatStatus.executeInternal
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] RepeatTemplate:161 - RepeatStatus.executeInternal
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] FlatFileItemReader:188 - Reading customer1,10
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] FlatFileItemReader:188 - Reading customer3,30
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] FlatFileItemReader:188 - Reading customer2,20
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] FlatFileItemReader:188 - Reading customer5,50
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] FlatFileItemReader:188 - Reading customer4,40
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] FlatFileItemReader:188 - Reading customer6,60
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] RepeatTemplate:224 - result is CONTINUABLE
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] RepeatTemplate:224 - result is CONTINUABLE
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:224 - result is CONTINUABLE
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] RepeatTemplate:383 - Entering no-op waitForResults
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] RepeatTemplate:383 - Entering no-op waitForResults
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:383 - Entering no-op waitForResults
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] RepeatTemplate:229 - Get result CONTINUABLE
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] RepeatTemplate:229 - Get result CONTINUABLE
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:229 - Get result CONTINUABLE
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] RepeatTemplate:149 - Quit RepeatStatus.iterate
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] RepeatTemplate:149 - Quit RepeatStatus.iterate
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:149 - Quit RepeatStatus.iterate
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] AbstractFileItemWriter:76 - Writing CustomerCredit [id=0,name=customer1, credit=15] 
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] AbstractFileItemWriter:76 - Writing CustomerCredit [id=0,name=customer2, credit=25] 
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] AbstractFileItemWriter:76 - Writing CustomerCredit [id=0,name=customer3, credit=35] 
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] AbstractFileItemWriter:76 - Writing CustomerCredit [id=0,name=customer5, credit=55] 
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] AbstractFileItemWriter:76 - Writing CustomerCredit [id=0,name=customer6, credit=65] 
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] AbstractFileItemWriter:76 - Writing CustomerCredit [id=0,name=customer4, credit=45] 
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] TaskExecutorRepeatTemplate:272 - run method end try
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] ResultHolderResultQueue:99 - Put this in queue org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@1635497b
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:139 - Take value: isContinuable = true
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:140 - Queue size after take: 0
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:143 - Before count -- : 5
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:145 - After count -- : 4
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:178 - Get org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@1635497b from queue.take()
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] ResultHolderResultQueue:105 - Notify all
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:192 - Result status
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] TaskExecutorRepeatTemplate:272 - run method end try
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] TaskExecutorRepeatTemplate:285 - run method end finally
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:176 - Running queue.take()
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] ResultHolderResultQueue:99 - Put this in queue org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@62c14c4b
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] TaskExecutorRepeatTemplate:259 - run method start
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:137 - Queue size before take: 0
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:139 - Take value: isContinuable = true
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:140 - Queue size after take: 0
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:143 - Before count -- : 4
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:145 - After count -- : 3
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:133 - Entering RepeatStatus.iterate
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:161 - RepeatStatus.executeInternal
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:178 - Get org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@62c14c4b from queue.take()
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] ResultHolderResultQueue:105 - Notify all
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:192 - Result status
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-2] TaskExecutorRepeatTemplate:285 - run method end finally
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] FlatFileItemReader:188 - Reading customer7,10
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:176 - Running queue.take()
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:137 - Queue size before take: 0
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] TaskExecutorRepeatTemplate:272 - run method end try
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] FlatFileItemReader:188 - Reading customer8,20
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] ResultHolderResultQueue:99 - Put this in queue org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@77723a53
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:139 - Take value: isContinuable = true
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:140 - Queue size after take: 0
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:224 - result is CONTINUABLE
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:143 - Before count -- : 3
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:383 - Entering no-op waitForResults
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:145 - After count -- : 2
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:229 - Get result CONTINUABLE
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] ResultHolderResultQueue:105 - Notify all
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:178 - Get org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@77723a53 from queue.take()
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] RepeatTemplate:149 - Quit RepeatStatus.iterate
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-1] TaskExecutorRepeatTemplate:285 - run method end finally
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:192 - Result status
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:176 - Running queue.take()
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] AbstractFileItemWriter:76 - Writing CustomerCredit [id=0,name=customer7, credit=15] 
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:137 - Queue size before take: 0
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] AbstractFileItemWriter:76 - Writing CustomerCredit [id=0,name=customer8, credit=25] 
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] TaskExecutorRepeatTemplate:272 - run method end try
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] ResultHolderResultQueue:99 - Put this in queue org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@70eedc13
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:139 - Take value: isContinuable = true
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:140 - Queue size after take: 0
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:143 - Before count -- : 2
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:145 - After count -- : 1
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:178 - Get org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@70eedc13 from queue.take()
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] ResultHolderResultQueue:105 - Notify all
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:192 - Result status
2025-07-06 19:03:56 INFO  [threadPoolTaskExecutor-3] TaskExecutorRepeatTemplate:285 - run method end finally
2025-07-06 19:03:56 INFO  [main] TaskExecutorRepeatTemplate:176 - Running queue.take()
2025-07-06 19:03:56 INFO  [main] ResultHolderResultQueue:137 - Queue size before take: 0

1.5 Running explanation

  1. Main thread goes to TaskExecutorRepeatTemplate
  2. Main thread submit 4 times task, see TaskExecutorRepeatTemplate
  3. When reached the 5th task, it goes error due to ThreadPool raised TaskRejectedException, but Spring Batch did nothing. 
    1
    2
    3
    2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:127 - Submitting jobs org.springframework.batch.core.step.tasklet.TaskletStep$2@2ee39e73
    2025-07-06 17:59:55 INFO  [threadPoolTaskExecutor-3] TaskExecutorRepeatTemplate:259 - run method start
    2025-07-06 17:59:55 INFO  [main] TaskExecutorRepeatTemplate:298 - Handling exception: org.springframework.core.task.TaskRejectedException, caused by: org.springframework.core.task.TaskRejectedException: ExecutorService in active state did not accept task: org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable@c4d2c44
  4. Main thread goes in running queue.take() and stay in waiting
  5. Main thread was waked up due to other three finished thread runs notifyAll()
  6. Main thread expecting 5 result to arrive, currently only 4. It went into waiting or taking then no thread wakes it up.

2. ThrottleLimit is unreliable

According to the source code hint of untrustworthy ThottleLimit

It sais: when used with a thread pooled TaskExecutor the thread pool might prevent the throttle limit actually being reached (so make the core pool size larger than the throttle limit if possible).

Actually we can also adjust the rejection policy to CallerRunPolicy.

3. Change to CallerRunPolicy

3.1 Source code branch

test/v5.1.0/deadlock-when-spring-batch-submit-more-task-than-core-pool-size.

3.2 Source code change

One line change

3.3 Diagram

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Normal Case:
+-------------+         +-------------------+         +---------------------+
| Main Thread |         | TaskExecutor      |         | Worker Thread(s)    |
+-------------+         +-------------------+         +---------------------+
      |                        |                               |
      | for (i < throttle)     |                               |
      |----------------------->|                               |
      | execute(runnable)      |                               |
      |  in TaskExecutorRepeatTemplate                         |    
      |----------------------->|                               |
      |                        |  run()                        |
      |                        |-----------------------------> |
      |                        |                               | doInIteration()
      |                        |                               | queue.put(result)
      | <----------------------------------------------------- |
      |                        |                               |
      | waitForResults()       |                               |
      |-----                   |                               |
      |    |                   |                               |
      |<----                   |                               |
      | queue.take()           |                               |
      |-----                   |                               |
      |    |                   |                               |
      |<----                   |                               |
      |                        |                               |

Important conceptions (TBD)

RepeatOperator

Internal queue for storaging the result

How a Reader - Processor - Writer to be wrapped into the RepearOperator

Overall task submittion (TBD)