Key Default Type Description
execution.checkpointing.file-merging.enabled
false Boolean Whether to enable merging multiple checkpoint files into one, which will greatly reduce the number of small checkpoint files. This is an experimental feature under evaluation, make sure you're aware of the possible effects of enabling it.
execution.checkpointing.file-merging.across-checkpoint-boundary
false Boolean Only relevant if execution.checkpointing.file-merging.enabled is enabled.
Whether to allow merging data of multiple checkpoints into one physical file. If this option is set to false, only merge files within checkpoint boundaries. Otherwise, it is possible for the logical files of different checkpoints to share the same physical file.
execution.checkpointing.file-merging.max-file-size
32 mb MemorySize Max size of a physical file for merged checkpoints.
execution.checkpointing.file-merging.pool-blocking
false Boolean Whether to use Blocking or Non-Blocking pool for merging physical files. A Non-Blocking pool will always provide usable physical file without blocking. It may create many physical files if poll file frequently. When poll a small file from a Blocking pool, it may be blocked until the file is returned.
execution.checkpointing.file-merging.max-space-amplification
2.0 Float Space amplification stands for the magnification of the occupied space compared to the amount of valid data. The more space amplification is, the more waste of space will be. This configs a space amplification above which a re-uploading for physical files will be triggered to reclaim space. Any value below 1f means disabling the space control.