Key Default Type Description
taskmanager.network.compression.codec
LZ4

Enum

The codec to be used when compressing shuffle data. If it is "NONE", compression is disable. If it is not "NONE", only "LZ4", "LZO" and "ZSTD" are supported now. Through tpc-ds test of these three algorithms, the results show that "LZ4" algorithm has the highest compression and decompression speed, but the compression ratio is the lowest. "ZSTD" has the highest compression ratio, but the compression and decompression speed is the slowest, and LZO is between the two. Also note that this option is experimental and might be changed in the future.

Possible values:
  • "NONE"
  • "LZ4"
  • "LZO"
  • "ZSTD"
taskmanager.network.detailed-metrics
false Boolean Boolean flag to enable/disable more detailed metrics about inbound/outbound network queue lengths.
taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class
(none) String The option configures the class that is responsible for creating an external remote tier factory for hybrid shuffle. If configured, the hybrid shuffle will only initialize the specified remote tier according to the given class name. Currently, since the tier interfaces are not yet public and are still actively evolving, it is recommended that users do not independently implement the external remote tier until the tier interfaces are stabilized.
taskmanager.network.hybrid-shuffle.remote.path
(none) String The option is used to configure the base path of remote storage for hybrid shuffle. The shuffle data will be stored in remote storage when the disk space is not enough. Note: If this option is not configured the remote storage will be disabled.
taskmanager.network.memory.buffer-debloat.enabled
false Boolean The switch of the automatic buffered debloating feature. If enabled the amount of in-flight data will be adjusted automatically accordingly to the measured throughput.
taskmanager.network.memory.buffer-debloat.period
200 ms Duration The minimum period of time after which the buffer size will be debloated if required. The low value provides a fast reaction to the load fluctuation but can influence the performance.
taskmanager.network.memory.buffer-debloat.samples
20 Integer The number of the last buffer size values that will be taken for the correct calculation of the new one.
taskmanager.network.memory.buffer-debloat.target
1 s Duration The target total time after which buffered in-flight data should be fully consumed. This configuration option will be used, in combination with the measured throughput, to adjust the amount of in-flight data.
taskmanager.network.memory.buffer-debloat.threshold-percentages
25 Integer The minimum difference in percentage between the newly calculated buffer size and the old one to announce the new value. Can be used to avoid constant back and forth small adjustments.
taskmanager.network.memory.read-buffer.required-per-gate.max
(none) Integer The maximum number of network read buffers that are required by an input gate. (An input gate is responsible for reading data from all subtasks of an upstream task.) The number of buffers needed by an input gate is dynamically calculated in runtime, depending on various factors (e.g., the parallelism of the upstream task). Among the calculated number of needed buffers, the part below this configured value is required, while the excess part, if any, is optional. A task will fail if the required buffers cannot be obtained in runtime. A task will not fail due to not obtaining optional buffers, but may suffer a performance reduction. If not explicitly configured, the default value is Integer.MAX_VALUE for streaming workloads, and 1000 for batch workloads. If explicitly configured, the configured value should be at least 1.
taskmanager.network.netty.client.connectTimeoutSec
120 Integer The Netty client connection timeout.
taskmanager.network.netty.client.tcp.keepCount
(none) Integer The maximum number of keepalive probes TCP should send before Netty client dropping the connection. Note: This will not take effect when using netty transport type of nio with an older version of JDK 8, refer to https://bugs.openjdk.org/browse/JDK-8194298.
taskmanager.network.netty.client.tcp.keepIdleSec
(none) Integer The time (in seconds) the connection needs to remain idle before TCP starts sending keepalive probes. Note: This will not take effect when using netty transport type of nio with an older version of JDK 8, refer to https://bugs.openjdk.org/browse/JDK-8194298.
taskmanager.network.netty.client.tcp.keepIntervalSec
(none) Integer The time (in seconds) between individual keepalive probes. Note: This will not take effect when using netty transport type of nio with an older version of JDK 8, refer to https://bugs.openjdk.org/browse/JDK-8194298.
taskmanager.network.partition-request-timeout
10 s Duration Timeout for an individual partition request of remote input channels. The partition request will finally fail if the total wait time exceeds twice the value of taskmanager.network.request-backoff.max.
taskmanager.network.request-backoff.initial
100 Integer Minimum backoff in milliseconds for partition requests of local input channels.
taskmanager.network.request-backoff.max
10000 Integer Maximum backoff in milliseconds for partition requests of local input channels.
taskmanager.network.retries
0 Integer The number of retry attempts for network communication. Currently it's only used for establishing input/output channel connections
taskmanager.network.sort-shuffle.min-buffers
512 Integer Minimum number of network buffers required per blocking result partition for sort-shuffle. For production usage, it is suggested to increase this config value to at least 2048 (64M memory if the default 32K memory segment size is used) to improve the data compression ratio and reduce the small network packets. Usually, several hundreds of megabytes memory is enough for large scale batch jobs. Note: you may also need to increase the size of total network memory to avoid the 'insufficient number of network buffers' error if you are increasing this config value.
taskmanager.network.tcp-connection.enable-reuse-across-jobs
true Boolean Whether to reuse tcp connections across multi jobs. If set to true, tcp connections will not be released after job finishes. The subsequent jobs will be free from the overhead of the connection re-establish. However, this may lead to an increase in the total number of connections on your machine. When it reaches the upper limit, you can set it to false to release idle connections.