Package org.apache.hadoop.hdfs.server.balancer

Class Dispatcher

java.lang.Object

org.apache.hadoop.hdfs.server.balancer.Dispatcher

@Private
public class Dispatcher
extends Object

Dispatching block replica moves between datanodes.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	Dispatcher.DBlock	A class for keeping track of block locations in the dispatcher.
static class	Dispatcher.DBlockStriped
static class	Dispatcher.DDatanode	A class that keeps track of a datanode.
class	Dispatcher.PendingMove	This class keeps track of a scheduled reportedBlock move
class	Dispatcher.Source	A node that can be the sources of a block move
static class	Dispatcher.StorageGroupMap<G extends Dispatcher.DDatanode.StorageGroup>

Constructor Summary

Constructors

Constructor	Description
Dispatcher(NameNodeConnector nnc, Set<String> includedNodes, Set<String> excludedNodes, long movedWinWidth, int moverThreads, int dispatcherThreads, int maxConcurrentMovesPerNode, int maxNoMoveInterval, org.apache.hadoop.conf.Configuration conf)	Constructor called by Mover.

Method Summary

Modifier and Type	Method	Description
static void	checkForBlockPinningFailures(Map<Long,Set<org.apache.hadoop.hdfs.protocol.DatanodeInfo>> excludedPinnedBlocks, Iterable<? extends Dispatcher.DDatanode.StorageGroup> targets)	Check any of the block movements are failed due to block pinning errors.
static boolean	checkForSuccess(Iterable<? extends Dispatcher.DDatanode.StorageGroup> targets)
boolean	dispatchAndCheckContinue()
void	executePendingMove(Dispatcher.PendingMove p)
org.apache.hadoop.net.NetworkTopology	getCluster()
org.apache.hadoop.hdfs.DistributedFileSystem	getDistributedFileSystem()
Dispatcher.StorageGroupMap<Dispatcher.DDatanode.StorageGroup>	getStorageGroupMap()
List<org.apache.hadoop.hdfs.server.protocol.DatanodeStorageReport>	init()	Get live datanode storage reports and then build the network topology.
int	moveTasksTotal()
Dispatcher.DDatanode	newDatanode(org.apache.hadoop.hdfs.protocol.DatanodeInfo datanode)
static void	setDelayAfterErrors(long time)
void	shutdownNow()	shutdown thread pools
static boolean	waitForMoveCompletion(Iterable<? extends Dispatcher.DDatanode.StorageGroup> targets)	Wait for all reportedBlock move confirmations.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Dispatcher

public Dispatcher(NameNodeConnector nnc,
 Set<String> includedNodes,
 Set<String> excludedNodes,
 long movedWinWidth,
 int moverThreads,
 int dispatcherThreads,
 int maxConcurrentMovesPerNode,
 int maxNoMoveInterval,
 org.apache.hadoop.conf.Configuration conf)

Constructor called by Mover.

Method Details

getDistributedFileSystem

public org.apache.hadoop.hdfs.DistributedFileSystem getDistributedFileSystem()

getStorageGroupMap

public Dispatcher.StorageGroupMap<Dispatcher.DDatanode.StorageGroup> getStorageGroupMap()

getCluster

public org.apache.hadoop.net.NetworkTopology getCluster()

moveTasksTotal

public int moveTasksTotal()

init

public List<org.apache.hadoop.hdfs.server.protocol.DatanodeStorageReport> init()
                                                                        throws IOException

Get live datanode storage reports and then build the network topology.

Throws:

IOException

newDatanode

public Dispatcher.DDatanode newDatanode(org.apache.hadoop.hdfs.protocol.DatanodeInfo datanode)

executePendingMove

public void executePendingMove(Dispatcher.PendingMove p)

dispatchAndCheckContinue

public boolean dispatchAndCheckContinue()
                                 throws InterruptedException

Throws:

InterruptedException

waitForMoveCompletion

public static boolean waitForMoveCompletion(Iterable<? extends Dispatcher.DDatanode.StorageGroup> targets)

Wait for all reportedBlock move confirmations.

Returns:

true if there is failed move execution

checkForBlockPinningFailures

public static void checkForBlockPinningFailures(Map<Long,Set<org.apache.hadoop.hdfs.protocol.DatanodeInfo>> excludedPinnedBlocks,
 Iterable<? extends Dispatcher.DDatanode.StorageGroup> targets)

Check any of the block movements are failed due to block pinning errors. If yes, add the failed blockId and its respective source node location to the excluded list.

checkForSuccess

public static boolean checkForSuccess(Iterable<? extends Dispatcher.DDatanode.StorageGroup> targets)

Returns:

true if some moves are success.

setDelayAfterErrors

@VisibleForTesting
public static void setDelayAfterErrors(long time)

shutdownNow

public void shutdownNow()

shutdown thread pools