Package org.apache.hadoop.hdfs.server.blockmanagement

Class BlockPlacementPolicyWithUpgradeDomain

java.lang.Object
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithUpgradeDomain
@Private @Evolving public class BlockPlacementPolicyWithUpgradeDomain extends BlockPlacementPolicyDefault
The class is responsible for choosing the desired number of targets for placing block replicas that honors upgrade domain policy. Here is the replica placement strategy. If the writer is on a datanode, the 1st replica is placed on the local machine, otherwise a random datanode. The 2nd replica is placed on a datanode that is on a different rack. The 3rd replica is placed on a datanode which is on a different node of the rack as the second replica. All 3 replicas have unique upgrade domains.

  • Constructor Details

    • BlockPlacementPolicyWithUpgradeDomain

      public BlockPlacementPolicyWithUpgradeDomain()

  • Method Details

    • initialize

      public void initialize(org.apache.hadoop.conf.Configuration conf, FSClusterStats stats, org.apache.hadoop.net.NetworkTopology clusterMap, org.apache.hadoop.hdfs.server.blockmanagement.Host2NodesMap host2datanodeMap)
      Description copied from class: BlockPlacementPolicy
      Used to setup a BlockPlacementPolicy object. This should be defined by all implementations of a BlockPlacementPolicy.
      Overrides:
      initialize in class BlockPlacementPolicyDefault
      Parameters:
      conf - the configuration object
      stats - retrieve cluster status from here
      clusterMap - cluster topology

    • isGoodDatanode

      protected boolean isGoodDatanode(DatanodeDescriptor node, int maxTargetPerRack, boolean considerLoad, List<DatanodeStorageInfo> results, boolean avoidStaleNodes)

    • getUpgradeDomainWithDefaultValue

      public String getUpgradeDomainWithDefaultValue(org.apache.hadoop.hdfs.protocol.DatanodeInfo datanodeInfo)

    • verifyBlockPlacement

      public BlockPlacementStatus verifyBlockPlacement(org.apache.hadoop.hdfs.protocol.DatanodeInfo[] locs, int numberOfReplicas)
      Description copied from class: BlockPlacementPolicy
      Verify if the block's placement meets requirement of placement policy, i.e. replicas are placed on no less than minRacks racks in the system.
      Overrides:
      verifyBlockPlacement in class BlockPlacementPolicyDefault
      Parameters:
      locs - block with locations
      numberOfReplicas - replica number of file to be verified
      Returns:
      the result of verification

    • pickupReplicaSet

      protected Collection<DatanodeStorageInfo> pickupReplicaSet(Collection<DatanodeStorageInfo> moreThanOne, Collection<DatanodeStorageInfo> exactlyOne, Map<String,List<DatanodeStorageInfo>> rackMap)
      Description copied from class: BlockPlacementPolicyDefault
      Pick up replica node set for deleting replica as over-replicated. First set contains replica nodes on rack with more than one replica while second set contains remaining replica nodes. If only 1 rack, pick all. If 2 racks, pick all that have more than 1 replicas on the same rack; if no such replicas, pick all. If 3 or more racks, pick all.
      Overrides:
      pickupReplicaSet in class BlockPlacementPolicyDefault

    • isMovable

      public boolean isMovable(Collection<org.apache.hadoop.hdfs.protocol.DatanodeInfo> locs, org.apache.hadoop.hdfs.protocol.DatanodeInfo source, org.apache.hadoop.hdfs.protocol.DatanodeInfo target)
      Description copied from class: BlockPlacementPolicy
      Check if the move is allowed. Used by balancer and other tools.
      Overrides:
      isMovable in class BlockPlacementPolicyDefault
      Parameters:
      locs - all replicas including source and target
      source - source replica of the move
      target - target replica of the move