Package org.apache.hadoop.hdfs.server.blockmanagement

Class BlockPlacementPolicyRackFaultTolerant

java.lang.Object
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant
Direct Known Subclasses:
AvailableSpaceRackFaultTolerantBlockPlacementPolicy
@Private public class BlockPlacementPolicyRackFaultTolerant extends BlockPlacementPolicyDefault
The class is responsible for choosing the desired number of targets for placing block replicas. The strategy is that it tries its best to place the replicas to most racks.

  • Constructor Details

    • BlockPlacementPolicyRackFaultTolerant

      public BlockPlacementPolicyRackFaultTolerant()

  • Method Details

    • getMaxNodesPerRack

      protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas)
      Description copied from class: BlockPlacementPolicyDefault
      Calculate the maximum number of replicas to allocate per rack. It also limits the total number of replicas to the total number of nodes in the cluster. Caller should adjust the replica count to the return value.
      Overrides:
      getMaxNodesPerRack in class BlockPlacementPolicyDefault
      Parameters:
      numOfChosen - The number of already chosen nodes.
      numOfReplicas - The number of additional nodes to allocate.
      Returns:
      integer array. Index 0: The number of nodes allowed to allocate in addition to already chosen nodes. Index 1: The maximum allowed number of nodes per rack. This is independent of the number of chosen nodes, as it is calculated using the target number of replicas.

    • chooseTargetInOrder

      protected org.apache.hadoop.net.Node chooseTargetInOrder(int numOfReplicas, org.apache.hadoop.net.Node writer, Set<org.apache.hadoop.net.Node> excludedNodes, long blocksize, int maxNodesPerRack, List<DatanodeStorageInfo> results, boolean avoidStaleNodes, boolean newBlock, EnumMap<org.apache.hadoop.fs.StorageType,Integer> storageTypes) throws BlockPlacementPolicy.NotEnoughReplicasException
      Choose numOfReplicas in order: 1. If total replica expected is less than numOfRacks in cluster, it choose randomly. 2. If total replica expected is bigger than numOfRacks, it choose: 2a. Fill each rack exactly (maxNodesPerRack-1) replicas. 2b. For some random racks, place one more replica to each one of them, until numOfReplicas have been chosen.
      3. If after step 2, there are still replicas not placed (due to some racks have fewer datanodes than maxNodesPerRack), the rest of the replicas is placed evenly on the rest of the racks who have Datanodes that have not been placed a replica. 4. If after step 3, there are still replicas not placed. A BlockPlacementPolicy.NotEnoughReplicasException is thrown.

      For normal setups, step 2 would suffice. So in the end, the difference of the numbers of replicas for each two racks is no more than 1. Either way it always prefer local storage.

      Overrides:
      chooseTargetInOrder in class BlockPlacementPolicyDefault
      Returns:
      local node of writer
      Throws:
      BlockPlacementPolicy.NotEnoughReplicasException

    • verifyBlockPlacement

      public BlockPlacementStatus verifyBlockPlacement(org.apache.hadoop.hdfs.protocol.DatanodeInfo[] locs, int numberOfReplicas)
      Description copied from class: BlockPlacementPolicy
      Verify if the block's placement meets requirement of placement policy, i.e. replicas are placed on no less than minRacks racks in the system.
      Overrides:
      verifyBlockPlacement in class BlockPlacementPolicyDefault
      Parameters:
      locs - block with locations
      numberOfReplicas - replica number of file to be verified
      Returns:
      the result of verification

    • pickupReplicaSet

      protected Collection<DatanodeStorageInfo> pickupReplicaSet(Collection<DatanodeStorageInfo> moreThanOne, Collection<DatanodeStorageInfo> exactlyOne, Map<String,List<DatanodeStorageInfo>> rackMap)
      Description copied from class: BlockPlacementPolicyDefault
      Pick up replica node set for deleting replica as over-replicated. First set contains replica nodes on rack with more than one replica while second set contains remaining replica nodes. If only 1 rack, pick all. If 2 racks, pick all that have more than 1 replicas on the same rack; if no such replicas, pick all. If 3 or more racks, pick all.
      Overrides:
      pickupReplicaSet in class BlockPlacementPolicyDefault