Package org.apache.hadoop.hdfs.net

Class DFSNetworkTopology

java.lang.Object
org.apache.hadoop.net.NetworkTopology
org.apache.hadoop.hdfs.net.DFSNetworkTopology
public class DFSNetworkTopology extends org.apache.hadoop.net.NetworkTopology
The HDFS specific network topology class. The main purpose of doing this subclassing is to add storage-type-aware chooseRandom method. All the remaining parts should be the same. Currently a placeholder to test storage type info.

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.apache.hadoop.net.NetworkTopology

    org.apache.hadoop.net.NetworkTopology.InvalidTopologyException

  • Field Summary

    Fields inherited from class org.apache.hadoop.net.NetworkTopology

    DEFAULT_RACK, LOG, netlock, numOfRacks

  • Constructor Summary

    Constructors
    Constructor
    Description
    DFSNetworkTopology()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.hadoop.net.Node
    chooseRandomWithStorageType(String scope, Collection<org.apache.hadoop.net.Node> excludedNodes, org.apache.hadoop.fs.StorageType type)
    Randomly choose one node from scope, with specified storage type.
    org.apache.hadoop.net.Node
    chooseRandomWithStorageTypeTwoTrial(String scope, Collection<org.apache.hadoop.net.Node> excludedNodes, org.apache.hadoop.fs.StorageType type)
    Randomly choose one node from scope with the given storage type.
    static DFSNetworkTopology
    getInstance(org.apache.hadoop.conf.Configuration conf)
     

    Methods inherited from class org.apache.hadoop.net.NetworkTopology

    add, chooseRandom, chooseRandom, chooseRandom, contains, countNumOfAvailableNodes, decommissionNode, getDatanodesInRack, getDistance, getDistanceByPath, getFirstHalf, getInstance, getLastHalf, getLeaves, getNode, getNodeForNetworkLocation, getNumOfLeaves, getNumOfNonEmptyRacks, getNumOfRacks, getRack, getWeight, getWeightUsingNetworkLocation, hasClusterEverBeenMultiRack, incrementRacks, init, isChildScope, isNodeGroupAware, isNodeInScope, isOnSameNodeGroup, isOnSameRack, isSameParents, recommissionNode, remove, shuffle, sortByDistance, sortByDistance, sortByDistanceUsingNetworkLocation, sortByDistanceUsingNetworkLocation, toString

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Constructor Details

    • DFSNetworkTopology

      public DFSNetworkTopology()

  • Method Details

    • getInstance

      public static DFSNetworkTopology getInstance(org.apache.hadoop.conf.Configuration conf)

    • chooseRandomWithStorageType

      public org.apache.hadoop.net.Node chooseRandomWithStorageType(String scope, Collection<org.apache.hadoop.net.Node> excludedNodes, org.apache.hadoop.fs.StorageType type)
      Randomly choose one node from scope, with specified storage type. If scope starts with ~, choose one from the all nodes except for the ones in scope; otherwise, choose one from scope. If excludedNodes is given, choose a node that's not in excludedNodes.
      Parameters:
      scope - range of nodes from which a node will be chosen
      excludedNodes - nodes to be excluded from
      type - the storage type we search for
      Returns:
      the chosen node

    • chooseRandomWithStorageTypeTwoTrial

      public org.apache.hadoop.net.Node chooseRandomWithStorageTypeTwoTrial(String scope, Collection<org.apache.hadoop.net.Node> excludedNodes, org.apache.hadoop.fs.StorageType type)
      Randomly choose one node from scope with the given storage type. If scope starts with ~, choose one from the all nodes except for the ones in scope; otherwise, choose one from scope. If excludedNodes is given, choose a node that's not in excludedNodes. This call would make up to two calls. It first tries to get a random node (with old method) and check if it satisfies. If yes, simply return it. Otherwise, it make a second call (with the new method) by passing in a storage type. This is for better performance reason. Put in short, the key note is that the old method is faster but may take several runs, while the new method is somewhat slower, and always succeed in one trial. See HDFS-11535 for more detail.
      Parameters:
      scope - range of nodes from which a node will be chosen
      excludedNodes - nodes to be excluded from
      type - the storage type we search for
      Returns:
      the chosen node