Package org.apache.hadoop.hdfs.server.namenode

Class NameNode

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.hadoop.conf.ReconfigurableBase
org.apache.hadoop.hdfs.server.namenode.NameNode
All Implemented Interfaces:
org.apache.hadoop.conf.Configurable, org.apache.hadoop.conf.Reconfigurable, TokenVerifier<org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier>, NameNodeStatusMXBean
Direct Known Subclasses:
BackupNode
@Private @Metrics(context="dfs") public class NameNode extends org.apache.hadoop.conf.ReconfigurableBase implements NameNodeStatusMXBean, TokenVerifier<org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier>
NameNode serves as both directory namespace manager and "inode table" for the Hadoop DFS. There is a single NameNode running in any DFS deployment. (Well, except when there is a second backup/failover NameNode, or when using federated NameNodes.) The NameNode controls two critical tables: 1) filename->blocksequence (namespace) 2) block->machinelist ("inodes") The first table is stored on disk and is very precious. The second table is rebuilt every time the NameNode comes up. 'NameNode' refers to both this class as well as the 'NameNode server'. The 'FSNamesystem' class actually performs most of the filesystem management. The majority of the 'NameNode' class itself is concerned with exposing the IPC interface and the HTTP server to the outside world, plus some configuration management. NameNode implements the ClientProtocol interface, which allows clients to ask for DFS services. ClientProtocol is not designed for direct use by authors of DFS client code. End-users should instead use the FileSystem class. NameNode also implements the DatanodeProtocol interface, used by DataNodes that actually store DFS data blocks. These methods are invoked repeatedly and automatically by all the DataNodes in a DFS deployment. NameNode also implements the NamenodeProtocol interface, used by secondary namenodes or rebalancing processes to get partial NameNode state, for example partial blocksMap etc.

  • Field Details

    • NAMENODE_SPECIFIC_KEYS

      public static final String[] NAMENODE_SPECIFIC_KEYS
      HDFS configuration can have three types of parameters:
      1. Parameters that are common for all the name services in the cluster.
      2. Parameters that are specific to a name service. These keys are suffixed with nameserviceId in the configuration. For example, "dfs.namenode.rpc-address.nameservice1".
      3. Parameters that are specific to a single name node. These keys are suffixed with nameserviceId and namenodeId in the configuration. for example, "dfs.namenode.rpc-address.nameservice1.namenode1"
      In the latter cases, operators may specify the configuration without any suffix, with a nameservice suffix, or with a nameservice and namenode suffix. The more specific suffix will take precedence. These keys are specific to a given namenode, and thus may be configured globally, for a nameservice, or for a specific namenode within a nameservice.

    • NAMESERVICE_SPECIFIC_KEYS

      public static final String[] NAMESERVICE_SPECIFIC_KEYS
      See Also:

    • DEFAULT_PORT

      @Deprecated public static final int DEFAULT_PORT
      Deprecated.
      Use HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT instead.
      See Also:

    • LOG

      public static final org.slf4j.Logger LOG

    • stateChangeLog

      public static final org.slf4j.Logger stateChangeLog

    • blockStateChangeLog

      public static final org.slf4j.Logger blockStateChangeLog

    • ACTIVE_STATE

      public static final HAState ACTIVE_STATE

    • STANDBY_STATE

      public static final HAState STANDBY_STATE

    • OBSERVER_STATE

      public static final HAState OBSERVER_STATE

    • METRICS_LOG_NAME

      public static final String METRICS_LOG_NAME
      See Also:

    • namesystem

      protected FSNamesystem namesystem

    • role

      protected final HdfsServerConstants.NamenodeRole role

    • allowStaleStandbyReads

      protected final boolean allowStaleStandbyReads

    • httpServer

      protected NameNodeHttpServer httpServer
      httpServer

    • stopRequested

      protected boolean stopRequested
      only used for testing purposes

    • nodeRegistration

      protected NamenodeRegistration nodeRegistration
      Registration information of this name-node

    • tracer

      protected final org.apache.hadoop.tracing.Tracer tracer

  • Constructor Details

    • NameNode

      public NameNode(org.apache.hadoop.conf.Configuration conf) throws IOException
      Start NameNode.

      The name-node can be started with one of the following startup options:

      • REGULAR - normal name node startup
      • FORMAT - format name node
      • BACKUP - start backup node
      • CHECKPOINT - start checkpoint node
      • UPGRADE - start the cluster
      • UPGRADEONLY - upgrade the cluster upgrade and create a snapshot of the current file system state
      • RECOVERY - recover name node metadata
      • ROLLBACK - roll the cluster back to the previous state
      • IMPORT - import checkpoint
      The option is passed via configuration field: dfs.namenode.startup The conf will be modified to reflect the actual ports on which the NameNode is up and running if the user passes the port as zero in the conf.
      Parameters:
      conf - confirguration
      Throws:
      IOException

    • NameNode

      protected NameNode(org.apache.hadoop.conf.Configuration conf, HdfsServerConstants.NamenodeRole role) throws IOException
      Throws:
      IOException

  • Method Details

    • getProtocolVersion

      public long getProtocolVersion(String protocol, long clientVersion) throws IOException
      Throws:
      IOException

    • format

      public static void format(org.apache.hadoop.conf.Configuration conf) throws IOException
      Format a new filesystem. Destroys any filesystem that may already exist at this location.
      Throws:
      IOException

    • getNamesystem

      public FSNamesystem getNamesystem()
      Return the FSNamesystem object.
      Returns:
      FSNamesystem object.

    • getRpcServer

      public NamenodeProtocols getRpcServer()

    • getHttpServer

      @VisibleForTesting public org.apache.hadoop.http.HttpServer2 getHttpServer()

    • queueExternalCall

      public void queueExternalCall(org.apache.hadoop.ipc.ExternalCall<?> extCall) throws IOException, InterruptedException
      Throws:
      IOException
      InterruptedException

    • initMetrics

      public static void initMetrics(org.apache.hadoop.conf.Configuration conf, HdfsServerConstants.NamenodeRole role)

    • getNameNodeMetrics

      public static NameNodeMetrics getNameNodeMetrics()

    • parseSpecialValue

      @VisibleForTesting public static String parseSpecialValue(String content, String key)
      Try to obtain the value corresponding to the key by parsing the content.
      Parameters:
      content - the full content to be parsed.
      key - trying to obtain the value of the key.
      Returns:
      the value corresponding to the key.

    • getClientMachine

      public static String getClientMachine(String[] ipProxyUsers)
      Try to obtain the actual client's machine according to the current user.
      Parameters:
      ipProxyUsers - Users who can override client infos.
      Returns:
      The actual client's machine.

    • getClientIdAndCallId

      public static org.apache.commons.lang3.tuple.Pair<byte[],Integer> getClientIdAndCallId(String[] ipProxyUsers)
      Try to obtain the actual client's id and call id according to the current user.
      Parameters:
      ipProxyUsers - Users who can override client infos
      Returns:
      The actual client's id and call id.

    • getStartupProgress

      public static StartupProgress getStartupProgress()
      Returns object used for reporting namenode startup progress.
      Returns:
      StartupProgress for reporting namenode startup progress

    • getTokenServiceName

      public String getTokenServiceName()
      Return the service name of the issued delegation token.
      Returns:
      The name service id in HA-mode, or the rpc address in non-HA mode

    • getClientNamenodeAddress

      public String getClientNamenodeAddress()
      Get the namenode address to be used by clients.
      Returns:
      nn address

    • setServiceAddress

      public static void setServiceAddress(org.apache.hadoop.conf.Configuration conf, String address)
      Set the configuration property for the service rpc address to address

    • getServiceAddress

      public static InetSocketAddress getServiceAddress(org.apache.hadoop.conf.Configuration conf, boolean fallback)
      Fetches the address for services to use when connecting to namenode based on the value of fallback returns null if the special address is not specified or returns the default namenode address to be used by both clients and services. Services here are datanodes, backup node, any non client connection

    • getRole

      public HdfsServerConstants.NamenodeRole getRole()

    • composeNotStartedMessage

      public static String composeNotStartedMessage(HdfsServerConstants.NamenodeRole role)

    • getServiceRpcServerAddress

      protected InetSocketAddress getServiceRpcServerAddress(org.apache.hadoop.conf.Configuration conf)
      Given a configuration get the address of the service rpc server If the service rpc is not configured returns null

    • getRpcServerAddress

      protected InetSocketAddress getRpcServerAddress(org.apache.hadoop.conf.Configuration conf)

    • getServiceRpcServerBindHost

      protected String getServiceRpcServerBindHost(org.apache.hadoop.conf.Configuration conf)
      Given a configuration get the bind host of the service rpc server If the bind host is not configured returns null.

    • getRpcServerBindHost

      protected String getRpcServerBindHost(org.apache.hadoop.conf.Configuration conf)
      Given a configuration get the bind host of the client rpc server If the bind host is not configured returns null.

    • setRpcServiceServerAddress

      protected void setRpcServiceServerAddress(org.apache.hadoop.conf.Configuration conf, InetSocketAddress serviceRPCAddress)
      Modifies the configuration passed to contain the service rpc address setting

    • setRpcServerAddress

      protected void setRpcServerAddress(org.apache.hadoop.conf.Configuration conf, InetSocketAddress rpcAddress)

    • getHttpServerAddress

      protected InetSocketAddress getHttpServerAddress(org.apache.hadoop.conf.Configuration conf)

    • getHttpServerBindAddress

      protected InetSocketAddress getHttpServerBindAddress(org.apache.hadoop.conf.Configuration conf)
      HTTP server address for binding the endpoint. This method is for use by the NameNode and its derivatives. It may return a different address than the one that should be used by clients to connect to the NameNode. See DFSConfigKeys.DFS_NAMENODE_HTTP_BIND_HOST_KEY
      Parameters:
      conf -
      Returns:

    • getHttpAddress

      public static InetSocketAddress getHttpAddress(org.apache.hadoop.conf.Configuration conf)
      Returns:
      the NameNode HTTP address.

    • loadNamesystem

      protected void loadNamesystem(org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • getRemoteUser

      public static org.apache.hadoop.security.UserGroupInformation getRemoteUser() throws IOException
      Throws:
      IOException

    • verifyToken

      public void verifyToken(org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier id, byte[] password) throws IOException
      Specified by:
      verifyToken in interface TokenVerifier<org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier>
      Throws:
      IOException

    • initialize

      protected void initialize(org.apache.hadoop.conf.Configuration conf) throws IOException
      Initialize name-node.
      Parameters:
      conf - the configuration
      Throws:
      IOException

    • getAliasMapServer

      @VisibleForTesting public InMemoryLevelDBAliasMapServer getAliasMapServer()

    • startMetricsLogger

      protected void startMetricsLogger(org.apache.hadoop.conf.Configuration conf)
      Start a timer to periodically write NameNode metrics to the log file. This behavior can be disabled by configuration.
      Parameters:
      conf -

    • stopMetricsLogger

      protected void stopMetricsLogger()

    • createRpcServer

      protected NameNodeRpcServer createRpcServer(org.apache.hadoop.conf.Configuration conf) throws IOException
      Create the RPC server implementation. Used as an extension point for the BackupNode.
      Throws:
      IOException

    • createHAState

      protected HAState createHAState(org.apache.hadoop.conf.Configuration conf)

    • createHAContext

      protected HAContext createHAContext()

    • join

      public void join()
      Wait for service to finish. (Normally, it runs forever.)

    • stop

      public void stop()
      Stop all NameNode threads and wait for all to finish.

    • isInSafeMode

      public boolean isInSafeMode()
      Is the cluster currently in safe mode?

    • getFSImage

      @VisibleForTesting public FSImage getFSImage()
      get FSImage

    • getNameNodeAddress

      public InetSocketAddress getNameNodeAddress()
      Returns:
      NameNode RPC address

    • getAuxiliaryNameNodeAddresses

      public Set<InetSocketAddress> getAuxiliaryNameNodeAddresses()
      Returns:
      The auxiliary nameNode RPC addresses, or empty set if there is none.

    • getNameNodeAddressHostPortString

      public String getNameNodeAddressHostPortString()
      Returns:
      NameNode RPC address in "host:port" string form

    • getNNAuxiliaryRpcAddress

      @VisibleForTesting public String getNNAuxiliaryRpcAddress()
      Return a host:port format string corresponds to an auxiliary port configured on NameNode. If there are multiple auxiliary ports, an arbitrary one is returned. If there is no auxiliary listener, returns null.
      Returns:
      a string of format host:port that points to an auxiliary NameNode address, or null if there is no such address.

    • getServiceRpcAddress

      public InetSocketAddress getServiceRpcAddress()
      Returns:
      NameNode service RPC address if configured, the NameNode RPC address otherwise

    • getHttpAddress

      public InetSocketAddress getHttpAddress()
      Returns:
      NameNode HTTP address, used by the Web UI, image transfer, and HTTP-based file system clients like WebHDFS

    • getHttpsAddress

      public InetSocketAddress getHttpsAddress()
      Returns:
      NameNode HTTPS address, used by the Web UI, image transfer, and HTTP-based file system clients like WebHDFS

    • joinHttpServer

      @VisibleForTesting public void joinHttpServer()
      NameNodeHttpServer, used by unit tests to ensure a full shutdown, so that no bind exception is thrown during restart.

    • checkAllowFormat

      public static void checkAllowFormat(org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • initializeSharedEdits

      @VisibleForTesting public static boolean initializeSharedEdits(org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • initializeSharedEdits

      @VisibleForTesting public static boolean initializeSharedEdits(org.apache.hadoop.conf.Configuration conf, boolean force) throws IOException
      Throws:
      IOException

    • doRollback

      @VisibleForTesting public static boolean doRollback(org.apache.hadoop.conf.Configuration conf, boolean isConfirmationNeeded) throws IOException
      Throws:
      IOException

    • getStartupOption

      public static HdfsServerConstants.StartupOption getStartupOption(org.apache.hadoop.conf.Configuration conf)

    • createNameNode

      public static NameNode createNameNode(String[] argv, org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • initializeGenericKeys

      public static void initializeGenericKeys(org.apache.hadoop.conf.Configuration conf, String nameserviceId, String namenodeId)
      In federation configuration is set for a set of namenode and secondary namenode/backup/checkpointer, which are grouped under a logical nameservice ID. The configuration keys specific to them have suffix set to configured nameserviceId. This method copies the value from specific key of format key.nameserviceId to key, to set up the generic configuration. Once this is done, only generic version of the configuration is read in rest of the code, for backward compatibility and simpler code changes.
      Parameters:
      conf - Configuration object to lookup specific key and to set the value to the key passed. Note the conf object is modified
      nameserviceId - name service Id (to distinguish federated NNs)
      namenodeId - the namenode ID (to distinguish HA NNs)
      See Also:

    • getNameServiceId

      protected String getNameServiceId(org.apache.hadoop.conf.Configuration conf)
      Get the name service Id for the node
      Returns:
      name service Id or null if federation is not configured

    • main

      public static void main(String[] argv) throws Exception
      Throws:
      Exception

    • getNameNodeState

      @Metric({"NameNodeState","Namenode HA service state"}) public int getNameNodeState()
      Emit Namenode HA service state as an integer so that one can monitor NN HA state based on this metric.
      Returns:
      0 when not fully started 1 for active or standalone (non-HA) NN 2 for standby 3 for observer These are the same integer values for the HAServiceState enum.

    • getNNRole

      public String getNNRole()
      Description copied from interface: NameNodeStatusMXBean
      Gets the NameNode role.
      Specified by:
      getNNRole in interface NameNodeStatusMXBean
      Returns:
      the NameNode role.

    • getState

      public String getState()
      Description copied from interface: NameNodeStatusMXBean
      Gets the NameNode state.
      Specified by:
      getState in interface NameNodeStatusMXBean
      Returns:
      the NameNode state.

    • getHostAndPort

      public String getHostAndPort()
      Description copied from interface: NameNodeStatusMXBean
      Gets the host and port colon separated.
      Specified by:
      getHostAndPort in interface NameNodeStatusMXBean
      Returns:
      host and port colon separated.

    • isSecurityEnabled

      public boolean isSecurityEnabled()
      Description copied from interface: NameNodeStatusMXBean
      Gets if security is enabled.
      Specified by:
      isSecurityEnabled in interface NameNodeStatusMXBean
      Returns:
      true, if security is enabled.

    • getLastHATransitionTime

      public long getLastHATransitionTime()
      Description copied from interface: NameNodeStatusMXBean
      Gets the most recent HA transition time in milliseconds from the epoch.
      Specified by:
      getLastHATransitionTime in interface NameNodeStatusMXBean
      Returns:
      the most recent HA transition time in milliseconds from the epoch.

    • getBytesWithFutureGenerationStamps

      public long getBytesWithFutureGenerationStamps()
      Description copied from interface: NameNodeStatusMXBean
      Gets number of bytes in blocks with future generation stamps.
      Specified by:
      getBytesWithFutureGenerationStamps in interface NameNodeStatusMXBean
      Returns:
      number of bytes that can be deleted if exited from safe mode.

    • getSlowPeersReport

      public String getSlowPeersReport()
      Description copied from interface: NameNodeStatusMXBean
      Retrieves information about slow DataNodes, if the feature is enabled. The report is in a JSON format.
      Specified by:
      getSlowPeersReport in interface NameNodeStatusMXBean

    • getSlowDisksReport

      public String getSlowDisksReport()
      Description copied from interface: NameNodeStatusMXBean
      Gets the topN slow disks in the cluster, if the feature is enabled.
      Specified by:
      getSlowDisksReport in interface NameNodeStatusMXBean
      Returns:
      JSON string of list of diskIDs and latencies

    • doImmediateShutdown

      protected void doImmediateShutdown(Throwable t) throws org.apache.hadoop.util.ExitUtil.ExitException
      Shutdown the NN immediately in an ungraceful way. Used when it would be unsafe for the NN to continue operating, e.g. during a failed HA state transition.
      Parameters:
      t - exception which warrants the shutdown. Printed to the NN log before exit.
      Throws:
      org.apache.hadoop.util.ExitUtil.ExitException - thrown only for testing.

    • isStandbyState

      public boolean isStandbyState()

    • isActiveState

      public boolean isActiveState()

    • isObserverState

      public boolean isObserverState()

    • getReconfigurableProperties

      public Collection<String> getReconfigurableProperties()
      Specified by:
      getReconfigurableProperties in interface org.apache.hadoop.conf.Reconfigurable
      Specified by:
      getReconfigurableProperties in class org.apache.hadoop.conf.ReconfigurableBase

    • reconfigurePropertyImpl

      protected String reconfigurePropertyImpl(String property, String newVal) throws org.apache.hadoop.conf.ReconfigurationException
      Specified by:
      reconfigurePropertyImpl in class org.apache.hadoop.conf.ReconfigurableBase
      Throws:
      org.apache.hadoop.conf.ReconfigurationException

    • getNewConf

      protected org.apache.hadoop.conf.Configuration getNewConf()
      Specified by:
      getNewConf in class org.apache.hadoop.conf.ReconfigurableBase