Package org.apache.hadoop.yarn.server.federation.utils

Class FederationStateStoreFacade

java.lang.Object
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade
public final class FederationStateStoreFacade extends Object
The FederationStateStoreFacade is an utility wrapper that provides singleton access to the Federation state store. It abstracts out retries and in addition, it also implements the caching for various objects.

  • Method Details

    • reinitialize

      @VisibleForTesting public void reinitialize(FederationStateStore store, org.apache.hadoop.conf.Configuration config)
      Delete and re-initialize the cache, to force it to use the given configuration.
      Parameters:
      store - the FederationStateStore instance to reinitialize with
      config - the updated configuration to reinitialize with

    • createRetryPolicy

      public static org.apache.hadoop.io.retry.RetryPolicy createRetryPolicy(org.apache.hadoop.conf.Configuration conf)
      Create a RetryPolicy for FederationStateStoreFacade. In case of failure, it retries for:
      • FederationStateStoreRetriableException
      • CacheLoaderException
      Parameters:
      conf - the updated configuration
      Returns:
      the RetryPolicy for FederationStateStoreFacade

    • getInstance

      public static FederationStateStoreFacade getInstance()
      Returns the singleton instance of the FederationStateStoreFacade object.
      Returns:
      the singleton FederationStateStoreFacade instance

    • getInstance

      public static FederationStateStoreFacade getInstance(org.apache.hadoop.conf.Configuration conf)
      Returns the singleton instance of the FederationStateStoreFacade object.
      Parameters:
      conf - configuration.
      Returns:
      the singleton FederationStateStoreFacade instance

    • getSubCluster

      public SubClusterInfo getSubCluster(SubClusterId subClusterId) throws org.apache.hadoop.yarn.exceptions.YarnException
      Returns the SubClusterInfo for the specified SubClusterId.
      Parameters:
      subClusterId - the identifier of the sub-cluster
      Returns:
      the sub cluster information, or null if there is no mapping for the subClusterId
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful

    • getSubCluster

      public SubClusterInfo getSubCluster(SubClusterId subClusterId, boolean flushCache) throws org.apache.hadoop.yarn.exceptions.YarnException
      Updates the cache with the central FederationStateStore and returns the SubClusterInfo for the specified SubClusterId.
      Parameters:
      subClusterId - the identifier of the sub-cluster
      flushCache - flag to indicate if the cache should be flushed or not
      Returns:
      the sub cluster information
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful

    • getSubClusters

      public Map<SubClusterId,SubClusterInfo> getSubClusters(boolean filterInactiveSubClusters) throws org.apache.hadoop.yarn.exceptions.YarnException
      Returns the SubClusterInfo of all active sub cluster(s).
      Parameters:
      filterInactiveSubClusters - whether to filter out inactive sub-clusters
      Returns:
      the information of all active sub cluster(s)
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful

    • getSubClusters

      public Map<SubClusterId,SubClusterInfo> getSubClusters(boolean filterInactiveSubClusters, boolean flushCache) throws org.apache.hadoop.yarn.exceptions.YarnException
      Updates the cache with the central FederationStateStore and returns the SubClusterInfo of all active sub cluster(s).
      Parameters:
      filterInactiveSubClusters - whether to filter out inactive sub-clusters
      flushCache - flag to indicate if the cache should be flushed or not
      Returns:
      the sub cluster information
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful

    • getPolicyConfiguration

      public SubClusterPolicyConfiguration getPolicyConfiguration(String queue) throws org.apache.hadoop.yarn.exceptions.YarnException
      Returns the SubClusterPolicyConfiguration for the specified queue.
      Parameters:
      queue - the queue whose policy is required
      Returns:
      the corresponding configured policy, or null if there is no mapping for the queue
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful

    • setPolicyConfiguration

      public void setPolicyConfiguration(SubClusterPolicyConfiguration policyConf) throws org.apache.hadoop.yarn.exceptions.YarnException
      Set a policy configuration into the state store.
      Parameters:
      policyConf - the policy configuration to set
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the request is invalid/fails

    • getPoliciesConfigurations

      public Map<String,SubClusterPolicyConfiguration> getPoliciesConfigurations() throws org.apache.hadoop.yarn.exceptions.YarnException
      Get the policies that is represented as SubClusterPolicyConfiguration for all currently active queues in the system.
      Returns:
      the policies for all currently active queues in the system
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful

    • addApplicationHomeSubCluster

      public SubClusterId addApplicationHomeSubCluster(ApplicationHomeSubCluster appHomeSubCluster) throws org.apache.hadoop.yarn.exceptions.YarnException
      Adds the home SubClusterId for the specified ApplicationId.
      Parameters:
      appHomeSubCluster - the mapping of the application to it's home sub-cluster
      Returns:
      the stored Subcluster from StateStore
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful

    • updateApplicationHomeSubCluster

      public void updateApplicationHomeSubCluster(ApplicationHomeSubCluster appHomeSubCluster) throws org.apache.hadoop.yarn.exceptions.YarnException
      Updates the home SubClusterId for the specified ApplicationId.
      Parameters:
      appHomeSubCluster - the mapping of the application to it's home sub-cluster
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful

    • getApplicationHomeSubCluster

      public SubClusterId getApplicationHomeSubCluster(org.apache.hadoop.yarn.api.records.ApplicationId appId) throws org.apache.hadoop.yarn.exceptions.YarnException
      Returns the home SubClusterId for the specified ApplicationId.
      Parameters:
      appId - the identifier of the application
      Returns:
      the home sub cluster identifier
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful

    • getSubClusterResolver

      public SubClusterResolver getSubClusterResolver()
      Get the singleton instance of SubClusterResolver.
      Returns:
      SubClusterResolver instance

    • getConf

      public org.apache.hadoop.conf.Configuration getConf()
      Get the configuration.
      Returns:
      configuration object

    • addReservationHomeSubCluster

      public SubClusterId addReservationHomeSubCluster(ReservationHomeSubCluster appHomeSubCluster) throws org.apache.hadoop.yarn.exceptions.YarnException
      Adds the home SubClusterId for the specified ReservationId.
      Parameters:
      appHomeSubCluster - the mapping of the reservation to it's home sub-cluster
      Returns:
      the stored subCluster from StateStore
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful

    • getReservationHomeSubCluster

      public SubClusterId getReservationHomeSubCluster(org.apache.hadoop.yarn.api.records.ReservationId reservationId) throws org.apache.hadoop.yarn.exceptions.YarnException
      Returns the home SubClusterId for the specified ReservationId.
      Parameters:
      reservationId - the identifier of the reservation
      Returns:
      the home subCluster identifier
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful

    • updateReservationHomeSubCluster

      public void updateReservationHomeSubCluster(ReservationHomeSubCluster appHomeSubCluster) throws org.apache.hadoop.yarn.exceptions.YarnException
      Updates the home SubClusterId for the specified ReservationId.
      Parameters:
      appHomeSubCluster - the mapping of the reservation to it's home sub-cluster
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful

    • deleteReservationHomeSubCluster

      public void deleteReservationHomeSubCluster(org.apache.hadoop.yarn.api.records.ReservationId reservationId) throws org.apache.hadoop.yarn.exceptions.YarnException
      Delete the home SubClusterId for the specified ReservationId.
      Parameters:
      reservationId - the identifier of the reservation
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful

    • createRetryInstance

      public static <T> Object createRetryInstance(org.apache.hadoop.conf.Configuration conf, String configuredClassName, String defaultValue, Class<T> type, org.apache.hadoop.io.retry.RetryPolicy retryPolicy)
      Helper method to create instances of Object using the class name defined in the configuration object. The instances creates RetryProxy using the specific RetryPolicy.
      Type Parameters:
      T - The type of the instance.
      Parameters:
      conf - the yarn configuration
      configuredClassName - the configuration provider key
      defaultValue - the default implementation for fallback
      type - the class for which a retry proxy is required
      retryPolicy - the policy for retrying method call failures
      Returns:
      a retry proxy for the specified interface

    • createInstance

      public static <T> T createInstance(org.apache.hadoop.conf.Configuration conf, String configuredClassName, String defaultValue, Class<T> type)
      Helper method to create instances of Object using the class name specified in the configuration object.
      Type Parameters:
      T - The type of the instance to create
      Parameters:
      conf - the yarn configuration
      configuredClassName - the configuration provider key
      defaultValue - the default implementation class
      type - the required interface/base class
      Returns:
      the instances created

    • getStateStore

      @VisibleForTesting public FederationStateStore getStateStore()

    • storeNewMasterKey

      public RouterMasterKeyResponse storeNewMasterKey(org.apache.hadoop.security.token.delegation.DelegationKey newKey) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      The Router Supports Store NewMasterKey (RouterMasterKeyRouterMasterKey).
      Parameters:
      newKey - Key used for generating and verifying delegation tokens
      Returns:
      RouterMasterKeyResponse
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful
      IOException - An IO Error occurred

    • removeStoredMasterKey

      public void removeStoredMasterKey(org.apache.hadoop.security.token.delegation.DelegationKey newKey) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      The Router Supports Remove MasterKey (RouterMasterKeyRouterMasterKey).
      Parameters:
      newKey - Key used for generating and verifying delegation tokens
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful
      IOException - An IO Error occurred

    • getMasterKeyByDelegationKey

      public RouterMasterKeyResponse getMasterKeyByDelegationKey(org.apache.hadoop.security.token.delegation.DelegationKey newKey) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      The Router Supports GetMasterKeyByDelegationKey.
      Parameters:
      newKey - Key used for generating and verifying delegation tokens
      Returns:
      RouterMasterKeyResponse
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful
      IOException - An IO Error occurred

    • storeNewToken

      public void storeNewToken(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier identifier, long renewDate) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      The Router Supports Store RMDelegationTokenIdentifierRMDelegationTokenIdentifier.
      Parameters:
      identifier - delegation tokens from the RM
      renewDate - renewDate
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful
      IOException - An IO Error occurred

    • storeNewToken

      public void storeNewToken(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier identifier, long renewDate, String tokenInfo) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      The Router Supports Store RMDelegationTokenIdentifierRMDelegationTokenIdentifier.
      Parameters:
      identifier - delegation tokens from the RM.
      renewDate - renewDate.
      tokenInfo - tokenInfo.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful.
      IOException - An IO Error occurred.

    • updateStoredToken

      public void updateStoredToken(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier identifier, long renewDate) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      The Router Supports Update RMDelegationTokenIdentifierRMDelegationTokenIdentifier.
      Parameters:
      identifier - delegation tokens from the RM
      renewDate - renewDate
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful
      IOException - An IO Error occurred

    • updateStoredToken

      public void updateStoredToken(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier identifier, long renewDate, String tokenInfo) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      The Router Supports Update RMDelegationTokenIdentifierRMDelegationTokenIdentifier.
      Parameters:
      identifier - delegation tokens from the RM
      renewDate - renewDate
      tokenInfo - tokenInfo.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful.
      IOException - An IO Error occurred.

    • removeStoredToken

      public void removeStoredToken(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier identifier) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      The Router Supports Remove RMDelegationTokenIdentifierRMDelegationTokenIdentifier.
      Parameters:
      identifier - delegation tokens from the RM
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful
      IOException - An IO Error occurred

    • getTokenByRouterStoreToken

      public RouterRMTokenResponse getTokenByRouterStoreToken(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier identifier) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      The Router Supports GetTokenByRouterStoreTokenRMDelegationTokenIdentifier.
      Parameters:
      identifier - delegation tokens from the RM
      Returns:
      RouterStoreToken
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful
      IOException - An IO Error occurred

    • incrementDelegationTokenSeqNum

      public int incrementDelegationTokenSeqNum()
      stateStore provides DelegationTokenSeqNum increase.
      Returns:
      delegationTokenSequenceNumber.

    • getDelegationTokenSeqNum

      public int getDelegationTokenSeqNum()
      Get SeqNum from stateStore.
      Returns:
      delegationTokenSequenceNumber.

    • setDelegationTokenSeqNum

      public void setDelegationTokenSeqNum(int seqNum)
      Set SeqNum from stateStore.
      Parameters:
      seqNum - delegationTokenSequenceNumber.

    • getCurrentKeyId

      public int getCurrentKeyId()
      Get CurrentKeyId from stateStore.
      Returns:
      currentKeyId.

    • incrementCurrentKeyId

      public int incrementCurrentKeyId()
      stateStore provides CurrentKeyId increase.
      Returns:
      currentKeyId.

    • getActiveSubClustersCount

      public int getActiveSubClustersCount() throws org.apache.hadoop.yarn.exceptions.YarnException
      Get the number of active cluster nodes.
      Returns:
      number of active cluster nodes.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the call to the state store is unsuccessful.

    • getRandomActiveSubCluster

      public static SubClusterId getRandomActiveSubCluster(Map<SubClusterId,SubClusterInfo> activeSubClusters, List<SubClusterId> blackList) throws org.apache.hadoop.yarn.exceptions.YarnException
      Randomly pick ActiveSubCluster. During the selection process, we will exclude SubClusters from the blacklist.
      Parameters:
      activeSubClusters - List of active subClusters.
      blackList - blacklist.
      Returns:
      Active SubClusterId.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - When there is no Active SubCluster, an exception will be thrown (No active SubCluster available to submit the request.)

    • getRetryNumbers

      public int getRetryNumbers(int configRetries) throws org.apache.hadoop.yarn.exceptions.YarnException
      Get the number of retries.
      Parameters:
      configRetries - User-configured number of retries.
      Returns:
      number of retries.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - yarn exception.

    • existsApplicationHomeSubCluster

      public boolean existsApplicationHomeSubCluster(org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
      Query SubClusterId By applicationId. If SubClusterId is not empty, it means it exists and returns true; if SubClusterId is empty, it means it does not exist and returns false.
      Parameters:
      applicationId - applicationId
      Returns:
      true, SubClusterId exists; false, SubClusterId not exists.

    • addApplicationHomeSubCluster

      public void addApplicationHomeSubCluster(org.apache.hadoop.yarn.api.records.ApplicationId applicationId, ApplicationHomeSubCluster homeSubCluster) throws org.apache.hadoop.yarn.exceptions.YarnException
      Add ApplicationHomeSubCluster to FederationStateStore.
      Parameters:
      applicationId - applicationId.
      homeSubCluster - homeSubCluster, homeSubCluster selected according to policy.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - yarn exception.

    • getApplicationsHomeSubCluster

      public List<ApplicationHomeSubCluster> getApplicationsHomeSubCluster() throws org.apache.hadoop.yarn.exceptions.YarnException
      Get the ApplicationHomeSubCluster list representing the mapping of all submitted applications to it's home sub-cluster.
      Returns:
      the mapping of all submitted application to it's home sub-cluster
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the request is invalid/fails

    • deleteApplicationHomeSubCluster

      public void deleteApplicationHomeSubCluster(org.apache.hadoop.yarn.api.records.ApplicationId applicationId) throws org.apache.hadoop.yarn.exceptions.YarnException
      Delete the mapping of home SubClusterId of a previously submitted ApplicationId. Currently response is empty if the operation is successful, if not an exception reporting reason for a failure.
      Parameters:
      applicationId - the application to delete the home sub-cluster of
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if the request is invalid/fails

    • updateApplicationHomeSubCluster

      public void updateApplicationHomeSubCluster(SubClusterId subClusterId, org.apache.hadoop.yarn.api.records.ApplicationId applicationId, ApplicationHomeSubCluster homeSubCluster) throws org.apache.hadoop.yarn.exceptions.YarnException
      Update ApplicationHomeSubCluster to FederationStateStore.
      Parameters:
      subClusterId - homeSubClusterId
      applicationId - applicationId.
      homeSubCluster - homeSubCluster, homeSubCluster selected according to policy.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - yarn exception.

    • addOrUpdateApplicationHomeSubCluster

      public void addOrUpdateApplicationHomeSubCluster(org.apache.hadoop.yarn.api.records.ApplicationId applicationId, SubClusterId subClusterId, int retryCount, org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext) throws org.apache.hadoop.yarn.exceptions.YarnException
      Add or Update ApplicationHomeSubCluster.
      Parameters:
      applicationId - applicationId, is the id of the application.
      subClusterId - homeSubClusterId, this is selected by strategy.
      retryCount - number of retries.
      appSubmissionContext - appSubmissionContext.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - yarn exception.

    • existsReservationHomeSubCluster

      public boolean existsReservationHomeSubCluster(org.apache.hadoop.yarn.api.records.ReservationId reservationId)
      Exists ReservationHomeSubCluster Mapping.
      Parameters:
      reservationId - reservationId
      Returns:
      true - exist, false - not exist

    • addReservationHomeSubCluster

      public void addReservationHomeSubCluster(org.apache.hadoop.yarn.api.records.ReservationId reservationId, ReservationHomeSubCluster homeSubCluster) throws org.apache.hadoop.yarn.exceptions.YarnException
      Save Reservation And HomeSubCluster Mapping.
      Parameters:
      reservationId - reservationId
      homeSubCluster - homeSubCluster
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - on failure

    • updateReservationHomeSubCluster

      public void updateReservationHomeSubCluster(SubClusterId subClusterId, org.apache.hadoop.yarn.api.records.ReservationId reservationId, ReservationHomeSubCluster homeSubCluster) throws org.apache.hadoop.yarn.exceptions.YarnException
      Update Reservation And HomeSubCluster Mapping.
      Parameters:
      subClusterId - subClusterId
      reservationId - reservationId
      homeSubCluster - homeSubCluster
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - on failure

    • addOrUpdateReservationHomeSubCluster

      public void addOrUpdateReservationHomeSubCluster(org.apache.hadoop.yarn.api.records.ReservationId reservationId, SubClusterId subClusterId, int retryCount) throws org.apache.hadoop.yarn.exceptions.YarnException
      Add or Update ReservationHomeSubCluster.
      Parameters:
      reservationId - reservationId.
      subClusterId - homeSubClusterId, this is selected by strategy.
      retryCount - number of retries.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - yarn exception.

    • deregisterSubCluster

      public boolean deregisterSubCluster(SubClusterId subClusterId, SubClusterState subClusterState) throws org.apache.hadoop.yarn.exceptions.YarnException
      Deregister subCluster, Update the subCluster state to SC_LOST、SC_DECOMMISSIONED etc.
      Parameters:
      subClusterId - subClusterId.
      subClusterState - The state of the subCluster to be updated.
      Returns:
      If Deregister subCluster is successful, return true, otherwise, return false.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - yarn exception.

    • getActiveSubClusters

      public Collection<SubClusterInfo> getActiveSubClusters() throws org.apache.hadoop.yarn.webapp.NotFoundException
      Get active subclusters.
      Returns:
      We will return a list of active subclusters as a Collection.
      Throws:
      org.apache.hadoop.yarn.webapp.NotFoundException

    • getApplicationSubmissionContext

      public org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext getApplicationSubmissionContext(org.apache.hadoop.yarn.api.records.ApplicationId appId)
      Get ApplicationSubmissionContext according to ApplicationId. We don't throw exceptions. If the application cannot be found, we return null.
      Parameters:
      appId - ApplicationId
      Returns:
      ApplicationSubmissionContext of ApplicationId

    • deleteAllPoliciesConfigurations

      public void deleteAllPoliciesConfigurations() throws Exception
      Throws:
      Exception

    • getFederationCache

      @VisibleForTesting public FederationCache getFederationCache()

    • deleteStore

      public void deleteStore() throws Exception
      Throws:
      Exception

    • deletePolicyConfigurations

      public void deletePolicyConfigurations(List<String> queuesList) throws org.apache.hadoop.yarn.exceptions.YarnException
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException