Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf

Class YarnConfigurationStore

java.lang.Object
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.YarnConfigurationStore
All Implemented Interfaces:
AutoCloseable
Direct Known Subclasses:
FSSchedulerConfigurationStore, InMemoryConfigurationStore, LeveldbConfigurationStore, ZKConfigurationStore
public abstract class YarnConfigurationStore extends Object implements AutoCloseable
YarnConfigurationStore exposes the methods needed for retrieving and persisting CapacityScheduler configuration via key-value using write-ahead logging. When configuration mutation is requested, caller should first log it with logMutation, which persists this pending mutation. This mutation is merged to the persisted configuration only after confirmMutation is called. On startup/recovery, caller should call retrieve to get all confirmed mutations, then get pending mutations which were not confirmed via getPendingMutations, and replay/confirm them via confirmMutation as in the normal case.

  • Field Details

    • LOG

      public static final org.slf4j.Logger LOG

  • Constructor Details

    • YarnConfigurationStore

      public YarnConfigurationStore()

  • Method Details

    • initialize

      public abstract void initialize(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.conf.Configuration schedConf, RMContext rmContext) throws Exception
      Initialize the configuration store, with schedConf as the initial scheduler configuration. If a persisted store already exists, use the scheduler configuration stored there, and ignore schedConf.
      Parameters:
      conf - configuration to initialize store with
      schedConf - Initial key-value scheduler configuration to persist.
      rmContext - RMContext for this configuration store
      Throws:
      IOException - if initialization fails
      Exception

    • close

      public abstract void close() throws IOException
      Closes the configuration store, releasing any required resources.
      Specified by:
      close in interface AutoCloseable
      Throws:
      IOException - on failure to close

    • logMutation

      public abstract void logMutation(YarnConfigurationStore.LogMutation logMutation) throws Exception
      Logs the configuration change to backing store.
      Parameters:
      logMutation - configuration change to be persisted in write ahead log
      Throws:
      IOException - if logging fails
      Exception

    • confirmMutation

      public abstract void confirmMutation(YarnConfigurationStore.LogMutation pendingMutation, boolean isValid) throws Exception
      Should be called after logMutation. Gets the pending mutation last logged by logMutation and marks the mutation as persisted (no longer pending). If isValid is true, merge the mutation with the persisted configuration.
      Parameters:
      pendingMutation - the log mutation to apply
      isValid - if true, update persisted configuration with pending mutation.
      Throws:
      Exception - if mutation confirmation fails

    • retrieve

      public abstract org.apache.hadoop.conf.Configuration retrieve() throws IOException
      Retrieve the persisted configuration.
      Returns:
      configuration as key-value
      Throws:
      IOException - an I/O exception has occurred.

    • format

      public abstract void format() throws Exception
      Format the persisted configuration.
      Throws:
      IOException - on failure to format
      Exception

    • getConfigVersion

      public abstract long getConfigVersion() throws Exception
      Get the last updated config version.
      Returns:
      Last updated config version.
      Throws:
      Exception - On version fetch failure.

    • getConfirmedConfHistory

      public abstract List<YarnConfigurationStore.LogMutation> getConfirmedConfHistory(long fromId)
      Get a list of confirmed configuration mutations starting from a given id.
      Parameters:
      fromId - id from which to start getting mutations, inclusive
      Returns:
      list of configuration mutations

    • getConfStoreVersion

      protected abstract org.apache.hadoop.yarn.server.records.Version getConfStoreVersion() throws Exception
      Get schema version of persisted conf store, for detecting compatibility issues when changing conf store schema.
      Returns:
      Schema version currently used by the persisted configuration store.
      Throws:
      Exception - On version fetch failure

    • getLogs

      protected abstract LinkedList<YarnConfigurationStore.LogMutation> getLogs() throws Exception
      Get a list of configuration mutations.
      Returns:
      list of configuration mutations.
      Throws:
      Exception - On mutation fetch failure

    • storeVersion

      protected abstract void storeVersion() throws Exception
      Persist the hard-coded schema version to the conf store.
      Throws:
      Exception - On storage failure

    • getCurrentVersion

      protected abstract org.apache.hadoop.yarn.server.records.Version getCurrentVersion()
      Get the hard-coded schema version, for comparison against the schema version currently persisted.
      Returns:
      Current hard-coded schema version

    • checkVersion

      public void checkVersion() throws Exception
      Throws:
      Exception