Package org.apache.hadoop.yarn.server.sharedcachemanager.store

Class InMemorySCMStore

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.service.CompositeService
org.apache.hadoop.yarn.server.sharedcachemanager.store.SCMStore
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.service.Service
@Private @Evolving public class InMemorySCMStore extends SCMStore
A thread safe version of an in-memory SCM store. The thread safety is implemented with two key pieces: (1) at the mapping level a ConcurrentHashMap is used to allow concurrency to resources and their associated references, and (2) a key level lock is used to ensure mutual exclusion between any operation that accesses a resource with the same key.

To ensure safe key-level locking, we use the original string key and intern it weakly using hadoop's StringInterner. It avoids the pitfalls of using built-in String interning. The interned strings are also weakly referenced, so it can be garbage collected once it is done. And there is little risk of keys being available for other parts of the code so they can be used as locks accidentally.

Resources in the in-memory store are evicted based on a time staleness criteria. If a resource is not referenced (i.e. used) for a given period, it is designated as a stale resource and is considered evictable.

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService

    org.apache.hadoop.service.CompositeService.CompositeServiceShutdownHook

    Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

    org.apache.hadoop.service.Service.STATE

  • Field Summary

    Fields inherited from class org.apache.hadoop.yarn.server.sharedcachemanager.store.SCMStore

    appChecker

    Fields inherited from class org.apache.hadoop.service.CompositeService

    STOP_ONLY_STARTED_SERVICES

  • Constructor Summary

    Constructors
    Constructor
    Description
    InMemorySCMStore()
     
    InMemorySCMStore(AppChecker appChecker)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    String
    addResource(String key, String fileName)
    Adds the given resource to the store under the key and the filename.
    String
    addResourceReference(String key, SharedCacheResourceReference ref)
    Adds the provided resource reference to the cache resource under the key, and updates the access time.
    void
    cleanResourceReferences(String key)
    Provides atomicity for the method.
    Collection<SharedCacheResourceReference>
    getResourceReferences(String key)
    Returns the list of resource references currently registered under the cache entry.
    boolean
    isResourceEvictable(String key, org.apache.hadoop.fs.FileStatus file)
    Check if a specific resource is evictable according to the store's enabled cache eviction policies.
    boolean
    removeResource(String key)
    Removes the given resource from the store.
    boolean
    removeResourceReference(String key, SharedCacheResourceReference ref, boolean updateAccessTime)
    Removes the provided resource reference from the resource.
    void
    removeResourceReferences(String key, Collection<SharedCacheResourceReference> refs, boolean updateAccessTime)
    Removes the provided collection of resource references from the resource.
    protected void
    serviceInit(org.apache.hadoop.conf.Configuration conf)
    The in-memory store bootstraps itself from the shared cache entries that exist in HDFS.
    protected void
    serviceStart()
     
    protected void
    serviceStop()
     

    Methods inherited from class org.apache.hadoop.yarn.server.sharedcachemanager.store.SCMStore

    createAppCheckerService

    Methods inherited from class org.apache.hadoop.service.CompositeService

    addIfService, addService, getServices, removeService

    Methods inherited from class org.apache.hadoop.service.AbstractService

    close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Constructor Details

    • InMemorySCMStore

      public InMemorySCMStore()

    • InMemorySCMStore

      @VisibleForTesting public InMemorySCMStore(AppChecker appChecker)

  • Method Details

    • serviceInit

      protected void serviceInit(org.apache.hadoop.conf.Configuration conf) throws Exception
      The in-memory store bootstraps itself from the shared cache entries that exist in HDFS.
      Overrides:
      serviceInit in class SCMStore
      Throws:
      Exception

    • serviceStart

      protected void serviceStart() throws Exception
      Overrides:
      serviceStart in class org.apache.hadoop.service.CompositeService
      Throws:
      Exception

    • serviceStop

      protected void serviceStop() throws Exception
      Overrides:
      serviceStop in class org.apache.hadoop.service.CompositeService
      Throws:
      Exception

    • addResource

      public String addResource(String key, String fileName)
      Adds the given resource to the store under the key and the filename. If the entry is already found, it returns the existing filename. It represents the state of the store at the time of this query. The entry may change or even be removed once this method returns. The caller should be prepared to handle that situation.
      Specified by:
      addResource in class SCMStore
      Parameters:
      key - a unique identifier for a resource
      fileName - the filename of the resource
      Returns:
      the filename of the newly inserted resource or that of the existing resource

    • addResourceReference

      public String addResourceReference(String key, SharedCacheResourceReference ref)
      Adds the provided resource reference to the cache resource under the key, and updates the access time. If it returns a non-null value, the caller may safely assume that the resource will not be removed at least until the app in this resource reference has terminated.
      Specified by:
      addResourceReference in class SCMStore
      Parameters:
      key - a unique identifier for a resource
      ref - the SharedCacheResourceReference to add
      Returns:
      the filename of the resource, or null if the resource is not found

    • getResourceReferences

      public Collection<SharedCacheResourceReference> getResourceReferences(String key)
      Returns the list of resource references currently registered under the cache entry. If the list is empty, it returns an empty collection. The returned collection is unmodifiable and a snapshot of the information at the time of the query. The state may change after this query returns. The caller should handle the situation that some or all of these resource references are no longer relevant.
      Specified by:
      getResourceReferences in class SCMStore
      Parameters:
      key - a unique identifier for a resource
      Returns:
      the collection that contains the resource references associated with the resource; or an empty collection if no resource references are registered under this resource

    • removeResourceReference

      public boolean removeResourceReference(String key, SharedCacheResourceReference ref, boolean updateAccessTime)
      Removes the provided resource reference from the resource. If the resource does not exist, nothing will be done.
      Specified by:
      removeResourceReference in class SCMStore
      Parameters:
      key - a unique identifier for a resource
      ref - the SharedCacheResourceReference to remove
      updateAccessTime - true if the call should update the access time for the resource
      Returns:
      true if the reference was removed, false otherwise

    • removeResourceReferences

      public void removeResourceReferences(String key, Collection<SharedCacheResourceReference> refs, boolean updateAccessTime)
      Removes the provided collection of resource references from the resource. If the resource does not exist, nothing will be done.
      Specified by:
      removeResourceReferences in class SCMStore
      Parameters:
      key - a unique identifier for a resource
      refs - the collection of SharedCacheResourceReferences to remove
      updateAccessTime - true if the call should update the access time for the resource

    • cleanResourceReferences

      public void cleanResourceReferences(String key) throws org.apache.hadoop.yarn.exceptions.YarnException
      Provides atomicity for the method.
      Overrides:
      cleanResourceReferences in class SCMStore
      Parameters:
      key - a unique identifier for a resource
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException

    • removeResource

      public boolean removeResource(String key)
      Removes the given resource from the store. Returns true if the resource is found and removed or if the resource is not found. Returns false if it was unable to remove the resource because the resource reference list was not empty.
      Specified by:
      removeResource in class SCMStore
      Parameters:
      key - a unique identifier for a resource
      Returns:
      true if the resource was removed or did not exist, false if the resource existed, contained at least one SharedCacheResourceReference and was not removed.

    • isResourceEvictable

      public boolean isResourceEvictable(String key, org.apache.hadoop.fs.FileStatus file)
      Description copied from class: SCMStore
      Check if a specific resource is evictable according to the store's enabled cache eviction policies.
      Specified by:
      isResourceEvictable in class SCMStore
      Parameters:
      key - a unique identifier for a resource
      file - the FileStatus object for the resource file in the file system.
      Returns:
      true if the resource is evicatble, false otherwise