Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity

Class CapacityScheduler

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.service.Service, org.apache.hadoop.yarn.event.EventHandler<SchedulerEvent>, Recoverable, CapacitySchedulerContext, ResourceAllocationCommitter, MutableConfScheduler, PreemptableResourceScheduler, ResourceScheduler, YarnScheduler

@LimitedPrivate("yarn")
@Evolving
public class CapacityScheduler
extends AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>
implements PreemptableResourceScheduler, CapacitySchedulerContext, org.apache.hadoop.conf.Configurable, ResourceAllocationCommitter, MutableConfScheduler

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler

AbstractYarnScheduler.ContainerObjectType

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler

activitiesManager, applications, EMPTY_ALLOCATION, lastNodeUpdateTime, minimumAllocation, nmExpireInterval, nmHeartbeatInterval, nodeTracker, readLock, rmContext, schedulerHealth, schedulingMonitorManager, THREAD_JOIN_TIMEOUT_MS, updateInterval, writeLock

Constructor Summary

Constructors

Constructor	Description
CapacityScheduler()

Method Summary

Modifier and Type	Method	Description
void	addNodes(List<FiCaSchedulerNode> nodesToAdd)	Add node to nodeTracker.
void	addQueue(Queue queue)	Add to the scheduler a new Queue.
Allocation	allocate(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, List<org.apache.hadoop.yarn.api.records.ResourceRequest> ask, List<org.apache.hadoop.yarn.api.records.SchedulingRequest> schedulingRequests, List<org.apache.hadoop.yarn.api.records.ContainerId> release, List<String> blacklistAdditions, List<String> blacklistRemovals, ContainerUpdates updateRequests)	The main API between the ApplicationMaster and the Scheduler.
boolean	attemptAllocationOnNode(SchedulerApplicationAttempt appAttempt, org.apache.hadoop.yarn.api.records.SchedulingRequest schedulingRequest, SchedulerNode schedulerNode)	Default implementation.
boolean	checkAccess(org.apache.hadoop.security.UserGroupInformation callerUGI, org.apache.hadoop.yarn.api.records.QueueACL acl, String queueName)	Check if the user has permission to perform the operation.
long	checkAndGetApplicationLifetime(String queueName, long lifetimeRequestedByApp, RMAppImpl app)	Verify whether a submitted application lifetime is valid as per configured Queue lifetime.
org.apache.hadoop.yarn.api.records.Priority	checkAndGetApplicationPriority(org.apache.hadoop.yarn.api.records.Priority priorityRequestedByApp, org.apache.hadoop.security.UserGroupInformation user, String queuePath, org.apache.hadoop.yarn.api.records.ApplicationId applicationId)	Verify whether a submitted application priority is valid as per configured Queue
protected void	completedContainerInternal(RMContainer rmContainer, org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus, RMContainerEventType event)
ResourceCommitRequest<FiCaSchedulerApp,FiCaSchedulerNode>	createResourceCommitRequest(CSAssignment csAssignment)
List<FiCaSchedulerNode>	getAllNodes()
FiCaSchedulerApp	getApplicationAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId)
List<org.apache.hadoop.yarn.api.records.ApplicationAttemptId>	getAppsInQueue(String queueName)	Gets the apps under a given queue
List<org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.AsyncScheduleThread>	getAsyncSchedulerThreads()
int	getAsyncSchedulingPendingBacklogs()
CapacitySchedulerQueueManager	getCapacitySchedulerQueueManager()
ResourceUsage	getClusterResourceUsage()
org.apache.hadoop.conf.Configuration	getConf()	Get the yarn configuration.
CapacitySchedulerConfiguration	getConfiguration()	Get the scheduler configuration.
RMContainerTokenSecretManager	getContainerTokenSecretManager()
CSConfigurationProvider	getCsConfProvider()
PlacementRule	getCSMappingPlacementRule()
long	getMaximumApplicationLifetime(String queueName)	Get maximum lifetime for a queue.
org.apache.hadoop.yarn.api.records.Resource	getMaximumResourceCapability(String queueName)	Get maximum allocatable Resource for the queue specified.
MutableConfigurationProvider	getMutableConfProvider()	Get scheduler's configuration provider, so other classes can directly call mutation APIs on configuration provider.
FiCaSchedulerNode	getNode(org.apache.hadoop.yarn.api.records.NodeId nodeId)
int	getNumAsyncSchedulerThreads()
int	getNumClusterNodes()	Get the number of nodes available in the cluster.
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.PendingApplicationComparator	getPendingApplicationComparator()
Set<String>	getPlanQueues()	Gets the list of names for queues managed by the Reservation System.
PreemptionManager	getPreemptionManager()
CSQueue	getQueue(String queueName)	Get queue object based on queue name.
CapacitySchedulerQueueContext	getQueueContext()
org.apache.hadoop.yarn.api.records.QueueInfo	getQueueInfo(String queueName, boolean includeChildQueues, boolean recursive)	Get queue information.
List<org.apache.hadoop.yarn.api.records.QueueUserACLInfo>	getQueueUserAclInfo()	Get acls for queues for current user.
org.apache.hadoop.yarn.util.resource.ResourceCalculator	getResourceCalculator()
RMContext	getRMContext()
CSQueue	getRootQueue()
QueueMetrics	getRootQueueMetrics()	Get the root queue for the scheduler.
EnumSet<org.apache.hadoop.yarn.proto.YarnServiceProtos.SchedulerResourceTypes>	getSchedulingResourceTypes()	Return a collection of the resource types that are considered when scheduling
WorkflowPriorityMappingsManager	getWorkflowPriorityMappingsManager()
void	handle(SchedulerEvent event)
boolean	isAmbiguous(String queueName)	Determines if a short queue name reference is ambiguous, if there are at least two queues with the same name, it is considered ambiguous.
boolean	isConfigurationMutable()	Returns if configuration is mutable.
boolean	isMultiNodePlacementEnabled()
boolean	isSystemAppsLimitReached()
void	killContainer(RMContainer container)	Kill a RMContainer.
void	killReservedContainer(RMContainer container)	If the scheduler support container reservations, this method is used to ask the scheduler to drop the reservation for the given container.
void	markContainerForKillable(RMContainer killableContainer)	Ask the scheduler to forcibly interrupt the container given as input.
void	markContainerForPreemption(org.apache.hadoop.yarn.api.records.ApplicationAttemptId aid, RMContainer cont)	Ask the scheduler to obtain back the container from a specific application by issuing a preemption request
String	moveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId, String targetQueueName)	Moves the given application to the given queue.
boolean	moveReservedContainer(RMContainer toBeMovedContainer, FiCaSchedulerNode targetNode)	Try to move a reserved container to a targetNode.
protected void	nodeUpdate(RMNode rmNode)	Process a heartbeat update from a node.
String	normalizeQueueName(String name)	Returns the normalized queue name, which should be used for internal queue references.
boolean	placementConstraintEnabled()	Returning true as capacity scheduler supports placement constraints.
void	preValidateMoveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId, String newQueue)
void	recover(RMStateStore.RMState state)
void	reinitialize(org.apache.hadoop.conf.Configuration newConf, RMContext rmContext)	Re-initialize the ResourceScheduler.
void	reinitialize(org.apache.hadoop.conf.Configuration newConf, RMContext rmContext, boolean validation)
void	removeQueue(String queueName)	Remove an existing queue.
void	removeQueue(CSQueue queue)
void	resetSchedulerMetrics()	Reset scheduler metrics.
void	serviceInit(org.apache.hadoop.conf.Configuration conf)
void	serviceStart()
void	serviceStop()
void	setAsyncSchedulingConf(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.AsyncSchedulingConfiguration conf)
void	setConf(org.apache.hadoop.conf.Configuration conf)
void	setEntitlement(String inQueue, QueueEntitlement entitlement)	This method increase the entitlement for current queue (must respect invariants, e.g., no overcommit of parents, non negative, etc.).
void	setMaxRunningAppsEnforcer(CSMaxRunningAppsEnforcer enforcer)
void	setQueueManager(CapacitySchedulerQueueManager qm)
void	setResourceCalculator(org.apache.hadoop.yarn.util.resource.ResourceCalculator rc)
void	setRMContext(RMContext rmContext)	Set RMContext for ResourceScheduler.
static boolean	shouldSkipNodeSchedule(FiCaSchedulerNode node, CapacityScheduler cs, boolean printVerboseLog)
void	submitResourceCommitRequest(org.apache.hadoop.yarn.api.records.Resource cluster, CSAssignment csAssignment)
boolean	tryCommit(org.apache.hadoop.yarn.api.records.Resource cluster, ResourceCommitRequest r, boolean updatePending)	Try to commit the allocation Proposal.
org.apache.hadoop.yarn.api.records.Priority	updateApplicationPriority(org.apache.hadoop.yarn.api.records.Priority newPriority, org.apache.hadoop.yarn.api.records.ApplicationId applicationId, org.apache.hadoop.thirdparty.com.google.common.util.concurrent.SettableFuture<Object> future, org.apache.hadoop.security.UserGroupInformation user)	Change application priority of a submitted application at runtime
void	updatePlacementRules()

Methods inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler

asyncContainerRelease, autoCorrectContainerAllocation, clearPendingContainerCache, completedContainer, containerIncreasedOnNode, containerLaunchedOnNode, createReleaseCache, createSchedContainerChangeRequests, getActivitiesManager, getAppResourceUsageReport, getBlacklistedNodes, getClock, getClusterResource, getCurrentAttemptForContainer, getLastNodeUpdateTime, getMaxClusterLevelAppPriority, getMaximumAllocation, getMaximumResourceCapability, getMinimumAllocation, getMinimumResourceCapability, getNodeIds, getNodeReport, getNodes, getNodeTracker, getNormalizedResource, getPendingResourceRequestsForAttempt, getPendingSchedulingRequestsForAttempt, getRMContainer, getSchedulerAppInfo, getSchedulerApplications, getSchedulerHealth, getSchedulerNode, getSchedulingMonitorManager, getSkipNodeInterval, getTransferredContainers, handleContainerUpdates, initMaximumResourceCapability, killAllAppsInQueue, moveAllApps, normalizeResourceRequests, normalizeResourceRequests, recoverContainersOnNode, refreshMaximumAllocation, releaseContainers, rollbackContainerUpdate, setClock, setClusterMaxPriority, setLastNodeUpdateTime, shouldContainersBeAutoUpdated, triggerUpdate, update, updateNodeResource, updateNodeResourceUtilization, updateSchedulerHealthInformation

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerContext

getActivitiesManager, getClock, getClusterResource, getLastNodeUpdateTime, getMaxClusterLevelAppPriority, getMaximumResourceCapability, getMinimumResourceCapability, getSchedulerHealth

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler

getNodeIds

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler

getAppResourceUsageReport, getClusterResource, getMaxClusterLevelAppPriority, getMaximumResourceCapability, getMinimumResourceCapability, getNodeReport, getNormalizedResource, getPendingResourceRequestsForAttempt, getPendingSchedulingRequestsForAttempt, getRMContainer, getSchedulerAppInfo, getSchedulerNode, getTransferredContainers, killAllAppsInQueue, moveAllApps, setClusterMaxPriority

Constructor Details

CapacityScheduler

public CapacityScheduler()

Method Details

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Description copied from interface: CapacitySchedulerContext

Get the yarn configuration.

Specified by:

getConf in interface CapacitySchedulerContext

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

Returns:

yarn configuration.

getRootQueueMetrics

public QueueMetrics getRootQueueMetrics()

Description copied from interface: YarnScheduler

Get the root queue for the scheduler.

Specified by:

getRootQueueMetrics in interface YarnScheduler

Returns:

the root queue for the scheduler.

getRootQueue

public CSQueue getRootQueue()

getConfiguration

public CapacitySchedulerConfiguration getConfiguration()

Description copied from interface: MutableConfScheduler

Get the scheduler configuration.

Specified by:

getConfiguration in interface CapacitySchedulerContext

Specified by:

getConfiguration in interface MutableConfScheduler

Returns:

the scheduler configuration

getQueueContext

public CapacitySchedulerQueueContext getQueueContext()

Specified by:

getQueueContext in interface CapacitySchedulerContext

getContainerTokenSecretManager

public RMContainerTokenSecretManager getContainerTokenSecretManager()

Specified by:

getContainerTokenSecretManager in interface CapacitySchedulerContext

getResourceCalculator

public org.apache.hadoop.yarn.util.resource.ResourceCalculator getResourceCalculator()

Specified by:

getResourceCalculator in interface CapacitySchedulerContext

Specified by:

getResourceCalculator in interface YarnScheduler

setResourceCalculator

@VisibleForTesting
public void setResourceCalculator(org.apache.hadoop.yarn.util.resource.ResourceCalculator rc)

getNumClusterNodes

public int getNumClusterNodes()

Description copied from interface: YarnScheduler

Get the number of nodes available in the cluster.

Specified by:

getNumClusterNodes in interface CapacitySchedulerContext

Specified by:

getNumClusterNodes in interface YarnScheduler

Returns:

the number of available nodes.

getRMContext

public RMContext getRMContext()

Specified by:

getRMContext in interface CapacitySchedulerContext

setRMContext

public void setRMContext(RMContext rmContext)

Description copied from interface: ResourceScheduler

Set RMContext for ResourceScheduler. This method should be called immediately after instantiating a scheduler once.

Specified by:

setRMContext in interface ResourceScheduler

Parameters:

rmContext - created by ResourceManager

serviceInit

public void serviceInit(org.apache.hadoop.conf.Configuration conf)
                 throws Exception

Overrides:

serviceInit in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Throws:

Exception

serviceStart

public void serviceStart()
                  throws Exception

Overrides:

serviceStart in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Throws:

Exception

serviceStop

public void serviceStop()
                 throws Exception

Overrides:

serviceStop in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Throws:

Exception

reinitialize

public void reinitialize(org.apache.hadoop.conf.Configuration newConf,
 RMContext rmContext,
 boolean validation)
                  throws IOException

Throws:

IOException

reinitialize

public void reinitialize(org.apache.hadoop.conf.Configuration newConf,
 RMContext rmContext)
                  throws IOException

Description copied from interface: ResourceScheduler

Re-initialize the ResourceScheduler.

Specified by:

reinitialize in interface ResourceScheduler

Overrides:

reinitialize in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

newConf - configuration

rmContext - RMContext.

Throws:

IOException - an I/O exception has occurred.

shouldSkipNodeSchedule

@VisibleForTesting
public static boolean shouldSkipNodeSchedule(FiCaSchedulerNode node,
 CapacityScheduler cs,
 boolean printVerboseLog)

setAsyncSchedulingConf

@VisibleForTesting
public void setAsyncSchedulingConf(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.AsyncSchedulingConfiguration conf)

getCSMappingPlacementRule

@VisibleForTesting
public PlacementRule getCSMappingPlacementRule()
                                        throws IOException

Throws:

IOException

updatePlacementRules

@VisibleForTesting
public void updatePlacementRules()
                          throws IOException

Throws:

IOException

getQueue

public CSQueue getQueue(String queueName)

Description copied from interface: MutableConfScheduler

Get queue object based on queue name.

Specified by:

getQueue in interface MutableConfScheduler

Parameters:

queueName - the queue name

Returns:

the queue object

normalizeQueueName

public String normalizeQueueName(String name)

Returns the normalized queue name, which should be used for internal queue references. Currently this is the fullQueuename which disambiguously identifies a queue.

Parameters:

name - Name of the queue to be normalized

Returns:

The normalized (full name) of the queue

isAmbiguous

public boolean isAmbiguous(String queueName)

Determines if a short queue name reference is ambiguous, if there are at least two queues with the same name, it is considered ambiguous. Otherwise it is not.

Parameters:

queueName - The name of the queue to check for ambiguity

Returns:

true if there are at least 2 queues with the same name

allocate

@Lock(org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public Allocation allocate(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId,
 List<org.apache.hadoop.yarn.api.records.ResourceRequest> ask,
 List<org.apache.hadoop.yarn.api.records.SchedulingRequest> schedulingRequests,
 List<org.apache.hadoop.yarn.api.records.ContainerId> release,
 List<String> blacklistAdditions,
 List<String> blacklistRemovals,
 ContainerUpdates updateRequests)

Description copied from interface: YarnScheduler

The main API between the ApplicationMaster and the Scheduler. The ApplicationMaster may request/update container resources, number of containers, node/rack preference for allocations etc. to the Scheduler.

Specified by:

allocate in interface YarnScheduler

Parameters:

applicationAttemptId - the id of the application attempt.

ask - the request made by an application to obtain various allocations like host/rack, resource, number of containers, relaxLocality etc., see ResourceRequest.

schedulingRequests - similar to ask, but with added ability to specify allocation tags etc., see SchedulingRequest.

release - the list of containers to be released.

blacklistAdditions - places (node/rack) to be added to the blacklist.

blacklistRemovals - places (node/rack) to be removed from the blacklist.

updateRequests - container promotion/demotion updates.

Returns:

the Allocation for the application.

getQueueInfo

@Lock(org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public org.apache.hadoop.yarn.api.records.QueueInfo getQueueInfo(String queueName,
 boolean includeChildQueues,
 boolean recursive)
                                                          throws IOException

Description copied from interface: YarnScheduler

Get queue information.

Specified by:

getQueueInfo in interface YarnScheduler

Parameters:

queueName - queue name

includeChildQueues - include child queues?

recursive - get children queues?

Returns:

queue information

Throws:

IOException - an I/O exception has occurred.

getQueueUserAclInfo

@Lock(org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public List<org.apache.hadoop.yarn.api.records.QueueUserACLInfo> getQueueUserAclInfo()

Description copied from interface: YarnScheduler

Get acls for queues for current user.

Specified by:

getQueueUserAclInfo in interface YarnScheduler

Returns:

acls for queues for current user

nodeUpdate

protected void nodeUpdate(RMNode rmNode)

Description copied from class: AbstractYarnScheduler

Process a heartbeat update from a node.

Overrides:

nodeUpdate in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

rmNode - The RMNode corresponding to the NodeManager

handle

public void handle(SchedulerEvent event)

Specified by:

handle in interface org.apache.hadoop.yarn.event.EventHandler<SchedulerEvent>

addNodes

public void addNodes(List<FiCaSchedulerNode> nodesToAdd)

Add node to nodeTracker. Used when validating CS configuration by instantiating a new CS instance.

Parameters:

nodesToAdd - node to be added

completedContainerInternal

protected void completedContainerInternal(RMContainer rmContainer,
 org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus,
 RMContainerEventType event)

Specified by:

completedContainerInternal in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

getApplicationAttempt

@Lock(org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
@VisibleForTesting
public FiCaSchedulerApp getApplicationAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId)

Specified by:

getApplicationAttempt in interface CapacitySchedulerContext

Overrides:

getApplicationAttempt in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

getNode

@Lock(org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public FiCaSchedulerNode getNode(org.apache.hadoop.yarn.api.records.NodeId nodeId)

Specified by:

getNode in interface CapacitySchedulerContext

Overrides:

getNode in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

getAllNodes

@Lock(org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public List<FiCaSchedulerNode> getAllNodes()

recover

@Lock(org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public void recover(RMStateStore.RMState state)
             throws Exception

Specified by:

recover in interface Recoverable

Throws:

Exception

killReservedContainer

public void killReservedContainer(RMContainer container)

Description copied from interface: PreemptableResourceScheduler

If the scheduler support container reservations, this method is used to ask the scheduler to drop the reservation for the given container.

Specified by:

killReservedContainer in interface PreemptableResourceScheduler

Parameters:

container - Reference to reserved container allocation.

markContainerForPreemption

public void markContainerForPreemption(org.apache.hadoop.yarn.api.records.ApplicationAttemptId aid,
 RMContainer cont)

Description copied from interface: PreemptableResourceScheduler

Ask the scheduler to obtain back the container from a specific application by issuing a preemption request

Specified by:

markContainerForPreemption in interface PreemptableResourceScheduler

Parameters:

aid - the application from which we want to get a container back

cont - the container we want back

killContainer

@VisibleForTesting
public void killContainer(RMContainer container)

Description copied from class: AbstractYarnScheduler

Kill a RMContainer. This is meant to be called in tests only to simulate AM container failures.

Specified by:

killContainer in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

container - the container to kill

markContainerForKillable

public void markContainerForKillable(RMContainer killableContainer)

Description copied from interface: PreemptableResourceScheduler

Ask the scheduler to forcibly interrupt the container given as input.

Specified by:

markContainerForKillable in interface PreemptableResourceScheduler

Parameters:

killableContainer - RMContainer.

checkAccess

public boolean checkAccess(org.apache.hadoop.security.UserGroupInformation callerUGI,
 org.apache.hadoop.yarn.api.records.QueueACL acl,
 String queueName)

Description copied from interface: YarnScheduler

Check if the user has permission to perform the operation. If the user has QueueACL.ADMINISTER_QUEUE permission, this user can view/modify the applications in this queue.

Specified by:

checkAccess in interface YarnScheduler

Parameters:

callerUGI - caller UserGroupInformation.

acl - queue ACL.

queueName - queue Name.

Returns:

true if the user has the permission, false otherwise

getAppsInQueue

public List<org.apache.hadoop.yarn.api.records.ApplicationAttemptId> getAppsInQueue(String queueName)

Description copied from interface: YarnScheduler

Gets the apps under a given queue

Specified by:

getAppsInQueue in interface YarnScheduler

Parameters:

queueName - the name of the queue.

Returns:

a collection of app attempt ids in the given queue.

isSystemAppsLimitReached

public boolean isSystemAppsLimitReached()

removeQueue

public void removeQueue(String queueName)
                 throws SchedulerDynamicEditException

Description copied from interface: YarnScheduler

Remove an existing queue. Implementations might limit when a queue could be removed (e.g., must have zero entitlement, and no applications running, or must be a leaf, etc..).

Specified by:

removeQueue in interface YarnScheduler

Overrides:

removeQueue in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

queueName - name of the queue to remove

Throws:

SchedulerDynamicEditException

removeQueue

public void removeQueue(CSQueue queue)
                 throws SchedulerDynamicEditException

Throws:

SchedulerDynamicEditException

addQueue

public void addQueue(Queue queue)
              throws SchedulerDynamicEditException,
IOException

Description copied from interface: YarnScheduler

Add to the scheduler a new Queue. Implementations might limit what type of queues can be dynamically added (e.g., Queue must be a leaf, must be attached to existing parent, must have zero entitlement).

Specified by:

addQueue in interface YarnScheduler

Overrides:

addQueue in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

queue - the queue being added.

Throws:

IOException - when io exception occur.

SchedulerDynamicEditException

setEntitlement

public void setEntitlement(String inQueue,
 QueueEntitlement entitlement)
                    throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

This method increase the entitlement for current queue (must respect invariants, e.g., no overcommit of parents, non negative, etc.). Entitlement is a general term for weights in FairScheduler, capacity for the CapacityScheduler, etc.

Specified by:

setEntitlement in interface YarnScheduler

Overrides:

setEntitlement in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

inQueue - the queue for which we change entitlement

entitlement - the new entitlement for the queue (capacity, maxCapacity, etc..)

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

moveApplication

public String moveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId,
 String targetQueueName)
                       throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Moves the given application to the given queue.

Specified by:

moveApplication in interface YarnScheduler

Overrides:

moveApplication in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

appId - application Id

targetQueueName - the given queue.

Returns:

the name of the queue the application was placed into

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if the move cannot be carried out

preValidateMoveApplication

public void preValidateMoveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId,
 String newQueue)
                                throws org.apache.hadoop.yarn.exceptions.YarnException

Specified by:

preValidateMoveApplication in interface YarnScheduler

Overrides:

preValidateMoveApplication in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

appId - Application ID

newQueue - Target QueueName

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if the pre-validation for move cannot be carried out

getSchedulingResourceTypes

public EnumSet<org.apache.hadoop.yarn.proto.YarnServiceProtos.SchedulerResourceTypes> getSchedulingResourceTypes()

Return a collection of the resource types that are considered when scheduling

Specified by:

getSchedulingResourceTypes in interface YarnScheduler

Overrides:

getSchedulingResourceTypes in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Returns:

an EnumSet containing the resource types

getMaximumResourceCapability

public org.apache.hadoop.yarn.api.records.Resource getMaximumResourceCapability(String queueName)

Description copied from interface: YarnScheduler

Get maximum allocatable Resource for the queue specified.

Specified by:

getMaximumResourceCapability in interface CapacitySchedulerContext

Specified by:

getMaximumResourceCapability in interface YarnScheduler

Overrides:

getMaximumResourceCapability in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

queueName - queue name

Returns:

maximum allocatable resource

getPlanQueues

public Set<String> getPlanQueues()

Description copied from interface: YarnScheduler

Gets the list of names for queues managed by the Reservation System.

Specified by:

getPlanQueues in interface YarnScheduler

Overrides:

getPlanQueues in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Returns:

the list of queues which support reservations

checkAndGetApplicationPriority

public org.apache.hadoop.yarn.api.records.Priority checkAndGetApplicationPriority(org.apache.hadoop.yarn.api.records.Priority priorityRequestedByApp,
 org.apache.hadoop.security.UserGroupInformation user,
 String queuePath,
 org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
                                                                           throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Verify whether a submitted application priority is valid as per configured Queue

Specified by:

checkAndGetApplicationPriority in interface YarnScheduler

Overrides:

checkAndGetApplicationPriority in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

priorityRequestedByApp - Submitted Application priority.

user - User who submitted the Application

queuePath - Name of the Queue

applicationId - Application ID

Returns:

Updated Priority from scheduler

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

updateApplicationPriority

public org.apache.hadoop.yarn.api.records.Priority updateApplicationPriority(org.apache.hadoop.yarn.api.records.Priority newPriority,
 org.apache.hadoop.yarn.api.records.ApplicationId applicationId,
 org.apache.hadoop.thirdparty.com.google.common.util.concurrent.SettableFuture<Object> future,
 org.apache.hadoop.security.UserGroupInformation user)
                                                                      throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Change application priority of a submitted application at runtime

Specified by:

updateApplicationPriority in interface YarnScheduler

Overrides:

updateApplicationPriority in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

newPriority - Submitted Application priority.

applicationId - Application ID

future - Sets any type of exception happened from StateStore

user - who submitted the application

Returns:

updated priority

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

getPreemptionManager

public PreemptionManager getPreemptionManager()

Specified by:

getPreemptionManager in interface CapacitySchedulerContext

getClusterResourceUsage

public ResourceUsage getClusterResourceUsage()

Specified by:

getClusterResourceUsage in interface CapacitySchedulerContext

Returns:

QueueCapacities root queue of the Capacity Scheduler Queue, root queue used capacities for different labels are same as that of the cluster.

submitResourceCommitRequest

@VisibleForTesting
public void submitResourceCommitRequest(org.apache.hadoop.yarn.api.records.Resource cluster,
 CSAssignment csAssignment)

attemptAllocationOnNode

public boolean attemptAllocationOnNode(SchedulerApplicationAttempt appAttempt,
 org.apache.hadoop.yarn.api.records.SchedulingRequest schedulingRequest,
 SchedulerNode schedulerNode)

Description copied from class: AbstractYarnScheduler

Default implementation. Always returns false.

Specified by:

attemptAllocationOnNode in interface ResourceScheduler

Overrides:

attemptAllocationOnNode in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

appAttempt - ApplicationAttempt.

schedulingRequest - SchedulingRequest.

schedulerNode - SchedulerNode.

Returns:

Success or not.

createResourceCommitRequest

@VisibleForTesting
public ResourceCommitRequest<FiCaSchedulerApp,FiCaSchedulerNode> createResourceCommitRequest(CSAssignment csAssignment)

tryCommit

public boolean tryCommit(org.apache.hadoop.yarn.api.records.Resource cluster,
 ResourceCommitRequest r,
 boolean updatePending)

Description copied from interface: ResourceAllocationCommitter

Try to commit the allocation Proposal. This also gives the option of not updating a pending queued request.

Specified by:

tryCommit in interface ResourceAllocationCommitter

Parameters:

cluster - Cluster Resource.

r - Proposal.

updatePending - Decrement pending if successful.

Returns:

Is successful or not.

getAsyncSchedulingPendingBacklogs

public int getAsyncSchedulingPendingBacklogs()

getCapacitySchedulerQueueManager

public CapacitySchedulerQueueManager getCapacitySchedulerQueueManager()

Specified by:

getCapacitySchedulerQueueManager in interface CapacitySchedulerContext

getWorkflowPriorityMappingsManager

public WorkflowPriorityMappingsManager getWorkflowPriorityMappingsManager()

moveReservedContainer

public boolean moveReservedContainer(RMContainer toBeMovedContainer,
 FiCaSchedulerNode targetNode)

Try to move a reserved container to a targetNode. If the targetNode is reserved by another application (other than this one). The previous reservation will be cancelled.

Parameters:

toBeMovedContainer - reserved container will be moved

targetNode - targetNode

Returns:

true if move succeeded. Return false if the targetNode is reserved by a different container or move failed because of any other reasons.

checkAndGetApplicationLifetime

public long checkAndGetApplicationLifetime(String queueName,
 long lifetimeRequestedByApp,
 RMAppImpl app)

Description copied from interface: YarnScheduler

Verify whether a submitted application lifetime is valid as per configured Queue lifetime.

Specified by:

checkAndGetApplicationLifetime in interface YarnScheduler

Overrides:

checkAndGetApplicationLifetime in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

queueName - Name of the Queue

lifetimeRequestedByApp - configured application lifetime

app - details of app

Returns:

valid lifetime as per queue

getMaximumApplicationLifetime

public long getMaximumApplicationLifetime(String queueName)

Description copied from interface: YarnScheduler

Get maximum lifetime for a queue.

Specified by:

getMaximumApplicationLifetime in interface YarnScheduler

Overrides:

getMaximumApplicationLifetime in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

queueName - to get lifetime

Returns:

maximum lifetime in seconds

isConfigurationMutable

public boolean isConfigurationMutable()

Description copied from interface: CapacitySchedulerContext

Returns if configuration is mutable.

Specified by:

isConfigurationMutable in interface CapacitySchedulerContext

Specified by:

isConfigurationMutable in interface MutableConfScheduler

Returns:

if configuration is mutable

getMutableConfProvider

public MutableConfigurationProvider getMutableConfProvider()

Description copied from interface: MutableConfScheduler

Get scheduler's configuration provider, so other classes can directly call mutation APIs on configuration provider.

Specified by:

getMutableConfProvider in interface MutableConfScheduler

Returns:

scheduler's configuration provider

getCsConfProvider

public CSConfigurationProvider getCsConfProvider()

resetSchedulerMetrics

public void resetSchedulerMetrics()

Description copied from interface: ResourceScheduler

Reset scheduler metrics.

Specified by:

resetSchedulerMetrics in interface ResourceScheduler

Overrides:

resetSchedulerMetrics in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

isMultiNodePlacementEnabled

public boolean isMultiNodePlacementEnabled()

getNumAsyncSchedulerThreads

public int getNumAsyncSchedulerThreads()

setMaxRunningAppsEnforcer

@VisibleForTesting
public void setMaxRunningAppsEnforcer(CSMaxRunningAppsEnforcer enforcer)

placementConstraintEnabled

public boolean placementConstraintEnabled()

Returning true as capacity scheduler supports placement constraints.

Overrides:

placementConstraintEnabled in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Returns:

enabled or not

setQueueManager

@VisibleForTesting
public void setQueueManager(CapacitySchedulerQueueManager qm)

getAsyncSchedulerThreads

@VisibleForTesting
public List<org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.AsyncScheduleThread> getAsyncSchedulerThreads()

getPendingApplicationComparator

public org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.PendingApplicationComparator getPendingApplicationComparator()

Specified by:

getPendingApplicationComparator in interface CapacitySchedulerContext