Package org.apache.hadoop.hdfs.server.federation.router.async

Class RouterAsyncRpcClient

java.lang.Object
org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient
org.apache.hadoop.hdfs.server.federation.router.async.RouterAsyncRpcClient
public class RouterAsyncRpcClient extends RouterRpcClient
The RouterAsyncRpcClient class extends the functionality of the base RouterRpcClient class to provide asynchronous remote procedure call (RPC) capabilities for communication with the Hadoop Distributed File System (HDFS) NameNodes in a federated environment.

This class is responsible for managing the asynchronous execution of RPCs to multiple NameNodes, which can improve performance and scalability in large HDFS deployments.

The class also includes methods for handling failover scenarios, where it can automatically retry operations on alternative NameNodes if the primary NameNode is unavailable or in standby mode.

See Also:

  • Constructor Details

    • RouterAsyncRpcClient

      public RouterAsyncRpcClient(org.apache.hadoop.conf.Configuration conf, Router router, ActiveNamenodeResolver resolver, RouterRpcMonitor monitor, RouterStateIdContext routerStateIdContext)
      Create a router async RPC client to manage remote procedure calls to NNs.
      Parameters:
      conf - Hdfs Configuration.
      router - A router using this RPC client.
      resolver - A NN resolver to determine the currently active NN in HA.
      monitor - Optional performance monitor.
      routerStateIdContext - the router state context object to hold the state ids for all namespaces.

  • Method Details

    • initConcurrentCallExecutorService

      protected void initConcurrentCallExecutorService(org.apache.hadoop.conf.Configuration conf)
      Overrides:
      initConcurrentCallExecutorService in class RouterRpcClient

    • invokeAll

      public <T extends RemoteLocationContext> boolean invokeAll(Collection<T> locations, RemoteMethod method) throws IOException
      Invoke method in all locations and return success if any succeeds.
      Overrides:
      invokeAll in class RouterRpcClient
      Type Parameters:
      T - The type of the remote location.
      Parameters:
      locations - List of remote locations to call concurrently.
      method - The remote method and parameters to invoke.
      Returns:
      If the call succeeds in any location.
      Throws:
      IOException - If any of the calls return an exception.

    • invokeMethod

      public Object invokeMethod(org.apache.hadoop.security.UserGroupInformation ugi, List<? extends FederationNamenodeContext> namenodes, boolean useObserver, Class<?> protocol, Method method, Object... params) throws IOException
      Invokes a method against the ClientProtocol proxy server. If a standby exception is generated by the call to the client, retries using the alternate server.

      Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

      Overrides:
      invokeMethod in class RouterRpcClient
      Parameters:
      ugi - User group information.
      namenodes - A prioritized list of namenodes within the same nameservice.
      useObserver - Whether to use observer namenodes.
      protocol - the protocol of the connection.
      method - Remote ClientProtocol method to invoke.
      params - Variable list of parameters matching the method.
      Returns:
      The result of invoking the method.
      Throws:
      ConnectException - If it cannot connect to any Namenode.
      org.apache.hadoop.ipc.StandbyException - If all Namenodes are in Standby.
      IOException - If it cannot invoke the method.

    • invoke

      protected Object invoke(FederationNamenodeContext namenode, Boolean listObserverFirst, int retryCount, Method method, Object obj, Object... params) throws IOException
      Asynchronously invokes a method on a specified NameNode in the context of the given namespace and NameNode information. This method is designed to handle the invocation in a non-blocking manner, allowing for improved performance and scalability when interacting with the NameNode.
      Overrides:
      invoke in class RouterRpcClient
      Parameters:
      namenode - The context information for the NameNode.
      listObserverFirst - Whether to list the observer node first in the invocation list.
      retryCount - The current retry count for the operation.
      method - The method to be invoked on the NameNode.
      obj - The proxy object through which the method will be invoked.
      params - The parameters for the method invocation.
      Returns:
      Response from the remote server
      Throws:
      IOException - If error occurs.

    • invokeSequential

      public <T> T invokeSequential(List<? extends RemoteLocationContext> locations, RemoteMethod remoteMethod, Class<T> expectedResultClass, Object expectedResultValue) throws IOException
      Invokes sequential proxy calls to different locations. Continues to invoke calls until the success condition is met, or until all locations have been attempted. The success condition may be specified by:
      • An expected result class
      • An expected result value
      If no expected result class/values are specified, the success condition is a call that does not throw a remote exception.
      Overrides:
      invokeSequential in class RouterRpcClient
      Type Parameters:
      T - The type of the remote method return.
      Parameters:
      locations - List of locations/nameservices to call concurrently.
      remoteMethod - The remote method and parameters to invoke.
      expectedResultClass - In order to be considered a positive result, the return type must be of this class.
      expectedResultValue - In order to be considered a positive result, the return value must equal the value of this object.
      Returns:
      The result of the first successful call, or if no calls are successful, the result of the first RPC call executed.
      Throws:
      IOException - if the success condition is not met, return the first remote exception generated.

    • invokeSequential

      public <R extends RemoteLocationContext, T> RemoteResult invokeSequential(RemoteMethod remoteMethod, List<R> locations, Class<T> expectedResultClass, Object expectedResultValue) throws IOException
      Invokes sequential proxy calls to different locations. Continues to invoke calls until the success condition is met, or until all locations have been attempted. The success condition may be specified by:
      • An expected result class
      • An expected result value
      If no expected result class/values are specified, the success condition is a call that does not throw a remote exception. This returns RemoteResult, which contains the invoked location as well as the result.
      Overrides:
      invokeSequential in class RouterRpcClient
      Type Parameters:
      R - The type of the remote location.
      T - The type of the remote method return.
      Parameters:
      remoteMethod - The remote method and parameters to invoke.
      locations - List of locations/nameservices to call concurrently.
      expectedResultClass - In order to be considered a positive result, the return type must be of this class.
      expectedResultValue - In order to be considered a positive result, the return value must equal the value of this object.
      Returns:
      The result of the first successful call, or if no calls are successful, the result of the first RPC call executed, along with the invoked location in form of RemoteResult.
      Throws:
      IOException - if the success condition is not met, return the first remote exception generated.

    • invokeConcurrent

      public <T extends RemoteLocationContext, R> Map<T,R> invokeConcurrent(Collection<T> locations, RemoteMethod method, boolean requireResponse, boolean standby, long timeOutMs, Class<R> clazz) throws IOException
      Invokes multiple concurrent proxy calls to different clients. Returns an array of results.

      Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

      Overrides:
      invokeConcurrent in class RouterRpcClient
      Type Parameters:
      T - The type of the remote location.
      R - The type of the remote method return.
      Parameters:
      locations - List of remote locations to call concurrently.
      method - The remote method and parameters to invoke.
      requireResponse - If true an exception will be thrown if all calls do not complete. If false exceptions are ignored and all data results successfully received are returned.
      standby - If the requests should go to the standby namenodes too.
      timeOutMs - Timeout for each individual call.
      clazz - Type of the remote return type.
      Returns:
      Result of invoking the method per subcluster: nsId to result.
      Throws:
      IOException - If requiredResponse=true and any of the calls throw an exception.

    • getRemoteResults

      protected <T extends RemoteLocationContext, R> List<RemoteResult<T,R>> getRemoteResults(RemoteMethod method, long timeOutMs, RouterRpcFairnessPolicyController controller, List<T> orderedLocations, List<Callable<Object>> callables) throws IOException
      Invokes multiple concurrent proxy calls to different clients. Returns an array of results.
      Overrides:
      getRemoteResults in class RouterRpcClient
      Type Parameters:
      T - The type of the remote location.
      R - The type of the remote method return.
      Parameters:
      method - The remote method and parameters to invoke.
      timeOutMs - Timeout for each individual call.
      controller - Fairness manager to control handlers assigned per NS.
      orderedLocations - List of remote locations to call concurrently.
      callables - Invoke method for each NameNode.
      Returns:
      Result of invoking the method per subcluster (list of results), This includes the exception for each remote location.
      Throws:
      IOException - If there are errors invoking the method.

    • invokeSingle

      public <T extends RemoteLocationContext, R> List<RemoteResult<T,R>> invokeSingle(T location, RemoteMethod method) throws IOException
      Invokes a ClientProtocol method against the specified namespace.

      Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

      Overrides:
      invokeSingle in class RouterRpcClient
      Type Parameters:
      T - The type of the remote location.
      R - The type of the remote method return.
      Parameters:
      location - RemoteLocation to invoke.
      method - The remote method and parameters to invoke.
      Returns:
      Result of invoking the method per subcluster (list of results), This includes the exception for each remote location.
      Throws:
      IOException - If there are errors invoking the method.

    • invokeSingle

      public Object invokeSingle(String nsId, RemoteMethod method) throws IOException
      Invokes a ClientProtocol method against the specified namespace.

      Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

      Overrides:
      invokeSingle in class RouterRpcClient
      Parameters:
      nsId - Target namespace for the method.
      method - The remote method and parameters to invoke.
      Returns:
      The result of invoking the method.
      Throws:
      IOException - If the invoke generated an error.

    • invokeSingle

      public <T> T invokeSingle(RemoteLocationContext location, RemoteMethod remoteMethod, Class<T> clazz) throws IOException
      Invokes a single proxy call for a single location.

      Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

      Overrides:
      invokeSingle in class RouterRpcClient
      Type Parameters:
      T - The type of the remote method return.
      Parameters:
      location - RemoteLocation to invoke.
      remoteMethod - The remote method and parameters to invoke.
      clazz - Class for the return type.
      Returns:
      The result of invoking the method if successful.
      Throws:
      IOException - If the invoke generated an error.

    • releasePermit

      protected void releasePermit(String nsId, org.apache.hadoop.security.UserGroupInformation ugi, Method m, RouterRpcFairnessPolicyController controller)
      Release permit for specific nsId after processing against downstream nsId is completed.
      Parameters:
      nsId - Identifier of the block pool.
      ugi - UserGroupIdentifier associated with the user.
      m - Remote method that needs to be invoked.
      controller - fairness policy controller to release permit from