Package org.apache.hadoop.log

Class LogThrottlingHelper

java.lang.Object
org.apache.hadoop.log.LogThrottlingHelper
public class LogThrottlingHelper extends Object
This is a class to help easily throttle log statements, so that they will not be emitted more frequently than a certain rate. It is useful to help prevent flooding the application logs with redundant messages. The instantiator specifies a minimum period at which statements should be logged. When record(double...) is called, if enough time has elapsed since the last time it was called, the return value will indicate to the caller that it should write to its actual log. Note that this class does not write to any actual log; it only records information about how many times record has been called and with what arguments, and indicates to the caller whether or not it should write to its log. If not enough time has yet elapsed, this class records the arguments and updates its summary information, and indicates to the caller that it should not log. For example, say that you want to know whenever too large of a request is received, but want to avoid flooding the logs if many such requests are received. 

   // Helper with a minimum period of 5 seconds
   private LogThrottlingHelper helper = new LogThrottlingHelper(5000);

   public void receiveRequest(int requestedObjects) {
     if (requestedObjects > MAXIMUM_REQUEST_SIZE) {
       LogAction logAction = helper.record(requestedObjects);
       if (logAction.shouldLog()) {
         LOG.warn("Received {} large request(s) with a total of {} objects " +
             "requested; maximum objects requested was {}",
             logAction.getCount(), logAction.getStats(0).getSum(),
             logAction.getStats(0).getMax());
       }
     }
   }
The above snippet allows you to record extraneous events, but if they become frequent, to limit their presence in the log to only every 5 seconds while still maintaining overall information about how many large requests were received.

This class can also be used to coordinate multiple logging points; see record(String, long, double...) for more details.

This class is thread-safe.

  • Field Details

  • Constructor Details

    • LogThrottlingHelper

      public LogThrottlingHelper(long minLogPeriodMs)
      Create a log helper without any primary recorder.
      Parameters:
      minLogPeriodMs - input minLogPeriodMs.
      See Also:

    • LogThrottlingHelper

      public LogThrottlingHelper(long minLogPeriodMs, String primaryRecorderName)
      Create a log helper with a specified primary recorder name; this can be used in conjunction with record(String, long, double...) to set up primary and dependent recorders. See record(String, long, double...) for more details.
      Parameters:
      minLogPeriodMs - The minimum period with which to log; do not log more frequently than this.
      primaryRecorderName - The name of the primary recorder.

  • Method Details

    • record

      public LogThrottlingHelper.LogAction record(double... values)
      Record some set of values at the current time into this helper. Note that this does not actually write information to any log. Instead, this will return a LogAction indicating whether or not the caller should write to its own log. The LogAction will additionally contain summary information about the values specified since the last time the caller was expected to write to its log.

      Specifying multiple values will maintain separate summary statistics about each value. For example: 

      
   helper.record(1, 0);
   LogAction action = helper.record(3, 100);
   action.getStats(0); // == 2
   action.getStats(1); // == 50
      Parameters:
      values - The values about which to maintain summary information. Every time this method is called, the same number of values must be specified.
      Returns:
      A LogAction indicating whether or not the caller should write to its log.

    • record

      public LogThrottlingHelper.LogAction record(String recorderName, long currentTimeMs, double... values)
      Record some set of values at the specified time into this helper. This can be useful to avoid fetching the current time twice if the caller has already done so for other purposes. This additionally allows the caller to specify a name for this recorder. When multiple names are used, one is denoted as the primary recorder. Only recorders named as the primary will trigger logging; other names not matching the primary can only be triggered by following the primary. This is used to coordinate multiple logging points. A primary can be set via the LogThrottlingHelper(long, String) constructor. If no primary is set in the constructor, then the first recorder name used becomes the primary. If multiple names are used, they maintain entirely different sets of values and summary information. For example: 
      
   // Initialize "pre" as the primary recorder name
   LogThrottlingHelper helper = new LogThrottlingHelper(1000, "pre");
   LogAction preLog = helper.record("pre", Time.monotonicNow());
   if (preLog.shouldLog()) {
     // ...
   }
   double eventsProcessed = ... // perform some action
   LogAction postLog =
       helper.record("post", Time.monotonicNow(), eventsProcessed);
   if (postLog.shouldLog()) {
     // ...
     // Can use postLog.getStats(0) to access eventsProcessed information
   }
      Since "pre" is the primary recorder name, logging to "pre" will trigger a log action if enough time has elapsed. This will indicate that "post" should log as well. This ensures that "post" is always logged in the same iteration as "pre", yet each one is able to maintain its own summary information.

      Other behavior is the same as record(double...).

      Parameters:
      recorderName - The name of the recorder. This is used to check if the current recorder is the primary. Other names are arbitrary and are only used to differentiate between distinct recorders.
      currentTimeMs - The current time.
      values - The values to log.
      Returns:
      The LogAction for the specified recorder.
      See Also:

    • getCurrentStats

      public org.apache.commons.math3.stat.descriptive.SummaryStatistics getCurrentStats(String recorderName, int idx)
      Return the summary information for given index.
      Parameters:
      recorderName - The name of the recorder.
      idx - The index value.
      Returns:
      The summary information.

    • getLogSupressionMessage

      public static String getLogSupressionMessage(LogThrottlingHelper.LogAction action)
      Helper function to create a message about how many log statements were suppressed in the provided log action. If no statements were suppressed, this returns an empty string. The message has the format (without quotes):

      ' (suppressed logging {suppression_count} times)'

      Parameters:
      action - The log action to produce a message about.
      Returns:
      A message about suppression within this action.

    • reset

      @VisibleForTesting public void reset()