BarrierTaskContext (Spark 2.4.8 JavaDoc)

Object
- org.apache.spark.TaskContext
- - org.apache.spark.BarrierTaskContext

All Implemented Interfaces:

java.io.Serializable, Logging
```
public class BarrierTaskContext
extends TaskContext
implements Logging
```
A TaskContext with extra contextual info and tooling for tasks in a barrier stage. Use get() to obtain the barrier context for a running barrier task.

See Also:

Serialized Form

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`BarrierTaskContext`	`addTaskCompletionListener(TaskCompletionListener listener)` Adds a (Java friendly) listener to be executed on task completion.
`BarrierTaskContext`	`addTaskFailureListener(TaskFailureListener listener)` Adds a listener to be executed on task failure.
`int`	`attemptNumber()` How many times this task has been attempted.
`void`	Experimental`barrier()` Sets a global barrier and waits until all tasks in this stage hit this barrier.
`static BarrierTaskContext`	Experimental`get()` Returns the currently active BarrierTaskContext.
`String`	`getLocalProperty(String key)` Get a local property set upstream in the driver, or null if it is missing.
`scala.collection.Seq<Source>`	`getMetricsSources(String sourceName)` ::DeveloperApi:: Returns all metrics sources with the given name which are associated with the instance which runs the task.
`BarrierTaskInfo[]`	Experimental`getTaskInfos()` Returns `BarrierTaskInfo` for all tasks in this barrier stage, ordered by partition ID.
`boolean`	`isCompleted()` Returns true if the task has completed.
`boolean`	`isInterrupted()` Returns true if the task has been killed.
`boolean`	`isRunningLocally()` Returns true if the task is running locally in the driver program.
`int`	`partitionId()` The ID of the RDD partition that is computed by this task.
`int`	`stageAttemptNumber()` How many times the stage that this task belongs to has been attempted.
`int`	`stageId()` The ID of the stage that this task belong to.
`long`	`taskAttemptId()` An ID that is unique to this task attempt (within the same SparkContext, no two task attempts will share the same attempt ID).
`org.apache.spark.executor.TaskMetrics`	`taskMetrics()`

Methods inherited from class org.apache.spark.TaskContext
addTaskCompletionListener, addTaskFailureListener, getPartitionId

Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging
initializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

- Method Detail
  - Experimental
    get
```
public static BarrierTaskContext get()
```
    Returns the currently active BarrierTaskContext. This can be called inside of user functions to access contextual information about running barrier tasks.
    
    Returns:
    
    (undocumented)
  - Experimental
    barrier
```
public void barrier()
```
    Sets a global barrier and waits until all tasks in this stage hit this barrier. Similar to MPI_Barrier function in MPI, the barrier() function call blocks until all tasks in the same stage have reached this routine.
    CAUTION! In a barrier stage, each task must have the same number of barrier() calls, in all possible code branches. Otherwise, you may get the job hanging or a SparkException after timeout. Some examples of '''misuses''' are listed below: 1. Only call barrier() function on a subset of all the tasks in the same barrier stage, it shall lead to timeout of the function call.
```
   rdd.barrier().mapPartitions { iter =>
       val context = BarrierTaskContext.get()
       if (context.partitionId() == 0) {
           // Do nothing.
       } else {
           context.barrier()
       }
       iter
   }
 
```
    2. Include barrier() function in a try-catch code block, this may lead to timeout of the second function call.
```
   rdd.barrier().mapPartitions { iter =>
       val context = BarrierTaskContext.get()
       try {
           // Do something that might throw an Exception.
           doSomething()
           context.barrier()
       } catch {
           case e: Exception => logWarning("...", e)
       }
       context.barrier()
       iter
   }
 
```
  - Experimental
    getTaskInfos
```
public BarrierTaskInfo[] getTaskInfos()
```
    Returns BarrierTaskInfo for all tasks in this barrier stage, ordered by partition ID.
    
    Returns:
    
    (undocumented)
  - isCompleted
```
public boolean isCompleted()
```
    Description copied from class: TaskContext
    
    Returns true if the task has completed.
    
    Specified by:
    
    isCompleted in class TaskContext
    
    Returns:
    
    (undocumented)
  - isInterrupted
```
public boolean isInterrupted()
```
    Description copied from class: TaskContext
    
    Returns true if the task has been killed.
    
    Specified by:
    
    isInterrupted in class TaskContext
    
    Returns:
    
    (undocumented)
  - isRunningLocally
```
public boolean isRunningLocally()
```
    Description copied from class: TaskContext
    
    Returns true if the task is running locally in the driver program.
    
    Specified by:
    
    isRunningLocally in class TaskContext
    
    Returns:
    
    false
  - addTaskCompletionListener
```
public BarrierTaskContext addTaskCompletionListener(TaskCompletionListener listener)
```
    Description copied from class: TaskContext
    
    Adds a (Java friendly) listener to be executed on task completion. This will be called in all situations - success, failure, or cancellation. Adding a listener to an already completed task will result in that listener being called immediately.
    An example use is for HadoopRDD to register a callback to close the input stream.
    Exceptions thrown by the listener will result in failure of the task.
    
    Specified by:
    
    addTaskCompletionListener in class TaskContext
    
    Parameters:
    
    listener - (undocumented)
    
    Returns:
    
    (undocumented)
  - addTaskFailureListener
```
public BarrierTaskContext addTaskFailureListener(TaskFailureListener listener)
```
    Description copied from class: TaskContext
    
    Adds a listener to be executed on task failure. Adding a listener to an already failed task will result in that listener being called immediately.
    
    Specified by:
    
    addTaskFailureListener in class TaskContext
    
    Parameters:
    
    listener - (undocumented)
    
    Returns:
    
    (undocumented)
  - stageId
```
public int stageId()
```
    Description copied from class: TaskContext
    
    The ID of the stage that this task belong to.
    
    Specified by:
    
    stageId in class TaskContext
    
    Returns:
    
    (undocumented)
  - stageAttemptNumber
```
public int stageAttemptNumber()
```
    Description copied from class: TaskContext
    
    How many times the stage that this task belongs to has been attempted. The first stage attempt will be assigned stageAttemptNumber = 0, and subsequent attempts will have increasing attempt numbers.
    
    Specified by:
    
    stageAttemptNumber in class TaskContext
    
    Returns:
    
    (undocumented)
  - partitionId
```
public int partitionId()
```
    Description copied from class: TaskContext
    
    The ID of the RDD partition that is computed by this task.
    
    Specified by:
    
    partitionId in class TaskContext
    
    Returns:
    
    (undocumented)
  - attemptNumber
```
public int attemptNumber()
```
    Description copied from class: TaskContext
    
    How many times this task has been attempted. The first task attempt will be assigned attemptNumber = 0, and subsequent attempts will have increasing attempt numbers.
    
    Specified by:
    
    attemptNumber in class TaskContext
    
    Returns:
    
    (undocumented)
  - taskAttemptId
```
public long taskAttemptId()
```
    Description copied from class: TaskContext
    
    An ID that is unique to this task attempt (within the same SparkContext, no two task attempts will share the same attempt ID). This is roughly equivalent to Hadoop's TaskAttemptID.
    
    Specified by:
    
    taskAttemptId in class TaskContext
    
    Returns:
    
    (undocumented)
  - getLocalProperty
```
public String getLocalProperty(String key)
```
    Description copied from class: TaskContext
    
    Get a local property set upstream in the driver, or null if it is missing. See also org.apache.spark.SparkContext.setLocalProperty.
    
    Specified by:
    
    getLocalProperty in class TaskContext
    
    Parameters:
    
    key - (undocumented)
    
    Returns:
    
    (undocumented)
  - taskMetrics
```
public org.apache.spark.executor.TaskMetrics taskMetrics()
```
    Specified by:
    
    taskMetrics in class TaskContext
  - getMetricsSources
```
public scala.collection.Seq<Source> getMetricsSources(String sourceName)
```
    Description copied from class: TaskContext
    
    ::DeveloperApi:: Returns all metrics sources with the given name which are associated with the instance which runs the task. For more information see org.apache.spark.metrics.MetricsSystem.
    
    Specified by:
    
    getMetricsSources in class TaskContext
    
    Parameters:
    
    sourceName - (undocumented)
    
    Returns:
    
    (undocumented)

Class BarrierTaskContext

Method Summary

Methods inherited from class org.apache.spark.TaskContext

Methods inherited from class Object

Methods inherited from interface org.apache.spark.internal.Logging

Method Detail

get

barrier

getTaskInfos

isCompleted

isInterrupted

isRunningLocally

addTaskCompletionListener

addTaskFailureListener

stageId

stageAttemptNumber

partitionId

attemptNumber

taskAttemptId

getLocalProperty

taskMetrics

getMetricsSources