Packages

abstract class WriteAheadLog extends AnyRef

:: DeveloperApi ::

This abstract class represents a write ahead log (aka journal) that is used by Spark Streaming to save the received data (by receivers) and associated metadata to a reliable storage, so that they can be recovered after driver failures. See the Spark documentation for more information on how to plug in your own custom implementation of a write ahead log.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. WriteAheadLog
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new WriteAheadLog()

Abstract Value Members

  1. abstract def clean(threshTime: Long, waitForCompletion: Boolean): Unit

    Clean all the records that are older than the threshold time.

    Clean all the records that are older than the threshold time. It can wait for the completion of the deletion.

  2. abstract def close(): Unit

    Close this log and release any resources.

    Close this log and release any resources. It must be idempotent.

  3. abstract def read(handle: WriteAheadLogRecordHandle): ByteBuffer

    Read a written record based on the given record handle.

  4. abstract def readAll(): Iterator[ByteBuffer]

    Read and return an iterator of all the records that have been written but not yet cleaned up.

  5. abstract def write(record: ByteBuffer, time: Long): WriteAheadLogRecordHandle

    Write the record to the log and return a record handle, which contains all the information necessary to read back the written record.

    Write the record to the log and return a record handle, which contains all the information necessary to read back the written record. The time is used to the index the record, such that it can be cleaned later. Note that implementations of this abstract class must ensure that the written data is durable and readable (using the record handle) by the time this function returns.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  16. def toString(): String
    Definition Classes
    AnyRef → Any
  17. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  19. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped