org.apache.iceberg.spark

Class SparkReadConf

java.lang.Object

org.apache.iceberg.spark.SparkReadConf

public class SparkReadConf
extends java.lang.Object

A class for common Iceberg configs for Spark reads.

If a config is set at multiple levels, the following order of precedence is used (top to bottom):

1. Read options
2. Session configuration
3. Table metadata

The most specific value is set in read options and takes precedence over all other configs. If no read option is provided, this class checks the session configuration for any overrides. If no applicable value is found in the session configuration, this class uses the table metadata.

Note this class is NOT meant to be serialized and sent to executors.

Constructor Summary

Constructors

Constructor and Description
SparkReadConf(org.apache.spark.sql.SparkSession spark, Table table, java.util.Map<java.lang.String,java.lang.String> readOptions)
SparkReadConf(org.apache.spark.sql.SparkSession spark, Table table, java.lang.String branch, java.util.Map<java.lang.String,java.lang.String> readOptions)

Method Summary

Modifier and Type	Method and Description
boolean	aggregatePushDownEnabled()
java.lang.Long	asOfTimestamp()
java.lang.String	branch()
boolean	caseSensitive()
java.lang.Long	endSnapshotId()
java.lang.Long	endTimestamp()
boolean	localityEnabled()
int	maxFilesPerMicroBatch()
int	maxRecordsPerMicroBatch()
int	orcBatchSize()
boolean	orcVectorizationEnabled()
int	parquetBatchSize()
boolean	parquetVectorizationEnabled()
boolean	preserveDataGrouping()
java.lang.String	scanTaskSetId()
java.lang.Long	snapshotId()
int	splitLookback()
java.lang.Integer	splitLookbackOption()
long	splitOpenFileCost()
java.lang.Long	splitOpenFileCostOption()
long	splitSize()
java.lang.Long	splitSizeOption()
java.lang.Long	startSnapshotId()
java.lang.Long	startTimestamp()
long	streamFromTimestamp()
boolean	streamingSkipDeleteSnapshots()
boolean	streamingSkipOverwriteSnapshots()
java.lang.String	tag()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkReadConf

public SparkReadConf(org.apache.spark.sql.SparkSession spark,
                     Table table,
                     java.util.Map<java.lang.String,java.lang.String> readOptions)

SparkReadConf

public SparkReadConf(org.apache.spark.sql.SparkSession spark,
                     Table table,
                     java.lang.String branch,
                     java.util.Map<java.lang.String,java.lang.String> readOptions)

Method Detail

caseSensitive

public boolean caseSensitive()

localityEnabled

public boolean localityEnabled()

snapshotId

public java.lang.Long snapshotId()

asOfTimestamp

public java.lang.Long asOfTimestamp()

startSnapshotId

public java.lang.Long startSnapshotId()

endSnapshotId

public java.lang.Long endSnapshotId()

branch

public java.lang.String branch()

tag

public java.lang.String tag()

scanTaskSetId

public java.lang.String scanTaskSetId()

streamingSkipDeleteSnapshots

public boolean streamingSkipDeleteSnapshots()

streamingSkipOverwriteSnapshots

public boolean streamingSkipOverwriteSnapshots()

parquetVectorizationEnabled

public boolean parquetVectorizationEnabled()

parquetBatchSize

public int parquetBatchSize()

orcVectorizationEnabled

public boolean orcVectorizationEnabled()

orcBatchSize

public int orcBatchSize()

splitSizeOption

public java.lang.Long splitSizeOption()

splitSize

public long splitSize()

splitLookbackOption

public java.lang.Integer splitLookbackOption()

splitLookback

public int splitLookback()

splitOpenFileCostOption

public java.lang.Long splitOpenFileCostOption()

splitOpenFileCost

public long splitOpenFileCost()

streamFromTimestamp

public long streamFromTimestamp()

startTimestamp

public java.lang.Long startTimestamp()

endTimestamp

public java.lang.Long endTimestamp()

maxFilesPerMicroBatch

public int maxFilesPerMicroBatch()

maxRecordsPerMicroBatch

public int maxRecordsPerMicroBatch()

preserveDataGrouping

public boolean preserveDataGrouping()

aggregatePushDownEnabled

public boolean aggregatePushDownEnabled()