org.apache.iceberg.spark

Class SparkUtil

java.lang.Object

org.apache.iceberg.spark.SparkUtil

public class SparkUtil
extends java.lang.Object

Method Summary

Modifier and Type	Method and Description
static boolean	caseSensitive(org.apache.spark.sql.SparkSession spark)
static <C,T> Pair<C,T>	catalogAndIdentifier(java.util.List<java.lang.String> nameParts, java.util.function.Function<java.lang.String,C> catalogProvider, java.util.function.BiFunction<java.lang.String[],java.lang.String,T> identiferProvider, C currentCatalog, java.lang.String[] currentNamespace) A modified version of Spark's LookupCatalog.CatalogAndIdentifier.unapply Attempts to find the catalog and identifier a multipart identifier represents
static org.apache.hadoop.conf.Configuration	hadoopConfCatalogOverrides(org.apache.spark.sql.SparkSession spark, java.lang.String catalogName) Pulls any Catalog specific overrides for the Hadoop conf from the current SparkSession, which can be set via `spark.sql.catalog.$catalogName.hadoop.*`
static java.util.List<org.apache.spark.sql.catalyst.expressions.Expression>	partitionMapToExpression(org.apache.spark.sql.types.StructType schema, java.util.Map<java.lang.String,java.lang.String> filters) Get a List of Spark filter Expression.
static java.lang.String	toColumnName(org.apache.spark.sql.connector.expressions.NamedReference ref)
static void	validatePartitionTransforms(PartitionSpec spec) Check whether the partition transforms in a spec can be used to write data.
static void	validateTimestampWithoutTimezoneConfig(org.apache.spark.sql.RuntimeConfig conf)
static void	validateTimestampWithoutTimezoneConfig(org.apache.spark.sql.RuntimeConfig conf, java.util.Map<java.lang.String,java.lang.String> options) Checks for properties both supplied by Spark's RuntimeConfig and the read or write options

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

validatePartitionTransforms

public static void validatePartitionTransforms(PartitionSpec spec)

Check whether the partition transforms in a spec can be used to write data.

Parameters:

spec - a PartitionSpec

Throws:

java.lang.UnsupportedOperationException - if the spec contains unknown partition transforms

catalogAndIdentifier

public static <C,T> Pair<C,T> catalogAndIdentifier(java.util.List<java.lang.String> nameParts,
                                                   java.util.function.Function<java.lang.String,C> catalogProvider,
                                                   java.util.function.BiFunction<java.lang.String[],java.lang.String,T> identiferProvider,
                                                   C currentCatalog,
                                                   java.lang.String[] currentNamespace)

A modified version of Spark's LookupCatalog.CatalogAndIdentifier.unapply Attempts to find the catalog and identifier a multipart identifier represents

Parameters:

nameParts - Multipart identifier representing a table

Returns:

The CatalogPlugin and Identifier for the table

hadoopConfCatalogOverrides

public static org.apache.hadoop.conf.Configuration hadoopConfCatalogOverrides(org.apache.spark.sql.SparkSession spark,
                                                                              java.lang.String catalogName)

Pulls any Catalog specific overrides for the Hadoop conf from the current SparkSession, which can be set via `spark.sql.catalog.$catalogName.hadoop.*`

Mirrors the override of hadoop configurations for a given spark session using `spark.hadoop.*`.

The SparkCatalog allows for hadoop configurations to be overridden per catalog, by setting them on the SQLConf, where the following will add the property "fs.default.name" with value "hdfs://hanksnamenode:8020" to the catalog's hadoop configuration. SparkSession.builder() .config(s"spark.sql.catalog.$catalogName.hadoop.fs.default.name", "hdfs://hanksnamenode:8020") .getOrCreate()

Parameters:

spark - The current Spark session

catalogName - Name of the catalog to find overrides for.

Returns:

the Hadoop Configuration that should be used for this catalog, with catalog specific overrides applied.

validateTimestampWithoutTimezoneConfig

public static void validateTimestampWithoutTimezoneConfig(org.apache.spark.sql.RuntimeConfig conf)

validateTimestampWithoutTimezoneConfig

public static void validateTimestampWithoutTimezoneConfig(org.apache.spark.sql.RuntimeConfig conf,
                                                          java.util.Map<java.lang.String,java.lang.String> options)

Checks for properties both supplied by Spark's RuntimeConfig and the read or write options

Parameters:

conf - The RuntimeConfig of the active Spark session

options - The read or write options supplied when reading/writing a table

partitionMapToExpression

public static java.util.List<org.apache.spark.sql.catalyst.expressions.Expression> partitionMapToExpression(org.apache.spark.sql.types.StructType schema,
                                                                                                            java.util.Map<java.lang.String,java.lang.String> filters)

Get a List of Spark filter Expression.

Parameters:

schema - table schema

filters - filters in the format of a Map, where key is one of the table column name, and value is the specific value to be filtered on the column.

Returns:

a List of filters in the format of Spark Expression.

toColumnName

public static java.lang.String toColumnName(org.apache.spark.sql.connector.expressions.NamedReference ref)

caseSensitive

public static boolean caseSensitive(org.apache.spark.sql.SparkSession spark)