public class HadoopCatalog extends BaseMetastoreCatalog implements java.io.Closeable, SupportsNamespaces, org.apache.hadoop.conf.Configurable
The HadoopCatalog now supports Catalog.createTable(org.apache.iceberg.catalog.TableIdentifier, org.apache.iceberg.Schema, org.apache.iceberg.PartitionSpec, java.lang.String, java.util.Map<java.lang.String, java.lang.String>)
, Catalog.dropTable(org.apache.iceberg.catalog.TableIdentifier)
, the Catalog.renameTable(org.apache.iceberg.catalog.TableIdentifier, org.apache.iceberg.catalog.TableIdentifier)
is not supported yet.
Note: The HadoopCatalog requires that the underlying file system supports atomic rename.
BaseMetastoreCatalog.BaseMetastoreCatalogTableBuilder
Catalog.TableBuilder
Constructor and Description |
---|
HadoopCatalog() |
HadoopCatalog(org.apache.hadoop.conf.Configuration conf,
java.lang.String warehouseLocation)
The constructor of the HadoopCatalog.
|
Modifier and Type | Method and Description |
---|---|
Catalog.TableBuilder |
buildTable(TableIdentifier identifier,
Schema schema)
/** Instantiate a builder to either create a table or start a create/replace transaction.
|
void |
close() |
void |
createNamespace(Namespace namespace,
java.util.Map<java.lang.String,java.lang.String> meta)
Create a namespace in the catalog.
|
protected java.lang.String |
defaultWarehouseLocation(TableIdentifier tableIdentifier) |
boolean |
dropNamespace(Namespace namespace)
Drop a namespace.
|
boolean |
dropTable(TableIdentifier identifier,
boolean purge)
Drop a table; optionally delete data and metadata files.
|
org.apache.hadoop.conf.Configuration |
getConf() |
void |
initialize(java.lang.String name,
java.util.Map<java.lang.String,java.lang.String> properties)
Initialize a catalog given a custom name and a map of catalog properties.
|
protected boolean |
isValidIdentifier(TableIdentifier identifier) |
java.util.List<Namespace> |
listNamespaces(Namespace namespace)
List namespaces from the namespace.
|
java.util.List<TableIdentifier> |
listTables(Namespace namespace)
Return all the identifiers under this namespace.
|
java.util.Map<java.lang.String,java.lang.String> |
loadNamespaceMetadata(Namespace namespace)
Load metadata properties for a namespace.
|
java.lang.String |
name()
Return the name for this catalog.
|
protected TableOperations |
newTableOps(TableIdentifier identifier) |
protected java.util.Map<java.lang.String,java.lang.String> |
properties() |
boolean |
removeProperties(Namespace namespace,
java.util.Set<java.lang.String> properties)
Remove a set of property keys from a namespace in the catalog.
|
void |
renameTable(TableIdentifier from,
TableIdentifier to)
Rename a table.
|
void |
setConf(org.apache.hadoop.conf.Configuration conf) |
boolean |
setProperties(Namespace namespace,
java.util.Map<java.lang.String,java.lang.String> properties)
Set a collection of properties on a namespace in the catalog.
|
java.lang.String |
toString() |
fullTableName, loadTable, registerTable
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
createNamespace, listNamespaces, namespaceExists
createTable, createTable, createTable, createTable, dropTable, invalidateTable, newCreateTableTransaction, newCreateTableTransaction, newCreateTableTransaction, newCreateTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, tableExists
public HadoopCatalog()
public HadoopCatalog(org.apache.hadoop.conf.Configuration conf, java.lang.String warehouseLocation)
conf
- The Hadoop configurationwarehouseLocation
- The location used as warehouse directorypublic void initialize(java.lang.String name, java.util.Map<java.lang.String,java.lang.String> properties)
Catalog
A custom Catalog implementation must have a no-arg constructor. A compute engine like Spark or Flink will first initialize the catalog without any arguments, and then call this method to complete catalog initialization with properties passed into the engine.
initialize
in interface Catalog
name
- a custom name for the catalogproperties
- catalog propertiespublic java.lang.String name()
Catalog
public java.util.List<TableIdentifier> listTables(Namespace namespace)
Catalog
listTables
in interface Catalog
namespace
- a namespaceprotected boolean isValidIdentifier(TableIdentifier identifier)
isValidIdentifier
in class BaseMetastoreCatalog
protected TableOperations newTableOps(TableIdentifier identifier)
newTableOps
in class BaseMetastoreCatalog
protected java.lang.String defaultWarehouseLocation(TableIdentifier tableIdentifier)
defaultWarehouseLocation
in class BaseMetastoreCatalog
public boolean dropTable(TableIdentifier identifier, boolean purge)
Catalog
If purge is set to true the implementation should delete all data and metadata files.
public void renameTable(TableIdentifier from, TableIdentifier to)
Catalog
renameTable
in interface Catalog
from
- identifier of the table to renameto
- new table namepublic void createNamespace(Namespace namespace, java.util.Map<java.lang.String,java.lang.String> meta)
SupportsNamespaces
createNamespace
in interface SupportsNamespaces
namespace
- a multi-part namespacemeta
- a string Map of properties for the given namespacepublic java.util.List<Namespace> listNamespaces(Namespace namespace)
SupportsNamespaces
For example, if table a.b.t exists, use 'SELECT NAMESPACE IN a' this method must return
Namepace.of("a","b") Namespace
.
listNamespaces
in interface SupportsNamespaces
Namespace
namespublic boolean dropNamespace(Namespace namespace)
SupportsNamespaces
dropNamespace
in interface SupportsNamespaces
namespace
- a namespace. Namespace
public boolean setProperties(Namespace namespace, java.util.Map<java.lang.String,java.lang.String> properties)
SupportsNamespaces
Properties that are not in the given map are not modified or removed by this method.
setProperties
in interface SupportsNamespaces
namespace
- a namespace. Namespace
properties
- a collection of metadata to apply to the namespacepublic boolean removeProperties(Namespace namespace, java.util.Set<java.lang.String> properties)
SupportsNamespaces
Properties that are not in the given set are not modified or removed by this method.
removeProperties
in interface SupportsNamespaces
namespace
- a namespace. Namespace
properties
- a collection of metadata to apply to the namespacepublic java.util.Map<java.lang.String,java.lang.String> loadNamespaceMetadata(Namespace namespace)
SupportsNamespaces
loadNamespaceMetadata
in interface SupportsNamespaces
namespace
- a namespace. Namespace
public void close() throws java.io.IOException
close
in interface java.io.Closeable
close
in interface java.lang.AutoCloseable
java.io.IOException
public java.lang.String toString()
toString
in class BaseMetastoreCatalog
public Catalog.TableBuilder buildTable(TableIdentifier identifier, Schema schema)
Catalog
buildTable
in interface Catalog
buildTable
in class BaseMetastoreCatalog
identifier
- a table identifierschema
- a schemapublic void setConf(org.apache.hadoop.conf.Configuration conf)
setConf
in interface org.apache.hadoop.conf.Configurable
public org.apache.hadoop.conf.Configuration getConf()
getConf
in interface org.apache.hadoop.conf.Configurable
protected java.util.Map<java.lang.String,java.lang.String> properties()
properties
in class BaseMetastoreCatalog