public class HadoopFileIO extends java.lang.Object implements FileIO, HadoopConfigurable, SupportsPrefixOperations, SupportsBulkOperations
Constructor and Description |
---|
HadoopFileIO()
Constructor used for dynamic FileIO loading.
|
HadoopFileIO(org.apache.hadoop.conf.Configuration hadoopConf) |
HadoopFileIO(SerializableSupplier<org.apache.hadoop.conf.Configuration> hadoopConf) |
Modifier and Type | Method and Description |
---|---|
org.apache.hadoop.conf.Configuration |
conf() |
void |
deleteFile(java.lang.String path)
Delete the file at the given path.
|
void |
deleteFiles(java.lang.Iterable<java.lang.String> pathsToDelete)
Delete the files at the given paths.
|
void |
deletePrefix(java.lang.String prefix)
Delete all files under a prefix.
|
org.apache.hadoop.conf.Configuration |
getConf() |
void |
initialize(java.util.Map<java.lang.String,java.lang.String> props)
Initialize File IO from catalog properties.
|
java.lang.Iterable<FileInfo> |
listPrefix(java.lang.String prefix)
Return an iterable of all files under a prefix.
|
InputFile |
newInputFile(java.lang.String path)
Get a
InputFile instance to read bytes from the file at the given path. |
InputFile |
newInputFile(java.lang.String path,
long length)
Get a
InputFile instance to read bytes from the file at the given path, with a known
file length. |
OutputFile |
newOutputFile(java.lang.String path)
Get a
OutputFile instance to write bytes to the file at the given path. |
java.util.Map<java.lang.String,java.lang.String> |
properties()
Returns the property map used to configure this FileIO
|
void |
serializeConfWith(java.util.function.Function<org.apache.hadoop.conf.Configuration,SerializableSupplier<org.apache.hadoop.conf.Configuration>> confSerializer)
Take a function that serializes Hadoop configuration into a supplier.
|
void |
setConf(org.apache.hadoop.conf.Configuration conf) |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
close, deleteFile, deleteFile
public HadoopFileIO()
Hadoop configuration
must be set through setConf(Configuration)
public HadoopFileIO(org.apache.hadoop.conf.Configuration hadoopConf)
public HadoopFileIO(SerializableSupplier<org.apache.hadoop.conf.Configuration> hadoopConf)
public org.apache.hadoop.conf.Configuration conf()
public void initialize(java.util.Map<java.lang.String,java.lang.String> props)
FileIO
initialize
in interface FileIO
props
- catalog propertiespublic InputFile newInputFile(java.lang.String path)
FileIO
InputFile
instance to read bytes from the file at the given path.newInputFile
in interface FileIO
public InputFile newInputFile(java.lang.String path, long length)
FileIO
InputFile
instance to read bytes from the file at the given path, with a known
file length.newInputFile
in interface FileIO
public OutputFile newOutputFile(java.lang.String path)
FileIO
OutputFile
instance to write bytes to the file at the given path.newOutputFile
in interface FileIO
public void deleteFile(java.lang.String path)
FileIO
deleteFile
in interface FileIO
public java.util.Map<java.lang.String,java.lang.String> properties()
FileIO
properties
in interface FileIO
public void setConf(org.apache.hadoop.conf.Configuration conf)
setConf
in interface org.apache.hadoop.conf.Configurable
public org.apache.hadoop.conf.Configuration getConf()
getConf
in interface org.apache.hadoop.conf.Configurable
public void serializeConfWith(java.util.function.Function<org.apache.hadoop.conf.Configuration,SerializableSupplier<org.apache.hadoop.conf.Configuration>> confSerializer)
HadoopConfigurable
serializeConfWith
in interface HadoopConfigurable
confSerializer
- A function that takes Hadoop configuration and returns a serializable
supplier of it.public java.lang.Iterable<FileInfo> listPrefix(java.lang.String prefix)
SupportsPrefixOperations
Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.
listPrefix
in interface SupportsPrefixOperations
prefix
- prefix to listpublic void deletePrefix(java.lang.String prefix)
SupportsPrefixOperations
Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.
deletePrefix
in interface SupportsPrefixOperations
prefix
- prefix to deletepublic void deleteFiles(java.lang.Iterable<java.lang.String> pathsToDelete) throws BulkDeletionFailureException
SupportsBulkOperations
deleteFiles
in interface SupportsBulkOperations
pathsToDelete
- The paths to deleteBulkDeletionFailureException
- in case of failure to delete at least 1 file