public class S3FileIO extends java.lang.Object implements FileIO, SupportsBulkOperations, SupportsPrefixOperations, CredentialSupplier
Locations used must follow the conventions for S3 URIs (e.g. s3://bucket/path...). URIs with
schemes s3a, s3n, https are also treated as s3 file paths. Using this FileIO with other schemes
will result in ValidationException
.
Constructor and Description |
---|
S3FileIO()
No-arg constructor to load the FileIO dynamically.
|
S3FileIO(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3)
Constructor with custom s3 supplier and S3FileIO properties.
|
S3FileIO(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3,
S3FileIOProperties s3FileIOProperties)
Constructor with custom s3 supplier and S3FileIO properties.
|
Modifier and Type | Method and Description |
---|---|
void |
close()
Close File IO to release underlying resources.
|
void |
deleteFile(java.lang.String path)
Delete the file at the given path.
|
void |
deleteFiles(java.lang.Iterable<java.lang.String> paths)
Deletes the given paths in a batched manner.
|
void |
deletePrefix(java.lang.String prefix)
This method provides a "best-effort" to delete all objects under the given prefix.
|
protected void |
finalize() |
java.lang.String |
getCredential()
Returns the credential string
|
void |
initialize(java.util.Map<java.lang.String,java.lang.String> props)
Initialize File IO from catalog properties.
|
java.lang.Iterable<FileInfo> |
listPrefix(java.lang.String prefix)
Return an iterable of all files under a prefix.
|
InputFile |
newInputFile(java.lang.String path)
Get a
InputFile instance to read bytes from the file at the given path. |
InputFile |
newInputFile(java.lang.String path,
long length)
Get a
InputFile instance to read bytes from the file at the given path, with a known
file length. |
OutputFile |
newOutputFile(java.lang.String path)
Get a
OutputFile instance to write bytes to the file at the given path. |
java.util.Map<java.lang.String,java.lang.String> |
properties()
Returns the property map used to configure this FileIO
|
clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
deleteFile, deleteFile
public S3FileIO()
All fields are initialized by calling initialize(Map)
later.
public S3FileIO(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3)
Calling initialize(Map)
will overwrite information set in this constructor.
s3
- s3 supplierpublic S3FileIO(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3, S3FileIOProperties s3FileIOProperties)
Calling initialize(Map)
will overwrite information set in this constructor.
s3
- s3 suppliers3FileIOProperties
- S3 FileIO propertiespublic InputFile newInputFile(java.lang.String path)
FileIO
InputFile
instance to read bytes from the file at the given path.newInputFile
in interface FileIO
public InputFile newInputFile(java.lang.String path, long length)
FileIO
InputFile
instance to read bytes from the file at the given path, with a known
file length.newInputFile
in interface FileIO
public OutputFile newOutputFile(java.lang.String path)
FileIO
OutputFile
instance to write bytes to the file at the given path.newOutputFile
in interface FileIO
public void deleteFile(java.lang.String path)
FileIO
deleteFile
in interface FileIO
public java.util.Map<java.lang.String,java.lang.String> properties()
FileIO
properties
in interface FileIO
public void deleteFiles(java.lang.Iterable<java.lang.String> paths) throws BulkDeletionFailureException
The paths are grouped by bucket, and deletion is triggered when we either reach the configured batch size or have a final remainder batch for each bucket.
deleteFiles
in interface SupportsBulkOperations
paths
- paths to deleteBulkDeletionFailureException
- in case of failure to delete at least 1 filepublic java.lang.Iterable<FileInfo> listPrefix(java.lang.String prefix)
SupportsPrefixOperations
Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.
listPrefix
in interface SupportsPrefixOperations
prefix
- prefix to listpublic void deletePrefix(java.lang.String prefix)
Bulk delete operations are used and no reattempt is made for deletes if they fail, but will log any individual objects that are not deleted as part of the bulk operation.
deletePrefix
in interface SupportsPrefixOperations
prefix
- prefix to deletepublic java.lang.String getCredential()
CredentialSupplier
getCredential
in interface CredentialSupplier
public void initialize(java.util.Map<java.lang.String,java.lang.String> props)
FileIO
initialize
in interface FileIO
props
- catalog propertiespublic void close()
FileIO
Calling this method is only required when this FileIO instance is no longer expected to be used, and the resources it holds need to be explicitly released to avoid resource leaks.
protected void finalize() throws java.lang.Throwable
finalize
in class java.lang.Object
java.lang.Throwable