public class BaseRewriteManifests extends java.lang.Object implements RewriteManifests
Modifier and Type | Method and Description |
---|---|
RewriteManifests |
addManifest(ManifestFile manifest)
Adds a
manifest file to the table. |
Snapshot |
apply()
Apply the pending changes and return the uncommitted changes for validation.
|
java.util.List<ManifestFile> |
apply(TableMetadata base,
Snapshot snapshot)
Apply the update's changes to the given metadata and snapshot.
|
protected void |
cleanAll() |
protected void |
cleanUncommitted(java.util.Set<ManifestFile> committed)
Clean up any uncommitted manifests that were created.
|
RewriteManifests |
clusterBy(java.util.function.Function<DataFile,java.lang.Object> func)
Groups an existing
DataFile by a cluster key produced by a function. |
void |
commit()
Apply the pending changes and commit.
|
protected CommitMetrics |
commitMetrics() |
protected TableMetadata |
current() |
protected void |
deleteFile(java.lang.String path) |
RewriteManifests |
deleteManifest(ManifestFile manifest)
Deletes a
manifest file from the table. |
ThisT |
deleteWith(java.util.function.Consumer<java.lang.String> deleteCallback)
Set a callback to delete files instead of the table's default.
|
protected OutputFile |
manifestListPath() |
protected ManifestReader<DeleteFile> |
newDeleteManifestReader(ManifestFile manifest) |
protected ManifestWriter<DeleteFile> |
newDeleteManifestWriter(PartitionSpec spec) |
protected OutputFile |
newManifestOutput() |
protected ManifestReader<DataFile> |
newManifestReader(ManifestFile manifest) |
protected ManifestWriter<DataFile> |
newManifestWriter(PartitionSpec spec) |
protected java.lang.String |
operation()
A string that describes the action that produced the new snapshot.
|
protected TableMetadata |
refresh() |
protected ThisT |
reportWith(MetricsReporter newReporter) |
RewriteManifests |
rewriteIf(java.util.function.Predicate<ManifestFile> pred)
Determines which existing
ManifestFile for the table should be rewritten. |
ThisT |
scanManifestsWith(java.util.concurrent.ExecutorService executorService)
Use a particular executor to scan manifests.
|
protected RewriteManifests |
self() |
RewriteManifests |
set(java.lang.String property,
java.lang.String value)
Set a summary property in the snapshot produced by this update.
|
protected long |
snapshotId() |
ThisT |
stageOnly()
Called to stage a snapshot in table metadata, but not update the current snapshot id.
|
protected java.util.Map<java.lang.String,java.lang.String> |
summary() |
protected java.lang.String |
targetBranch() |
protected void |
targetBranch(java.lang.String branch)
A setter for the target branch on which snapshot producer operation should be performed
|
protected void |
validate(TableMetadata currentMetadata,
Snapshot snapshot)
Validate the current metadata.
|
protected java.util.concurrent.ExecutorService |
workerPool() |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
deleteWith, scanManifestsWith, stageOnly, toBranch
apply, commit, updateEvent
protected RewriteManifests self()
protected java.lang.String operation()
public RewriteManifests set(java.lang.String property, java.lang.String value)
SnapshotUpdate
set
in interface SnapshotUpdate<RewriteManifests>
property
- a String property namevalue
- a String property valueprotected java.util.Map<java.lang.String,java.lang.String> summary()
public RewriteManifests clusterBy(java.util.function.Function<DataFile,java.lang.Object> func)
RewriteManifests
DataFile
by a cluster key produced by a function. The cluster key
will determine which data file will be associated with a particular manifest. All data files
with the same cluster key will be written to the same manifest (unless the file is large and
split into multiple files). Manifests deleted via RewriteManifests.deleteManifest(ManifestFile)
or
added via RewriteManifests.addManifest(ManifestFile)
are ignored during the rewrite process.clusterBy
in interface RewriteManifests
func
- Function used to cluster data files to manifests.public RewriteManifests rewriteIf(java.util.function.Predicate<ManifestFile> pred)
RewriteManifests
ManifestFile
for the table should be rewritten. Manifests
that do not match the predicate are kept as-is. If this is not called and no predicate is set,
then all manifests will be rewritten.rewriteIf
in interface RewriteManifests
pred
- Predicate used to determine which manifests to rewrite. If true then the
manifest file will be included for rewrite. If false then then manifest is kept as-is.public RewriteManifests deleteManifest(ManifestFile manifest)
RewriteManifests
manifest file
from the table.deleteManifest
in interface RewriteManifests
manifest
- a manifest to deletepublic RewriteManifests addManifest(ManifestFile manifest)
RewriteManifests
manifest file
to the table. The added manifest cannot contain new
or deleted files.
By default, the manifest will be rewritten to ensure all entries have explicit snapshot IDs. In that case, it is always the responsibility of the caller to manage the lifecycle of the original manifest.
If manifest entries are allowed to inherit the snapshot ID assigned on commit, the manifest should never be deleted manually if the commit succeeds as it will become part of the table metadata and will be cleaned up on expiry. If the manifest gets merged with others while preparing a new snapshot, it will be deleted automatically if this operation is successful. If the commit fails, the manifest will never be deleted and it is up to the caller whether to delete or reuse it.
addManifest
in interface RewriteManifests
manifest
- a manifest to addpublic java.util.List<ManifestFile> apply(TableMetadata base, Snapshot snapshot)
base
- the base table metadata to apply changes tosnapshot
- snapshot to apply the changes toprotected void cleanUncommitted(java.util.Set<ManifestFile> committed)
Manifests may not be committed if apply is called more because a commit conflict has occurred. Implementations may keep around manifests because the same changes will be made by both apply calls. This method instructs the implementation to clean up those manifests and passes the paths of the manifests that were actually committed.
committed
- a set of manifest paths that were actually committedpublic ThisT stageOnly()
SnapshotUpdate
stageOnly
in interface SnapshotUpdate<ThisT>
public ThisT scanManifestsWith(java.util.concurrent.ExecutorService executorService)
SnapshotUpdate
scanManifestsWith
in interface SnapshotUpdate<ThisT>
executorService
- the provided executorprotected CommitMetrics commitMetrics()
protected ThisT reportWith(MetricsReporter newReporter)
protected void targetBranch(java.lang.String branch)
branch
- to set as target branchprotected java.lang.String targetBranch()
protected java.util.concurrent.ExecutorService workerPool()
public ThisT deleteWith(java.util.function.Consumer<java.lang.String> deleteCallback)
SnapshotUpdate
deleteWith
in interface SnapshotUpdate<ThisT>
deleteCallback
- a String consumer used to delete locations.protected void validate(TableMetadata currentMetadata, Snapshot snapshot)
Child operations can override this to add custom validation.
currentMetadata
- current table metadata to validatesnapshot
- ending snapshot on the lineage which is being validatedpublic Snapshot apply()
PendingUpdate
This does not result in a permanent update.
apply
in interface PendingUpdate<Snapshot>
PendingUpdate.commit()
protected TableMetadata current()
protected TableMetadata refresh()
public void commit()
PendingUpdate
Changes are committed by calling the underlying table's commit method.
Once the commit is successful, the updated table will be refreshed.
commit
in interface PendingUpdate<Snapshot>
protected void cleanAll()
protected void deleteFile(java.lang.String path)
protected OutputFile manifestListPath()
protected OutputFile newManifestOutput()
protected ManifestWriter<DataFile> newManifestWriter(PartitionSpec spec)
protected ManifestWriter<DeleteFile> newDeleteManifestWriter(PartitionSpec spec)
protected ManifestReader<DataFile> newManifestReader(ManifestFile manifest)
protected ManifestReader<DeleteFile> newDeleteManifestReader(ManifestFile manifest)
protected long snapshotId()