API for replacing files in a table. More...

#include <rewrite_files.h>

Inheritance diagram for iceberg::RewriteFiles:

Public Member Functions
RewriteFiles &	DeleteDataFile (const std::shared_ptr< DataFile > &data_file)
	Remove a data file from the current table state.

RewriteFiles &	DeleteDeleteFile (const std::shared_ptr< DataFile > &delete_file)
	Remove a delete file from the table state.

RewriteFiles &	AddDataFile (const std::shared_ptr< DataFile > &file)
	Add a new data file.

RewriteFiles &	AddDeleteFile (const std::shared_ptr< DataFile > &delete_file)
	Add a new delete file.

RewriteFiles &	AddDeleteFile (const std::shared_ptr< DataFile > &delete_file, int64_t data_sequence_number)
	Add a new delete file with the given data sequence number.

RewriteFiles &	SetDataSequenceNumber (int64_t sequence_number)
	Configure the data sequence number for this rewrite operation.

RewriteFiles &	RewriteDataFiles (const std::vector< std::shared_ptr< DataFile > > &files_to_delete, const std::vector< std::shared_ptr< DataFile > > &files_to_add, int64_t sequence_number)
	Add a rewrite that replaces one set of data files with another set that contains the same data. The sequence number provided will be used for all the data files added.

RewriteFiles &	Rewrite (const std::vector< std::shared_ptr< DataFile > > &data_files_to_replace, const std::vector< std::shared_ptr< DataFile > > &delete_files_to_replace, const std::vector< std::shared_ptr< DataFile > > &data_files_to_add, const std::vector< std::shared_ptr< DataFile > > &delete_files_to_add)
	Add a rewrite that replaces one set of files with another set that contains the same data.

RewriteFiles &	ValidateFromSnapshot (int64_t snapshot_id)
	Set the snapshot ID used in any reads for this operation.

std::string	operation () override
	A string that describes the action that produced the new snapshot.

Public Member Functions inherited from iceberg::MergingSnapshotUpdate
Result< std::vector< ManifestFile > >	Apply (const TableMetadata &metadata_to_update, const std::shared_ptr< Snapshot > &snapshot) override
	Apply the update's changes to the given metadata and snapshot.

Status	CleanUncommitted (const std::unordered_set< std::string > &committed) override
	Clean up any uncommitted manifests that were created.

std::unordered_map< std::string, std::string >	Summary () override
	Get the summary map for this operation.

Public Member Functions inherited from iceberg::SnapshotUpdate
Kind	kind () const override
	Return the kind of this pending update.

bool	IsRetryable () const override
	Whether this update can be retried after a commit conflict.

auto &	DeleteWith (this auto &self, std::function< Status(const std::string &)> delete_func)
	Set a callback to delete files instead of the table's default.

auto &	StageOnly (this auto &self)
	Stage a snapshot in table metadata, but do not make it current.

auto &	ScanManifestsWith (this auto &self, Executor &executor)
	Configure an executor for manifest planning work.

auto &	ToBranch (this auto &self, const std::string &branch)
	Perform operations on a particular branch.

auto &	Set (this auto &self, const std::string &property, const std::string &value)
	Set a summary property in the snapshot produced by this update.

auto &	WriteManifestsWith (this auto &self, Executor &executor, int32_t parallelism)
	Configure an executor and max writer count for writing new manifests.

Result< ApplyResult >	Apply ()
	Apply the update's changes to create a new snapshot.

Status	Finalize (Result< const TableMetadata * > commit_result) override
	Finalize the snapshot update, cleaning up any uncommitted files.

Public Member Functions inherited from iceberg::PendingUpdate
virtual Status	Commit ()
	Apply the pending changes and commit.

	PendingUpdate (const PendingUpdate &)=delete

PendingUpdate &	operator= (const PendingUpdate &)=delete

	PendingUpdate (PendingUpdate &&) noexcept=default

PendingUpdate &	operator= (PendingUpdate &&) noexcept=default

Public Member Functions inherited from iceberg::ErrorCollector
	ErrorCollector (ErrorCollector &&)=default

ErrorCollector &	operator= (ErrorCollector &&)=default

	ErrorCollector (const ErrorCollector &)=default

ErrorCollector &	operator= (const ErrorCollector &)=default

template<typename... Args>
auto &	AddError (this auto &self, ErrorKind kind, const std::format_string< Args... > fmt, Args &&... args)
	Add a specific error and return reference to derived class.

auto &	AddError (this auto &self, Error err)
	Add an existing error object and return reference to derived class.

auto &	AddError (this auto &self, std::unexpected< Error > err)
	Add an unexpected result's error and return reference to derived class.

bool	has_errors () const
	Check if any errors have been collected.

size_t	error_count () const
	Get the number of errors collected.

Status	CheckErrors () const
	Check for accumulated errors and return them if any exist.

void	ClearErrors ()
	Clear all accumulated errors.

const std::vector< Error > &	errors () const
	Get read-only access to all collected errors.

Static Public Member Functions
static Result< std::unique_ptr< RewriteFiles > >	Make (std::string table_name, std::shared_ptr< TransactionContext > ctx)
	Create a new RewriteFiles operation.

Protected Member Functions
Status	Validate (const TableMetadata &current_metadata, const std::shared_ptr< Snapshot > &snapshot) override
	Validate the current metadata.

	RewriteFiles (std::string table_name, std::shared_ptr< TransactionContext > ctx)

Protected Member Functions inherited from iceberg::MergingSnapshotUpdate
	MergingSnapshotUpdate (std::string table_name, std::shared_ptr< TransactionContext > ctx)
	Constructor; reads merge configuration from table properties.

Status	AddDataFile (std::shared_ptr< DataFile > file)
	Stage a data file to be added to the table.

Status	AddDeleteFile (std::shared_ptr< DataFile > file)
	Stage a delete file to be added to the table.

Status	ValidateNewDeleteFile (const TableMetadata &metadata, const DataFile &file)
	Validate a delete file against the table format version rules.

Status	AddDeleteFile (std::shared_ptr< DataFile > file, int64_t data_sequence_number)
	Stage a delete file with an explicit data sequence number.

Status	AddManifest (ManifestFile manifest)
	Add all files in a pre-existing data manifest to the new snapshot.

Status	DeleteDataFile (std::shared_ptr< DataFile > file)
	Register a data file (by object) to be deleted from the table.

Status	DeleteDeleteFile (std::shared_ptr< DataFile > file)
	Register a delete file (by object) to be removed from the table.

Status	DeleteByPath (std::string_view path)
	Register a data file path to be deleted from the table.

Status	DeleteByRowFilter (std::shared_ptr< Expression > expr)
	Register an expression to delete matching rows.

Status	DropPartition (int32_t spec_id, PartitionValues partition)
	Register a partition to be dropped.

void	FailMissingDeletePaths ()
	Fail if any registered delete path is not found in any manifest.

void	FailAnyDelete ()
	Fail if any manifest entry matches a delete condition.

void	SetNewDataFilesDataSequenceNumber (int64_t sequence_number)
	Override the data sequence number assigned to all newly-added data files.

bool	HasDataSequenceNumber () const
	Returns true if SetNewDataFilesDataSequenceNumber was called.

void	CaseSensitive (bool case_sensitive)
	Set case sensitivity for row filter and expression evaluation.

bool	IsCaseSensitive () const
	Returns true if case-sensitive matching is enabled (default: true).

bool	AddsDataFiles () const
	Returns true if any data files have been staged for addition.

bool	AddsDeleteFiles () const
	Returns true if any delete files have been staged for addition.

bool	DeletesDataFiles () const
	Returns true if any data files have been registered for deletion.

bool	DeletesDeleteFiles () const
	Returns true if any delete files have been registered for removal.

const std::shared_ptr< Expression > &	RowFilter () const
	Returns the row-filter expression set via DeleteByRowFilter, or nullptr.

Result< std::shared_ptr< PartitionSpec > >	DataSpec () const
	Returns the single partition spec for all staged data files.

std::vector< std::shared_ptr< DataFile > >	AddedDataFiles () const
	Returns all data files staged for addition.

Status	ValidateNoNewDeletesForDataFiles (const TableMetadata &metadata, std::optional< int64_t > starting_snapshot_id, const DataFileSet &replaced_files, const std::shared_ptr< Snapshot > &parent, std::shared_ptr< FileIO > io) const
	Return an error if any snapshot after starting_snapshot_id, or from the beginning if unset, added a delete file that covers a file in replaced_files.

Status	ValidateAddedDVs (const TableMetadata &metadata, std::optional< int64_t > starting_snapshot_id, std::shared_ptr< Expression > conflict_filter, const std::shared_ptr< Snapshot > &parent, std::shared_ptr< FileIO > io) const
	Return an error if a staged deletion vector conflicts with a deletion vector added since starting_snapshot_id.

Protected Member Functions inherited from iceberg::SnapshotUpdate
	SnapshotUpdate (std::shared_ptr< TransactionContext > ctx)

Result< std::vector< ManifestFile > >	WriteDataManifests (std::span< const std::shared_ptr< DataFile > > files, const std::shared_ptr< PartitionSpec > &spec, std::optional< int64_t > data_sequence_number=std::nullopt)
	Write data manifests for the given data files.

Result< std::vector< ManifestFile > >	WriteDeleteManifests (std::span< const ContentFileWithSequenceNumber > files, const std::shared_ptr< PartitionSpec > &spec)

const std::string &	target_branch () const

bool	can_inherit_snapshot_id () const

const std::string &	commit_uuid () const

int32_t	manifest_count () const

int32_t	attempt () const

int64_t	target_manifest_size_bytes () const

virtual bool	CleanupAfterCommit () const
	Check if cleanup should happen after commit.

int64_t	SnapshotId ()
	Get or generate the snapshot ID for the new snapshot.

Status	DeleteFile (const std::string &path)
	Delete a file at the given path.

std::string	ManifestPath ()

std::string	ManifestListPath ()

SnapshotSummaryBuilder &	summary_builder ()

SnapshotSummaryBuilder	BuildManifestCountSummary (std::span< const ManifestFile > manifests, int32_t replaced_manifests_count)

Protected Member Functions inherited from iceberg::PendingUpdate
	PendingUpdate (std::shared_ptr< TransactionContext > ctx)

const TableMetadata &	base () const

Additional Inherited Members
Public Types inherited from iceberg::PendingUpdate
enum class	Kind : uint8_t { kExpireSnapshots , kSetSnapshot , kUpdateLocation , kUpdatePartitionSpec , kUpdatePartitionStatistics , kUpdateProperties , kUpdateSchema , kUpdateSnapshot , kUpdateSnapshotReference , kUpdateSortOrder , kUpdateStatistics }

Static Protected Member Functions inherited from iceberg::MergingSnapshotUpdate
static Status	ValidateAddedDataFiles (const TableMetadata &metadata, std::optional< int64_t > starting_snapshot_id, std::shared_ptr< Expression > filter, const std::shared_ptr< Snapshot > &parent, std::shared_ptr< FileIO > io, bool case_sensitive=true)
	Return an error if any snapshot after starting_snapshot_id, or from the beginning if unset, added a data file matching the given filter expression.

static Status	ValidateAddedDataFiles (const TableMetadata &metadata, std::optional< int64_t > starting_snapshot_id, const PartitionSet &partition_set, const std::shared_ptr< Snapshot > &parent, std::shared_ptr< FileIO > io)
	Return an error if any snapshot after starting_snapshot_id, or from the beginning if unset, added a data file in any partition of the given partition set.

static Status	ValidateDataFilesExist (const TableMetadata &metadata, std::optional< int64_t > starting_snapshot_id, const std::unordered_set< std::string > &file_paths, bool skip_deletes, std::shared_ptr< Expression > filter, const std::shared_ptr< Snapshot > &parent, std::shared_ptr< FileIO > io, bool case_sensitive=true)
	Return an error if any snapshot after starting_snapshot_id, or from the beginning if unset, removed a file whose path is in file_paths (and skip_deletes is false).

static Status	ValidateNoNewDeletesForDataFiles (const TableMetadata &metadata, std::optional< int64_t > starting_snapshot_id, const DataFileSet &replaced_files, const std::shared_ptr< Snapshot > &parent, std::shared_ptr< FileIO > io, bool ignore_equality_deletes=false)
	Return an error if any snapshot after starting_snapshot_id, or from the beginning if unset, added a delete file that covers a file in replaced_files.

static Status	ValidateNoNewDeletesForDataFiles (const TableMetadata &metadata, std::optional< int64_t > starting_snapshot_id, std::shared_ptr< Expression > data_filter, const DataFileSet &replaced_files, const std::shared_ptr< Snapshot > &parent, std::shared_ptr< FileIO > io, bool case_sensitive=true)
	Return an error if any snapshot after starting_snapshot_id, or from the beginning if unset, added a delete file matching the data filter that covers a file in replaced_files.

static Status	ValidateNoNewDeleteFiles (const TableMetadata &metadata, std::optional< int64_t > starting_snapshot_id, std::shared_ptr< Expression > data_filter, const std::shared_ptr< Snapshot > &parent, std::shared_ptr< FileIO > io, bool case_sensitive=true)
	Return an error if any snapshot after starting_snapshot_id, or from the beginning if unset, added a delete file matching the given row filter.

static Status	ValidateNoNewDeleteFiles (const TableMetadata &metadata, std::optional< int64_t > starting_snapshot_id, const PartitionSet &partition_set, const std::shared_ptr< Snapshot > &parent, std::shared_ptr< FileIO > io)
	Return an error if any snapshot after starting_snapshot_id, or from the beginning if unset, added a delete file matching any partition in the given partition set.

static Status	ValidateDeletedDataFiles (const TableMetadata &metadata, std::optional< int64_t > starting_snapshot_id, std::shared_ptr< Expression > data_filter, const std::shared_ptr< Snapshot > &parent, std::shared_ptr< FileIO > io, bool case_sensitive=true)
	Return an error if any snapshot after starting_snapshot_id, or from the beginning if unset, deleted a data file matching the given row filter.

static Status	ValidateDeletedDataFiles (const TableMetadata &metadata, std::optional< int64_t > starting_snapshot_id, const PartitionSet &partition_set, const std::shared_ptr< Snapshot > &parent, std::shared_ptr< FileIO > io)
	Return an error if any snapshot after starting_snapshot_id, or from the beginning if unset, deleted a data file in any partition of the given partition set.

static Result< std::unique_ptr< DeleteFileIndex > >	AddedDeleteFiles (const TableMetadata &metadata, std::optional< int64_t > starting_snapshot_id, std::shared_ptr< Expression > data_filter, std::shared_ptr< PartitionSet > partition_set, const std::shared_ptr< Snapshot > &parent, std::shared_ptr< FileIO > io, bool case_sensitive=true)
	Build a DeleteFileIndex of delete files added since starting_snapshot_id.

static Status	ValidateAddedDVs (const TableMetadata &metadata, std::optional< int64_t > starting_snapshot_id, std::shared_ptr< Expression > conflict_filter, const std::unordered_set< std::string > &referenced_data_files, const std::shared_ptr< Snapshot > &parent, std::shared_ptr< FileIO > io, bool case_sensitive=true)
	Return an error if any snapshot after starting_snapshot_id, or from the beginning if unset, added a deletion vector that conflicts with DVs being written.

Protected Attributes inherited from iceberg::SnapshotUpdate
SnapshotSummaryBuilder	summary_

Protected Attributes inherited from iceberg::PendingUpdate
std::shared_ptr< TransactionContext >	ctx_

Protected Attributes inherited from iceberg::ErrorCollector
std::vector< Error >	errors_

Detailed Description

API for replacing files in a table.

This API accumulates file additions and deletions, produces a new Snapshot of the changes, and commits that snapshot as the current.

When committing, these changes will be applied to the latest table snapshot. Commit conflicts will be resolved by applying the changes to the new latest snapshot and reattempting the commit. If any of the deleted files are no longer in the latest snapshot when reattempting, the commit will fail validation.

Note that the new state of the table after each rewrite must be logically equivalent to the original table state.

Member Function Documentation

◆ AddDataFile()

RewriteFiles & iceberg::RewriteFiles::AddDataFile ( const std::shared_ptr< DataFile > & file )

Add a new data file.

This rewrite operation may change the size or layout of the data files. When applicable, it is also recommended to discard already deleted records while rewriting data files. However, the set of live data records must never change.

Parameters

file	a new data file

Returns: this for method chaining

◆ AddDeleteFile() [1/2]

RewriteFiles & iceberg::RewriteFiles::AddDeleteFile ( const std::shared_ptr< DataFile > & delete_file )

Add a new delete file.

This rewrite operation may change the size or layout of the delete files. When applicable, it is also recommended to discard delete records for files that are no longer part of the table state. However, the set of applicable delete records must never change.

Parameters

delete_file a new delete file

Returns: this for method chaining

◆ AddDeleteFile() [2/2]

RewriteFiles & iceberg::RewriteFiles::AddDeleteFile	(	const std::shared_ptr< DataFile > &	delete_file,
		int64_t	data_sequence_number
	)

Add a new delete file with the given data sequence number.

This rewrite operation may change the size or layout of the delete files. When applicable, it is also recommended to discard delete records for files that are no longer part of the table state. However, the set of applicable delete records must never change.

To ensure equivalence in the set of applicable delete records, the sequence number of the delete file must be the max sequence number of the delete files that it is replacing. Rewriting equality deletes that belong to different sequence numbers is not allowed.

Parameters

delete_file	a new delete file
data_sequence_number	data sequence number to append on the file

Returns: this for method chaining

◆ DeleteDataFile()

RewriteFiles & iceberg::RewriteFiles::DeleteDataFile ( const std::shared_ptr< DataFile > & data_file )

Remove a data file from the current table state.

This rewrite operation may change the size or layout of the data files. When applicable, it is also recommended to discard already deleted records while rewriting data files. However, the set of live data records must never change.

Parameters

data_file a rewritten data file

Returns: this for method chaining

◆ DeleteDeleteFile()

RewriteFiles & iceberg::RewriteFiles::DeleteDeleteFile ( const std::shared_ptr< DataFile > & delete_file )

Remove a delete file from the table state.

This rewrite operation may change the size or layout of the delete files. When applicable, it is also recommended to discard delete records for files that are no longer part of the table state. However, the set of applicable delete records must never change.

Parameters

delete_file a rewritten delete file

Returns: this for method chaining

◆ Make()

Result< std::unique_ptr< RewriteFiles > > iceberg::RewriteFiles::Make	(	std::string	table_name,
		std::shared_ptr< TransactionContext >	ctx
	)

static

Create a new RewriteFiles operation.

Parameters

table_name	The name of the table
ctx	The transaction context

Returns: A unique pointer to the new RewriteFiles operation

◆ operation()

std::string iceberg::RewriteFiles::operation ( )

overridevirtual

A string that describes the action that produced the new snapshot.

Returns: A string operation name

Implements iceberg::SnapshotUpdate.

◆ Rewrite()

RewriteFiles & iceberg::RewriteFiles::Rewrite	(	const std::vector< std::shared_ptr< DataFile > > &	data_files_to_replace,
		const std::vector< std::shared_ptr< DataFile > > &	delete_files_to_replace,
		const std::vector< std::shared_ptr< DataFile > > &	data_files_to_add,
		const std::vector< std::shared_ptr< DataFile > > &	delete_files_to_add
	)

Add a rewrite that replaces one set of files with another set that contains the same data.

Parameters

data_files_to_replace	data files that will be replaced (deleted)
delete_files_to_replace	delete files that will be replaced (deleted)
data_files_to_add	data files that will be added
delete_files_to_add	delete files that will be added

Returns: this for method chaining

◆ RewriteDataFiles()

RewriteFiles & iceberg::RewriteFiles::RewriteDataFiles	(	const std::vector< std::shared_ptr< DataFile > > &	files_to_delete,
		const std::vector< std::shared_ptr< DataFile > > &	files_to_add,
		int64_t	sequence_number
	)

Add a rewrite that replaces one set of data files with another set that contains the same data. The sequence number provided will be used for all the data files added.

Parameters

files_to_delete	files that will be replaced (deleted), cannot be null or empty
files_to_add	files that will be added, cannot be null or empty
sequence_number	sequence number to use for all data files added

Returns: this for method chaining

◆ SetDataSequenceNumber()

RewriteFiles & iceberg::RewriteFiles::SetDataSequenceNumber ( int64_t sequence_number )

Configure the data sequence number for this rewrite operation.

This data sequence number will be used for all new data files that are added in this rewrite. This method is helpful to avoid commit conflicts between data compaction and adding equality deletes.

Parameters

sequence_number a data sequence number

Returns: this for method chaining

◆ Validate()

Status iceberg::RewriteFiles::Validate	(	const TableMetadata &	current_metadata,
		const std::shared_ptr< Snapshot > &	snapshot
	)

overrideprotectedvirtual

Validate the current metadata.

Child operations can override this to add custom validation.

Parameters

current_metadata	Current table metadata to validate
snapshot	Ending snapshot on the lineage which is being validated

Reimplemented from iceberg::SnapshotUpdate.

◆ ValidateFromSnapshot()

RewriteFiles & iceberg::RewriteFiles::ValidateFromSnapshot ( int64_t snapshot_id )

Set the snapshot ID used in any reads for this operation.

Validations will check changes after this snapshot ID. If this is not called, all ancestor snapshots through the table's initial snapshot are validated.

Parameters

snapshot_id a snapshot ID

Returns: this for method chaining

The documentation for this class was generated from the following files:

iceberg/update/rewrite_files.h
iceberg/update/rewrite_files.cc

Public Member Functions

Static Public Member Functions

Protected Member Functions

Additional Inherited Members

Detailed Description

Member Function Documentation

◆ AddDataFile()

◆ AddDeleteFile() [1/2]

◆ AddDeleteFile() [2/2]

◆ DeleteDataFile()

◆ DeleteDeleteFile()

◆ Make()

◆ operation()

◆ Rewrite()

◆ RewriteDataFiles()

◆ SetDataSequenceNumber()

◆ Validate()

◆ ValidateFromSnapshot()