|
iceberg-cpp
|
Filters an existing snapshot's manifest list. More...
#include <manifest_filter_manager.h>
Public Types | |
| using | PartitionSpecsById = std::unordered_map< int32_t, std::shared_ptr< PartitionSpec > > |
Public Member Functions | |
| ManifestFilterManager (const ManifestFilterManager &)=delete | |
| ManifestFilterManager & | operator= (const ManifestFilterManager &)=delete |
| Status | DeleteByRowFilter (std::shared_ptr< Expression > expr) |
| Register a row-filter expression. | |
| void | CaseSensitive (bool case_sensitive) |
| Set whether row-filter field binding is case-sensitive. | |
| Status | DeleteFile (std::string_view path) |
| Register an exact file path for deletion. | |
| Status | DeleteFile (std::shared_ptr< DataFile > file) |
| Register a file object for deletion. | |
| const DataFileSet & | FilesToBeDeleted () const |
| Returns the set of file objects marked for deletion by this manager. | |
| const std::vector< std::shared_ptr< DataFile > > & | DeletedFiles () const |
| Returns content-file objects deleted by the most recent FilterManifests() call, deduplicated by content-file identity. | |
| int32_t | DuplicateDeletesCount () const |
| Returns how many duplicate file deletes were found in the most recent FilterManifests() call. | |
| Result< SnapshotSummaryBuilder > | BuildSummary (const std::vector< ManifestFile > &manifests, const PartitionSpecsById &specs_by_id) const |
| Build a snapshot-summary fragment from filtered manifests. | |
| Status | DropPartition (int32_t spec_id, PartitionValues partition) |
| Register a partition for dropping. | |
| void | FailMissingDeletePaths () |
| Set a flag that makes FilterManifests() fail if any registered delete path was not found in any manifest entry. | |
| void | FailAnyDelete () |
| Set a flag that makes FilterManifests() return an error if any manifest entry matches a delete condition. | |
| int32_t | ReplacedManifestsCount () const |
| Returns the number of manifests rewritten (replaced) by the last FilterManifests() call. A manifest is replaced when it contained deleted entries and was rewritten with those entries marked DELETED. | |
| bool | ContainsDeletes () const |
| Returns true if any delete condition has been registered. | |
| Status | DropDeleteFilesOlderThan (int64_t sequence_number) |
| Set the minimum data sequence number for delete files to retain. | |
| void | RemoveDanglingDeletesFor (const DataFileSet &deleted_files) |
| Register data files that have been removed so their dangling DVs can be cleaned up. | |
| Result< std::vector< ManifestFile > > | FilterManifests (const TableMetadata &metadata, const std::shared_ptr< Snapshot > &base_snapshot, const ManifestWriterFactory &writer_factory) |
| Apply all accumulated delete conditions to the base snapshot's manifests. | |
| Result< std::vector< ManifestFile > > | FilterManifests (const std::shared_ptr< Schema > &schema, const TableMetadata &metadata, const std::shared_ptr< Snapshot > &base_snapshot, const ManifestWriterFactory &writer_factory) |
| Apply all accumulated delete conditions using an explicit schema. | |
| Result< std::vector< ManifestFile > > | FilterManifests (const std::shared_ptr< Schema > &schema, const PartitionSpecsById &specs_by_id, const std::vector< const ManifestFile * > &manifests, const ManifestWriterFactory &writer_factory) |
| Apply all accumulated delete conditions to the provided manifests. | |
| Status | CleanUncommitted (const std::unordered_set< std::string > &committed) |
| Delete cached filtered manifests that were not committed and roll back replaced-manifest accounting. | |
Static Public Member Functions | |
| static Result< std::unique_ptr< ManifestFilterManager > > | Make (ManifestContent content, std::shared_ptr< FileIO > file_io, std::function< Status(const std::string &)> delete_file={}) |
Filters an existing snapshot's manifest list.
The manager accumulates delete conditions incrementally, then applies them all at once in a single FilterManifests() call. Manifests that contain no deleted entries are returned unchanged (no I/O). Manifests that do contain deleted entries are rewritten with those entries marked DELETED.
TODO(Guotao): For ManifestContent::kDeletes, implement cleanup for orphan delete files and dangling deletion vectors.
| Status iceberg::ManifestFilterManager::DeleteByRowFilter | ( | std::shared_ptr< Expression > | expr | ) |
Register a row-filter expression.
Any manifest entry whose column metrics indicate the file may satisfy the expression will be marked DELETED.
| expr | The expression to match files against |
| Status iceberg::ManifestFilterManager::DeleteFile | ( | std::shared_ptr< DataFile > | file | ) |
Register a file object for deletion.
Any manifest entry whose file_path matches file->file_path will be marked DELETED. Duplicate registrations (same path) are silently ignored.
| file | The data/delete file to delete (must not be null) |
| Status iceberg::ManifestFilterManager::DeleteFile | ( | std::string_view | path | ) |
Register an exact file path for deletion.
Any manifest entry whose file_path matches this path will be marked DELETED.
| path | The exact file path to delete |
| Status iceberg::ManifestFilterManager::DropDeleteFilesOlderThan | ( | int64_t | sequence_number | ) |
Set the minimum data sequence number for delete files to retain.
Only valid for ManifestContent::kDeletes managers. Delete entries whose data_sequence_number is positive and less than sequence_number will be marked DELETED. This continuously removes delete files that cannot match any remaining data rows (i.e. all data written before that sequence number has itself been deleted).
| sequence_number | the inclusive lower bound; delete files older than this value are dropped |
| Status iceberg::ManifestFilterManager::DropPartition | ( | int32_t | spec_id, |
| PartitionValues | partition | ||
| ) |
Register a partition for dropping.
Any manifest entry whose (spec_id, partition) pair matches will be marked DELETED.
| spec_id | The partition spec ID |
| partition | The partition values to drop |
| const DataFileSet & iceberg::ManifestFilterManager::FilesToBeDeleted | ( | ) | const |
Returns the set of file objects marked for deletion by this manager.
Includes file objects explicitly registered for deletion plus files deleted while filtering manifests.
| Result< std::vector< ManifestFile > > iceberg::ManifestFilterManager::FilterManifests | ( | const std::shared_ptr< Schema > & | schema, |
| const PartitionSpecsById & | specs_by_id, | ||
| const std::vector< const ManifestFile * > & | manifests, | ||
| const ManifestWriterFactory & | writer_factory | ||
| ) |
Apply all accumulated delete conditions to the provided manifests.
This overload accepts only the context needed for filtering. It is intended for callers that already have the active schema, partition specs, and manifest list.
| schema | Active schema to bind row-filter expressions and metrics evaluators |
| specs_by_id | All partition specs keyed by spec ID |
| manifests | Manifest descriptors to filter |
| writer_factory | Factory to create new ManifestWriter instances |
| Result< std::vector< ManifestFile > > iceberg::ManifestFilterManager::FilterManifests | ( | const std::shared_ptr< Schema > & | schema, |
| const TableMetadata & | metadata, | ||
| const std::shared_ptr< Snapshot > & | base_snapshot, | ||
| const ManifestWriterFactory & | writer_factory | ||
| ) |
Apply all accumulated delete conditions using an explicit schema.
This overload is used when callers need row-filter evaluation bound against a schema other than metadata.Schema(), such as the schema at a branch head.
| Result< std::vector< ManifestFile > > iceberg::ManifestFilterManager::FilterManifests | ( | const TableMetadata & | metadata, |
| const std::shared_ptr< Snapshot > & | base_snapshot, | ||
| const ManifestWriterFactory & | writer_factory | ||
| ) |
Apply all accumulated delete conditions to the base snapshot's manifests.
Manifests that cannot possibly contain deleted files are returned unchanged. Manifests that do contain deleted files are rewritten using writer_factory.
| metadata | Table metadata (provides specs and schema for evaluators) |
| base_snapshot | The snapshot whose manifests to filter (may be null) |
| writer_factory | Factory to create new ManifestWriter instances |
| void iceberg::ManifestFilterManager::RemoveDanglingDeletesFor | ( | const DataFileSet & | deleted_files | ) |
Register data files that have been removed so their dangling DVs can be cleaned up.
Only valid for ManifestContent::kDeletes managers. For each DV whose referenced_data_file path appears in deleted_files, the DV entry is marked DELETED because the data file it targets no longer exists.
| deleted_files | set of data files that have been marked for deletion |
1.9.8