FileIO¶

FileIO reads, writes, and deletes Iceberg data and metadata files.

Built-in implementations¶

Call iceberg::arrow::RegisterAll() to register the Arrow-backed FileIO implementations:

Registry name	Schemes
`arrow-fs-local`	paths without a scheme, `file`
`arrow-fs-s3`	`s3`, `s3a`, `s3n`, `oss`

The S3 implementation requires Arrow S3 support.

Select an implementation¶

Load a registered implementation directly:

#include "iceberg/arrow/arrow_register.h"
#include "iceberg/arrow/s3/s3_properties.h"
#include "iceberg/file_io_registry.h"
#include "iceberg/resolving_file_io.h"

iceberg::arrow::RegisterAll();

auto file_io = iceberg::FileIORegistry::Load(
    iceberg::FileIORegistry::kArrowS3FileIO,
    {{std::string(iceberg::arrow::S3Properties::kEndpoint),
      "https://s3.example.com"},
     {std::string(iceberg::arrow::S3Properties::kClientRegion), "us-east-1"}});

For a REST catalog, set io-impl to the registry name. If it is omitted, the REST catalog uses ResolvingFileIO and selects a registered implementation for each file location's scheme.

Configure S3¶

Key	Example	Description
`s3.access-key-id`	`admin`	Static access key ID; must be set together with the secret key
`s3.secret-access-key`	`password`	Static secret access key
`s3.session-token`	`AQoDYXdzEJr...`	Session token, for temporary credentials. Ignored unless both static keys are set
`s3.session-token-expires-at-ms`	`1767225600000`	When the session token expires, in epoch milliseconds. Vended credentials are refreshed from the catalog five minutes before, including for files already open; once they have expired and cannot be refreshed, new operations under their prefix fail
`client.region`	`us-east-1`	Region to sign requests for
`s3.endpoint`	`https://127.0.0.1:9000`	Endpoint to use instead of the AWS one. When absent, the `AWS_ENDPOINT_URL_S3` / `AWS_ENDPOINT_URL` environment variables are consulted
`s3.path-style-access`	`true`	Address buckets as a path (`endpoint/bucket`) instead of a virtual host (`bucket.endpoint`). Only takes effect together with a custom endpoint
`s3.delete.num-threads`	`8`	Maximum worker tasks per `DeleteFiles` call; defaults to the hardware thread count. Requires a delete executor

The following keys are specific to iceberg-cpp; they are not part of the Java Iceberg or REST specification property set:

Key	Example	Description
`s3.ssl.enabled`	`true`	Scheme to use for the endpoint, overriding the one it carries
`s3.connect-timeout-ms`	`1000`	Connection timeout
`s3.socket-timeout-ms`	`5000`	Request timeout. Ignored outside Windows and macOS

Without credentials, the AWS default credential chain is used, which covers environment variables, the shared configuration file, and the various role and identity providers.

Bulk deletion¶

S3 DeleteFiles attempts each file and returns the failure count, including exceptions thrown by a delete. Deletion is sequential by default. To enable concurrency, call iceberg::arrow::InitializeS3FileIODeleteExecutor(executor) from iceberg/arrow/arrow_io_util.h before the first delete. The application owns the executor; the setting applies to direct and registry-created FileIOs.

Initialization is permanent. Repeating the same pointer is allowed, but changing it returns an error. The first delete freezes the default nullptr if no executor was initialized. Keep the executor alive until the last delete finishes, and do not start more deletes after destroying it.

s3.delete.num-threads limits workers within one call, not across concurrent calls. The shared executor controls total concurrency; this property does not create or resize a thread pool. Workers share the file list, so task and future storage is bounded by the configured parallelism.

DeleteFiles waits for accepted work before returning, even when submission fails or throws. A submission failure is returned without deleting on the caller. The executor must make progress while callers wait; do not call DeleteFiles from its workers unless it supports nested blocking work.

S3-compatible storage¶

Stores that speak the S3 API are served by the same implementation. The scheme selects it; s3.endpoint decides where requests actually go. A location keeps its own scheme and is canonicalized internally, so a credential vended for the s3 prefix applies to it.

For Alibaba Cloud OSS, point s3.endpoint at the S3-compatible endpoint of the bucket's region and set s3.path-style-access to false: with a custom endpoint, buckets are addressed as a path unless told otherwise, and the service rejects that with SecondLevelDomainForbidden: Please use virtual hosted style to access:

auto file_io = iceberg::FileIORegistry::Load(
    iceberg::FileIORegistry::kArrowS3FileIO,
    {{std::string(iceberg::arrow::S3Properties::kEndpoint),
      "https://s3.oss-cn-hangzhou.aliyuncs.com"},
     {std::string(iceberg::arrow::S3Properties::kClientRegion), "cn-hangzhou"},
     {std::string(iceberg::arrow::S3Properties::kPathStyleAccess), "false"}});

file_io.value()->NewInputFile("oss://bucket/path/to/file.parquet");

Register a custom FileIO¶

Register the factory before creating the catalog or resolver:

#include <string_view>

iceberg::FileIORegistry::Register(
    "my-file-io",
    {.create = [](const iceberg::FileIORegistry::Properties& properties)
                   -> iceberg::Result<std::unique_ptr<iceberg::FileIO>> {
       return MakeMyFileIO(properties);
     },
     .accepts = [](std::string_view scheme) { return scheme == "myfs"; }});

create is required. Set accepts to enable automatic selection; it receives the normalized lower-case scheme. Leave it empty for an implementation selected only by io-impl.

iceberg::FileIORegistry::Load("my-file-io", {});
auto file_io = std::make_unique<iceberg::ResolvingFileIO>(
    iceberg::FileIORegistry::Properties{});
file_io->NewInputFile("myfs://bucket/path/file.parquet");

Registrations are process-wide and must be completed before creating catalogs or resolvers. When multiple implementations accept the same scheme, the last registration takes precedence. ResolvingFileIO lazily creates and reuses one FileIO instance per registry name.

Storage credentials¶

When a REST catalog returns vended storage credentials for a table, it applies them to the table's FileIO. A custom FileIO selected through io-impl must implement SupportsStorageCredentials; otherwise table access with vended credentials is unsupported. With automatic resolution, ResolvingFileIO forwards credentials to registered delegates that support them.