FileIO¶

FileIO reads, writes, and deletes Iceberg data and metadata files.

Built-in implementations¶

Call iceberg::arrow::RegisterAll() to register the Arrow-backed FileIO implementations:

Registry name	Schemes
`arrow-fs-local`	paths without a scheme, `file`
`arrow-fs-s3`	`s3`, `s3a`, `s3n`, `oss`

The S3 implementation requires Arrow S3 support.

Select an implementation¶

Load a registered implementation directly:

#include "iceberg/arrow/arrow_register.h"
#include "iceberg/arrow/s3/s3_properties.h"
#include "iceberg/file_io_registry.h"
#include "iceberg/resolving_file_io.h"

iceberg::arrow::RegisterAll();

auto file_io = iceberg::FileIORegistry::Load(
    iceberg::FileIORegistry::kArrowS3FileIO,
    {{std::string(iceberg::arrow::S3Properties::kEndpoint),
      "https://s3.example.com"},
     {std::string(iceberg::arrow::S3Properties::kClientRegion), "us-east-1"}});

For a REST catalog, set io-impl to the registry name. If it is omitted, the REST catalog uses ResolvingFileIO and selects a registered implementation for each file location's scheme.

Configure S3¶

Key	Example	Description
`s3.access-key-id`	`admin`	Static access key ID; must be set together with the secret key
`s3.secret-access-key`	`password`	Static secret access key
`s3.session-token`	`AQoDYXdzEJr...`	Session token, for temporary credentials. Ignored unless both static keys are set
`client.region`	`us-east-1`	Region to sign requests for
`s3.endpoint`	`https://127.0.0.1:9000`	Endpoint to use instead of the AWS one. When absent, the `AWS_ENDPOINT_URL_S3` / `AWS_ENDPOINT_URL` environment variables are consulted
`s3.path-style-access`	`true`	Address buckets as a path (`endpoint/bucket`) instead of a virtual host (`bucket.endpoint`). Only takes effect together with a custom endpoint

The following keys are specific to iceberg-cpp; they are not part of the Java Iceberg or REST specification property set:

Key	Example	Description
`s3.ssl.enabled`	`true`	Scheme to use for the endpoint, overriding the one it carries
`s3.connect-timeout-ms`	`1000`	Connection timeout
`s3.socket-timeout-ms`	`5000`	Request timeout. Ignored outside Windows and macOS

Without credentials, the AWS default credential chain is used, which covers environment variables, the shared configuration file, and the various role and identity providers.

S3-compatible storage¶

Stores that speak the S3 API are served by the same implementation. The scheme selects it; s3.endpoint decides where requests actually go. A location keeps its own scheme and is canonicalized internally, so a credential vended for the s3 prefix applies to it.

For Alibaba Cloud OSS, point s3.endpoint at the S3-compatible endpoint of the bucket's region and set s3.path-style-access to false: with a custom endpoint, buckets are addressed as a path unless told otherwise, and the service rejects that with SecondLevelDomainForbidden: Please use virtual hosted style to access:

auto file_io = iceberg::FileIORegistry::Load(
    iceberg::FileIORegistry::kArrowS3FileIO,
    {{std::string(iceberg::arrow::S3Properties::kEndpoint),
      "https://s3.oss-cn-hangzhou.aliyuncs.com"},
     {std::string(iceberg::arrow::S3Properties::kClientRegion), "cn-hangzhou"},
     {std::string(iceberg::arrow::S3Properties::kPathStyleAccess), "false"}});

file_io.value()->NewInputFile("oss://bucket/path/to/file.parquet");

Register a custom FileIO¶

Register the factory before creating the catalog or resolver:

#include <string_view>

iceberg::FileIORegistry::Register(
    "my-file-io",
    {.create = [](const iceberg::FileIORegistry::Properties& properties)
                   -> iceberg::Result<std::unique_ptr<iceberg::FileIO>> {
       return MakeMyFileIO(properties);
     },
     .accepts = [](std::string_view scheme) { return scheme == "myfs"; }});

create is required. Set accepts to enable automatic selection; it receives the normalized lower-case scheme. Leave it empty for an implementation selected only by io-impl.

iceberg::FileIORegistry::Load("my-file-io", {});
auto file_io = std::make_unique<iceberg::ResolvingFileIO>(
    iceberg::FileIORegistry::Properties{});
file_io->NewInputFile("myfs://bucket/path/file.parquet");

Registrations are process-wide and must be completed before creating catalogs or resolvers. When multiple implementations accept the same scheme, the last registration takes precedence. ResolvingFileIO lazily creates and reuses one FileIO instance per registry name.

Storage credentials¶

When a REST catalog returns vended storage credentials for a table, it applies them to the table's FileIO. A custom FileIO selected through io-impl must implement SupportsStorageCredentials; otherwise table access with vended credentials is unsupported. With automatic resolution, ResolvingFileIO forwards credentials to registered delegates that support them.