Load Data from Object Storage and HDFS

You can use the datalake_fdw extension to load data from an object storage (such as Amazon S3), HDFS, and ORC tables in Hive into SynxDB for data query and access.

Currently, supported data formats are CSV, TEXT, ORC, and PARQUET.

Note

datalake_fdw does not support loading data in parallel.

Attention

  • The default port for the data_lake agent has been changed from 5888 to 3888 to avoid conflict with PXF.

  • The datalake_agent is a background service that must run on each SynxDB node (coordinator and all segments). By default, this agent binds to localhost and listens on port 3888, meaning it only accepts connections from the local database instance. This secure design ensures that the agent’s service port is not exposed to the network.

For information on how to load tables from Hive into SynxDB, see Load Data from Hive.

Install the extension

To install the datalake_fdw extension to the database, execute the SQL statement CREATE EXTENSION datalake_fdw;.

CREATE EXTENSION datalake_fdw;

Instructions

This section explains how to use datalake_fdw to load data from object storage and HDFS into SynxDB.

To load data using datalake_fdw, you need to create a foreign data wrapper (FDW) first. This includes creating an FDW server and user mapping.

Load data from object storage

You can load data from Amazon S3 or other object storages into SynxDB. Follow these steps:

  1. Create a foreign table wrapper FOREIGN DATA WRAPPER. Note that there are no options in the SQL statement below, and you need to execute it exactly as provided.

    CREATE FOREIGN DATA WRAPPER datalake_fdw
HANDLER datalake_fdw_handler
VALIDATOR datalake_fdw_validator
OPTIONS ( mpp_execute 'all segments' );

  2. Create an external server foreign_server.

    CREATE SERVER foreign_server
FOREIGN DATA WRAPPER datalake_fdw
OPTIONS (host 'xxx', protocol 's3b', isvirtual 'false', ishttps 'false');

    The options in the above SQL statement are explained as follows:

    Option name

    Description

    Details

    host

    Sets the host information for accessing the object storage.

    Required: Must be set

    Example:

    • Host for private cloud: 192.168.1.1:9000

    protocol

    Specifies the cloud platform for the object storage.

    Required: Must be set

    Options:

    • s3b: Amazon Cloud (uses v2 signature)

    • s3: Amazon Cloud (uses v4 signature)

    • azure: Azure Blob Storage

    • gcs: Google Cloud Storage

    Note: azure and gcs reuse the same option set as the other object storage protocols. Azure uses its native backend, and GCS uses the GCS XML API.

    isvirtual

    Use virtual-host-style or path-host-style to parse the host of the object storage.

    Required: Optional

    Options:

    • true: Uses virtual-host-style.

    • false: Uses path-host-style.

    Default value: false

    ishttps

    Whether to use HTTPS to access the object storage.

    Required: Optional

    Options:

    • true: Uses HTTPS.

    • false: Does not use HTTPS.

    Default value: false

  3. Create a user mapping.

    CREATE USER MAPPING FOR gpadmin SERVER foreign_server
OPTIONS (user 'gpadmin', accesskey 'xxx', secretkey 'xxx');

    The options in the above SQL statement are explained as follows:

    Option name

    Description

    Required

    user

    Creates the specific user specified by foreign_server.

    Yes

    accesskey

    The key needed to access the object storage.

    Yes

    secretkey

    The secret key needed to access the object storage.

    Yes

  4. Create a foreign table example. After creating it, the data on the object storage is loaded into SynxDB, and you can query this table.

    CREATE FOREIGN TABLE example(
a text,
b text
)
SERVER foreign_server
OPTIONS (filePath '/test/parquet/', compression 'none', enableCache 'false', format 'parquet');

    The options in the SQL statement above are explained as follows:

    Option name

    Description

    Details

    filePath

    Sets the specific path for the target foreign table.

    • Required: Must be set

    • Path format should be /bucket/prefix.

    • Example:

      • If the bucket name is test-bucket and the path is bucket/test/orc_file_folder/, and there are files like 0000_0, 0001_1, 0002_2, then to access file 0000_0, set filePath to filePath '/test-bucket/test/orc_file_folder/0000_0'.

      • To access all files in test/orc_file_folder/, set filePath to filePath '/test-bucket/test/orc_file_folder/'.

    • Note: filePath is parsed in the format /bucket/prefix/. Incorrect formats might lead to errors, such as:

      • filePath 'test-bucket/test/orc_file_folder/'

      • filePath '/test-bucket/test/orc_file_folder/0000_0'

    compression

    Sets the write compression format. Currently supports gzip, zstd, lz4.

    • Required: Optional

    • Options:

      • none: Supports CSV, ORC, TEXT, PARQUET.

      • gzip: Supports CSV, TEXT, PARQUET.

      • zstd: Supports PARQUET.

      • lz4: Supports PARQUET.

    • Default value: none, which means no compression. Not setting this value means no compression.

    enableCache

    Specifies whether to use Gopher caching.

    • Required: Optional

    • Options:

      • true: Enables Gopher caching.

      • false: Disables Gopher caching.

    • Default value: false

    • Deleting the foreign table does not automatically clear its cache. To clear the cache, you need to manually run a specific SQL function, such as: select gp_toolkit.__gopher_cache_free_relation_name(text);

    format

    The file format supported by FDW.

    • Required: Must be set

    • Options:

      • csv: Read, Write

      • text: Read, Write

      • orc: Read, Write

      • parquet: Read, Write

Use Iceberg tables on object storage

To work with Apache Iceberg tables on S3 or HDFS, use the dedicated Iceberg flow with CREATE FOREIGN CATALOG, CREATE FOREIGN VOLUME, and CREATE ICEBERG TABLE. The older CREATE FOREIGN TABLE ... OPTIONS (format 'iceberg', ...) syntax is no longer the recommended path. See Create and Manage Apache Iceberg Tables.

Load data from HDFS

You can load data from HDFS into SynxDB. The following sections explain how to load data from an HDFS cluster without authentication and how to load data from an HDFS cluster with Kerberos authentication. SynxDB also supports loading data from an HDFS HA (High Availability) cluster, which is also explained below.

Load HDFS data without authentication

Load data from HDFS in the simple mode, which is the basic HDFS mode without using complex security authentication. For details, see the Hadoop documentation: Hadoop in Secure Mode. The steps are as follows:

  1. Create an external table wrapper FOREIGN DATA WRAPPER. Note that there are no options in the SQL statement below, and you need to execute the statement exactly as provided.

    CREATE FOREIGN DATA WRAPPER datalake_fdw
HANDLER datalake_fdw_handler
VALIDATOR datalake_fdw_validator
OPTIONS ( mpp_execute 'all segments' );

  2. Create an external server. In this step, you can create an external server for a single-node HDFS or for HA (High Availability) HDFS.

    • Create an external server foreign_server for a single-node HDFS:

    CREATE SERVER foreign_server FOREIGN DATA WRAPPER datalake_fdw
OPTIONS (
       protocol 'hdfs',
       hdfs_namenodes '192.168.178.95',
       hdfs_port '9000',
       hdfs_auth_method 'simple',
       hadoop_rpc_protection 'authentication');

    The options in the above SQL statement are explained as follows:

    Option name

    Description

    Details

    protocol

    Specifies the Hadoop platform.

    • Required: Must be set

    • Setting: Fixed as hdfs, which means Hadoop platform, cannot be changed.

    • Default value: hdfs

    hdfs_namenodes

    Specifies the namenode host for accessing HDFS.

    • Required: Must be set

    • Example: For example, hdfs_namenodes '192.168.178.95:9000'

    hdfs_auth_method

    Specifies the authentication mode for accessing HDFS.

    • Required: Must be set

    • Options:

      • simple: Uses Simple authentication to access HDFS.

      • kerberos: Uses Kerberos authentication to access HDFS.

    • Note: To access in Simple mode, set the value to simple, for example, hdfs_auth_method 'simple'.

    hadoop_rpc_protection

    Configures the authentication mechanism for setting up a SASL connection.

    • Required: Must be set

    • Options: Three values are available: authentication, integrity, and privacy.

    • Note: This option must match the hadoop.rpc.protection setting in the HDFS configuration file core-site.xml. For more details, see the Hadoop documentation Explanation of core-site.xml.

    data_transfer_protection

    Sets the quality of protection for data transfer between the client and the DataNodes.

    • Required: Optional

    • Options: Three values are available: authentication, integrity, and privacy. Any other value is rejected when you create the server.

    • Note: This option must match the dfs.data.transfer.protection setting in the HDFS configuration file hdfs-site.xml. If you leave it unset, the client negotiates the level with the DataNode. Set it explicitly for clusters that enforce a specific level, where negotiation alone leaves the connection unable to read data.

    • Note: The option only takes effect when hdfs_auth_method is kerberos; under simple authentication it is ignored.

    • Note: Do not confuse this option with data_transfer_protocol, which takes true or false and only switches SASL data transfer on or off. Any value other than true turns it off without raising an error.

    To load an HDFS HA cluster, you can create an external server using the following template:

    CREATE SERVER foreign_server
        FOREIGN DATA WRAPPER datalake_fdw
        OPTIONS (
        protocol 'hdfs',
        hdfs_namenodes 'mycluster',
        hdfs_auth_method 'simple',
        hadoop_rpc_protection 'authentication',
        is_ha_supported 'true',
        dfs_nameservices 'mycluster',
        dfs_ha_namenodes 'nn1,nn2,nn3',
        dfs_namenode_rpc_address '192.168.178.95:9000,192.168.178.160:9000,192.168.178.186:9000',
        dfs_client_failover_proxy_provider 'org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider');

    In the above SQL statement, protocol, hdfs_namenodes, hdfs_auth_method, and hadoop_rpc_protection are the same as in the single-node example. The HA-specific options are explained as follows:

    Option name

    Description

    Details

    is_ha_supported

    Specifies whether to access the HDFS HA service (high availability).

    • Required: Must be set

    • Setting: Set to true.

    • Default value: /

    dfs_nameservices

    When is_ha_supported is true, specify the name of the HDFS HA service to access.

    • Required: If using an HDFS HA cluster, must be set.

    • Matches the dfs.ha.namenodes.mycluster item in the HDFS config file hdfs-site.xml.

    • Note: For example, if dfs.ha.namenodes.mycluster is cluster, set this option as dfs_nameservices 'mycluster'.

    dfs_ha_namenodes

    When is_ha_supported is true, specify the accessible nodes for HDFS HA.

    • Required: If using an HDFS HA cluster, must be set.

    • Setting: Matches the value of the dfs.ha.namenodes.mycluster item in the HDFS config file hdfs-site.xml.

    • Note: For example, dfs_ha_namenodes 'nn1,nn2,nn3'.

    dfs_namenode_rpc_address

    When is_ha_supported is true, specifies the IP addresses of the high availability nodes in HDFS HA.

    • Required: If using an HDFS HA cluster, must be set.

    • Setting: Refer to the dfs.ha_namenodes configuration in the HDFS hdfs-site.xml file. The node address matches the namenode address in the configuration.

    • Note: For example, if dfs.ha.namenodes.mycluster has three namenodes named nn1, nn2, nn3, find their addresses from the HDFS configuration file and enter them into this field.

      dfs_namenode_rpc_address '192.168.178.95:9000,192.168.178.160:9000,192.168.178.186:9000'

    dfs_client_failover_proxy

    Specifies whether HDFS HA has failover enabled.

    • Required: If using an HDFS HA cluster, must be set.

    • Setting: Set to the default value: dfs_client_failover_proxy_provider 'org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider'.

    • Default value: /

    Note

    When Kerberos authentication is enabled, you need to renew the Kerberos ticket periodically. If the ticket is not renewed, database performance will gradually degrade until the database eventually crashes and becomes unavailable. Add a renewal command to an operating-system scheduler (such as crontab) or to your orchestration platform, for example:

    kinit -kt /yourpath/greenplum.keytab bdms_greerplum/dev@BDMS.COM

    In the command above, /yourpath/greenplum.keytab is the path to your keytab file. The principal bdms_greerplum/dev@BDMS.COM is an example, where bdms_greerplum is the user, dev is the instance, and BDMS.COM is the REALM. Replace these with your actual values.

    It is recommended to add a renewal script to monitoring to ensure that the ticket remains valid. For more examples of ticket-renewal configuration, see the Greenplum documentation: <https://techdocs.broadcom.com/us/en/vmware-tanzu/data-solutions/tanzu-greenplum-platform-extension-framework/6-7/gp-pxf/pxf_kerbhdfs.html>.

  3. Create a user mapping.

    CREATE USER MAPPING FOR gpadmin SERVER foreign_server
OPTIONS (user 'gpadmin');

    In the above statement, the user option specifies the specific user for foreign_server and must be set.

  4. Create the foreign table example. After creating it, the data from object storage is already loaded into SynxDB, and you can query this table.

    CREATE FOREIGN TABLE example(
a text,
b text
)
SERVER foreign_server
OPTIONS (filePath '/test/parquet/', compression 'none', enableCache 'false', format 'parquet');

    The options in the above SQL statement are explained as follows:

    Option name

    Description

    Details

    filePath

    Sets the specific path of the target foreign table.

    • Required: Must be set

    • Setting: The path format should be /hdfs_prefix.

      Example: If the hdfs_prefix to be accessed is named test-example, the access path is /test-example/. If there are multiple files in the path (for example, 0000_0, 0001_1, 0002_2), you can set the filePath of 0000_0 file as filePath '/test-example/0000_0'. To access all the files in /test-example/, you can set the filePath as filePath '/test_example/'.

    • Note: filePath should follow the /hdfs_prefix/ format. Incorrect formats might lead to errors, such as:

      • filePath 'test/test/orc_file_folder/'

      • filePath '/test/test/orc_file_folder/0000_0'

    compression

    Sets the compression format for writing. Currently supports gzip, zstd, lz4 formats.

    • Required: Optional

    • Setting:

      • none: Supports CSV, ORC, TEXT, PARQUET formats.

      • gzip: Supports CSV, TEXT, PARQUET formats.

      • zstd: Supports PARQUET format.

      • lz4: Supports PARQUET format.

    • Default value: none, which means no compression. Not setting this value also means no compression.

    enableCache

    Specifies whether to use the Gopher cache.

    • Required: Optional

    • Setting:

      • true: Enables Gopher cache.

      • false: Disables Gopher cache.

    • Default: false

    • Note: Deleting a foreign table does not automatically clear the cache. To clear the cache for this table, you need to manually run a specific SQL function, for example:

      select gp_toolkit.__gopher_cache_free_relation_name(text);

    format

    The file format currently supported by FDW.

    • Required: Must be set

    • Setting:

      • csv: Readable, writable

      • text: Readable, writable

      • orc: Readable, writable

      • parquet: Readable, writable

Load HDFS data with Kerberos authentication

If the target HDFS cluster uses Kerberos for authentication, you can follow these steps to load data from HDFS to SynxDB.

  1. Create a foreign data wrapper FOREIGN DATA WRAPPER. Note that there are no options in the following SQL statement, and you need to execute it exactly as provided.

    CREATE FOREIGN DATA WRAPPER datalake_fdw
HANDLER datalake_fdw_handler
VALIDATOR datalake_fdw_validator
OPTIONS ( mpp_execute 'all segments' );

  2. Create an external server. In this step, you can create an external server for a single-node HDFS or for an HA (High Availability) HDFS.

    • Create an external server foreign_server for a single-node HDFS:

    DROP SERVER foreign_server;

CREATE SERVER foreign_server
        FOREIGN DATA WRAPPER datalake_fdw
        OPTIONS (hdfs_namenodes '192.168.3.32',
        hdfs_port '9000',
        protocol 'hdfs',
        hdfs_auth_method 'kerberos',
        krb_principal 'gpadmin/hdw-68212a9a-0@GPADMINCLUSTER2.COM',
        krb_principal_keytab '/home/gpadmin/hadoop.keytab',
        hadoop_rpc_protection 'privacy'
        );

    The options in the above SQL statement are explained as follows:

    Option name

    Description

    Details

    hdfs_namenodes

    Specifies the namenode host for accessing HDFS.

    • Required: Must be set

    • Setting: /

    • Example: For example, hdfs_namenodes '192.168.178.95:9000'

    protocol

    Specifies the Hadoop platform.

    • Required: Must be set

    • Setting: Fixed to hdfs, meaning the Hadoop platform. Cannot be changed.

    • Default: hdfs

    hdfs_auth_method

    Specifies the authentication method for accessing HDFS, which is Kerberos.

    • Required: Must be set

    • Setting: kerberos, to access HDFS using Kerberos authentication.

    krb_principal

    Specifies the primary user set in the HDFS keytab.

    • Required: Must be set

    • Setting: Must match the user information in the keytab. You need to check the relevant user information and set this value accordingly.

    krb_principal_keytab

    Specifies the path to the HDFS keytab.

    • Required: Must be set

    • Setting: The value must match the actual path of the keytab in HDFS.

    hadoop_rpc_protection

    Configures the authentication mechanism for establishing a SASL connection.

    • Required: Must be set

    • Setting: Three options are available: authentication, integrity, and privacy.

    • Note: This option must match the hadoop.rpc.protection setting in the HDFS configuration file core-site.xml. For more details, see the Hadoop core-site.xml documentation.

    To load an HDFS HA cluster, you can use the following template to create an external server:

    CREATE SERVER foreign_server
        FOREIGN DATA WRAPPER datalake_fdw
        OPTIONS (hdfs_namenodes 'mycluster',
        protocol 'hdfs',
        hdfs_auth_method 'kerberos',
        krb_principal 'gpadmin/hdw-68212a9a-0@GPADMINCLUSTER2.COM',
        krb_principal_keytab '/home/gpadmin/hadoop.keytab',
        hadoop_rpc_protection 'privacy',
        is_ha_supported 'true',
        dfs_nameservices 'mycluster',
        dfs_ha_namenodes 'nn1,nn2,nn3',
        dfs_namenode_rpc_address '192.168.178.95:9000,192.168.178.160:9000,192.168.178.186:9000',
        dfs_client_failover_proxy_provider 'org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider'
        );

    In the above SQL statement, the explanations for hdfs_namenodes, protocol, hdfs_auth_method, krb_principal, krb_principal_keytab, and hadoop_rpc_protection are the same as for the single-node example. The HA-specific options are explained as follows:

    Option name

    Description

    Details

    is_ha_supported

    Specifies whether to access the HDFS HA service, which means high availability.

    • Required: Must be set

    • Setting: Set to true.

    dfs_nameservices

    When is_ha_supported is true, specifies the name of the HDFS HA service to access.

    • Required: Must be set if using an HDFS HA cluster.

    • Setting: Should match the dfs.ha.namenodes.mycluster item in the HDFS configuration file hdfs-site.xml.

    • Note: For example, if dfs.ha.namenodes.mycluster is cluster, set this parameter to dfs_nameservices 'mycluster'.

    dfs_ha_namenodes

    When is_ha_supported is true, specifies the accessible nodes for HDFS HA.

    • Required: Must be set if using an HDFS HA cluster.

    • Setting: Should match the value of the dfs.ha.namenodes.mycluster item in the HDFS configuration file hdfs-site.xml.

    • Note: For example, dfs_ha_namenodes 'nn1,nn2,nn3'

    dfs_namenode_rpc_address

    When is_ha_supported is true, specifies the IP addresses of the high availability nodes in HDFS HA.

    • Required: Must be set if using an HDFS HA cluster.

    • Setting: Refer to the dfs.ha_namenodes configuration in the HDFS hdfs-site.xml file. The node address should match the namenode address in the configuration.

    • Example: If dfs.ha.namenodes.mycluster has three namenodes nn1, nn2, nn3, you can find the addresses like dfs.namenode.rpc-address.mycluster.nn1 in the HDFS config and fill them in.

      dfs_namenode_rpc_address '192.168.178.95:9000,192.168.178.160:9000,192.168.178.186:9000'

    dfs_client_failover_proxy_provider

    Specifies whether HDFS HA has failover enabled.

    • Required: Must be set if using an HDFS HA cluster.

    • Setting: Set to the default value dfs_client_failover_proxy_provider 'org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider'.

  3. Create a user mapping.

    CREATE USER MAPPING FOR gpadmin SERVER foreign_server
OPTIONS (user 'gpadmin');

    In the above statement, the user option specifies the specific user for foreign_server and must be set.

  4. Create the foreign table example. After creating it, the data from object storage is already loaded into SynxDB, and you can query this table.

    CREATE FOREIGN TABLE example(
a text,
b text
)
SERVER foreign_server
OPTIONS (filePath '/test/parquet/', compression 'none', enableCache 'false', format 'parquet');

    The options in the above SQL statement are explained as follows:

    Option name

    Description

    Details

    filePath

    Sets the specific path of the target foreign table.

    • Required: Must be set

    • Setting: The path format should be /hdfs_prefix.

      Example: If the hdfs_prefix to be accessed is named test-example, the access path is /test-example/. If there are multiple files in the path (for example, 0000_0, 0001_1, 0002_2), you can set the filePath of 0000_0 file as filePath '/test-example/0000_0'. To access all the files in /test-example/, you can set the filePath as filePath '/test_example/'.

    • Note: filePath should follow the /hdfs_prefix/ format. Incorrect formats might lead to errors, such as:

      • filePath 'test/test/orc_file_folder/'

      • filePath '/test/test/orc_file_folder/0000_0'

    compression

    Sets the compression format for writing, supports gzip, zstd, lz4 formats.

    • Required: Optional

    • Setting:

      • none: Supports CSV, ORC, TEXT, PARQUET formats.

      • gzip: Supports CSV, TEXT, PARQUET formats.

      • zstd: Supports PARQUET format.

      • lz4: Supports PARQUET format.

    • Default: none, which means no compression. Not setting this value also means no compression.

    enableCache

    Specifies whether to use the Gopher cache.

    • Required: Optional

    • Setting:

      • true: Enables Gopher cache.

      • false: Disables Gopher cache.

    • Default: false

    • Note: Deleting a foreign table does not automatically clear the cache. To clear the cache, you need to manually run an SQL function, for example:

      select gp_toolkit.__gopher_cache_free_relation_name(text);

    format

    The file format currently supported by FDW.

    • Required: Must be set

    • Setting:

      • csv: Readable, writable

      • text: Readable, writable

      • orc: Readable, writable

      • parquet: Readable, writable

Cache foreign table data and metadata

When you query a datalake_fdw foreign table, each segment caches the data files it reads on local disk, so repeated queries over the same data do not go back to the object storage or HDFS. This cache is on by default. To turn it off for a session, set datalake.disable_cache_file to on. The session setting has no effect on a foreign table created with enableCache 'true'.

Stale data after an in-place file replacement

Warning

The cache decides whether a file has changed from its path and its length. After you write a file of the same length but different content to the same path, queries still return the data from before the replacement, and raise no error. On object storage, a change in file length invalidates the cache as expected. On HDFS, the cache stays valid even when the file length changes.

Write updated data to a new file path. When you must replace a file in place, or replace any file on HDFS, clear the cache for that table afterwards:

SELECT gp_toolkit.__gopher_cache_free_relation_name('sales_parquet');

Cache the metadata of ORC and Parquet files

Before reading an ORC or Parquet file, the reader first fetches the metadata at the end of the file: the ORC tail or the Parquet footer. Turn on datalake.enable_file_meta_cache to cache that metadata in Gopher, so that later opens of the same file read it from the cache instead of going to the object storage or HDFS. The metadata cache takes effect only while the data file cache is on:

SET datalake.enable_file_meta_cache = on;

The parameter is off by default and covers only ORC and Parquet foreign tables created through datalake_fdw. Iceberg tables, Hudi tables, and CSV or text foreign tables use a different reader stack, so turning it on leaves their behavior unchanged.

To confirm that the cache is working, raise the log level to debug1 for the session and then run the query:

SET log_min_messages = debug1;

The segment logs then carry one line per file. The first open reports miss and writes the metadata to the cache; later opens report hit:

datalake file-meta-cache: miss for /datalake/sales/part-00000.parquet (hit=0 tombstone=0 type=0)
datalake file-meta-cache: backfill ok (producer=1145849431 type=2 offset=45781035 len=65536)
datalake file-meta-cache: hit for /datalake/sales/part-00000.parquet (hit=1 tombstone=0 type=2)

datalake.file_meta_max_size caps the cached metadata of a single file at 512 kB by default. A file whose metadata exceeds the cap stays out of the cache and is read again on every open, with no effect on query results. ORC metadata grows with the number of stripes and columns, so a wide table can exceed the default. The log reports the actual size:

datalake file-meta-cache: blob 838164 bytes exceeds datalake.file_meta_max_size (512 kB), skipping backfill (producer=1145849431 type=1)

For Parquet, the cached range is always the last 64 kB of the file. The minimum of this parameter is therefore 64 kB.

Note

The Gopher server applies its own cap on a single metadata blob, 1 MB by default. Raising datalake.file_meta_max_size above the server cap does not bring larger files into the cache. Gopher marks those files as uncacheable and stops retrying them.

Site configuration file key names

Instead of repeating the same storage settings in every CREATE SERVER statement, you can keep them in a site configuration file and reference a section of it through the server_name option:

File

Used by

s3.conf

S3 and S3-compatible OSS servers, and Iceberg S3 volumes

gphdfs.conf

HDFS servers, and Iceberg HDFS volumes

gphive.conf

Hive Metastore connections

Place these files in the coordinator’s data directory ($COORDINATOR_DATA_DIRECTORY). SynxDB reads them on each request, so you do not need to restart the cluster after editing one.

The keys inside a section are the SQL option names of the corresponding SERVER or USER MAPPING object, spelled identically. What you write as endpoint '...' in CREATE SERVER you write as endpoint: ... in the file. When a SERVER names a section, the two sources merge per key: an option written in SQL wins, and any key you leave out falls back to the file.

The fallback covers metadata access on the coordinator, so a FOREIGN CATALOG can take url and the rest from a gphive.conf section with only type and server_name in SQL.

An Iceberg FOREIGN VOLUME needs separate treatment: the segments read and write its data files, and a key from a section reaches them only in some combinations. An HDFS volume can carry nothing but server_name for tables on an external catalog (hive, hadoop, or polaris). For every other combination, write the connection options in SQL.

Warning

An S3 volume’s endpoint, access_key_id, and secret_access_key must appear on the SERVER and USER MAPPING. If you leave those keys in s3.conf alone and create an Iceberg table through an external catalog, the gopher service on the coordinator exits. Every later read and write on any Iceberg table then fails to connect to gopher, and gpstop -ar is required to recover.

For the full reach, see Create and Manage Apache Iceberg Tables.

Key name reference

If you maintain site configuration files written for an earlier release, rename their keys as follows.

s3.conf:

Earlier key name

Key name to use

fs.gopher.ufs_type

type

fs.s3a.endpoint

endpoint

fs.s3a.endpoint.region

region

fs.s3a.access.key

access_key_id

fs.s3a.secret.key

secret_access_key

fs.s3a.path.style.access

path_style_access

A section can now also carry bucket_name and username. A section’s bucket_name applies to metadata only: CREATE ICEBERG TABLE still requires it as a SQL option on the volume SERVER, otherwise failing with bucket_name is required for volume server type "s3". When neither source sets it, the bucket is derived from the location URL as before.

gphdfs.conf:

Earlier key name

Key name to use

hdfs_namenode_host

hdfs_namenodes

hdfs_namenode_port

hdfs_port

dfs.nameservices

dfs_nameservices

dfs.ha.namenodes

dfs_ha_namenodes

dfs.namenode.rpc-address

dfs_namenode_rpc_address

dfs.client.failover.proxy.provider.<nameservice>

dfs_client_failover_proxy_provider

Other dfs.-prefixed keys

The underscore spelling of the same key

hdfs_namenodes accepts a host:port form, in which case the spliced port takes precedence over hdfs_port. Outside HA mode, the port defaults to 8020. A section can now also carry dfs_client_use_datanode_hostname and username.

gphive.conf:

Earlier key name

Key name to use

uris

url

gphive.conf sections can now also carry username.

Error message for outdated key names

Iceberg catalogs and volumes reject the earlier key names. The error names the replacement for every offending key in the section, so you can correct the file in one pass:

configuration file gphdfs.conf, server "my_hdfs_site": legacy key(s) no longer
supported: "hdfs_namenode_host" (use "hdfs_namenodes"), "hdfs_namenode_port"
(use "hdfs_port"), "dfs.nameservices" (use "dfs_nameservices"); keys must match
the SQL OPTION names

External tables created with datalake_fdw and tables synchronized by the Hive Connector continue to accept both spellings, so a file shared between an Iceberg volume and existing external tables keeps working once you rename its keys. If a section carries both spellings of the same setting, the new key wins.