Upgrade with gpupgrade

gpupgrade upgrades a SynxDB cluster between two versions that have different catalog versions. It initializes a new cluster from the target version, runs pg_upgrade on the coordinator and on all segments, and migrates the data of the source cluster into the new one, without a full data export and reimport. To decide whether you need gpupgrade, see Upgrade the Cluster.

This document describes the upgrade in copy mode. Copy mode upgrades a copy of the source cluster data and leaves the data files of the source cluster unchanged. For the limitations of link mode, see Known limitations.

Before you begin

The preparation has two parts. First complete the checks and settings in the environment of the source cluster. Then clear the environment variables and prepare the shell that runs gpupgrade.

Prepare in the source cluster environment

  • Contact Synx Data Labs technical support to confirm that gpupgrade supports the upgrade from your source version to the target version.

  • Ensure the current user can connect to all cluster hosts via SSH without a password prompt.

  • Ensure sufficient disk space. The file systems that hold the coordinator and segment data directories, including user-defined tablespaces, need at least 60% free space. gpupgrade checks this ratio during initialization, and the configuration parameter disk_free_ratio sets the threshold.

  • Install the target version of SynxDB on all cluster hosts. The source and target versions share the same RPM package name, so you must install with rpm -ivh to keep both versions installed side by side:

    rpm -ivh <target_version_rpm>
rpm -q <package_name>

    The output of rpm -q must list both the source and the target versions.

    Warning

    Do not install the target version with yum install, dnf install, or rpm -U. These commands treat the target version as an update of the source version and remove the source version after the installation. The removal deletes the executables of the source cluster, so the source cluster cannot start again after it stops.

    Warning

    A deployment usually creates a convenience symbolic link next to the installation directories, without a version number, that points at the version in use. If this link already points at the source version, installing the target version leaves a symbolic link to the target version inside the source installation directory, while the convenience link itself still points at the source version.

    Command-line utilities of the source cluster, such as gpinitsystem, resolve an executable to the last match inside the installation directory. While the stray link exists, these utilities silently use the target version binaries.

    After you install the target version, check for the stray link on every host and remove it:

    ls -l <convenience_link>
ls -l <source_install_dir>
rm -f <source_install_dir>/<target_install_dir_name>

    To confirm the removal, run find $GPHOME -follow -name initdb | tail -1 in the source version environment. The result must be under the source installation directory.

  • Back up postgresql.conf and pg_hba.conf from the coordinator data directory and from each segment data directory. The upgrade replaces these files with the defaults of the target version, so you must restore them manually afterward. You can also run gpconfig -s <parameter_name> to record the value of each parameter you have changed.

  • Check each database for directory tables. gpupgrade does not carry directory tables over to the target cluster. For details, see Known limitations.

  • Turn off GopherMeta registration, then restart the cluster:

    gpconfig -c pg_gophermeta.register_gophermeta -v false
gpstop -ar

    In certain time windows, the GopherMeta process might not respond to a shutdown request. The shutdown then times out, and the upgrade substep that issued it fails. Every upgrade stage starts and stops database instances, so any stage can hit this problem.

    With registration turned off, the source cluster starts no GopherMeta process, which prevents the problem when the finalize stage upgrades the mirrors. The finalize stage cannot be reverted, so you must complete this setting before the upgrade.

    The setting applies only to the source cluster. It has no effect on the target cluster that the initialization creates. The initialize and execute stages can still hit the problem when they start and stop target cluster instances. To handle it, see Revert the upgrade. After the upgrade, the parameter on the target cluster has its default value of on, so you do not need to change it back.

Prepare the shell for gpupgrade

  1. On all cluster hosts, remove the source command for the environment script and environment variable declarations such as COORDINATOR_DATA_DIRECTORY from ~/.bashrc and ~/.bash_profile. The environment script is named cloudberry-env.sh, and greenplum_path.sh is a symbolic link to it.

    gpupgrade initialize logs in to each host through SSH and checks PATH and LD_LIBRARY_PATH of the login session. If either contains the installation directory of the source or the target version, the check fails with on host <hostname> PATH contains GPHOME.

    Note

    A cluster deployed by following Deploy Multiple Nodes carries these settings by default. You add them back after the upgrade. See Update the convenience link and environment variables.

  2. On the coordinator host, open a new shell and add the bin directory of the target version to PATH temporarily. Run all gpupgrade commands from this shell:

    export PATH=<target_install_dir>/bin:$PATH
which gpupgrade

    The output of which must be under the target installation directory. The setting lasts only for the current shell and does not affect the check in the previous step. Do not write it to ~/.bashrc or ~/.bash_profile. You only need it on the coordinator host, because the coordinator starts the gpupgrade processes on the other hosts through the installation path of the target version.

Configure gpupgrade

Create a configuration file named gpupgrade_config with the following parameters:

# Port of the source cluster coordinator
source_master_port = 5432

# Installation directory of the source cluster. Use the actual path, not the convenience link.
source_gphome = <source_install_dir>

# Installation directory of the target cluster. Use the actual path.
target_gphome = <target_install_dir>

# Upgrade mode: copy or link. The default is copy.
# For the limitations of link mode, see Known limitations.
mode = copy

# Optional. Fraction of disk space that must be free on every host. The default is 0.6.
# disk_free_ratio = 0.6

Perform the upgrade

The upgrade runs in three sequential stages. Before each stage starts, gpupgrade lists the steps it will run and asks for confirmation. Enter y when prompted.

  1. Initialize

    gpupgrade initialize --file gpupgrade_config --verbose

    This stage runs the pre-upgrade checks, including version compatibility, environment variables, and disk space, and then creates the target cluster. The source cluster remains available during this stage.

  2. Execute

    gpupgrade execute --verbose

    This stage stops the source cluster and runs pg_upgrade on the coordinator and all segments. Downtime begins at this stage.

    When the stage completes, the target cluster runs on a temporary port, and the output shows how to connect to it. You can connect and run validation queries at this point, but do not modify data. Otherwise, the target cluster becomes inconsistent with the source cluster.

  3. Finalize

    gpupgrade finalize --verbose

    This stage upgrades the mirrors and the standby, moves the target cluster to the ports and data directories of the source cluster, and removes intermediate files.

    Caution

    After finalization, the upgrade cannot be reverted.

If you rerun an interrupted stage and it fails with connecting to hub on port 7527: context deadline exceeded, the gpupgrade hub or agent processes are no longer running, for example after a host restart. Restart them with the following command, then rerun the interrupted command:

gpupgrade restart-services

Post-upgrade tasks

After finalize completes, do the following tasks to return the cluster to the state it was in before the upgrade.

Restore the database configuration

gpinitsystem initializes the target cluster, so both postgresql.conf and pg_hba.conf contain the default content of the target version. The upgrade silently resets the parameters that you set through gpconfig on the source cluster, and it resets timezone to the default value of the target version.

Set the parameters again from the backup you made before the upgrade, then reload the configuration:

gpconfig -c <parameter_name> -v <parameter_value>
gpstop -u

Restore pg_hba.conf manually from your backup and reload it with gpstop -u. Otherwise, existing clients lose access because the authentication rules they rely on are missing, and they receive the error FATAL: no pg_hba.conf entry for host ....

pg_gophermeta.register_gophermeta, which you turned off before the upgrade, also does not carry over to the new cluster. It returns to its default of on, which is the required state, so leave it as it is.

Confirm the index rebuild

Two groups of indexes might use a storage format that differs from the one in the source version: bitmap indexes, and BRIN indexes on append-optimized tables, both row-oriented and column-oriented. The upgrade rebuilds these indexes automatically on the coordinator and on every segment when needed, so they are ready when the upgrade completes. BRIN indexes on heap tables are unaffected.

Before you restore application access, check a few of these indexes. Run the following query on each user database to list both groups:

SELECT n.nspname AS schema_name,
       ic.relname AS index_name,
       tc.relname AS table_name,
       ia.amname AS index_type,
       i.indisvalid,
       pg_relation_size(ic.oid) AS index_size
FROM pg_index i
    JOIN pg_class ic ON ic.oid = i.indexrelid
    JOIN pg_class tc ON tc.oid = i.indrelid
    JOIN pg_namespace n ON n.oid = ic.relnamespace
    JOIN pg_am ia ON ia.oid = ic.relam
    LEFT JOIN pg_am ta ON ta.oid = tc.relam
WHERE ic.relkind = 'i'
  AND (ia.amname = 'bitmap'
       OR (ia.amname = 'brin' AND ta.amname IN ('ao_row', 'ao_column')));

indisvalid must be t, and index_size must be greater than 0. Then pick one table and compare the result with and without the index. Turn off GPORCA before the comparison:

SET optimizer = off;

-- With the index
SET enable_seqscan = off;
SELECT count(*) FROM <table_name> WHERE <indexed_column> = <value>;
RESET enable_seqscan;

-- Without the index
SET enable_bitmapscan = off;
SET enable_indexscan = off;
SELECT count(*) FROM <table_name> WHERE <indexed_column> = <value>;
RESET enable_bitmapscan;
RESET enable_indexscan;

RESET optimizer;

The two results must match. If an index reports an index_size of 0, or the two results differ, rebuild that index with REINDEX INDEX <index_name>;.

Update extensions

The upgrade does not change the version of an installed extension. During the execute stage, pg_upgrade checks all databases and generates update_extensions.sql for the extensions that need an update. The file lists an ALTER EXTENSION ... UPDATE statement for each affected database. Find the file on the coordinator host and run it:

find ~/gpAdminLogs -name update_extensions.sql
psql -d postgres -f <path_to_update_extensions.sql>

If the file does not exist, no extension needs an update.

Warning

When datalake_fdw is updated from 1.0 to 1.1, the update script omits pg_iceberg_alter_column_type_local(). After the update, the extension reports version 1.1, but the function is still missing, and ALTER TABLE ... ALTER COLUMN TYPE on an Iceberg native table fails:

ERROR:  function pg_catalog.pg_iceberg_alter_column_type_local(integer, unknown, integer, integer) does not exist

Other ALTER TABLE subcommands are unaffected, and a freshly installed database is unaffected as well. Create the function manually in each database where datalake_fdw was updated to restore the operation:

CREATE FUNCTION pg_catalog.pg_iceberg_alter_column_type_local(oid, text, oid, integer)
RETURNS void AS '$libdir/datalake_fdw', 'pg_iceberg_alter_column_type_local' LANGUAGE C;

After datalake_fdw is updated to 1.1, the features that depend on version 1.1 in Create and Manage Apache Iceberg Tables become available.

Restore privileges on the public schema

The upgrade re-creates the public schema and drops the USAGE and CREATE privileges that the PUBLIC role held on it. As a result, regular users cannot create tables in the public schema after the upgrade. They receive the error ERROR: no schema has been selected to create in, which does not mention privileges directly.

To restore these privileges, run the following statement on each user database:

GRANT USAGE, CREATE ON SCHEMA public TO PUBLIC;

If the source cluster used more granular privileges on the public schema, restore each grant individually instead of granting privileges to the PUBLIC role.

Delete the source cluster data directories

In copy mode, the data directories of the source cluster remain on each host after the upgrade, with names that end in .old. The output of finalize shows the naming pattern. After you confirm that the target cluster works, you can delete these directories to reclaim disk space. After the deletion, the source cluster can no longer start.

Revert the upgrade

If a problem occurs, you can revert the upgrade after initialize or execute by running:

gpupgrade revert --verbose

The revert stops and deletes the target cluster, and then restores and starts the source cluster.

Caution

Reverting is not supported after finalize.

If the initialize or execute stage fails because GopherMeta blocks a shutdown, pg_upgrade_server.log of the target cluster instance contains pg_ctl: server does not shut down. The log is in pg_upgrade_output.d/<timestamp>/log/ under the data directory of each target cluster instance, and the target cluster data directories are named like gpseg.<upgrade_id>.<content_id>.

In this case, run gpupgrade revert and then start the upgrade again.

After the revert, pg_gophermeta.register_gophermeta on the source cluster is still off. If you do not plan to upgrade right away, restore it:

gpconfig -c pg_gophermeta.register_gophermeta -v true
gpstop -ar

Known limitations

  • The current release of gpupgrade does not support directory tables and does not carry them over to the target cluster. Before the upgrade, run \dY on each database. If directory tables exist, keep a copy of the files they manage, and re-create the tables after the upgrade.

  • In the current release, link mode applies only to a cluster deployed on a single host. For a cluster that spans multiple hosts, the initialize stage checks the data directories of all segments on the coordinator host. The directories on the other hosts do not exist there, so the check fails with stat "<segment_data_directory>": no such file or directory. Upgrade such a cluster in copy mode. Link mode also has the following limitations:

    • It requires mirrors for all segments and a standby coordinator. On a cluster that lacks either, you cannot revert once execute starts, and the source cluster can no longer start.

    • It requires the source and target data directories to reside on the same file system, because it reuses the data files of the source cluster through hard links. A cluster whose data directories span multiple disk devices must use copy mode.

  • Some SQL statements that were valid in the source version might not execute in the target version. Review your application workloads before the upgrade.