[ovs-dev,v2,15/17] ovsdb-server: Allow user-provided config files.

OVSDB server maintains a temporary file with the current database
configuration for the case it is restarted by a monitor process
after a crash.  On startup the configuration from command line
arguments is stored there in a JSON format, also whenever user
changes the configuration with different UnixCtl commands, those
changes are getting added to the file.  When restarted from the
crash it reads the configuration from the file and continues
with all the necessary remotes and databases.

This change allows it to be an external user-provided file that
OVSDB server will read the configuration from.  The file can be
specified with a --config-file command line argument and it is
mutually exclusive with most other command line arguments that
set up remotes or databases, it is also mutually exclusive with
use of appctl commands that modify same configurations, e.g.
add/remove-db or add/remove-remote.

If the user wants to change the configuration of a running server,
they may change the file and call ovsdb-server/reload appctl.
OVSDB server will open a file, read and parse it, compare the
new configuration with the current one and adjust the running
configuration as needed.  OVSDB server will try to keep existing
databases and connections intact, if the change can be applied
without disrupting the normal operation.

User-provided files are not trustworthy, so extra checks were
added to ensure a correct file format.  If the file cannot be
correctly parsed, e.g. contains invalid JSON, no changes will
be applied and the server will keep using the previous
configuration until the next reload.

If config-file is provided for active-backup databases, permanent
disconnection of one of the backup databases no longer leads to
switching all other databases to 'active'.  Only the disconnected
one will transition, since all of them have their own records in
the configuration file.

With this change, users can run all types of databases within
the same ovsdb-server process at the same time.

Simple configuration may look like this:

 {
    "remotes": {
        "punix:db.sock": {},
        "pssl:6641": {
            "inactivity-probe": 16000,
            "read-only": false,
            "role": "ovn-controller"
        }
    },
    "databases": {
        "conf.db": {},
        "sb.db": {
            "service-model": "active-backup",
            "backup": true,
            "source": {
                "tcp:127.0.0.1:6644": null
            }
        },
        "OVN_Northbound": {
            "service-model": "relay",
            "source": {
                "ssl:[fe:::1]:6642,ssl:[fe:::2]:6642": {
                    "max-backoff": 8000,
                    "inactivity-probe": 10000
                }
            }
        }
    }
 }

Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
---
 Documentation/ref/ovsdb.7.rst        |  86 +++++-
 Documentation/topics/ovsdb-relay.rst |  19 ++
 NEWS                                 |   4 +
 ovsdb/ovsdb-server.1.in              |  96 ++++++-
 ovsdb/ovsdb-server.c                 | 394 ++++++++++++++++++++++-----
 5 files changed, 521 insertions(+), 78 deletions(-)

On 1/9/24 23:49, Ilya Maximets wrote:
> OVSDB server maintains a temporary file with the current database
> configuration for the case it is restarted by a monitor process
> after a crash.  On startup the configuration from command line
> arguments is stored there in a JSON format, also whenever user
> changes the configuration with different UnixCtl commands, those
> changes are getting added to the file.  When restarted from the
> crash it reads the configuration from the file and continues
> with all the necessary remotes and databases.
> 
> This change allows it to be an external user-provided file that
> OVSDB server will read the configuration from.  The file can be
> specified with a --config-file command line argument and it is
> mutually exclusive with most other command line arguments that
> set up remotes or databases, it is also mutually exclusive with
> use of appctl commands that modify same configurations, e.g.
> add/remove-db or add/remove-remote.
> 
> If the user wants to change the configuration of a running server,
> they may change the file and call ovsdb-server/reload appctl.
> OVSDB server will open a file, read and parse it, compare the
> new configuration with the current one and adjust the running
> configuration as needed.  OVSDB server will try to keep existing
> databases and connections intact, if the change can be applied
> without disrupting the normal operation.
> 
> User-provided files are not trustworthy, so extra checks were
> added to ensure a correct file format.  If the file cannot be
> correctly parsed, e.g. contains invalid JSON, no changes will
> be applied and the server will keep using the previous
> configuration until the next reload.
> 
> If config-file is provided for active-backup databases, permanent
> disconnection of one of the backup databases no longer leads to
> switching all other databases to 'active'.  Only the disconnected
> one will transition, since all of them have their own records in
> the configuration file.
> 
> With this change, users can run all types of databases within
> the same ovsdb-server process at the same time.
> 
> Simple configuration may look like this:
> 
>  {
>     "remotes": {
>         "punix:db.sock": {},
>         "pssl:6641": {
>             "inactivity-probe": 16000,
>             "read-only": false,
>             "role": "ovn-controller"
>         }
>     },
>     "databases": {
>         "conf.db": {},
>         "sb.db": {
>             "service-model": "active-backup",
>             "backup": true,
>             "source": {
>                 "tcp:127.0.0.1:6644": null
>             }
>         },
>         "OVN_Northbound": {
>             "service-model": "relay",
>             "source": {
>                 "ssl:[fe:::1]:6642,ssl:[fe:::2]:6642": {
>                     "max-backoff": 8000,
>                     "inactivity-probe": 10000
>                 }
>             }
>         }
>     }
>  }
> 
> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
> ---

I left a bunch of minor comments in the man pages below; with that:

Acked-by: Dumitru Ceara <dceara@redhat.com>

>  Documentation/ref/ovsdb.7.rst        |  86 +++++-
>  Documentation/topics/ovsdb-relay.rst |  19 ++
>  NEWS                                 |   4 +
>  ovsdb/ovsdb-server.1.in              |  96 ++++++-
>  ovsdb/ovsdb-server.c                 | 394 ++++++++++++++++++++++-----
>  5 files changed, 521 insertions(+), 78 deletions(-)
> 
> diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst
> index 84b153d24..93df00c89 100644
> --- a/Documentation/ref/ovsdb.7.rst
> +++ b/Documentation/ref/ovsdb.7.rst
> @@ -155,6 +155,22 @@ standalone database, configure the server to listen on a "connection method"
>  that the client can reach, then point the client to that connection method.
>  See `Connection Methods`_ below for information about connection methods.
>  
> +Open vSwitch 3.3 introduced support for configuration files via
> +``--config-file`` command line option.  The configuration file for a server
> +with a **standalone** database may look like this::
> +
> +  {
> +      "remotes": { "<connection method>": {} },
> +      "databases": { "<database file>": {} }
> +  }
> +
> +``ovsdb-server`` will infer the service model from the database file itself.
> +However, if additional verification is desired, an optional
> +``"service-model": "standalone"`` can be provided for a database file inside

s/for a database file/for the database file/

> +the inner curly braces.  If the specified ``service-model`` will not match the
> +content of the database file, ``ovsdb-server`` will refuse to open this
> +database.
> +
>  Active-Backup Database Service Model
>  ------------------------------------
>  
> @@ -177,10 +193,36 @@ database file from the active server.  Then use
>  connects to the active server.  At that point, the backup server will fetch a
>  copy of the active database and keep it up-to-date until it is killed.
>  
> +Open vSwitch 3.3 introduced support for configuration files via
> +``--config-file`` command line option.  The configuration file for a backup
> +server in this case may look like this::
> +
> +  {
> +      "remotes": { "<connection method>": {} },
> +      "databases": {
> +          "<database file>": {
> +              "service-model": "active-backup",
> +              "backup": true,
> +              "source": {
> +                  "<active>": {
> +                      "inactivity-probe": <integer>,
> +                      "max-backoff": <integer>
> +                  }
> +              }
> +          }
> +      }
> +  }
> +
> +All the fields in the ``"<database file>"`` description above are required.
> +Options for the ``"<active>"`` connection method (``"inactivity-probe"``, etc.)
> +can be omitted.
> +
>  When the active server in an active-backup server pair fails, an administrator
>  can switch the backup server to an active role with the ``ovs-appctl`` command
>  ``ovsdb-server/disconnect-active-ovsdb-server``.  Clients then have read/write
> -access to the now-active server.  Of course, administrators are slow to respond
> +access to the now-active server.  When the ``--config-file`` is in use, the
> +same can be achieved by changing the ``"backup"`` value in the file and running
> +``ovsdb-server/reload`` command.  Of course, administrators are slow to respond
>  compared to software, so in practice external management software detects the
>  active server's failure and changes the backup server's role.  For example, the
>  "Integration Guide for Centralized Control" in the OVN documentation describes
> @@ -236,6 +278,22 @@ To set up a clustered database, first initialize it on a single node by running
>  arguments, the ``create-cluster`` command can create an empty database or copy
>  a standalone database's contents into the new database.
>  
> +Open vSwitch 3.3 introduced support for configuration files via
> +``--config-file`` command line option.  The configuration file for a server
> +with a **clustered** database may look like this::
> +
> +  {
> +      "remotes": { "<connection method>": {} },
> +      "databases": { "<database file>": {} }
> +  }
> +
> +``ovsdb-server`` will infer the service model from the database file itself.
> +However, if additional verification is desired, an optional
> +``"service-model": "clustered"`` can be provided for a database file inside
> +the inner curly braces.  If the specified ``service-model`` will not match the
> +content of the database file, ``ovsdb-server`` will refuse to open this
> +database.
> +
>  To configure a client to use a clustered database, first configure all of the
>  servers to listen on a connection method that the client can reach, then point
>  the client to all of the servers' connection methods, comma-separated.  See
> @@ -505,6 +563,29 @@ server.  ``<relay source>`` could contain a comma-separated list of connection
>  methods, e.g. to connect to any server of the clustered database.
>  Multiple relay servers could be started for the same relay source.
>  
> +Open vSwitch 3.3 introduced support for configuration files via
> +``--config-file`` command line option.  The configuration file for a relay
> +database server in this case may look like this::
> +
> +  {
> +      "remotes": { "<connection method>": {} },
> +      "databases": {
> +          "<DB_NAME>": {
> +              "service-model": "relay",
> +              "source": {
> +                  "<relay source>": {
> +                      "inactivity-probe": <integer>,
> +                      "max-backoff": <integer>
> +                  }
> +              }
> +          }
> +      }
> +  }
> +
> +Both the ``"service-model"`` and the ``"source"`` are required.  Options for
> +the ``"<relay source>"`` connection method (``"inactivity-probe"``, etc.)
> +can be omitted.
> +
>  Since the way relays handle read and write transactions is very similar
>  to the clustered model where "cluster" means "set of relay servers connected
>  to the same relay source", "follower" means "relay server" and the "leader"
> @@ -629,7 +710,8 @@ Creating a Database
>  
>  Creating and starting up the service for a new database was covered
>  separately for each database service model in the `Service
> -Models`_ section, above.
> +Models`_ section, above.  Single ``ovsdb-server`` process may serve

s/Single /A single /

> +any number of databases with different service models at the same time.
>  
>  Backing Up and Restoring a Database
>  -----------------------------------
> diff --git a/Documentation/topics/ovsdb-relay.rst b/Documentation/topics/ovsdb-relay.rst
> index 50a3c6d07..63ea5329b 100644
> --- a/Documentation/topics/ovsdb-relay.rst
> +++ b/Documentation/topics/ovsdb-relay.rst
> @@ -105,6 +105,25 @@ started like this::
>    $ ...
>    $ ovsdb-server --remote=ptcp:6642:172.16.0.K relay:OVN_Southbound:$REMOTES
>  
> +Open vSwitch 3.3 introduced support for configuration files via
> +``--config-file`` command line option.  The configuration file for a relay
> +database servers in this case may look like this::
> +

Either "for a relay database server" or "for relay database servers" I
think.

> +  {
> +      "remotes": { "ptcp:6642:172.16.0.X": {} },
> +      "databases": {
> +          "OVN_Southbound": {
> +              "service-model": "relay",
> +              "source": {
> +                  "$REMOTES": {}
> +              }
> +          }
> +      }
> +  }
> +
> +See ``ovsdb-server(1)`` and  ``Relay Service Model`` in ``ovsdb(7)`` for more
> +configuration options.
> +
>  Every relay server could connect to any of the cluster members of their choice,
>  fairness of load distribution is achieved by shuffling remotes.
>  
> diff --git a/NEWS b/NEWS
> index 270ed6673..e075ecbb7 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -6,6 +6,10 @@ Post-v3.2.0
>         from older version is supported but it may trigger more leader elections
>         during the process, and error logs complaining unrecognized fields may
>         be observed on old nodes.
> +     * New command line option --config-file that allows a fine control over
> +       remotes and database configuration, including setting options for
> +       connection methods for relays and active-backup replication.
> +       For more details see ovsdb-server(1) and ovsdb(7).
>     - OpenFlow:
>       * NXT_CT_FLUSH extension is updated to support flushing connections
>         based on mark and labels.  'ct-flush' command of ovs-ofctl updated
> diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in
> index da7a6fd5d..91d6a6e0f 100644
> --- a/ovsdb/ovsdb-server.1.in
> +++ b/ovsdb/ovsdb-server.1.in
> @@ -12,6 +12,7 @@ ovsdb\-server \- Open vSwitch database server
>  [\fIdatabase\fR]\&...
>  [\fIrelay:schema_name:remote\fR]\&...
>  [\fB\-\-remote=\fIremote\fR]\&...
> +[\fB\-\-config\-file=\fIfile\fR]
>  [\fB\-\-run=\fIcommand\fR]
>  .so lib/daemon-syn.man
>  .so lib/service-syn.man
> @@ -44,6 +45,11 @@ If none of database files or relay databases is specified, the default is
>  initialized using, for example, \fBovsdb\-tool\fR's \fBcreate\fR,
>  \fBcreate\-cluster\fR, or \fBjoin\-cluster\fR command.
>  .PP
> +All types of databases can alternatively be added using a configuration
> +file provided via \fB\-\-config\-file\fR option.  This option is mutually
> +exclusive with specifying \fIdatabase\fR on the command line.  For a detailed
> +description of a configuration file format see \fBovsdb\fR(7).

s/of a configuration/of the configuration/

> +.PP
>  This OVSDB implementation supports standalone, active-backup, relay and
>  clustered database service models, as well as database replication.
>  See the Service Models section of \fBovsdb\fR(7) for more information.
> @@ -105,6 +111,74 @@ It is an error for \fIcolumn\fR to have another type.
>  .IP
>  To connect or listen on multiple connection methods, use multiple
>  \fB\-\-remote\fR options.
> +.IP
> +Alternatively, remotes can be specified in a "remotes" section of a

'in the "remotes" section of the configuration file'

> +configuration file, if provided using \fB\-\-config\-file\fR option.
> +\fB\-\-config\-file\fR and \fB\-\-remote\fR options are mutually
> +exclusive.
> +.
> +.IP "\fB\-\-config-file=\fIfile\fR"
> +Specifies a configuration file for \fBovsdb\-server\fR.  This \fIfile\fR
> +can contain connection methods and databases used by the server.
> +The \fIfile\fR contains a JSON object with two main elements:
> +.RS
> +.IP "\fBremotes\fR"
> +JSON object that contains a set of connection methods in a following format:

s/in a following format/in the following format/

> +"\fItarget\fR": { "\fIoption\fR": \fIvalue\fR, ... }.  Where \fItarget\fR
> +is in the same format as \fIremote\fR in \fB\-\-remote\fR option.
> +\fIoption\fR can be \fBmax-backoff\fR (integer), \fBinactivity-probe\fR
> +(integer), \fBread-only\fR (boolean), \fBrole\fR (string) or \fBdscp\fR
> +(integer) with their allowed \fIvalue\fRs respectively.  The meaning of these
> +\fIoption\fRs is the same as in configuration of \fIremote\fR via a database
> +row with \fB\-\-remote\fR option.
> +.IP "\fBdatabases\fR"
> +JSON object that describes databases that should be added to the
> +\fBovsdb\-server\fR in the following format: "\fIname\fR":{ "\fIoption\fR":
> +\fIvalue\fR, ... }.  Where \fIname\fR is either a file name of a previously
> +created and initialized database or a schema name in case of relay
> +databases.  Available \fIoption\fRs are:
> +.RS
> +.IP "\fBservice-model\fR (string)"
> +Describes the service model of this database.  One of: \fBstandalone\fR,
> +\fBclustered\fR, \fBactive-backup\fR or \fBrelay\fR.  This option is
> +required for all types, except for standalone and clustered.  For these
> +databases the service model will be inferred from the file, if not
> +specified explicitly.  \fBovsdb-server\fR will refuse to add a database
> +if the specified \fBservice-model\fR doesn't match with the provided file.
> +.IP "\fBsource\fR (JSON object; active-backup or relay)"
> +Describes the connection method to the active database or to the relay
> +source.  It is a JSON object with exactly one element in the same format
> +as elements of "\fBremotes\fR", except that \fBread-only\fR and \fBrole\fR
> +options are not applicable.  E.g. \fB"source": { "unix:db.sock": {
> +"inactivity-probe": 10000, "max-backoff": 8000 } }\fR
> +.IP "\fBbackup\fR (boolean; active-backup only)"
> +If set to \fBtrue\fR, \fBovsdb-server\fR will use this database as a
> +backup for the specified \fBsource\fR.  Will be served as an active
> +database otherwise.
> +.IP "\fBexclude-tables\fR (JSON array of strings; active-backup only)"
> +List of table names that should be excluded from replication in backup mode,
> +e.g. \fB"exclude-tables": [ "Table_One", "Table_Two" ]\fR.
> +.RE
> +.RE
> +.IP
> +Content of the most basic configuration file may look like this:
> +\fB{ "remotes": { "pssl:6640": {} }, "databases": { "conf.db": {} } }\fR
> +.IP
> +Examples of configuration files for different service models can be
> +found in in \fBovsdb\fR(7).
> +.IP
> +\fB\-\-config-file\fR option is mutually exclusive with \fB\-\-remote\fR

s/with/ with the/

> +as well as with specifying \fIdatabase\fR on a command line.  It is also
> +mutually exclusive with all the \fBActive-Backup Options\fR and all the
> +\fBRUNTIME MANAGEMENT COMMANDS\fR that can change the configuration of
> +the server in conflict with the content of the file, i.e. all the commands
> +that manipulate with remotes and databases.  Read-only commands can still
> +be used.
> +.IP
> +In case of changes in the \fIfile\fR, users should run

s/should run/should run the/

> +\fBovsdb-server/reload\fR command with \fBovs-appctl\fR(8) in order for
> +changes to take effect.
> +.RE
>  .
>  .IP "\fB\-\-run=\fIcommand\fR]"
>  Ordinarily \fBovsdb\-server\fR runs forever, or until it is told to
> @@ -178,6 +252,8 @@ allow the syncing options to be specified using command line options,
>  yet start the server, as the default, active server.  To switch the
>  running server to backup mode, use \fBovs-appctl(1)\fR to execute the
>  \fBovsdb\-server/connect\-active\-ovsdb\-server\fR command.
> +.PP
> +These options are mutually exclusive with \fB\-\-config\-file\fR.

s/with /with the/

There are 5 more occurences below.

>  .SS "Public Key Infrastructure Options"
>  The options described below for configuring the SSL public key
>  infrastructure accept a special syntax for obtaining their
> @@ -230,6 +306,8 @@ clients.
>  Adds a remote, as if \fB\-\-remote=\fIremote\fR had been specified on
>  the \fBovsdb\-server\fR command line.  (If \fIremote\fR is already a
>  remote, this command succeeds without changing the configuration.)
> +.IP
> +Mutually exclusive with \fB\-\-config\-file\fR option.
>  .
>  .IP "\fBovsdb\-server/remove\-remote \fIremote\fR"
>  Removes the specified \fIremote\fR from the configuration, failing
> @@ -241,6 +319,8 @@ configuring a \fBdb:\fIdb\fB,\fItable\fB,\fIcolumn\fR remote.
>  (You can remove a database source with \fBovsdb\-server/remove\-remote
>  \fBdb:\fIdb\fB,\fItable\fB,\fIcolumn\fR, but not individual
>  remotes found indirectly through the database.)
> +.IP
> +Mutually exclusive with \fB\-\-config\-file\fR option.
>  .
>  .IP "\fBovsdb\-server/list\-remotes"
>  Outputs a list of the currently configured remotes named on
> @@ -254,6 +334,8 @@ Adds the \fIdatabase\fR to the running \fBovsdb\-server\fR.  \fIdatabase\fR
>  could be a database file or a relay description in the following format:
>  \fIrelay:schema_name:remote\fR.  The database file must already have been
>  created and initialized using, for example, \fBovsdb\-tool create\fR.
> +.IP
> +Mutually exclusive with \fB\-\-config\-file\fR option.
>  .
>  .IP "\fBovsdb\-server/remove\-db \fIdatabase\fR"
>  Removes \fIdatabase\fR from the running \fBovsdb\-server\fR.  \fIdatabase\fR
> @@ -268,6 +350,8 @@ Any public key infrastructure options specified through this database
>  (e.g. \fB\-\-private\-key=db:\fIdatabase,\fR... on the command line)
>  will be disabled until another database with the same name is added
>  again (with \fBovsdb\-server/add\-db\fR).
> +.IP
> +Mutually exclusive with \fB\-\-config\-file\fR option.
>  .
>  .IP "\fBovsdb\-server/list\-dbs"
>  Outputs a list of the currently configured databases added either through
> @@ -286,6 +370,9 @@ These commands query and update the role of \fBovsdb\-server\fR within
>  an active-backup pair of servers.  See \fBActive-Backup Options\fR,
>  above, and \fBActive-Backup Database Service Model\fR in
>  \fBovsdb\fR(7) for more information.
> +.PP
> +All \fBActive-Backup Commands\fR that change the state of \fBovsdb\-server\fR
> +are mutually exclusive with \fB\-\-config\-file\fR option.
>  .
>  .IP "\fBovsdb\-server/set\-active\-ovsdb\-server \fIserver"
>  Sets  the active \fIserver\fR from which \fBovsdb\-server\fR connects through
> @@ -324,11 +411,10 @@ Gets  the  tables  that are currently excluded from synchronization.
>  Prints a summary of replication run time information. The \fBstate\fR
>  information is always provided, indicating whether the server is running
>  in the \fIactive\fR or the \fIbackup\fR mode.
> -When running in backup mode, replication connection status, which
> -can be either \fIconnecting\fR, \fIreplicating\fR or \fIerror\fR, are shown.
> -When the connection is in \fIreplicating\fR state, further output shows
> -the list of databases currently replicating, and the tables that are
> -excluded.
> +For all databases with active-backup service model, replication connection
> +status, which can be either \fIconnecting\fR, \fIreplicating\fR or
> +\fIerror\fR, are shown.  When the connection is in \fIreplicating\fR state,
> +further output shows the tables that are currently excluded from replication.
>  .
>  .SS "Cluster Commands"
>  These commands support the \fBovsdb\-server\fR clustered service model.
> diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c
> index 9726a8d72..f10e06ae0 100644
> --- a/ovsdb/ovsdb-server.c
> +++ b/ovsdb/ovsdb-server.c
> @@ -195,6 +195,13 @@ static void add_server_db(struct server_config *);
>  static void remove_db(struct server_config *, struct shash_node *db, char *);
>  static void close_db(struct server_config *, struct db *, char *);
>  
> +static struct ovsdb_error *update_schema(struct ovsdb *,
> +                                         const struct ovsdb_schema *,
> +                                         const struct uuid *txnid,
> +                                         bool conversion_with_no_data,
> +                                         void *aux)
> +    OVS_WARN_UNUSED_RESULT;
> +
>  static void parse_options(int argc, char *argvp[],
>                            struct shash *db_conf, struct shash *remotes,
>                            char **unixctl_pathp, char **run_command,
> @@ -223,7 +230,7 @@ static void save_config__(FILE *config_file, const struct shash *remotes,
>                            const char *sync_from, const char *sync_exclude,
>                            bool is_backup);
>  static void save_config(struct server_config *);
> -static void load_config(FILE *config_file, struct shash *remotes,
> +static bool load_config(FILE *config_file, struct shash *remotes,
>                          struct shash *db_conf, char **sync_from,
>                          char **sync_exclude, bool *is_backup);
>  
> @@ -263,8 +270,9 @@ ovsdb_server_replication_run(struct server_config *config)
>      }
>  
>      /* If one connection is broken, switch all databases to active,
> -     * since they are configured via the same command line / appctl. */
> -    if (!all_alive && *config->is_backup) {
> +     * if they are configured via the command line / appctl and so have
> +     * shared configuration. */
> +    if (!config_file_path && !all_alive && *config->is_backup) {
>          *config->is_backup = false;
>  
>          SHASH_FOR_EACH (node, config->all_dbs) {
> @@ -513,6 +521,196 @@ free_database_configs(struct shash *db_conf)
>      shash_clear(db_conf);
>  }
>  
> +static bool
> +service_model_can_convert(enum service_model a, enum service_model b)
> +{
> +    ovs_assert(a != SM_UNDEFINED);
> +
> +    if (a == b) {
> +        return true;
> +    }
> +
> +    if (b == SM_UNDEFINED) {
> +        return a == SM_STANDALONE || a == SM_CLUSTERED;
> +    }
> +
> +    /* Conversion can happen only between standalone and active-backup. */
> +    return (a == SM_STANDALONE && b == SM_ACTIVE_BACKUP)
> +            || (a == SM_ACTIVE_BACKUP && b == SM_STANDALONE);
> +}
> +
> +static void
> +database_update_config(struct server_config *server_config,
> +                       struct db *db, const struct db_config *new_conf)
> +{
> +    struct db_config *conf = db->config;
> +    enum service_model model = conf->model;
> +
> +    /* Stop replicating when transitioning to active or standalone. */
> +    if (conf->model == SM_ACTIVE_BACKUP && conf->ab.backup
> +        && (new_conf->model == SM_STANDALONE || !new_conf->ab.backup)) {
> +        ovsdb_server_replication_remove_db(db);
> +    }
> +
> +    db_config_destroy(conf);
> +    conf = db->config = db_config_clone(new_conf);
> +
> +    if (conf->model == SM_UNDEFINED) {
> +        /* We're operating on the same file, the model is the same. */
> +        conf->model = model;
> +    }
> +
> +    if (conf->model == SM_RELAY) {
> +        ovsdb_relay_add_db(db->db, conf->source, update_schema, server_config,
> +                           &conf->options->rpc);
> +    }
> +    if (conf->model == SM_ACTIVE_BACKUP && conf->ab.backup) {
> +        const struct uuid *server_uuid;
> +
> +        server_uuid = ovsdb_jsonrpc_server_get_uuid(server_config->jsonrpc);
> +        replication_set_db(db->db, conf->source, conf->ab.sync_exclude,
> +                           server_uuid, &conf->options->rpc);
> +    }
> +}
> +
> +static bool
> +reconfigure_databases(struct server_config *server_config,
> +                      struct shash *db_conf)
> +{
> +    struct db_config *cur_conf, *new_conf;
> +    struct shash_node *node, *conf_node;
> +    bool res = true;
> +    struct db *db;
> +
> +    /* Remove databases that are no longer in the configuration or have
> +     * incompatible configuration.  Update compatible ones. */
> +    SHASH_FOR_EACH_SAFE (node, server_config->all_dbs) {
> +        db = node->data;
> +
> +        if (node->name[0] == '_') {
> +            /* Skip internal databases. */
> +            continue;
> +        }
> +
> +        cur_conf = db->config;
> +        conf_node = shash_find(db_conf, db->filename);
> +        new_conf = conf_node ? conf_node->data : NULL;
> +
> +        if (!new_conf) {
> +            remove_db(server_config, node,
> +                      xasprintf("database %s removed from configuration",
> +                                node->name));
> +            continue;
> +        }
> +        if (!service_model_can_convert(cur_conf->model, new_conf->model)) {
> +            remove_db(server_config, node,
> +                      xasprintf("service model changed for database %s",
> +                                node->name));
> +            continue;
> +        }
> +        database_update_config(server_config, db, new_conf);
> +
> +        db_config_destroy(new_conf);
> +        shash_delete(db_conf, conf_node);
> +    }
> +
> +    /* Create new databases. */
> +    SHASH_FOR_EACH (node, db_conf) {
> +        struct ovsdb_error *error = open_db(server_config,
> +                                            node->name, node->data);
> +        if (error) {
> +            char *s = ovsdb_error_to_string_free(error);
> +
> +            VLOG_WARN("failed to open database '%s': %s", node->name, s);
> +            free(s);
> +            res = false;
> +        }
> +        db_config_destroy(node->data);
> +    }
> +    shash_clear(db_conf);
> +
> +    return res;
> +}
> +
> +static bool
> +reconfigure_ovsdb_server(struct server_config *server_config)
> +{
> +    char *sync_from = NULL, *sync_exclude = NULL;
> +    bool is_backup = false;
> +    struct shash remotes;
> +    struct shash db_conf;
> +    bool res = true;
> +
> +    FILE *file = NULL;
> +
> +    if (config_file_path) {
> +        file = fopen(config_file_path, "r+b");
> +        if (!file) {
> +            VLOG_ERR("failed to open configuration file '%s': %s",
> +                     config_file_path, ovs_strerror(errno));
> +            return false;
> +        } else {
> +            VLOG_INFO("loading configuration from '%s'", config_file_path);
> +        }
> +    } else {
> +        file = server_config->config_tmpfile;
> +    }
> +    ovs_assert(file);
> +
> +    shash_init(&remotes);
> +    shash_init(&db_conf);
> +
> +    if (!load_config(file, &remotes, &db_conf,
> +                     &sync_from, &sync_exclude, &is_backup)) {
> +        if (config_file_path) {
> +            VLOG_WARN("failed to load configuration from %s",
> +                      config_file_path);
> +        } else {
> +            VLOG_FATAL("failed to load configuration from a temporary file");
> +        }
> +        res = false;
> +        goto exit_close;
> +    }
> +
> +    /* Parsing was successful.  Update the server configuration. */
> +    shash_swap(server_config->remotes, &remotes);
> +    free(*server_config->sync_from);
> +    *server_config->sync_from = sync_from;
> +    free(*server_config->sync_exclude);
> +    *server_config->sync_exclude = sync_exclude;
> +    *server_config->is_backup = is_backup;
> +
> +    if (!reconfigure_databases(server_config, &db_conf)) {
> +        VLOG_WARN("failed to configure databases");
> +        res = false;
> +    }
> +
> +    char *error = reconfigure_remotes(server_config->jsonrpc,
> +                                      server_config->all_dbs,
> +                                      server_config->remotes);
> +    if (error) {
> +        VLOG_WARN("failed to configure remotes: %s", error);
> +        res = false;
> +    } else {
> +        error = reconfigure_ssl(server_config->all_dbs);
> +        if (error) {
> +            VLOG_WARN("failed to configure SSL: %s", error);
> +            res = false;
> +        }
> +    }
> +    free(error);
> +
> +exit_close:
> +    if (config_file_path) {
> +        fclose(file);
> +    }
> +    free_remotes(&remotes);
> +    free_database_configs(&db_conf);
> +    shash_destroy(&remotes);
> +    shash_destroy(&db_conf);
> +    return res;
> +}
> +
>  int
>  main(int argc, char *argv[])
>  {
> @@ -527,13 +725,22 @@ main(int argc, char *argv[])
>      struct process *run_process;
>      bool exiting;
>      int retval;
> -    FILE *config_tmpfile;
> -    struct server_config server_config;
> +    FILE *config_tmpfile = NULL;
>      struct shash all_dbs;
>      struct shash_node *node;
>      int replication_probe_interval = REPLICATION_DEFAULT_PROBE_INTERVAL;
>      int relay_source_probe_interval = RELAY_SOURCE_DEFAULT_PROBE_INTERVAL;
>  
> +    struct server_config server_config = {
> +        .remotes = &remotes,
> +        .all_dbs = &all_dbs,
> +        .sync_from = &sync_from,
> +        .sync_exclude = &sync_exclude,
> +        .is_backup = &is_backup,
> +        .replication_probe_interval = &replication_probe_interval,
> +        .relay_source_probe_interval = &relay_source_probe_interval,
> +    };
> +
>      ovs_cmdl_proctitle_init(argc, argv);
>      set_program_name(argv[0]);
>      service_start(&argc, &argv);
> @@ -548,64 +755,39 @@ main(int argc, char *argv[])
>  
>      daemon_become_new_user(false, false);
>  
> -    /* Create and initialize 'config_tmpfile' as a temporary file to hold
> -     * ovsdb-server's most basic configuration, and then save our initial
> -     * configuration to it.  When --monitor is used, this preserves the effects
> -     * of ovs-appctl commands such as ovsdb-server/add-remote (which saves the
> -     * new configuration) across crashes. */
> -    config_tmpfile = tmpfile();
> -    if (!config_tmpfile) {
> -        ovs_fatal(errno, "failed to create temporary file");
> +    if (!config_file_path) {
> +         /* Create and initialize 'config_tmpfile' as a temporary file to hold
> +         * ovsdb-server's most basic configuration, and then save our initial
> +         * configuration to it.  When --monitor is used, this preserves the
> +         * effects of ovs-appctl commands such as ovsdb-server/add-remote
> +         * (which saves the new configuration) across crashes. */
> +        config_tmpfile = tmpfile();
> +        if (!config_tmpfile) {
> +            ovs_fatal(errno, "failed to create temporary file");
> +        }
> +        server_config.config_tmpfile = config_tmpfile;
> +        save_config__(config_tmpfile, &remotes, &db_conf, sync_from,
> +                      sync_exclude, is_backup);
>      }
>  
> -    server_config.remotes = &remotes;
> -    server_config.config_tmpfile = config_tmpfile;
> -
> -    save_config__(config_tmpfile, &remotes, &db_conf, sync_from,
> -                  sync_exclude, is_backup);
>      free_remotes(&remotes);
>      free_database_configs(&db_conf);
>  
>      daemonize_start(false, false);
>  
> -    /* Load the saved config. */
> -    load_config(config_tmpfile, &remotes, &db_conf, &sync_from,
> -                &sync_exclude, &is_backup);
> -
> -    /* Start ovsdb jsonrpc server. When running as a backup server,
> -     * jsonrpc connections are read only. Otherwise, both read
> -     * and write transactions are allowed.  */
> -    jsonrpc = ovsdb_jsonrpc_server_create(is_backup);
> +    perf_counters_init();
>  
> -    shash_init(&all_dbs);
> -    server_config.all_dbs = &all_dbs;
> +    /* Start ovsdb jsonrpc server.  Both read and write transactions are
> +     * allowed by default, individual remotes and databases will be configured
> +     * as read-only, if necessary. */
> +    jsonrpc = ovsdb_jsonrpc_server_create(false);
>      server_config.jsonrpc = jsonrpc;
> -    server_config.sync_from = &sync_from;
> -    server_config.sync_exclude = &sync_exclude;
> -    server_config.is_backup = &is_backup;
> -    server_config.replication_probe_interval = &replication_probe_interval;
> -    server_config.relay_source_probe_interval = &relay_source_probe_interval;
>  
> -    perf_counters_init();
> -
> -    SHASH_FOR_EACH (node, &db_conf) {
> -        struct ovsdb_error *error = open_db(&server_config,
> -                                            node->name, node->data);
> -        if (error) {
> -            char *s = ovsdb_error_to_string_free(error);
> -            ovs_fatal(0, "%s", s);
> -        }
> -        db_config_destroy(node->data);
> -    }
> -    shash_clear(&db_conf);
> +    shash_init(&all_dbs);
>      add_server_db(&server_config);
>  
> -    char *error = reconfigure_remotes(jsonrpc, &all_dbs, &remotes);
> -    if (!error) {
> -        error = reconfigure_ssl(&all_dbs);
> -    }
> -    if (error) {
> -        ovs_fatal(0, "%s", error);
> +    if (!reconfigure_ovsdb_server(&server_config)) {
> +        ovs_fatal(0, "server configuration failed");
>      }
>  
>      retval = unixctl_server_create(unixctl_path, &unixctl);
> @@ -2057,14 +2239,21 @@ ovsdb_server_reconnect(struct unixctl_conn *conn, int argc OVS_UNUSED,
>   * 'config_file_path', read it and sync the runtime configuration with it. */
>  static void
>  ovsdb_server_reload(struct unixctl_conn *conn, int argc OVS_UNUSED,
> -                    const char *argv[] OVS_UNUSED, void *config_ OVS_UNUSED)
> +                    const char *argv[] OVS_UNUSED, void *config_)
>  {
> +    struct server_config *config = config_;
> +
>      if (!config_file_path) {
>          unixctl_command_reply_error(conn,
>              "Configuration file was not specified on command line");
> -    } else {
> +        return;
> +    }
> +
> +    if (!reconfigure_ovsdb_server(config)) {
>          unixctl_command_reply_error(conn,
> -            "Configuration file support is not implemented yet");
> +            "Configuration failed.  See the log file for details.");
> +    } else {
> +        unixctl_command_reply(conn, NULL);
>      }
>  }
>  
> @@ -2739,6 +2928,10 @@ save_config(struct server_config *config)
>      struct shash_node *node;
>      struct shash db_conf;
>  
> +    if (config_file_path) {
> +        return;
> +    }
> +
>      shash_init(&db_conf);
>      SHASH_FOR_EACH (node, config->all_dbs) {
>          struct db *db = node->data;
> @@ -2755,7 +2948,7 @@ save_config(struct server_config *config)
>      shash_destroy(&db_conf);
>  }
>  
> -static void
> +static bool
>  remotes_from_json(struct shash *remotes, const struct json *json)
>  {
>      struct ovsdb_jsonrpc_options *options;
> @@ -2765,14 +2958,31 @@ remotes_from_json(struct shash *remotes, const struct json *json)
>      free_remotes(remotes);
>  
>      ovs_assert(json);
> -    ovs_assert(json->type == JSON_OBJECT);
> +    if (json->type == JSON_NULL) {
> +        return true;
> +    }
> +    if (json->type != JSON_OBJECT) {
> +        VLOG_WARN("config: 'remotes' is not a JSON object");
> +        return false;
> +    }
>  
>      object = json_object(json);
>      SHASH_FOR_EACH (node, object) {
>          options = ovsdb_jsonrpc_default_options(node->name);
> -        ovsdb_jsonrpc_options_update_from_json(options, node->data, false);
>          shash_add(remotes, node->name, options);
> +
> +        json = node->data;
> +        if (json->type == JSON_OBJECT) {
> +            ovsdb_jsonrpc_options_update_from_json(options, node->data, false);
> +        } else if (json->type != JSON_NULL) {
> +            VLOG_WARN("%s: JSON-RPC options are not a JSON object or null",
> +                      node->name);
> +            free_remotes(remotes);
> +            return false;
> +        }
>      }
> +
> +    return true;
>  }
>  
>  static struct db_config *
> @@ -2783,12 +2993,24 @@ db_config_from_json(const char *name, const struct json *json)
>      struct ovsdb_parser parser;
>      struct ovsdb_error *error;
>  
> +    conf->model = SM_UNDEFINED;
> +
> +    ovs_assert(json);
> +    if (json->type == JSON_NULL) {
> +        return conf;
> +    }
> +
>      ovsdb_parser_init(&parser, json, "database %s", name);
>  
>      model = ovsdb_parser_member(&parser, "service-model",
>                                  OP_STRING | OP_OPTIONAL);
> -    conf->model = model ? service_model_from_string(json_string(model))
> -                        : SM_UNDEFINED;
> +    if (model) {
> +        conf->model = service_model_from_string(json_string(model));
> +        if (conf->model == SM_UNDEFINED) {
> +            ovsdb_parser_raise_error(&parser,
> +                "'%s' is not a valid service model", json_string(model));
> +        }
> +    }
>  
>      if (conf->model == SM_ACTIVE_BACKUP) {
>          backup = ovsdb_parser_member(&parser, "backup", OP_BOOLEAN);
> @@ -2859,7 +3081,7 @@ db_config_from_json(const char *name, const struct json *json)
>  }
>  
>  
> -static void
> +static bool
>  databases_from_json(struct shash *db_conf, const struct json *json)
>  {
>      const struct shash_node *node;
> @@ -2868,7 +3090,12 @@ databases_from_json(struct shash *db_conf, const struct json *json)
>      free_database_configs(db_conf);
>  
>      ovs_assert(json);
> -    ovs_assert(json->type == JSON_OBJECT);
> +    if (json->type == JSON_NULL) {
> +        return true;
> +    }
> +    if (json->type != JSON_OBJECT) {
> +        VLOG_WARN("config: 'databases' is not a JSON object or null");
> +    }
>  
>      object = json_object(json);
>      SHASH_FOR_EACH (node, object) {
> @@ -2876,13 +3103,19 @@ databases_from_json(struct shash *db_conf, const struct json *json)
>  
>          if (conf) {
>              shash_add(db_conf, node->name, conf);
> +        } else {
> +            free_database_configs(db_conf);
> +            return false;
>          }
>      }
> +    return true;
>  }
>  
> -/* Clears and replaces 'remotes' and 'dbnames' by a configuration read from
> - * 'config_file', which must have been previously written by save_config(). */
> -static void
> +/* Clears and replaces 'remotes' and 'db_conf' by a configuration read from
> + * 'config_file', which must have been previously written by save_config()
> + * or provided by the user with --config-file.
> + * Returns 'true', if parsing was successful, 'false' otherwise. */
> +static bool
>  load_config(FILE *config_file, struct shash *remotes,
>              struct shash *db_conf, char **sync_from,
>              char **sync_exclude, bool *is_backup)
> @@ -2890,17 +3123,34 @@ load_config(FILE *config_file, struct shash *remotes,
>      struct json *json;
>  
>      if (fseek(config_file, 0, SEEK_SET) != 0) {
> -        VLOG_FATAL("seek failed in temporary file (%s)", ovs_strerror(errno));
> +        VLOG_WARN("config: file seek failed (%s)", ovs_strerror(errno));
> +        return false;
>      }
>      json = json_from_stream(config_file);
>      if (json->type == JSON_STRING) {
> -        VLOG_FATAL("reading json failed (%s)", json_string(json));
> +        VLOG_WARN("config: reading JSON failed (%s)", json_string(json));
> +        json_destroy(json);
> +        return false;
> +    }
> +    if (json->type != JSON_OBJECT) {
> +        VLOG_WARN("configuration in a file must be a JSON object");
> +        json_destroy(json);
> +        return false;
>      }
> -    ovs_assert(json->type == JSON_OBJECT);
>  
> -    remotes_from_json(remotes, shash_find_data(json_object(json), "remotes"));
> -    databases_from_json(db_conf,
> -                        shash_find_data(json_object(json), "databases"));
> +    if (!remotes_from_json(remotes,
> +                           shash_find_data(json_object(json), "remotes"))) {
> +        VLOG_WARN("config: failed to parse 'remotes'");
> +        json_destroy(json);
> +        return false;
> +    }
> +    if (!databases_from_json(db_conf, shash_find_data(json_object(json),
> +                                                      "databases"))) {
> +        VLOG_WARN("config: failed to parse 'databases'");
> +        free_remotes(remotes);
> +        json_destroy(json);
> +        return false;
> +    }
>  
>      struct json *string;
>      string = shash_find_data(json_object(json), "sync_from");
> @@ -2911,7 +3161,9 @@ load_config(FILE *config_file, struct shash *remotes,
>      free(*sync_exclude);
>      *sync_exclude = string ? xstrdup(json_string(string)) : NULL;
>  
> -    *is_backup = json_boolean(shash_find_data(json_object(json), "is_backup"));
> +    struct json *boolean = shash_find_data(json_object(json), "is_backup");
> +    *is_backup = boolean ? json_boolean(boolean) : false;
>  
>      json_destroy(json);
> +    return true;
>  }