diff mbox series

[ovs-dev] ovsdb: Clarify that a server that leaves a cluster may never rejoin.

Message ID 20181029233711.11712-1-blp@ovn.org
State Accepted
Headers show
Series [ovs-dev] ovsdb: Clarify that a server that leaves a cluster may never rejoin. | expand

Commit Message

Ben Pfaff Oct. 29, 2018, 11:37 p.m. UTC
This wasn't clear from the documentation.

Reported-by; Paul Greenberg <greenpau@outlook.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
---
 Documentation/ref/ovsdb.7.rst | 3 +++
 ovsdb/ovsdb-server.1.in       | 3 +++
 2 files changed, 6 insertions(+)

Comments

Yifeng Sun Nov. 1, 2018, 6:25 p.m. UTC | #1
Looks good to me, thanks.
Reviewed-by: Yifeng Sun <pkusunyifeng@gmail.com>

On Mon, Oct 29, 2018 at 4:37 PM Ben Pfaff <blp@ovn.org> wrote:

> This wasn't clear from the documentation.
>
> Reported-by; Paul Greenberg <greenpau@outlook.com>
> Signed-off-by: Ben Pfaff <blp@ovn.org>
> ---
>  Documentation/ref/ovsdb.7.rst | 3 +++
>  ovsdb/ovsdb-server.1.in       | 3 +++
>  2 files changed, 6 insertions(+)
>
> diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst
> index 39d85b6e5774..f891309fc05d 100644
> --- a/Documentation/ref/ovsdb.7.rst
> +++ b/Documentation/ref/ovsdb.7.rst
> @@ -255,6 +255,9 @@ cluster: transactions not yet replicated to the server
> will be lost, and
>  transactions not yet applied to the cluster may be committed.  Afterward,
> any
>  servers in its former cluster will regard the server to have failed.
>
> +Once a server leaves a cluster, it may never rejoin it.  Instead, create
> a new
> +server and join it to the cluster.
> +
>  The servers in a cluster synchronize data over a cluster management
> protocol
>  that is specific to Open vSwitch; it is not the same as the OVSDB protocol
>  specified in RFC 7047.  For this purpose, a server in a cluster is tied
> to a
> diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in
> index 2d80dafffb7d..9f78e87f655c 100644
> --- a/ovsdb/ovsdb-server.1.in
> +++ b/ovsdb/ovsdb-server.1.in
> @@ -339,6 +339,9 @@ executed.
>  .IP
>  Use \fBovsdb\-client wait\fR (see \fBovsdb\-client\fR(1)) to wait
>  until the server has left the cluster.
> +.IP
> +Once a server leaves a cluster, it may never rejoin it.  Instead,
> +create a new server and join it to the cluster.
>  .
>  .IP "\fBcluster/kick \fIdb server\fR"
>  Start graceful removal of \fIserver\fR from \fIdb\fR's cluster, like
> --
> 2.16.1
>
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
Paul Greenberg Nov. 1, 2018, 7:03 p.m. UTC | #2
I would also add that when a server leaves a cluster, it must be removed using “ovs-appctl -t db.sock.ctl cluster/kick <dead-server-id>”

Best Regards,
Paul Greenberg
Ben Pfaff Nov. 2, 2018, 6:29 p.m. UTC | #3
Thanks, applied.

On Thu, Nov 01, 2018 at 11:25:10AM -0700, Yifeng Sun wrote:
> Looks good to me, thanks.
> Reviewed-by: Yifeng Sun <pkusunyifeng@gmail.com>
> 
> On Mon, Oct 29, 2018 at 4:37 PM Ben Pfaff <blp@ovn.org> wrote:
> 
> > This wasn't clear from the documentation.
> >
> > Reported-by; Paul Greenberg <greenpau@outlook.com>
> > Signed-off-by: Ben Pfaff <blp@ovn.org>
> > ---
> >  Documentation/ref/ovsdb.7.rst | 3 +++
> >  ovsdb/ovsdb-server.1.in       | 3 +++
> >  2 files changed, 6 insertions(+)
> >
> > diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst
> > index 39d85b6e5774..f891309fc05d 100644
> > --- a/Documentation/ref/ovsdb.7.rst
> > +++ b/Documentation/ref/ovsdb.7.rst
> > @@ -255,6 +255,9 @@ cluster: transactions not yet replicated to the server
> > will be lost, and
> >  transactions not yet applied to the cluster may be committed.  Afterward,
> > any
> >  servers in its former cluster will regard the server to have failed.
> >
> > +Once a server leaves a cluster, it may never rejoin it.  Instead, create
> > a new
> > +server and join it to the cluster.
> > +
> >  The servers in a cluster synchronize data over a cluster management
> > protocol
> >  that is specific to Open vSwitch; it is not the same as the OVSDB protocol
> >  specified in RFC 7047.  For this purpose, a server in a cluster is tied
> > to a
> > diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in
> > index 2d80dafffb7d..9f78e87f655c 100644
> > --- a/ovsdb/ovsdb-server.1.in
> > +++ b/ovsdb/ovsdb-server.1.in
> > @@ -339,6 +339,9 @@ executed.
> >  .IP
> >  Use \fBovsdb\-client wait\fR (see \fBovsdb\-client\fR(1)) to wait
> >  until the server has left the cluster.
> > +.IP
> > +Once a server leaves a cluster, it may never rejoin it.  Instead,
> > +create a new server and join it to the cluster.
> >  .
> >  .IP "\fBcluster/kick \fIdb server\fR"
> >  Start graceful removal of \fIserver\fR from \fIdb\fR's cluster, like
> > --
> > 2.16.1
> >
> > _______________________________________________
> > dev mailing list
> > dev@openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >
Ben Pfaff Nov. 2, 2018, 6:30 p.m. UTC | #4
I'm not sure what you mean.  Do you mean that the commands for removing
a server from a cluster aren't documented well enough?  Or something
else?

On Thu, Nov 01, 2018 at 07:03:07PM +0000, Paul Greenberg wrote:
> I would also add that when a server leaves a cluster, it must be removed using “ovs-appctl -t db.sock.ctl cluster/kick <dead-server-id>”
> 
> Best Regards,
> Paul Greenberg
> 
> ________________________________
> From: ovs-dev-bounces@openvswitch.org on behalf of Yifeng Sun <pkusunyifeng@gmail.com>
> Sent: Thursday, November 1, 2018 2:25 PM
> To: Ben Pfaff
> Cc: ovs dev
> Subject: Re: [ovs-dev] [PATCH] ovsdb: Clarify that a server that leaves a cluster may never rejoin.
> 
> Looks good to me, thanks.
> Reviewed-by: Yifeng Sun <pkusunyifeng@gmail.com>
> 
> On Mon, Oct 29, 2018 at 4:37 PM Ben Pfaff <blp@ovn.org> wrote:
> 
> > This wasn't clear from the documentation.
> >
> > Reported-by; Paul Greenberg <greenpau@outlook.com>
> > Signed-off-by: Ben Pfaff <blp@ovn.org>
> > ---
> > Documentation/ref/ovsdb.7.rst | 3 +++
> > ovsdb/ovsdb-server.1.in | 3 +++
> > 2 files changed, 6 insertions(+)
> >
> > diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst
> > index 39d85b6e5774..f891309fc05d 100644
> > --- a/Documentation/ref/ovsdb.7.rst
> > +++ b/Documentation/ref/ovsdb.7.rst
> > @@ -255,6 +255,9 @@ cluster: transactions not yet replicated to the server
> > will be lost, and
> > transactions not yet applied to the cluster may be committed. Afterward,
> > any
> > servers in its former cluster will regard the server to have failed.
> >
> > +Once a server leaves a cluster, it may never rejoin it. Instead, create
> > a new
> > +server and join it to the cluster.
> > +
> > The servers in a cluster synchronize data over a cluster management
> > protocol
> > that is specific to Open vSwitch; it is not the same as the OVSDB protocol
> > specified in RFC 7047. For this purpose, a server in a cluster is tied
> > to a
> > diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in
> > index 2d80dafffb7d..9f78e87f655c 100644
> > --- a/ovsdb/ovsdb-server.1.in
> > +++ b/ovsdb/ovsdb-server.1.in
> > @@ -339,6 +339,9 @@ executed.
> > .IP
> > Use \fBovsdb\-client wait\fR (see \fBovsdb\-client\fR(1)) to wait
> > until the server has left the cluster.
> > +.IP
> > +Once a server leaves a cluster, it may never rejoin it. Instead,
> > +create a new server and join it to the cluster.
> > .
> > .IP "\fBcluster/kick \fIdb server\fR"
> > Start graceful removal of \fIserver\fR from \fIdb\fR's cluster, like
> > --
> > 2.16.1
> >
> > _______________________________________________
> > dev mailing list
> > dev@openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Paul Greenberg Nov. 2, 2018, 7:08 p.m. UTC | #5
Hi Ben,

Let me clarify. Based on my observation, once a server loses touch with the rest of the cluster you have to rejoin it. At the same time it is not readily apparent that you have to do the clean up (removal) yourself. For example, you had a cluster of 3 nodes, then after some time one node is out. You go to that node and do a cluster join. My understanding is that a new server id gets generated.

Now, if you did not do a cleanup, you end up with 4 node cluster. When 3 out of 4 are working, there is no issue. It is almost mandatory to do a cleanup after join.

Best Regards,
Paul Greenberg
Paul Greenberg Nov. 2, 2018, 10:08 p.m. UTC | #6
RAFT requires odd number of nodes. With 2 out of 4 down, there never be a majority vote.

Best Regards,
Paul Greenberg
Ben Pfaff Nov. 5, 2018, 9:21 p.m. UTC | #7
On Fri, Nov 02, 2018 at 07:08:34PM +0000, Paul Greenberg wrote:
> Let me clarify. Based on my observation, once a server loses touch
> with the rest of the cluster you have to rejoin it.
> At the same time it is not readily apparent that you have to do the
> clean up (removal) yourself. For example, you had a cluster of 3
> nodes, then after some time one node is out. You go to that node and
> do a cluster join. My understanding is that a new server id gets
> generated.
> 
> Now, if you did not do a cleanup, you end up with 4 node cluster. When 3 out of 4 are working, there is no issue. It is almost mandatory to do a cleanup after join.

The design intent is that, if a server goes out of contact with the rest
of the cluster, and later comes back into contact, then it gracefully
catches up and becomes a productive member of the cluster.

It seems like you're encountering bugs that I don't understand yet.  Can
you help me to reproduce them?
Paul Greenberg Nov. 9, 2018, 4:22 p.m. UTC | #8
Hi Ben,

The issues are sporadic. I got prometheus ovn_exporter running to capture relevant RAFT metrics. Once I see them reappearing, I will post log entries around the time when a cluster is in Leader, Candidate, Follower states. (During normal operation, it is Leader, and 2 Followers).

Best Regards,
Paul Greenberg
Ben Pfaff Nov. 15, 2018, 6:04 p.m. UTC | #9
I found and fixed some bugs in the Raft implementation.  The patch
series is currently waiting for review:
        https://patchwork.ozlabs.org/project/openvswitch/list/?series=76115

On Fri, Nov 09, 2018 at 04:22:51PM +0000, Paul Greenberg wrote:
> Hi Ben,
> 
> The issues are sporadic. I got prometheus ovn_exporter running to capture relevant RAFT metrics. Once I see them reappearing, I will post log entries around the time when a cluster is in Leader, Candidate, Follower states. (During normal operation, it is Leader, and 2 Followers).
> 
> Best Regards,
> Paul Greenberg
> 
> ________________________________
> From: Ben Pfaff <blp@ovn.org>
> Sent: Monday, November 5, 2018 4:21 PM
> To: Paul Greenberg
> Cc: Yifeng Sun; ovs dev
> Subject: Re: [ovs-dev] [PATCH] ovsdb: Clarify that a server that leaves a cluster may never rejoin.
> 
> On Fri, Nov 02, 2018 at 07:08:34PM +0000, Paul Greenberg wrote:
> > Let me clarify. Based on my observation, once a server loses touch
> > with the rest of the cluster you have to rejoin it.
> > At the same time it is not readily apparent that you have to do the
> > clean up (removal) yourself. For example, you had a cluster of 3
> > nodes, then after some time one node is out. You go to that node and
> > do a cluster join. My understanding is that a new server id gets
> > generated.
> >
> > Now, if you did not do a cleanup, you end up with 4 node cluster. When 3 out of 4 are working, there is no issue. It is almost mandatory to do a cleanup after join.
> 
> The design intent is that, if a server goes out of contact with the rest
> of the cluster, and later comes back into contact, then it gracefully
> catches up and becomes a productive member of the cluster.
> 
> It seems like you're encountering bugs that I don't understand yet. Can
> you help me to reproduce them?
diff mbox series

Patch

diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst
index 39d85b6e5774..f891309fc05d 100644
--- a/Documentation/ref/ovsdb.7.rst
+++ b/Documentation/ref/ovsdb.7.rst
@@ -255,6 +255,9 @@  cluster: transactions not yet replicated to the server will be lost, and
 transactions not yet applied to the cluster may be committed.  Afterward, any
 servers in its former cluster will regard the server to have failed.
 
+Once a server leaves a cluster, it may never rejoin it.  Instead, create a new
+server and join it to the cluster.
+
 The servers in a cluster synchronize data over a cluster management protocol
 that is specific to Open vSwitch; it is not the same as the OVSDB protocol
 specified in RFC 7047.  For this purpose, a server in a cluster is tied to a
diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in
index 2d80dafffb7d..9f78e87f655c 100644
--- a/ovsdb/ovsdb-server.1.in
+++ b/ovsdb/ovsdb-server.1.in
@@ -339,6 +339,9 @@  executed.
 .IP
 Use \fBovsdb\-client wait\fR (see \fBovsdb\-client\fR(1)) to wait
 until the server has left the cluster.
+.IP
+Once a server leaves a cluster, it may never rejoin it.  Instead,
+create a new server and join it to the cluster.
 .
 .IP "\fBcluster/kick \fIdb server\fR"
 Start graceful removal of \fIserver\fR from \fIdb\fR's cluster, like