| Message ID | 20181029233711.11712-1-blp@ovn.org |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [ovs-dev] ovsdb: Clarify that a server that leaves a cluster may never rejoin. | expand |
Looks good to me, thanks. Reviewed-by: Yifeng Sun <pkusunyifeng@gmail.com> On Mon, Oct 29, 2018 at 4:37 PM Ben Pfaff <blp@ovn.org> wrote: > This wasn't clear from the documentation. > > Reported-by; Paul Greenberg <greenpau@outlook.com> > Signed-off-by: Ben Pfaff <blp@ovn.org> > --- > Documentation/ref/ovsdb.7.rst | 3 +++ > ovsdb/ovsdb-server.1.in | 3 +++ > 2 files changed, 6 insertions(+) > > diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst > index 39d85b6e5774..f891309fc05d 100644 > --- a/Documentation/ref/ovsdb.7.rst > +++ b/Documentation/ref/ovsdb.7.rst > @@ -255,6 +255,9 @@ cluster: transactions not yet replicated to the server > will be lost, and > transactions not yet applied to the cluster may be committed. Afterward, > any > servers in its former cluster will regard the server to have failed. > > +Once a server leaves a cluster, it may never rejoin it. Instead, create > a new > +server and join it to the cluster. > + > The servers in a cluster synchronize data over a cluster management > protocol > that is specific to Open vSwitch; it is not the same as the OVSDB protocol > specified in RFC 7047. For this purpose, a server in a cluster is tied > to a > diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in > index 2d80dafffb7d..9f78e87f655c 100644 > --- a/ovsdb/ovsdb-server.1.in > +++ b/ovsdb/ovsdb-server.1.in > @@ -339,6 +339,9 @@ executed. > .IP > Use \fBovsdb\-client wait\fR (see \fBovsdb\-client\fR(1)) to wait > until the server has left the cluster. > +.IP > +Once a server leaves a cluster, it may never rejoin it. Instead, > +create a new server and join it to the cluster. > . > .IP "\fBcluster/kick \fIdb server\fR" > Start graceful removal of \fIserver\fR from \fIdb\fR's cluster, like > -- > 2.16.1 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >
I would also add that when a server leaves a cluster, it must be removed using “ovs-appctl -t db.sock.ctl cluster/kick <dead-server-id>” Best Regards, Paul Greenberg
Thanks, applied. On Thu, Nov 01, 2018 at 11:25:10AM -0700, Yifeng Sun wrote: > Looks good to me, thanks. > Reviewed-by: Yifeng Sun <pkusunyifeng@gmail.com> > > On Mon, Oct 29, 2018 at 4:37 PM Ben Pfaff <blp@ovn.org> wrote: > > > This wasn't clear from the documentation. > > > > Reported-by; Paul Greenberg <greenpau@outlook.com> > > Signed-off-by: Ben Pfaff <blp@ovn.org> > > --- > > Documentation/ref/ovsdb.7.rst | 3 +++ > > ovsdb/ovsdb-server.1.in | 3 +++ > > 2 files changed, 6 insertions(+) > > > > diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst > > index 39d85b6e5774..f891309fc05d 100644 > > --- a/Documentation/ref/ovsdb.7.rst > > +++ b/Documentation/ref/ovsdb.7.rst > > @@ -255,6 +255,9 @@ cluster: transactions not yet replicated to the server > > will be lost, and > > transactions not yet applied to the cluster may be committed. Afterward, > > any > > servers in its former cluster will regard the server to have failed. > > > > +Once a server leaves a cluster, it may never rejoin it. Instead, create > > a new > > +server and join it to the cluster. > > + > > The servers in a cluster synchronize data over a cluster management > > protocol > > that is specific to Open vSwitch; it is not the same as the OVSDB protocol > > specified in RFC 7047. For this purpose, a server in a cluster is tied > > to a > > diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in > > index 2d80dafffb7d..9f78e87f655c 100644 > > --- a/ovsdb/ovsdb-server.1.in > > +++ b/ovsdb/ovsdb-server.1.in > > @@ -339,6 +339,9 @@ executed. > > .IP > > Use \fBovsdb\-client wait\fR (see \fBovsdb\-client\fR(1)) to wait > > until the server has left the cluster. > > +.IP > > +Once a server leaves a cluster, it may never rejoin it. Instead, > > +create a new server and join it to the cluster. > > . > > .IP "\fBcluster/kick \fIdb server\fR" > > Start graceful removal of \fIserver\fR from \fIdb\fR's cluster, like > > -- > > 2.16.1 > > > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > >
I'm not sure what you mean. Do you mean that the commands for removing a server from a cluster aren't documented well enough? Or something else? On Thu, Nov 01, 2018 at 07:03:07PM +0000, Paul Greenberg wrote: > I would also add that when a server leaves a cluster, it must be removed using “ovs-appctl -t db.sock.ctl cluster/kick <dead-server-id>” > > Best Regards, > Paul Greenberg > > ________________________________ > From: ovs-dev-bounces@openvswitch.org on behalf of Yifeng Sun <pkusunyifeng@gmail.com> > Sent: Thursday, November 1, 2018 2:25 PM > To: Ben Pfaff > Cc: ovs dev > Subject: Re: [ovs-dev] [PATCH] ovsdb: Clarify that a server that leaves a cluster may never rejoin. > > Looks good to me, thanks. > Reviewed-by: Yifeng Sun <pkusunyifeng@gmail.com> > > On Mon, Oct 29, 2018 at 4:37 PM Ben Pfaff <blp@ovn.org> wrote: > > > This wasn't clear from the documentation. > > > > Reported-by; Paul Greenberg <greenpau@outlook.com> > > Signed-off-by: Ben Pfaff <blp@ovn.org> > > --- > > Documentation/ref/ovsdb.7.rst | 3 +++ > > ovsdb/ovsdb-server.1.in | 3 +++ > > 2 files changed, 6 insertions(+) > > > > diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst > > index 39d85b6e5774..f891309fc05d 100644 > > --- a/Documentation/ref/ovsdb.7.rst > > +++ b/Documentation/ref/ovsdb.7.rst > > @@ -255,6 +255,9 @@ cluster: transactions not yet replicated to the server > > will be lost, and > > transactions not yet applied to the cluster may be committed. Afterward, > > any > > servers in its former cluster will regard the server to have failed. > > > > +Once a server leaves a cluster, it may never rejoin it. Instead, create > > a new > > +server and join it to the cluster. > > + > > The servers in a cluster synchronize data over a cluster management > > protocol > > that is specific to Open vSwitch; it is not the same as the OVSDB protocol > > specified in RFC 7047. For this purpose, a server in a cluster is tied > > to a > > diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in > > index 2d80dafffb7d..9f78e87f655c 100644 > > --- a/ovsdb/ovsdb-server.1.in > > +++ b/ovsdb/ovsdb-server.1.in > > @@ -339,6 +339,9 @@ executed. > > .IP > > Use \fBovsdb\-client wait\fR (see \fBovsdb\-client\fR(1)) to wait > > until the server has left the cluster. > > +.IP > > +Once a server leaves a cluster, it may never rejoin it. Instead, > > +create a new server and join it to the cluster. > > . > > .IP "\fBcluster/kick \fIdb server\fR" > > Start graceful removal of \fIserver\fR from \fIdb\fR's cluster, like > > -- > > 2.16.1 > > > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Hi Ben, Let me clarify. Based on my observation, once a server loses touch with the rest of the cluster you have to rejoin it. At the same time it is not readily apparent that you have to do the clean up (removal) yourself. For example, you had a cluster of 3 nodes, then after some time one node is out. You go to that node and do a cluster join. My understanding is that a new server id gets generated. Now, if you did not do a cleanup, you end up with 4 node cluster. When 3 out of 4 are working, there is no issue. It is almost mandatory to do a cleanup after join. Best Regards, Paul Greenberg
RAFT requires odd number of nodes. With 2 out of 4 down, there never be a majority vote. Best Regards, Paul Greenberg
On Fri, Nov 02, 2018 at 07:08:34PM +0000, Paul Greenberg wrote: > Let me clarify. Based on my observation, once a server loses touch > with the rest of the cluster you have to rejoin it. > At the same time it is not readily apparent that you have to do the > clean up (removal) yourself. For example, you had a cluster of 3 > nodes, then after some time one node is out. You go to that node and > do a cluster join. My understanding is that a new server id gets > generated. > > Now, if you did not do a cleanup, you end up with 4 node cluster. When 3 out of 4 are working, there is no issue. It is almost mandatory to do a cleanup after join. The design intent is that, if a server goes out of contact with the rest of the cluster, and later comes back into contact, then it gracefully catches up and becomes a productive member of the cluster. It seems like you're encountering bugs that I don't understand yet. Can you help me to reproduce them?
Hi Ben, The issues are sporadic. I got prometheus ovn_exporter running to capture relevant RAFT metrics. Once I see them reappearing, I will post log entries around the time when a cluster is in Leader, Candidate, Follower states. (During normal operation, it is Leader, and 2 Followers). Best Regards, Paul Greenberg
I found and fixed some bugs in the Raft implementation. The patch
series is currently waiting for review:
https://patchwork.ozlabs.org/project/openvswitch/list/?series=76115
On Fri, Nov 09, 2018 at 04:22:51PM +0000, Paul Greenberg wrote:
> Hi Ben,
>
> The issues are sporadic. I got prometheus ovn_exporter running to capture relevant RAFT metrics. Once I see them reappearing, I will post log entries around the time when a cluster is in Leader, Candidate, Follower states. (During normal operation, it is Leader, and 2 Followers).
>
> Best Regards,
> Paul Greenberg
>
> ________________________________
> From: Ben Pfaff <blp@ovn.org>
> Sent: Monday, November 5, 2018 4:21 PM
> To: Paul Greenberg
> Cc: Yifeng Sun; ovs dev
> Subject: Re: [ovs-dev] [PATCH] ovsdb: Clarify that a server that leaves a cluster may never rejoin.
>
> On Fri, Nov 02, 2018 at 07:08:34PM +0000, Paul Greenberg wrote:
> > Let me clarify. Based on my observation, once a server loses touch
> > with the rest of the cluster you have to rejoin it.
> > At the same time it is not readily apparent that you have to do the
> > clean up (removal) yourself. For example, you had a cluster of 3
> > nodes, then after some time one node is out. You go to that node and
> > do a cluster join. My understanding is that a new server id gets
> > generated.
> >
> > Now, if you did not do a cleanup, you end up with 4 node cluster. When 3 out of 4 are working, there is no issue. It is almost mandatory to do a cleanup after join.
>
> The design intent is that, if a server goes out of contact with the rest
> of the cluster, and later comes back into contact, then it gracefully
> catches up and becomes a productive member of the cluster.
>
> It seems like you're encountering bugs that I don't understand yet. Can
> you help me to reproduce them?
diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst index 39d85b6e5774..f891309fc05d 100644 --- a/Documentation/ref/ovsdb.7.rst +++ b/Documentation/ref/ovsdb.7.rst @@ -255,6 +255,9 @@ cluster: transactions not yet replicated to the server will be lost, and transactions not yet applied to the cluster may be committed. Afterward, any servers in its former cluster will regard the server to have failed. +Once a server leaves a cluster, it may never rejoin it. Instead, create a new +server and join it to the cluster. + The servers in a cluster synchronize data over a cluster management protocol that is specific to Open vSwitch; it is not the same as the OVSDB protocol specified in RFC 7047. For this purpose, a server in a cluster is tied to a diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in index 2d80dafffb7d..9f78e87f655c 100644 --- a/ovsdb/ovsdb-server.1.in +++ b/ovsdb/ovsdb-server.1.in @@ -339,6 +339,9 @@ executed. .IP Use \fBovsdb\-client wait\fR (see \fBovsdb\-client\fR(1)) to wait until the server has left the cluster. +.IP +Once a server leaves a cluster, it may never rejoin it. Instead, +create a new server and join it to the cluster. . .IP "\fBcluster/kick \fIdb server\fR" Start graceful removal of \fIserver\fR from \fIdb\fR's cluster, like
This wasn't clear from the documentation. Reported-by; Paul Greenberg <greenpau@outlook.com> Signed-off-by: Ben Pfaff <blp@ovn.org> --- Documentation/ref/ovsdb.7.rst | 3 +++ ovsdb/ovsdb-server.1.in | 3 +++ 2 files changed, 6 insertions(+)