From patchwork Tue May 14 16:33:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 1099604 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 453NbG4bYPz9s9T for ; Wed, 15 May 2019 02:35:38 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 84BADE5C; Tue, 14 May 2019 16:33:50 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 9E855E57 for ; Tue, 14 May 2019 16:33:48 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 361B583A for ; Tue, 14 May 2019 16:33:48 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BBF1281DE5; Tue, 14 May 2019 16:33:47 +0000 (UTC) Received: from dmarchan.remote.csb (ovpn-116-80.ams2.redhat.com [10.36.116.80]) by smtp.corp.redhat.com (Postfix) with ESMTP id 479451975C; Tue, 14 May 2019 16:33:46 +0000 (UTC) From: David Marchand To: dev@openvswitch.org Date: Tue, 14 May 2019 18:33:28 +0200 Message-Id: <1557851610-5602-4-git-send-email-david.marchand@redhat.com> In-Reply-To: <1557851610-5602-1-git-send-email-david.marchand@redhat.com> References: <1557851610-5602-1-git-send-email-david.marchand@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Tue, 14 May 2019 16:33:47 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: i.maximets@samsung.com Subject: [ovs-dev] [RFC v2 3/5] dpif-netdev: Do not sleep when swapping queues. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org When swapping queues from a pmd thread to another (q0 polled by pmd0/q1 polled by pmd1 -> q1 polled by pmd0/q0 polled by pmd1), the current "Step 5" puts both pmds to sleep waiting for the control thread to wake them up later. Prefer to make them spin in such a case to avoid sleeping an undeterministic amount of time. Signed-off-by: David Marchand --- lib/dpif-netdev.c | 47 ++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 42 insertions(+), 5 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 4febd8b..254a50e 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -684,6 +684,7 @@ struct dp_netdev_pmd_thread { struct seq *reload_seq; uint64_t last_reload_seq; atomic_bool reload; /* Do we need to reload ports? */ + atomic_bool wait_for_reload; /* Can we busy wait for the next reload? */ atomic_bool exit; /* For terminating the pmd thread. */ pthread_t thread; unsigned core_id; /* CPU core id of this pmd thread. */ @@ -4897,6 +4898,33 @@ reconfigure_datapath(struct dp_netdev *dp) HMAP_FOR_EACH_SAFE (poll, poll_next, node, &pmd->poll_list) { if (poll->rxq->pmd != pmd) { dp_netdev_del_rxq_from_pmd(pmd, poll); + + /* This pmd might sleep after this step reload if it has no + * rxq remaining. Can we tell it to busy wait for new rxq at + * Step 6 ? */ + if (hmap_count(&pmd->poll_list) == 0) { + HMAP_FOR_EACH (port, node, &dp->ports) { + int qid; + + if (!netdev_is_pmd(port->netdev)) { + continue; + } + + for (qid = 0; qid < port->n_rxq; qid++) { + struct dp_netdev_rxq *q = &port->rxqs[qid]; + + if (q->pmd == pmd) { + atomic_store_relaxed(&q->pmd->wait_for_reload, + true); + break; + } + } + + if (qid != port->n_rxq) { + break; + } + } + } } } ovs_mutex_unlock(&pmd->port_mutex); @@ -5414,7 +5442,9 @@ pmd_thread_main(void *f_) struct pmd_perf_stats *s = &pmd->perf_stats; unsigned int lc = 0; struct polled_queue *poll_list; + bool wait_for_reload = false; bool exiting; + bool reload; int poll_cnt; int i; int process_packets = 0; @@ -5442,9 +5472,16 @@ reload: } if (!poll_cnt) { - while (seq_read(pmd->reload_seq) == pmd->last_reload_seq) { - seq_wait(pmd->reload_seq, pmd->last_reload_seq); - poll_block(); + /* Don't sleep, control thread will ask for a reload shortly. */ + if (wait_for_reload) { + do { + atomic_read_relaxed(&pmd->reload, &reload); + } while (!reload); + } else { + while (seq_read(pmd->reload_seq) == pmd->last_reload_seq) { + seq_wait(pmd->reload_seq, pmd->last_reload_seq); + poll_block(); + } } lc = UINT_MAX; } @@ -5483,8 +5520,6 @@ reload: } if (lc++ > 1024) { - bool reload; - lc = 0; coverage_try_clear(); @@ -5504,6 +5539,7 @@ reload: ovs_mutex_unlock(&pmd->perf_stats.stats_mutex); poll_cnt = pmd_load_queues_and_ports(pmd, &poll_list); + atomic_read_relaxed(&pmd->wait_for_reload, &wait_for_reload); atomic_read_relaxed(&pmd->exit, &exiting); /* Signal here to make sure the pmd finishes * reloading the updated configuration. */ @@ -5840,6 +5876,7 @@ dp_netdev_pmd_reload_done(struct dp_netdev_pmd_thread *pmd) { uint32_t old; + atomic_store_relaxed(&pmd->wait_for_reload, false); atomic_store_relaxed(&pmd->reload, false); pmd->last_reload_seq = seq_read(pmd->reload_seq); atomic_sub_explicit(&pmd->dp->reloading_pmds, 1, &old,