From patchwork Wed Apr 21 12:43:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Seth Forshee X-Patchwork-Id: 1468710 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FQKx4206jz9ssP; Wed, 21 Apr 2021 22:43:52 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1lZCCq-00027T-Qt; Wed, 21 Apr 2021 12:43:48 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1lZCCn-00026p-LM for kernel-team@lists.ubuntu.com; Wed, 21 Apr 2021 12:43:45 +0000 Received: from mail-ot1-f72.google.com ([209.85.210.72]) by youngberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1lZCCn-0003jS-8W for kernel-team@lists.ubuntu.com; Wed, 21 Apr 2021 12:43:45 +0000 Received: by mail-ot1-f72.google.com with SMTP id h10-20020a9d554a0000b02901d8bed80c43so13487549oti.22 for ; Wed, 21 Apr 2021 05:43:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2fUVPXklZhqAfs0rzecTzaaSHYgtQESQucTgV9H9vMs=; b=BlyyCtrKKwZvz9zEF4lpYKYUK32nUiLHve6i2x7fzbqmPvd6GCPeaaJu8RbMiE/VGd AeoBsB7QFz1BJTb4jjW8rAGBp0vvJV04XwyHWjRVQEUICeJF/0rZqgwFjC7nhHbAhwds Ogqsg2PAwrvTLnZLlwlTjRPHXfhVMVSJ/wVFd6xZp3bfhFSmZVoAq8D8wFTQ6Cdwfzi8 53PUY4ZRbg6d7mHTpZ/fOuuPJaDOD8zQfFZFNw5Gt7eFc8WWj7x0w2ZZlOR4m5yFLsLN RMPicRDs8SlLtKXjWrZvqV1zgzgTETq/EUnk/nCFp0zL92dmVFCIhXKuSBS/YhJFaVEa 8Byw== X-Gm-Message-State: AOAM532npBCg62VUSVHspE7ffUmmcCGmMUwUPD/THnnUkqwyjFnZ39O4 bWJSt7Nn8rd9RrONzPeRYt7NNOD/7teQVcSvu0AzjMo8irepTmVpPnLmz1IJKGOfx/pkQ5DcWQ4 rlhZoaquEImLgNVamo+CGGuIShZSbkdkAlWsDlrfY6g== X-Received: by 2002:a05:6808:1412:: with SMTP id w18mr6584563oiv.50.1619009024134; Wed, 21 Apr 2021 05:43:44 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxE64ZMKUFPPd+ZCPFZjURVFarEdJ0/wb6lmw4neWTBxy3U+DAkklby7eEJUeOVkT4rDyQUsg== X-Received: by 2002:a05:6808:1412:: with SMTP id w18mr6584549oiv.50.1619009023799; Wed, 21 Apr 2021 05:43:43 -0700 (PDT) Received: from localhost ([2605:a601:ac0f:820:2704:71f7:acb6:2135]) by smtp.gmail.com with ESMTPSA id s131sm448211oib.14.2021.04.21.05.43.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Apr 2021 05:43:43 -0700 (PDT) From: Seth Forshee To: kernel-team@lists.ubuntu.com Subject: [PATCH 1/1][H] UBUNTU: SAUCE: Revert "s390/cio: remove pm support from ccw bus driver" Date: Wed, 21 Apr 2021 07:43:40 -0500 Message-Id: <20210421124340.301916-2-seth.forshee@canonical.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210421124340.301916-1-seth.forshee@canonical.com> References: <20210421124340.301916-1-seth.forshee@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" BugLink: https://bugs.launchpad.net/bugs/1925211 This reverts commit 8cc0dcfdc1c0e0be107d0288f9c0cf1f4201be62, which introduces a regression with hot-unplug of virtio disks from VMs. Roll back these changes until we have a proper fix. Signed-off-by: Seth Forshee --- arch/s390/include/asm/ccwdev.h | 10 ++ drivers/s390/cio/cmf.c | 5 + drivers/s390/cio/device.c | 247 ++++++++++++++++++++++++++++++++- drivers/s390/cio/device.h | 1 + drivers/s390/cio/device_fsm.c | 6 + drivers/s390/cio/io_sch.h | 1 + 6 files changed, 268 insertions(+), 2 deletions(-) diff --git a/arch/s390/include/asm/ccwdev.h b/arch/s390/include/asm/ccwdev.h index 778247bb1d61..0495ac635ed5 100644 --- a/arch/s390/include/asm/ccwdev.h +++ b/arch/s390/include/asm/ccwdev.h @@ -126,6 +126,11 @@ enum uc_todo { * @notify: notify driver of device state changes * @path_event: notify driver of channel path events * @shutdown: called at device shutdown + * @prepare: prepare for pm state transition + * @complete: undo work done in @prepare + * @freeze: callback for freezing during hibernation snapshotting + * @thaw: undo work done in @freeze + * @restore: callback for restoring after hibernation * @uc_handler: callback for unit check handler * @driver: embedded device driver structure * @int_class: interruption class to use for accounting interrupts @@ -139,6 +144,11 @@ struct ccw_driver { int (*notify) (struct ccw_device *, int); void (*path_event) (struct ccw_device *, int *); void (*shutdown) (struct ccw_device *); + int (*prepare) (struct ccw_device *); + void (*complete) (struct ccw_device *); + int (*freeze)(struct ccw_device *); + int (*thaw) (struct ccw_device *); + int (*restore)(struct ccw_device *); enum uc_todo (*uc_handler) (struct ccw_device *, struct irb *); struct device_driver driver; enum interruption_class int_class; diff --git a/drivers/s390/cio/cmf.c b/drivers/s390/cio/cmf.c index b7b590646d58..72dd2471ec1e 100644 --- a/drivers/s390/cio/cmf.c +++ b/drivers/s390/cio/cmf.c @@ -1109,6 +1109,11 @@ static ssize_t cmb_enable_store(struct device *dev, } DEVICE_ATTR_RW(cmb_enable); +int ccw_set_cmf(struct ccw_device *cdev, int enable) +{ + return cmbops->set(cdev, enable ? 2 : 0); +} + /** * enable_cmf() - switch on the channel measurement for a specific device * @cdev: The ccw device to be enabled diff --git a/drivers/s390/cio/device.c b/drivers/s390/cio/device.c index 4b0a7cbb2096..45d68e8b3449 100644 --- a/drivers/s390/cio/device.c +++ b/drivers/s390/cio/device.c @@ -1421,7 +1421,7 @@ static enum io_sch_action sch_get_action(struct subchannel *sch) } if (device_is_disconnected(cdev)) return IO_SCH_REPROBE; - if (cdev->online) + if (cdev->online && !cdev->private->flags.resuming) return IO_SCH_VERIFY; if (cdev->private->state == DEV_STATE_NOT_OPER) return IO_SCH_UNREG_ATTACH; @@ -1513,6 +1513,11 @@ static int io_subchannel_sch_event(struct subchannel *sch, int process) break; case IO_SCH_UNREG_ATTACH: spin_lock_irqsave(sch->lock, flags); + if (cdev->private->flags.resuming) { + /* Device will be handled later. */ + rc = 0; + goto out_unlock; + } sch_set_cdev(sch, NULL); spin_unlock_irqrestore(sch->lock, flags); /* Unregister ccw device. */ @@ -1525,7 +1530,7 @@ static int io_subchannel_sch_event(struct subchannel *sch, int process) switch (action) { case IO_SCH_ORPH_UNREG: case IO_SCH_UNREG: - if (!cdev) + if (!cdev || !cdev->private->flags.resuming) css_sch_device_unregister(sch); break; case IO_SCH_ORPH_ATTACH: @@ -1684,6 +1689,14 @@ void ccw_device_wait_idle(struct ccw_device *cdev) udelay(100); } } + +static int ccw_device_pm_restore(struct device *dev); + +int ccw_device_force_console(struct ccw_device *cdev) +{ + return ccw_device_pm_restore(&cdev->dev); +} +EXPORT_SYMBOL_GPL(ccw_device_force_console); #endif /** @@ -1784,6 +1797,235 @@ static void ccw_device_shutdown(struct device *dev) __disable_cmf(cdev); } +static int ccw_device_pm_prepare(struct device *dev) +{ + struct ccw_device *cdev = to_ccwdev(dev); + + if (work_pending(&cdev->private->todo_work)) + return -EAGAIN; + /* Fail while device is being set online/offline. */ + if (atomic_read(&cdev->private->onoff)) + return -EAGAIN; + + if (cdev->online && cdev->drv && cdev->drv->prepare) + return cdev->drv->prepare(cdev); + + return 0; +} + +static void ccw_device_pm_complete(struct device *dev) +{ + struct ccw_device *cdev = to_ccwdev(dev); + + if (cdev->online && cdev->drv && cdev->drv->complete) + cdev->drv->complete(cdev); +} + +static int ccw_device_pm_freeze(struct device *dev) +{ + struct ccw_device *cdev = to_ccwdev(dev); + struct subchannel *sch = to_subchannel(cdev->dev.parent); + int ret, cm_enabled; + + /* Fail suspend while device is in transistional state. */ + if (!dev_fsm_final_state(cdev)) + return -EAGAIN; + if (!cdev->online) + return 0; + if (cdev->drv && cdev->drv->freeze) { + ret = cdev->drv->freeze(cdev); + if (ret) + return ret; + } + + spin_lock_irq(sch->lock); + cm_enabled = cdev->private->cmb != NULL; + spin_unlock_irq(sch->lock); + if (cm_enabled) { + /* Don't have the css write on memory. */ + ret = ccw_set_cmf(cdev, 0); + if (ret) + return ret; + } + /* From here on, disallow device driver I/O. */ + spin_lock_irq(sch->lock); + ret = cio_disable_subchannel(sch); + spin_unlock_irq(sch->lock); + + return ret; +} + +static int ccw_device_pm_thaw(struct device *dev) +{ + struct ccw_device *cdev = to_ccwdev(dev); + struct subchannel *sch = to_subchannel(cdev->dev.parent); + int ret, cm_enabled; + + if (!cdev->online) + return 0; + + spin_lock_irq(sch->lock); + /* Allow device driver I/O again. */ + ret = cio_enable_subchannel(sch, (u32)(addr_t)sch); + cm_enabled = cdev->private->cmb != NULL; + spin_unlock_irq(sch->lock); + if (ret) + return ret; + + if (cm_enabled) { + ret = ccw_set_cmf(cdev, 1); + if (ret) + return ret; + } + + if (cdev->drv && cdev->drv->thaw) + ret = cdev->drv->thaw(cdev); + + return ret; +} + +static void __ccw_device_pm_restore(struct ccw_device *cdev) +{ + struct subchannel *sch = to_subchannel(cdev->dev.parent); + + spin_lock_irq(sch->lock); + if (cio_is_console(sch->schid)) { + cio_enable_subchannel(sch, (u32)(addr_t)sch); + goto out_unlock; + } + /* + * While we were sleeping, devices may have gone or become + * available again. Kick re-detection. + */ + cdev->private->flags.resuming = 1; + cdev->private->path_new_mask = LPM_ANYPATH; + css_sched_sch_todo(sch, SCH_TODO_EVAL); + spin_unlock_irq(sch->lock); + css_wait_for_slow_path(); + + /* cdev may have been moved to a different subchannel. */ + sch = to_subchannel(cdev->dev.parent); + spin_lock_irq(sch->lock); + if (cdev->private->state != DEV_STATE_ONLINE && + cdev->private->state != DEV_STATE_OFFLINE) + goto out_unlock; + + ccw_device_recognition(cdev); + spin_unlock_irq(sch->lock); + wait_event(cdev->private->wait_q, dev_fsm_final_state(cdev) || + cdev->private->state == DEV_STATE_DISCONNECTED); + spin_lock_irq(sch->lock); + +out_unlock: + cdev->private->flags.resuming = 0; + spin_unlock_irq(sch->lock); +} + +static int resume_handle_boxed(struct ccw_device *cdev) +{ + cdev->private->state = DEV_STATE_BOXED; + if (ccw_device_notify(cdev, CIO_BOXED) == NOTIFY_OK) + return 0; + ccw_device_sched_todo(cdev, CDEV_TODO_UNREG); + return -ENODEV; +} + +static int resume_handle_disc(struct ccw_device *cdev) +{ + cdev->private->state = DEV_STATE_DISCONNECTED; + if (ccw_device_notify(cdev, CIO_GONE) == NOTIFY_OK) + return 0; + ccw_device_sched_todo(cdev, CDEV_TODO_UNREG); + return -ENODEV; +} + +static int ccw_device_pm_restore(struct device *dev) +{ + struct ccw_device *cdev = to_ccwdev(dev); + struct subchannel *sch; + int ret = 0; + + __ccw_device_pm_restore(cdev); + sch = to_subchannel(cdev->dev.parent); + spin_lock_irq(sch->lock); + if (cio_is_console(sch->schid)) + goto out_restore; + + /* check recognition results */ + switch (cdev->private->state) { + case DEV_STATE_OFFLINE: + case DEV_STATE_ONLINE: + cdev->private->flags.donotify = 0; + break; + case DEV_STATE_BOXED: + ret = resume_handle_boxed(cdev); + if (ret) + goto out_unlock; + goto out_restore; + default: + ret = resume_handle_disc(cdev); + if (ret) + goto out_unlock; + goto out_restore; + } + /* check if the device type has changed */ + if (!ccw_device_test_sense_data(cdev)) { + ccw_device_update_sense_data(cdev); + ccw_device_sched_todo(cdev, CDEV_TODO_REBIND); + ret = -ENODEV; + goto out_unlock; + } + if (!cdev->online) + goto out_unlock; + + if (ccw_device_online(cdev)) { + ret = resume_handle_disc(cdev); + if (ret) + goto out_unlock; + goto out_restore; + } + spin_unlock_irq(sch->lock); + wait_event(cdev->private->wait_q, dev_fsm_final_state(cdev)); + spin_lock_irq(sch->lock); + + if (ccw_device_notify(cdev, CIO_OPER) == NOTIFY_BAD) { + ccw_device_sched_todo(cdev, CDEV_TODO_UNREG); + ret = -ENODEV; + goto out_unlock; + } + + /* reenable cmf, if needed */ + if (cdev->private->cmb) { + spin_unlock_irq(sch->lock); + ret = ccw_set_cmf(cdev, 1); + spin_lock_irq(sch->lock); + if (ret) { + CIO_MSG_EVENT(2, "resume: cdev 0.%x.%04x: cmf failed " + "(rc=%d)\n", cdev->private->dev_id.ssid, + cdev->private->dev_id.devno, ret); + ret = 0; + } + } + +out_restore: + spin_unlock_irq(sch->lock); + if (cdev->online && cdev->drv && cdev->drv->restore) + ret = cdev->drv->restore(cdev); + return ret; + +out_unlock: + spin_unlock_irq(sch->lock); + return ret; +} + +static const struct dev_pm_ops ccw_pm_ops = { + .prepare = ccw_device_pm_prepare, + .complete = ccw_device_pm_complete, + .freeze = ccw_device_pm_freeze, + .thaw = ccw_device_pm_thaw, + .restore = ccw_device_pm_restore, +}; + static struct bus_type ccw_bus_type = { .name = "ccw", .match = ccw_bus_match, @@ -1791,6 +2033,7 @@ static struct bus_type ccw_bus_type = { .probe = ccw_device_probe, .remove = ccw_device_remove, .shutdown = ccw_device_shutdown, + .pm = &ccw_pm_ops, }; /** diff --git a/drivers/s390/cio/device.h b/drivers/s390/cio/device.h index 24b2fce69590..853b6a8ca095 100644 --- a/drivers/s390/cio/device.h +++ b/drivers/s390/cio/device.h @@ -143,5 +143,6 @@ void retry_set_schib(struct ccw_device *cdev); void cmf_retry_copy_block(struct ccw_device *); int cmf_reenable(struct ccw_device *); void cmf_reactivate(void); +int ccw_set_cmf(struct ccw_device *cdev, int enable); extern struct device_attribute dev_attr_cmb_enable; #endif diff --git a/drivers/s390/cio/device_fsm.c b/drivers/s390/cio/device_fsm.c index 6420b197bb05..8fc267324ebb 100644 --- a/drivers/s390/cio/device_fsm.c +++ b/drivers/s390/cio/device_fsm.c @@ -224,6 +224,12 @@ ccw_device_recog_done(struct ccw_device *cdev, int state) wake_up(&cdev->private->wait_q); return; } + if (cdev->private->flags.resuming) { + cdev->private->state = state; + cdev->private->flags.recog_done = 1; + wake_up(&cdev->private->wait_q); + return; + } switch (state) { case DEV_STATE_NOT_OPER: break; diff --git a/drivers/s390/cio/io_sch.h b/drivers/s390/cio/io_sch.h index 85a11c1836e5..c03b4a19974e 100644 --- a/drivers/s390/cio/io_sch.h +++ b/drivers/s390/cio/io_sch.h @@ -160,6 +160,7 @@ struct ccw_device_private { unsigned int donotify:1; /* call notify function */ unsigned int recog_done:1; /* dev. recog. complete */ unsigned int fake_irb:2; /* deliver faked irb */ + unsigned int resuming:1; /* recognition while resume */ unsigned int pgroup:1; /* pathgroup is set up */ unsigned int mpath:1; /* multipathing is set up */ unsigned int pgid_unknown:1;/* unknown pgid state */