[SRU,Pull,Bionic] Fix deadlock in ceph

BugLink: https://bugs.launchpad.net/bugs/1834235

[Impact]

Deadlock may occur if iput_final() decides to wait for readahead pages 
while a lock is held.

In order to resolve this, the following two patches install an 
asynchronous "iput" for the ceph inodes so that a hold-and-wait deadlock 
doesn't occur. A more detailed example is shown in the original patch: 
https://github.com/ceph/ceph-client/commit/093ea205acd4b047cf5aacabc0c6ffecf198d2a9

Requested patches:

3e1d0452edcee ceph: avoid iput_final() while holding mutex or in dispatch thread
1cf89a8dee5e6 ceph: single workqueue for inode related works

[Test Case]

These changes were tested by the original requester with positive results over a few
days in their own environment where they first experienced the regression. They have
determined they are no longer experiencing the regression with this patch set applied
to a test kernel.

[Regression Potential] 

Several patches were required in order to cleanly cherry pick the requested patches.
A large number of changes increases the regression potential, however, these 
pre-requisite patches have been in mainline since early 2018 and the blast radius
is localized only to ceph.

----------------------------------------------------------------

The following changes since commit bb936a8a0dbbc727533e953071862a228044fa9f:

  block/bio: Do not zero user pages (2019-06-28 18:27:34 +0200)

are available in the Git repository at:

  git+ssh://connork@git.launchpad.net/~connork/+git/bionic ceph

for you to fetch changes up to 4a7619c37800ae7433f2bd5e2d66ce4a3c60d23e:

  ceph: avoid iput_final() while holding mutex or in dispatch thread (2019-06-28 09:32:06 -0700)

----------------------------------------------------------------
Luis Henriques (6):
      ceph: quota: add initial infrastructure to support cephfs quotas
      ceph: quota: support for ceph.quota.max_files
      ceph: quota: don't allow cross-quota renames
      ceph: quota: support for ceph.quota.max_bytes
      ceph: quota: update MDS when max_bytes is approaching
      ceph: quota: add counter for snaprealms with quota

Yan, Zheng (8):
      ceph: use atomic_t for ceph_inode_info::i_shared_gen
      ceph: define argument structure for handle_cap_grant
      ceph: flush pending works before shutdown super
      ceph: send cap releases more aggressively
      ceph: single workqueue for inode related works
      ceph: avoid dereferencing invalid pointer during cached readdir
      ceph: fix root quota realm check
      ceph: avoid iput_final() while holding mutex or in dispatch thread

 Documentation/filesystems/ceph.txt |  12 ++
 fs/ceph/Makefile                   |   2 +-
 fs/ceph/caps.c                     | 154 ++++++++++---------
 fs/ceph/dir.c                      |  75 +++++++---
 fs/ceph/file.c                     |  23 ++-
 fs/ceph/inode.c                    | 211 ++++++++++++++++----------
 fs/ceph/mds_client.c               | 113 +++++++++++---
 fs/ceph/mds_client.h               |  14 +-
 fs/ceph/quota.c                    | 298 +++++++++++++++++++++++++++++++++++++
 fs/ceph/snap.c                     |  16 +-
 fs/ceph/super.c                    |  38 ++---
 fs/ceph/super.h                    |  62 ++++++--
 fs/ceph/xattr.c                    |  44 ++++++
 include/linux/ceph/ceph_features.h |   1 +
 include/linux/ceph/ceph_fs.h       |  17 +++
 net/ceph/ceph_common.c             |   1 +
 16 files changed, 847 insertions(+), 234 deletions(-)
 create mode 100644 fs/ceph/quota.c

On 6/28/19 8:09 PM, Connor Kuehl wrote:
> BugLink: https://bugs.launchpad.net/bugs/1834235
> 
> [Impact]
> 
> Deadlock may occur if iput_final() decides to wait for readahead pages 
> while a lock is held.
> 
> In order to resolve this, the following two patches install an 
> asynchronous "iput" for the ceph inodes so that a hold-and-wait deadlock 
> doesn't occur. A more detailed example is shown in the original patch: 
> https://github.com/ceph/ceph-client/commit/093ea205acd4b047cf5aacabc0c6ffecf198d2a9
> 
> Requested patches:
> 
> 3e1d0452edcee ceph: avoid iput_final() while holding mutex or in dispatch thread
> 1cf89a8dee5e6 ceph: single workqueue for inode related works
> 
> [Test Case]
> 
> These changes were tested by the original requester with positive results over a few
> days in their own environment where they first experienced the regression. They have
> determined they are no longer experiencing the regression with this patch set applied
> to a test kernel.
> 
> [Regression Potential] 
> 
> Several patches were required in order to cleanly cherry pick the requested patches.
> A large number of changes increases the regression potential, however, these 
> pre-requisite patches have been in mainline since early 2018 and the blast radius
> is localized only to ceph.

Hard to review due to the number of patches, but clean cherry-picks
of commits that have been around for a while and tested to fix
the issue reported.


Acked-by: Kleber Sacilotto de Souza <kleber.souza@canonical.com>

Thank you,
Kleber

> 
> ----------------------------------------------------------------
> 
> The following changes since commit bb936a8a0dbbc727533e953071862a228044fa9f:
> 
>   block/bio: Do not zero user pages (2019-06-28 18:27:34 +0200)
> 
> are available in the Git repository at:
> 
>   git+ssh://connork@git.launchpad.net/~connork/+git/bionic ceph
> 
> for you to fetch changes up to 4a7619c37800ae7433f2bd5e2d66ce4a3c60d23e:
> 
>   ceph: avoid iput_final() while holding mutex or in dispatch thread (2019-06-28 09:32:06 -0700)
> 
> ----------------------------------------------------------------
> Luis Henriques (6):
>       ceph: quota: add initial infrastructure to support cephfs quotas
>       ceph: quota: support for ceph.quota.max_files
>       ceph: quota: don't allow cross-quota renames
>       ceph: quota: support for ceph.quota.max_bytes
>       ceph: quota: update MDS when max_bytes is approaching
>       ceph: quota: add counter for snaprealms with quota
> 
> Yan, Zheng (8):
>       ceph: use atomic_t for ceph_inode_info::i_shared_gen
>       ceph: define argument structure for handle_cap_grant
>       ceph: flush pending works before shutdown super
>       ceph: send cap releases more aggressively
>       ceph: single workqueue for inode related works
>       ceph: avoid dereferencing invalid pointer during cached readdir
>       ceph: fix root quota realm check
>       ceph: avoid iput_final() while holding mutex or in dispatch thread
> 
>  Documentation/filesystems/ceph.txt |  12 ++
>  fs/ceph/Makefile                   |   2 +-
>  fs/ceph/caps.c                     | 154 ++++++++++---------
>  fs/ceph/dir.c                      |  75 +++++++---
>  fs/ceph/file.c                     |  23 ++-
>  fs/ceph/inode.c                    | 211 ++++++++++++++++----------
>  fs/ceph/mds_client.c               | 113 +++++++++++---
>  fs/ceph/mds_client.h               |  14 +-
>  fs/ceph/quota.c                    | 298 +++++++++++++++++++++++++++++++++++++
>  fs/ceph/snap.c                     |  16 +-
>  fs/ceph/super.c                    |  38 ++---
>  fs/ceph/super.h                    |  62 ++++++--
>  fs/ceph/xattr.c                    |  44 ++++++
>  include/linux/ceph/ceph_features.h |   1 +
>  include/linux/ceph/ceph_fs.h       |  17 +++
>  net/ceph/ceph_common.c             |   1 +
>  16 files changed, 847 insertions(+), 234 deletions(-)
>  create mode 100644 fs/ceph/quota.c
> 
>

On 28.06.19 20:09, Connor Kuehl wrote:
>  git+ssh://connork@git.launchpad.net/~connork/+git/bionic ceph

This is more that normally sounds safe to pull into a stable update but as its
claimed to be tested well and isolated to that area...

Acked-by: Stefan Bader <stefan.bader@canonical.com>

On 6/28/19 8:09 PM, Connor Kuehl wrote:
> BugLink: https://bugs.launchpad.net/bugs/1834235
> 
> [Impact]
> 
> Deadlock may occur if iput_final() decides to wait for readahead pages 
> while a lock is held.
> 
> In order to resolve this, the following two patches install an 
> asynchronous "iput" for the ceph inodes so that a hold-and-wait deadlock 
> doesn't occur. A more detailed example is shown in the original patch: 
> https://github.com/ceph/ceph-client/commit/093ea205acd4b047cf5aacabc0c6ffecf198d2a9
> 
> Requested patches:
> 
> 3e1d0452edcee ceph: avoid iput_final() while holding mutex or in dispatch thread
> 1cf89a8dee5e6 ceph: single workqueue for inode related works
> 
> [Test Case]
> 
> These changes were tested by the original requester with positive results over a few
> days in their own environment where they first experienced the regression. They have
> determined they are no longer experiencing the regression with this patch set applied
> to a test kernel.
> 
> [Regression Potential] 
> 
> Several patches were required in order to cleanly cherry pick the requested patches.
> A large number of changes increases the regression potential, however, these 
> pre-requisite patches have been in mainline since early 2018 and the blast radius
> is localized only to ceph.
> 
> ----------------------------------------------------------------
> 
> The following changes since commit bb936a8a0dbbc727533e953071862a228044fa9f:
> 
>   block/bio: Do not zero user pages (2019-06-28 18:27:34 +0200)
> 
> are available in the Git repository at:
> 
>   git+ssh://connork@git.launchpad.net/~connork/+git/bionic ceph
> 
> for you to fetch changes up to 4a7619c37800ae7433f2bd5e2d66ce4a3c60d23e:
> 
>   ceph: avoid iput_final() while holding mutex or in dispatch thread (2019-06-28 09:32:06 -0700)

Applied to bionic/master-next branch.

Thanks,
Kleber

> 
> ----------------------------------------------------------------
> Luis Henriques (6):
>       ceph: quota: add initial infrastructure to support cephfs quotas
>       ceph: quota: support for ceph.quota.max_files
>       ceph: quota: don't allow cross-quota renames
>       ceph: quota: support for ceph.quota.max_bytes
>       ceph: quota: update MDS when max_bytes is approaching
>       ceph: quota: add counter for snaprealms with quota
> 
> Yan, Zheng (8):
>       ceph: use atomic_t for ceph_inode_info::i_shared_gen
>       ceph: define argument structure for handle_cap_grant
>       ceph: flush pending works before shutdown super
>       ceph: send cap releases more aggressively
>       ceph: single workqueue for inode related works
>       ceph: avoid dereferencing invalid pointer during cached readdir
>       ceph: fix root quota realm check
>       ceph: avoid iput_final() while holding mutex or in dispatch thread
> 
>  Documentation/filesystems/ceph.txt |  12 ++
>  fs/ceph/Makefile                   |   2 +-
>  fs/ceph/caps.c                     | 154 ++++++++++---------
>  fs/ceph/dir.c                      |  75 +++++++---
>  fs/ceph/file.c                     |  23 ++-
>  fs/ceph/inode.c                    | 211 ++++++++++++++++----------
>  fs/ceph/mds_client.c               | 113 +++++++++++---
>  fs/ceph/mds_client.h               |  14 +-
>  fs/ceph/quota.c                    | 298 +++++++++++++++++++++++++++++++++++++
>  fs/ceph/snap.c                     |  16 +-
>  fs/ceph/super.c                    |  38 ++---
>  fs/ceph/super.h                    |  62 ++++++--
>  fs/ceph/xattr.c                    |  44 ++++++
>  include/linux/ceph/ceph_features.h |   1 +
>  include/linux/ceph/ceph_fs.h       |  17 +++
>  net/ceph/ceph_common.c             |   1 +
>  16 files changed, 847 insertions(+), 234 deletions(-)
>  create mode 100644 fs/ceph/quota.c
> 
>