From patchwork Wed Mar 21 11:10:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 888702 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 405nPt0JSYz9s0n for ; Wed, 21 Mar 2018 22:19:30 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="nx4FdlCz"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 405nPs5GbLzF1b3 for ; Wed, 21 Mar 2018 22:19:29 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="nx4FdlCz"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:4002:c09::241; helo=mail-yb0-x241.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="nx4FdlCz"; dkim-atps=neutral Received: from mail-yb0-x241.google.com (mail-yb0-x241.google.com [IPv6:2607:f8b0:4002:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 405nD65lVgzDrFR for ; Wed, 21 Mar 2018 22:11:01 +1100 (AEDT) Received: by mail-yb0-x241.google.com with SMTP id i13-v6so1569021ybl.9 for ; Wed, 21 Mar 2018 04:11:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=O3X4etx49cZQ7IaCBhfWC+i2pkx9C/dE/bjl2rWmf30=; b=nx4FdlCzZumNNGw7qGcwRHanrhHyokXxsCDJ+wmZEzUcTVW5/Y1CNXP5Xt/5yGkOqb DPjj1kB55sf1UQVuZBh12fu5LSXzMD4sMdRZDmnWWnoj4vR5/2MqK1tQuNJ/c8x+Q0Ok EHEytpU49SmhxX8y0/letWViEbvikW1tnz9vw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=O3X4etx49cZQ7IaCBhfWC+i2pkx9C/dE/bjl2rWmf30=; b=gVvAgDoWDHL2O95BESpEESUJ4u5TCGNdrIEtG0KmU08zKnMq8ZARZ+YkyffHVIjbgR W+ucMxMZOFvKcP142W5h5czqfH1NDM0GHM9sZxFP4g74vCX/i24foGcHSWcOgG5KfzUr 1evtLq08cMgEjrHG1Yv7gjtDyhAqYa8TSB8oV8CTicfLE0LxmwXd5sSPOxyAa7kAfwPk vsfATIROFMUqAuTbvp/AiDvF2AT2nIZFdjf8+yTCzQ7r+F5uPdZVKE+72SEId28RsJil bQ/ZeDpo6XcYoRGSZmCSiUD08V/2Os4sLA/ptmNN9Dfmjtl2hS3oSlr4H5dikHy7o8CZ iO0w== X-Gm-Message-State: AElRT7GQ238aSH3VHvkR+WyWfkie63t+/cARldwLmt9RHKX3Cr1kjf46 ybK6A68S3ftZpvTH8MkvmCziwepOkWg= X-Google-Smtp-Source: AG47ELuI1vzhZXe9mZEpIYIEg1VfjDGeb6FV4EwTanoi2PZYhdeKt2hgpqzI+2lYFxi1QbLuOIwVZw== X-Received: by 2002:a25:62cc:: with SMTP id w195-v6mr8621469ybb.342.1521630658114; Wed, 21 Mar 2018 04:10:58 -0700 (PDT) Received: from linkitivity.nonius.hsia ([88.157.201.114]) by smtp.gmail.com with ESMTPSA id g3sm1456400ywk.24.2018.03.21.04.10.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Mar 2018 04:10:57 -0700 (PDT) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH v2 2/2] api: Only provide JSON version of events list Date: Wed, 21 Mar 2018 22:10:33 +1100 Message-Id: <20180321111033.15385-2-dja@axtens.net> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180321111033.15385-1-dja@axtens.net> References: <20180321111033.15385-1-dja@axtens.net> X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" Something is very, very slow in the d-r-f browsable API events renderer. In my MySQL test (~33k patches), the CPU time to render the events list is ~11s, and the time taken by SQL queries is only ~3s. If the JSON renderer is used, that drops to 0.2s for the entire page (because less CPU is used, and - for some as yet unknown reason - a *very* expensive db query is dropped.) In my PostgreSQL test (~100k patches), the results are even more stark: 30s of CPU time and 0.2s of DB time goes to 0.25s for the entire page. Something is seriously, seriously wrong with whatever d-r-f is doing. So, simply render the event list as HTML-ised, unlinked JSON for now. There are a few followups we should do, but this is an important start - no-one should be able to DoS a patchwork server by just enumerating the events! In particular, we should find out: - why postgres and mysql behaviour is so different. - what on earth d-r-f is doing that makes rendering the pretty-printed version so incredibly slow. Signed-off-by: Daniel Axtens --- v2: Make it a bit nicer than just pure json. --- patchwork/api/event.py | 15 +++++++++++++++ patchwork/templates/patchwork/event-list.html | 17 +++++++++++++++++ 2 files changed, 32 insertions(+) create mode 100644 patchwork/templates/patchwork/event-list.html diff --git a/patchwork/api/event.py b/patchwork/api/event.py index 7e04b716af1a..9879a9f670a8 100644 --- a/patchwork/api/event.py +++ b/patchwork/api/event.py @@ -18,10 +18,13 @@ # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA from collections import OrderedDict +import json from rest_framework.generics import ListAPIView from rest_framework.serializers import ModelSerializer from rest_framework.serializers import SerializerMethodField +from rest_framework.renderers import JSONRenderer +from rest_framework.renderers import TemplateHTMLRenderer from patchwork.api.embedded import CheckSerializer from patchwork.api.embedded import CoverLetterSerializer @@ -85,9 +88,21 @@ class EventSerializer(ModelSerializer): read_only_fields = fields +# The standard template html renderer is broken: +# https://github.com/encode/django-rest-framework/issues/5236 +class JSONListHTMLRenderer(TemplateHTMLRenderer): + def get_template_context(self, data, renderer_context): + response = renderer_context['response'] + if response.exception: + data['status_code'] = response.status_code + return {'data': json.dumps(data, indent=4)} + + class EventList(ListAPIView): """List events.""" + renderer_classes = (JSONRenderer, JSONListHTMLRenderer) + template_name = 'patchwork/event-list.html' serializer_class = EventSerializer filter_class = EventFilter page_size_query_param = None # fixed page size diff --git a/patchwork/templates/patchwork/event-list.html b/patchwork/templates/patchwork/event-list.html new file mode 100644 index 000000000000..821c6897388e --- /dev/null +++ b/patchwork/templates/patchwork/event-list.html @@ -0,0 +1,17 @@ +{% extends "base.html" %} + +{% load person %} +{% load static %} + +{% block title %}Event List{% endblock %} +{% block patch_active %}active{% endblock %} + +{% block body %} + +

Due to a currently undiagnosed issue with django-rest-framework, the browsable API is very CPU intensive and has been disabled. The JSON output is:

+ +
+{{data}}
+
+ +{% endblock %}