From patchwork Thu Jan 18 20:21:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Beier X-Patchwork-Id: 863129 X-Patchwork-Delegate: blogic@openwrt.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.infradead.org (client-ip=65.50.211.133; helo=bombadil.infradead.org; envelope-from=lede-dev-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="SWoJOLXm"; dkim-atps=neutral Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zMwNm1hxCz9sDB for ; Fri, 19 Jan 2018 07:22:16 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Subject:References: In-Reply-To:Message-Id:Date:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=poRix4Yu/PJ4PZuR87/fAWxFkW8sEK6fZJpE1z3sfgc=; b=SWoJOLXm4kxZLD aljsTOqm3tUQug6Eu0CkphsRSVVWjdMYddaVOPdWvqPrJOWrYrI4GVQ6cLapzWaRM3CHe4DdBd5hZ 20jrmsMEQyxofZgZJqds9T8NMXCznJ0IIqCpYHtvNmhzfJUf1GXLnufYI9EgV2pIMCBNCggO0TE+e njwG8MF1++YO+CTCqwVnyABLpw/fk0ykypzls3JBmKE3fBJthP67VY4SMWSKL0t0y8qA1B5Qbge6s bnZ3NBhKQFJP0UmSNwHboTzKtux5lvRI8bE1uzd/cvBvMrVTFQwKVBnPPBz1bAaOvPE3TsQZ8KAq7 E+Jak8uaOkOTN4pL6mDA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.89 #1 (Red Hat Linux)) id 1ecGhR-0003iI-MN; Thu, 18 Jan 2018 20:22:13 +0000 Received: from mail-wm0-f67.google.com ([74.125.82.67]) by bombadil.infradead.org with esmtps (Exim 4.89 #1 (Red Hat Linux)) id 1ecGhM-0003WR-PA for lede-dev@lists.infradead.org; Thu, 18 Jan 2018 20:22:11 +0000 Received: by mail-wm0-f67.google.com with SMTP id f71so24330823wmf.0 for ; Thu, 18 Jan 2018 12:21:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mfyBkJfQvuYb4dqiyHOYit+Ebu1uqIQNPgMFvn0lxgY=; b=pjs4UVRJ7H5YUiH3yEujp6BTM1+uKCssvGbns+TZBsk/3Fw+X5tO5nWKFdpjjScsJV nPwWzGkS9giUvaBnM4R2csvmQjSJ0pwq8T/XxERjBY7BqQxbRjApwL4Rjq7o3HQ52CMS 5Zkspchy424sift94cpHFKT7Py2mho+76pjgxnf2mb0UhE1Ba0elVnbOLW019fa0Zykc w0XhL1fOpTbL7qH+2INn+TUS22k0wAvWKHjTNA+p4K25JxaW1J4QQx9E1/Ls7cUOr6PK uB1WBoKOxcLvVG5qdUsoMnJ37JzrqETExY22DubhhS3AD4UJOXkuulre1CfCKvHuuF+K avIQ== X-Gm-Message-State: AKwxytcPSXqQNvmnzCsoebWG+3jqp+eottb5OAtJrHasEjVQXTZNPMpR rfrvrIhi7o8qNTtLWkXwnXo= X-Google-Smtp-Source: ACJfBou1M10zSjuRQUoSUUWRV0+4V/9ykWiWr1rbpXduKAvJMgQqmi3mo6p4kmi18B00h9JPMac7iw== X-Received: by 10.80.145.115 with SMTP id f48mr9475835eda.92.1516306916824; Thu, 18 Jan 2018 12:21:56 -0800 (PST) Received: from denkpad ([178.19.214.170]) by smtp.gmail.com with ESMTPSA id s33sm4899945edd.10.2018.01.18.12.21.55 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 18 Jan 2018 12:21:56 -0800 (PST) Received: from bk by denkpad with local (Exim 4.89) (envelope-from ) id 1ecGh9-00030p-Ix; Thu, 18 Jan 2018 21:21:55 +0100 From: Christian Beier To: john@phrozen.org, lede-dev@lists.infradead.org Date: Thu, 18 Jan 2018 21:21:09 +0100 Message-Id: <20180118202109.11459-2-dontmind@freeshell.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180118202109.11459-1-dontmind@freeshell.org> References: <63befaf5-77e6-17d4-1ad6-683d9fd65e59@phrozen.org> <20180118202109.11459-1-dontmind@freeshell.org> X-Spam-Note: CRM114 invocation failed X-Spam-Score: -1.4 (-) X-Spam-Report: SpamAssassin version 3.4.1 on bombadil.infradead.org summary: Content analysis details: (-1.4 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [74.125.82.67 listed in list.dnswl.org] -0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) [74.125.82.67 listed in wl.mailspike.net] 0.2 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level mail domains are different -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (christian.cb.beier[at]googlemail.com) -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 RCVD_IN_MSPIKE_WL Mailspike good senders 0.2 FREEMAIL_FORGED_FROMDOMAIN 2nd level domains in From and EnvelopeFrom freemail headers are different Subject: [LEDE-DEV] [PATCH v3] jshn: add functionality to read big JSON X-BeenThere: lede-dev@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Christian Beier MIME-Version: 1.0 Sender: "Lede-dev" Errors-To: lede-dev-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org The existing read functionality feeds the complete JSON to jshn as a cmdline argument, leading to `-ash: jshn: Argument list too long` errors for JSONs bigger than ca. 100KB. This commit adds the ability to read the JSON directly from a file if wanted, removing this shell-imposed size limit. Tested on x86-64 and ar71xx. An mmap()-based solution was also evaluated, but found to make no performance difference on either platform. Signed-off-by: Christian Beier --- jshn.c | 35 +++++++++++++++++++++++++++++++++-- sh/jshn.sh | 4 ++++ 2 files changed, 37 insertions(+), 2 deletions(-) diff --git a/jshn.c b/jshn.c index 3188af5..7f67746 100644 --- a/jshn.c +++ b/jshn.c @@ -25,6 +25,8 @@ #include #include #include +#include +#include #include "list.h" #include "avl.h" @@ -305,7 +307,7 @@ out: static int usage(const char *progname) { - fprintf(stderr, "Usage: %s [-n] [-i] -r |-w\n", progname); + fprintf(stderr, "Usage: %s [-n] [-i] -r |-R |-w\n", progname); return 2; } @@ -338,6 +340,10 @@ int main(int argc, char **argv) struct env_var *vars; int i; int ch; + int fd; + struct stat sb; + char *fbuf; + int ret; avl_init(&env_vars, avl_strcmp_var, false, NULL); for (i = 0; environ[i]; i++); @@ -359,7 +365,7 @@ int main(int argc, char **argv) avl_insert(&env_vars, &vars[i].avl); } - while ((ch = getopt(argc, argv, "p:nir:w")) != -1) { + while ((ch = getopt(argc, argv, "p:nir:R:w")) != -1) { switch(ch) { case 'p': var_prefix = optarg; @@ -367,6 +373,31 @@ int main(int argc, char **argv) break; case 'r': return jshn_parse(optarg); + case 'R': + if ((fd = open(optarg, O_RDONLY)) == -1) { + fprintf(stderr, "Error opening %s\n", optarg); + return 3; + } + if (fstat(fd, &sb) == -1) { + fprintf(stderr, "Error getting size of %s\n", optarg); + close(fd); + return 3; + } + if (!(fbuf = malloc(sb.st_size))) { + fprintf(stderr, "Error allocating memory for %s\n", optarg); + close(fd); + return 3; + } + if (read(fd, fbuf, sb.st_size) != sb.st_size) { + fprintf(stderr, "Error reading %s\n", optarg); + free(fbuf); + close(fd); + return 3; + } + ret = jshn_parse(fbuf); + free(fbuf); + close(fd); + return ret; case 'w': return jshn_format(no_newline, indent); case 'n': diff --git a/sh/jshn.sh b/sh/jshn.sh index 1090814..66baccb 100644 --- a/sh/jshn.sh +++ b/sh/jshn.sh @@ -180,6 +180,10 @@ json_load() { eval "`jshn -r "$1"`" } +json_load_file() { + eval "`jshn -R "$1"`" +} + json_dump() { jshn "$@" ${JSON_PREFIX:+-p "$JSON_PREFIX"} -w }