From patchwork Sun Jan 7 17:08:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Beier X-Patchwork-Id: 856571 X-Patchwork-Delegate: blogic@openwrt.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.infradead.org (client-ip=65.50.211.133; helo=bombadil.infradead.org; envelope-from=lede-dev-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="dicJI51E"; dkim-atps=neutral Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zF4dN3FpTz9s7n for ; Mon, 8 Jan 2018 04:09:28 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Subject:References: In-Reply-To:Message-Id:Date:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=2vYm61s+R7dkU1fETeNKjb3RQQVT5yCa+py8TIcIx5M=; b=dicJI51El5snBV LGJdwhtsgr5lyXbu6yu6t0+qZndyhDB0lpr8+rYMzOSqSMfTjfwgeI2Mo8/533ulZi5qCDV/I/Kui DqSn3g56igY5v5IMCdcL6g/u902An3Xjrrj2sT/ufBRnzPX1NVStxG5dCGUADiB0T3S8Ftp56tsA4 a4MdGYrp+ZT4dvEpb9t9WPY0bBnIkWGrpIJ8JDAQd6e4Uq7ZoMFGTdZLUu2BnZQcKT9nWmjmVv14g AyblAj+fax6r0oY4e2PNEy0mBcydJDJ3A6xHhLH2yzB7NfAOjz8hie1aHpBcVgStnRcULEN5cXzY5 iODJyVng9JDnet8korqw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.89 #1 (Red Hat Linux)) id 1eYERj-0007Z8-TR; Sun, 07 Jan 2018 17:09:19 +0000 Received: from mail-wm0-f68.google.com ([74.125.82.68]) by bombadil.infradead.org with esmtps (Exim 4.89 #1 (Red Hat Linux)) id 1eYERg-0007Uz-D5 for lede-dev@lists.infradead.org; Sun, 07 Jan 2018 17:09:18 +0000 Received: by mail-wm0-f68.google.com with SMTP id 64so10326459wme.3 for ; Sun, 07 Jan 2018 09:09:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GTLWU17Ow+kU+hUoLYWWoRRRNy259puQ/GYaeS9TsDU=; b=nOLwcKUj+gjDiB9vhE1sEUmyM0r8knmS6qaHZqmjzbM2lrSIEf4yNbemypLpDri6Ep DoMG3ZNlrNeLZJnJ8w+j+cHmYRFICZvlFV/rcjeef5qtcCIoFvOzTSX35mue5qRlU9UP QDZgo3pN0NeSseSxzp4JFcRZt78hWIETFFzn14VlPuOf7ylOylLTnMUxXdjj4jjmwo7y uOl3d2eDMqbX49O8Xuz8VY7cQ6/jinKt3n7nsbtCTRZ+pWkNg3lbZQvevlkBSaw4r+Ek 4GkOBYmREKv3PrhoSiyjzVpGpc2qwd0hSZtPkyEvHkafhCAtSalLRI6DwPtfuwt8QZ9r WovA== X-Gm-Message-State: AKwxytcBJzymnx2bX5oLos3ATvP+yfH7cU+fZPjVM0iaKz5jtPgL62wn pCGQ6XdO0KHBtBTE754RFMQoVw== X-Google-Smtp-Source: ACJfBosizFB1sl7jaW34WXKG1LUVP0fmxAXsVYexX2xmz9wrbseBMhXzH6ICrFZhU1+F7GtN5/QBYg== X-Received: by 10.28.214.68 with SMTP id n65mr6307789wmg.18.1515344942776; Sun, 07 Jan 2018 09:09:02 -0800 (PST) Received: from denkpad (ip5f5bd6ef.dynamic.kabel-deutschland.de. [95.91.214.239]) by smtp.gmail.com with ESMTPSA id u54sm10983854wrf.77.2018.01.07.09.09.01 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 07 Jan 2018 09:09:01 -0800 (PST) Received: from bk by denkpad with local (Exim 4.89) (envelope-from ) id 1eYERQ-0008Ox-Qr; Sun, 07 Jan 2018 18:09:00 +0100 From: Christian Beier To: lede-dev@lists.infradead.org Date: Sun, 7 Jan 2018 18:08:09 +0100 Message-Id: <20180107170809.31401-2-dontmind@freeshell.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180107170809.31401-1-dontmind@freeshell.org> References: <3a7b3048-1930-c3fd-de35-ec48841fad5c@phrozen.org> <20180107170809.31401-1-dontmind@freeshell.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180107_090916_458817_709C4BA3 X-CRM114-Status: GOOD ( 12.42 ) X-Spam-Score: -1.4 (-) X-Spam-Report: SpamAssassin version 3.4.1 on bombadil.infradead.org summary: Content analysis details: (-1.4 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [74.125.82.68 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (christian.cb.beier[at]googlemail.com) 0.2 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level mail domains are different -0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) [74.125.82.68 listed in wl.mailspike.net] -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 RCVD_IN_MSPIKE_WL Mailspike good senders 0.2 FREEMAIL_FORGED_FROMDOMAIN 2nd level domains in From and EnvelopeFrom freemail headers are different Subject: [LEDE-DEV] [PATCH] jshn: add functionality to read big JSON X-BeenThere: lede-dev@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Christian Beier MIME-Version: 1.0 Sender: "Lede-dev" Errors-To: lede-dev-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org The existing read functionality feeds the complete JSON to jshn as a cmdline argument, leading to `-ash: jshn: Argument list too long` errors for JSONs bigger than ca. 100KB. This commit adds the ability to read the JSON directly from a file if wanted, removing this shell-imposed size limit. Tested on x86-64 and ar71xx. An mmap()-based solution was also evaluated, but found to make no performance difference on either platform. Signed-off-by: Christian Beier --- jshn.c | 30 ++++++++++++++++++++++++++++-- sh/jshn.sh | 4 ++++ 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/jshn.c b/jshn.c index 3188af5..eb72fb7 100644 --- a/jshn.c +++ b/jshn.c @@ -25,6 +25,8 @@ #include #include #include +#include +#include #include "list.h" #include "avl.h" @@ -305,7 +307,7 @@ out: static int usage(const char *progname) { - fprintf(stderr, "Usage: %s [-n] [-i] -r |-w\n", progname); + fprintf(stderr, "Usage: %s [-n] [-i] -r |-R |-w\n", progname); return 2; } @@ -338,6 +340,10 @@ int main(int argc, char **argv) struct env_var *vars; int i; int ch; + int fd; + struct stat sb; + char *fbuf; + int ret; avl_init(&env_vars, avl_strcmp_var, false, NULL); for (i = 0; environ[i]; i++); @@ -359,7 +365,7 @@ int main(int argc, char **argv) avl_insert(&env_vars, &vars[i].avl); } - while ((ch = getopt(argc, argv, "p:nir:w")) != -1) { + while ((ch = getopt(argc, argv, "p:nir:R:w")) != -1) { switch(ch) { case 'p': var_prefix = optarg; @@ -367,6 +373,26 @@ int main(int argc, char **argv) break; case 'r': return jshn_parse(optarg); + case 'R': + if ((fd = open(optarg, O_RDONLY)) == -1) { + fprintf(stderr, "Error opening %s\n", optarg); + return 3; + } + if (fstat(fd, &sb) == -1) { + fprintf(stderr, "Error getting size of %s\n", optarg); + close(fd); + return 3; + } + if (!(fbuf = malloc(sb.st_size)) || read(fd, fbuf, sb.st_size) != sb.st_size) { + fprintf(stderr, "Error reading %s\n", optarg); + free(fbuf); + close(fd); + return 3; + } + ret = jshn_parse(fbuf); + free(fbuf); + close(fd); + return ret; case 'w': return jshn_format(no_newline, indent); case 'n': diff --git a/sh/jshn.sh b/sh/jshn.sh index 1090814..66baccb 100644 --- a/sh/jshn.sh +++ b/sh/jshn.sh @@ -180,6 +180,10 @@ json_load() { eval "`jshn -r "$1"`" } +json_load_file() { + eval "`jshn -R "$1"`" +} + json_dump() { jshn "$@" ${JSON_PREFIX:+-p "$JSON_PREFIX"} -w }